Extracted Python-Related Data Engineering Interview Questions

Below is a comprehensive list of Python-related questions and topics from the provided
interview guides and resources. These questions cover general Python programming, data
manipulation, ETL, PySpark, and automation as relevant to data engineering roles.

General Python Programming and Coding

Which scripting language are you most comfortable with?
How would you check if a given string is a palindrome?
Write a program to count the number of vowels in a string.
Could you walk me through the logic behind your code?
What's the most challenging Python problem you've tackled so far? Can you write that code
for me? [1]
Function to find the top 3 largest numbers in a list.
def top_3_largest(numbers):
return sorted(numbers, reverse=True)[:3]

Implement a Python function to count unique words from a file and write them to another
file.
Write a decorator function to log the execution time of a function.
Create a Python program to demonstrate the use of set operations (union, intersection).
Implement file handling in Python to read a CSV and store only specific columns in a
dictionary.
Explain the difference between mutable and immutable objects in Python. [2] [3]
How would you handle an exception in Python? Provide an example.
What are lambda functions in Python? How are they different from regular functions?
How would you iterate over a dictionary in Python and print its keys and values?
Explain the concept of generators in Python. Provide an example of a generator function.
How would you sort a list of dictionaries based on a specific key in Python?
What is the difference between shallow copy and deep copy in Python? When would you
use each?
How can you read data from a CSV file in Python? Provide an example.
Explain the concept of object-oriented programming (OOP) in Python. Give an example of a
class and its usage.
 How would you handle memory management in Python? What is the purpose of garbage
collection? [3]

Python for Data Engineering & ETL

What are the different ways to read a CSV file in Python?
How do you interact with Google BigQuery using Python?
How can you automate data insertion into BigQuery using Python? [4]
Write a Python script to process raw JSON files containing sales data and load them into a
relational database.
Describe how you would debug a failing ETL pipeline in production.
How would you handle duplicate or corrupted data in a batch ETL job?
Create a function to detect anomalies in sales trends using Pandas and NumPy.
Write a Python function to merge and deduplicate two sorted lists of sales data.
How would you build a reusable ETL framework using Airflow?
Explain how to implement schema validation for incoming data streams.
Describe how you would monitor ETL job performance and handle long-running tasks. [5] [6]
Handling data validation using SQL or Python.
How do you handle missing data in a DataFrame in Python?
Can you explain the concept of a data pipeline and how you would build one in Python? [7]

PySpark and Spark with Python

Managing schema changes in PySpark over time.
Why is RDD considered resilient and fault-tolerant?
Lazy evaluation in Spark and its impact on performance.
Difference between persist() and cache() in Spark.
Difference between reduceByKey() and groupByKey().
DataFrames vs. RDDs in PySpark.
What are the key differences between DataFrames and RDDs in PySpark?
How do you manage schema changes in PySpark when processing data over time?
Write PySpark code to filter and count records.
Write PySpark code to filter records based on specific conditions and add a calculated
column.
Write a PySpark script to filter out invalid records from a dataset and calculate the average
for a specific column, ensuring the schema is strictly defined at runtime. [8] [9] [10]
Automation, Data Pipelines, and Airflow
How would you automate a data pipeline deployment using GitHub Actions or another CI/CD
tool?
Explain how to schedule an automated task using Apache Airflow.
How would you build a reusable ETL framework using Airflow? [11]

Python Data Structures & Algorithms

Data Structures: List, Set, Tuple, Dictionary, String.
Write a Python script to merge two sorted lists.
Implement a function to find duplicate records in a large dataset using Python.
Create a script to parse and transform a JSON file into a structured CSV.
Merge two dictionaries and remove keys with null values. [2] [11]
Odd Number Sorting: Write a function to sort an array, returning only odd numbers.
Unique Values Preservation: Find non-duplicate numbers from a list while preserving the
original order.
Maximum Occurrences: Given a list, return the numbers with the highest count.
JSON Flattening: Write a function to flatten nested JSON objects into a single key-value
dictionary.
Array Pair Sum: Write code to find two numbers in an array that sum up to x.
Stack Implementation: Implement a stack using a linked list. [12]

Python in Data Engineering Contexts

Use libraries like requests or urllib in Python for API data ingestion, then transform and load
it into the target system. [13]
Handling data validation and schema management using Python.
Using Pandas and Numpy for data preprocessing and transformation.
Handling null values in a single column using fillna or replace in PySpark:
df.fillna({'column_name': 'value'}).show()

Moving files in Databricks using dbutils in Python:

dbutils.fs.mv('/source/path', '/destination/path')

Scheduling jobs in Databricks and defining tasks using Python scripts. [13]
Sample Scenario-Based and Conceptual Questions
How would you handle missing data in a DataFrame in Python?
How do you handle duplicate or corrupted data in a batch ETL job?
How would you debug a failing ETL pipeline in production?
How do you automate data insertion into BigQuery using Python?
How do you interact with Google BigQuery using Python?
How would you merge and deduplicate two sorted lists in Python?
How would you create a function to detect anomalies in sales trends using Pandas and
NumPy?
How would you build a reusable ETL framework using Airflow and Python?
How do you handle schema changes in PySpark over time?
How do you monitor and troubleshoot data pipeline failures using Python-based tools?
How do you manage memory allocation in Spark using PySpark?
How do you handle skewed data in a Spark job using PySpark?
How do you validate data using Python in ETL pipelines?
How do you implement file handling in Python for reading and writing CSVs?
How do you implement object-oriented programming concepts in Python for data
engineering tasks? [4] [8] [5] [13] [12] [1] [2] [9] [11] [10] [14] [6] [15] [3] [7]

This list covers the breadth of Python-related questions you may encounter in data engineering
interviews, including practical coding, data pipeline design, ETL automation, PySpark, and data
manipulation tasks.
⁂

Extracted Python-Related Data Engineering Interview Questions

Below is a comprehensive list of all Python-related questions found in the provided interview
guides and attachments. The questions span from core Python programming and scripting to
Python’s use in ETL, automation, PySpark, and data engineering scenarios.

General Python Programming and Scripting

What are the different ways to read a CSV file in Python? [16]
How do you interact with Google BigQuery using Python? [16]
How can you automate data insertion into BigQuery using Python? [16]
Which scripting language are you most comfortable with? [17]
How would you check if a given string is a palindrome? [17]
Write a program to count the number of vowels in a string. [17]
 Could you walk me through the logic behind your code? [17]
What's the most challenging Python problem you've tackled so far? Can you write that code
for me? [17]
Function to find the top 3 largest numbers in a list. [18]
def top_3_largest(numbers):
return sorted(numbers, reverse=True)[:3]

Implement a Python function to count unique words from a file and write them to another
file. [19]
Write a decorator function to log the execution time of a function. [19]
Create a Python program to demonstrate the use of set operations (union, intersection). [19]
Implement file handling in Python to read a CSV and store only specific columns in a
dictionary. [19]
Explain the difference between mutable and immutable objects in Python. [19]

Python for Data Engineering, ETL, and Automation

Write a Python script to process raw JSON files containing sales data and load them into a
relational database. [20]
Describe how you would debug a failing ETL pipeline in production. [20]
How would you handle duplicate or corrupted data in a batch ETL job? [20]
Create a function to detect anomalies in sales trends using Pandas and NumPy. [20]
Write a Python function to merge and deduplicate two sorted lists of sales data. [20]
How would you build a reusable ETL framework using Airflow? [20]
Explain how to implement schema validation for incoming data streams. [20]
Describe how you would monitor ETL job performance and handle long-running tasks. [20]
Handling data validation using SQL or Python. [18]
How do you handle missing data in a DataFrame in Python? [20]
Can you explain the concept of a data pipeline and how you would build one in Python? [20]
How would you automate a data pipeline deployment using GitHub Actions or another CI/CD
tool? [21]
Explain how to schedule an automated task using Apache Airflow. [21]
Python Data Structures & Algorithms
Data Structures: List, Set, Tuple, Dictionary, String. [19]
Odd Number Sorting: Write a function to sort an array, returning only odd numbers. [22]
Unique Values Preservation: Find non-duplicate numbers from a list while preserving the
original order. [22]
Maximum Occurrences: Given a list, return the numbers with the highest count. [22]
JSON Flattening: Write a function to flatten nested JSON objects into a single key-value
dictionary. [22]
Array Pair Sum: Write code to find two numbers in an array that sum up to x. [22]
Stack Implementation: Implement a stack using a linked list. [22]
Write a Python script to merge two sorted lists. [21]
Implement a function to find duplicate records in a large dataset using Python. [21]
Create a script to parse and transform a JSON file into a structured CSV. [21]
Merge two dictionaries and remove keys with null values. [21]

PySpark and Spark with Python

Managing schema changes in PySpark over time. [18]
Write PySpark code to filter and count records. [23]
Write PySpark code to filter records based on specific conditions and add a calculated
column. [24]
Write a PySpark script to filter out invalid records from a dataset and calculate the average
for a specific column, ensuring the schema is strictly defined at runtime. [24]
Null Value Handling in a Single Column: Use fillna or replace in PySpark:
df.fillna({'column_name': 'value'}).show()

Left Anti Join in PySpark:

df1.join(df2, df1['id'] == df2['id'], 'left_anti').show()

Python in Data Engineering Contexts

Use libraries like requests or urllib in Python for API data ingestion, then transform and load
it into the target system. [25]
Using Pandas and Numpy for data preprocessing and transformation. [20] [24]
Handling data validation and schema management using Python. [18] [20]
Implement file handling in Python for reading and writing CSVs. [19]
Implement object-oriented programming concepts in Python for data engineering tasks. [19]
Scenario-Based and Conceptual Python Questions
How would you handle missing data in a DataFrame in Python? [20]
How do you handle duplicate or corrupted data in a batch ETL job? [20]
How would you debug a failing ETL pipeline in production? [20]
How do you automate data insertion into BigQuery using Python? [16]
How do you interact with Google BigQuery using Python? [16]
How would you merge and deduplicate two sorted lists in Python? [20]
How would you create a function to detect anomalies in sales trends using Pandas and
NumPy? [20]
How would you build a reusable ETL framework using Airflow and Python? [20]
How do you handle schema changes in PySpark over time? [18]
How do you monitor and troubleshoot data pipeline failures using Python-based tools? [20]
How do you manage memory allocation in Spark using PySpark? [23]
How do you handle skewed data in a Spark job using PySpark? [23]
How do you validate data using Python in ETL pipelines? [18] [20]
How do you implement file handling in Python for reading and writing CSVs? [19]
How do you implement object-oriented programming concepts in Python for data
engineering tasks? [19]

This list comprehensively covers all Python-related questions and scenarios from the provided
interview materials, including coding, ETL, automation, PySpark, and data pipeline design.
⁂

1. Amazon-Fresher.pdf
2. American-Express.pdf
3. https://www.careerflow.ai/blog/amazon-data-engineer-interview
4. Aarate_1.pdf
5. Adidas.pdf
6. https://www.linkedin.com/posts/prakhar-srivastava-615922150_dataengineer-adidasinterview-bigdata-
activity-7280997662022144000-DfmK
7. https://dataengineeracademy.com/blog/data-engineer-interview-questions-with-python-detailed-answ
ers/
8. Accenture-Azure-Data-Engineer-3.pdf
9. Bitwise.pdf
10. Bristol-Myers-Squibb.pdf
11. Boston-Consulting-Group-_BCG.pdf
12. Amazon-Experienced.pdf
13. Altimetrik.pdf
14. https://www.interviewquery.com/p/data-engineer-python-questions
15. https://www.interviewquery.com/interview-guides/altimetrik-data-engineer
16. Aarate_1.pdf
17. Amazon-Fresher.pdf
18. Accenture-Azure-Data-Engineer-3.pdf
19. American-Express.pdf
20. Adidas.pdf
21. Boston-Consulting-Group-_BCG.pdf
22. Amazon-Experienced.pdf
23. Bitwise.pdf
24. Bristol-Myers-Squibb.pdf
25. Altimetrik.pdf