What feedback did your manager give you
Senior Data Engineer Interview Questions
2,393 senior data engineer interview questions shared by candidates
A developer on the team was running the following line in a function for logging purposes, and the job crashed with an "out of memory" exception. The developer says that the cluster has many workers with a lot of memory and disk and still the job crashes. Can you help explain how come to this line makes the job crash with OOM even though the cluster is huge? def someFunc(): for row in df.collect(): print(f'Customerr{row["customer"]} => Paid {row["price"]}')
Describe a project you are particularly proud of
sql queries and pyspark questions and on your projects
SQL queries medium to complex Spark concepts and scenario based questions
persist vs cache? what does init do? args vs kargs? json vs parquet? columnar storage? distributed systems? spark dataframe vs pandas dataframe
How did you design data lake what are the three layers. How did you design using what services. why this services over the other Difference between schema in data warehouse Whats the purpose of this service. How you would store some secret keys and how you would secure the data. what are the methods and how you did it. whats the typical project activities dev,test,prod the cycle.
Spark and Scala, detecting bank fraud
How do you manage conflict? How do you break down technical issues to non-technical audience? Have you implemented any Data Governance tools?
What does diversity and inclusion mean to you? Name a time when you identified and solved a problem How did you know when the solution was a success?
Viewing 1731 - 1740 interview questions