How to understand, debug, and optimize large-scale data pipelines using Spark and Kafka.
They covered investigating slow Spark jobs, identifying data skew and slow executors, optimizing joins on skewed keys using AQE and key salting, and validating performance improvements with Spark UI metrics.
They also covered data formats and platforms, including Parquet, ORC, Iceberg, Hive, Snowflake, and Trino; Spark driver, workers, executors, and Flink TaskManagers; Kafka partitions and consumer groups; and how to discover dataset meaning and data lineage.
This is a systems-oriented interview topic focused on debugging and optimizing large-scale data pipelines rather than a classic algorithm problem. The key themes are identifying the root cause of slow Spark jobs, diagnosing data skew and slow executors, applying techniques such as AQE and key salting to handle skewed joins, and validating improvements with Spark UI metrics. It also touches on practical platform knowledge across data formats, Spark/Flink components, Kafka partitions and consumer groups, and data lineage discovery.
