How to understand, debug, and optimize large-scale data pipelines using Spark and Kafka.
They covered investigating slow Spark jobs, identifying data skew and slow executors, optimizing joins on skewed keys using AQE and key salting, and validating performance improvements with Spark UI metrics.
They also covered data formats and platforms, including Parquet, ORC, Iceberg, Hive, Snowflake, and Trino; Spark driver, workers, executors, and Flink TaskManagers; Kafka partitions and consumer groups; and how to discover dataset meaning and data lineage.
这道题更像是一次围绕大数据平台排障与优化的面试题，核心在于如何定位 Spark 任务变慢的根因，并针对数据倾斜、慢执行器、倾斜 join 等典型问题做优化。常见思路包括借助 Spark UI 分析 stage、task 和 shuffle 指标，结合 AQE、自定义分区或 key salting 缓解热点 key 带来的倾斜；同时还要理解 Parquet、ORC、Iceberg 等数据格式，以及 Kafka 分区、consumer group、Spark/Flink 组件之间的数据流转，最终用指标验证优化是否真正生效。
