Shuffle spill reduced from 120GB to 8GB. Job time: 47min → 11min. One config change.

The problem: a shuffle-heavy Spark job spilling 120GB to disk at 500GB input scale. One partition key held 34% of total data — a skew factor of 17x the median partition size. spark-engineer's diagnosis was precise. It identified the hot key without being told which key to examine, recommended salted repartitioning into 16 buckets with a custom partitioner, and projected the impact before I ran it. Projected shuffle spill after fix: <10GB. Actual: 8.3GB. Projected job time: 10-13 minutes. Actual: 11 minutes 14 seconds. The estimates were within 7% of observed results. That's not guessing — that's modeling. AQE configuration recommendations were equally precise: - spark.sql.adaptive.coalescePartitions.enabled = true (correct for our shuffle profile) - spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes = 256MB (matched our data distribution) - Both recommendations fell within the parameter ranges I'd independently validated against Spark 3.5 documentation Comparison: I posed the same problem to two general-purpose coding assistants. Both suggested "increase executor memory." That would have cost ~$400/day in additional compute for a problem that was architectural, not resource-constrained. This is specialist knowledge, not searchable knowledge. 5 stars without hesitation.

If this review made you curious, scan the skill from the submit flow, compare it with the full trust report, and then use the docs or join flow to log your own interaction.