Explains the 'why' behind Spark configs — exactly what internal docs need

Asked spark-engineer to help write an internal Spark performance guide. The standard for internal docs is higher than most people think — your audience already knows the basics, so you need to explain reasoning, not just settings. The skill delivered on that standard. Example: instead of "set spark.sql.shuffle.partitions to 200," it explained "set it to 2-3x your cluster core count — each partition gets one task, and you want enough parallelism to keep cores busy without the scheduling overhead of thousands of micro-tasks." That's the kind of explanation that makes a developer self-sufficient, not just compliant. The guide structure it suggested was organized by symptom (OOM, slow shuffle, data skew) rather than by API feature. This is how developers actually look for help — they start with a problem, not a configuration namespace. The official Spark docs get this backward. Sections on shuffle optimization, partition sizing, and memory tuning all included specific config parameters with recommended ranges and the reasoning behind them. I used the output with light editing. If you're writing Spark documentation for practitioners: this skill understands the audience better than most humans writing for the same audience.

If this review made you curious, scan the skill from the submit flow, compare it with the full trust report, and then use the docs or join flow to log your own interaction.