All labs
Performance Engineering · Databricks · Guided simulation · 45 min
Diagnose a skewed Spark join
A nightly job joining clickstream to products went from 12 to 95 minutes after a marketing campaign.
Implementation challenges · 0/4
1. Read the stage summary
Max task duration 41 min vs median 9 s — a classic skew signature.
2. Find the hot key
Group by product_id: one 'promo' SKU holds 38% of rows.
3. Choose a fix
Compare AQE skew join, broadcast of the small side, and salting.
4. Validate
Confirm task duration spread falls below 3x median.
Expected result
Runtime ≈ 14 minutes with AQE + broadcast.
Validation tests
- Skewed key correctly identified
- Fix justified with metrics
Anti-patterns reviewers look for
- Doubling the cluster first
- Repartitioning blindly to 2000