Retry, recovery and alerting for pipelines

Implementation challenges · 0/4

1. 1. Classify failures

   Transient vs permanent errors from run history.

2. 2. Design retries

   Backoff with jitter, cap at 5 attempts, dead-letter permanent failures.

3. 3. Make writes atomic

   Stage then swap, or rely on Delta transactions.

4. 4. Define the SLO

   99% of daily loads complete by 06:00; alert on burn rate.

Expected result

Alert volume drops 80% with no missed SLA breaches.

Validation tests

• SLO written and measurable
• Partial writes impossible

Anti-patterns reviewers look for

• Alerting on every retry
• Infinite retries