Airflow retries are not a retry strategy

For a long time our answer to flaky pipelines was "just add retries". Three retries, then five. It made the failure rate look better and made everything else worse.

What actually happened:

• Real failures got buried. A task that failed because of a schema change looked identical to a task that failed because BigQuery hiccuped, and both eventually "succeeded" after a retry.
• Runtime crept up. Retries with exponential backoff on a slow dbt model turned a 40 minute DAG into a 90 minute one on bad days.
• The on-call runbook was useless. "It recovers on its own" is not a runbook.

What we changed:

1. Classified failures. Timeouts, rate limits and transient network errors get retries. Data issues, schema changes and null explosions do not — they fail fast with a clear message.
2. Added an explicit retry_exponential_backoff and a max retry window, so a DAG can't quietly burn two hours.
3. Wrote down the expected runtime per task and alert when it exceeds it. That single alert has caught more real problems than any retry ever did.

Retries are fine for flaky infrastructure. They are terrible as a substitute for knowing why something broke, and even worse when they hide the fact that your data arrived wrong three times in a row.

One thing I still haven't figured out: where to draw the line for semi-transient errors like a query that times out once every few hours. Right now we retry it, but it feels like we're just postponing the investigation.