Check the output interval and event timestamps before the report becomes available to consumers.
The dashboard says the overnight pipeline succeeded. The BI report opened at 8:00 a.m. People are already discussing the morning numbers.
Someone from operations asks why yesterday’s transactions are still the newest records.
Pipeline finished successfully.
In this example, every expected task succeeded. The pipeline copied an old snapshot into a valid new file, and no task checked its event timestamps. I would put that check at the boundary where the report becomes available to consumers.
Three clocks, three different questions
Teams often compress distinct timestamps into a single idea of “latest”:
The run clock: when the orchestration system started and finished a run. This helps answer whether the workflow executed.
The interval clock: the logical period of data assigned to the run. In Airflow, a time-scheduled run has a data interval; its logical date commonly denotes that interval’s start, not the moment the tasks physically execute. A run for Monday’s data may legitimately execute after Monday has ended.
The event clock: the newest business event actually represented in the output. If an hourly orders dataset finishes at 8:00 but its newest order_created_at is 5:00, you have a freshness question even if the DAG status says success.
The three clocks can agree, disagree, or be incomparable depending on what the source produces. Never use a scheduler timestamp as a substitute for inspecting output events.
How a green DAG can deliver old data
Consider an hourly DAG that extracts orders, transforms them, and writes a report table. A source API is delayed but still returns a well-formed response. The transform accepts the response. The write succeeds. A terminal task sends the success notification. Yesterday’s data has completed today’s commute.
Nothing in that story proves that the output contains events within the business SLA.
There is another nuance: Airflow’s DAG-run status is derived from task states, including leaf tasks. Its documentation warns that certain trigger-rule arrangements can produce an apparently successful DAG run even when an intermediate task failed. A status light is useful, but it is not an end-to-end assertion about the delivered dataset.
This is why I do not recommend solving freshness incidents by simply adding more retries. Retrying a successful extraction of an old snapshot may give you the same old snapshot again.
Define freshness from the consumer backward
For this hourly orders example, I would agree on the following contract with the consumer:
The hourly orders dataset must contain at least one valid row, must be produced for the required interval, and must contain business event timestamps no older than 90 minutes at the 8:00 reporting checkpoint.
The 90-minute number is a made-up example SLA , not a recommendation for every company. For a dataset that receives no orders on quiet days, a mandatory recent event may be an invalid contract; use source heartbeat, completeness markers, or expected arrival windows instead. For batch snapshots, partition publication time may be more informative than event time. The right contract reflects producer behavior and consumer expectations.
Before marking a dataset ready, I want answers to five specific questions: Did the job publish the required data interval, or did it succeed while reprocessing an older one? Is the newest event recent enough for this consumer? Is the expected partition or count present, or are missing records being silently tolerated? Did the agreed validation checks pass before the table became visible downstream? Are timestamps timezone-aware, is event time distinct from ingestion time, and is clock skew bounded?
These are properties of the output and its contract, not properties of the colored dot next to a task.
Fail at the publication boundary
The companion lab contains a small Python contract that demonstrates the idea without requiring Airflow, Spark, or cloud credentials:
Open the freshness-contract lab and tests on GitHub .
The core function accepts a snapshot description, the current UTC instant, the required interval end, and a maximum event age. It returns specific breaches for an old interval, stale events, empty output, failed quality checks, or implausibly future-dated events. A strict wrapper raises an error so the publishing task can fail closed rather than completing with bad data.
For example, the synthetic case is:
from datetime import datetime, timedelta, timezone
from freshness_contract import Snapshot, assess_snapshot
t = datetime(2026, 10, 20, 8, tzinfo=timezone.utc)
snapshot = Snapshot(
dataset="orders_hourly",
dag_state="success",
data_interval_end=t,
latest_event_at=t - timedelta(hours=2),
row_count=100,
quality_checks_passed=True,
)
print(assess_snapshot(
snapshot,
now=t,
required_interval_end=t,
max_age=timedelta(minutes=90),
))
# ('event timestamp too old',)
The output reports event timestamp too old: the latest event is 120 minutes old against the example limit of 90 minutes, even though dag_state is success.
Run the standalone tests with Python 3.12+:
python -m unittest discover -s airflow/day_001_freshness_contract -p 'test_*.py' -v
The tests cover stale-but-green output, the wrong interval, missing timestamps, empty output, quality failures, invalid timezone handling, clock skew and a healthy dataset.
Scope: this is an original pure-Python illustration of a guard. It is not an installed Airflow operator and has not been exercised against an Airflow scheduler, Databricks, cloud storage, or production data. To deploy it, fetch real metadata from the produced dataset and invoke the strict check before marking the artifact ready for downstream use.
What about asset-aware orchestration?
Airflow 3 includes asset-aware scheduling, which is valuable when downstream workflows should respond to upstream asset events instead of an arbitrary timer. For task-declared outlets, Airflow marks an asset as updated when the producer task completes successfully. Externally emitted asset events need their own producer-side validation. Neither path inherently certifies every business-quality dimension of the data. Put the contract check before the event is emitted, or make consumers verify the contract independently.
Asset-aware scheduling solves when to trigger. Data contracts answer what is safe to consume. They complement each other; they are not substitutes.
The incident-review questions I would ask
Did the source publish new business events? What is the newest event timestamp and what does that field actually represent? Which interval did the job write? Can we distinguish a source outage from a legitimate zero-activity period? Was the output made visible only after validation passed? What alert goes to the owner when the contract is breached?
Those questions lead to an operational action. “Make the DAG green” usually leads to another green DAG.
If I had to choose one dashboard tile for an executive review, I would show data availability against the consumer’s agreed freshness contract , backed by evidence, alongside workflow status. The scheduler should remain visible. It should not be the sole source of truth.
A green DAG means the orchestration ran. A trustworthy dataset means the data met its promises.
References
- Apache Airflow DAG runs: status, logical date, and data intervals.
- Apache Airflow asset-aware scheduling.
- Reproducible freshness-contract lab.
AI-generated illustration of a successful workflow delivering stale data.
Top comments (0)