How Old Is My Data? The Missing OpenTelemetry Signal

A proposal — and a working reference implementation — for making data freshness a portable, first-class telemetry signal.

The failure mode nobody's dashboard catches

Picture a data pipeline at 3 a.m. Every service is green. Latency is low, error rates are zero, throughput is normal. Every SLO is met.

And the executive dashboard it feeds is quietly six hours out of date, because one upstream producer stalled and nothing downstream noticed. There was no error to fire on. No latency spike. The data just… stopped being fresh.

This is one of the most common and most invisible failure modes in modern data systems, and it has a precise name from networked-systems research: the Age of Information (AoI) — how much time has elapsed since the freshest piece of data was generated at its source:

age = now − event_time(freshest record)

If a source stops producing, its age climbs without bound while every other signal stays perfectly healthy. That's exactly the condition freshness monitoring exists to catch — and it's exactly the signal our observability stacks don't have a standard way to express.

We standardized everything except freshness

OpenTelemetry has become the de facto standard for telemetry. It gives us portable conventions for latency, errors, throughput, database calls, and messaging. You can move a dashboard or an alert between backends and vendors because the names are standardized.

But there is no OpenTelemetry semantic convention for data freshness. db.* and messaging.* describe operations — how long a query took, how many messages were published — not how old the data itself is. There isn't even a standardized consumer-lag metric.

So today, every team reinvents freshness in a silo:

• Data-observability vendors (Monte Carlo, Metaplane, Sifflet, …) each compute it internally, incompatibly.
• dbt source freshness computes it for dbt.
• Kafka-lag tools (Burrow) compute it for Kafka.
• Everyone else writes a bespoke SELECT MAX(updated_at) and a Grafana panel.

None of it is comparable across systems or portable across backends. Freshness is a well-understood quantity trapped in a dozen proprietary shapes.

A proposal: data.staleness.*

I've been working on a small, vendor-neutral semantic convention for data staleness / freshness, built on OpenTelemetry, plus a complete reference implementation. To be clear about status: this is an independent, community project — not an official OpenTelemetry project, and not endorsed by or affiliated with the OpenTelemetry project or the CNCF. The convention is a proposal, at OpenTelemetry "Development" stability, currently under discussion with the Semantic Conventions SIG (issue #3909). It is not (yet) an accepted standard, and names may change.

The shape is deliberately compact — a handful of metrics grounded in AoI:

Metric	Meaning
data.staleness.age	now − event_time of the freshest record (the primary signal)
data.staleness.lag	processing_time − event_time of the most recent record
data.staleness.last_update.timestamp	Unix time of the last successful update
data.staleness.records.behind	positional backlog (e.g. Kafka consumer lag)
data.staleness.sla.*	configured threshold + breach evaluation

…with attributes like data.source.system, data.source.name, data.staleness.method, and data.staleness.partition — orthogonal to, and composable with, the existing db.* and messaging.* conventions.

The key insight is age vs lag: if a source stops producing, the last record's lag stays constant while age keeps rising. That divergence is the alert you actually want.

The design principle: never fabricate a number

One rule runs through the whole implementation: an unmeasurable freshness value must be visible, never faked. An empty table, a NULL MAX(), a timeout, an empty Kafka partition — each surfaces as data.staleness.probe.errors with a specific error.type, not as a fabricated "0 seconds old." A broken freshness check that silently reports "fresh" is worse than no check at all.

Try it in two minutes

There are two ways to adopt it — a Python SDK for code you own, and a zero-code Collector for infrastructure you don't.

Python SDK — you give it a probe that returns the freshest event time; it emits the standardized metrics:

pip install data-staleness-otel

from otel_staleness import StalenessMonitor, conventions as sc
from otel_staleness.probes import SQLFreshnessProbe

monitor = StalenessMonitor()
monitor.add_probe(SQLFreshnessProbe(
    fetch_max_epoch=lambda: latest_updated_at(),   # Unix seconds or datetime
    source_name="orders", system=sc.System.POSTGRESQL,
    namespace="public", sla_threshold_seconds=300))
monitor.start()

Zero-code Collector — a custom OpenTelemetry Collector receiver scrapes SQL, Kafka, Kinesis, files, HTTP, schema registries, and DB-migration versions directly from YAML, with no application code:

receivers:
  datastaleness:
    collection_interval: 30s
    sources:
      - type: sql
        name: orders
        system: postgresql
        dsn: "postgres://user:pass@db:5432/app?sslmode=disable"
        table: orders
        timestamp_column: updated_at
        sla_threshold_seconds: 300

Both emit the same data.staleness.* metrics into the same backend, so they interoperate.

It's more than a spec

Because a convention is only credible if it's real and checkable, the project ships:

• A written spec plus a machine-readable Weaver model that passes weaver registry check — the same tool the OpenTelemetry SIG uses to validate and generate conventions.
• A language-agnostic conformance suite so the numbers aren't one library's opinion.
• A two-language reference implementation (Python SDK + Go Collector receiver & processor), validated end-to-end on real backends — a suite stands up real Postgres, Kafka, Redis, LocalStack Kinesis, and a schema registry and asserts the emitted metrics are numerically correct (age equal to a known injected offset, exact consumer lag), and that a stopped source surfaces probe.errors rather than a fabricated value.

Why standardize instead of ship-another-library?

Because the value is portability. If freshness has one vocabulary, then a freshness alert, a Grafana panel, or an SLO travels across Postgres, Kafka, dbt, a vector index, and a replica — and across Prometheus, an OTLP store, or any vendor backend — unchanged. This isn't a new metric; it's the consolidation of a well-studied one (AoI, and what every data-observability tool already computes) into the framework that already standardizes latency and errors.

Get involved

If "how old is my data?" is a question you've had to answer with a bespoke query and a fragile dashboard, I'd love your input:

• The convention proposal is open in the OpenTelemetry Semantic Conventions repo (#3909) — thumbs-up, comments, and use cases genuinely help gauge demand.
• The code, spec, model, and validation live here: https://github.com/anirudhrajreliability/otel-data-staleness (Apache-2.0).
• Try it (pip install data-staleness-otel), tell me where it breaks, and — if this is your problem space — consider co-owning the effort.

Freshness is a known quantity that, paradoxically, has no standard way to be emitted. Let's fix that.