DEV Community: Anirudh Rajmohan

Your RAG Index Might Be Lying to You: Data Freshness Is the Missing Signal for AI Systems

Anirudh Rajmohan — Wed, 29 Jul 2026 00:28:10 +0000

The failure mode that gets worse when a machine is reading the data

In a classic dashboard, stale data is a human problem: someone looks at a number that's six hours old and makes a slightly worse decision. Annoying, rarely catastrophic.

Now hand that same data to a retrieval-augmented-generation (RAG) pipeline, or to an autonomous agent. The stakes change. The system doesn't pause to sanity-check the timestamp — it acts. And when the data it acts on is stale, three things are true at once:

The answer is confidently wrong.
There is no error to fire on — the query succeeded, the model responded, latency was normal.
Every other signal on your dashboard is green.

That's the worst combination in observability: a real failure that is completely invisible to the signals we currently emit.

Where staleness hides in AI systems

RAG: index vs. corpus. Your vector index was built from a corpus at some point in time. The corpus keeps changing — documents get added, edited, retracted. If the re-embedding job stalls or falls behind, the index quietly drifts out of date. The retriever still returns plausible chunks; the model still writes a fluent answer. It's just answering from a version of reality that no longer exists. The quantity you care about is the age of the index relative to its source — not the age of either one alone.

Feature stores: online–offline skew. The features your model trained on and the features it serves on are supposed to match. When the online store lags the offline pipeline, predictions degrade in a way that looks like model drift but is actually data staleness wearing a costume.

Agents: stale shared state. Multi-agent systems coordinate through shared memory, scratchpads, and context. An agent reasoning over state that another agent updated ten steps ago — but which never propagated — makes locally reasonable, globally wrong decisions. This isn't a new or exotic problem: it's exactly the regime that Age of Information theory was invented to analyze in real-time control and networked systems, where an actuator acting on a stale sensor reading is a known hazard.

Freshness is old; its absence from our telemetry is what's newly expensive

None of this is a new metric. Freshness — the time since the most recent update was generated at its source — has a precise name in networked-systems research: the Age of Information:

age = now − event_time(freshest record)

If a source stops producing, its age climbs without bound while every other signal stays healthy. Data-observability vendors already compute this internally. dbt source freshness computes it for dbt. Kafka-lag tools compute it for Kafka.

What's missing is a standard, portable way to emit it — one vocabulary that works whether the "source" is a Postgres table, a Kafka topic, a feature store, or a vector index, and that lands in any backend unchanged. That gap was tolerable when humans were the only consumers. It stops being tolerable when the consumer is a model that acts without a second look.

A proposal: `data.staleness.*`

I've been working on a small, vendor-neutral semantic convention for data staleness, built on OpenTelemetry, plus a working reference implementation. To be clear about status: this is an independent, community project — not an official OpenTelemetry project, and not endorsed by or affiliated with OpenTelemetry or the CNCF. The convention is a proposal at "Development" stability, under discussion with the Semantic Conventions SIG (issue #3909). Names may change.

The shape is deliberately compact:

Metric	Meaning
`data.staleness.age`	`now − event_time` of the freshest record (the primary signal)
`data.staleness.lag`	`processing_time − event_time` of the most recent record
`data.staleness.last_update.timestamp`	Unix time of the last successful update
`data.staleness.records.behind`	positional backlog (e.g. Kafka consumer lag)
`data.staleness.sla.*`	configured threshold + breach evaluation

The key insight for AI systems is the differential case: the age of an index relative to the corpus it was built from, or a replica relative to its primary. A single number ("the index is 40 minutes behind its source") is exactly the alert a RAG pipeline needs and rarely has.

The design rule that matters most for AI: never fabricate a number

One principle runs through the whole implementation: an unmeasurable freshness value must be visible, never faked. An empty table, a NULL MAX(), a timeout, an empty partition — each surfaces as data.staleness.probe.errors with a specific error.type, not as a fabricated "0 seconds old."

This matters doubly for AI. A freshness check that silently reports "fresh" when it actually failed is worse than no check at all — because it will reassure you precisely when your RAG index or agent state is most broken.

Try it against one AI workload

Python SDK — point a differential probe at your index and its source:

pip install data-staleness-otel

from otel_staleness import StalenessMonitor, conventions as sc
from otel_staleness.probes import SQLFreshnessProbe

monitor = StalenessMonitor()
# Freshest source document vs. freshest embedded doc in the index:
monitor.add_probe(SQLFreshnessProbe(
    fetch_max_epoch=lambda: latest_source_doc_ts(),
    source_name="kb_corpus", system=sc.System.POSTGRESQL,
    sla_threshold_seconds=1800))
monitor.start()

Zero-code Collector — scrape SQL, Kafka, Kinesis, files, HTTP, and schema registries from YAML alone, no application code, and emit the same data.staleness.* metrics into the same backend.

Get involved

If "how stale is the data my model is answering from?" is a question you've had to answer with a bespoke query and a fragile dashboard, I'd value your input:

The convention proposal is open in the OpenTelemetry Semantic Conventions repo (#3909) — thumbs-up, comments, and concrete AI use cases genuinely help gauge demand.
Code, spec, model, and validation: https://github.com/anirudhrajreliability/otel-data-staleness (Apache-2.0).
Try it (pip install data-staleness-otel) and tell me where it breaks.

We standardized latency and errors because machines needed portable signals to reason about. AI systems reasoning over data need a portable signal for freshness for exactly the same reason. Let's give them one.

How Old Is My Data? The Missing OpenTelemetry Signal

Anirudh Rajmohan — Wed, 22 Jul 2026 21:14:54 +0000

A proposal — and a working reference implementation — for making data freshness a portable, first-class telemetry signal.

The failure mode nobody's dashboard catches

Picture a data pipeline at 3 a.m. Every service is green. Latency is low, error rates are zero, throughput is normal. Every SLO is met.

And the executive dashboard it feeds is quietly six hours out of date, because one upstream producer stalled and nothing downstream noticed. There was no error to fire on. No latency spike. The data just… stopped being fresh.

This is one of the most common and most invisible failure modes in modern data systems, and it has a precise name from networked-systems research: the Age of Information (AoI) — how much time has elapsed since the freshest piece of data was generated at its source:

age = now − event_time(freshest record)

If a source stops producing, its age climbs without bound while every other signal stays perfectly healthy. That's exactly the condition freshness monitoring exists to catch — and it's exactly the signal our observability stacks don't have a standard way to express.

We standardized everything except freshness

OpenTelemetry has become the de facto standard for telemetry. It gives us portable conventions for latency, errors, throughput, database calls, and messaging. You can move a dashboard or an alert between backends and vendors because the names are standardized.

But there is no OpenTelemetry semantic convention for data freshness. db.* and messaging.* describe operations — how long a query took, how many messages were published — not how old the data itself is. There isn't even a standardized consumer-lag metric.

So today, every team reinvents freshness in a silo:

Data-observability vendors (Monte Carlo, Metaplane, Sifflet, …) each compute it internally, incompatibly.
dbt source freshness computes it for dbt.
Kafka-lag tools (Burrow) compute it for Kafka.
Everyone else writes a bespoke SELECT MAX(updated_at) and a Grafana panel.

None of it is comparable across systems or portable across backends. Freshness is a well-understood quantity trapped in a dozen proprietary shapes.

A proposal: `data.staleness.*`

I've been working on a small, vendor-neutral semantic convention for data staleness / freshness, built on OpenTelemetry, plus a complete reference implementation. To be clear about status: this is an independent, community project — not an official OpenTelemetry project, and not endorsed by or affiliated with the OpenTelemetry project or the CNCF. The convention is a proposal, at OpenTelemetry "Development" stability, currently under discussion with the Semantic Conventions SIG (issue #3909). It is not (yet) an accepted standard, and names may change.

The shape is deliberately compact — a handful of metrics grounded in AoI:

Metric	Meaning
`data.staleness.age`	`now − event_time` of the freshest record (the primary signal)
`data.staleness.lag`	`processing_time − event_time` of the most recent record
`data.staleness.last_update.timestamp`	Unix time of the last successful update
`data.staleness.records.behind`	positional backlog (e.g. Kafka consumer lag)
`data.staleness.sla.*`	configured threshold + breach evaluation

…with attributes like data.source.system, data.source.name, data.staleness.method, and data.staleness.partition — orthogonal to, and composable with, the existing db.* and messaging.* conventions.

The key insight is age vs lag: if a source stops producing, the last record's lag stays constant while age keeps rising. That divergence is the alert you actually want.

The design principle: never fabricate a number

One rule runs through the whole implementation: an unmeasurable freshness value must be visible, never faked. An empty table, a NULL MAX(), a timeout, an empty Kafka partition — each surfaces as data.staleness.probe.errors with a specific error.type, not as a fabricated "0 seconds old." A broken freshness check that silently reports "fresh" is worse than no check at all.

Try it in two minutes

There are two ways to adopt it — a Python SDK for code you own, and a zero-code Collector for infrastructure you don't.

Python SDK — you give it a probe that returns the freshest event time; it emits the standardized metrics:

pip install data-staleness-otel

from otel_staleness import StalenessMonitor, conventions as sc
from otel_staleness.probes import SQLFreshnessProbe

monitor = StalenessMonitor()
monitor.add_probe(SQLFreshnessProbe(
    fetch_max_epoch=lambda: latest_updated_at(),   # Unix seconds or datetime
    source_name="orders", system=sc.System.POSTGRESQL,
    namespace="public", sla_threshold_seconds=300))
monitor.start()

Zero-code Collector — a custom OpenTelemetry Collector receiver scrapes SQL, Kafka, Kinesis, files, HTTP, schema registries, and DB-migration versions directly from YAML, with no application code:

receivers:
  datastaleness:
    collection_interval: 30s
    sources:
      - type: sql
        name: orders
        system: postgresql
        dsn: "postgres://user:pass@db:5432/app?sslmode=disable"
        table: orders
        timestamp_column: updated_at
        sla_threshold_seconds: 300

Both emit the same data.staleness.* metrics into the same backend, so they interoperate.

It's more than a spec

Because a convention is only credible if it's real and checkable, the project ships:

A written spec plus a machine-readable Weaver model that passes weaver registry check — the same tool the OpenTelemetry SIG uses to validate and generate conventions.
A language-agnostic conformance suite so the numbers aren't one library's opinion.
A two-language reference implementation (Python SDK + Go Collector receiver & processor), validated end-to-end on real backends — a suite stands up real Postgres, Kafka, Redis, LocalStack Kinesis, and a schema registry and asserts the emitted metrics are numerically correct (age equal to a known injected offset, exact consumer lag), and that a stopped source surfaces probe.errors rather than a fabricated value.

Why standardize instead of ship-another-library?

Because the value is portability. If freshness has one vocabulary, then a freshness alert, a Grafana panel, or an SLO travels across Postgres, Kafka, dbt, a vector index, and a replica — and across Prometheus, an OTLP store, or any vendor backend — unchanged. This isn't a new metric; it's the consolidation of a well-studied one (AoI, and what every data-observability tool already computes) into the framework that already standardizes latency and errors.

Get involved

If "how old is my data?" is a question you've had to answer with a bespoke query and a fragile dashboard, I'd love your input:

The convention proposal is open in the OpenTelemetry Semantic Conventions repo (#3909) — thumbs-up, comments, and use cases genuinely help gauge demand.
The code, spec, model, and validation live here: https://github.com/anirudhrajreliability/otel-data-staleness (Apache-2.0).
Try it (pip install data-staleness-otel), tell me where it breaks, and — if this is your problem space — consider co-owning the effort.

Freshness is a known quantity that, paradoxically, has no standard way to be emitted. Let's fix that.

DEV Community: Anirudh Rajmohan

Your RAG Index Might Be Lying to You: Data Freshness Is the Missing Signal for AI Systems

The failure mode that gets worse when a machine is reading the data

Where staleness hides in AI systems

Freshness is old; its absence from our telemetry is what's newly expensive

A proposal: data.staleness.*

The design rule that matters most for AI: never fabricate a number

Try it against one AI workload

Get involved

How Old Is My Data? The Missing OpenTelemetry Signal

The failure mode nobody's dashboard catches

We standardized everything except freshness

A proposal: data.staleness.*

The design principle: never fabricate a number

Try it in two minutes

It's more than a spec

Why standardize instead of ship-another-library?

Get involved

A proposal: `data.staleness.*`

A proposal: `data.staleness.*`