When AI Hallucinations Are Really Data Contract Failures

Short version of what I took away from this one: when a RAG assistant quotes a confidently wrong number, pull the retrieved records before you blame the model.

Often the model was handed the wrong number and repeated it faithfully. A new prompt can't fix that. Neither can a new vendor.

Here's the mechanism, with an invented example. An insurer's claims copilot retrieves from its policy database. A migration for a new mobile app renames max_coverage_limit_usd to per_incident_deductible_usd. Unit tests pass, the ETL job exits cleanly, no alert fires. The type didn't change, so a type check sees nothing wrong.

Days later an adjuster asks for a policy's coverage cap. The retrieved chunk has no coverage limit field, only dollar figures under deductible labels. The copilot answers $12,500 instead of the real $1,250,000. It didn't invent that number. It retrieved it.

Classic BI had a buffer: a bad record shows up as an odd chart and someone notices. In RAG, the pipeline's output is all the assistant sees. A green job that writes reinterpreted fields into the index ships the problem straight to the user.

Kleppmann's framing in DDIA is the useful lens: schema evolution is compatibility in two directions, new code reading old data and old code reading new data. The format matters for renames too. Avro matches fields by name, so a safe rename keeps the old name as an alias. Protobuf matches by tag number, so names can change but tags can't. A raw JSON dump checks nothing.

What I'd put between the source and the model:

• Compatibility checks in CI. Register every schema an AI system consumes. Confluent's default mode is BACKWARD; for AI consumers I'd set FULL or FULL_TRANSITIVE explicitly. Renames or removals of fields an AI consumer reads go through a deprecation window.
• A validator before embedding. Pydantic mode="after" validators for invariants like coverage limit >= deductible. Use raise ValueError, not assert, because -O strips asserts. Rejects go to a dead-letter queue with a named owner.
• Provenance on every chunk. Schema version, source, as-of timestamp, so you can filter stale chunks and trace an incident to the exact record.

Plus one team rule: when a wrong answer traces back to bad source data, the fix goes into the contract or the pipeline, not the system prompt.

To be clear about the limits. Contracts don't make answers correct. If the contract maps "coverage cap" to the wrong field, you get a wrong answer with a clean audit trail. They also cost something: pushback from backend teams, a quarantine backlog someone has to work through, real effort on legacy systems. Start where a wrong answer costs money or creates compliance risk.

The full article has the diagrams, the sources and the complete four-step audit: https://medium.com/towards-data-engineering/the-contaminated-fuel-trap-when-ai-hallucinations-are-data-contract-failures-55cca4d41377