DEV Community: Muhammad Ikram UL Mustafa

Context Rot: Why Your RAG Pipeline Degrades Without Anyone Deploying Anything

Muhammad Ikram UL Mustafa — Mon, 02 Nov 2026 02:19:34 +0000

Nobody changed the model. Nobody touched the prompt. Accuracy dropped anyway.

This is the most common production RAG failure I see, and it is the hardest to notice, because there is no deploy to correlate it with. The system that passed every test in March is quietly wrong in September.

The cause is almost always the same: your index drifted away from your source of truth, and nothing told you.

The failure nobody instruments for

A RAG pipeline has two copies of your knowledge. The source documents, and the embedded chunks derived from them. Those are updated at different times, by different code paths, and often by different people.

Every gap between them is a window where retrieval returns something that is no longer true.

Here is what actually happens over six months:

Documents get updated, chunks do not. Someone edits a policy document. The text store updates. The embedding job is queued behind something else, or failed silently three weeks ago, or was never wired up for edits because it was built for initial ingestion. Retrieval now returns the old content, confidently.

Documents get deleted, chunks survive. A product is discontinued and its documentation is removed. Nobody deletes the chunks, because deletion was never part of the ingestion path. The model is still happily answering questions about it.

New documents change the neighbourhood. You add two hundred new documents. Nothing about the old ones changed, but they are no longer in the top five for the queries they used to answer, because something more semantically similar now outranks them. Your accuracy on existing questions drops and nobody deployed anything.

The embedding model changes under you. Provider updates the model behind the same API name. New queries are embedded with the new version, your stored chunks with the old one. Now you are comparing vectors from two different spaces. This one is particularly nasty because similarity scores still come back, they are just meaningless.

Query distribution shifts. Your users start asking about things the corpus never covered well. Retrieval still returns the closest match, because that is what it does. The closest match to a question you have no answer for is simply the wrong answer with high confidence.

None of these throw an error.

Why your tests did not catch it

Most teams test retrieval once, at build time, with a set of questions written by the person who built it. That tells you the pipeline works. It tells you nothing about whether it still works.

The second problem is what gets measured. Teams evaluate the final answer, because that is what the user sees. But the answer is downstream of retrieval, so a retrieval failure and a generation failure look identical from the output. You cannot tell whether the model reasoned badly or whether it reasoned correctly over the wrong chunks.

Instrument the gap, not the output

1. Track whether chunks match their source

The cheapest useful thing you can do is hash the source content at ingestion and store it with the chunk.

import hashlib

def content_hash(text: str) -> str:
    return hashlib.sha256(text.encode()).hexdigest()

async def ingest_document(doc: Document, tenant_id: UUID):
    doc_hash = content_hash(doc.content)

    existing = await db.fetchval(
        "SELECT source_hash FROM doc_chunks "
        "WHERE document_id = $1 LIMIT 1",
        doc.id
    )

    if existing == doc_hash:
        return  # nothing changed, skip re-embedding

    chunks = chunk_text(doc.content)
    embeddings = await embed_batch(chunks)

    async with db.transaction():
        await db.execute(
            "DELETE FROM doc_chunks WHERE document_id = $1", doc.id
        )
        for chunk, embedding in zip(chunks, embeddings):
            await db.execute(
                "INSERT INTO doc_chunks "
                "(document_id, tenant_id, content, embedding, "
                " source_hash, embedding_model, indexed_at) "
                "VALUES ($1, $2, $3, $4, $5, $6, now())",
                doc.id, tenant_id, chunk, embedding,
                doc_hash, EMBEDDING_MODEL
            )

Three things worth noting. The hash check means re-ingestion is idempotent, so you can run it on a schedule without re-embedding everything. The delete and insert are in one transaction, so a query cannot land in a window where the old chunks are gone and the new ones are not written yet. And embedding_model is stored per row, which matters for the next part.

2. Find the drift

Once you have the hash and the model version stored, staleness becomes a query rather than a mystery.

-- chunks whose source document has since changed
SELECT d.id, d.title, c.indexed_at
FROM documents d
JOIN doc_chunks c ON c.document_id = d.id
WHERE c.source_hash != encode(sha256(d.content::bytea), 'hex')
GROUP BY d.id, d.title, c.indexed_at;

-- orphaned chunks: source document is gone
SELECT DISTINCT c.document_id
FROM doc_chunks c
LEFT JOIN documents d ON d.id = c.document_id
WHERE d.id IS NULL;

-- chunks embedded with a different model than you are querying with
SELECT embedding_model, count(*)
FROM doc_chunks
GROUP BY embedding_model;

That last one should return exactly one row. If it returns two, you have vectors from two different embedding spaces in the same index and your similarity scores are not comparable.

Run these on a schedule and alert on them. They are cheap and they catch the majority of real drift.

3. Evaluate retrieval separately from generation

This is the change that matters most, and it is the one teams skip.

Build a set of questions where you know which document should be retrieved. Not which answer is correct, which source is correct. Then measure whether it still comes back.

RETRIEVAL_CASES = [
    {
        "query": "what is the refund window for annual plans",
        "must_retrieve": "billing-policy-v4",
    },
    {
        "query": "how do I rotate an API key",
        "must_retrieve": "api-authentication",
    },
]

async def check_retrieval(tenant_id: UUID) -> list[dict]:
    failures = []
    for case in RETRIEVAL_CASES:
        results = await search(case["query"], tenant_id=tenant_id, limit=5)
        retrieved_docs = {r["document_slug"] for r in results}

        if case["must_retrieve"] not in retrieved_docs:
            failures.append({
                "query": case["query"],
                "expected": case["must_retrieve"],
                "got": list(retrieved_docs),
            })
    return failures

Run this nightly. When it starts failing, you know the problem is retrieval and not the model, which is half the debugging done before you have opened anything.

The useful property of this test is that it fails for reasons that have nothing to do with your code. New documents outranking old ones, an embedding model change, a chunking tweak that seemed harmless. Those are exactly the changes that have no deploy attached.

4. Treat an empty or weak retrieval as a real state

If nothing crosses your similarity threshold, that is information. Most pipelines throw it away and send whatever came back anyway.

MIN_SIMILARITY = 0.35

async def answer(query: str, tenant_id: UUID):
    results = await search(query, tenant_id=tenant_id, limit=6)
    relevant = [r for r in results if r["similarity"] >= MIN_SIMILARITY]

    if not relevant:
        await log_retrieval_miss(query, tenant_id)
        return "I don't have information about that."

    return await generate(query, context=relevant)

Those logged misses are the most valuable dataset you will build. They are a direct record of what your users want and your corpus does not cover, and the rate of them climbing over time is drift showing up in your metrics before it shows up in complaints.

The shape of the problem

A RAG system is not a feature you ship. It is two stores that have to agree, updated on different schedules by different code, with a model in front making whatever they return sound reasonable.

Everything above is the same idea in different places: make the disagreement visible. Hash so you can detect it, version so you can detect it, test retrieval so you can detect it, log misses so you can detect it.

Because the default behaviour of this failure is a well-written answer, delivered confidently, that happens to be six months out of date. Nothing in your stack is going to tell you about it.

I write about production AI systems and what breaks in them. More at ikramulmustafa.com, and a RAG agent with documented failure handling at github.com/ikramulmustafa/n8n-rag-lead-agent.

Your RAG Pipeline Is a Data Leak Waiting to Happen

Muhammad Ikram UL Mustafa — Fri, 09 Oct 2026 06:18:08 +0000

Your multi-tenant app is probably well isolated. Every query is scoped to a tenant, enforced centrally, and no individual developer has to remember it.

Then you add retrieval, and you quietly create a second data path that never gets the same discipline.

The failure mode

Tenant A asks a question. Semantic search finds the most relevant chunks. One of them came from Tenant B's document. The model reads it and answers fluently, using data the user was never allowed to see.

No error. No exception. No alert. Just a confident, correct-sounding answer built on someone else's data.

This is worse than an ordinary data leak, because there is no artifact. The user never saw Tenant B's document. They saw a paragraph of generated text that happened to contain its contents. Your logs show a successful request.

Why it happens

Most teams build the application layer first and get isolation right there. In Laravel it is a global scope. In Django it is a manager. Somewhere central, every query gets WHERE tenant_id = ? attached, and developers stop thinking about it because they no longer have to.

Then retrieval arrives as a separate feature. Documents get chunked, embedded, and written to a vector store. The code that does this often lives in a background job or an ingestion script, written by whoever was working on the AI feature that sprint.

Now you have two paths to the same data, and only one of them inherited the rules.

A concrete example

Here is ingestion that looks fine:

async def ingest_document(doc: Document):
    chunks = chunk_text(doc.content)
    embeddings = await embed_batch(chunks)

    for chunk, embedding in zip(chunks, embeddings):
        await db.execute(
            "INSERT INTO doc_chunks (document_id, content, embedding) "
            "VALUES ($1, $2, $3)",
            doc.id, chunk, embedding
        )

Spot the problem. There is no tenant_id on the row. The document has one, but it did not make it into the chunk table. You can still get back to it by joining through documents, which means somebody has to remember to join, which means eventually somebody does not.

And here is the retrieval that goes with it:

async def search(query: str, limit: int = 6):
    embedding = await embed(query)
    return await db.fetch(
        "SELECT content FROM doc_chunks "
        "ORDER BY embedding <=> $1 LIMIT $2",
        embedding, limit
    )

Clean, fast, and it will happily return any tenant's content.

What actually fixes it

1. Denormalise the tenant onto the chunk

Put tenant_id directly on the vector rows, even though it is already on the parent document. This is deliberate duplication. It removes the join, and it means the filter cannot be forgotten by accident.

ALTER TABLE doc_chunks ADD COLUMN tenant_id uuid NOT NULL;
CREATE INDEX ON doc_chunks (tenant_id);

2. Make the tenant filter impossible to omit

Do not leave it to the caller. Scope it centrally, the same way your ORM already does for SQL.

async def search(query: str, tenant_id: UUID, limit: int = 6):
    if tenant_id is None:
        raise ValueError("tenant_id is required for retrieval")

    embedding = await embed(query)
    return await db.fetch(
        "SELECT content FROM doc_chunks "
        "WHERE tenant_id = $1 "
        "ORDER BY embedding <=> $2 LIMIT $3",
        tenant_id, embedding, limit
    )

A required argument with no default is cheap and effective. A developer who forgets gets an error, not a leak.

If you are on Postgres, row level security is stronger, because it holds even if someone writes a raw query:

ALTER TABLE doc_chunks ENABLE ROW LEVEL SECURITY;

CREATE POLICY tenant_isolation ON doc_chunks
  USING (tenant_id = current_setting('app.tenant_id')::uuid);

3. Filter before the similarity search, not after

Retrieving the top 20 chunks globally and then discarding the ones belonging to other tenants is a correctness bug waiting to happen, and it also degrades quality. You are spending your result budget on rows you are going to throw away, so a tenant with little data gets worse answers.

Filter first. Let the index do the work.

4. Test isolation as a real case

Most teams test that retrieval returns relevant results. Very few test that it refuses to return someone else's.

async def test_tenant_cannot_retrieve_other_tenant_data():
    await ingest_for_tenant(TENANT_B, "The launch date is March 14th.")

    results = await search(
        "when is the launch",
        tenant_id=TENANT_A
    )

    assert all(r["tenant_id"] == TENANT_A for r in results)
    assert not any("March 14th" in r["content"] for r in results)

Write the test where Tenant A asks for something only Tenant B has, and assert the answer is empty. If that test does not exist, you do not know your isolation works. You are assuming it.

The harder version

Everything above is filtering, which is an application-level control. It runs in the same process as the bug it is guarding against.

If your tenant count is low and the data is sensitive enough, physical separation is stronger. A separate index, collection or namespace per tenant means a dropped filter cannot leak anything, because the other tenant's vectors are not reachable from that connection at all.

The tradeoff is operational. Per tenant indexes get expensive at thousands of small tenants, migrations have to run per tenant, and cross tenant analytics become their own problem. For enterprise B2B with tens or hundreds of tenants, especially under compliance pressure, it is usually the right call. For a product with a large free tier, it usually is not.

Filters are the floor. Physical isolation is the ceiling. Most teams should at least be standing on the floor.

The thing to take away

You can have perfect isolation at the SQL layer and still leak through retrieval, because they are two data paths and only one of them inherited your rules.

Treat the embedding pipeline as a data path with the same requirements as your database, because that is exactly what it is.

I write about production AI systems and what breaks in them. More at ikramulmustafa.com, and there's a RAG agent with the failure handling documented at github.com/ikramulmustafa/n8n-rag-lead-agent.