Your RAG bot is wrong and the person who knows it cannot fix it

Your retrieval agent gives a wrong answer. Refunds take 14 days now, and it is confidently telling people 30, because that is what the help page said when you built the index.

Here is the uncomfortable part. The person who knows it is wrong is on the support team, reading the transcript, right now. And they cannot do anything about it.

They can file a ticket. An engineer picks it up, edits a markdown file, runs the ingest, waits for a deploy. Best case that is hours. Meanwhile the agent gives that same wrong answer to everyone who asks the same question.

That is not a prompt problem or a retrieval problem. It is an architecture problem, and it is worth naming because almost every RAG tutorial builds it in without mentioning it.

Why the index cannot be the fix

A knowledge index is built at deploy time. It is a file, produced by a build step, shipped by a release. That makes it exactly as editable as your source code, which is to say: not by the support lead, and not this afternoon.

So the obvious fix, "just update the docs", inherits the whole deployment pipeline. Rebuild, re-embed, redeploy. For a typo. For a number that changed.

You can make that pipeline faster. You cannot make it something a non-engineer runs at 4pm on a Friday.

An override layer instead

Add a second lookup that runs before retrieval and can be written at runtime by a human with no deploy:

import { createAgent, memoryCorrections } from '@recourse-ai/core'

const corrections = memoryCorrections()

createAgent({ index, model, corrections })

Writing one is a POST:

curl -X POST https://support.example/admin/corrections \
  -H 'Authorization: Bearer <token>' \
  -H 'Content-Type: application/json' \
  -d '{
        "question": "how do I get a refund?",
        "answer": "Refunds now take 14 days, not 30.",
        "author": "[e-mail]"
      }'

Nothing is rebuilt. Nothing is re-embedded. The next person to ask gets the corrected answer.

The index stays the source of truth for everything nobody has had to override. Corrections are the thin layer of "we know better than the docs right now", which is an accurate description of most support knowledge in the days after something changes.

Three details that make it work in practice

Store the author, and edit rather than replace

PATCH /corrections/<id>   { "answer": "..." }

The temptation is to skip PATCH and let people remove and re-add. It is less code and it looks equivalent.

It is not. Remove-then-add mints a new id, a new date and usually a new author. So fixing a typo silently rewrites the record of who decided this and when. That record is the only reason to store an author at all: six weeks later, when somebody asks why the bot says 14 days, you want a name and a date rather than a mystery.

Make update optional on the interface so a simple store can answer 501 and remove-then-add still works. But implement it where you can.

Wire it to the failures, not to a blank form

The feature that makes corrections actually get used is not the form. It is putting an Answer it button next to every question the agent could not answer, which opens the form with that question already filled in.

A page that lists questions your agent failed, and gives the reader no way to act on them, is worse than not having the page. You have shown someone a problem and handed them nothing.

Corrections are a signal, not just a patch

Every correction is a sentence saying "the documentation is wrong or missing here", written by someone who knows. Read them as a queue for your docs, not just as overrides. A correction that has been live for a month is a documentation bug nobody has fixed.

The workflow this actually creates

The point is not the endpoint. It is who is holding the pen.

Before: customer gets a wrong answer, agent notices, files a ticket, engineer schedules it, deploy goes out, fixed tomorrow. Five people, one deploy.

After: customer gets a wrong answer, agent writes the right one, next customer gets it. One person, no deploy.

That is a support lead's workflow rather than a developer's, and it is the difference between a retrieval agent your team can actually own and one that is permanently somebody else's problem.

Where this does not help

Honest limits, because an override layer invites overuse:

• It is a patch, not a rewrite. Fifty corrections means your documentation has drifted and you should fix the documentation.
• Matching is only as good as the question you wrote. A correction keyed to "how do I get a refund?" will not fire on a question phrased very differently, so keep an eye on which ones are actually being hit.
• Anyone who can write a correction can make the agent say anything. It needs the same authentication as your admin surface, and it should record who wrote what.

The implementation is MIT at github.com/ibrahimhajjaj/recourse. There is a live demo at recourse-demo.ibrhajjaj.workers.dev that will answer questions about its own corrections layer, which is either a good sign or an elaborate way to fail publicly.

Comments

[Ahmet Özel]

PATCH preserves the correction identity, but concurrent edits need another boundary. If two support leads open the same correction and save different answers, a last-write-wins update can quietly erase one decision while retaining the original ID and author.

An expected revision on the update would let the second writer receive a conflict and review the newer answer. I would keep the original creator separately from the editor and append the change history, including the reason. That makes the operational workflow you describe accountable without forcing every small correction back through an engineering deployment.