I Corrected My AI's Memory. It Cited the Original Mistake as Proof.

aryapathak — Mon, 19 Oct 2026 02:58:20 +0000

The correction went through. The claim now said the launch was in July. I asked the agent to re-check the plan, and it agreed: July, definitely. Then I opened the evidence panel to see why it was so sure.

It was quoting the original document. The one that said June. The one I had just corrected.

My memory repair had fixed the sentence and kept the lie underneath it, at confidence: 1.0.

The scene nobody demos

Every agent-memory demo I've seen shows the satisfying part: the agent remembers something. You tell it a fact on Monday, it brings it up on Thursday, everyone claps.

Nobody shows the next scene. The fact turns out to be wrong, and the agent has already built a decision on top of it. Maybe it drafted a launch plan. Maybe it proposed "book the venue for June 12." The wrong fact is no longer just sitting in memory. Things depend on it now.

Deleting the fact takes one line. Finding everything that quietly depended on it is the actual problem.

That's what I spent the WeMakeDevs Graph Hacks weekend on. The project started out as RECALL and ended up as Afterhack, because the question I kept coming back to wasn't "how does an agent remember?" It was "what do you do after it remembered wrong?"

Why a graph, and not a vector store

My first instinct was the usual agent-memory setup: chunk documents, embed them, retrieve by similarity. That's fine for finding things. It's useless for answering "if this is wrong, what else is wrong?", because similarity isn't dependency. Two chunks can be about the same launch without one being the reason for the other.

What I needed was the chain of because:

Source ──▶ Fragment ──▶ Claim ──▶ Decision ──▶ Action
                          ▲
                      Correction
         (everything to the right of it might now be wrong)

Once you write it down like that, "what did this wrong fact touch?" stops being a search problem and becomes a traversal, which is exactly what a graph database is for. I used FalkorDB, with one graph per workspace. The graph name is derived from a server-generated UUID, so no request can ever pick which graph it talks to.

The first rule: a quote or it didn't happen

Before worrying about corrections, I needed claims I could trust in the first place. So extraction got one hard rule: the model has to return an evidence_quote with every claim, and the API checks that the quote is an exact substring of the source. Not "close enough." Not "semantically similar." Character for character.

If the model paraphrases, the claim is dropped. If it invents a quote, the claim is dropped. The ones that survive get stored with start and end offsets, so the UI can highlight exactly which words a claim came from.

It's a blunt rule, and it throws away some claims that were technically fine. I was okay with that. A memory system that's occasionally forgetful is annoying. One that's confidently making things up is much worse.

The database that returned everything and nothing

My first real end-to-end run against FalkorDB went like this: every query succeeded, and none of them returned anything I could use. Workspaces I had just created were "not found." Claim lists came back as dictionaries with nonsense keys.

My row mapper was the obvious, boring thing:

return [dict(zip(result.header, row)) for row in result.result_set]

It took me an embarrassingly long time to actually print result.header:

>>> result.header
[[1, 'id'], [1, 'name'], [1, 'revision']]

Each column name wasn't a string. It was a pair: a type code, then the alias. I was using lists as dictionary keys and wondering why nothing matched.

# FalkorDB's client exposes each header as [type_code, alias], not just alias.
headers = [item[-1] if isinstance(item, (list, tuple)) else item for item in result.header]
return [dict(zip(headers, row)) for row in result.result_set]

A one-line fix. It was also the first of several times this weekend that the bug wasn't in my logic. It was in what I assumed a tool was handing me.

"A search hit is not evidence"

The agent side has two roles making two separate model calls. An Analyst gets tools to search claims in the graph and pull the exact source fragments behind them. It hands a proposal to a Coordinator, which reviews it and writes a draft decision.

Two things went wrong the first time I watched it run live.

First, the Analyst searched like a person typing into Google: whole phrases like "launch date for the product". My Cypher used CONTAINS, and no claim contains that exact phrase, so search returned nothing and the agent concluded there was no evidence. I added a back-off: if the full phrase misses, try the meaningful individual words and drop stopwords like "the", "find" and "date".

The second problem was worse. Sometimes the Analyst would find a claim and then skip the step of retrieving its source fragment. It would go straight to "here's my decision, based on claim X." It had seen the claim's ID in a search result, and that was enough for it.

That's when I wrote the comment I now think of as the project's motto:

# A search hit is not evidence by itself. Ensure the agent retrieves its persisted
# source fragments before the handoff even if the model omits the second tool call.

And then the part that actually enforces it:

allowed_ids = searched_claim_ids & retrieved_claim_ids
claim_ids = [str(i) for i in proposal.get("claim_ids", []) if str(i) in allowed_ids]

[CODE SNIPPET] this one, highlighted. It's the moment the post shifts from "I asked the model nicely" to "the model can't get around it."

The Coordinator can only cite a claim if that claim was both found and had its source pulled during this run. It can't invent an ID, and it can't reuse one it remembers from somewhere else. If nothing survives the filter, the decision gets written as needs_review, titled "No source-backed answer yet", and no action is created.

I tested that with a deliberately vague prompt in a workspace containing no relevant evidence. It did exactly that: an honest "I don't know", saved to the graph with the handoff trail, and no invented next step. Weirdly, that was my favorite result of the weekend.

The "oh…" moment

Now back to the opening.

Correction was the feature the whole project existed for. The flow was:

A reviewer types the corrected statement.
Preview shows which decisions and actions depend on the claim.
Apply marks the old claim superseded, creates a replacement linked with SUPERSEDES, and updates the decisions that depended on it.

I'd built it, it worked, the replacement claim showed up with the right text. Then I looked at what that replacement claim was supported by.

The first version did this:

CREATE (replacement:Claim {
  statement: $statement,
  evidence_quote: old.evidence_quote,  -- 👀
  confidence: 1.0,
  ...
})
CREATE (replacement)-[:SUPPORTED_BY {role:'correction_context'}]->(f)  -- the OLD fragment

I had copied the old claim's evidence onto the new claim. It made sense at the time: keep the context, so you can see where this came from. In practice, the replacement claim said "July" and pointed to a document that said "June" as its proof. At full confidence.

So when the agent later went looking for evidence about the launch date, it found the corrected claim, retrieved its "source", and quoted the original mistake back at me as support for the fix. Every individual step was doing what I told it to. The graph was faithfully keeping a link that was a lie.

That's when the thing I'd actually been building came into focus. A correction isn't an edit. It's a new piece of evidence from a different source, a human reviewer, and it needs its own provenance like any other source.

[BEFORE/AFTER] two cards side by side. Before: replacement claim "July", evidence quote "…launch in June…", confidence 1.0. After: same claim, evidence quote is the reviewer's own text, origin "Workspace reviewer correction", confidence_kind human_correction. This is the most important visual in the post. If you only add one, add this.

The fix:

CREATE (f:SourceFragment {
  text: $statement,
  origin_type: 'human_correction',
  origin_name: 'Workspace reviewer correction'
})
CREATE (corr)-[:HAS_FRAGMENT]->(f)
CREATE (replacement:Claim {
  statement: $statement,
  evidence_quote: $statement,
  confidence: 0.0,
  confidence_kind: 'human_correction',
  ...
})
CREATE (replacement)-[:SUPPORTED_BY {role:'reviewer_correction'}]->(f)

The correction now creates its own fragment, owned by the Correction node and labelled as coming from a human. The replacement claim cites that. Confidence starts at 0.0 with an explicit confidence_kind, because a reviewer typing a sentence isn't the same kind of confidence as a model matching a quote, and I didn't want one number pretending to mean both.

I reran the whole journey live: ingest the source, extract the claim, run the agent, correct the claim, then start a fresh agent task. This time the agent retrieved the replacement claim and quoted the reviewer's correction, not the superseded source.

Reachable isn't wrong

Fixing the evidence exposed a second mistake in the same feature.

My first propagation rule was blunt. If a decision used the corrected claim, mark it needs_review. If an action required that decision, mark it held. Simple, cautious, and wrong in a way that would make people stop using the tool.

Picture a decision backed by three claims. One gets corrected, and the other two still fully support it. Flagging that decision is like a smoke alarm that goes off whenever you make toast. After the third false alarm, nobody looks anymore.

Being connected to a wrong fact doesn't make a decision wrong. What matters is whether it still has enough support. So apply now re-checks:

def _reconcile_correction_dependencies(self, graph, claim_id):
    decisions = self.rows(graph.ro_query(
        "MATCH (c:Claim {id:$claim_id})<-[:USED_CLAIM]-(d:Decision) "
        "RETURN DISTINCT d.id AS id, d.status AS status LIMIT 100",
        {"claim_id": claim_id},
    ))
    review_ids = []
    for decision in decisions:
        alternatives = graph.ro_query(
            "MATCH (d:Decision {id:$decision_id})-[:USED_CLAIM]->(other:Claim) "
            "WHERE other.id <> $claim_id AND other.status IN ['proposed','accepted'] "
            "RETURN other.id AS id LIMIT 1",
            {"decision_id": decision["id"], "claim_id": claim_id},
        )
        if not alternatives.result_set:
            graph.query("MATCH (d:Decision {id:$decision_id}) SET d.status='needs_review'",
                        {"decision_id": decision["id"]})
            review_ids.append(decision["id"])
    # ...then hold proposed/approved actions that REQUIRE those decisions

Only decisions that lost their last supporting claim get flagged. Only actions that depend on those decisions get held.

The part that broke

If that function looks a bit clumsy, with a loop and a separate query per decision where you'd expect one neat query, there's a reason.

The first version was one neat query. I used a NOT EXISTS { ... } subquery to say "decisions with no other supporting claim" in one go. It passed the unit tests, because the unit tests used test doubles that never actually ran the Cypher.

The live correction test returned HTTP 500.

FalkorDB doesn't support that subquery form. My elegant query was valid Cypher in my head and nowhere else. I rewrote it as small, bounded, read-only queries that FalkorDB definitely supports, and the full live journey passed: apply, re-check, fresh agent retrieval.

Lesson: when your tests fake the database, they're testing your assumptions about the database. The only test that catches this kind of bug is the one that talks to the real thing.

Two people hit Apply at the same time

Corrections change shared state, so I had to think about the boring distributed-systems stuff.

What if someone previews a correction, a teammate changes the same claim, and then the first person hits Apply based on a preview that's now out of date? What if Apply gets double-clicked, or a request is retried after a timeout?

Every workspace and every claim carries a revision number. Preview returns the revisions it saw. Apply has to send them back, and the write only goes through if they still match:

MATCH (w:Workspace {id:$workspace_id}) MATCH (old:Claim {id:$id})
WHERE w.revision = $workspace_revision
  AND old.revision = $claim_revision
  AND old.status <> 'superseded'
...
SET old.status = 'superseded', w.revision = w.revision + 1

If anything changed in between, you get a 409 stale_correction_preview and have to look again. If the same correction was already applied, you get back the existing replacement with status: "already_applied", and the duplicate request also re-runs the dependency check, in case the first attempt died halfway.

That last part matters because of a tradeoff I chose not to hide: correction plus repair is several FalkorDB calls, not one atomic transaction. If the process crashes between "supersede the claim" and "hold the actions", you can end up with partial state. Retrying the apply cleans it up. That's a recovery path, not a guarantee, and turning it into a guarantee is the next thing I'd build.

Hiding the database from the user

Last surprise, and it had nothing to do with AI.

When I first showed the whole graph in the UI, it looked like a crime-scene evidence board: Workspace, SourceVersion, SourceFragment, AuditEvent, AgentRun, Handoff, arrows everywhere. Technically accurate. Completely useless for someone asking "what breaks if this fact is wrong?"

So the backend now returns a product-level view of the graph: Sources → Claims → Decisions → Actions, plus Corrections, with relationships named things like SUPPORTS instead of SUPPORTED_BY. All the plumbing is still stored in FalkorDB and still used for provenance. The user just doesn't have to read it.

[BEFORE/AFTER] the full internal graph next to the cleaned-up product graph, both rendered in React Flow. A quick visual win, and it makes the point that the hard part of graph UX is deciding what not to show.

What it looks like now

[ARCHITECTURE DIAGRAM] `

React + React Flow frontend that renders whatever graph view the backend returns
FastAPI backend that owns every rule: quote checks, the evidence gate, revisions, repair
FalkorDB as the graph and source of truth, one isolated graph per workspace
Model calls go through an OpenAI-compatible gateway on the server only, so no key ever reaches the browser
docker compose up brings up the whole thing locally, with FalkorDB on a persistent volume

The full loop runs live: add a source → claims extracted with exact quotes → Analyst searches and retrieves → Coordinator drafts a decision linked to real claim IDs → correct a claim → preview the impact → apply → a fresh agent task cites the correction, not the mistake.

What it doesn't do (yet)

In my live runs the Coordinator never actually proposed an action, so I've only seen "hold the action" work in store-level tests, not end to end.
"Alternative support" currently means "at least one other current claim". It doesn't yet ask whether that claim comes from an independent source, and it doesn't follow chains more than one step deep.
One fragment per source. No chunking yet.
Repair isn't atomic (see above).

What I'd tell myself on Friday night

A correction is evidence, not an edit. If your memory system lets a fix inherit the provenance of the mistake, the mistake never really leaves. It just gets a new headline.

Enforce evidence in code, not in the prompt. "Please only cite retrieved evidence" is a request. searched & retrieved is a rule. The model has no choice about the second one.

Connected isn't the same as broken. Flag everything near a wrong fact and people learn to ignore the flags. Re-check support instead.

Print the thing. The header bug, the NOT EXISTS 500, the copied quote: each one was me assuming what something looked like instead of looking at it.

I started the weekend thinking agent memory was about remembering. I finished it thinking the interesting part is what happens after you remember wrong, and how many things already acted on it.

DEV Community: aryapathak