An OCR'd wrong date becomes a confidently wrong Hindsight memory

V.karthik sai ram — Tue, 29 Sep 2026 04:07:31 +0000

A vision model reads a smudged diary page and gets the 8 in "18 June" wrong. It reads it as a 3. On its own, that's a typo. Once it's stored in an agent's long-term memory, it's something worse: a fact the agent will cite, with a source chip, in a calm voice, to a lawyer standing in front of a judge.
I spent more time on the path between "the model read a photo" and "this is now a memory" than on any other part of Tareekh. This article is about that path, and why I put a person in the middle of it.
What goes in
Tareekh is a memory for a litigator's practice. The inputs are whatever the lawyer already produces, which in an Indian district court means a lot of paper:
phone photos of pocket-diary pages, handwritten, often shot at an angle under a tube light
certified copies of order sheets, with several dated rows per page
deeds, 1-B land records, encumbrance certificates, legal notices, police petition receipts, a WhatsApp export
typed notes and pasted text
Each upload is read, split into one entry per hearing, filed under the right case and retained in Hindsight. An agent answers questions from that memory with citations.

Tareekh's architecture. Everything on the left passes through extract, segment, repair and review before anything is retained.
The test practice I use has 79 uploads across five cases. The diary photos are deliberately rough: barrel distortion, chromatic aberration, vignetting, soft corners, sensor noise in the shadows, and the occasional hand-shake blur. If the pipeline only works on clean scans, it doesn't work.

A diary page from the test practice, as a phone camera would take it.
Why memory makes extraction errors worse
In a search system, an OCR error costs you a missed hit. In an agent with memory, it's much more expensive, for three reasons.
First, retain is not a dumb insert. Hindsight runs extraction on what you store, pulls out facts and entities, and later consolidates them into observations and mental models. A wrong date doesn't sit in one row. It gets built into a summary of how the case has gone.
Second, the agent is good at sounding sure. Tareekh's bank has a "Cite sources" directive, and every factual line in an answer carries a [n] marker that resolves to a real upload. That's the feature. It also means a wrong fact arrives with a citation attached.
Third, nobody re-reads their own diary. The whole point is that the lawyer stops keeping it in their head. If the memory drifts from the page, nobody finds out until it matters.
So I treated the retain call as a commit to a system of record, not a cache write.
The pipeline, stage by stage
Extract
Text files and .docx are read directly. PDFs use their text layer if there is one; scanned PDFs are rendered page by page and sent to OCR. Images go to a vision model with a prompt that's mostly about what not to do:
OCR_PROMPT = (
"Transcribe all text in this image exactly as written. It is either a lawyer's handwritten court diary page, "
"a notebook page, or a scanned court document. Rules: keep original line order and wording, keep abbreviations "
"and numbers exactly (case numbers, dates, amounts, diary numbers), SKIP words that are struck through, "
"keep any printed date header. Output only the transcription, no commentary."
)
"Skip struck-through words" is there because diary pages are full of corrections. If a lawyer crosses out "Tuesday" and writes "Thursday", a model that faithfully transcribes both has recorded two days for one hearing.
Segment and resolve
One LLM call gets the raw text and a compact listing of the case registry. It returns entries shaped like {case_id, hearing_date, author, doc_type, text, confidence, reason}.
A diary page covering three matters becomes three entries, and an order sheet with three dated rows becomes three entries. Lawyers write "beach land" or "Gorle" or "OS 214/24", and the model has to map those to a case id without inventing one.
Repair
Then plain code checks the model's work against things that are known to be true:
def repair(entries, source_file, hints, known_ids):
"""Validate the LLM's output against the registry and fill gaps from hints/filename. Pure function."""
for e in entries:
conf = float(e.get("confidence") or 0.5)
cid = e.get("case_id")
if cid not in known_ids:
cid = None # the model named a case that doesn't exist
if not cid:
found = registry.find_cases(text[:300], limit=1)
if found:
cid = found[0]["case_id"]
conf = min(conf, found[0]["score"] / 100)
date = valid_date(e.get("hearing_date"))
if not date and fallback_date: # IMG_20250618... gives us a date
date = fallback_date
conf = min(conf, 0.75)
if not cid or not date:
conf = min(conf, 0.3)
(Trimmed from ingest/segment.py .) Notice that repair only ever lowers confidence. A fuzzy alias match can't claim more certainty than its match score. A date taken from the filename is capped at 0.75. An entry missing either a case or a date drops to 0.3.
The file type also overrides the model on one point. If the upload is an image, the entry is a handwritten note. The model is only trusted to tell order sheets and documents apart from notes, because that's a judgment about content, and "was this handwritten" is a fact about the file.
Review
This is the part I'd defend hardest.
The review gate
Every entry lands in a review state first. Auto-confirm only happens when the upload asked for it and every entry in it clears the bar:

ingest/pipeline.py

confident = entries and all(e["case_id"] and e["hearing_date"] and e["confidence"] >=
settings.auto_confirm_threshold
for e in entries)
if hints.get("auto_confirm") and confident:
confirm(upload_id, [])
else:
_status(upload_id, "review")
The threshold is 0.8 by default. It's all or nothing per upload: one doubtful entry on a diary page holds back the other two as well, because they came from the same photo and the same OCR pass, and a page that confused the model once may have confused it twice.
Confirm accepts edits (case, date, text, author, or reject) and only then builds Hindsight items. Two details there came from things breaking. A multi-page document yields one entry per page with the same case and date, and Hindsight rejects two items with the same document_id in one batch, so entries are merged by (case_id, hearing_date) before retaining. And retain always runs with retain_async=True, so Hindsight queues the work and retries on provider rate limits instead of failing the confirm.

The Add notes screen. Tareekh reads the upload, files each hearing under its case, and asks you to check anything it isn't sure about.
Before and after
Here's the failure the gate exists for. A diary page has a faint header, and the model takes its best guess at the hearing date. The most tempting guess is the next date written at the bottom, because it's often the most prominent date on the page ("next date 5 Oct", in capitals). Without a gate, that becomes a memory saying a hearing happened on a day it was only scheduled for. Ask "what happened last time" and you get a cited, fluent account of a hearing on the wrong day. Nothing in the system would ever flag it.
After the gate, the same page shows up in review, flagged. The reason field says "date from filename/hint", confidence is 0.75, and the lawyer taps once to accept or fix. The segmentation prompt also now says in so many words: Dates like "next date 5 Oct" are NOT the hearing date.
The uploads that should end up in review are exactly the ones I'd want a human to look at anyway: blurred headers, pages mentioning two cases by the same party name, documents that concern three suits at once.
Source-linked memories
Every memory keeps a link to its source. The note reader shows the original photo above the text that was read from it, so a wrong read is easy to spot.

What I learned
Treat retain like a database commit. The memory layer is going to extract, link and summarize whatever you give it. Garbage in turns into confident garbage out. Validate before you retain, not after.
Let code lower the model's confidence, never raise it. The LLM's self-reported confidence isn't calibrated. I don't trust it as a number. What I do trust is a rule like "if the case id isn't in the registry, it's wrong." Deterministic checks should only ever pull the number down.
Keep the source attached forever. metadata.source_file and upload_id go into every Hindsight item and come back with every recalled fact. When something looks off, the lawyer is one tap from the original photo. That's the real safety net, more than any threshold.
Review is a cost, so spend it carefully. A tool that asks you to confirm every entry won't get used. All-or-nothing per upload, with a 0.8 bar, is my current compromise, and I'm not sure it's the right number. It's an environment variable for that reason.
Make re-uploads safe. document_id = upload:case:date means fixing a mistake and re-confirming replaces the memory rather than adding a second, contradictory one next to it.
If you're feeding documents into agent memory, the extraction step deserves as much design as retrieval. The Hindsight docs explain what retain does with an item once it's stored, which is the best argument for being careful about what you store. For the bigger picture of why this layer is different from a vector index, Vectorize's piece on what agent memory is is worth ten minutes.

DEV Community: V.karthik sai ram

An OCR'd wrong date becomes a confidently wrong Hindsight memory

ingest/pipeline.py