Tag filter, metadata cites: how I structured Hindsight memories

Harriss — Tue, 29 Sep 2026 03:29:50 +0000

A lawyer asks: "Did Seabreeze start any work on the land before they had permissions?" The right answer isn't a paragraph. It's a paragraph with a finger pointing at the exact diary page where the company's own site manager admitted, under cross-examination, that survey pegs and poles went in during April 2025.

Without the finger it's gossip. With it, it's something you can put to a witness.

I built Tareekh, a memory for a litigator's practice, on Hindsight. This article covers the least glamorous part of it and the one I'd copy first into any other project: how each memory is shaped, what goes in tags versus metadata, and what I deliberately kept out of agent memory altogether.

The system in one paragraph

The lawyer uploads diary photos, certified order sheets, deeds and typed notes. Tareekh reads each upload, splits it into one entry per hearing, resolves which case each entry belongs to, and retains it in a single Hindsight bank for that lawyer. An agent answers questions with a small tool loop (find_case, recall_memories, reflect_on_memories, case_timeline), and every factual line in the answer carries a [n] marker pointing to a source.

Three questions for every field

When I designed the stored item, I kept asking three questions about every piece of information:

Will I need to filter recall by it?
Will I need to show it to the user next to a fact?
Does it need to be exact every time?

The answers sorted everything into three places: Hindsight tags, Hindsight metadata, and a plain SQLite registry outside Hindsight.

Tags are for filtering

Hindsight's recall and reflect both accept a list of tags and a match mode. That's how the agent narrows a search to one case, or to one judge across every case. Each retained hearing carries tags for all four entities it touches, plus two about the record itself:

def case_tags(case_id: str) -> list[str]:
    c = get_case(case_id)
    if not c: return []
    return [f"case:{c['id']}", f"judge:{c['judge_id']}", f"counsel:{c['opposing_counsel_id']}", f"client:{c['client_id']}"]

# in build_item():
"tags": registry.case_tags(cid) + [f"type:{entry.get('doc_type', 'other')}", f"author:{author}"]

The agent's tools expose exactly those axes and nothing else:

def _tags(case_id=None, judge_id=None, counsel_id=None) -> list[str] | None:
    t = []
    if case_id: t.append(f"case:{case_id}")
    if judge_id: t.append(f"judge:{judge_id}")
    if counsel_id: t.append(f"counsel:{counsel_id}")
    return t or None

resp = client().recall(
    bank_id=settings.bank_id,
    query=query,
    max_tokens=max_tokens,
    budget=budget,
    tags=_tags(case_id, judge_id, counsel_id),
    tags_match="any",
)

Tags are short, opaque ids ( judge: J1, not judge: Sri P. Suryanarayana Murthy ). Names change spelling between an order sheet and a diary page. Ids don't.

One trap I hit: tags also affect how Hindsight groups memories when it forms observations. By default it consolidates over an item's full tag set, which with four entity tags meant effectively one bucket per case. I had to add explicit observation_scores per judge, counsel and case before cross-case patterns could form. If you use tags heavily, read that part of the Hindsight documentation early.

Metadata is for citing

Metadata can't be filtered on. But it comes back, untouched, with every recalled fact. That makes it the right place for everything the UI needs to render a source chip:

"metadata": {"source_file": source_file, "upload_id": upload_id, "doc_type": entry.get("doc_type", "other"), "case_id": cid, "hearing_date": date, "author": author},

On the answering side, a small class called FactBook numbers every fact the tools return. The model sees them as 1 ... and is told to cite with those numbers. After the answer comes back, the markers are parsed and resolved to metadata:

def citations(self, answer: str) -> list[dict]:
    num = [n for grp in re.findall(r"\[(\d+?:\s*)\]", answer) for n in re.split(r"\s*", grp)]
    used = sorted({int(n) for n in nums if 0 < int(n) <= len(self.facts)})
    out = []
    for n in used:
        f = self.facts[n - 1]
        md = f.get("metadata") or {}
        out.append({"n": n, "text": f.get("text"), "case_id": md.get("case_id"), "hearing_date": md.get("hearing_date"), "source_file": md.get("source_file"), "doc_type": md.get("doc_type"), "upload_id": md.get("upload_id")})
    return out

The model never writes a filename or a date into a citation. It writes [3], and code looks up what 3 was. A marker pointing at a number that doesn't exist is simply dropped. That's a small thing, and it removes a whole class of fabricated sources.

The same metadata drives the note reader. Tap a citation and you get the original photo, the text that was read from it, the case, the hearing date and who wrote it.

The registry is for being exact

This was the decision I went back and forth on most. Hindsight does entity resolution, and it's good at it. Why keep a separate SQLite registry of cases, aliases, judges, counsel and clients at all?

Because some lookups must be deterministic. When a diary page says "Greenfield", it has to resolve to case C5 every single time, before anything is stored. When a cause list says "O.S. No. 57 of 2025", the match has to be exact. A semantic "probably this one" is fine for recall and it isn't fine for filing. Case resolution in the registry runs in tiers: an exact case-number match scores 100, a whole-word alias match scores 95 and up, and fuzzy matching only kicks in if neither hit, with a floor of 82.

The registry is also the source of truth for tags. A tag like case:C5 is only meaningful if C5 means one specific suit forever.

But I didn't keep the registry only in SQLite. At onboarding, each case is also retained into Hindsight as a stub:

"content": {f"Case {c['case_number']} ({c['short_name']}): {c['title']}. Nature: {c['nature']}. "f" {lawyer} represents the {c['represents']}; client {c['client_name']}. Court: {c['court']}, "f"before {c['judge_name']}. Opposing counsel: {c['counsel_name']}."),
"context": "case registry",
"tags": [f"case:{c['id']}", f"judge:{c['judge_id']}", ..., "type:registry"],
"document_id": f"registry:{c['id']}",

That gives Hindsight's own entity linking something to anchor names to, which improves the odds that "Ramesh garu" in a diary note and "Ramesh Gorle" in a sale deed get tied to the same cases.

What I left out of the registry on purpose: anything about how a judge or counsel behaves. The seed data has it. The bank doesn't get it. That has to be learned from the notes, through observations and mental models, or the claim that the agent learns means nothing.

Two kinds of source, ranked

Tareekh also remembers durable things the lawyer says in chat, like "we won't settle for Srinivas's half only." Those go into the same bank, but everything about them is marked:

"content": f"{{today}} CHAT MEMORY (said by {p['lawyer']} in a chat with Tareekh, not a court record): {text}",
"tags": tags + ["type:chat_memory", f"author:{lawyer}"],
"metadata": {"source_file": f"chat: {chat['title']}", "doc_type": "chat_memory", ...},

FactBook.render sorts chat memories after records and notes, and labels them CHAT MEMORY; in the numbered list. The system prompt says that if a chat memory conflicts with the record, the record wins and the answer says so. The label is in the content (so Hindsight's extraction sees it), in a tag (so it can be filtered) and in metadata (so the UI can style it). I'd rather over-label than have a remark from a chat quoted back as a court finding.

Before and after

Picture the naive version: store the hearing text with a case tag and nothing else. Answers would often be right, but the lawyer would have no way to check them short of digging out the diary page by hand. "The site manager admitted work began in April 2025" is a claim. It isn't evidence.

With the current structure, the answer to the Seabreeze question leads with the admission, cites it as [n], and the chip opens the 19 August 2026 page in the lawyer's own handwriting. The same question filtered by judge_id instead of case_id finds everything in front of that judge. Different question, same memories, and the change is one argument.

Takeaways

Separate filtering from display from truth. Tags filter, metadata displays, a registry is exact. Any time I was tempted to make one mechanism do two of those jobs, the design got worse.

Use ids in tags, names in content. Content is what Hindsight extracts facts from, so write it for a reader, with full names and case numbers. Tags are for machines.

Never let the model write a citation. Let it write a number, and resolve the number in code.

Label second-class sources everywhere. Content, tags and metadata. It costs nothing and saves you from your own agent.

Retain your reference data too. A registry stub in memory helps the memory layer link names you'd otherwise have to resolve yourself.

If you're deciding how to structure memories for your own agent, the Hindsight repo has the full retain schema, and Vectorize's explainer on agent memory is a good frame for why structure at write time pays off at read time

DEV Community: Harriss

Tag filter, metadata cites: how I structured Hindsight memories