With Hindsight, My Agent Says "We Fixed This in May"

Akepati Thrisaileswari — Sat, 07 Nov 2026 14:50:07 +0000

With Hindsight, My Agent Says "We Fixed This in May"
At 07:02 on a Monday, a recon check fails: fact_supplier_commit rows=0 for fiscal_week=FY27W06. The load job reports SUCCESS and says it wrote 60 rows. Staging has all 60. So where did they go?
I gave the same incident to the same model twice, with the same prompt, the same warehouse, and the same SQL tool. The first run worked out that the rows had landed under next week, FY27W07, and suggested "fixing the week lookup". The second run said the loader uses a week_start > today rule that picks the following week on the Monday a new week begins. It said this exact bug took down supplier_forecast_merge in May (INC-2026-0505), that reloading the staging file from SFTP didn't help back then, and that the FY27W07 rows need deleting after the fix. It got there in four queries instead of six. Neither run could see the loader's code at that point. The second one recognised the bug from history alone.
The only difference between the runs was memory. That second agent is PipelineMD, and this is what I learned building it on Hindsight, the open-source agent memory system from Vectorize.

What PipelineMD does
PipelineMD is an on-call triage agent for data pipelines. You paste an alert, a log excerpt, or a failed reconciliation. It:
Recalls similar past incidents from a Hindsight memory bank: postmortems, plus the team's verdicts on earlier triages.
Verifies the leading hypothesis with read-only SQL against the warehouse, and reads the failing pipeline's load SQL.
Answers with a root cause, a fix, the past incidents it matched, and a list of things not to try because they wasted time before.
Learns: the engineer marks the diagnosis Correct or Wrong, and that verdict is retained for the next time.
The stack is small on purpose: Groq running openai/gpt-oss-120b for the agent loop, Hindsight Cloud for memory, SQLite for a demo warehouse, and Streamlit plus a CLI on top. The memory bank holds 24 postmortems from five pipelines at a fictional logistics company. The failures are ones I've seen in real warehouse work: join fan-out on mapping tables, fiscal-week boundary rules, NULL-versus-zero mismatches, PK collisions after a constraint change, and ORA-12839 parallel-DML errors.

The part a database can't tell you
Here's the idea the whole project rests on: the most useful thing on-call knows isn't in the data. The data told the baseline that rows went to the wrong week. Only history could tell it which rule breaks on Monday rolls, that it's happened before, and what was a waste of time last time.
So I designed the memory around that. Every postmortem is retained as prose, not JSON, and the line I care about most is the first response and whether it worked:

def to_memory_text(self) -> str:
    """Render as a short postmortem. Hindsight extracts facts from prose better than from JSON."""
    first = "worked" if self.first_response_worked else "did NOT work"
    return (
        f"Incident {self.id} on pipeline {self.pipeline} ({self.severity}, category {self.category}).\n"
        f"Symptom: {self.symptom}\n"
        f"Error: {self.error_excerpt}\n"
        f"First response: {self.first_response} This {first}.\n"
        f"Root cause: {self.root_cause}\n"
        f"Fix: {self.fix}\n"
        f"Resolved by {self.resolved_by} in {self.time_to_resolve_min} minutes."
    )

"Reran the load. This did NOT work." is what turns into the agent's "skip this" list. Without it, an agent with memory only knows the answers. With it, it knows the dead ends too, and that's what actually saves time at 07:02.
Each postmortem goes in with its real incident date, its incident ID as the document_id, and tags for pipeline and failure category:

items = [
    {
        "content": inc.to_memory_text(),
        "timestamp": inc.opened_at,
        "context": "data pipeline incident postmortem",
        "document_id": inc.id,
        "metadata": {"incident_id": inc.id, "pipeline": inc.pipeline, "severity": inc.severity},
        "tags": inc.tags(),
    }
    for inc in incidents
]

The bank also gets a retain_mission telling Hindsight what to extract and to keep table names, error codes, plant IDs and fiscal-week keys verbatim. That matters more than I expected. dim_plant_region and FY27W06 are exactly the strings an incident gets matched on, and a paraphrase like "a regional mapping table" matches nothing. The Hindsight docs on retain and memory banks cover these options, and the mission text is worth tuning.
Recall doesn't filter by pipeline, on purpose
My first instinct was to filter recall to the failing pipeline's tag. I'm glad I didn't. The best match is often the same bug on a different pipeline that shares a table. A shipments overcount in EU-W matched INC-2026-0412, a fan-out on dim_plant_region that hit the inventory pipeline in April. Same two plants, 3104 and 5862, same two-rows-per-plant problem, different pipeline. A filter by pipeline would have hidden it.
Recall has to come back as incidents, not fragments
This is where my first version fell over. Hindsight's recall is good: in the Memory explorer, the top results for an EU-W overcount were exactly the right facts about dim_plant_region. But it returns facts that Hindsight extracted, and one postmortem becomes a dozen of them ("Root cause: …", "Fix: …", "First response … failed"), most without the incident ID. My first agent got 84 of them in a flat list, mixed in with week-boundary and PK-collision facts, and it latched onto a parallel-load fact from the same pipeline. Its answer was confidently wrong.
The fix used two things Hindsight already returns. Recall with include_chunks=True gives the original retained text behind each fact, and every fact carries the document_id I retained it under. So I group facts back into incidents and show the model the top four as whole postmortems:

INCIDENT_ID = re.compile(r"\bINC-\d{4}-\d{4}\b")

for h in hits:
    if h.document_id and INCIDENT_ID.fullmatch(h.document_id):
        incidents.setdefault(h.document_id, []).append(h)
    elif h.text not in {x.text for x in loose}:
        loose.append(h)  # triage feedback and observations: not an incident, whatever their document id

That regex is there because of a bug I hit live. Feedback memories have no incident ID, so Hindsight gives them a generated UUID as their document ID. My first grouping treated every document ID as an incident, the model dutifully reported "Matches past incidents: d6e6d866-6208-…", and my own citation check let it through. Now only incident-shaped IDs can be shown or cited as incidents, and an ID the model cites survives only if it actually came back from Hindsight. That also means the memory run can't make up a match. When nothing matches, it says so:

matches = _verified_matches(answer.get("similar_incidents", []), hits) if has_memory else []
if has_memory and not matches and not root_cause.startswith(NO_MATCH):
    root_cause = f"{NO_MATCH} {root_cause}"

On a TLS failure after a base-image rebuild, a failure mode with no history in the bank, the memory run starts its answer with "No matching incident in team memory." and still gets the fix right: the rebuilt image is missing the CA bundle. I find that sentence as valuable as a match. It tells the engineer they're on new ground.
It learns from the team, and you can watch it happen
Every triage ends with Correct or Wrong and an optional note. That goes back into the same bank:

client.retain(
    bank_id=self.bank_id,
    content=content,
    timestamp=datetime.now(timezone.utc).isoformat(),
    context="on-call triage feedback",
    tags=[f"pipeline:{pipeline}", "kind:triage_feedback", f"outcome:{'accepted' if accepted else 'rejected'}"],
)

After I confirmed one EU-W diagnosis, the next recall for that incident put my own verdict first, above the April postmortem: "dim_plant_region contained duplicate rows for plant_id 3104 and 5862, causing a join on plant_id only to multiply shipped_qty." The agent now sees it as [triage feedback] next to the matched incidents. That's the loop I wanted: the second engineer to hit a bug benefits from the first one's verdict, without anyone writing a wiki page.

The fan-out, with the loader in view
Incident 01, the EU-W overcount (193,280 shipped against a source total of 115,023, a ratio of 1.68), taught me one more thing. Memory got the agent to the right table quickly, but it kept getting the direction wrong: it called the source inflated, or treated 193,280 as the true total, because it couldn't see the load that wrote the numbers. So I gave both runs a read-only get_pipeline_sql tool that returns the load each pipeline really runs.
With the loader in view, both runs found the faulty line, JOIN dim_plant_region m ON m.plant_id = s.plant_id, and both proposed joining through a deduplicated mapping. That's the right outcome. A tool that shows the code should help an agent with or without memory. The difference was in what came with it. The memory run got there in three queries instead of four, and it tied the line to INC-2026-0412, where the same two plants caused the same fan-out in April, offering that incident's fix: join on (plant_id, lob) and enforce uniqueness on the mapping table. Knowing it's a repeat is what turns "fix this join" into "fix this join, and find out why the April uniqueness check didn't catch it".
What I'd tell someone building this
Retain the dead ends, not just the answers. "This did NOT work" is the highest-value line in every postmortem. It's what makes the memory run's answer shorter than the baseline's, not just more accurate.
Make memory the only variable. PipelineMD's baseline is a NoMemory class with the same interface. Same model, prompt, tools and warehouse. Without that, every "memory helped" claim is a guess. With it, you can point at query 1 versus query 4.
Don't hand the model a pile of fragments. Recall that's good at retrieval can still drown the model. Group by the ID you retained under, give it whole documents with include_chunks, and cap how many.
Check what the model cites. Models will cite whatever looks like an ID, including UUIDs. Only accept citations that came back from memory, and require an explicit "no match" when nothing did.
Free tiers are a design constraint. Groq's free tier gives 200k tokens a day per model, and an agent loop resends its whole prompt every step. A compact schema and capped tool output were what got me enough comparison runs to record a demo.
If you're new to the idea, Vectorize has a good primer on what agent memory is and how it differs from RAG. For me the short version is this: RAG would have found the April postmortem. Memory is what put "reloading from SFTP didn't work" and my own verdict from last week in front of the agent, and that's what made the 07:02 page a four-query problem.
The code for PipelineMD, including the seeded incident history, the demo warehouse, and the tests, is on GitHub: Thrisha999/pipeline-incident-agent.

DEV Community: Akepati Thrisaileswari

With Hindsight, My Agent Says "We Fixed This in May"