I Gave My Data Pipelines a Memory: Building PipeMind with Hindsight and Groq

Hindsight Taught My On-Call Agent Which Fixes Don't Work

A Synapse load fails on a Monday morning with a timeout. I paste the error into a generic LLM and get the answer it always gives: "increase the timeout." The agent I built for the same failure said something different: we already tried that, it didn't help, and the real cause was a paused SQL pool.

The difference isn't a better model. It's that the second agent remembers what happened last time, including the fixes that failed.

The problem: the fix exists, just not where you're looking

Every data team has a small set of failures that repeat. Upstream teams rename columns at month-end. A SQL pool gets paused right before an hourly job. A CRM export gets cut off halfway and nulls spike. The fix for each is known, but it lives in one person's head or an old Slack thread. At 2 AM, neither is available.

A stateless LLM makes this worse. It gives plausible, generic advice, and it has no idea that the plausible advice was already tried and didn't work. So I built PipeMind, an on-call agent for pipeline failures that uses Hindsight agent memory as its long-term memory. The source is on GitHub.

How it hangs together

The stack is deliberately small:

• Hindsight is the memory layer, in a single bank called pipeline-ops.
• Groq does the reasoning: openai/gpt-oss-120b, with openai/gpt-oss-20b as a fallback.
• Streamlit is the UI, and there's a terminal script (demo.py) for the same flow.

The Hindsight documentation describes three operations. I think of them in SQL terms:

Hindsight	SQL analogy	What PipeMind uses it for
retain	INSERT	Save an incident and its outcome
recall	SELECT ... WHERE relevant	Fetch similar past incidents
reflect	GROUP BY across history	Summarise fragile pipelines and recurring causes

The whole agent is one file, agent.py, built around those three calls.

The core idea: store the verdict, not just the incident

Storing incidents is easy, and any vector database does it. What changes the agent's behavior is storing whether the fix worked. When an engineer clicks Fix worked or Fix did not work, this runs:

def record_outcome(failure: dict, resolution: str, worked: bool) -> bool:
    verdict = "WORKED" if worked else "DID NOT WORK"
    content = (
        f"On {failure['when']}, pipeline {failure['pipeline']} failed with: "
        f"{failure['error']}. Resolution attempted: {resolution}. "
        f"Outcome: fix {verdict}."
    )
    hs.retain(bank_id=BANK, content=content)

There's no schema and no custom embedding step. The outcome is written as a plain sentence, and Hindsight handles extraction and indexing. Because "DID NOT WORK" is part of the memory, a failed fix becomes retrievable knowledge instead of something that gets forgotten.

That's the line I'd draw between retrieval and learning. Plain RAG can tell you what your docs say. An agent with outcome memory can tell you what your team actually tried.

Recall: what the agent sees before it answers

When a new failure arrives, the query is just the pipeline name plus the error:

def recall_context(failure: dict, k: int = 8) -> list[str]:
    query = f"{failure['pipeline']} failed: {failure['error']}"
    resp = hs.recall(bank_id=BANK, query=query)
    return [r.text for r in resp.results][:k]

I didn't build a hybrid search stack. Per the Hindsight docs, recall combines semantic, keyword, graph and temporal retrieval, and that combination matters for incident data. Error messages are full of exact tokens like table names and status codes, where keyword matching helps. Root causes are conceptual, where semantic matching helps. "Same pipeline, same month-end" is a time and graph relationship.

The recalled incidents go into the prompt with one instruction that carries most of the weight:

"Use PAST INCIDENTS if relevant: cite which incident, prefer fixes that "
"worked, and never repeat a fix that failed."

Asking the model to cite which incident it's relying on has a useful side effect. The UI has a Memories used panel that shows the exact memory units Hindsight returned, so the engineer can check the reasoning instead of trusting it.

Before and after

The UI has a Memory ON/OFF toggle and a side-by-side mode, so one failure gets answered both ways. For a Synapse timeout:

• Memory off: a reasonable generic answer. Raise the timeout, check for locks, retry.
• Memory on: a diagnosis that points to past incidents where a paused SQL pool caused the same error, notes that raising the timeout didn't resolve it, and recommends checking pool state first.

[Replace the two bullets above with your real output from side-by-side mode and add the screenshot. Real output beats my paraphrase.]

The generic answer isn't wrong. It's just the answer with the most common cause, not the answer with the cause we've actually seen here.

Reflect: turning history into a checklist

Once enough outcomes are stored, reflect synthesizes across them:

r = hs.reflect(
    bank_id=BANK,
    query=(
        "Which pipelines are most fragile? What root causes recur, which fixes "
        "worked vs failed, and what should we check before the next month-end run?"
    ),
)

This powers the Team insights button. Instead of one failure, you get a picture across pipelines: which ones break most, which root causes keep recurring, and what to verify before the next month-end run. That's the part that moves the agent from reactive to preventive.

Making it survive 3 AM

An on-call tool that fails when its dependencies fail is worse than useless, so agent.py has several defensive layers:

• The Hindsight client is configured with a timeout and a retry count (HINDSIGHT_TIMEOUT, HINDSIGHT_MAX_ATTEMPTS).
• If recall fails, the error is caught, the UI says the memory server is unreachable, and the diagnosis still runs. The offline notice is filtered out before it reaches the prompt, so the model never treats an error string as an incident.
• The LLM call retries twice on the primary model, then falls back to the second one.

The fallback taught me something. My connectivity test showed that one model I'd configured as a fallback had been deprecated. A fallback that silently doesn't exist is worse than none, so I replaced it. Test the failure path before you need it.

The data

Memory only looks convincing if the history looks real. I generated 48 synthetic incidents across 6 pipelines on Airflow, Azure Data Factory and dbt: 8 seeded baseline records plus 40 from gen_data.py. They include recurring patterns like month-end schema drift, paused SQL pools and truncated CRM exports, plus many "first fix failed, second fix worked" pairs. Those pairs are the useful ones, because they're what teach the agent which advice to avoid. The data is synthetic, so treat the demo as a proof of the mechanism and not a benchmark. The design is what I'd expect to carry over to real incident history.

What I learned

1. Failures are as valuable as successes. Storing "this did not work" is what stops the agent from repeating bad advice.
2. The outcome is the feature. Retrieving similar incidents gets you most of the way. Retaining the verdict is what makes the agent improve with use.
3. Make the memory visible. The Memories used panel builds trust because engineers can see why the agent said what it said.
4. Test your fallbacks. Mine was broken and I only found out because I checked.
5. Realistic data beats clever code. Believable pipeline names, error messages and dates made the whole system easier to evaluate.

The limitation: quality depends on engineers clicking the feedback buttons. If nobody records outcomes, the agent only knows the seeded history. Making that step nearly free is the most important thing to get right.

What's next

• Slack alerts so the diagnosis lands where on-call already is
• Real webhooks from Airflow and Azure Data Factory
• Separate memory banks per team

Try it

The code is at PipeMind on GitHub. To run it, seed the memory bank with python seed.py and start the UI with streamlit run app.py. You'll need a Groq API key and a Hindsight server, either self-hosted or in the cloud.

To learn more about the memory layer, see the Hindsight repo, the docs, and Vectorize's explainer on what agent memory is.

If your team has its own "only Priya knows how to fix this" stories, I'd like to hear them.