My CI Bot Remembers Every Failure, Thanks to Hindsight

The third time a build failed on the same flaky integration test, I realized the one component in my pipeline that hadn't learned anything was the tool I built to analyze failures.

That tool, PipelineDoctor, would read a failed log, call an LLM, and return a confident summary. Every time. From scratch. It had no idea it had explained this exact timeout twice before, or that the fix last time was pinning a dependency. So I gave it memory using Hindsight, an open-source agent memory system, and the way it behaves changed more than I expected.

What the system does

PipelineDoctor is a Python service built on FastAPI. When a CI run fails, a webhook sends the run to the service. The service pulls the logs through the GitHub API, extracts the failing step and error output, and produces a diagnosis with a suggested fix. Results land in a small dashboard and in SQLite for the run history. For unfamiliar errors it can also look up outside documentation before answering.

The flow is simple:

1. Webhook receives a failed run
2. Parser extracts the failing step, error lines, and changed files
3. Recall from Hindsight: have I seen something like this before?
4. LLM diagnoses the failure, with recalled history in the prompt
5. Retain the diagnosis, then later retain whether the fix worked

Steps 3 and 5 are where Hindsight lives. Everything else is ordinary plumbing.

The problem with stateless failure analysis

A stateless analyzer treats every failure as a first encounter. For CI that is the wrong model, because most failures are not novel. Flaky tests, expired tokens, a transitive dependency that shipped a breaking patch: these repeat, and the team's accumulated knowledge about them lives in people's heads and old pull request comments.

I tried the obvious fix first: store past diagnoses in SQLite and search them by error string. It worked for byte-identical errors and fell apart otherwise. The same timeout shows up with different hostnames, request IDs, and line numbers. Exact matching misses almost everything, and I did not want to build and maintain my own embedding and retrieval layer just to fix that.

Where Hindsight fits

Hindsight gives the agent a memory bank that you write to with retain and read from with recall. I write memories in natural language, including the context a human would use to describe the failure, and recall by describing the new failure the same way.

Here is the shape of the integration (simplified from the service):

from hindsight_client import Hindsight

memory = Hindsight(base_url=HINDSIGHT_URL)
BANK = "pipelinedoctor"

def recall_similar(failure):
    query = (
        f"Failing step: {failure.step}. "
        f"Error: {failure.error_summary}. "
        f"Repo: {failure.repo}."
    )
    return memory.recall(bank_id=BANK, query=query)

def retain_diagnosis(failure, diagnosis):
    memory.retain(
        bank_id=BANK,
        content=(
            f"Run {failure.run_id} in {failure.repo} failed at "
            f"'{failure.step}'. Root cause: {diagnosis.root_cause}. "
            f"Suggested fix: {diagnosis.fix}."
        ),
    )

The important design decision was what to store. My first version stored raw logs. That was a mistake: logs are noisy, and recall surfaced memories that matched on boilerplate. Storing a short, human-readable statement of the step, the root cause, and the fix produced far better matches, because that is what I want to compare against later.

The part that made it useful: retaining outcomes

Recalling past diagnoses is helpful. Recalling past diagnoses with knowledge of whether they worked is the real feature.

When the next run on the same branch finishes, PipelineDoctor compares it against the earlier failure. If the build is green, it retains something like:

memory.retain(
    bank_id=BANK,
    content=(
        f"The fix suggested for run {prev.run_id} ('{prev.fix}') "
        f"resolved the failure: the next run passed."
    ),
)

If the build fails again at the same step, it retains that the suggested fix did not work. Now the memory bank contains more than a record of problems. It contains evidence about which remedies hold up in this particular codebase.

In the diagnosis prompt, I include recalled memories and tell the model to prefer fixes that have a recorded success and to avoid ones that have a recorded failure.

What it does now

Here is a representative pair of interactions on the same underlying problem.

First time, empty memory bank:

The integration test job timed out while waiting for the database container. This is commonly caused by slow service startup. Consider increasing the timeout or adding a health check.

Reasonable, generic, and the kind of answer any model gives.

Later, after similar failures have been retained:

This matches a failure from earlier in this repo: the same step timed out waiting on the database container. The fix recorded for that failure, adding a health check before the test step, resolved it on the next run. A previous attempt that only raised the timeout did not. I suggest the health check.

The second answer cites history, names what worked, and names what did not. It is not smarter in the model sense. It simply knows things about this team's pipeline that the first answer could not.

Honest limitations

A few things did not go smoothly, and I would rather say so.

• Cold start is real. With an empty bank, the agent is exactly as generic as before. The value compounds with history, so early runs look unremarkable.
• Outcome tracking is heuristic. "The next run on the branch passed" is not proof that my suggested fix caused the pass. A developer may have changed three other things. I record the outcome as evidence, not as fact, and word it that way.
• Memory quality depends on what you write. Garbage in, garbage out. The raw-log mistake taught me this faster than anything else.
• Recall is not a guarantee. Sometimes a relevant memory is not returned. I treat recalled history as context for the model, never as a hard rule.

Lessons learned

1. Store conclusions, not transcripts. Retain what a human would want to remember: the cause and the fix, written plainly.
2. Retain outcomes, not just events. Knowing a fix worked or failed is worth more than knowing it was suggested.
3. Describe the new problem the way you described the old one. Consistent phrasing between retain and recall noticeably improved matches.
4. Treat memory as evidence in the prompt. Tell the model how to weigh it, including that a recorded failure should count against a fix.
5. Don't build the retrieval layer yourself. Letting Hindsight handle storage and retrieval saved me from maintaining embeddings and ranking code, so I could spend my time on what to remember.

Closing thought

The most useful thing about agent memory was not that it made answers longer or fancier. It made them specific. A CI tool that says "this happened before, here is what worked" is closer to a teammate than a log summarizer. If you want to try the same pattern, start with the Hindsight docs and retain your next failure's root cause in one plain sentence.

Built with Code.in.