Joseph Yeo

Posted on Sep 8

The Commit Survived. The Reason It Failed Didn't.

A few months ago, one of my agent runs failed.

The code survived.

The commit survived.

The reason it failed did not.

That turned out to be a much bigger problem than the failed run itself.

Git Remembered the Code

At the time, I was already treating Git as an important part of the agent system.

That felt obvious.

If an agent changes code, I want to know:

what changed;
when it changed;
what came before it;
whether I can reproduce the diff;
whether I can go back.

Git is extremely good at this.

After the failed run, I could still inspect the candidate code.

Nothing had vanished.

From a normal software-development perspective, that sounds fine.

It wasn't.

Because the question I needed to answer later was not:

What code did the agent produce?

It was:

What actually happened during the run?

Those are different questions.

And I had preserved only one of them.

The Missing Part Wasn't the Error Message

At first, this sounded like a logging problem.

Maybe I just needed better logs.

But the more I looked at it, the less useful that framing became.

An agent run produces more than code and more than logs.

There is an entire surrounding context:

what the agent was trying to do

what it was allowed to do

what it actually changed

which checks ran

what those checks observed

where the run stopped

what was known at that moment

what was still uncertain

If that context disappears, the next session has to reconstruct the story.

And reconstruction is dangerous.

Especially when an LLM is doing it.

A model is very good at turning incomplete evidence into a coherent explanation.

Sometimes that explanation is correct.

Sometimes it is simply the most plausible story that fits what survived.

Those are not the same thing.

A Failed Run Is Data

This changed how I thought about failure.

Until then, I mostly treated a failed run as something to fix.

run
↓
failure
↓
diagnose
↓
retry

The retry was the useful part.

The failure was an obstacle on the way there.

But if you are building an agent system that is supposed to improve over time, the failed run is also an observation.

It tells you something about:

the task;
the environment;
the agent;
the verification process;
the assumptions surrounding all of them.

And once the retry succeeds, that earlier failure may become more valuable than the successful run.

The success tells you that one path worked.

The failure may tell you why the previous architecture didn't.

So I started thinking about the lifecycle differently:

run
↓
observation
↓
interpretation
↓
next action

The important thing was no longer just preserving the output.

It was preserving enough reality that the interpretation could be challenged later.

This Is Where Agent Systems Get Weird

In normal development, a human often carries enormous amounts of context implicitly.

You remember:

That test wasn't meaningful because the environment was wrong.

Or:

That implementation technically passed, but we discovered the assumption was invalid.

Or:

Don't reuse that approach. We already tried it.

A team may carry this information through code review, Slack, issue trackers, meetings, or simply memory.

An autonomous system doesn't get that for free.

A new session sees artifacts.

It does not automatically inherit the meaning humans attached to them.

This becomes more important as the agent works for longer periods.

Suppose an agent comes back tomorrow and sees:

commit A
commit B
commit C

It can inspect the code.

But which of these represents progress?

Which one was rejected?

Which one was only an experiment?

Which failure discovered something important?

Which apparent success later turned out to be misleading?

A Git graph cannot answer all of that.

Because a history of source code is not automatically a history of the development process.

That distinction became one of the early reasons I started building Bezalel.

Source and Evidence Are Different Things

I had been collapsing several concepts into one vague idea of "history."

That was a mistake.

The source tells me:

This code existed.

Evidence tells me:

This was observed.

Those are not interchangeable.

A commit can survive while the evidence that explains it disappears.

Evidence can survive while later investigation shows that the interpretation was wrong.

And a perfectly preserved narrative can still disagree with the repository itself.

That last one became important later.

But the first lesson was simpler:

If future work depends on understanding what happened, preserve the observation—not just the artifact that resulted from it.

This sounds obvious in retrospect.

Most useful engineering lessons do.

More Logs Would Not Have Solved It

There is an easy overreaction here:

Record everything.

Every command.

Every prompt.

Every intermediate message.

Every token.

Every state transition.

I don't think that's the answer.

More data can make reconstruction harder, not easier.

A million lines of agent transcript are not automatically better evidence than ten machine-observed facts.

I became much more interested in selective evidence.

What facts would a future run actually need in order to understand the previous one?

What can the machine observe directly?

What should never depend on the agent's own description of what it did?

That last question has followed me through nearly every system I've built since.

If a machine can observe a fact directly, I increasingly don't want a model to be the canonical source of that fact.

For example:

bad:

"What files did you modify?"

agent:
"I modified A and B."


better:

observe repository state directly

The explanation can still be useful.

It just shouldn't replace the observable fact.

The Real Requirement Was Recovery

Eventually I realized this was not primarily an observability problem either.

It was a recovery problem.

What does a future process need to continue safely after the previous process disappears?

Not:

Can it reconstruct a convincing story?

But:

Can it recover enough verified state to know what happened and what remains unresolved?

That became a recurring theme in Bezalel.

The process should be able to stop.

The chat can disappear.

The model can change.

The machine can restart.

And the next run should not depend on me remembering the missing parts.

This is much harder than adding memory to an agent.

Memory tries to answer:

What happened before?

Recovery asks:

What previous facts are safe to depend on now?

I care much more about the second question.

What This Changed for Me

The failure left me with three simple rules.

1. Preserve observations, not just outputs

The code is one artifact of a run.

It is not the run itself.

2. Prefer machine-observed facts over agent narration

If a system can measure something directly, don't make the agent's report the only surviving version of that fact.

3. Design for the next session

A system that works only while the current conversation is alive is not yet a long-horizon autonomous system.

The next session should inherit enough reality to continue without inventing the missing story.

What This Does Not Prove

I am not claiming every agent run needs permanent forensic storage.

That would be expensive and probably useless.

I am not claiming more evidence automatically makes a system more trustworthy.

Bad evidence can be preserved perfectly.

I am not claiming Git is insufficient.

Git is excellent at preserving source history.

The mistake was expecting source history to preserve information it was never designed to represent.

And I am not claiming I have found the perfect boundary between useful evidence and noise.

I haven't.

That is still an active design problem.

The Commit Wasn't the History

The failed run eventually got fixed.

That part was ordinary.

What stayed with me was what I couldn't recover afterward.

The code was there.

The explanation wasn't.

More importantly, I could no longer distinguish with confidence between:

what actually happened

and

what I could plausibly reconstruct from what remained.

For an agent system, that gap is dangerous.

Because the longer the system runs, the more future decisions are built on previous ones.

And if previous reality disappears, autonomy slowly turns into archaeology.

So one of the first questions I started asking while building Bezalel was not:

How do I make the agent remember more?