DEV Community: Guillermo Leyendeker

Everything that breaks once you stop watching the screen

Guillermo Leyendeker — Mon, 05 Oct 2026 12:31:11 +0000

An agent that works fine in a ten-minute demo and one that runs on its own for months are not the same system. The difference isn't architectural: it's dozens of edge cases that only surface with time on the clock.

Clocks drifting out of sync. A machine that suspends overnight. A parser that breaks on an unexpected prefix. A browser that grows without a ceiling until it eats the memory. A rhetorical question, written by the agent itself, that stalled the entire pipeline waiting for an answer nobody was going to give.

None of these appears in a diagram, and together they consumed as much time as every design decision before them. If you're about to leave something running unattended, this is the list you'd want to have read first.

Clocks

My favourite, because it's the hardest to suspect. For a while, the same task could read as alive or dead depending on which part of the system was looking at it.

The cause: the log file's name was generated in one timezone and its status was evaluated in another. Neither was "wrong" — they were simply different. The result was that a perfectly active run showed up as stopped, or the reverse, depending on the time of day.

Timezone mismatches are the kind of bug you don't find by reading code, because each piece looks reasonable on its own. You find it by watching the screen and noticing it says one thing while the system does another.

The machine that goes to sleep

The watchdog kills processes that have been running too long. Simple, until the machine suspends overnight.

On waking, the watchdog computed elapsed time in wall-clock hours and found processes with eight or ten hours of "activity" that hadn't executed a single cycle. And it killed them, correctly from its own point of view.

It had to be taught to subtract host suspension time from the calculation. It's a three-line adjustment that only occurs to you after losing work to it.

Fragile parsers

The end-to-end test progress bar sat frozen at zero. The suite ran fine, the tests passed, but the indicator never moved.

The reason: the test runner includes the project name on each output line when more than one project is configured, and my parser expected the format without that prefix. One extra token and the counter stayed at zero forever.

It's a reminder of something I badly underestimated early on: almost all integration with external tools runs through parsing their text output, and that output changes without notice, through configuration or version.

Memory that grows on its own

The browser running the interface reviews grew in memory without a ceiling. It wasn't a leak in my code: it was the normal behaviour of a browser held open for hours, accumulating tabs, contexts and cache.

The fix wasn't elegant and didn't need to be: recycle the containers when they cross a RAM threshold. Long-lived processes don't get fixed, they get restarted periodically.

States that lie on screen

There was an entire family of bugs around the interface showing things that weren't so.

Rows spinning indefinitely with an activity indicator, because the active task's identifier was compared incorrectly and no match ever switched it off. Live runs reading as stopped. Steps that did nothing but claimed the run button.

They all look cosmetic and none of them are, because the state on screen is what you use to decide whether to step in. An indicator that lies makes you wait for something already finished, or interrupt something that was working fine.

Operations that can't tell empty from broken

Read-only steps — the ones that explore code without modifying anything — still triggered the automatic push on completion. Since there was nothing to push, the operation failed, and that failure was reported as though the task had gone wrong.

It's the same pattern as the exhausted turns from the previous entry: the system couldn't distinguish "there was nothing to do" from "something went wrong". And that confusion generates noise which eventually teaches you to ignore the warnings, the worst possible outcome.

The rhetorical question that jammed everything

One skill ended its runs with a question along the lines of "shall we continue with the next part?". It wasn't expecting an answer: it was a closing formula.

But the orchestrator detects runs left waiting on user input, so they don't hang indefinitely. A question at the end, however rhetorical, triggers that detection and aborts the whole sequence.

The fix was to explicitly forbid closing on a question. It's also the case that gave rise to the self-improvement loop from the fourth entry: it was the canonical example of "I caught this by hand, the system should be catching it".

Other people's infrastructure falling over

When the provider's API returned a server error, the entire queue stopped. A thirty-second blip left the afternoon's work standing still.

Now those errors retry the step instead of killing the sequence. It's the same idea as in the entry on failure policy: you have to separate failures of the work from failures of the environment, because they call for different responses.

The pattern behind all of them

Looking at the full list, half of these cases weren't discovered by reading code. They were discovered by watching the screen and noticing something said one thing while the system did another.

And almost none of them are agent problems. They're old distributed-systems problems: desynchronised clocks, miscalculated timeouts, long-lived processes accumulating state, fragile parsers, external dependencies failing. They reappear intact because an agent running for hours is, for all practical purposes, an unreliable long-lived process.

If I had one piece of advice for someone about to build something like this, it would be: the interesting part — the prompts, the architecture, the retry policy — is the part you'll solve quickly. The part that eats your time is what keeps the system truthful when nobody is watching it.

The full arc

That's the path of these four months. It started as a screen with buttons for launching agents, with two commits devoted to one button's icon, and ended as a control system with distrust, budgets and auditing.

Today it has closed 2,095 fixes and features, plus 654 audits, on real projects.

The screen is still there. It's no longer the important part.

Your agent ran out of turns. Your system logs it as an error.

Guillermo Leyendeker — Wed, 30 Sep 2026 18:00:30 +0000

An agent that runs out of budget and an agent that gets it wrong end the same way in your terminal: exit code 1.

For two months my system treated them as the same case, which is wrong, because they call for opposite responses: one resumes, the other gets fixed.

Behind that confusion sits a number almost nobody calibrates and everybody leaves as it came: how many turns to give the agent before cutting it off. With no limit, it can spend hours circling something it won't solve. With too low a limit, it abandons work that was two steps from done.

That number isn't a guess. Only in August, with 30 days of real runs in the database, could I build a table of budgets per task type instead of using the same one for everything.

How it was before: no limit and a hammer

At first there was no turn cap at all. Every step ran with the global default configuration, and the only brake was a watchdog that kills stuck processes after an hour.

A time-based watchdog is a blunt instrument. It can't tell an agent working well on something hard from one stuck in an unproductive loop: both burn an hour the same way. All it guarantees is that nothing runs forever.

That watchdog also had its own operational lesson, which I cover in the final entry: the machine suspends overnight, so suspended time had to be subtracted from the calculation. Otherwise it killed healthy runs over hours that had never actually elapsed.

In June I added a global turn cap as a backstop against runaway runs. Better than nothing, but still a single number for tasks that differ wildly from one another.

The table the data produced

In August the cap became per task type, calibrated by looking at how the last 30 days of runs had actually behaved.

What I like about the result is that the table has a logic you can explain without looking at the numbers:

The verifier is the most generous, at 160 turns. That makes sense: it reviews someone else's work, and reviewing properly means reading far more than you write.
The fix implementer follows at 150. A fix usually starts without knowing where the problem is, so a good share of the budget goes into finding it.
The frontend implementers sit around 90 to 110, depending on whether they work on the admin or the public application.
The explorer gets 60. It only has to map the terrain and return the touch points: if it needs more than that, something is wrongly framed.

Above all of it sits a global safety ceiling of 200 turns, which no task should ever reach and is there just in case.

The mistake that took me two months to understand

And now the part I consider most important in this entry, because it isn't about the number but about what running out of it means.

For the first two months, an agent that exhausted its turns showed up in the system as "exit code 1". That is: exactly like any other error.

That's a serious diagnostic problem. Running out of budget and failing are completely different things:

A real failure means something went wrong and you need to investigate what.
Exhausting the budget means the work was in progress and got interrupted. Nothing is broken: something is unfinished.

Confusing them sent me hunting for bugs that didn't exist, more than once. And the inverse, which is worse: assuming certain failures were "the turn limit again" when they were legitimate errors.

In August I separated them. Running out of budget now has its own message, states clearly what happened, and no longer kills the sequence: the step goes back to the queue instead of dragging the whole task into failure. Which is correct, because the work done up to that point still stands.

Budgets are editable without touching code

A small decision with a large impact: the per-task caps are editable from the interface, with no restart.

The reason is that calibrating them isn't a one-time job. The model changes, project complexity changes, the skills change. A number you adjust by editing code and redeploying is a number that in practice never gets adjusted.

The same criterion, applied to the model

There's a related decision I made at the same time with the same shape: not every step deserves the same model.

The verifier — the most expensive step in the system, that 29% of the bill I described in the entry about measurement — deliberately runs on a cheaper model. It verifies against explicit criteria, and that doesn't require the most capable model available.

Meanwhile, the component making autonomous decisions about what to do next runs on the highest tier with maximum reasoning effort. It's the one that executes least often and has the worst consequences when it gets things wrong.

It's the same idea as the turn budgets, applied along another dimension: budget should follow consequence, not volume.

What I learned

A turn budget looks like a technical parameter and is a product decision: it defines how much autonomy each task gets before someone interrupts it.

And the expensive mistake isn't getting the number wrong — that's corrected with data, and the data shows up on its own if you're measuring. The expensive mistake is failing to distinguish "ran out of budget" from "failed", because the first is resumed and the second is investigated, and confusing them wastes your time in both directions.

What's left to tell

Everything I've covered so far is design decisions: things you think about, plan and implement.

The final entry is about the other half of the work, the half that appears in no architecture diagram and consumed half the time: everything that breaks when you leave the system actually running, for months, on a machine that suspends, restarts and runs out of memory.

You restart the server and the agent loses all ten tasks

Guillermo Leyendeker — Mon, 28 Sep 2026 12:30:19 +0000

You leave the agent working through a queue of ten tasks. You need to deploy a server improvement, you restart it, and all ten are gone.

Not just the pending work: also any trace of which one was running and what state it was left in. The queue lived in the process's memory, so any restart — a deploy, an update, a power cut — wiped it entirely.

And the server you're restarting is the very one you're improving every day. Every improvement forced a choice: wait for the queue to drain, or lose the plan.

The fix isn't "put the queue in a database" and done. What you actually have to decide is what a half-finished task means, and who gets to change its state when the process comes back up.

How it used to work

The original queue was an in-memory list. You ticked the tasks you wanted to run on screen, the order was the order you'd selected them in, and the system processed them one by one.

It worked as long as nothing interrupted it. With two serious problems.

The first is the one above: every time I wanted to deploy an improvement to the orchestrator, I had to choose between waiting for the queue to drain or losing the plan.

The second, quieter one: the first failure aborted the whole queue. An error in the third of ten tasks cancelled the remaining seven. And those seven had nothing to do with the one that failed: they were simply behind it in line.

The redesign

The queue moved to the database, with one design decision I particularly like: position is a real number, not an integer.

That allows inserting between two elements without renumbering the whole list. If you want to slot something between positions 3 and 4, you assign it 3.5. It's an old technique and it solves at the root the problem of reordering a persisted list without writing fifty rows every time someone drags one.

The order is curated by hand, mixing fixes, features and audits into whatever sequence makes sense that day. You reorder by dragging, as long as tasks are still waiting. Whatever is running is pinned and locked, which is the bare minimum: you can't reorder something that already started.

What to do when a task fails

There was a product decision disguised as a technical detail here, and I resolved it by not resolving it: I made it a per-run option.

You can choose for the queue to carry on with the rest, flagging the failed one, or to stop and leave everything else waiting. Both positions are defensible depending on the case. If you're processing ten independent fixes, you want it to continue. If the third failed because the environment broke, you want it to stop before the remaining seven fail for the same reason.

Later I added a stricter rule that turned out to be the most useful in practice: the queue doesn't resume past an unresolved failure. You can continue, but someone has to have looked at that failure and decided what to do about it. Without that, the queue stays put.

It's deliberately inconvenient. The alternative — always continue — lets failures pile up unseen, which is exactly the problem behind the 1,771 unprocessed incidents I described a few entries back.

A single execution mechanism

A side effect of the redesign I hadn't anticipated: the "run everything" and "repair everything" buttons stopped being separate mechanisms.

Before, each was its own pipeline, with its own advancement logic and its own bugs. Now they're simply shortcuts that bulk-enqueue with a preset order and start the queue. One execution mechanism, all of it visible and reorderable.

It's the kind of simplification that appears on its own once the right abstraction finally exists. While the queue was ephemeral, having parallel pipelines seemed reasonable. With a persistent, curatable queue, it makes no sense at all.

Actually surviving a restart

Persisting the queue solves half the problem. The other half is what happens to whatever was running at the moment of the restart.

That task lands in limbo: the database says it's in progress, but the process executing it no longer exists. If nobody does anything, it sits there forever, occupying the head of the queue and blocking everything behind it.

In August I added the two missing pieces: graceful shutdown, so in-flight runs are marked before dying, and startup reconciliation, which checks what's declared as running with no process behind it and resolves it.

And a rule that seems obvious and took me a while to find: re-read the task's status before launching its row. Because between enqueuing and its turn arriving, anything could have happened — the task might have been completed by another path, or postponed. Launching blind against a stale snapshot is a constant source of duplicated work.

Why this sums up the whole project

Of everything I built over these months, the persistent queue is what best captures the thesis: in a system with agents, the state of in-flight work matters as much as the work itself.

An in-memory queue works perfectly until the first restart, and the first restart always arrives at the worst moment. It isn't an optimisation: it's the difference between a tool you use while you're watching and a system that runs when you're not.

Today the queue is the system's only execution path. There's no way to run a task outside it, and that — which felt like an annoying restriction at first — turned out to be what makes everything else observable.

With the queue surviving restarts, the system could run on its own for hours with me nowhere near it. And that raised a question I'd never asked, because I'd never let it run that long: how long to let it think before interrupting. That's what the next entry is about.

Verifying two lines from the agent cost me an hour

Guillermo Leyendeker — Wed, 23 Sep 2026 18:01:19 +0000

The agent closes a two-line task. Verifying it hadn't broken anything took me close to an hour.

And not one check was superfluous. I'd added the end-to-end tests because they caught a real bug. The security review, because it caught another one. Each had earned its place, and for exactly that reason none of them could be skipped.

It's a natural slope: you add a check because it helps; because it helps, you don't want it skipped; so you make it mandatory. Repeat that five times over three months and you end up with an hour of checks hanging off every task, however small.

What unblocked it wasn't deleting checks or making them faster. It was changing the question: not is this check useful? — they all were — but what decision does it block?. Some have to stop the task. For others, stopping the deploy is enough.

The expensive part was the interface review

Within the gate, the slowest check was the model-assisted interface review: an agent that opens the browser, walks the application and judges whether what was implemented looks and behaves as it should.

It's valuable. It catches things no automated test detects. And it's enormously expensive in time, because it means bringing up the environment, navigating, waiting on every screen and reasoning about what it sees.

Running that on every task was the equivalent of asking a designer to review the entire application every time someone changes a line of code.

The right question isn't whether the check is worth it

The change of question reordered everything, and the reason is economic before it is conceptual: a control that stops a task from closing and one that stops a deploy to production are not the same thing. Their waiting costs are completely different. The first is paid dozens of times a week; the second, once per release. Mixing them is what leads a team to eventually disable both.

That's where the two tiers came from.

Tier 1: what blocks each task

Fast, deterministic and per task. Three things: the full backend suite, the end-to-end tests belonging to that task running headless, and a security trace.

The key part is "belonging to that task". Not the full end-to-end suite: only the specs covering what was just touched. That's what takes the time from an hour down to minutes.

Tier 2: what blocks the deploy

Slow, expensive and batched. That's where the full end-to-end suite and the model-driven interface review went, the latter running over tasks Tier 1 didn't cover.

That batch runs at two moments: before deploying, blocking, and every night as a safety net. If something fails, it doesn't jam the day's work — it jams the release, which is where that control actually matters.

On screen the split was made visible with two separate columns, one per tier, each with its own state. Visibility matters: if a check doesn't block, it has to be crystal clear that its result still exists and that someone is going to look at it.

The delicate part: changing something everything depends on

This is the part that gave me the most trouble and the one least often written about.

The quality gate is a load-bearing piece. The completion steps refuse to mark a task as done if the gate didn't pass. And that pass, as it was written, explicitly required the interface review's result.

Which means: if I did the intuitive thing — pull the interface review out of the per-task gate — the entire system would stop being able to close tasks. Not some of them: all of them. Every completion would start failing while looking for data nobody produced anymore.

That's why the plan carries a rule written in capitals: if you stop producing the signal before you stop requiring it, every completion breaks.

The implementation order was fixed in five stages, and the first is the counterintuitive one:

First, decouple the requirement. Make the pass stop requiring the interface review. It's a backward-compatible change: if the signal is present it's ignored, if it's absent it no longer blocks. At this point the system behaves exactly as before.
Then add the new Tier 1 checks, which is purely additive.
Only at the end, make the skill stop producing the per-task signal. It's safe now, because nothing has required it for two steps.
The screen columns and the batch runner, which are net-new and don't touch the existing path.

It's the kind of sequence that looks bureaucratic when you write it and saves your day when you execute it. Had I started with the obvious step — removing the interface review — I would have broken task completion across every managed project simultaneously.

A detail that avoided a lot of pain

Every new check reads the project's configuration and skips itself gracefully when the project doesn't declare what it needs.

Some managed projects have no end-to-end tests at all. If the gate demanded them, those projects would be permanently stuck behind a requirement that doesn't apply. The rule that stuck is: the gate can't demand what the project doesn't declare it has.

It sounds obvious written down, but it's the kind of thing you discover when the same mechanism runs across projects of different shapes.

The outcome and what followed

The goal stated in the plan was to bring the per-task gate down from roughly an hour to minutes, across every project at once. That was achieved.

And with the system finally running fast, a problem the slowness had been masking came into view: every time I restarted the server, I lost the entire work plan. That's what the next entry is about.

I used my agent's memory 1,112 times and I don't know if it helped

Guillermo Leyendeker — Mon, 21 Sep 2026 12:31:15 +0000

Your agent has memory: it stores what went well and what went wrong, and injects it into itself when it starts a new task. It's one of those features you switch on once and never look at again.

The uncomfortable question is simple: if you turned it off tomorrow, would you notice?

Mine ran 1,112 times in 30 days. When I sat down to evaluate it I couldn't say whether it helped, and not for lack of instrumentation. Memory is the one component in the system that never complains, and that's why more than a month went by without my looking at it once.

What it actually does

The system stores two kinds of thing. Patterns that worked — ways of solving something that produced good results — and known failures, so they aren't repeated. At the time of measuring there were 127 patterns and 180 failures on record.

When an agent starts a task, the system searches that store for whatever resembles the work about to be done and injects it into the context. The search is by trigram similarity, with a threshold of 0.30 and a cap of three results per query.

It's simple, it's cheap and it runs without errors. That last point turned out to be the problem.

The component that never complains

Everything else in the system makes it onto my review list because it makes noise. The quality gate fails and stops a task. The watchdog kills a process. An agent runs out of turns. The queue jams. Each of those events produces a signal that forces me to look at it.

Memory doesn't. It gets injected, the agent works, the task closes. No error, no alert, nothing to review. It became invisible through good behaviour.

And so a good month went by with over a thousand accumulated injections, without anyone — me — asking whether those injections were improving anything or simply taking up room in every agent's context.

The textbook technical answer, and why I didn't take it

When I finally sat down to look at it, my first reaction was the obvious one: trigram search is primitive, this should move to embeddings.

It's true that it's primitive. It compares character sequences, not meaning. It's the kind of retrieval anyone would replace today with a vector search, and there are libraries to do it in an afternoon.

I didn't, and I think that was the right call. Improving retrieval without an impact metric is betting blind, and paying for the privilege: embeddings carry a per-query cost, so I'd be spending more to improve something I don't even know contributes anything.

The right question wasn't "how do I make retrieval better?" but "is retrieval making any difference?". And I had never asked that second one.

The infrastructure to measure it already existed

Here's what surprised me most: I could measure it, and I didn't know.

Every memory injection records the identifier of the session it happened in. That, which I'd added without thinking about it much, enables a natural experiment: there are sessions with memory injected and sessions without, on the same system, over the same period, doing comparable work.

It's an A/B that's been running for months. All that was missing was looking at it.

The cross-reference to make is against two things also on record: each session's incidents and the quality gate's outcome. The questions are concrete. Do sessions with memory injected repeat fewer reads of the same file? Do they run fewer searches that find nothing? Do they pass on the first attempt more often?

If the answer is yes, memory helps, and only then does investing in better retrieval make sense — with the cost justified by a measured effect.

The question almost nobody asks

But there's a third possible outcome, and it's the one I find most interesting: that memory doesn't move the needle at all.

If that were the case, the correct conclusion isn't to improve it. It's the opposite: raise the similarity threshold, lower the result cap, inject less. Because every injection carries a cost that is certain — it consumes context, and context is finite — against a benefit that would be zero.

That's the uncomfortable part of measuring seriously: one of the possible outcomes is having to dismantle something that took work to build. I suspect the resistance to measuring comes largely from there, not from technical difficulty.

The criterion I took away

A component that never fails isn't the same as a component that helps. They're two different things and they're easy to confuse, because day to day you operate by exception: you look at whatever complains.

Now, before adding any new piece to the system, I try to answer up front: what would the world look like if this piece didn't exist? If I can't answer that, it isn't that the piece is wrong — it's that I'll be stuck with it, unable to ever justify it.

With memory I arrived at that question late. I'm 1,112 injections in and still don't have the answer.

By July the system was running fast and verifying well, but every task still dragged close to an hour of checks behind it before it could close. The problem was no longer what to check, but what should block the work and what shouldn't: that's what the next entry is about.

Your agent says the tests pass. Did it run them?

Guillermo Leyendeker — Mon, 14 Sep 2026 12:31:16 +0000

Your agent finishes a task and tells you the tests pass. It's worth asking whether it ran them or is reading the result from the previous run.

Caching that result when the code hasn't changed is an obvious optimisation: an afternoon's work and the pipeline flies. I did it the same day it occurred to me.

What nobody writes down is what you're asserting when you do: that the code that was tested is the code you're about to ship. When that doesn't hold, the system doesn't get slow. It signs off on code that was never tested, and you don't find out.

What looked like a trivial change ended up resting on two conditions, plus a marker that makes it auditable afterwards. Miss any of the three and the cache stops being an optimisation.

The starting point

The gate ran its checks serially, one after another. The full suite, with a 40-minute timeout. Then each frontend build, at 5 minutes apiece. And finally mutation analysis, at 20.

That 40-minute timeout says a lot: back in May I had raised it from 10 to 40, because the suite no longer fit. The curve pointed in one direction only.

But duration wasn't the worst of it. This was: when a task failed the gate and had to be retried touching only the frontend, the re-verification ran the entire backend suite again, which hadn't been modified at all. Minutes of compute to confirm something we already knew.

Two changes, one obvious and one delicate

The obvious one: run the suite and the frontend builds in parallel instead of serially. There's no reason for them to wait on each other beyond the fact that's how it was written.

The delicate one: cache the green result of the suite by the combination of project, repository and commit hash. If we already ran the tests against exactly this code and they passed, there's no need to run them again.

Put that way it sounds harmless. This is exactly where the cache stops being an optimisation: from this point on, the green the system reports doesn't come from running the tests. It comes from trusting that the code is the same.

The two conditions holding the promise up

That's why the cache has two locks, and neither is optional.

The working tree has to be clean. If there are uncommitted changes, the commit hash no longer describes what's on disk. The code about to be evaluated isn't the code that was tested. In that case the cache isn't even consulted: the suite runs.

The result is valid for 24 hours. Not because code changes on its own, but because the environment does: dependencies, containers, test databases, external services. A green from a week ago says very little about today's state.

There's a third, smaller detail I like: the suite's exclusion lock — the one preventing two runs from stepping on each other — is only acquired if something is actually going to run. When the cache hits, the function returns before reaching that line. Without it, we'd be serialising runs that will never happen.

The marker that makes auditing possible

This is the part I consider most important in the whole implementation, and the one I see written down least often elsewhere.

Every result coming from the cache travels flagged as such inside the quality gate's data. It's on record that this task didn't run the tests: it leaned on an earlier run.

What's that for? For the day a green shows up that shouldn't have. On that day I'll be able to reconstruct exactly which tasks leaned on the cache and which genuinely ran, and check whether that's where the problem was.

Without that marker, a false green is indistinguishable from a legitimate one. You have a bug you can't even scope, because you don't know which universe of tasks to search.

It's a principle I now apply to any optimisation replacing a real check with an inference: every automated promise needs to leave the marker that lets you audit it when it breaks. Because it will break eventually, and on that day the marker is all you have.

What was deliberately left out

Mutation analysis — which checks whether the tests actually detect changes in the code, not merely whether they pass — was kept out of the cache on purpose.

The reason is that it measures something different. The suite is evaluated against the state of the repository; mutations are evaluated against the specific task's diff. Two tasks on the same base commit can have completely different diffs, so caching by commit hash would make no sense: they're orthogonal.

What I did add was a short-circuit from the other side: if the task's diff contains no mutable code, the analysis simply doesn't run. There's nothing to mutate. That resolved a large share of cases without touching the underlying logic.

The outcome

Between the parallelism and the cache, the quality gate stopped being the obvious bottleneck it had been. It's still the most expensive step in the system — that 29% of the bill didn't move as much as I expected — but the wait per task dropped noticeably.

What stayed with me above all was a criterion I've reused several times since: speeding up a check is fine as long as you can prove afterwards that the check existed. If the speed-up leaves you unable to audit what was verified and what wasn't, you're not optimising — you're loosening the check and calling it something else.

The front left open

That same June review left another item on the table, one that wasn't about speed and was considerably more uncomfortable.

There was a component I'd been using for over a month, running more than a thousand times a month, and I couldn't say whether it improved anything. It never failed. That was exactly the problem.

That's what the next entry is about.

29% of my bill was in the step I never looked at

Guillermo Leyendeker — Mon, 07 Sep 2026 12:30:35 +0000

If you use agents daily you already have an intuition about where the money goes. Mine pointed at the retry: the agent fails a check, starts over from scratch, and you watch it on screen redoing something it had already done.

I wrote a five-point optimisation plan based on that intuition. Then I went to the database for the baseline I was going to measure the improvement against, and found the retry had fired exactly once in the system's entire history.

The spend was somewhere else: in the verification that runs before each task closes, which took $3,463 of the $11,968 the system spent in total. 29% of the bill in the dullest step there is, the one I never looked at because it always worked.

The lesson holds for anything you automate with models: what annoys you are the exceptions, because they interrupt you. What costs you is the routine, because it happens every time.

The obvious suspect

All five points of the plan aimed at the same place: the retry loop.

The reasoning was solid. When work fails the quality gate and has to be retried, you pay for everything again: another agent spun up, another exploration of the code, another implementation, another gate run. It's by far the most expensive operation in the system. And on top of that it was badly implemented: it discarded the previous session, so the second attempt started with no memory of the first.

All of that was true. The five diagnostic points were correctly identified. The plan opened with making retries smarter and continued with minor optimisations.

Then I went to the database to find the baseline number I'd be measuring the improvement against.

What the data said

I found three things that fit nothing I had written:

A single item, across the system's entire history, had ever produced retry feedback.
Zero surrenders. The give-up path, carefully designed with its structured incident report, had never once fired.
The first-attempt pass rate was effectively 100%.

The retry loop I was about to optimise barely ran. I had written an improvement plan for the least-travelled path in the system.

The explanation, once I looked for it, made sense: errors weren't being corrected between orchestrator steps. They were corrected inside each agent's own loop, before the work ever reached the quality gate. The agent iterated on its own output, caught its mistakes and fixed them before handing anything over. To the orchestrator that's invisible: all it sees is a step that finished cleanly.

So my intuition wasn't just wrong about the size of the problem. I was looking at a layer of the system where the problem simply didn't happen.

The number that reordered the plan

The same query surfaced the figure that changed everything. The quality gate — the step that verifies each task before closing it — was running around 157 times a week, averaging 5 to 8 minutes each time.

That was the dominant cost. Not retries, which barely happened: verification, which happened every time.

And the finding held up over time. Today, looking at the cumulative figures, the quality gate has consumed 3,463 dollars out of the 11,968 the system has spent in total. Twenty-nine percent of the bill concentrated in one step.

The plan was rewritten that same afternoon. First, parallelise the gate and cache the test suite. Retry work was demoted to a cheap robustness improvement, for when the worst case actually occurs.

Why intuition pointed the wrong way

I've thought a fair bit about why I got it so clearly wrong, because the error wasn't analytical: it was perceptual.

Retries hurt. You see them on screen, they show up as a loop repeating, they feel like waste while they happen. They're visible, noisy and frustrating.

The quality gate didn't hurt because it was routine. It ran every time, it was part of normal operation, and for exactly that reason it was invisible. Nobody looks twice at a step doing what it's supposed to do.

Which is why what annoys you most and what costs you most are almost never the same thing. The exception interrupts you and never shows up on the invoice. The routine never bothers you and is the whole invoice.

A note on how I recorded the mistake

When I rewrote the plan, I didn't delete the original diagnosis. I added a dated note above it with the recalibration: what I had believed, what the data showed, and how that changed the priority.

I could have rewritten the document cleanly, with the correct diagnosis from the start. I didn't, for a practical reason: the wrong diagnosis still has value. It explains why the system is built the way it is, and it reminds me that the reasoning behind it was sound — reading the code, anyone would have reached the same conclusion.

A plan that hides its own diagnostic errors becomes an internal marketing document. One that leaves them visible, with the date attached, is a record of how something was learned.

Where to start in your own system

The order isn't guessed, it's queried. Before writing an optimisation plan, list your pipeline's steps with two columns — what each run costs and how many times it runs per week — and sort by the product of the two. Frequency is what surprises you, almost never unit cost: the wildly expensive step that runs once a month isn't your problem.

The first task in the reordered plan sounded trivial: stop running the full suite when the code hadn't changed. It had a trap that nearly had me certify as passing a codebase that was never tested, and that's what the next entry is about.

I asked the AI to fix its own instructions and it barely helped

Guillermo Leyendeker — Mon, 31 Aug 2026 12:30:30 +0000

Every time the system gave up on a task, it left a record of what it had tried and why it failed. One day I looked at the pile of those reports and thought the obvious thing: the AI can read this and fix its own instructions.

I built the whole thing, with one rule from day one: the routine edits nothing on its own. It proposes the change, I approve it with a click.

It works. I collected 1,771 recorded failures over three months, it produced 13 proposed fixes, and the system barely improved through that path.

The model didn't fail and neither did the human approval. Something far more boring did, and it's a mistake that repeats in any system that tries to learn from its own logs.

The case that started it

It wasn't an abstract idea: it came from a concrete, repeated problem.

One of the system's skills had a habit of ending its runs with a question. Something along the lines of "shall I continue with the next part?". To the orchestrator, a run that ends waiting for input is a stuck run: it aborted the whole sequence.

I fixed it by hand, editing that skill's instructions. And while doing it I thought the obvious thing: this pattern is on record. The system knows this skill failed for this reason, several times over. Why do I have to be the one who notices?

The goal ended up written like this: replace the cycle human detects, human edits, human commits with the system detects, proposes a patch, the human approves with one click.

The constraint I set on day one

Before writing a line, I decided something I still consider the best decision in the whole subsystem: the routine never edits files or commits on its own.

Its output isn't a change, it's a proposal: the diff plus the reasoning behind it. It's stored in the database and waits. A human approves it and only then is it applied, or rejects it and it's archived.

The temptation to skip that is strong, because manual approval is precisely the bottleneck you're trying to remove. But a system that modifies its own instructions without supervision has no brake at all: if the criteria it judges by are wrong, every iteration makes them worse, and there's nothing outside the loop to catch it.

Four phases, all implemented

The design ended up in four parts, and all four are running:

A unified incidents table, consolidating signals that previously lived scattered: orchestrator surrenders, skills that stall waiting for input, pre-flight checks that fail.
Recurring-pattern detection: it groups by skill and reason, and flags anything appearing three or more times in seven days.
The routine that proposes the patch, which decides whether the fix belongs in the skill's instructions or in code, computes the diff and persists it.
The approval interface, with a 24-hour limit so the same proposal doesn't come back over and over.

None of this is broken. It all runs.

The numbers, three months later

When I went to measure the outcome, this is what I found:

1,901 incidents recorded in total.
1,771 of them still open, unprocessed.
13 proposals generated across the system's entire history. Seven approved, six rejected.

Thirteen. In three months, with nearly two thousand incidents of raw material.

And what stings most is that the most frequent incidents are exactly the ones you'd want to turn into rules. Over fourteen days: 351 repeated reads of the same file, 154 searches that found nothing and 151 failed commands. All actionable patterns. All sitting there, unprocessed.

Where it failed, which isn't where you'd expect

My first reaction was to assume the model wasn't up to the task. That's the comfortable explanation, and it was wrong.

The loop doesn't fail at the intelligent part. It fails in the plumbing. The hypotheses I still have open — and haven't finished ruling out — are three, and none of them involve the model's capability:

That the proposal routine is barely firing at all, because of how its scheduling ended up configured.
That the recurring-pattern view isn't grouping correctly, so it finds nothing to report.
That the threshold of three occurrences in seven days is unreachable — not because patterns don't repeat, but because there's so much background noise that the real cases get diluted.

That last one interests me most, because it implies the problem isn't detection but signal quality. If the inefficiency detector produces false positives at scale, it doesn't matter how good the next stage is: you're asking it to find patterns inside a pile of noise.

So the next step isn't improving the routine that proposes patches. It's triaging the 1,771 open incidents and answering something more basic: how many are real signal and how many are noise from the detector itself. If the bulk is noise, detection has to be tightened before asking anyone for proposals.

What I took from this

A self-improvement loop doesn't fail for lack of intelligence. It fails in the plumbing, which is the part nobody shows in demos.

Recording signals is easy and cheap, which is why it's the first thing you build. It feels productive: the numbers go up, the table fills, it looks like the system is learning. Turning that signal into action is the real work, and it's exactly where the project stalled.

Accumulating 1,771 incidents without processing them isn't telemetry. It's an expensive log that looks like a dashboard.

I now apply the lesson before instrumenting anything new: if I can't say what decision someone — or something — will make with that data, the instrumentation can wait.

The second time the data contradicted me

This was the first result that forced me to go look at the database instead of trusting what I thought was happening.

A month later I did the same with the rest of the system, in a general review aimed at improving pipeline speed. I had a diagnosis put together and a plan written on top of it.

The data made me throw that plan out the same day. That's what the next entry is about.

Your agent fails and sits there waiting for you

Guillermo Leyendeker — Sun, 23 Aug 2026 09:00:59 +0000

You hand the agent a task, go do something else, and when you come back it's sitting there waiting for you: something failed and it didn't know what to do.

That moment decides whether the agent saves you time or eats it. If the answer when work comes out wrong is "it stops and waits for you", you can leave it alone for ten minutes, never for an afternoon.

That's what I had by early May. The system could already detect that a piece of work was wrong, and what it did next was nothing. The problem wasn't technical: I had never decided what should happen after a failing verdict.

And there the options aren't two: fix what it did, start over with a different approach, or give up and leave a record of what happened. Having it written down when each one applies is what separates an experiment from something you can leave running alone.

Separating the one who does from the one who judges

The first change was structural: explicitly separating two roles. One generates the work, the other evaluates it. This isn't an organisational nicety — it changes what information travels between steps.

As long as the evaluator returns "passed" or "failed", the only thing you can do with a failure is retry blindly. If instead it returns structured feedback, the system can decide what to do with it.

Evaluation became multidimensional. Instead of a single verdict, four axes: correctness, security, user experience and completeness, each with its own score. And something that turned out more useful than I expected: regression detection against previous attempts, to tell "still wrong" apart from "worse than before", which are different problems.

Three paths, not one

With that information in hand, the system picks between three exits. This is the part I think is missing from most orchestrators, where failure handling boils down to retrying.

Refine: the work is on track but incomplete or has specific defects. It goes back to the same implementer with feedback about exactly what to fix.
Pivot: the work is wrongly framed. Fixing on top of it won't help, because the starting point is the problem. The task is handed back a step up the chain.
Surrender: after three attempts, the system stops pushing.

Surrendering isn't failing silently, which would be the worst of both worlds. It writes a structured incident report: what was attempted, with what feedback, which dimensions failed each time. That report is stored and becomes material to review later.

In fact, looking at the pile of those accumulated reports is how I came up with the most ambitious idea of the project, which is the subject of the next entry and turned out considerably worse than I expected.

The mistake that took me a month to see

There's a detail in this implementation that was wrong for a full month, and it illustrates how easy it is to waste context without noticing.

When the system decided to refine, it relaunched the implementer. But in doing so it discarded the previous session. The agent started from scratch: no memory of which files it had touched, which paths it had ruled out, or why it made the decisions it made. All it received was a block of text with the evaluator's feedback.

Put another way: I was asking it to fix work it didn't remember doing.

The fix, once I finally saw it, was one line: keep the session when the decision is to refine, and discard it only when pivoting, which is the one case where forgetting is deliberate — there you want the model starting without the bias of the failed attempt.

What makes that safe is a limit that already existed: since surrender cuts in at the third attempt, a refine with a live session can happen at most twice. Context doesn't grow indefinitely; it has a structural ceiling.

Telling apart failures that aren't the agent's

Something else I learned while operating this: not every failure is the worker's fault.

When the provider's API returned a network error, the system counted it as a failed attempt. The agent had done nothing wrong — it hadn't even started — but one of its three attempts was consumed. With two consecutive network blips, a perfectly healthy task reached surrender without anyone having evaluated anything.

Today those errors don't consume an attempt and are retried with backoff. Penalising the model for an infrastructure outage is, quite simply, measuring the wrong thing.

The fourth path

In August I added one more exit, for a case I hadn't foreseen: the impossible pivot.

Sometimes the evaluator determines the work is wrongly framed, but handing it a step up makes no sense because that previous step doesn't exist or is already exhausted. Before, that fell straight through to surrender. Now, instead of giving up, the system re-explores the terrain: it relaunches the code reconnaissance to rebuild context and retries from there.

It's an example of something that recurs throughout the project: failure policies aren't designed well up front. They're discovered by operating, when the case appears that doesn't fit any of the exits you'd anticipated.

What this buys you

A system with agents needs a failure policy as explicit as its happy path. And that policy has to distinguish three things that look alike at first: work that's close and deserves another pass, work that's wrongly framed and needs to go back, and work that has to be handed to a person.

Retrying without that distinction is burning money: you repeat the same mistake with the same information and expect a different outcome. And failing to separate surrender from the rest is worse still, because you lose the one case that actually deserved your attention.

With the three exits implemented, the system started resolving most of its own stumbles on its own. And what settled at the bottom were the reports of the ones it couldn't.

That pile of reports was the starting point for what comes next: asking the system to improve itself.

Your agent can exit clean without writing a single line

Guillermo Leyendeker — Sun, 16 Aug 2026 09:00:36 +0000

Try this with whatever agent you use: give it a concrete task and, when it reports back that it's done, don't believe it. Go read the diff.

An agent can explore the repository, reason about the problem, write a convincing summary of what it would do, and exit with status zero without touching a file. It isn't lying: as far as it's concerned the run went fine. What lies is the criterion you're measuring it with.

That criterion — if the process returned no error, the task is done — is the one my system had in April, when it was already closing tasks one after another. Written out like that it sounds reasonable. In practice it's taking the suspect's word for it and filing it as evidence.

Four months and 2,095 tasks later it's still the hardest problem in the system, and I have no reason to think it's only mine.

The evidence is in the repository, not in the summary

The idea was simple and it changed the whole system: to mark something as done, you have to find evidence in the repository. Not read a claim in the agent's output. Find commits.

It sounds like a minor detail. It isn't, because it flips the burden of proof. Before, the system assumed work was done unless something failed. From that day on, it assumed work wasn't done until a real change showed up in the code.

Three days later I tightened the criterion further: the quality gate, until then a step you could skip, became mandatory between implementation and completion. It was no longer an optional check that ran when I remembered. Without a passing gate, the task didn't close.

The holes that showed up afterwards

What's interesting about this problem is that it wasn't solved by those two changes. It was solved — partially — over four months, and every hole I plugged taught me something about the difference between verifying and appearing to verify.

In early May I added a check comparing the files the spec declared it would touch against the ones that actually changed in the commits. Because finding "a commit" isn't enough: the agent might have committed something, somewhere, with nothing to do with what was asked.

That same day a case I hadn't anticipated appeared: an old gate verdict certifying new work. The task had passed the gate on an earlier attempt, was then modified, and the stored verdict still said everything was fine. The rule I added was to require a fresh gate before marking anything as done.

And there was a scoping problem, duller but just as important: the commit search only looked at the main repository. Since changes are spread across the backend and the various frontends, there were perfectly real implementations the system couldn't find, and tasks that got stuck for no reason. The search had to be extended to every repository in the workspace.

Four months on, the problem still surfaces

This is the part I least expected when I started.

On August 3rd — nearly four months after the first check — I found a new case: the gate was certifying tasks it had never actually looked at. It wasn't that verification failed; it was that on a certain code path verification never ran, and the result defaulted to passed.

That's the kind of bug that worries me most, because it's invisible. A check that fails makes noise. A check that doesn't run and returns green is indistinguishable from one that ran correctly, unless you go looking for it specifically.

Nine days later I took the most recent step: each task's implementation stage stopped being declared by what the system believes and started being derived from already-verified change evidence. In other words, state is no longer something written when a step finishes; it's something computed from what can be proven.

Looking at the project's full history, this is the only line of work running end to end across all four months. Everything else had a beginning and an end. This didn't.

Why it's the hardest problem in this space

I think there's an underlying reason, and it's that trusting is the default behaviour of almost any system orchestrating agents.

When you wire a model into a pipeline, what's naturally available to you is the process outcome: it finished, it didn't, it errored. Building real verification — going to the repository, finding commits, cross-referencing files against the spec, confirming the gate ran against this version and not another — is extra work nobody asks for and that goes unnoticed when it works.

It gets noticed when it's missing. And it gets noticed late, which is the worst part: a wrongly closed task doesn't produce an error, it produces a false foundation for everything built on top of it. The cost shows up weeks later, when something that depended on it doesn't work and nobody understands why.

So what I took from these four months isn't a technique but a criterion: trust in an agent isn't configured, it's built from evidence external to the agent. And useful evidence isn't that the process finished without error. It's that commits exist, with specific files, matching what was asked, verified after the last modification.

Everything else is taking the suspect's word for it.

What this made possible

Once the system could detect that a piece of work was wrong, the next question was exposed, and it was considerably more uncomfortable: now what do I do about it?

At that point, a failing verdict simply stopped the chain and waited for me. The agent had failed, the gate had caught it, and the system sat still until I showed up to look at what happened.

A pipeline that needs you watching isn't worth much. That's what the next entry is about: what the system does when an agent fails, and how to decide between pushing on, changing approach, or giving up.

You're paying a model to work as an if statement

Guillermo Leyendeker — Thu, 13 Aug 2026 15:53:47 +0000

When I built my agent orchestrator, the first step in the chain was this: an agent read a list of tasks and picked which one came next.

Sounds reasonable until you look at the rule up close. Take the first pending task with all its prerequisites met, mark it in progress. There's nothing ambiguous about that: it's an if, and I was asking a model to run it.

I moved it to a script three days in and noted the saving in the commit: around $0.25 and 40 seconds per run. I did the same with the closing step, which decides nothing either: another $0.30 and 20 seconds. Sounds like nothing.

Four months and 2,095 tasks later, those two scripts add up to more than a thousand dollars and more than thirty hours never spent. And the saving didn't come from a cheaper model or better prompts: it came from giving it fewer things to decide.

The first two days were all interface

Before that chain there was a screen. I started on April 7th, 2026, and the first thing I built was a server managing several projects at once, with a dashboard in the middle showing which stage everything was in. Looking back at that first week's history, it's almost funny. The width of the sidebar. Tooltips showing full dates on hover. "In progress" badges on each stage. Colours per pipeline stage, so they'd be distinguishable at a glance.

There's a pair of consecutive commits I particularly like as a portrait of that moment. The first sets an emoji as the icon for the loop button. The second replaces it with a Unicode character, because the emoji ignored the colour I was setting in CSS and always looked the same, regardless of state.

I don't bring this up to mock myself. I bring it up because it's exactly what a project looks like when you don't yet know what the problem is. I was polishing the system's surface very carefully because the surface was the only thing in front of me. The underlying question — which parts of this flow need a model to decide, and which don't — hadn't even occurred to me.

Two steps that became scripts

By then the pipeline had taken shape: a chain of steps where an agent picked the next pending task, implemented it, and then marked it complete. All three steps ran through the model, because all three were written as instructions inside the skills I handed it.

The uncomfortable question came from looking at the first of the three. There's an index of items, each with a status and a list of prerequisites, and the whole rule is: take the first one that's pending with all prerequisites met, mark it in progress, and record the change.

That day I turned it into a deterministic script. It parses the index, finds the first pending item with its prerequisites in order, marks it in progress and commits the change. It also handles resuming an item that was already underway, which was the only part with any subtlety to it.

The same day I did the same at the other end of the chain. The step that closes a task — updating the status, recalculating the summary counters, deleting the spec file and committing — also stopped going through the model. It's a sequence of operations over files and state. There's nothing to interpret.

The numbers I wrote down at the time

What keeps this from being an anecdote is that I measured before and after, and left it written in those commit messages.

Automatic picking cost roughly 0.25 dollars and 40 seconds every time it ran. Completion, roughly 0.30 dollars and 20 seconds. As a script, picking became step zero of the pipeline: instant and free, running before any agent was spun up.

A little under sixty cents and a minute per task. Seen one at a time, it's negligible. That's precisely the problem with this kind of waste: it never hurts enough for you to look at it.

Four months on, the system has completed 2,095 features and fixes. At that rate, those two scripts add up to more than a thousand dollars and over thirty hours of waiting that were never paid. And not because the model was doing its job badly — it was doing it well. It was simply doing a job that didn't require a model.

You pay for the model where there's ambiguity

The conclusion I drew from those two days is still the criterion I apply to every new step I add to the pipeline. A model earns its keep where something has to be interpreted: understanding a spec written in prose, deciding how to structure a change, judging whether a result is acceptable. Where the rule can be written down, the rule wins.

And it doesn't win on price alone. It wins for three reasons that weigh more heavily as the system grows:

It's cheaper, which is the obvious point and the least important one.
It's faster: a script takes milliseconds where an agent takes tens of seconds, and that gap multiplies across every task in the backlog.
It's deterministic, which is what actually matters. A pipeline that has to run unattended can't have steps that sometimes do one thing and sometimes another. Every point where a model intervenes is a point where the outcome can vary, and that variance compounds along the chain.

Put differently: every deterministic step you take away from the agent isn't just savings, it's a source of uncertainty you remove from the system.

Telemetry is what made it visible

There's a detail without which none of this would have happened: to claim a step cost twenty-five cents, I first had to measure it.

That instrumentation — recording cost and duration for every single step — was among the first things I built, and today the system holds 5,698 skill runs and 9,109 recorded steps, each with what it cost and how long it took. It's the foundation under nearly every decision I'll describe in this series.

Without those numbers, the reasoning of that April 10th would have been one more intuition, of the same kind as the ones that later turned out to be wrong. With the numbers, it was a ten-minute decision.

Where yours are

The exercise requires building nothing: list the steps in your flow and mark the ones with a single correct answer given the input. Picking the next item, moving a state, recalculating a counter, checking whether the prerequisites are met. Every one of those a model resolves today you're paying for twice — in tokens and in variance — to get something a conditional returns identically every time.

With the deterministic steps out of the way the pipeline started producing at a good pace, and the opposite problem showed up, the one that occupied me for the next four months: the agent would finish, report that it had implemented what was asked, and the only criterion for believing it was that the step hadn't returned an error. That's what the next entry is about.