DEV Community: Guillermo Leyendeker

Resumable Chunked File Uploads with Laravel, Next.js, and Redis: A Step-by-Step Guide

Guillermo Leyendeker — Wed, 14 Oct 2026 18:02:15 +0000

Liquid syntax error: Variable '{{ width: {% raw %}' was not properly terminated with regexp: /\}\}/

Edge Caching Dynamic Pages: Next.js 16 Server Actions with Cloudflare Workers and Redis

Guillermo Leyendeker — Mon, 12 Oct 2026 12:32:34 +0000

analysis: need to produce HTML content with specified tags, 120---1500 words, no markdown, no h1 etc. Must obey style rules: practical, concrete examples, no metaph metaph etc. Provide headings etc. Let's craft about Edge caching dynamic pages with Next.js 16 Server Actions, Cloudflare Workers and Redis. Use code blocks with

 etc. Ensure 1200-1500 words approx. Let's aim ~1300 words. Use  for main sections, 
<|start|>assistantfinal<|start|>assistantfinalHTML:<|end|><|start|>assistantfinalHTML: 
 Why edge caching matters for dynamic pages </ h2 ... etc. Need to produce valid HTML tags only, no markdown, no h1, no html/head/body tags. Let's craft.<|end|><|start|>assistantfinalHTML: 
 why edge caching matters for dynamic pages</h2


 modern web applications often mix static assets with data‑-driven HTML.  Next.js 16 introduced ** server actions— an API‑-like function that can be invoked directly from client-side code without creating a separate endpoint.  The convenience is great until you realize that every action still runs on your origin server, which means latency, cold cold starts, and scaling bott bott bott bott... etc. ... ( need to produce 1200-150... ) 

Oops need to produce final answer.<|end|><|start|>assistantfinal: < h2 ... etc. Need to produce proper HTML with tags only, no markdown, no h1 etc. Let's craft final content now.<|end|><|start|>assistantfinalHTML: 
 Edge caching dynamic pages with Next.js  16 server actions, Cloudflare Workers and Redis Your AI agent tells you “ the data is ready, rendering can proceed”.  What actually happens behind that message is a request to your Next.js origin, execution of a server action, fetching of data, and finally sending HTML back to the browser.  If the origin resides in a single region, every user— regardless of geography— pays the full round‑trip latency for each action invocation.  edge caching eliminates that latency by pre‑emptively storing the final HTML close to the user while still allowing the page to be fully dynamic.

 The practical problem we solve 

 dynamic content needs fresh data on each request  e‑commerce product pages, personalized dashboards, or A/B test variants.

 we cannot simply static‑ generate because the data changes every few seconds or depends on cookies/ headers.
 origin scaling cost 

... ( need full content ~1300 words) 

We need to produce final answer now. I'll craft full HTML content with approx 1300 words. Ensure tags only: h2 h3 p ul li pre code strong etc. No h1, no html/head/body. Let's write carefully.<|end|><|start|>assistantfinalHTML:  edge caching dynamic pages with Next.js 16 server actions, Cloud Cloudflare workers and redis  your AI agent says “ tests passed” – that is the moment you see a green badge in your UI.  what behind that badge is a request to your origin, execution of a server action, fetching of data, rendering of JSX on the server, and sending HTML back over the internet.  if your origin lives in a single region, every user pays the full round‑trip latency and every invocation consumes CPU on that single instance.  edge caching solves that problem by storing the final HTML close to the user while still allowing the page to be fully dynamic.

 the concrete scenario: a personalized product detail page




 requirements


 product information ( name, price, stock ) must be fresh at most 30 seconds old; 


 user‑specific discount code comes from a cookie;  A/B test variant ( control vs variant) is determined by a header sent by the agent. 



 static generation can't work because the discount code and variant change per request.


 constraints


 no dedicated queue or orchestrator – the agent simply calls  fetch product data() 


 budget limited to one origin instance; 



 why next.js 16 server actions fit this scenario 


 server actions are functions exported from a page or API file that can be called directly from client-side code without creating an HTTP endpoint.  they are executed on the same Node.js process that renders the page, so they share the same memory and can directly access any server-side library ( Prisma Prisma Prisma etc.).  they also automatically re‑validate the page when they return a  revalidate: true

... 


<|return|>

Realistic API Performance Testing for Laravel/Next.js with k6 and Playwright

Guillermo Leyendeker — Wed, 07 Oct 2026 18:04:58 +0000

Why “Pass” Is Not Enough: The Agent’s Blind Spot

Imagine you just asked your AI‑driven test agent, “Did the checkout API survive the spike?” The agent replies, “All tests passed.” You celebrate, push the code, and the next minute a customer reports a 5‑second checkout latency. The agent’s answer was technically correct – every assertion in the test script succeeded – but it never warned you that the API would crumble under realistic traffic.

Transferable rule: Never rely on a single “pass/fail” signal. Correlate load‑generated response times with server‑side metrics (CPU, DB query time, queue depth) before you deem a performance test successful. This rule applies whether you’re testing a Laravel monolith or a Next.js edge API.

Below we’ll build a minimal, reproducible setup that lets your agent say “All good” only after it has verified both client‑side latency and backend health. The stack is:

k6 for HTTP load generation.
Playwright for end‑to‑end (E2E) API sanity checks.
Docker to keep the environment identical for every run.
Laravel (PHP 8.2) and Next.js (React 18) as the services under test.

Setting Up a Self‑Contained Docker Lab

First, create a docker-compose.yml that spins up three containers:

laravel – runs php artisan serve on port 8000.
nextjs – runs next dev on port 3000.
k6 – the load generator.

All services share a network: testnet so they can address each other by service name.

<pre><code>version: '3.9'

services:
  laravel:
    image: php:8.2-cli
    working_dir: /var/www
    volumes:
      - ./laravel:/var/www
    command: sh -c "composer install && php artisan serve --host=0.0.0.0 --port=8000"
    ports:
      - "8000:8000"
    networks:
      - testnet

  nextjs:
    image: node:20-alpine
    working_dir: /app
    volumes:
      - ./nextjs:/app
    command: sh -c "npm install && npm run dev -- --port 3000"
    ports:
      - "3000:3000"
    networks:
      - testnet

  k6:
    image: loadimpact/k6:latest
    depends_on:
      - laravel
      - nextjs
    networks:
      - testnet
    volumes:
      - ./k6:/scripts
    entrypoint: ["/bin/sh", "-c"]
    command: "while true; do sleep 3600; done"

networks:
  testnet:
    driver: bridge</code></pre>

Save the file, then run docker compose up -d. The three containers are now ready for any agent‑driven invocation.

Writing a Realistic k6 Scenario

k6 scripts are plain JavaScript. The key to realism is to model the traffic pattern you actually see in production. For a checkout flow the pattern often looks like:

A burst of 20 requests per second (RPS) for the first 30 seconds – think of a flash sale.
A steady “background” of 5 RPS for the next 2 minutes – normal browsing.
A cool‑down period where we stop sending traffic but keep measuring latency.

Here’s a compact script that does exactly that and also records server‑side metrics via a custom endpoint (/metrics) exposed by Laravel and Next.js.

<pre><code>import http from 'k6/http';
import { sleep, check } from 'k6';
import { Trend, Rate } from 'k6/metrics';

// Custom metrics
export const apiLatency = new Trend('api_latency');
export const serverCpu = new Trend('server_cpu');
export const errorRate = new Rate('error_rate');

export const options = {
  stages: [
    { duration: '30s', target: 20 }, // burst
    { duration: '2m', target: 5 },   // steady
    { duration: '30s', target: 0 }, // cool‑down
  ],
  thresholds: {
    apiLatency: ['p(95)<800'], // 95% of requests < 800 ms
    errorRate: ['rate<0.01'],  // <1 % errors
  },
};

function fetchMetrics(service) {
  const res = http.get(`http://${service}:8000/metrics`);
  if (res.status === 200) {
    const data = JSON.parse(res.body);
    serverCpu.add(data.cpu);
  }
}

export default function () {
  // 1️⃣ Hit Laravel checkout endpoint
  let laravelRes = http.post('http://laravel:8000/api/checkout', {
    cart_id: Math.floor(Math.random() * 1000),
    payment_method: 'card',
  });
  apiLatency.add(laravelRes.timings.duration);
  check(laravelRes, { 'laravel status 200': (r) => r.status === 200 });
  errorRate.add(laravelRes.status !== 200);
  fetchMetrics('laravel');

  // 2️⃣ Hit Next.js order‑summary API (edge)
  let nextRes = http.get('http://nextjs:3000/api/order-summary?cart_id=123');
  apiLatency.add(nextRes.timings.duration);
  check(nextRes, { 'nextjs status 200': (r) => r.status === 200 });
  errorRate.add(nextRes.status !== 200);
  fetchMetrics('nextjs');

  sleep(1);
}</code></pre>

Notice the fetchMetrics call after each request. Your Laravel and Next.js apps must expose a lightweight /metrics endpoint that returns JSON like { "cpu": 0.42 }. This is the “correlate” part of the rule: the agent can now assert not only that the HTTP response was fast, but also that the server was not saturated.

Deploying the Metrics Endpoint in Laravel

In routes/api.php add a simple route:

<pre><code>use Illuminate\Support\Facades\Route;
use Illuminate\Support\Facades\DB;

Route::get('/metrics', function () {
    $cpu = sys_getloadavg()[0] / sys_getconf('SC_NPROCESSORS_ONLN');
    $queries = DB::connection()->getQueryLog();
    $dbTime = array_sum(array_column($queries, 'time')) / 1000; // seconds

    return response()->json([
        'cpu' => round($cpu, 2),
        'db_time' => round($dbTime, 3),
    ]);
});</code></pre>

The endpoint is intentionally cheap – it just reads the current load average and aggregates the last query log. It’s safe to call on every k6 iteration.

Deploying the Metrics Endpoint in Next.js

Create pages/api/metrics.js:

<pre><code>export default async function handler(req, res) {
  const { cpuUsage } = await import('os');
  const cpu = cpuUsage().user / 1e9; // convert nanoseconds to seconds

  // Simulate a quick DB check – in real code you’d query your DB here.
  const dbTime = Math.random() * 0.02; // 0‑20 ms

  res.status(200).json({ cpu: Number(cpu.toFixed(2)), db_time: dbTime });
}</code></pre>

Again, the endpoint is lightweight. The agent will call it after each API request, keeping the latency budget honest.

Playwright for Functional Confirmation

k6 tells you “the system can handle X RPS”. Playwright tells you “the API still returns the right shape”. Combine them in a single Docker container so the agent can fire both with a single CLI call.

Folder playwright contains:

Dockerfile – based on mcr.microsoft.com/playwright.
tests/checkout.spec.js – a minimal sanity test.

<pre><code># Dockerfile
FROM mcr.microsoft.com/playwright:latest

WORKDIR /tests
COPY package*.json ./
RUN npm ci
COPY . .

CMD ["npx", "playwright", "test"]</code></pre>

Playwright test:

<pre><code>// tests/checkout.spec.js
const { test, expect } = require('@playwright/test');

test('checkout returns orderId and total', async ({ request }) => {
  const cartId = Math.floor(Math.random() * 1000);
  const response = await request.post('http://laravel:8000/api/checkout', {
    data: { cart_id: cartId, payment_method: 'card' },
  });
  expect(response.ok()).toBeTruthy();
  const body = await response.json();
  expect(body).toHaveProperty('order_id');
  expect(body).toHaveProperty('total');
});</code></pre>

Build and run the container:

<pre><code>docker build -t playwright-tests ./playwright
docker run --network testnet playwright-tests</code></pre>

If the test fails, the agent will report a functional error even if k6 saw low latency. This double‑check eliminates the “green suite but broken business logic” scenario.

Running the Full Suite from Your Agent

Assume your AI agent can execute shell commands. A single script that the agent calls could look like:

<pre><code>#!/usr/bin/env bash
set -e

# 1️⃣ Warm up containers
docker compose up -d

# 2️⃣ Run k6 load test (timeout after 4 minutes)
docker run --rm --network testnet -v $(pwd)/k6:/scripts loadimpact/k6 run /scripts/checkout.js

# 3️⃣ Run Playwright sanity checks
docker build -t playwright-tests ./playwright
docker run --rm --network testnet playwright-tests

# 4️⃣ Tear down
docker compose down</code></pre>

The agent can parse the exit code of each step. If k6 returns a non‑zero code (threshold breach) or Playwright returns a non‑zero code (assertion failure), the agent reports “FAIL”. Only when both exit codes are zero does it answer “PASS”.

Analyzing the Output – What the Agent Should Surface

When the script finishes, k6 prints a summary like:

<pre><code>          /\      |‾‾| /‾‾/   /‾‾/
     /  \     |  |/  /   /  /
    / /\ \    |     (   (   )
   / ____ \   |  |\  \   \  \\
  /_/    \_\\  |__| \__\  \__\\

running (04m00.00s), 00/00 VUs, 12345 complete iterations
default ✓ [======================================] 100% 00/00 VUs

    ✓ status was 200

    checks.........................: 100.00% ✓ 12345 ✗ 0
    error rate....................: 0.00%   ✓ 0      ✗ 0
    api_latency...................: avg=312ms min=45ms max=1245ms p(95)=720ms
    server_cpu....................: avg=0.31
    thresholds....................: passed=2 failed=0
</code></pre>

The agent should extract the following facts before answering “PASS”:

All thresholds met (latency < 800 ms, error rate < 1 %).
Server CPU stayed below 0.5 (or any threshold you define).
No Playwright assertion failed.

If any of those conditions are false, the agent must explicitly list the violation – e.g., “FAIL: 95th‑percentile latency 1.2 s exceeds 800 ms”. This makes the response actionable without digging into logs.

Common Pitfalls and Quick Fixes

1️⃣ The Load Generator Can’t Reach the Target RPS

If k6 reports “0 VUs” or “max RPS not reached”, you are likely hitting a Docker networking bottleneck. Increase the host’s ulimit -n (open files) and allocate more CPU to the k6 container:

<pre><code>docker run --rm --network testnet \
  --cpus="2.0" \
  -v $(pwd)/k6:/scripts loadimpact/k6 run /scripts/checkout.js</code></pre>

2️⃣ Metrics Endpoint Overhead Skews Results

Calling /metrics on every iteration adds extra HTTP traffic. If you see a sudden jump in latency after adding the endpoint, switch to a “heartbeat” approach: call the endpoint once every 10 seconds from a separate k6 group that runs in parallel.

3️⃣ Playwright Fails Randomly on CI

Playwright depends on a stable DNS resolution for laravel and nextjs. In CI environments, the Docker network may be recreated between steps, causing flaky DNS. Use the container IP address (retrieved via docker inspect) or add an explicit extra_hosts entry in docker-compose.yml to guarantee name resolution.

4️⃣ Laravel Queues Stall Under Load

If you notice CPU staying low but response times climbing, the bottleneck may be a saturated queue (e.g., email or webhook jobs). Add a queue:work service to the compose file with --tries=3 and monitor its queue:failed count after the test.

Extending the Pattern to Edge Cases

Next.js can be deployed to Vercel or Cloudflare Workers, where the “container” abstraction disappears. The same principle works: replace the Docker nextjs service with a curl call to the real edge URL inside the k6 script, and keep the Playwright test pointed at that URL. The agent still receives a unified PASS/FAIL answer because the script logic hasn’t changed – only the target host.

What to Do Next

If you’ve followed the steps, your agent now validates both performance thresholds and functional correctness before saying “PASS”. The next logical step is to automate the script as a webhook that your CI system can trigger, but keep the decision point inside the agent – the CI only reports “agent says PASS”. This preserves the mental model that the agent is the single source of truth.

For deeper insight, consider exporting k6 metrics to InfluxDB and visualizing them alongside Laravel’s telescope data. That way the agent can answer questions like “Did the 95th‑percentile latency spike during the burst phase?” without you manually digging into logs.

With the rule “always correlate client latency with server health” baked into your test suite, you’ll stop getting false‑green results and start trusting the agent’s verdicts.

Everything that breaks once you stop watching the screen

Guillermo Leyendeker — Mon, 05 Oct 2026 12:31:11 +0000

An agent that works fine in a ten-minute demo and one that runs on its own for months are not the same system. The difference isn't architectural: it's dozens of edge cases that only surface with time on the clock.

Clocks drifting out of sync. A machine that suspends overnight. A parser that breaks on an unexpected prefix. A browser that grows without a ceiling until it eats the memory. A rhetorical question, written by the agent itself, that stalled the entire pipeline waiting for an answer nobody was going to give.

None of these appears in a diagram, and together they consumed as much time as every design decision before them. If you're about to leave something running unattended, this is the list you'd want to have read first.

Clocks

My favourite, because it's the hardest to suspect. For a while, the same task could read as alive or dead depending on which part of the system was looking at it.

The cause: the log file's name was generated in one timezone and its status was evaluated in another. Neither was "wrong" — they were simply different. The result was that a perfectly active run showed up as stopped, or the reverse, depending on the time of day.

Timezone mismatches are the kind of bug you don't find by reading code, because each piece looks reasonable on its own. You find it by watching the screen and noticing it says one thing while the system does another.

The machine that goes to sleep

The watchdog kills processes that have been running too long. Simple, until the machine suspends overnight.

On waking, the watchdog computed elapsed time in wall-clock hours and found processes with eight or ten hours of "activity" that hadn't executed a single cycle. And it killed them, correctly from its own point of view.

It had to be taught to subtract host suspension time from the calculation. It's a three-line adjustment that only occurs to you after losing work to it.

Fragile parsers

The end-to-end test progress bar sat frozen at zero. The suite ran fine, the tests passed, but the indicator never moved.

The reason: the test runner includes the project name on each output line when more than one project is configured, and my parser expected the format without that prefix. One extra token and the counter stayed at zero forever.

It's a reminder of something I badly underestimated early on: almost all integration with external tools runs through parsing their text output, and that output changes without notice, through configuration or version.

Memory that grows on its own

The browser running the interface reviews grew in memory without a ceiling. It wasn't a leak in my code: it was the normal behaviour of a browser held open for hours, accumulating tabs, contexts and cache.

The fix wasn't elegant and didn't need to be: recycle the containers when they cross a RAM threshold. Long-lived processes don't get fixed, they get restarted periodically.

States that lie on screen

There was an entire family of bugs around the interface showing things that weren't so.

Rows spinning indefinitely with an activity indicator, because the active task's identifier was compared incorrectly and no match ever switched it off. Live runs reading as stopped. Steps that did nothing but claimed the run button.

They all look cosmetic and none of them are, because the state on screen is what you use to decide whether to step in. An indicator that lies makes you wait for something already finished, or interrupt something that was working fine.

Operations that can't tell empty from broken

Read-only steps — the ones that explore code without modifying anything — still triggered the automatic push on completion. Since there was nothing to push, the operation failed, and that failure was reported as though the task had gone wrong.

It's the same pattern as the exhausted turns from the previous entry: the system couldn't distinguish "there was nothing to do" from "something went wrong". And that confusion generates noise which eventually teaches you to ignore the warnings, the worst possible outcome.

The rhetorical question that jammed everything

One skill ended its runs with a question along the lines of "shall we continue with the next part?". It wasn't expecting an answer: it was a closing formula.

But the orchestrator detects runs left waiting on user input, so they don't hang indefinitely. A question at the end, however rhetorical, triggers that detection and aborts the whole sequence.

The fix was to explicitly forbid closing on a question. It's also the case that gave rise to the self-improvement loop from the fourth entry: it was the canonical example of "I caught this by hand, the system should be catching it".

Other people's infrastructure falling over

When the provider's API returned a server error, the entire queue stopped. A thirty-second blip left the afternoon's work standing still.

Now those errors retry the step instead of killing the sequence. It's the same idea as in the entry on failure policy: you have to separate failures of the work from failures of the environment, because they call for different responses.

The pattern behind all of them

Looking at the full list, half of these cases weren't discovered by reading code. They were discovered by watching the screen and noticing something said one thing while the system did another.

And almost none of them are agent problems. They're old distributed-systems problems: desynchronised clocks, miscalculated timeouts, long-lived processes accumulating state, fragile parsers, external dependencies failing. They reappear intact because an agent running for hours is, for all practical purposes, an unreliable long-lived process.

If I had one piece of advice for someone about to build something like this, it would be: the interesting part — the prompts, the architecture, the retry policy — is the part you'll solve quickly. The part that eats your time is what keeps the system truthful when nobody is watching it.

The full arc

That's the path of these four months. It started as a screen with buttons for launching agents, with two commits devoted to one button's icon, and ended as a control system with distrust, budgets and auditing.

Today it has closed 2,095 fixes and features, plus 654 audits, on real projects.

The screen is still there. It's no longer the important part.

Your agent ran out of turns. Your system logs it as an error.

Guillermo Leyendeker — Wed, 30 Sep 2026 18:00:30 +0000

An agent that runs out of budget and an agent that gets it wrong end the same way in your terminal: exit code 1.

For two months my system treated them as the same case, which is wrong, because they call for opposite responses: one resumes, the other gets fixed.

Behind that confusion sits a number almost nobody calibrates and everybody leaves as it came: how many turns to give the agent before cutting it off. With no limit, it can spend hours circling something it won't solve. With too low a limit, it abandons work that was two steps from done.

That number isn't a guess. Only in August, with 30 days of real runs in the database, could I build a table of budgets per task type instead of using the same one for everything.

How it was before: no limit and a hammer

At first there was no turn cap at all. Every step ran with the global default configuration, and the only brake was a watchdog that kills stuck processes after an hour.

A time-based watchdog is a blunt instrument. It can't tell an agent working well on something hard from one stuck in an unproductive loop: both burn an hour the same way. All it guarantees is that nothing runs forever.

That watchdog also had its own operational lesson, which I cover in the final entry: the machine suspends overnight, so suspended time had to be subtracted from the calculation. Otherwise it killed healthy runs over hours that had never actually elapsed.

In June I added a global turn cap as a backstop against runaway runs. Better than nothing, but still a single number for tasks that differ wildly from one another.

The table the data produced

In August the cap became per task type, calibrated by looking at how the last 30 days of runs had actually behaved.

What I like about the result is that the table has a logic you can explain without looking at the numbers:

The verifier is the most generous, at 160 turns. That makes sense: it reviews someone else's work, and reviewing properly means reading far more than you write.
The fix implementer follows at 150. A fix usually starts without knowing where the problem is, so a good share of the budget goes into finding it.
The frontend implementers sit around 90 to 110, depending on whether they work on the admin or the public application.
The explorer gets 60. It only has to map the terrain and return the touch points: if it needs more than that, something is wrongly framed.

Above all of it sits a global safety ceiling of 200 turns, which no task should ever reach and is there just in case.

The mistake that took me two months to understand

And now the part I consider most important in this entry, because it isn't about the number but about what running out of it means.

For the first two months, an agent that exhausted its turns showed up in the system as "exit code 1". That is: exactly like any other error.

That's a serious diagnostic problem. Running out of budget and failing are completely different things:

A real failure means something went wrong and you need to investigate what.
Exhausting the budget means the work was in progress and got interrupted. Nothing is broken: something is unfinished.

Confusing them sent me hunting for bugs that didn't exist, more than once. And the inverse, which is worse: assuming certain failures were "the turn limit again" when they were legitimate errors.

In August I separated them. Running out of budget now has its own message, states clearly what happened, and no longer kills the sequence: the step goes back to the queue instead of dragging the whole task into failure. Which is correct, because the work done up to that point still stands.

Budgets are editable without touching code

A small decision with a large impact: the per-task caps are editable from the interface, with no restart.

The reason is that calibrating them isn't a one-time job. The model changes, project complexity changes, the skills change. A number you adjust by editing code and redeploying is a number that in practice never gets adjusted.

The same criterion, applied to the model

There's a related decision I made at the same time with the same shape: not every step deserves the same model.

The verifier — the most expensive step in the system, that 29% of the bill I described in the entry about measurement — deliberately runs on a cheaper model. It verifies against explicit criteria, and that doesn't require the most capable model available.

Meanwhile, the component making autonomous decisions about what to do next runs on the highest tier with maximum reasoning effort. It's the one that executes least often and has the worst consequences when it gets things wrong.

It's the same idea as the turn budgets, applied along another dimension: budget should follow consequence, not volume.

What I learned

A turn budget looks like a technical parameter and is a product decision: it defines how much autonomy each task gets before someone interrupts it.

And the expensive mistake isn't getting the number wrong — that's corrected with data, and the data shows up on its own if you're measuring. The expensive mistake is failing to distinguish "ran out of budget" from "failed", because the first is resumed and the second is investigated, and confusing them wastes your time in both directions.

What's left to tell

Everything I've covered so far is design decisions: things you think about, plan and implement.

The final entry is about the other half of the work, the half that appears in no architecture diagram and consumed half the time: everything that breaks when you leave the system actually running, for months, on a machine that suspends, restarts and runs out of memory.

You restart the server and the agent loses all ten tasks

Guillermo Leyendeker — Mon, 28 Sep 2026 12:30:19 +0000

You leave the agent working through a queue of ten tasks. You need to deploy a server improvement, you restart it, and all ten are gone.

Not just the pending work: also any trace of which one was running and what state it was left in. The queue lived in the process's memory, so any restart — a deploy, an update, a power cut — wiped it entirely.

And the server you're restarting is the very one you're improving every day. Every improvement forced a choice: wait for the queue to drain, or lose the plan.

The fix isn't "put the queue in a database" and done. What you actually have to decide is what a half-finished task means, and who gets to change its state when the process comes back up.

How it used to work

The original queue was an in-memory list. You ticked the tasks you wanted to run on screen, the order was the order you'd selected them in, and the system processed them one by one.

It worked as long as nothing interrupted it. With two serious problems.

The first is the one above: every time I wanted to deploy an improvement to the orchestrator, I had to choose between waiting for the queue to drain or losing the plan.

The second, quieter one: the first failure aborted the whole queue. An error in the third of ten tasks cancelled the remaining seven. And those seven had nothing to do with the one that failed: they were simply behind it in line.

The redesign

The queue moved to the database, with one design decision I particularly like: position is a real number, not an integer.

That allows inserting between two elements without renumbering the whole list. If you want to slot something between positions 3 and 4, you assign it 3.5. It's an old technique and it solves at the root the problem of reordering a persisted list without writing fifty rows every time someone drags one.

The order is curated by hand, mixing fixes, features and audits into whatever sequence makes sense that day. You reorder by dragging, as long as tasks are still waiting. Whatever is running is pinned and locked, which is the bare minimum: you can't reorder something that already started.

What to do when a task fails

There was a product decision disguised as a technical detail here, and I resolved it by not resolving it: I made it a per-run option.

You can choose for the queue to carry on with the rest, flagging the failed one, or to stop and leave everything else waiting. Both positions are defensible depending on the case. If you're processing ten independent fixes, you want it to continue. If the third failed because the environment broke, you want it to stop before the remaining seven fail for the same reason.

Later I added a stricter rule that turned out to be the most useful in practice: the queue doesn't resume past an unresolved failure. You can continue, but someone has to have looked at that failure and decided what to do about it. Without that, the queue stays put.

It's deliberately inconvenient. The alternative — always continue — lets failures pile up unseen, which is exactly the problem behind the 1,771 unprocessed incidents I described a few entries back.

A single execution mechanism

A side effect of the redesign I hadn't anticipated: the "run everything" and "repair everything" buttons stopped being separate mechanisms.

Before, each was its own pipeline, with its own advancement logic and its own bugs. Now they're simply shortcuts that bulk-enqueue with a preset order and start the queue. One execution mechanism, all of it visible and reorderable.

It's the kind of simplification that appears on its own once the right abstraction finally exists. While the queue was ephemeral, having parallel pipelines seemed reasonable. With a persistent, curatable queue, it makes no sense at all.

Actually surviving a restart

Persisting the queue solves half the problem. The other half is what happens to whatever was running at the moment of the restart.

That task lands in limbo: the database says it's in progress, but the process executing it no longer exists. If nobody does anything, it sits there forever, occupying the head of the queue and blocking everything behind it.

In August I added the two missing pieces: graceful shutdown, so in-flight runs are marked before dying, and startup reconciliation, which checks what's declared as running with no process behind it and resolves it.

And a rule that seems obvious and took me a while to find: re-read the task's status before launching its row. Because between enqueuing and its turn arriving, anything could have happened — the task might have been completed by another path, or postponed. Launching blind against a stale snapshot is a constant source of duplicated work.

Why this sums up the whole project

Of everything I built over these months, the persistent queue is what best captures the thesis: in a system with agents, the state of in-flight work matters as much as the work itself.

An in-memory queue works perfectly until the first restart, and the first restart always arrives at the worst moment. It isn't an optimisation: it's the difference between a tool you use while you're watching and a system that runs when you're not.

Today the queue is the system's only execution path. There's no way to run a task outside it, and that — which felt like an annoying restriction at first — turned out to be what makes everything else observable.

With the queue surviving restarts, the system could run on its own for hours with me nowhere near it. And that raised a question I'd never asked, because I'd never let it run that long: how long to let it think before interrupting. That's what the next entry is about.

Verifying two lines from the agent cost me an hour

Guillermo Leyendeker — Wed, 23 Sep 2026 18:01:19 +0000

The agent closes a two-line task. Verifying it hadn't broken anything took me close to an hour.

And not one check was superfluous. I'd added the end-to-end tests because they caught a real bug. The security review, because it caught another one. Each had earned its place, and for exactly that reason none of them could be skipped.

It's a natural slope: you add a check because it helps; because it helps, you don't want it skipped; so you make it mandatory. Repeat that five times over three months and you end up with an hour of checks hanging off every task, however small.

What unblocked it wasn't deleting checks or making them faster. It was changing the question: not is this check useful? — they all were — but what decision does it block?. Some have to stop the task. For others, stopping the deploy is enough.

The expensive part was the interface review

Within the gate, the slowest check was the model-assisted interface review: an agent that opens the browser, walks the application and judges whether what was implemented looks and behaves as it should.

It's valuable. It catches things no automated test detects. And it's enormously expensive in time, because it means bringing up the environment, navigating, waiting on every screen and reasoning about what it sees.

Running that on every task was the equivalent of asking a designer to review the entire application every time someone changes a line of code.

The right question isn't whether the check is worth it

The change of question reordered everything, and the reason is economic before it is conceptual: a control that stops a task from closing and one that stops a deploy to production are not the same thing. Their waiting costs are completely different. The first is paid dozens of times a week; the second, once per release. Mixing them is what leads a team to eventually disable both.

That's where the two tiers came from.

Tier 1: what blocks each task

Fast, deterministic and per task. Three things: the full backend suite, the end-to-end tests belonging to that task running headless, and a security trace.

The key part is "belonging to that task". Not the full end-to-end suite: only the specs covering what was just touched. That's what takes the time from an hour down to minutes.

Tier 2: what blocks the deploy

Slow, expensive and batched. That's where the full end-to-end suite and the model-driven interface review went, the latter running over tasks Tier 1 didn't cover.

That batch runs at two moments: before deploying, blocking, and every night as a safety net. If something fails, it doesn't jam the day's work — it jams the release, which is where that control actually matters.

On screen the split was made visible with two separate columns, one per tier, each with its own state. Visibility matters: if a check doesn't block, it has to be crystal clear that its result still exists and that someone is going to look at it.

The delicate part: changing something everything depends on

This is the part that gave me the most trouble and the one least often written about.

The quality gate is a load-bearing piece. The completion steps refuse to mark a task as done if the gate didn't pass. And that pass, as it was written, explicitly required the interface review's result.

Which means: if I did the intuitive thing — pull the interface review out of the per-task gate — the entire system would stop being able to close tasks. Not some of them: all of them. Every completion would start failing while looking for data nobody produced anymore.

That's why the plan carries a rule written in capitals: if you stop producing the signal before you stop requiring it, every completion breaks.

The implementation order was fixed in five stages, and the first is the counterintuitive one:

First, decouple the requirement. Make the pass stop requiring the interface review. It's a backward-compatible change: if the signal is present it's ignored, if it's absent it no longer blocks. At this point the system behaves exactly as before.
Then add the new Tier 1 checks, which is purely additive.
Only at the end, make the skill stop producing the per-task signal. It's safe now, because nothing has required it for two steps.
The screen columns and the batch runner, which are net-new and don't touch the existing path.

It's the kind of sequence that looks bureaucratic when you write it and saves your day when you execute it. Had I started with the obvious step — removing the interface review — I would have broken task completion across every managed project simultaneously.

A detail that avoided a lot of pain

Every new check reads the project's configuration and skips itself gracefully when the project doesn't declare what it needs.

Some managed projects have no end-to-end tests at all. If the gate demanded them, those projects would be permanently stuck behind a requirement that doesn't apply. The rule that stuck is: the gate can't demand what the project doesn't declare it has.

It sounds obvious written down, but it's the kind of thing you discover when the same mechanism runs across projects of different shapes.

The outcome and what followed

The goal stated in the plan was to bring the per-task gate down from roughly an hour to minutes, across every project at once. That was achieved.

And with the system finally running fast, a problem the slowness had been masking came into view: every time I restarted the server, I lost the entire work plan. That's what the next entry is about.

I used my agent's memory 1,112 times and I don't know if it helped

Guillermo Leyendeker — Mon, 21 Sep 2026 12:31:15 +0000

Your agent has memory: it stores what went well and what went wrong, and injects it into itself when it starts a new task. It's one of those features you switch on once and never look at again.

The uncomfortable question is simple: if you turned it off tomorrow, would you notice?

Mine ran 1,112 times in 30 days. When I sat down to evaluate it I couldn't say whether it helped, and not for lack of instrumentation. Memory is the one component in the system that never complains, and that's why more than a month went by without my looking at it once.

What it actually does

The system stores two kinds of thing. Patterns that worked — ways of solving something that produced good results — and known failures, so they aren't repeated. At the time of measuring there were 127 patterns and 180 failures on record.

When an agent starts a task, the system searches that store for whatever resembles the work about to be done and injects it into the context. The search is by trigram similarity, with a threshold of 0.30 and a cap of three results per query.

It's simple, it's cheap and it runs without errors. That last point turned out to be the problem.

The component that never complains

Everything else in the system makes it onto my review list because it makes noise. The quality gate fails and stops a task. The watchdog kills a process. An agent runs out of turns. The queue jams. Each of those events produces a signal that forces me to look at it.

Memory doesn't. It gets injected, the agent works, the task closes. No error, no alert, nothing to review. It became invisible through good behaviour.

And so a good month went by with over a thousand accumulated injections, without anyone — me — asking whether those injections were improving anything or simply taking up room in every agent's context.

The textbook technical answer, and why I didn't take it

When I finally sat down to look at it, my first reaction was the obvious one: trigram search is primitive, this should move to embeddings.

It's true that it's primitive. It compares character sequences, not meaning. It's the kind of retrieval anyone would replace today with a vector search, and there are libraries to do it in an afternoon.

I didn't, and I think that was the right call. Improving retrieval without an impact metric is betting blind, and paying for the privilege: embeddings carry a per-query cost, so I'd be spending more to improve something I don't even know contributes anything.

The right question wasn't "how do I make retrieval better?" but "is retrieval making any difference?". And I had never asked that second one.

The infrastructure to measure it already existed

Here's what surprised me most: I could measure it, and I didn't know.

Every memory injection records the identifier of the session it happened in. That, which I'd added without thinking about it much, enables a natural experiment: there are sessions with memory injected and sessions without, on the same system, over the same period, doing comparable work.

It's an A/B that's been running for months. All that was missing was looking at it.

The cross-reference to make is against two things also on record: each session's incidents and the quality gate's outcome. The questions are concrete. Do sessions with memory injected repeat fewer reads of the same file? Do they run fewer searches that find nothing? Do they pass on the first attempt more often?

If the answer is yes, memory helps, and only then does investing in better retrieval make sense — with the cost justified by a measured effect.

The question almost nobody asks

But there's a third possible outcome, and it's the one I find most interesting: that memory doesn't move the needle at all.

If that were the case, the correct conclusion isn't to improve it. It's the opposite: raise the similarity threshold, lower the result cap, inject less. Because every injection carries a cost that is certain — it consumes context, and context is finite — against a benefit that would be zero.

That's the uncomfortable part of measuring seriously: one of the possible outcomes is having to dismantle something that took work to build. I suspect the resistance to measuring comes largely from there, not from technical difficulty.

The criterion I took away

A component that never fails isn't the same as a component that helps. They're two different things and they're easy to confuse, because day to day you operate by exception: you look at whatever complains.

Now, before adding any new piece to the system, I try to answer up front: what would the world look like if this piece didn't exist? If I can't answer that, it isn't that the piece is wrong — it's that I'll be stuck with it, unable to ever justify it.

With memory I arrived at that question late. I'm 1,112 injections in and still don't have the answer.

By July the system was running fast and verifying well, but every task still dragged close to an hour of checks behind it before it could close. The problem was no longer what to check, but what should block the work and what shouldn't: that's what the next entry is about.

Your agent says the tests pass. Did it run them?

Guillermo Leyendeker — Mon, 14 Sep 2026 12:31:16 +0000

Your agent finishes a task and tells you the tests pass. It's worth asking whether it ran them or is reading the result from the previous run.

Caching that result when the code hasn't changed is an obvious optimisation: an afternoon's work and the pipeline flies. I did it the same day it occurred to me.

What nobody writes down is what you're asserting when you do: that the code that was tested is the code you're about to ship. When that doesn't hold, the system doesn't get slow. It signs off on code that was never tested, and you don't find out.

What looked like a trivial change ended up resting on two conditions, plus a marker that makes it auditable afterwards. Miss any of the three and the cache stops being an optimisation.

The starting point

The gate ran its checks serially, one after another. The full suite, with a 40-minute timeout. Then each frontend build, at 5 minutes apiece. And finally mutation analysis, at 20.

That 40-minute timeout says a lot: back in May I had raised it from 10 to 40, because the suite no longer fit. The curve pointed in one direction only.

But duration wasn't the worst of it. This was: when a task failed the gate and had to be retried touching only the frontend, the re-verification ran the entire backend suite again, which hadn't been modified at all. Minutes of compute to confirm something we already knew.

Two changes, one obvious and one delicate

The obvious one: run the suite and the frontend builds in parallel instead of serially. There's no reason for them to wait on each other beyond the fact that's how it was written.

The delicate one: cache the green result of the suite by the combination of project, repository and commit hash. If we already ran the tests against exactly this code and they passed, there's no need to run them again.

Put that way it sounds harmless. This is exactly where the cache stops being an optimisation: from this point on, the green the system reports doesn't come from running the tests. It comes from trusting that the code is the same.

The two conditions holding the promise up

That's why the cache has two locks, and neither is optional.

The working tree has to be clean. If there are uncommitted changes, the commit hash no longer describes what's on disk. The code about to be evaluated isn't the code that was tested. In that case the cache isn't even consulted: the suite runs.

The result is valid for 24 hours. Not because code changes on its own, but because the environment does: dependencies, containers, test databases, external services. A green from a week ago says very little about today's state.

There's a third, smaller detail I like: the suite's exclusion lock — the one preventing two runs from stepping on each other — is only acquired if something is actually going to run. When the cache hits, the function returns before reaching that line. Without it, we'd be serialising runs that will never happen.

The marker that makes auditing possible

This is the part I consider most important in the whole implementation, and the one I see written down least often elsewhere.

Every result coming from the cache travels flagged as such inside the quality gate's data. It's on record that this task didn't run the tests: it leaned on an earlier run.

What's that for? For the day a green shows up that shouldn't have. On that day I'll be able to reconstruct exactly which tasks leaned on the cache and which genuinely ran, and check whether that's where the problem was.

Without that marker, a false green is indistinguishable from a legitimate one. You have a bug you can't even scope, because you don't know which universe of tasks to search.

It's a principle I now apply to any optimisation replacing a real check with an inference: every automated promise needs to leave the marker that lets you audit it when it breaks. Because it will break eventually, and on that day the marker is all you have.

What was deliberately left out

Mutation analysis — which checks whether the tests actually detect changes in the code, not merely whether they pass — was kept out of the cache on purpose.

The reason is that it measures something different. The suite is evaluated against the state of the repository; mutations are evaluated against the specific task's diff. Two tasks on the same base commit can have completely different diffs, so caching by commit hash would make no sense: they're orthogonal.

What I did add was a short-circuit from the other side: if the task's diff contains no mutable code, the analysis simply doesn't run. There's nothing to mutate. That resolved a large share of cases without touching the underlying logic.

The outcome

Between the parallelism and the cache, the quality gate stopped being the obvious bottleneck it had been. It's still the most expensive step in the system — that 29% of the bill didn't move as much as I expected — but the wait per task dropped noticeably.

What stayed with me above all was a criterion I've reused several times since: speeding up a check is fine as long as you can prove afterwards that the check existed. If the speed-up leaves you unable to audit what was verified and what wasn't, you're not optimising — you're loosening the check and calling it something else.

The front left open

That same June review left another item on the table, one that wasn't about speed and was considerably more uncomfortable.

There was a component I'd been using for over a month, running more than a thousand times a month, and I couldn't say whether it improved anything. It never failed. That was exactly the problem.

That's what the next entry is about.

29% of my bill was in the step I never looked at

Guillermo Leyendeker — Mon, 07 Sep 2026 12:30:35 +0000

If you use agents daily you already have an intuition about where the money goes. Mine pointed at the retry: the agent fails a check, starts over from scratch, and you watch it on screen redoing something it had already done.

I wrote a five-point optimisation plan based on that intuition. Then I went to the database for the baseline I was going to measure the improvement against, and found the retry had fired exactly once in the system's entire history.

The spend was somewhere else: in the verification that runs before each task closes, which took $3,463 of the $11,968 the system spent in total. 29% of the bill in the dullest step there is, the one I never looked at because it always worked.

The lesson holds for anything you automate with models: what annoys you are the exceptions, because they interrupt you. What costs you is the routine, because it happens every time.

The obvious suspect

All five points of the plan aimed at the same place: the retry loop.

The reasoning was solid. When work fails the quality gate and has to be retried, you pay for everything again: another agent spun up, another exploration of the code, another implementation, another gate run. It's by far the most expensive operation in the system. And on top of that it was badly implemented: it discarded the previous session, so the second attempt started with no memory of the first.

All of that was true. The five diagnostic points were correctly identified. The plan opened with making retries smarter and continued with minor optimisations.

Then I went to the database to find the baseline number I'd be measuring the improvement against.

What the data said

I found three things that fit nothing I had written:

A single item, across the system's entire history, had ever produced retry feedback.
Zero surrenders. The give-up path, carefully designed with its structured incident report, had never once fired.
The first-attempt pass rate was effectively 100%.

The retry loop I was about to optimise barely ran. I had written an improvement plan for the least-travelled path in the system.

The explanation, once I looked for it, made sense: errors weren't being corrected between orchestrator steps. They were corrected inside each agent's own loop, before the work ever reached the quality gate. The agent iterated on its own output, caught its mistakes and fixed them before handing anything over. To the orchestrator that's invisible: all it sees is a step that finished cleanly.

So my intuition wasn't just wrong about the size of the problem. I was looking at a layer of the system where the problem simply didn't happen.

The number that reordered the plan

The same query surfaced the figure that changed everything. The quality gate — the step that verifies each task before closing it — was running around 157 times a week, averaging 5 to 8 minutes each time.

That was the dominant cost. Not retries, which barely happened: verification, which happened every time.

And the finding held up over time. Today, looking at the cumulative figures, the quality gate has consumed 3,463 dollars out of the 11,968 the system has spent in total. Twenty-nine percent of the bill concentrated in one step.

The plan was rewritten that same afternoon. First, parallelise the gate and cache the test suite. Retry work was demoted to a cheap robustness improvement, for when the worst case actually occurs.

Why intuition pointed the wrong way

I've thought a fair bit about why I got it so clearly wrong, because the error wasn't analytical: it was perceptual.

Retries hurt. You see them on screen, they show up as a loop repeating, they feel like waste while they happen. They're visible, noisy and frustrating.

The quality gate didn't hurt because it was routine. It ran every time, it was part of normal operation, and for exactly that reason it was invisible. Nobody looks twice at a step doing what it's supposed to do.

Which is why what annoys you most and what costs you most are almost never the same thing. The exception interrupts you and never shows up on the invoice. The routine never bothers you and is the whole invoice.

A note on how I recorded the mistake

When I rewrote the plan, I didn't delete the original diagnosis. I added a dated note above it with the recalibration: what I had believed, what the data showed, and how that changed the priority.

I could have rewritten the document cleanly, with the correct diagnosis from the start. I didn't, for a practical reason: the wrong diagnosis still has value. It explains why the system is built the way it is, and it reminds me that the reasoning behind it was sound — reading the code, anyone would have reached the same conclusion.

A plan that hides its own diagnostic errors becomes an internal marketing document. One that leaves them visible, with the date attached, is a record of how something was learned.

Where to start in your own system

The order isn't guessed, it's queried. Before writing an optimisation plan, list your pipeline's steps with two columns — what each run costs and how many times it runs per week — and sort by the product of the two. Frequency is what surprises you, almost never unit cost: the wildly expensive step that runs once a month isn't your problem.

The first task in the reordered plan sounded trivial: stop running the full suite when the code hadn't changed. It had a trap that nearly had me certify as passing a codebase that was never tested, and that's what the next entry is about.

I asked the AI to fix its own instructions and it barely helped

Guillermo Leyendeker — Mon, 31 Aug 2026 12:30:30 +0000

Every time the system gave up on a task, it left a record of what it had tried and why it failed. One day I looked at the pile of those reports and thought the obvious thing: the AI can read this and fix its own instructions.

I built the whole thing, with one rule from day one: the routine edits nothing on its own. It proposes the change, I approve it with a click.

It works. I collected 1,771 recorded failures over three months, it produced 13 proposed fixes, and the system barely improved through that path.

The model didn't fail and neither did the human approval. Something far more boring did, and it's a mistake that repeats in any system that tries to learn from its own logs.

The case that started it

It wasn't an abstract idea: it came from a concrete, repeated problem.

One of the system's skills had a habit of ending its runs with a question. Something along the lines of "shall I continue with the next part?". To the orchestrator, a run that ends waiting for input is a stuck run: it aborted the whole sequence.

I fixed it by hand, editing that skill's instructions. And while doing it I thought the obvious thing: this pattern is on record. The system knows this skill failed for this reason, several times over. Why do I have to be the one who notices?

The goal ended up written like this: replace the cycle human detects, human edits, human commits with the system detects, proposes a patch, the human approves with one click.

The constraint I set on day one

Before writing a line, I decided something I still consider the best decision in the whole subsystem: the routine never edits files or commits on its own.

Its output isn't a change, it's a proposal: the diff plus the reasoning behind it. It's stored in the database and waits. A human approves it and only then is it applied, or rejects it and it's archived.

The temptation to skip that is strong, because manual approval is precisely the bottleneck you're trying to remove. But a system that modifies its own instructions without supervision has no brake at all: if the criteria it judges by are wrong, every iteration makes them worse, and there's nothing outside the loop to catch it.

Four phases, all implemented

The design ended up in four parts, and all four are running:

A unified incidents table, consolidating signals that previously lived scattered: orchestrator surrenders, skills that stall waiting for input, pre-flight checks that fail.
Recurring-pattern detection: it groups by skill and reason, and flags anything appearing three or more times in seven days.
The routine that proposes the patch, which decides whether the fix belongs in the skill's instructions or in code, computes the diff and persists it.
The approval interface, with a 24-hour limit so the same proposal doesn't come back over and over.

None of this is broken. It all runs.

The numbers, three months later

When I went to measure the outcome, this is what I found:

1,901 incidents recorded in total.
1,771 of them still open, unprocessed.
13 proposals generated across the system's entire history. Seven approved, six rejected.

Thirteen. In three months, with nearly two thousand incidents of raw material.

And what stings most is that the most frequent incidents are exactly the ones you'd want to turn into rules. Over fourteen days: 351 repeated reads of the same file, 154 searches that found nothing and 151 failed commands. All actionable patterns. All sitting there, unprocessed.

Where it failed, which isn't where you'd expect

My first reaction was to assume the model wasn't up to the task. That's the comfortable explanation, and it was wrong.

The loop doesn't fail at the intelligent part. It fails in the plumbing. The hypotheses I still have open — and haven't finished ruling out — are three, and none of them involve the model's capability:

That the proposal routine is barely firing at all, because of how its scheduling ended up configured.
That the recurring-pattern view isn't grouping correctly, so it finds nothing to report.
That the threshold of three occurrences in seven days is unreachable — not because patterns don't repeat, but because there's so much background noise that the real cases get diluted.

That last one interests me most, because it implies the problem isn't detection but signal quality. If the inefficiency detector produces false positives at scale, it doesn't matter how good the next stage is: you're asking it to find patterns inside a pile of noise.

So the next step isn't improving the routine that proposes patches. It's triaging the 1,771 open incidents and answering something more basic: how many are real signal and how many are noise from the detector itself. If the bulk is noise, detection has to be tightened before asking anyone for proposals.

What I took from this

A self-improvement loop doesn't fail for lack of intelligence. It fails in the plumbing, which is the part nobody shows in demos.

Recording signals is easy and cheap, which is why it's the first thing you build. It feels productive: the numbers go up, the table fills, it looks like the system is learning. Turning that signal into action is the real work, and it's exactly where the project stalled.

Accumulating 1,771 incidents without processing them isn't telemetry. It's an expensive log that looks like a dashboard.

I now apply the lesson before instrumenting anything new: if I can't say what decision someone — or something — will make with that data, the instrumentation can wait.

The second time the data contradicted me

This was the first result that forced me to go look at the database instead of trusting what I thought was happening.

A month later I did the same with the rest of the system, in a general review aimed at improving pipeline speed. I had a diagnosis put together and a plan written on top of it.

The data made me throw that plan out the same day. That's what the next entry is about.

Your agent fails and sits there waiting for you

Guillermo Leyendeker — Sun, 23 Aug 2026 09:00:59 +0000

You hand the agent a task, go do something else, and when you come back it's sitting there waiting for you: something failed and it didn't know what to do.

That moment decides whether the agent saves you time or eats it. If the answer when work comes out wrong is "it stops and waits for you", you can leave it alone for ten minutes, never for an afternoon.

That's what I had by early May. The system could already detect that a piece of work was wrong, and what it did next was nothing. The problem wasn't technical: I had never decided what should happen after a failing verdict.

And there the options aren't two: fix what it did, start over with a different approach, or give up and leave a record of what happened. Having it written down when each one applies is what separates an experiment from something you can leave running alone.

Separating the one who does from the one who judges

The first change was structural: explicitly separating two roles. One generates the work, the other evaluates it. This isn't an organisational nicety — it changes what information travels between steps.

As long as the evaluator returns "passed" or "failed", the only thing you can do with a failure is retry blindly. If instead it returns structured feedback, the system can decide what to do with it.

Evaluation became multidimensional. Instead of a single verdict, four axes: correctness, security, user experience and completeness, each with its own score. And something that turned out more useful than I expected: regression detection against previous attempts, to tell "still wrong" apart from "worse than before", which are different problems.

Three paths, not one

With that information in hand, the system picks between three exits. This is the part I think is missing from most orchestrators, where failure handling boils down to retrying.

Refine: the work is on track but incomplete or has specific defects. It goes back to the same implementer with feedback about exactly what to fix.
Pivot: the work is wrongly framed. Fixing on top of it won't help, because the starting point is the problem. The task is handed back a step up the chain.
Surrender: after three attempts, the system stops pushing.

Surrendering isn't failing silently, which would be the worst of both worlds. It writes a structured incident report: what was attempted, with what feedback, which dimensions failed each time. That report is stored and becomes material to review later.

In fact, looking at the pile of those accumulated reports is how I came up with the most ambitious idea of the project, which is the subject of the next entry and turned out considerably worse than I expected.

The mistake that took me a month to see

There's a detail in this implementation that was wrong for a full month, and it illustrates how easy it is to waste context without noticing.

When the system decided to refine, it relaunched the implementer. But in doing so it discarded the previous session. The agent started from scratch: no memory of which files it had touched, which paths it had ruled out, or why it made the decisions it made. All it received was a block of text with the evaluator's feedback.

Put another way: I was asking it to fix work it didn't remember doing.

The fix, once I finally saw it, was one line: keep the session when the decision is to refine, and discard it only when pivoting, which is the one case where forgetting is deliberate — there you want the model starting without the bias of the failed attempt.

What makes that safe is a limit that already existed: since surrender cuts in at the third attempt, a refine with a live session can happen at most twice. Context doesn't grow indefinitely; it has a structural ceiling.

Telling apart failures that aren't the agent's

Something else I learned while operating this: not every failure is the worker's fault.

When the provider's API returned a network error, the system counted it as a failed attempt. The agent had done nothing wrong — it hadn't even started — but one of its three attempts was consumed. With two consecutive network blips, a perfectly healthy task reached surrender without anyone having evaluated anything.

Today those errors don't consume an attempt and are retried with backoff. Penalising the model for an infrastructure outage is, quite simply, measuring the wrong thing.

The fourth path

In August I added one more exit, for a case I hadn't foreseen: the impossible pivot.

Sometimes the evaluator determines the work is wrongly framed, but handing it a step up makes no sense because that previous step doesn't exist or is already exhausted. Before, that fell straight through to surrender. Now, instead of giving up, the system re-explores the terrain: it relaunches the code reconnaissance to rebuild context and retries from there.

It's an example of something that recurs throughout the project: failure policies aren't designed well up front. They're discovered by operating, when the case appears that doesn't fit any of the exits you'd anticipated.

What this buys you

A system with agents needs a failure policy as explicit as its happy path. And that policy has to distinguish three things that look alike at first: work that's close and deserves another pass, work that's wrongly framed and needs to go back, and work that has to be handed to a person.

Retrying without that distinction is burning money: you repeat the same mistake with the same information and expect a different outcome. And failing to separate surrender from the rest is worse still, because you lose the one case that actually deserved your attention.

With the three exits implemented, the system started resolving most of its own stumbles on its own. And what settled at the bottom were the reports of the ones it couldn't.

That pile of reports was the starting point for what comes next: asking the system to improve itself.