What I learned building an agent pipeline — and where I stopped

Post 8 of 8 in the game-factory series.

The runaway

One afternoon I watched an agent spend three hours and 138 dollars trying to edit a single file.

It was the Builder — the agent whose job was to take a theme spec and turn my slot machine's React code into a themed variant. It had opened the main game component, a 1,900-line file, and started making small surgical edits. One of them broke the syntax. The model ran the compiler itself, read the error, tried to fix it, broke something else, and kept going — nothing in the loop forced it to stop on a failed build. By the time I killed it, it had made 868 calls to the model. The file was a wreck: two export default statements, duplicate return (...) blocks, orphaned JSX floating between them.

Nothing about that failure was exotic. The agent did exactly what I told it to do. The problem was what I told it to do.

What the factory was

The short version, if you didn't read post 1: I built a real slot machine on AWS to learn the iGaming domain, then turned the theming into a pipeline of agents. Describe a theme in a sentence, and six agents in sequence design a spec, generate icons and a background, rewrite the code, test it in a browser, and deploy it. Designer, Image-Gen, Background-Gen, Builder, Tester, Deployer.

Five of those six behaved. The Builder was the one that kept me up. Every other agent in the pipeline had a clean output — a spec, a set of images, a test report, a deployed URL. The Builder's output was a code change to a large, interconnected React app, and code changes are where a model's mistakes compound instead of just being wrong.

The fix that mattered most

The Builder's task was: take the casino's SlotGame.js and edit it to match the theme. The file was 1,900 lines. The model couldn't do it reliably — it guessed at whitespace, patched the wrong one of twenty near-identical calls, broke the parse with unescaped apostrophes, and each miss compounded.

The insight I was slow to reach: the model was bad at editing that file because the file was hard to edit. It mixed theme-specific content into structural code. So I stopped asking the model to edit it at all.

I moved the theme-specific text into a config file the component reads at runtime:

// before: baked into the component, edited by the model
setMessage(`🎉 JACKPOT! You won ${spinData.win} credits! 🎉`);

// after: read from a config the component loads
setMessage(formatMessage(themeConfig.winMessages.jackpot, { credits: spinData.win }));

Now the Builder's job for that file was: copy it unchanged, and write a theme-config.json next to it. Generating a JSON config from a spec is ordinary Python — no model involved. There is no 1,900-line edit anymore.

I made the config back-compatible — with no theme specified, the defaults are the original strings, so the untouched casino renders identically. That mattered, because the casino is a live thing and I wasn't going to break it to make theming easier.

The result: the Builder became almost entirely deterministic. Copy the components verbatim, generate the config, run the color and font and API passes in plain code. The only step left for the model was optional cosmetic CSS — and even that was still the flakiest step in the pipeline.

What I observed across all six agents

Each agent taught something specific (posts 3-7 cover those), but patterns emerged across the whole pipeline:

In this pipeline, generation was reliable; editing was not. The Designer generating a spec from a conversation: reliable. Image-Gen producing icons from prompts: reliable. The Builder patching a large, interleaved file with an exact-string-match tool: fragile. Models can edit effectively with better tooling — AST-aware editors, bounded context, validation loops. But with the tools I had, every time I moved work from "edit this" to "generate that," reliability improved.

Explicit, inspectable handoffs made debugging possible. Every stage's output was a file on disk — a JSON spec, a set of PNGs, a test report. When a build came out wrong, I could open the intermediate files and see exactly where the pipeline diverged. The requirement isn't "files specifically" — it's that each handoff is durable, immutable, and something you can inspect without running anything.

Human gates belong where mistakes are costly. Three approval gates in the pipeline: the spec, the build plan, and the deploy. The spec because a bad spec poisons five stages downstream. The build plan because code modifications are hard to inspect after the fact. The deploy because shipping has external impact and rollback — while possible — carries risk. Every other stage ran without asking, because its output was cheap to redo.

Prompt caching changes the economics. The Builder's 868-call runaway cost $138 at Sonnet's list rates, with Bedrock's prompt caching enabled — each call paid a fraction of the full input cost for the conversation prefix it had already seen. Without caching, the same run would have been far more expensive. Caching made agent loops economically viable for normal runs. It also masked how expensive a runaway was becoming until an hour had passed.

Turn caps are the first safety measure, not the last. Before I understood why the Builder was failing, I added a sixty-turn cap. It didn't make the agent smarter — it made its worst case bounded. The capped runs that hit the limit aborted in minutes instead of hours and cost single digits instead of triple. Bound the loop before you trust the loop.

Test the deterministic core without the model. After the Builder redesign, I ran the whole deterministic phase — copy, config, code passes — with the model stubbed out, and compiled the result. It found four real bugs for zero dollars. Every part of an agent pipeline that doesn't require the model is testable like ordinary code. Pull work out of the model's reach not only for reliability, but so you can test it at all.

The honest ceiling

The pipeline runs end to end. It produces themed, deployed, playable games from a sentence. That's real.

It's also not a product. Here's what's still true:

• Icon consistency isn't solved. Each image generation call is independent. Thirty icons come back in thirty slightly different styles. Prompt engineering narrows the range; it doesn't eliminate the variance. The games look themed. They don't look art-directed.
• The Tester's navigation is fragile. Vision-as-judge works — a model looking at a screenshot can tell you whether the game looks right. But the model guessing CSS selectors to drive Playwright breaks often enough to produce false negatives. Accessible labels and stable test hooks would fix this. I haven't shipped that.
• The cosmetic CSS pass still wobbles. Even after shrinking the model's job to "optionally adjust shadows and glows," it still hits the turn cap on some themes. The deterministic build is the floor. The model polish is unreliable upward pressure.

I stopped there. The deterministic Builder was proven — it produced a themed, compiling build with no model involvement. The lessons had stopped being about slot machines and started being about agents in general.

What transfers

Five things, distilled from months of building this:

• Decide what should be an agent, not what could be. If a step can be generation instead of surgery, make it generation. Restructure the target before you blame the model.
• Bound every model loop before you trust it. Unbounded while plus a paid API is a liability, not a feature.
• Make every handoff explicit and inspectable. Files, versioned objects, whatever — as long as you can look at what one stage produced without running the next one.
• Back-compat is how you refactor a live thing without fear. Defaults that reproduce the original behavior let you change everything around them safely. The casino still works without a theme config; that's what made the redesign safe.
• Know when the lessons have outpaced the project. That's the signal to bank them and move on.

What's next

The game factory was a learning project. What it taught me about scoping agent work, managing cost, and designing for inspectability carries into whatever I build next. The factory itself stays where it is — functional, proven for what it does, but not something I'm polishing further.

The next thing is something different: applying what I learned here to a problem that isn't themed slot machines. The agent patterns transfer. The domain won't.

---

This is the final post in the game-factory series. The earlier posts cover each agent individually: the factory overview, Designer, Image agents, Builder, Tester, Deployer.