What I learned building an agent pipeline — and where I stopped

Post 8 of 8 in the game-factory series.

The runaway

One afternoon I watched an agent spend three hours and 138 dollars trying to edit a single file.

It was the Builder — the agent whose job was to take a theme spec and turn my slot machine's React code into a themed variant. It had opened the main game component, a 1,959-line file, and started making small surgical edits. One of them broke the syntax. The model ran the compiler itself, read the error, tried to fix it, broke something else, and kept going — nothing in the loop forced it to stop on a failed build. By the time I killed it, it had made 868 calls to the model. The file was a wreck: two export default statements, duplicate return (...) blocks, orphaned JSX floating between them.

Nothing about that failure was exotic. The agent did exactly what I told it to do. The problem was what I told it to do.

What the factory was

The short version, if you didn't read post 1: I built a real slot machine on AWS to learn the iGaming domain, then turned the theming into a pipeline of agents. Describe a theme in a sentence, and six agents in sequence design a spec, generate icons and a background, rewrite the code, test it in a browser, and deploy it. Designer, Image-Gen, Background-Gen, Builder, Tester, Deployer.

Five of those six behaved. The Builder was the one that kept me up. Every other agent in the pipeline had a clean output — a spec, a set of images, a test report, a deployed URL. The Builder's output was a code change to a large, interconnected React app, and code changes are where a model's mistakes compound instead of just being wrong.

The fix that mattered most

The Builder's task was: take the casino's SlotGame.js and edit it to match the theme. The file was 1,959 lines. The model couldn't do it reliably — it guessed at whitespace, patched the wrong one of twenty near-identical calls, broke the parse with unescaped apostrophes, and each miss compounded.

The insight I was slow to reach: the model was bad at editing that file because the file was hard to edit. So I stopped trying to make the model better at the edit and changed the thing being edited.

Two changes, in this order.

I modularized the casino. SlotGame.js went from 1,959 lines to 715, with its concerns moved into focused modules — five hooks (useAudio, useAuth, useLeaderboard, useReelAnimation, useResponsibleGaming) and five presentational components (RiskGauge, TutorialModal, ResponsibleGamingModal, ProfileDropdown, WinEffects). TournamentGame.js, which had been a 90% copy-paste of SlotGame.js, dropped from 1,441 lines to 459 by consuming the same hooks. After the refactor, no file in the tree exceeded 234 lines.

That last number was the design target, not a side effect. Not "cleaner code" — a file size the agent could read, hold, and patch without losing its place.

The refactor had one hard constraint: the casino is the real game, the one people actually play, and it had to behave identically afterward. Same UI, same animations, same deploy. So the split moved code without changing behavior, and "the game still plays the same" was the acceptance test. Deduplicating TournamentGame.js onto shared hooks paid for the work on its own — every bug fix had previously needed applying twice.

I took the theming away from the model. Everything mechanical became a Python pass over a copy of the tree. The Builder mirrors the casino frontend into output/<theme_id>/, then runs its passes over the copy in a fixed order: background image, spin button, color palette, font families, UI strings, structured theme content, theme_id injection, admin-surface strip.

Swapping every casino hex for the spec's palette used to be 30 to 40 model patch calls per build. Now it is a function call:

# walks every .css / .js / .html file in the output tree
result = apply_color_palette(self.output_root, self.spec)

The mirror is a denylist, not an allowlist — two admin files are excluded and everything else flows through. When the casino gains a hook or a component, themed games inherit it with no Builder change.

The order of those passes is load-bearing in a way I didn't anticipate. The background and spin-button passes anchor on the casino's original hex values, so they have to run before the color pass rewrites those same hexes. Get the sequence wrong and the anchors silently stop matching. Deterministic code has dependencies too; they're just dependencies you can unit-test.

The result: the model's editable surface is now CSS, and only CSS. Every .js and .jsx write or patch is refused by a guard before it reaches disk. The model's budget is about six patches, spent on the one thing the passes genuinely cannot do — restyling keyframes, glows, and panel shapes so the game's mood matches the theme. That remains the flakiest step in the pipeline, and it is now also the smallest.

What "deterministic" didn't mean

Moving a swap into Python made it reliable. It did not automatically make it correct.

The themed strings come from the spec, and they get substituted into JavaScript source. One theme's tagline was Every Spin's a Touchdown!. A plain str.replace() put it where the casino's original tagline lived:

// casino source
const [message, setMessage] = useState('AWS Fortune Awaits!');

// after a naive swap
const [message, setMessage] = useState('Every Spin's a Touchdown!');

The apostrophe closes the string literal. SyntaxError. The compile gate caught it, but the model couldn't repair it — by then its patch anchors no longer matched the file either.

The fix was to make the substitution aware of the quote context it was landing in: escape for the delimiter that encloses each occurrence, whether that's ', ", or a backtick, and insert verbatim when the match isn't inside a string at all. Obvious in hindsight. The lesson isn't about escaping, though — it's that pulling work out of the model moves the correctness burden onto you. The model would have gotten the quoting right. My str.replace() didn't.

What I observed across all six agents

Each agent taught something specific (posts 3-7 cover those), but patterns emerged across the whole pipeline:

Editing got reliable when I changed the file, not the model. The Designer generating a spec from a conversation: reliable. Image-Gen producing icons from prompts: reliable. The Builder patching a 1,959-line interleaved file with an exact-string-match tool: fragile, and no amount of prompt work fixed it. What fixed it was splitting that file until nothing exceeded 234 lines, and then deleting most of the edits by making them mechanical. Better tooling would have helped too — AST-aware edits, bounded context, validation loops. But the cheapest lever I had was the shape of the target.

Explicit, inspectable handoffs made debugging possible. Every stage's output was a file on disk — a JSON spec, a set of PNGs, a test report. When a build came out wrong, I could open the intermediate files and see exactly where the pipeline diverged. The requirement isn't "files specifically" — it's that each handoff is durable, immutable, and something you can inspect without running anything.

Human gates belong where mistakes are costly. Three approval gates in the pipeline: the spec, the build plan, and the deploy. The spec because a bad spec poisons five stages downstream. The build plan because code modifications are hard to inspect after the fact. The deploy because shipping has external impact and rollback — while possible — carries risk. Every other stage ran without asking, because its output was cheap to redo.

Prompt caching changes the economics. The Builder's 868-call runaway cost $138 at Sonnet's list rates, with Bedrock's prompt caching enabled — each call paid a fraction of the full input cost for the conversation prefix it had already seen. Without caching, the same run would have been far more expensive. Caching made agent loops economically viable for normal runs. It also masked how expensive a runaway was becoming until an hour had passed.

Turn caps are the first safety measure, not the last. Before I understood why the Builder was failing, I added a sixty-turn cap. It didn't make the agent smarter — it made its worst case bounded. The capped runs that hit the limit aborted in minutes instead of hours and cost single digits instead of triple. Bound the loop before you trust the loop.

Work outside the model is work you can test. Every deterministic pass is a pure function over a directory, so every one of them has unit tests — the color swap, the string swaps, the mirror, the quote-escaping, the pass ordering, and the guards that keep the model out of JavaScript. The suite runs in about three seconds and costs nothing, because it never calls a model. That is the under-sold reason to pull work out of a model's reach: not just that deterministic code fails less, but that you can write a test that proves it.

The honest ceiling

The pipeline runs end to end. It produces themed, deployed, playable games from a sentence. That's real.

It's also not a product. Here's what's still true:

• Icon consistency isn't solved. Each image generation call is independent. Thirty icons come back in thirty slightly different styles. Prompt engineering narrows the range; it doesn't eliminate the variance. The games look themed. They don't look art-directed.
• The Tester's navigation is fragile. Vision-as-judge works — a model looking at a screenshot can tell you whether the game looks right. But the model guessing CSS selectors to drive Playwright breaks often enough to produce false negatives. Accessible labels and stable test hooks would fix this. I haven't shipped that.
• The cosmetic CSS pass still wobbles. Even after shrinking the model's job to "optionally adjust shadows and glows," it still hits the turn cap on some themes. The deterministic build is the floor. The model polish is unreliable upward pressure.

I stopped there. The deterministic Builder was proven — it produced a themed, compiling build with no model involvement. The lessons had stopped being about slot machines and started being about agents in general.

What transfers

Five things, distilled from months of building this:

• Restructure the target before you blame the model. If a file is too big for an agent to edit, that is a fact about the file. Split it, then convert what's left from surgery into generation.
• Bound every model loop before you trust it. Unbounded while plus a paid API is a liability, not a feature.
• Make every handoff explicit and inspectable. Files, versioned objects, whatever — as long as you can look at what one stage produced without running the next one.
• Refactor the target for the agent, but never let the agent near the original. Making the casino agent-friendly was a behavior-preserving refactor of a live game. Making it safe was structural: the factory mirrors the casino into an output tree and edits only the copy, so no build can touch the real thing.
• Know when the lessons have outpaced the project. That's the signal to bank them and move on.

What's next

The game factory was a learning project. What it taught me about scoping agent work, managing cost, and designing for inspectability carries into whatever I build next. The factory itself stays where it is — functional, proven for what it does, but not something I'm polishing further.

The next thing is something different: applying what I learned here to a problem that isn't themed slot machines. The agent patterns transfer. The domain won't.

The most reliable agent I built was the one I gradually turned back into a program. The best thing I did for it was to keep taking work away from the model until the only thing left was the thing a model is actually good at.

---

This is the final post in the game-factory series. The earlier posts cover each agent individually: the factory overview, Designer, Image agents, Builder, Tester, Deployer.