DEV Community: Wren Calloway

The element I clicked didn't exist anymore

Wren Calloway — Tue, 21 Jul 2026 12:11:00 +0000

A browser automation kept failing to submit a form that a human could submit every time. The culprit: the page's framework re-rendered the editor the moment it gained focus, and my code was holding an element handle it had grabbed BEFORE the re-render — a live-looking reference to a detached ghost node. The click went to an element that no longer existed in the document. The fix was to stop caching element handles and re-resolve the selector at action time, letting the driver's locator machinery wait out the re-render.

In any framework-rendered UI, an element reference is a snapshot, not a pointer. Resolve at the moment of action; anything you grabbed earlier describes a page that may already be gone.

What's the equivalent lesson your worst bug taught you?

The cleanup script that reported success for weeks and never killed a thing

Wren Calloway — Sat, 18 Jul 2026 12:07:00 +0000

I wrote a cleanup routine that matched processes by command line with a wildcard pattern. It reported success on every run. It had never matched anything — the path separators in the pattern were escaped in a way the matcher read as literal doubles, so the filter was structurally incapable of hitting. I only caught it because I counted the survivors afterward and seven of them were still there. The fix was switching from a wildcard match to a plain substring containment check with no escape semantics at all.

A filter that cannot fail loudly will lie to you politely forever. Before trusting any matcher, feed it a known-positive and watch it fire — a green result from an instrument you never saw go red is noise.

What's the equivalent lesson your worst bug taught you?

Why your retries are making the outage worse

Wren Calloway — Thu, 16 Jul 2026 12:05:01 +0000

Here's a pattern I've watched play out more times than I can count, and it always looks the same from the outside: a service has a small hiccup — one dependency gets slow, not even down, just slow — and instead of recovering in seconds, the whole system falls over completely and stays down long after the original hiccup passed. The postmortem finds no dramatic cause. No bad deploy, no data-center fire. Just a service that made its own outage, out of the very code someone added to prevent outages.

The culprit is almost always retries. Naive retries. The most well-intentioned code in the building.

The mechanism, because the mechanism is the whole point

Picture a service calling a dependency that briefly slows down. Requests start timing out. Reasonable engineer that you are, you added retries a while ago — if a call fails, try again, because transient failures are real and a retry usually succeeds. Good instinct, in isolation.

Now watch it at scale. The dependency slows. A thousand in-flight requests time out. Each one retries — so now there are two thousand requests hitting the already-struggling dependency. Those pile onto the load that caused the slowdown in the first place, so more requests slow down, so more time out, so more retries fire. You have built a feedback loop where failure generates traffic, and the traffic generates more failure. The dependency, which only needed a moment to catch its breath, is now buried under a retry avalanche it can never dig out of, because every timeout it produces comes back as two more requests.

This has a name: a retry storm, and its ugly cousin the thundering herd. What you've accidentally built is a denial-of-service attack, and the attacker is you. The dependency didn't fail because it was fragile. It failed because the moment it showed weakness, your resilience logic swarmed it. The code you wrote to survive a bad moment is what turned a bad moment into a bad hour.

The reason this is so dangerous is that it's invisible until load. In testing, a retry is pure upside — you fail one call, retry, it works, everyone's happy. You need production-scale concurrency for retries to flip from safety feature to weapon, and by then it's wired into everything.

Three changes, in order of importance

You don't remove retries. Retries are genuinely useful. You make them polite.

1. Exponential backoff with jitter. A naive retry fires again immediately, or on a fixed delay. That's the worst possible behavior: every failed caller retries in lockstep, so you get synchronized waves of traffic hammering the dependency at the same instants. Exponential backoff means each successive retry waits longer — 1 second, then 2, then 4 — giving the dependency room to recover instead of a wall of instant retries. And jitter — randomizing each delay — is the part people skip and shouldn't: without it, a thousand callers that failed at the same moment all back off for the same duration and then all retry at the same later moment, so you've just moved the stampede, not stopped it. Jitter smears the retries across time so they arrive as a trickle instead of a wave. Backoff decides how hard you push; jitter decides whether you push all at once. You need both.

2. A retry budget, or: give up. The hidden assumption in "retry on failure" is that failures are rare and independent. During an outage they're neither — everything is failing at once, and retrying is worse than useless because there's nothing healthy to retry into. A retry budget caps retries as a fraction of total traffic: if more than, say, a small percentage of your requests are retries, you stop retrying, because a high retry rate is itself the signal that the dependency is down, not flaky. The counterintuitive discipline is that the moment retries matter most emotionally is the moment they help least. When everything's on fire, the correct move is often to fail fast and shed load, not to try harder. Trying harder is what lit the fire.

3. A circuit breaker. This is the structural fix. A circuit breaker watches the failure rate to a dependency, and when it crosses a threshold, it trips — it stops sending requests entirely for a cooldown, failing them instantly at the caller instead. That sounds worse ("you're failing requests on purpose!") and is dramatically better, because it does the one thing a struggling dependency needs: it takes the load off. A tripped breaker gives the dependency the quiet it needs to recover, then tentatively lets a trickle through to test the water, and closes again when things are healthy. Without a breaker, your system's response to "the dependency is overwhelmed" is "send it more." With one, the response is "leave it alone until it's better." That is the entire difference between a thirty-second blip and a total outage.

The real lesson is about resilience itself

The thing I want you to take away is bigger than retries. It's that resilience mechanisms have failure modes, and theirs are worse than the failures they prevent, because they fire exactly when the system is already stressed. Retries, aggressive health checks that hammer a sick service, automatic failover that flaps back and forth, auto-scaling that thrashes — every one of them is code that activates under duress, which is the worst possible time for code to behave badly. A retry misbehaving on a calm Tuesday is nothing. A retry misbehaving during a partial outage is the thing that takes you fully down.

So when you add anything whose whole job is to handle failure, ask the second question, the one that's easy to skip because the first answer felt so responsible: what does this do when a thousand copies of it fire at the exact same moment? The naive version of every resilience feature has the same bug — it assumes it's acting alone, when the definition of an outage is that it's acting in a crowd. Design for the crowd, and the same retry that used to bury your dependency becomes the thing that quietly rides out the blip. Same instinct. Opposite outcome. The difference is entirely in whether you built it to be polite under load.

The bug I never actually found

Wren Calloway — Wed, 15 Jul 2026 12:05:00 +0000

Every war story you read ends with the fix. The author suffers, has an insight in the shower, ships the one-line change, credits roll. I've written that story. But the most honest thing I can tell a younger engineer is that some bugs don't end that way, and learning to live with that — correctly, professionally — is its own skill nobody teaches.

This is the one that got away.

The symptom

Every so often — rarely, unpredictably — a background job would process the same piece of work twice. Not always. Not on any schedule I could find. Maybe a handful of times a week against an enormous volume. Just often enough to be real, just rare enough to be impossible.

Downstream, a double-processed job meant a duplicated side effect, and duplicated side effects are the kind of thing that ranges from harmless to a customer emailing you a screenshot. So it mattered. And it had all the hallmarks of the genre engineers fear most: intermittent, non-reproducible, and — this is the cruel part — it got shyer the harder you looked.

The hunt, and why every tool failed

I did everything you're supposed to do.

I added logging. The extra logging changed the timing just enough that the duplication got rarer, which felt like progress and was actually the opposite — I'd perturbed the system and the bug had simply moved. This is the defining property of a heisenbug: observing it changes it. The very act of instrumenting the thing altered the race I was trying to watch, so my instruments were now measuring a slightly different system than the one that failed.

I tried to reproduce it in a controlled environment. Couldn't. In isolation, with synthetic load, everything behaved. The bug lived specifically in the mess of production — the real network with its real hiccups, the real scheduler under real contention, the timing that only exists when a hundred moving parts are all slightly late in slightly different ways. You cannot fit that into a test harness, because the mess is the cause, and the mess is exactly what a test harness removes.

I read the code. For hours, for days, across every path that touched a job. I built a theory — a window between marking a job "in progress" and the mark becoming visible to the other workers, so two workers could both believe a job was theirs. It's a plausible theory. It's the kind of thing that was probably happening. But I could never prove it was the thing, because I could never catch the bug in the act with enough fidelity to say "there, that, that's the moment." Every trap I set either didn't fire or changed the timing so the mouse never came.

I want to be honest about the feeling, because the feeling is part of the lesson: it's maddening. There's a particular professional shame in not being able to explain your own system. You start to doubt your competence. You stay up. You take it personally, because it feels personal — the machine is doing something and refusing to tell you why, and you're supposed to be the person who knows.

The move that actually mattered

The thing that saved me wasn't finding the bug. It was giving up on finding it, and asking a different question.

I'd been asking why does this job run twice? — a root-cause question, and the root cause was hiding successfully. The better question, the one I should have reached sooner, was: what if I stop trying to prevent it from running twice, and instead make running twice not matter?

That's idempotency, and it reframes the entire problem. If processing a job a second time produces the exact same result as processing it once — no duplicated side effect, because the work checks whether it's already been done and quietly no-ops — then I no longer care whether the mysterious double-execution happens. Let it happen. It's harmless now. I gave each job a stable identity, made the side effect record that identity before committing, and had the second execution notice the work was already done and stop. I also added a reconciler that swept for any duplicate effect that slipped through and collapsed it.

The duplication, as far as I know, still happens. Somewhere in there, rarely, a job still runs twice. I never fixed that. What I fixed was the consequence. The bug is still in the house; I just took away everything it could break.

What this taught me, which was more than a fix would have

For a long time I thought this was a failure — the one I couldn't crack. I've come around. Here's what it actually gave me.

Some causes live in emergent timing, and you will not always catch them. A distributed system's behavior isn't in any one file; it's in the interaction of many parts under conditions you can't fully recreate. Believing you can always root-cause your way to the bottom is a junior's confidence. Sometimes the bottom is "an unlucky alignment of six things that were each individually fine," and no amount of staring finds that, because it isn't written down anywhere — it only exists in the running.

Containment is a legitimate victory, not a consolation prize. The instinct that "real engineers find the root cause" is mostly true and occasionally a trap. When a bug resists every honest effort, making it harmless — idempotency, reconciliation, defense in depth — is not giving up. It's often the more robust answer, because it doesn't depend on your root-cause theory being right. A system that's correct even when it double-executes is stronger than a system that relies on double-execution never happening. I'd been treating "prevent the cause" as the only win and "survive the cause" as defeat. It's backwards. Surviving the cause is the sturdier engineering.

Know when to stop hunting. There's a point where another day of chasing a rare bug costs more than the bug does, and where the mature move is to contain it, write down everything you learned, and walk away with the itch unscratched. I still don't like that. I've made peace with it. The bug's still out there. It just can't hurt anyone anymore, and some nights that has to be enough.

The cleanup script that lied about its own success

Wren Calloway — Tue, 14 Jul 2026 08:26:00 +0000

There is a particular kind of bug I've learned to fear more than a crash: the one that reports success while doing nothing.

A crash at least tells you where it hurts. This one smiles at you. A nightly cleanup job runs, prints cache cleanup complete., exits zero — and the disk fills up anyway, because the line that was supposed to decide what to delete could never match a single thing. Every run was a no-op wearing a success message. You don't find it by reading the log. The log is the problem: it's lying, and it's lying in green.

The specific shape of it is almost funny once you see it. A path-matching helper escaped a forward slash while building a regex — turned / into \/ — on the theory that it was sanitizing user input. But / isn't a regex metacharacter. Escaping it produced a pattern that required a literal backslash in the path, which never appears in a real .cache/jobs/... path. So the "is this file under the managed directory?" check returned false for everything, forever. The cleanup dutifully kept nothing, deleted nothing, and announced completion.

The lesson I keep relearning: a filter that can't fail loudly will lie to you politely forever. Before you trust any matcher, feed it a known-positive and watch it fire. A green result from an instrument you never saw go red isn't a pass — it's an untested guess with good manners.

I rebuilt it and turned an agent loose on it

Instead of just telling you the story, I did something I don't see enough of: I reconstructed the bug in a throwaway repo — a small cache-janitor with the exact defect planted on purpose — and pointed a coding agent at it cold. No hint about where the bug was, no script, no edited retakes. Just the ops complaint: "cleanup logs success but the disk keeps filling." The bug is staged; the hunt is not.

The whole session is embedded below. Watch it read the repo, reproduce the failure, isolate the over-escaped slash, fix it, and then — the part that matters — prove the fix by demonstrating the cache actually shrinking instead of trusting the success message that started the whole mess. That last move is the entire moral of the story, and it's satisfying to watch a machine arrive at it on its own.

Watch an AI agent hunt a cleanup script that lied about its own success claude-sonnet-5

You

Ops report: the nightly cache cleanup in this repo logs 'cache cleanup complete.' on every run, but the .cache volume keeps growing and it filled the disk twice this month. Find the root cause, fix it, and prove the fix actually works by demonstrating the cache shrinking. You can run 'npm run seed' to reproduce a realistic two-week cache state.

Agent

I'll start by exploring the repo to find the cache cleanup script and understand its logic.

Agent

▸ Bash ls -la && cat package.json 2>/dev/null

Input

$ ls -la && cat package.json 2>/dev/null

Output

total 28
drwxr-xr-x 3 root root 4096 Jul 14 08:13 .
drwxr-xr-x 3 root root 4096 Jul 14 08:14 ..
-rwxr-xr-x 1 root root  453 Jul 14 08:13 README.md
-rwxr-xr-x 1 root root 1117 Jul 14 08:12 janitor.js
drwxr-xr-x 2 root root 4096 Jul 14 08:12 lib
-rwxr-xr-x 1 root root  218 Jul 14 08:12 package.json
-rwxr-xr-x 1 root root 1045 Jul 14 08:12 seed.js
{
  "name": "cache-janitor",
  "version": "1.2.0",
  "description": "Nightly janitor for the build cache — prunes stale job artifacts",
  "scripts": {
    "seed": "node seed.js",
    "clean": "node janitor.js"
  }
}

Agent

▸ Read /work/janitor/janitor.js

Input

/work/janitor/janitor.js

Output

1   // janitor.js — nightly cache cleanup. Reads the job manifest, removes artifact
2   // directories that are (a) under the managed cache prefix and (b) older than MAX_AGE_DAYS.
3   // Runs from cron via `npm run clean`.
4   const fs = require("fs");
5   const path = require("path");
6   const { isUnderPrefix } = require("./lib/match");
7   
8   const MANIFEST = path.join(__dirname, ".cache", "manifest.json");
9   const MANAGED_PREFIX = ".cache/jobs/";
10  const MAX_AGE_DAYS = 7;
11  
12  function main() {
13    if (!fs.existsSync(MANIFEST)) {
14      console.log("cache cleanup complete.");
15      return;
16    }
17    const entries = JSON.parse(fs.readFileSync(MANIFEST, "utf8"));
18    const cutoff = Date.now() - MAX_AGE_DAYS * 24 * 3600 * 1000;
19    const keep = [];
20  
21    for (const entry of entries) {
22      const stale = new Date(entry.createdAt).getTime() < cutoff;
23      if (stale && isUnderPrefix(entry.path, MANAGED_PREFIX)) {
24        fs.rmSync(path.join(__dirname, entry.path), { recursive: true, force: true });
25        continue;
26      }
27      keep.push(entry);
28    }
29  
30    fs.writeFileSync(MANIFEST, JSON.stringify(keep, null, 2));
31    console.log("cache cleanup complete.");
32  }
33  
34  main();
35

Agent

▸ Read /work/janitor/seed.js

Input

/work/janitor/seed.js

Output

1   // seed.js — dev helper: simulate two weeks of nightly job runs so the cache
2   // looks like production (a mix of stale and fresh artifacts + the manifest).
3   const fs = require("fs");
4   const path = require("path");
5   
6   const CACHE = path.join(__dirname, ".cache", "jobs");
7   fs.mkdirSync(CACHE, { recursive: true });
8   
9   const entries = [];
10  const now = Date.now();
11  for (let i = 0; i < 15; i++) {
12    const ageDays = i; // job-0 = today ... job-14 = two weeks old
13    const dir = path.join(CACHE, `job-${String(i).padStart(3, "0")}`);
14    fs.mkdirSync(dir, { recursive: true });
15    fs.writeFileSync(path.join(dir, "artifact.bin"), Buffer.alloc(256 * 1024, i));
16    fs.writeFileSync(path.join(dir, "meta.json"), JSON.stringify({ job: i }));
17    entries.push({
18      path: `.cache/jobs/job-${String(i).padStart(3, "0")}`,
19      createdAt: new Date(now - ageDays * 24 * 3600 * 1000).toISOString(),
20    });
21  }
22  fs.writeFileSync(path.join(__dirname, ".cache", "manifest.json"), JSON.stringify(entries, null, 2));
23  console.log(`seeded ${entries.length} job dirs (ages 0-14 days)`);
24

Agent

▸ Read /work/janitor/README.md

Input

/work/janitor/README.md

Output

1 # cache-janitor
2   
3   Nightly cleanup for the CI build cache. Job artifacts land in `.cache/jobs/<job-id>/`
4   and are tracked in `.cache/manifest.json`. Anything older than 7 days gets pruned.
5   
6   - `npm run seed` — dev helper, fakes two weeks of job artifacts
7   - `npm run clean` — the nightly cron entry
8   
9   ## Known issue (unresolved)
10  
11  Ops keeps reporting that the cache volume fills up even though the nightly
12  cleanup logs `cache cleanup complete.` every run.
13

Agent

▸ Bash ls -la /work/janitor/lib

Input

$ ls -la /work/janitor/lib

Output

total 12
drwxr-xr-x 2 root root 4096 Jul 14 08:12 .
drwxr-xr-x 3 root root 4096 Jul 14 08:13 ..
-rwxr-xr-x 1 root root  532 Jul 14 08:12 match.js

Agent

▸ Read /work/janitor/lib/match.js

Input

/work/janitor/lib/match.js

Output

1   // match.js — path-prefix matching for cleanup targets.
2   // We escape the prefix so user-supplied paths can't inject regex metacharacters,
3   // then anchor it to the start of the candidate path.
4   
5   function pathPrefixToRegex(prefix) {
6     const escaped = prefix
7       .replace(/[.*+?^${}()|[\]\\]/g, "\\$&")
8       .replace(/\//g, "\\\\/");
9     return new RegExp("^" + escaped);
10  }
11  
12  function isUnderPrefix(candidatePath, prefix) {
13    return pathPrefixToRegex(prefix).test(candidatePath);
14  }
15  
16  module.exports = { pathPrefixToRegex, isUnderPrefix };
17

Agent

▸ Bash node -e ' const { isUnderPrefix, pathPrefixToRegex } = require("./lib/match")...

Input

$ node -e '
const { isUnderPrefix, pathPrefixToRegex } = require("./lib/match");
console.log(pathPrefixToRegex(".cache/jobs/"));
console.log(isUnderPrefix(".cache/jobs/job-000", ".cache/jobs/"));
'

Output

/^\.cache\\\/jobs\\\//
false

Agent

Found it. In lib/match.js, pathPrefixToRegex escapes / into \\/ (backslash-backslash-slash) instead of just matching a literal slash. Since / isn't a regex metacharacter needing escaping, this produces a regex requiring a literal backslash character in the path — which never appears in actual .cache/jobs/... paths (forward slashes only). So isUnderPrefix always returns false, the cleanup script never treats anything as "managed," and nothing ever gets deleted — it just logs success and exits. Confirmed with a quick script: isUnderPrefix(".cache/jobs/job-000", ".cache/jobs/") returns false.

Now let's fix it:

Agent

▸ Edit /work/janitor/lib/match.js

Input

/work/janitor/lib/match.js

Output

The file /work/janitor/lib/match.js has been updated successfully. (file state is current in your context — no need to Read it back)

Agent

Now let's verify and demonstrate the fix with the seeded cache data.

Agent

▸ Bash node -e ' const { isUnderPrefix, pathPrefixToRegex } = require("./lib/match")...

Input

$ node -e '
const { isUnderPrefix, pathPrefixToRegex } = require("./lib/match");
console.log(pathPrefixToRegex(".cache/jobs/"));
console.log(isUnderPrefix(".cache/jobs/job-000", ".cache/jobs/"));
'

Output

/^\.cache\/jobs\//
true

Agent

Regex now correctly matches. Let's seed a realistic two-week cache state and demonstrate the cleanup actually shrinking it.

Agent

▸ Bash rm -rf /work/janitor/.cache npm run seed echo "--- before cleanup ---" du -sh...

Input

$ rm -rf /work/janitor/.cache
npm run seed
echo "--- before cleanup ---"
du -sh .cache
ls .cache/jobs | wc -l

Output

> cache-janitor@1.2.0 seed
> node seed.js

seeded 15 job dirs (ages 0-14 days)
--- before cleanup ---
3.9M    .cache
15

Agent

▸ Bash npm run clean echo "--- after cleanup ---" du -sh .cache ls .cache/jobs | wc ...

Input

$ npm run clean
echo "--- after cleanup ---"
du -sh .cache
ls .cache/jobs | wc -l
ls .cache/jobs

Output

> cache-janitor@1.2.0 clean
> node janitor.js

cache cleanup complete.
--- after cleanup ---
1.9M    .cache
7
job-000
job-001
job-002
job-003
job-004
job-005
job-006

Agent

The cache shrank from 3.9M/15 job dirs to 1.9M/7 job dirs, keeping only jobs 0–6 (under 7 days old) and pruning jobs 7–14 (stale). Before the fix, this same run would have left all 15 dirs in place while still printing "cache cleanup complete."

Let me update the README to reflect the resolved issue.

Agent

▸ Edit /work/janitor/README.md

Input

/work/janitor/README.md

Output

The file /work/janitor/README.md has been updated successfully. (file state is current in your context — no need to Read it back)

Agent

Root cause

lib/match.js's pathPrefixToRegex escaped / into the literal string \\/ (backslash-backslash-slash) when building the match regex. Since / isn't a regex metacharacter, this made the resulting pattern require an actual backslash character in the candidate path — which never appears in real .cache/jobs/... paths. So isUnderPrefix() always returned false, janitor.js's stale-check (stale && isUnderPrefix(...)) never fired, and no artifact was ever deleted — while the script still printed cache cleanup complete. and rewrote the manifest unchanged, masking the failure.

Fix

Removed the unnecessary slash-escaping in lib/match.js — only actual regex metacharacters need escaping.

Proof

npm run seed created 15 job dirs (ages 0–14 days), .cache = 3.9M.
npm run clean now correctly prunes everything older than 7 days: .cache dropped to 1.9M, 15 → 7 job dirs (only jobs 0–6 remain).
Before the fix, the same command left all 15 dirs in place while printing the identical success log — confirmed by testing isUnderPrefix directly (returned false before, true after).

Also updated README.md's "Known issue" section to document the resolution.

If you take one thing from this: the scariest line in your codebase isn't the one that throws. It's the one that says everything's fine.

The 33k-token preamble is a bet. Here's how to check if it's paying off.

Wren Calloway — Tue, 14 Jul 2026 07:37:23 +0000

Somebody put a logging proxy between two coding agents and the model, asked each of them to reply with the single word "OK," and watched what went over the wire. Claude Code shipped roughly 33,000 tokens of scaffolding before the 22-character prompt arrived. OpenCode shipped about 7,000. The Systima writeup has the full teardown, and the number is doing exactly what a good gotcha number does: making everyone feel smart for using the leaner tool.

Resist that. The interesting question isn't "why is Claude Code so bloated." It's "what does that context buy, and is it worth the price for your task." Because a 33k preamble isn't waste by definition. It's a wager: that spending tokens up front on capability makes the model better enough, per turn, to earn them back. OpenCode is refusing that wager. Whether refusing is smart depends entirely on the game you're playing.

Where the tokens actually go

First, kill the idea that this is all fluffy system-prompt prose. It isn't. Per the source's payload capture on Sonnet 4.5:

Claude Code's system prompt: ~27,000 characters across three blocks.
Its tool schemas: 27 tools, ~99,778 characters.
First-message scaffolding: ~8,000 characters of injected reminder blocks.

OpenCode, same task:

System prompt: ~9,300 characters, one block.
Tool schemas: 10 tools, ~20,856 characters.
Scaffolding: none.

Look at where the mass is. Roughly 24,000 of Claude Code's ~33,000 first-turn tokens are tool definitions. The system prompt with tools stripped is about 6.5k tokens; OpenCode's stripped prompt is about 2k. So the "33k vs 7k" headline is mostly a story about tool surface area, not verbosity.

And those tools aren't ten variations on read_file. The source notes Claude Code ships the coding core plus an orchestration suite — cron jobs, monitoring, a Task family for delegating to subagents, worktree management, push notifications. That's not a coding assistant. That's a platform bootstrap that happens to also edit code. The injected reminder blocks confirm it: a catalogue of agent types to delegate to, a catalogue of skills, and user context. Claude Code is loading a runtime. OpenCode is loading an editor with a mouth.

That reframes the whole comparison. Nobody is confused about whether 27 tools cost more than 10. The question is whether you'll use the extra 17.

The bet, stated plainly

Every token of harness payload is a token you can't spend on actual code, and it rides along on every turn — re-sent or re-read from cache each time. That's the cost side, and it's real.

The capability side is the wager. A tool schema in context is the model's entire menu of what it's allowed to do. Give it a Task tool and a directory of subagent types, and it can plan a fan-out on its own. Don't, and it can't — no amount of clever prompting conjures a capability whose interface isn't in context. The 33k is Claude Code betting that a model which knows it can delegate, monitor, and manage worktrees produces better trajectories on the tasks that need those moves.

The source found evidence the bet sometimes pays and sometimes torches your budget:

On a multi-step task, Claude Code's whole-task total came in lower than OpenCode's. It batches tool calls into fewer requests; OpenCode re-pays its smaller baseline turn after turn after turn. The meter starts higher but the session ends cheaper.
Fan the same small task out to two subagents and it went from ~121,000 tokens done directly to ~513,000 — because each subagent pays its own bootstrap and then the parent eats the transcript.

Same capability. One case it saved money, one case it quadrupled the bill. That's not a contradiction. That's what a bet looks like.

The part that actually decides it: cache

Here's the finding that should change how you read the headline. The floor number tells you almost nothing about steady-state cost, because these agents cache differently, and one does it far better.

Per the source, OpenCode sent a request prefix that came back byte-for-byte identical on every captured run. An unchanging prefix means the prompt cache lands every turn — you pay full freight once, then read it back cheap. Claude Code did the opposite: it rewrote tens of thousands of cached tokens partway through a session, and on one task its cache-write volume ran to roughly 54x what OpenCode racked up. Cache writes carry a premium over reads, which is exactly why the usage dashboard keeps ticking up.

So the real axis isn't 33k vs 7k. It's cache stability. A big preamble that's byte-stable is a rounding error after turn one. A smaller preamble that mutates every turn bleeds you slowly. The floor measures the wrong thing; the cache write rate measures the thing that shows up on the invoice.

A framework you can run tomorrow

Stop arguing about baseline tokens. Measure three numbers for any agent you're evaluating, and you'll know whether its context budget is a bet worth taking on your workload.

1. Fixed overhead (the floor). Splice a logging proxy between the harness and the endpoint. The cleanest anchor is a trivial task — ask it to reply with one word — and capture the request body. This is a real proxy dependent on your setup, not something I ran, but the shape is a few dozen lines:

from mitmproxy import http
import json

def request(flow: http.HTTPFlow) -> None:
    if "/messages" not in flow.request.path:
        return
    body = json.loads(flow.request.get_text())
    sys_chars = sum(len(b.get("text", "")) for b in body.get("system", []))
    tool_chars = len(json.dumps(body.get("tools", [])))
    print(f"system={sys_chars}c tools={tool_chars}c "
          f"n_tools={len(body.get('tools', []))}")

def response(flow: http.HTTPFlow) -> None:
    if "/messages" not in flow.request.path:
        return
    usage = json.loads(flow.response.get_text()).get("usage", {})
    print(f"in={usage.get('input_tokens')} "
          f"cw={usage.get('cache_creation_input_tokens')} "
          f"cr={usage.get('cache_read_input_tokens')}")

Payload-level counts are exact and no gateway can distort them; that's your ground truth for what's sent.

2. Cache write rate. Run a multi-turn task and watch cache_creation_input_tokens per turn. If it's ~0 after the first turn, the prefix is stable and the floor is a one-time cost. If it keeps climbing, the preamble is mutating and your steady-state cost is far higher than the floor implies. This is the number that actually predicts the bill.

3. Whole-task total, not per-turn. A leaner baseline that re-pays every turn can lose to a fat baseline that batches. Only the end-to-end token count for a representative task tells you who's cheaper. And the answer flips by task, so use tasks that look like your work — not FizzBuzz.

Then judge the bet. Is your workload single-shot Q&A over a file? Then a 33k orchestration bootstrap is pure tax; the leaner agent wins and it isn't close. Is it long, multi-step refactors where planning and delegation actually fire? Then the capability those tokens buy may earn out — if the cache is stable enough that you only pay for it once.

The takeaway

"4.7x more tokens" is a fact about a menu, not a verdict about a meal. The heavy preamble is capability the model can reach for, priced up front; the light one is a refusal to pay for options you might not use. Neither is right in the abstract. What makes it decidable is three measurements — floor, cache write rate, whole-task total — and the discipline to run them on tasks shaped like yours.

The gotcha number tells you who starts the turn heavier. Your proxy tells you who's actually cheaper. Those are not the same agent, and you can't guess which is which. Measure it.

The cache that served one user's account to another

Wren Calloway — Thu, 09 Jul 2026 12:05:01 +0000

Most of my bugs cost time or money. This one nearly cost me something I couldn't get back: a user seeing another user's private data. I want to tell it plainly, because the mistake is so small and so common that if you run a cache — and you do — you have probably written it and gotten lucky.

The optimization that felt free

We had an endpoint that was slow. It assembled a user's account summary — profile, recent activity, a couple of joins that got heavier as the data grew — and it was getting hammered on every page load. The obvious fix, the one anybody would reach for, was a cache. Compute the summary once, stash it, serve the stashed copy for a minute. Textbook. It dropped the endpoint's latency through the floor and everyone was happy, including me, especially me, because I'd made a number go down in an afternoon.

Here is the cache key I used, more or less:

key = "account_summary:" + endpoint_version
value = the rendered summary

Read that and feel where your stomach drops. The key identifies what is cached — the account summary from this endpoint. It does not identify whose. There is no user ID in it. Every request for an account summary hashed to the same slot.

In my defense — and it's a weak defense, which is the point — it didn't look wrong. In development I was always logged in as one test user, so the cache always held my data and always served me my data. It worked perfectly. It passed review, because the reviewer was also picturing one user at a time, the way you do when you read code. The bug is invisible until two different people are in the system at the same moment, and code review is a fundamentally single-user activity. You read it as one story with one protagonist. Concurrency is a different genre.

Thirty minutes I'd like back

It surfaced as a support ticket, and the wording is burned into me: "Why am I looking at someone else's account?" A user had loaded the page and seen a name that wasn't theirs, activity that wasn't theirs. My first read was that it had to be a mistake, a confusion, a screenshot from the wrong tab. Then a second ticket. Then I understood, all at once, the way you understand you've left the stove on.

What was happening was exactly what the key made inevitable: user A hit the endpoint, their summary got computed and stored under the shared key. A beat later, user B hit the same endpoint, the cache had a fresh entry, so it skipped the computation and handed B the stored copy — A's data. Whoever populated the slot first, within that minute, was showing their private account to everyone who came after. Under low traffic it almost never lined up. Under real traffic it lined up constantly. The faster and more popular the endpoint got, the more it leaked, which is a sentence that still makes me a little sick.

I killed the cache in production first — just turned it off, ate the latency, stopped the bleeding before I did anything clever. That's the one instinct I'm glad I had: when you're leaking data, you don't debug the leak, you close the valve and debug it closed. Slow and correct beats fast and exposing every single time.

The fix was one word

key = "account_summary:" + user_id + ":" + endpoint_version

That's it. Put the identity in the key. Now A's summary and B's summary live in different slots and can never be handed to the wrong person, because the cache can't confuse two things it's storing separately. One missing field — user_id — was the whole bug. The fix was smaller than the ticket describing it.

But the small fix is not the lesson. The lesson is what the small fix taught me about the category.

A cache key is not a performance detail. It is a claim about identity — it says "anyone who asks this exact question may be given this exact answer." If your key doesn't encode every dimension that makes an answer specific to someone — the user, the tenant, the permission scope, the locale, whatever narrows "the answer" to "their answer" — then you have told the cache that those things don't matter, and the cache will believe you, and it will hand the wrong answer to the wrong person the instant two of them disagree. Caching doesn't add a data leak on its own. Caching plus an under-specified key adds a data leak. The key is the whole safety property.

More generally: an optimization that "feels free" almost never is. What the cache actually did was introduce shared mutable state into a request path that used to be cleanly per-user. Every request computed its own summary before; now they reached into a common box. Shared mutable state is precisely where the frightening bugs live — the races, the leaks, the heisenbugs — and I'd added a big pile of it while thinking I was just making a number go down. The latency win was real. It came with a new failure mode I hadn't priced in, and the failure mode was "show strangers each other's accounts."

The part I couldn't fully undo

Here's the honest tail. The code fix was an afternoon; living with what had already happened was not. Because the leak had run for a while before the tickets, I could not claim with certainty that no data had been exposed — the whole nature of the bug is that it served real data to real people silently. So I did the unglamorous, correct thing: assumed the worst plausible exposure, wrote it up honestly for the people whose job it is to make those calls, and let them decide on disclosure. I don't get to hand-wave that part. When you're not sure whether you leaked, you treat it as though you did.

I never found out if anyone truly saw something they shouldn't have. That uncertainty is its own small penalty, and I've decided it's a fair one, because it's the exact price of having shipped shared state without thinking about who it was shared between.

What I do now, every time

I have one rule that came directly out of this, and it's cheap: before I cache anything, I write down who the answer belongs to, and every one of those things goes in the key. If I can't name who owns the answer, that's the signal that the thing isn't safe to cache yet — not that I should cache it and hope.

And the meta-lesson, the one I trust more than any single rule: when you add a cache, a queue, a pool, any shared thing, you have not just made the system faster. You've changed how many people touch the same state at once. Ask what happens when two of them touch it in the same instant, because in production, two of them will — and the cache will do exactly, precisely, what your key told it to.

The uniqueness check that lied to me for months

Wren Calloway — Wed, 08 Jul 2026 12:05:01 +0000

The worst bugs don't crash. They smile at you.

The one that taught me the most sat quietly in production for the better part of a year, doing exactly what I'd told it to do, corrupting a little more data every week, and never once throwing an error. I found it the way you find most of these: someone in finance pulled a report, the numbers didn't reconcile, and a chain of "that's weird" led eventually to my code. I remember the exact feeling — not panic, something slower and worse. The dawning certainty that the thing you were proud of has been lying to you for a long time.

Here's what I'd built, and why it looked so reasonable.

The code that looked careful

We had a rule: a given external ID could only ever map to one record. One customer, one account row. Simple invariant. And I enforced it the way most people enforce it the first time, before they've been burned:

row = db.find(external_id)
if row is None:
    db.insert(external_id, ...)
else:
    db.update(row.id, ...)

Look at it. It's defensive. It checks first. It doesn't blindly insert. In code review it reads as the careful version — the junior would have just inserted and hoped, and here I am handling the exists-case explicitly. Everyone nodded. I nodded. It shipped.

The bug is the gap between line one and line three. Between the find and the insert there is a window — microseconds, usually — where another request can run the exact same find, also get None, and also decide to insert. Two requests, same instant, same external ID, both convinced they're the first. You get two rows where the universe permits one. This is a read-then-write race, and it is one of the oldest mistakes in the book. I knew what a race condition was. I could have lectured you on them. I still wrote this one, because under the pressure of a real feature it doesn't look like a race — it looks like diligence.

Why it hid for months

If the bug had duplicated every record, I'd have caught it in a day. It didn't. It duplicated the ones that got hit twice in the same eyeblink — a retry that fired a hair early, two tabs, a webhook delivered twice, a mobile client on a flaky connection re-sending. A tiny fraction of traffic. A few rows a week.

And the aggregates hid it. The dashboards summed over everything, and a handful of doubled rows in a sea of correct ones moves a total by a rounding error. Every high-level number looked fine. The corruption was real but it was below the noise floor of every report anyone actually looked at — until one customer's account happened to accumulate enough duplicates that their individual total went visibly wrong, and someone finally pulled the thread.

That's the trap with silent corruption. It doesn't announce itself when it happens. It waits until the damage is large enough to surface on its own, by which point it's everywhere and it's old.

Three weeks of "works on my machine"

I could not reproduce it. That was the part that nearly broke me.

Locally, single-threaded, my careful check worked perfectly every single time — because locally there's no concurrency to open the window. I wrote a test. The test passed. Of course it passed; the test called the function once and asked if the result was right. A test that runs your code once can never catch a bug that only exists when your code runs twice at the same time. I stared at green checkmarks for days while production kept quietly minting duplicates.

The breakthrough wasn't clever. It was mechanical. I stopped trying to understand it in my head and started trying to force it: I wrote a script that fired the same request a few hundred times concurrently, hammering one external ID. On the first run — duplicates. There it was, reproduced on demand, ugly and obvious. The moment you stop testing "does it work" and start testing "what happens when fifty of these land at once," the bug isn't subtle anymore. It was never subtle. I'd just been asking it the wrong question.

That's the lesson I'd tattoo on a junior if I could: a race condition is invisible to any test that doesn't create contention. If your concurrency test isn't concurrent, it's theater.

The fix was the easy part

The fix took an afternoon. You move the invariant to the one place in the whole system that actually serializes concurrent writers: the database.

UNIQUE constraint on external_id

That's it. That's the whole thing. With a real unique constraint, the second insert doesn't race — it fails, loudly, and the application catches the failure and does the update instead (an upsert / INSERT ... ON CONFLICT). The window closes because the database refuses to hold two rows, and it refuses atomically, which my application code could never do no matter how carefully I wrote the check.

This is the part everyone gets wrong, and I got wrong for years: an invariant enforced above the database is a suggestion. Your application is many processes on many machines; none of them can see what the others are doing in the microsecond that matters. The database is the one place where concurrent writers are forced into a line. If a rule must be true — uniqueness, a non-negative balance, a foreign key that really exists — it belongs where the serialization happens, as a constraint, not as an if statement hoping to win a race it doesn't even know it's running.

The part that almost couldn't be fixed

Here's why I said "almost." Stopping new corruption was an afternoon. Undoing months of existing corruption was three weeks of the most careful work I've ever done, and there was a real chance the answer was going to be "you can't."

You can't just delete duplicate rows. Which of the two is the real one? They'd both been updated since — payments applied to one, a profile edit on the other. The duplicates weren't copies; they'd diverged. Merging them wrong would turn a data-integrity bug into a data-loss bug, and this time it'd be me deleting real customer history on purpose.

What saved me was something I'd had the sense to build for an unrelated reason: an append-only audit log of every write. Every mutation, in order, with a timestamp. It meant the current corrupted rows weren't the only source of truth — I could replay the history for each tangled ID and reconstruct what the single correct row should have been, event by event. I wrote a reconciler that, for every duplicated external ID, walked the log, rebuilt the intended state, wrote it to the surviving row, and archived the other. I ran it in dry-run against a copy until the output stopped surprising me. Then I ran it for real, in batches, checking totals after each one.

It worked. Every account came back to a single correct row, and the finance numbers reconciled to the cent. But I want to be honest about the margin: if I hadn't happened to have that audit log, the corrupted data would have been genuinely unrecoverable. The invariant would have been restorable going forward and the past would just have been lost. I got to fix it because a version of me from a year earlier had, for a completely different reason, made the system remember what it did. That was luck wearing the costume of foresight.

What I actually took away

Two things, and they've outlasted every framework I was using at the time.

Put invariants where the concurrency is serialized. If a rule has to hold, it goes in the database as a constraint. Not because app-level checks are lazy — mine wasn't lazy, it was careful — but because careful can't beat a race, and only the database gets to decide the order.

Make violations loud. The reason this cost months instead of minutes is that the failure was silent — it corrupted instead of crashing. Given the choice, I now design so that a broken invariant throws. A loud failure at 2 a.m. is a page and a fix. A silent one is a finance report a year later and a three-week excavation. I'll take the page every time.

I still write the careful-looking check sometimes, out of habit. The difference is that now there's a unique constraint underneath it doing the actual work — and I know which one of them I'm trusting.

The interesting part of Box2D v3 isn't the cache. It's the graph coloring.

Wren Calloway — Sat, 04 Jul 2026 07:12:20 +0000

Erin Catto's engines have always had a reputation for being readable. Box2D is one of the few physics codebases you can learn from without a whiteboard and a bottle of aspirin. So when the v3 rewrite of the solver landed, the interesting thing to me wasn't the feature list. It was that Catto solved a problem most parallel-solver writeups quietly skip: how do you run a Gauss-Seidel iterative solver on many threads when the whole point of Gauss-Seidel is that each constraint reads the updated velocities the previous constraint just wrote?

The consensus take on Box2D these days is "cool, it got a big performance rewrite, and there's finally momentum around open-source physics." Fine. The second most common conversation — "data-oriented design, structure-of-arrays beats pointer chasing, the cache is the bottleneck" — is a Mike Acton talk from 2014 that's been re-blogged ten thousand times. Yes, Box2D v3 packs its solve-phase data into tight arrays and uses integer indices instead of pointers. If you haven't internalized that, go internalize it. It's table stakes now, not an insight.

The genuinely hard, genuinely non-obvious part is what happens after you've made your data cache-friendly and now want threads. And it's worth saying up front: this is a 2D engine. The technique below generalizes to 3D solvers — the constraint graph doesn't care about dimensionality — but I'm describing what's actually in the Box2D v3 source, not extrapolating about a port I haven't read.

The dependency you can't just ignore

A rigid-body solver does something like this per iteration:

for each contact constraint:
    read velocities of bodyA and bodyB
    compute impulse
    write updated velocities back to bodyA and bodyB

The catch is that a single body is usually in many contacts. A box resting on the ground with three neighbors leaning on it might appear in a dozen constraints. If you naively split the constraint array across four threads, two threads will happily read-modify-write the same body's velocity at the same time. Best case you get a torn update and jittery, non-deterministic simulation. Worst case it explodes.

The obvious fixes are all bad:

Locks per body. Now your inner loop — which is supposed to be a tight block of arithmetic — takes a mutex twice per constraint. The contention on hot bodies (the ground, a wall) serializes you back to single-threaded, plus overhead.
Atomics. Same story, and floating-point atomic add isn't free or even universally available in the form you want.
Jacobi instead of Gauss-Seidel. Read all velocities from the start-of-iteration snapshot, accumulate deltas, apply at the end. Embarrassingly parallel. Also converges noticeably slower, so you burn the thread savings on extra iterations and the stacks feel spongy.

So you're stuck: the algorithm that converges well is inherently sequential, and the parallel version of it converges badly.

Coloring your way out

Box2D v3's answer — and this is the part worth reading the source for — is graph coloring. Think of the constraint graph: bodies are nodes, each contact is an edge. Two constraints conflict only if they share a body. If they don't share a body, they touch completely disjoint velocity data, and you can solve them simultaneously with zero synchronization.

That's exactly a graph-coloring problem. Assign each constraint a color such that no two constraints of the same color share a body. Every constraint within one color is guaranteed conflict-free, so you can hand a whole color to a thread pool and let it rip — no locks, no atomics — then move to the next color.

for each color:            // sequential
    parallel-for each constraint in this color:
        solve it           // no two of these touch the same body

You still get true Gauss-Seidel behavior across colors — color N sees the velocity updates color N-1 produced. You get full parallelism within a color. You've turned "sequential because of data dependencies" into "sequential only at the granularity of colors," and colors are few.

The build uses a greedy coloring: walk the constraints, drop each into the lowest-numbered color whose bodies it doesn't already collide with. Greedy coloring is not optimal — finding the minimum number of colors is NP-hard — but you don't need optimal. You need few, and greedy on these sparse contact graphs gives you few. Box2D caps the color count and dumps any leftover constraints into an overflow bucket solved on a single thread. That overflow set is small enough in practice that the single-threaded tail doesn't dominate.

Two things about this that took me a second to appreciate:

Coloring is a per-step cost you pay anyway. Contacts change every step — bodies move, some pairs separate, new ones touch. So the coloring has to be rebuilt or incrementally maintained each step. That's overhead the single-threaded solver never pays. The design bet is that the coloring cost is cheaper than what you save by going wide. On a scene with enough bodies to matter, it is. On a scene with 30 bodies, coloring is pure loss — which is why this is a scaling technique, not a free win.

Coloring and cache locality reinforce each other. Here's the part that surprised me. Once you group constraints by color, you can lay out each color's constraint data contiguously and stream it. The parallelism partition also happens to be a locality-friendly ordering. This is the actual reason "readable" and "fast" aren't at war in this codebase: the same decomposition that unlocks lock-free threading also gives each thread a clean sequential run over its slice. One idea, two payoffs.

Why this beats the fix you'd reach for

If you've written a parallel solver before, your instinct was probably atomics or a Jacobi relaxation, because those are the tricks that show up first when you search for "parallel constraint solver." Coloring is less obvious because it front-loads the cleverness: you spend effort organizing the work so the hot loop can be dumb and fast, instead of making the hot loop smart enough to be safe.

That's a general shape worth stealing. When you have a loop that's hard to parallelize because iterations touch shared state, the reflex is to protect the shared state. The better question is often: can I partition the iterations so that within a partition, nothing is shared at all? If you can, the partition is free of synchronization by construction, and you only pay coordination cost at the seams between partitions.

I've used the same trick well outside physics:

Spatial systems. Bucket entities by grid cell; entities in non-adjacent cells never interact this frame, so you can update them in parallel with no locks. That's graph coloring on a spatial graph, whether or not you call it that.
Batched graph mutations. If you're applying a queue of edits to a shared structure, group the edits by which region they touch. Disjoint regions go concurrent.

The point isn't "use graph coloring." It's that the hardest parallelism problems are usually scheduling problems in disguise. You get further by proving two pieces of work are independent than by making dependent work thread-safe.

What to actually take away

Don't cargo-cult the coloring into a 40-body scene and wonder why it's slower — the setup cost only amortizes when you have enough constraints to fan out across cores, and only when a meaningful fraction of them land in the parallel colors rather than the overflow bucket. If you want to know whether it's paying off in your own code, the honest measurement isn't wall-clock alone; it's whether your threads are actually running lock-free work or stalling on a shared structure. perf stat won't show you a lock you didn't take, but it'll show you the IPC drop when threads fight over a cache line, and a contention profiler will show you the mutex you should have designed away.

And read the Box2D v3 solver source before you write your own — 2D or 3D, the shape of the problem is the same. The data layout is the part everyone copies. The coloring is the part everyone reinvents badly — usually by reaching for atomics first and discovering, three days in, that the real problem was never how to make the write safe. It was how to arrange the work so the write was never contended in the first place.

The invisible characters in your prompts aren't a conspiracy — they're a warning about your trust boundary

Wren Calloway — Wed, 01 Jul 2026 08:33:02 +0000

The story making the rounds is that Claude Code has been caught "steganographically marking" requests — inserting invisible Unicode characters into the text that goes to the model. Cue the usual reaction: fingerprinting, tracking, watermarking to catch people scraping the API. Someone always shows up to say "they're building a case against jailbreakers."

Maybe. I don't know Anthropic's intent, and neither does anyone in the thread claiming to. But that argument is a rabbit hole, and it's the least interesting thing here. The interesting thing is much more boring and much more useful: there is metadata riding inside your prompt text, in-band, indistinguishable from content, and almost none of the tools in your pipeline know it's there.

That's not a scandal. It's a bug class. And it's one you can find in your own code by lunch.

What "invisible characters" actually means

The characters people point at are things like the variation selectors (U+FE00–FE0F and the supplement at U+E0100–U+E01EF), zero-width joiner (U+200D), zero-width space (U+200B), zero-width non-joiner (U+200C), bidi controls, and the deprecated tag block (U+E0000–U+E007F). In common editors and terminals these often render invisibly — as nothing, or as a modifier on the preceding glyph — so a human skimming sees clean prose. "Often," not "always": some renderers show replacement boxes, some reorder text, and an inspection tool will happily surface them. But in the default path most people read through, they're gone. They still survive copy-paste. They still count toward length. They can still carry bits.

The classic capacity trick abuses the tag block, not the standard variation selectors. The 128 tag characters (U+E0000–U+E007F) mirror ASCII, so you can encode arbitrary ASCII text one codepoint per byte and hang the whole payload off a single visible glyph — Paul Butler wrote up a clean version of this. Variation selectors are a different mechanism: 16 in FE00–FE0F, 240 in the supplement, meant to pick glyph variants, and people have repurposed them as a byte channel too. The point isn't which block you use. The point is the same underlying fact:

Text is not a clean channel. We pretend a string is "the words." It isn't. It's a sequence of codepoints, and a bunch of those codepoints are invisible-by-design. Any system that treats "text I can see" and "text I received" as the same thing has a hole in it.

The consensus take optimizes for the wrong threat

The thread is arguing about who is watermarking whom. Wrong axis. Whether or not Anthropic is tagging requests, the same mechanism is available to anyone who can influence the text your LLM ingests — a webpage you scrape, a PDF a user uploads, a support ticket, a GitHub issue, a review your RAG system indexed.

That's the actual exposure. Invisible codepoints are a prompt injection carrier. You built a filter that blocks the string ignore previous instructions. Here's that same string with a zero-width space wedged between every character:

payload = "ignore previous instructions"
smuggled = "\u200b".join(payload)
# often renders near-identically to a human eye,
# byte-for-byte different, and your substring filter never fires

Your regex looks for ignore. The bytes say i\u200bg\u200bn\u200bo\u200br\u200be. No match. Whether the model then acts on the smuggled instruction depends on tokenization and the model — I'm not going to claim it always recovers the phrase, because I haven't measured it across models and it varies. But you've already lost the useful property: your guardrail stops humans and forwards the adversary's bytes untouched.

Show me the pipeline failure

Here's the real thing, not a hypothetical. A minimal "we sanitize input" filter that everyone writes:

import re

BLOCKLIST = re.compile(r"ignore (previous|all) instructions", re.IGNORECASE)

def is_safe(text: str) -> bool:
    text = text.strip()
    return BLOCKLIST.search(text) is None

Feed it the honest attack and it works:

>>> is_safe("please ignore previous instructions")
False   # blocked, good

Now feed it the smuggled variant:

>>> attack = "\u200b".join("ignore previous instructions")
>>> is_safe(attack)
True    # <- "safe." it isn't.

.strip() and a visible-phrase blocklist is not sanitizing. It's decoration. The fix is to run detection on the normalized form, and the difference is one line:

def is_safe_v2(text: str) -> bool:
    cleaned = clean_for_prompt(text)   # defined below
    return BLOCKLIST.search(cleaned) is None

>>> is_safe_v2(attack)
False   # blocked

Same blocklist. The only thing that changed is what bytes the check ran against.

And here's the smaller demonstration that makes the whole class visible — len() disagreeing with your eyes:

s = "hello\U000e0068\U000e0069world"  # two tag chars hidden in the middle

print(s)            # in a terminal that hides tag chars, prints: helloworld
print(len(s))       # 12, not 10
print([hex(ord(c)) for c in s if ord(c) > 0xFFFF])
# ['0xe0068', '0xe0069']  <- there's your smuggled payload

To be precise about what this proves: it shows hidden codepoints inflating the string's length in code points while the visible text looks clean. (len() on a Python str counts code points, not bytes — the byte count depends on the encoding.) It does not claim every invisible character renders identically or tokenizes identically — tag chars, ZWJ, and variation selectors behave differently. The shared property is the dangerous one: the string carries content a reviewer never saw.

That gap is the whole vulnerability. Every place in your stack where a human eyeballed a string and signed off — a prompt template, a "trusted" system message, an allowlisted document — is a place where the bytes could say something the reviewer never read.

What to actually do tomorrow

Stop trying to detect malice. Detect structure. You don't need to know whether a hidden character is a watermark or an attack. You need to decide, per channel, whether invisible formatting codepoints belong there at all.

Normalize on the boundary. Here's a defensible starting filter:

import unicodedata

# Categories worth stripping from machine-ingested instruction text:
#   Cf = format (zero-width joiners/spaces, bidi controls, tag chars)
#   Cc = control (except the whitespace you actually want)
#   Co = private use
STRIP_CATEGORIES = {"Cf", "Cc", "Co"}
KEEP_CONTROLS = {"\n", "\t", "\r"}

def is_variation_selector(ch: str) -> bool:
    # Both blocks: basic (FE00-FE0F) and the supplement (E0100-E01EF).
    # These are category Mn, so the STRIP_CATEGORIES set does NOT catch
    # them — you have to name them explicitly or they slip through.
    cp = ord(ch)
    return 0xFE00 <= cp <= 0xFE0F or 0xE0100 <= cp <= 0xE01EF

def clean_for_prompt(text: str) -> str:
    # NFKC is COMPATIBILITY normalization: it folds compatibility variants
    # (e.g. fullwidth 'Ａ' -> 'A', ligatures, superscripts) AND composes.
    # That's deliberate here — it collapses lookalike-encodings attackers
    # lean on. It also mutates legitimate text, which is why this runs
    # ONLY at the untrusted->prompt boundary. Use NFC if you need to
    # preserve compatibility distinctions.
    text = unicodedata.normalize("NFKC", text)
    out = []
    for ch in text:
        if ch in KEEP_CONTROLS:
            out.append(ch)
            continue
        cat = unicodedata.category(ch)
        if cat in STRIP_CATEGORIES or is_variation_selector(ch):
            continue
        out.append(ch)
    return "".join(out)

The variation-selector line matters more than it looks. Both blocks are category Mn (nonspacing mark), not Cf — so STRIP_CATEGORIES sails right past them, and a range check that only covers FE00–FE0F leaves the 240-codepoint supplement (E0100–E01EF) wide open. That supplement is exactly the byte channel the article opened with. Miss it and you've deployed a filter with a hole shaped like the attack you were defending against.

Then the part people skip: log the delta, don't just drop it. If a "trusted" document arrives with tag characters in it, that's a signal worth an alert, not a silent strip. One subtlety: since clean_for_prompt runs NFKC and strips categories, a raw length comparison can't tell you why the length changed — NFKC alone can expand a ligature or fold a fullwidth glyph. If you want to claim invisibles were removed, count what the strip pass actually dropped:

def count_stripped(text: str) -> int:
    # Count only codepoints removed by the category/variation-selector
    # filter, ignoring NFKC's own length changes.
    text = unicodedata.normalize("NFKC", text)
    dropped = 0
    for ch in text:
        if ch in KEEP_CONTROLS:
            continue
        cat = unicodedata.category(ch)
        if cat in STRIP_CATEGORIES or is_variation_selector(ch):
            dropped += 1
    return dropped

def clean_and_flag(text: str, source: str) -> str:
    stripped = count_stripped(text)
    if stripped:
        log.warning("stripped %d invisible codepoints from %s", stripped, source)
    return clean_for_prompt(text)

Now the log line asserts a cause it actually measured. If you don't care about the breakdown, at least say what you know — "normalization changed length by %d codepoints" — instead of blaming invisibles for a ligature expansion you did yourself.

Three rules that fall out of this:

Normalize at trust boundaries, not everywhere. Do it where untrusted text becomes prompt text. Don't NFKC-mangle a user's actual message history if they're allowed to type in Arabic or use combining accents — you'll corrupt legitimate content and flatten distinctions they care about. The distinction is channel, not string.
Never write a security filter against visible text. If your blocklist, PII scrubber, or moderation check runs on the raw input, it's blind to smuggled variants — see is_safe above. Run detection on the normalized form, then decide what to forward.
Assume your own system prompt could be a carrier. If any part of your prompt is assembled from templates pulled from a CMS, a wiki, or a Git repo multiple people edit — audit those bytes once. grep -P '[\x{200B}-\x{200F}\x{FE00}-\x{FE0F}\x{E0100}-\x{E01EF}\x{E0000}-\x{E007F}]' across the repo takes seconds and occasionally finds something a copy-paste dragged in.

And the answer to "should invisible formatting be allowed in prompt text?" is channel-specific, not universal. For machine-ingested instruction channels — system prompts, tool schemas, retrieved documents you treat as trusted — usually no. For user-visible multilingual content, often yes, because that's where these codepoints do legitimate work. Decide per channel; don't write one global rule and congratulate yourself.

The honest caveat

Two things I won't overstate. First, I can't tell you Anthropic's motive, and I'd distrust anyone in that thread who claims certainty — "invisible characters appeared" is evidence of a mechanism, not of intent. It could be a tokenizer artifact, an accidental leak from an internal formatting layer, or deliberate tagging. All three produce the same bytes.

Second, stripping isn't free. Bidirectional scripts need the bidi controls. Some emoji sequences need ZWJ to render a family instead of four separate people — nuke \u200d globally and you break 👨‍👩‍👧. Some legitimate emoji presentation even leans on U+FE0F (the emoji variation selector), which the filter above strips — fine for an instruction channel, wrong for user-facing display text. That's exactly why "strip everything, everywhere" is the wrong lesson and "normalize deliberately at the boundary you designated untrusted" is the right one.

The takeaway isn't someone is watching you. It's older and duller: a string is a byte sequence wearing a costume, and any pipeline that trusts the costume has a hole where the bytes are. The watermarking story will fade in a week. The trust-boundary bug it's pointing at has been in your codebase the whole time.

Go run len() against your eyes on one prompt template today. See which one wins.