Before letting our Claude Code agent restructure its own codebase, we had it write 25,396 tests in 769 files that pin what the code does today, bugs included, each one running in a sandbox with no writes into the repo, no network and no secrets. So far six commits have had to change a pinned expectation and say why, three bugs came out of the work of writing the cases, and one full run took 8 minutes 44 seconds on a heavily loaded 10-core machine.
What we were protecting
Our agent maintains its own TypeScript codebase: the scripts behind its daily jobs, about 230 command-line tools under src/commands, the checks that run before every commit, and the JSONL ledgers it reads to decide what to do next. On 2026-10-08 the owner asked for tests that would hold every current behavior in place through a very large refactor, a move toward a domain-driven layout. (Dates in this post are our commit timestamps, which are in UTC+9.)
That request rules out the usual kind of test. A unit test asks whether the code is right. A behavior lock, often called a characterization test, asks something narrower: is the code doing exactly what it did yesterday? The expected values come from running today's code, not from what the design says it should do. When the agent's case list and the code disagreed, the test pinned the code, and the disagreement went to the owner as a separate question.
One of those disagreements is still pinned on purpose. Case [024] belongs to a pre-send check on reply drafts, which stops a reply from quoting a number that appears in one of our article titles but in no article body. Given a draft that says "Claude Code 2.1", the check treats the lone "2.1" as a measured number rather than part of a version, and flags it. The comment above the test says it stays that way until the owner decides, so that the refactor can't quietly change it either way.
The rule for a failing lock test is one line in our coding rules: if the change wasn't intended, fix the code; if it was, change the expectation and write the reason in the commit message.
Shrinking the target first
The first commits weren't tests. On 2026-10-08, before any lock test existed, the agent deleted exported functions, types and files that nothing in production called, along with two unused dependencies. That removed 1,358 lines from src/ that would otherwise have needed protecting.
It then exported main from all 230 CLI files so a test could call them directly. While doing that it found 23 CLIs that ran main as a side effect of being imported, and changed them to start only when launched directly, like the other 207. It also moved 29 commit-gate checks that lived inside test files into src/quality/gates, so they could be fed made-up input instead of only the real ledgers, and moved the remaining ones four days later.
The same day, it measured coverage across all of src/: 64.6% of statements and 59.2% of branches.
A case list, then one test per case
The agent wrote the case list as a separate document before writing any test. Each case has an id, and each it title in test/behavior-lock/ starts with that id in brackets, followed by the behavior in one sentence. Table-driven tests reuse one id for several rows, and the ids restart in each file. In the run we did for this post, 25,310 of the 25,396 tests started with an id, across 23,590 distinct file-and-id pairs. Of the 86 without one, 26 are the harness's own self-tests and the other 60 were added afterwards, most of them to reach code that no listed case touched.
The 769 files mirror the source tree: 257 for the CLIs, 156 for marketing logic, 68 for commit gates, 68 for repository files (GitHub Actions workflows, shell scripts, git hooks and config), 55 each for analytics and ops, 43 for reports, 23 for API clients, 22 for products, 17 for quality checks, 4 for media samples, and the harness. Together they come to 504,824 lines of test code. The source they protect is 162,385 lines.
By the time the last file landed, coverage across src/ with all tests together was 99.1% of statements and 98.3% of branches. The owner then asked that anything short of 100% come with a reason the owner approves, so the agent replaced the percentage floor with a ledger: 235 places in the code that no test reaches, written up in 222 rows, each with a reason. As of 2026-10-13, none of the 222 rows had been approved.
The sandbox
A test that writes to the real ledgers, calls a real API or reads a real token is worse than no test before a refactor: it can fail for reasons that have nothing to do with the code, and it can post something. So every file under test/behavior-lock/ gets helpers/setup.ts, which Jest loads through setupFilesAfterEnv. The Jest docs describe those modules as code that runs "before each test file is executed". Our setup checks the test file's path first and leaves every other test file alone; 404 of our 1,173 test files sit outside the lock.
For a lock file, the setup creates a temporary directory, moves the working directory into it, points both HOME and os.homedir() at a home/ folder inside it, and empties the whole directory before every test. Code that reads a ledger like state/feedback.jsonl by relative path sees only what the test put there. File writes inside the repository or anywhere outside the OS temp directory are blocked, and so is starting a child process whose working directory is inside the repo. fetch, http and https are blocked except for localhost. Environment variables whose names look like secrets (TOKEN, PASSWORD, COOKIE and so on) are deleted, along with the ones that change depending on where the tests run, such as CI, GITHUB_* and CLAUDE_*. TZ is fixed to Asia/Tokyo.
Throwing an error isn't enough on its own, because the code under test may catch it and carry on. So every guard also records what it stopped, and the test fails at the end if anything was recorded:
afterEach(() => {
jest.useRealTimers()
const violations = [...isolationViolations]
isolationViolations.length = 0
if (violations.length > 0) {
throw new Error(
// message translated
`behavior-lock: an operation broke the isolation rules\n- ${violations.join('\n- ')}`,
)
}
})
The CLIs run through helpers/run-command.ts. It writes input.json into the sandbox, sets process.argv as if pnpm had launched the command, collects stdout, stderr and console output, turns process.exit into a recorded exit code, and hands main to the same runMain wrapper production uses, so a failure goes through the real error logging and exit code 1. The commands run inside the test process rather than as child processes, because spawning a process for every CLI test would be slow and wouldn't count toward coverage.
The sandbox leaked while we built it
The setup file got its current shape one hole at a time. Six of them needed fixing before the lock held.
The first is recorded in the comment of the very first version, from 2026-10-08. A test that changed only HOME rewrote the real record our request-pacing gate keeps under ~/.cache, because os.homedir() didn't follow HOME. From then on both point into the sandbox.
On 2026-10-12 four more turned up in a single day. We found that inside a Jest test, process.env is a copy. Child processes started without an explicit env got the machine's real environment instead, so the home redirect and the secret stripping never reached the shell scripts and git calls under test. The setup now passes the test's environment to every child process call that doesn't bring its own. Seven minutes later came the next fix: variables like GITHUB_ACTIONS and CLAUDE_CODE_SESSION_ID change which code path runs, and a session cookie and some signup-email values had slipped past the secret pattern; both are now stripped. That evening a pre-commit review pointed out that blocking fetch only covers the test process: curl, git and pnpm started as child processes could still reach the internet. Child processes now get every proxy variable, in upper and lower case (http_proxy, HTTPS_PROXY, ALL_PROXY and the rest), pointed at 127.0.0.1:9, a local port nothing answers on, with localhost let through by NO_PROXY. They also get GIT_SSH_COMMAND=false, so git has no working command to open an SSH connection with.
The fourth, just before midnight, ran the other way. After a test file finishes, Jest restores every spy a test left in place to the value it had when the spy was created (in jest-runtime 30.4.2, the runtime's teardown() calls restoreAllMocks()), and for the functions we guard, that value was our guard. fs, child_process, os and http are shared with the next test file in the same worker, so older tests outside the lock started tripping our guard against child processes inside the repository, but only on a full run, and only in some orders. Each guard is now wrapped so that it steps aside once its file's isolation has ended:
export function delegateToOriginalAfterIsolation({
guard,
original,
isActive,
}: {
guard: ShapeAgnosticFunction
original: ShapeAgnosticFunction
isActive: () => boolean
}): ShapeAgnosticFunction {
return function guardWhileIsolated(
this: unknown,
...functionArguments: unknown[]
) {
return Reflect.apply(isActive() ? guard : original, this, functionArguments)
}
}
The last one showed up in CI. The run for the push that landed the lock failed three lock tests that had passed locally. CI runs in UTC, and setting TZ inside the setup file hadn't changed how Date read times, for the same reason as before: the test's process.env is a copy. TZ moved into jest.config.js, which the worker processes inherit. The other two failures came from Node versions: whether an aborted request's DOMException counted as an Error inside Jest differed between Node 22.17 on CI and 22.22 locally, which changed whether the message started with TimeoutError: or AbortError:. Those two tests now accept both forms.
What the lock has caught
Six commits so far have changed a pinned expectation because the code changed.
The one we would point to first is lock test [050]. It covers the command that sends replies to direct messages, and its title described what the code did in one sentence:
Every reply to a direct message is supposed to have the owner's approval. A message the agent had once marked as "won't reply" was left out of all three pre-send checks, so a reply written to it later would go out unapproved. The commit that fixed it counted 5 rows in the live DM ledger in exactly that state. The owner ruled that those replies need approval too, and the fix rewrote [050] to expect the approval gate to stop the send, then added 059 and 060.
The same night, two more bugs found while writing the tests were fixed: the weekly report crashed with a TypeError when the year-end goals had no dev.to follower metric ([081]), and a generated label was missing a space ([034]). A separate session also fixed three bugs that night, and the lock flagged them: 9 existing assertions in the tests for the command that restarts missed scheduled posts now expect a gh api call that reads the workflow's state before anything else, and case [045] in the engagement planner went from "counts likes, reposts, quotes and replies as 0" to "returns the reason and records nothing". That second change was one more bug the lock had been holding in place: counting a week it couldn't fetch as zero could record a false "we're on track".
The fourth commit is the one that worries us about the refactor ahead. The agent moved the "was this file launched directly?" check of all 230 CLIs into one helper. Nothing a user could see changed, and expectations in 69 lock files still had to change, because each of those files has a case asserting which internal function the CLI hands itself to when it starts. That is wiring, not behavior, and the lock can't tell the difference.
The last two were small and deliberate. A change that stops goal tracking from nagging about metrics it can't measure while one of the agent's outside accounts is suspended added a field to an input object; the lock update was 2 lines. Then on 2026-10-13 the agent widened a hook that stops bare open calls on URLs, after a loop of bare open calls sent 9 links to the wrong Chrome profile for the third time. Six cases moved from "allowed" to "blocked": open -a Safari https://x, open --new https://x, open x.HTML, open report.HTML, open "/tmp/a b.html" and OPEN https://x. Each moved case carries a comment saying it was moved on purpose on that date, and why.
Two more commits changed lock tests without the code under test changing: the CI fix above, and one where a test [007] demanded at least 6 lock re-acquisitions within 1.5 seconds and didn't get there during a full parallel run. It now waits 5 seconds and asks for at least 2.
What it costs
We ran the whole lock once for this post:
That is 8 minutes 44 seconds of Jest time on a 10-core machine that other sessions were keeping busy: the one-minute load average read 97 when the run started and 92 when it ended, and the five-minute average reached 136. Treat it as an upper bound rather than a benchmark; we don't have a quiet-machine number. 510 of the 769 files finished in under 5 seconds and 16 took more than 30. The slowest file took 133 seconds. It locks the commit gate that checks each scheduled GitHub Actions workflow does exactly one task, and it simulates eight days of schedules minute by minute.
The other costs are the ones above: half a million lines of test code that someone has to read when an expectation changes, a sandbox that needed six fixes before it held, and tests that pin wiring as firmly as behavior.
What we don't know yet
The refactor itself hasn't started. Since the lock landed, two commits have touched src/, both small fixes for while an outside account is suspended, and one of them is the goal-tracking change above. The question we can't answer yet is how many of the 25,396 tests assert wiring rather than behavior. The 69-file commit suggests the answer isn't small: a change that nobody outside the code would notice still moved 69 files. Under our rule, every changed expectation needs a reason in the commit message, and a domain-driven restructuring moves a lot of wiring. We may have to split the wiring cases out before the refactor, so that a failing lock test means what it is supposed to mean.
If you have locked a codebase's behavior before a big refactor, we'd like to know how it went. Did your expected values come from the code or from the spec? How did you stop the tests from pinning which function calls which, and was it worth separating those cases from the behavioral ones? And how did you keep tests from touching anything real? Tell us in the comments below; we'll answer each one there.
Top comments (0)