At 1:36 in the morning, a fix my team had built was merged into our main branch. Nineteen minutes later, the scheduled job it was built to rescue ran — and failed in exactly the old way, because the job does not run from main. It runs from a different checkout, and nobody had put the fix there. Merged is a state of a repository. Shipped is a state of the machine that does the work. Confusing the two is how "fixed" bugs survive the night.
The fix itself was good. One of our production services depends on a token that expires on a schedule, and a patrol job refreshes it and pushes the new value to production. That push kept dying against a platform upload quota — not a code error, a rationing wall: too many file uploads in the window, try again tomorrow. So we built a fallback with a simple rule: if the upload dies specifically of quota exhaustion, do not upload anything. Take the deployment that is already live and known-good, ask the platform to redeploy that exact artifact with the current environment, and verify the result. Tested by hand, it worked end to end: the redeploy came up healthy, the token read as fresh on both production domains, and every core tool the service exposes was present.
Then the code was merged, the report was written, and everyone — including me — believed the patrol was protected. It was not. The merge had landed in our main branch. The patrol, like most scheduled jobs in a small operation, runs from a long-lived working checkout that was sitting on a different branch for unrelated work. That checkout had never heard of the fallback. At 1:55 the patrol pushed, hit the same quota wall, had no fallback to fall back on, and stopped. The fix existed. It just did not exist anywhere that mattered at 1:55 a.m.
The repository is not the runtime
Our industry talks about delivery as if it were a property of version control: the pull request is green, the merge is in, therefore the system has changed. But a running system does not read your repository's default branch. It reads whatever bytes are in the directory it was started from, on the machine it runs on, under the configuration it was given. Every layer between "merged" and "running" — the checkout, the branch, the working tree, the deployed artifact — is a place where a fix can be completely real and completely absent at the same time.
This is the same disease I keep meeting in different subsystems. Last week it was evaluation scores attached to answers that never reached the disk. This week it is a patch attached to a branch the job never reads. In both cases the dashboard layer reported success and the layer that actually does the work had never been touched. The pattern is not carelessness; it is that "done" gets declared at the layer where the work is most visible — the merge, the score table — instead of the layer where the work takes effect.
The completion standard is a readback of the running tree
We changed the rule that night, and it is deliberately unglamorous: a fix to anything that runs on a schedule is complete only when someone reads it back from the exact tree the job executes — not from main, not from the pull request, from the working directory on disk. In our case that meant moving the change into the patrol's actual checkout and then proving it was there: the script counted at its expected length, the fallback section found in place by direct search, the syntax checked. Only after that readback, at 2:25, did the next patrol run actually carry the fallback. That is the moment the bug was fixed. The merge, fifty minutes earlier, was just paperwork.
Two honest limits belong in this story. First, the fallback is not magic: redeploying an existing artifact can refresh configuration, but it cannot carry new code — a release that changes the application still has to wait for the quota window like everyone else, and our tooling refuses to pretend otherwise by stopping, unchanged, on any failure that is not the quota wall. Second, none of this scales by adding a release manager. In a one-person company there is no release manager. There is only a discipline: never let the word "done" attach to the layer you can see, when the layer that matters is the one you can read back.
So here is the test I now apply to every "fixed" report, including my own team's: show me the fix in the tree that runs. If you cannot point at the bytes the machine will actually execute, you have not shipped a fix. You have shipped an intention — and intentions, like unmerged branches, do not run at 1:55 in the morning.
— Jiahui Miao, founder of vertciti, building a personal AI operating system for one.
Top comments (1)
The 'readback of the running tree' rule is the sharpest part here — declaring done at the layer where the work is most visible (the merge, the score table) instead of the layer where it takes effect. I'd extend it one step further: the readback should be automated, not a 2 a.m. manual check, or the next on-call will skip it under pressure. Do you now run a post-merge verification job that diffs the checkout against main on a schedule?