Originally published on hexisteme notes.
A few days before this, I'd fixed a bug in the same YouTube upload stage of my pipeline: the local record marking an episode as uploaded was written only after verification succeeded, so a verification failure left the video live on YouTube with nothing on disk that knew about it — and re-running the command would have uploaded it a second time. I wrote about that separately in The Upload Succeeded, the Record Did Not; the fix was to move the checkpoint earlier, writing the video ID to disk the instant it came back — marked verified: false — so a resume would finish the verification instead of re-uploading.
That design was correct. It also got exercised for real, on a live publish run — and the resume failed anyway.
The incident
I was publishing five episodes as public. One of them failed verification. The cause was mundane: right after upload, re-querying the video through videos.list hits eventual consistency — snippet.tags can read back empty for a moment — and it didn't propagate within the retry budget (5 attempts at 5 seconds each, 25 seconds total). This is exactly the situation the earlier fix was built for. The checkpoint was written as designed:
{ "video_id": "fzxvS2S3I7I", "privacy_status": "public", "verified": false }
I re-ran the command to resume. I didn't pass the visibility flag this time — it's a resume, why would I need to:
$ agent-youtube upload --episode EP-...-bolivia-navy
Upload verification failed (downgrade detected): requested privacyStatus='unlisted' but actual is 'public'.
"Downgrade detected" is about the most serious alarm this stage can raise — it means the platform silently overrode what I asked for and made the video more private than intended. Except nothing had happened. The video was exactly public, exactly as requested. The alarm was wrong.
A false alarm, not a real one
The resume path had re-derived what "correct" means from the current moment instead of from the checkpoint. The requested value — public — was sitting right there in upload.json. The code didn't read it. It recomputed the expected value from the current CLI flags and the config.yaml default, which is unlisted. The first run had passed --status public; the resume run hadn't, because a resume isn't supposed to need it. Same video, same code, a different yardstick.
Why this one was hard to catch
First, the symptom doesn't look different from a real alarm. "Downgrade detected" is the heaviest signal this stage can raise, and a false positive breaks two things at once: it blocks a perfectly good resume, and it trains you to shrug off the next real downgrade as "probably that same false alarm again." An alarm that cries wolf is worse than no alarm at all.
Second, it can't reproduce on the first run. When the request and the verification live inside the same process, they read the same variables — there's nothing to diverge. The mismatch only exists on the resume path, and the resume path only runs after something else has already failed. This bug needs a different bug to fire first before it can even show up.
Third, the tests were green. There was a test for exactly this case — if a checkpoint exists, does re-running resume verification instead of re-uploading? That test called the first and second run with the same config. When the config doesn't change, it doesn't matter where the yardstick comes from; the checkpoint and "now" already agree, so the result is identical either way. The test confirmed that resume runs. It never asked what resume uses to judge.
The fix: pin the request, not just the result
On resume, the verification target now gets overwritten with what the checkpoint recorded — privacy_status, title, tags, category_id, made_for_kids, video_language — instead of being re-derived from whatever the CLI flags and config happen to say today.
The regression test had to be built to deliberately disagree with itself: upload as public to force the original failure, then resume with no flags at all — meaning the config default of unlisted is now in play — and it still has to pass. A test that hands the same config to both runs is structurally incapable of catching this, so I pinned the mismatch into the test itself: assert config["default_status"] != "public". I also removed the fix to confirm the test goes red without it.
What generalizes
A checkpoint has to hold not just what happened, but what was asked for.
The earlier fix established that the moment you receive an identifier is the moment you record it. This is the layer on top of that one: a record holding only the identifier can't later tell you whether the thing it points at is correct, because correctness is only ever defined relative to the request. A result-only checkpoint makes resume half-possible — you know what got built, not whether it's what you meant to build.
The moment a resume path goes back to current config to fill that gap, it starts judging a past result by today's rules. Anything that shifts in between — a flag you didn't think to repeat, a config default someone changed, a deploy that landed while the job was sitting there — turns a perfectly fine piece of work into a reported failure. The gap between two runs isn't a precondition your resume logic gets to assume away. It's a variable.
The same shape shows up anywhere a checkpoint exists:
- A payment idempotency key that stores only the key, not the amount or currency, ends up comparing a retry against whatever the cart holds now. If the cart changed in between, the comparison means nothing.
- Infra provisioning that records only the instance ID, not the requested spec, has its resume report "drift" against whatever the current Terraform files say — not against what was actually asked for the first time.
- A deploy retry that never recorded the target version re-reads whatever HEAD happens to be when it wakes back up.
One question covers all of it: of the values this resume path reads, which ones could have changed since the first attempt? Anything that can belongs in the checkpoint, not in "look it up again." And the only way to test for it is to deliberately make the first attempt and the resume disagree on purpose — a test that calls both with identical conditions is structurally blind to this exact defect.
Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.
More notes at hexisteme.github.io/notes.
Top comments (0)