Originally published on hexisteme notes.
I built a YouTube upload stage for a video pipeline, and the flow looked clean enough on paper: start a resumable session, PUT the file, get back a video ID, verify the upload actually landed the way it was supposed to, then write a local record marking the episode as uploaded. Four steps, each one depending on the last. It was the dependency between the last two that turned out to be the problem.
The sequence, and where it breaks
Verification here means re-querying the video through videos.list after the upload finishes, to confirm the visibility wasn't silently demoted, the upload wasn't rejected, and the metadata actually propagated. That's a reasonable thing to check — YouTube's upload API can report success at the transport layer while the platform-side processing does something you didn't ask for. But if that verification call raises, the exception propagates straight up, and the local record — a JSON file I'll call upload.json — never gets written. Not "gets written with an error flag." Never written, period.
By the time that exception fires, though, the video already exists on YouTube. The PUT succeeded. The video ID is real. There's a public (or not-quite-public) video sitting on the channel, and there is exactly nothing on disk that knows about it.
Run the same command again after that, and the guard that's supposed to answer "have I already uploaded this?" — a check for whether upload.json exists — sails right through, because it doesn't exist. The result isn't a retry. It's a second, completely independent upload of the same video.
What "retries don't duplicate" actually meant
The module's docstring said retries don't create duplicate videos. That line wasn't wrong, exactly — it was scoped narrower than it read. It was true for retries inside the low-level file-PUT function, which reuses the same resumable session URI on retry, so transport-layer hiccups during the upload itself are genuinely safe to retry. What it didn't cover was someone re-running the whole command from the outside after a failure. That's a completely different code path, going through the duplicate guard instead of the resumable session, and nothing about the inner retry safety extended to it. One sentence in a docstring, two different meanings of "retry," and only one of them was actually protected.
A second copy of the same bug, already live
While tracking this down, I found a second instance of the identical root cause already sitting in the same repository. Before this upload stage existed, five episodes had already gone up through a different path — an MCP tool — and that path never wrote upload.json either. Which meant the brand-new duplicate guard wasn't just vulnerable to a future verification failure; it was already blind to five specific videos that were live on the channel right now. A single run of the new command, pointed at any of those five, would have found no record, seen no reason to stop, and put a duplicate of each one on the channel. The defect wasn't a bug in the guard's logic. The guard's logic was fine. The defect was that a second path could create the same kind of resource and never tell the first path's bookkeeping about it.
It's worth pausing on how this got past testing at all, because every relevant unit test was green the whole time. Happy path, the missing-file guard, the public-visibility check, the duplicate guard, even a test asserting that verification throws when it detects a demotion — all covered, all passing. What none of them checked was what state got left on disk after that exception fired. Tests tend to ask what a function returns and what it throws. Whether the right thing got thrown is easy to assert. What's still sitting on the filesystem once it has been thrown is a different question, and it's one that residual-state bugs like this one live in specifically because nobody's assertion is pointed at it.
The fix: write before you verify, not after
The fix moved the checkpoint, not the check. As soon as the video ID comes back — before verification runs at all — the code now writes upload.json immediately, with a field marking it verified: false. Verification then runs against that already-recorded video ID. If verification succeeds, the record gets updated to verified: true. If it fails or the process dies, there's already a record on disk saying "this video exists, id X, not yet confirmed."
That record turns the duplicate guard into three branches instead of one: --force means deliberately create another copy, a record with verified: false means resume verification against the existing video ID rather than uploading again, and anything else means block the run outright because the episode is already up. Re-running the command after a verification failure no longer re-uploads anything — it just finishes the check it didn't get to finish the first time.
The five already-live episodes needed a different move, since there was no failed-verification moment to catch retroactively — they'd gone up through a path that never wrote a record at all. For those, I backfilled upload.json entries using the real API values for each video, so the guard had something to check against going forward. Rearming a guard after the fact only works because the actual state (the video, its ID, its visibility) was still recoverable from the platform. If it hadn't been, there'd have been no way to reconstruct which videos were already live without visually checking the channel.
What generalizes
Any operation that creates a resource somewhere remote and then verifies it has a window — the gap between the moment the side effect is committed out there and the moment that fact gets recorded in here. Die inside that window, and two states become indistinguishable from the outside:
- the resource was created, and only the verification step failed
- the resource was never created at all
Retry logic is built entirely on the assumption that you can tell these apart. When you can't, "retry" and "duplicate" become the same button. This isn't specific to video uploads — payment authorization, cloud instance provisioning, creating an account on some third-party service, any file upload with a follow-up confirmation step: different resource, same shape of window, same failure mode if nothing gets written until after the confirmation.
The rule that falls out of it is simple to state and easy to skip in practice: the moment you receive an identifier is the moment you record it, before verification, before any post-processing, before cleanup. The record shouldn't only be capable of saying "done" — it needs a field for "this happened, not yet confirmed," because that's the actual state the system is in for however long the window stays open. And the diagnostic question is just as simple: if this function dies on any given line, does re-running it produce the same result? Ask it specifically for every line between the one where the remote write succeeded and the one where that fact gets written locally. The longer that stretch of code, the more likely something in it eventually dies mid-way.
One more piece, easy to miss because it isn't about the retry path at all: if two different code paths can create the same kind of resource, both of them owe the same bookkeeping. A guard built with only one creation path in mind will only ever know what that one path told it — which is exactly how five already-uploaded videos ended up invisible to a duplicate check that, on its own terms, was working correctly.
A related note on this site covers the same pipeline shipping a different kind of invisible bug: a width guard that measured text the renderer never actually drew, with every test staying green the whole time. Different mechanism, same shape — the tests weren't wrong about what they checked, they just weren't checking the thing that broke.
More notes at hexisteme.github.io/notes.
Top comments (0)