DEV Community

haruki
haruki

Posted on

Fixing one silent failure uncovered seven more

Last week I wrote about my YouTube automation that kept "running" for almost six months: 1,376 commits, every run green, zero uploads. An OAuth app left in Testing status meant every refresh token died after 7 days, and nothing turned red.

This is the follow-up. The first channel is posting again. I also built a small watchdog, and pointed it at my second channel, The Obscura Files. That one had been dead for 7 months (last video: April 16).

I expected one bug. I found seven, stacked on top of each other. Each fix let the pipeline run a little further, straight into the next silent failure.

Seven layers, one channel

Every layer below was hidden by the one above it. None of them produced an alert on its own.

# What broke Why nobody saw it
1 config/config.yaml was never committed. Every run failed at startup. It also broke the other channel's thumbnails, quietly.
2 Pillow was missing from requirements.txt. It only existed on branches that were never merged. My local venv had Pillow, so local checks passed. The first channel's thumbnail code caught the exception and wrote a warning to a log file that is only kept when a run fails.
3 The runner ran out of memory. ffmpeg was crossfading ~110 clips in a single call. The runner died so hard that GitHub kept no log at all. Just "lost communication with the server".
4 My fix (crossfade in chunks of 10) broke above 100 clips: the second level wrote to the same file it was reading. The original commit literally said UNVERIFIED. It only failed on the real clip count.
5 The memory cap wrapper (sudo systemd-run) dropped 66 environment variables, including my "skip upload" test flag. The test run was one step away from posting a real video. It was saved only by layer 6.
6 The OAuth token had 2 scopes. The code asked for 3 (youtube.force-ssl, for one comment after each upload). Google answered invalid_scope. This had been true since spring. Earlier layers failed before the code ever got here.
7 The first real upload in 7 months went up as private. The config had said private since April. Someone (me, months ago) used to publish by hand in YouTube Studio. When I stopped, the channel went dark while every run stayed green.

The fixes were small: one config file, one dependency, chunked crossfades, unique file names per level, passing the full environment through, a fresh login with all three scopes, and scheduled publishing via publishAt.

After layer 5, I changed how test runs work. A test run no longer receives the YouTube credentials at all. If the skip flag fails again, the uploader has nothing to log in with.

What the watchdog caught, and what it missed

The watchdog is quiet-pipelines. It does not ask "did the job succeed?". It asks "did a new public video actually appear?", from outside the pipeline, once a day.

On its first real run it opened two GitHub Issues:

  • The Obscura Files: no new output. It stayed open through all seven layers. When the private upload finally went up, it still stayed open, because it only counts public videos. That was the right call.
  • An expiry ledger item. The Claude model my script generator used retires on Nov 30. After I migrated, the next run closed that Issue by itself.

The migration had its own trap. Swapping only the model name would have failed silently: the new model thinks before answering, the old max_tokens=500 was used up by thinking, and the code fell back to a template script. Green run, worse videos.

What it missed: the first channel's thumbnails. Videos were going up, so the output check was happy, but the custom thumbnail step was failing behind a caught exception. The watchdog checks that output exists, not whether it is complete. A reader on my last post described a better check for this: compare the volume of output with the previous run, and fail when it drops by half. A job that exits clean with nothing in it passes every heartbeat.

How good is the AI diagnosis? Honest numbers

When a check fails, the watchdog can send an excerpt of the failed log (secrets masked twice) to Claude and ask for a category: expired auth, upstream change, quota, code bug, or unknown. Low-confidence answers are forced to "unknown" in code.

Set Correct Partly Wrong Unknown
3 real failures from my own pipelines 3 0 0 0
7 real failures from other people's public repos 3 1 0 3

The first row is in-sample: I looked at those cases while writing the prompt. The second row is the fairer measure, and it is biased toward easy cases. I could only score failures whose fixing commit made the right answer obvious, so it has no expired-credential or quota cases at all.

The three "unknown" answers all had the same cause: the deciding evidence was not in the log excerpt. For layer 3 above, there was no log at all. No model can diagnose that, which is why the outcome check matters more than the diagnosis.

What I'd tell myself six months ago

  1. A green run is a claim, not a result. The job is done when the video exists and is public, not when the process exits 0.
  2. Assume there is one more layer. Every fix moved the failure further down the pipeline. I stopped celebrating fixes and started waiting for the first real output.
  3. Swallowed exceptions are debt with no statement. try/except: log.warning(...) into a file that is deleted on success is the same as no log.
  4. Make test runs unable to do harm, not just told not to. A flag can get lost (mine did). Missing credentials cannot.
  5. Check what humans used to do by hand. The last layer was not a bug. It was a manual step that quietly stopped happening.

Where it stands, and a question for you

The Obscura Files uploaded again last night, its first upload in 7 months, and I published it by hand. Tonight's run is the first one with scheduled publishing (15:00 UTC the next day), so nobody has to touch YouTube Studio. I'll add a line here once it has actually gone public, because by now I don't trust a green run until I see the output.

quiet-pipelines is open source (MIT), v0.1.0: github.com/hisatsuneharukiq5-spec/quiet-pipelines. It runs as a GitHub Action with the built-in GITHUB_TOKEN. The AI diagnosis is optional and costs about $0.08 per failure. The README lists what it cannot see, as plainly as what it can.

I'm curious about yours: what was the most recent silent failure in something you run unattended, and how did you finally notice it? Especially if the answer was "a human looked".

Top comments (0)