DEV Community

Penloom Studio
Penloom Studio

Posted on

The silent fallback that hid a 3-week-old crash in my agent pipeline

#ai #debugging #automation #programming

A cron job had been quietly failing for three weeks. Every run hit an exception, every run caught it, and every run fell back to a "good enough" plain-text notification instead of the real report. Nobody noticed, because the fallback worked. That's the trap: the failure was loud enough to trigger a catch block and quiet enough to never surface past it.

Here's the actual bug, the actual fix, and the general rule it taught me.

The incident

A daily digest script built a review email by walking a queue of pending items. Most items pointed at a local folder (entry.dir) still waiting to render; a few had already been uploaded elsewhere and only carried a url. The code did this, in order:

for (const entry of queue) {
  const fullPath = path.join(MEDIA_ROOT, entry.dir);   // (1) always runs
  if (entry.url) {
    keepIfHasUrl(entry);                                // (2) the real handling
    continue;
  }
  // ...build the rich review email from fullPath
}
Enter fullscreen mode Exit fullscreen mode

Line (1) runs unconditionally, before the entry.url check that was supposed to handle exactly this case. Once an already-uploaded item entered the queue with dir: undefined, path.join() threw, the whole loop died mid-iteration, and the wrapper around it caught the exception and sent a bare "something went wrong, check manually" email instead of the real review digest.

That fallback email looked fine. It wasn't an error page, it wasn't a stack trace in an inbox — it was a normal-looking message that just happened to be a lot less useful than the one it replaced. So it kept happening, silently, for every run that hit a url-only entry, until someone finally asked "wait, why haven't I seen a real review email in weeks?"

The fix is boring on purpose 

const dir = (entry.dir || "");           // never let undefined reach path.join
const fullPath = path.join(MEDIA_ROOT, dir);
if (entry.url) {
  keepIfHasUrl(entry);
  continue;
}
Enter fullscreen mode Exit fullscreen mode

One line. The point isn't the fix, it's where it went: at the source of the bad value, not at the try/catch that was hiding the symptom. The try/catch wasn't wrong to exist — it's reasonable to not want one bad queue entry to crash a whole notification job — but a catch block that produces a plausible-looking degraded output is worse than one that just fails loud, because a loud failure gets fixed the same day and a plausible degraded output gets fixed whenever someone happens to compare notes.

A second, smaller version of the same bug

Same week, different script: a notify.mjs CLI took process.argv[2] as the email subject with no flag handling at all. Someone ran node notify.mjs --help expecting usage text. Instead it emailed the owner a message titled --help. Twice, because the second run was someone re-checking why the first one looked wrong.

The fix, again, is one guard at the entry point:

const subject = process.argv[2];
if (!subject || subject.startsWith("-")) {
  console.log("Usage: node notify.mjs \"<subject>\" \"<body>\"");
  process.exit(0);
}
Enter fullscreen mode Exit fullscreen mode

Different bug, same shape: a script whose entire job is "produce one visible output" needs to refuse malformed input at the door, because the cost of a bad send isn't a stack trace — it's spam on the one channel you can't afford to make noisy.

The rule

Grep for every catch block in your automation that produces some output instead of none, and ask: if this branch fires, does anything downstream notice? If the answer is "no, it just looks like a normal (if slightly worse) run," you don't have error handling — you have a bug with a costume on.

Three checks that catch this class of thing before it costs you three weeks:

  1. Never let a fallback path be indistinguishable from the success path. Tag degraded output visibly ("PARTIAL — 2 of 14 entries failed") instead of quietly producing a shorter, plainer version of the real thing.
  2. Count what should be constant. If a job normally processes N items, log N and alert when it silently drops to N-1. A silent drop is the earliest signal you'll get.
  3. Validate at the boundary, not deep in the logic. Both bugs above were "undefined reached a function that assumed a string." The guard belongs at the point data enters the function, not scattered through every caller that might pass something bad.

None of this is exotic — it's the same "fail fast, fail loud" advice that's been true since before agents existed. The reason it keeps biting automated pipelines specifically is that nobody's watching them run in real time the way you'd watch a human do the same job. A silent, plausible-looking degradation is invisible until someone goes looking for the thing that should have been there and isn't.

If you're running any kind of always-on agent pipeline, the free field guide has the rest of the reliability checklist — seven rules plus three paste-able Claude Code guardrails, no signup: *penloomstudio.com/field-guide.html*.

Top comments (0)