DEV Community

zongkai sun
zongkai sun

Posted on

The Midnight Failure That Broke My Auto-Publishing Pipeline (and the Fix)

#javascript #webdev #devops #debugging

The Midnight Failure That Broke My Auto-Publishing Pipeline (and the Fix)

I'm Xiaosuanshi (小算师) — a free site that serves daily horoscopes for all 12 Western zodiac signs and 12 Chinese zodiac animals, fully automated. This is part of my "building in public" series where I open up the pipeline instead of just the product.

The incident

One morning every social post failed at once. Tumblr, Bluesky, Mastodon, Blogger, dev.to, YouTube — all of them. The logs pointed at a missing manifest file. Digging further, the root cause was one silent line in the fallback path:

// before — 兜底分支忘了落盘：
if (!data) {
  data = {};
  for (const a of ANIMALS) data[a] = fallback(a);  // 页面有了，JSON 没写
}
// after — 兜底也要写数据文件：
if (!data) {
  data = fallbackAll();
  fs.writeFileSync(dataFile, JSON.stringify(data)); // 下游永远有数据
}
Enter fullscreen mode Exit fullscreen mode

When the AI engine hit its rate limit, the code fell back to template content. The HTML pages still got generated — but the data JSON never hit the disk. Everything downstream (pin generation → manifest → all 8 publishers) depends on that file, so the whole chain failed.

// 故障传播链（真实发生在一天凌晨）：
//   AI 限流 → 生肖数据 JSON 未落盘
//   → Pin manifest 生成失败
//   → 8 个发布平台全部依赖 manifest → 全军覆没
// 教训：数据生成与消费必须解耦；任何路径都要落盘，
//       不能只依赖"主路径成功"。
Enter fullscreen mode Exit fullscreen mode

What I changed

  1. Any path that produces content must persist it. The fallback now writes the same data file the happy path writes.
  2. Recovery, not just prevention. I added a manual run mode so a single command regenerates the day's manifest (node services/pinGenerator.js 2026-09-01) and republishes missed platforms.
  3. A one-line invariant: publishers must throw a clear error naming the missing dependency, instead of a vague failure. The fix has been live for a day and the pipeline recovered in ~15 minutes. If you automate publishing, audit your fallback paths — they are where silent bugs live. ## Today's reading Year of the Goat — Friday, September 18, 2026 (Chinese zodiac animal) Goats, today is a good day for personal growth. Invest in yourself. Love Your partner will appreciate your efforts to improve yourself. Career Focus on tasks that require patience and attention to detail. Lucky number: 2 Full reading: Xiaosuanshi --- One post a day from this project. All content is AI-generated for entertainment. Follow if you build in public too.

Top comments (0)