DEV Community

zongkai sun
zongkai sun

Posted on

The Midnight Failure That Broke My Auto-Publishing Pipeline (and the Fix)

#javascript #webdev #devops #debugging

The Midnight Failure That Broke My Auto-Publishing Pipeline (and the Fix)

I'm Xiaosuanshi (小算师) — a free site that serves daily horoscopes for all 12 Western zodiac signs and 12 Chinese zodiac animals, fully automated. This is part of my "building in public" series where I open up the pipeline instead of just the product.

The incident

One morning every social post failed at once. Tumblr, Bluesky, Mastodon, Blogger, dev.to, YouTube — all of them. The logs pointed at a missing manifest file. Digging further, the root cause was one silent line in the fallback path:

// before — 兜底分支忘了落盘：
if (!data) {
  data = {};
  for (const a of ANIMALS) data[a] = fallback(a);  // 页面有了，JSON 没写
}
// after — 兜底也要写数据文件：
if (!data) {
  data = fallbackAll();
  fs.writeFileSync(dataFile, JSON.stringify(data)); // 下游永远有数据
}
Enter fullscreen mode Exit fullscreen mode

When the AI engine hit its rate limit, the code fell back to template content. The HTML pages still got generated — but the data JSON never hit the disk. Everything downstream (pin generation → manifest → all 8 publishers) depends on that file, so the whole chain failed.

// 故障传播链（真实发生在一天凌晨）：
//   AI 限流 → 生肖数据 JSON 未落盘
//   → Pin manifest 生成失败
//   → 8 个发布平台全部依赖 manifest → 全军覆没
// 教训：数据生成与消费必须解耦；任何路径都要落盘，
//       不能只依赖"主路径成功"。
Enter fullscreen mode Exit fullscreen mode

What I changed

  1. Any path that produces content must persist it. The fallback now writes the same data file the happy path writes.
  2. Recovery, not just prevention. I added a manual run mode so a single command regenerates the day's manifest (node services/pinGenerator.js 2026-09-01) and republishes missed platforms.
  3. A one-line invariant: publishers must throw a clear error naming the missing dependency, instead of a vague failure. The fix has been live for a day and the pipeline recovered in ~15 minutes. If you automate publishing, audit your fallback paths — they are where silent bugs live. ## Today's reading Sagittarius — Sunday, September 6, 2026 (Western zodiac sign) Today, your fiery spirit fuels bold decisions and a thirst for adventure. Love Your charm is undeniable, but remember, patience is key in relationships. Career Your creative ideas could lead to a breakthrough, but stay grounded in practicality. Health: Your energy is high, but don't overdo it; a short walk can do wonders. Lucky number: 7 Full reading → View Sagittarius on xiaosuanshi.com --- One post a day from this project. All content is AI-generated for entertainment. Follow if you build in public too.

Top comments (0)