My site's deploys silently failed for two weeks. I didn't notice

This one is embarrassing, which is exactly why it's worth writing.

My portfolio is hosted on Vercel. For roughly two weeks, from about September 10 to September 24, every deploy failed, and the live site just kept serving an older version. I kept pushing changes. I kept wondering why nothing showed up. The site looked fine, because the old site looked fine.

Here's the full post-mortem: what broke, why I didn't notice, how I fixed it, and the one rule I took away.

What a "silent" failure looks like

Nothing screamed at me. My local build passed. My git push succeeded. The live URL loaded without errors. The only thing wrong was that none of my newer commits were live, and that's a hard thing to notice when you're the only person looking at the site and you're not comparing versions.

That's the nasty thing about a frozen deploy: the failure mode looks exactly like success until you specifically check.

The root cause

My build runs a prerender step after the main build. It uses Puppeteer to render pages to static HTML so they're quick to load and easy for crawlers to read. On my laptop, Puppeteer downloads a copy of Chrome when you install it.

On Vercel, npm enforces an allow-scripts policy: dependency postinstall scripts are skipped unless explicitly allowed. Puppeteer's Chrome download is a postinstall script. So on Vercel:

1. npm install ran, but skipped the Chrome download.
2. The prerender step tried to launch a browser that didn't exist.
3. It crashed.
4. The crash returned a non-zero exit code.
5. A non-zero exit code from the build command failed the entire deploy.

So a nice-to-have optimization step was able to block the thing that actually matters: shipping.

The fix, in two parts

Part 1: a browser that works on Vercel. On Linux (which is what Vercel runs), the prerender script falls back to @sparticuz/chromium, a build of Chromium made for serverless environments.

Part 2: never let the optimisation break the build. If no browser launches at all, the script now skips prerendering and exits 0. The deploy ships; the pages just aren't pre-rendered.

In pseudo-code the shape is:

// scripts/prerender.mjs (simplified)
async function main() {
  let browser;
  try {
    browser = await launchBrowser();   // tries puppeteer, then @sparticuz/chromium
  } catch (err) {
    console.warn('No browser available, skipping prerender:', err.message);
    process.exit(0);                   // IMPORTANT: do not fail the deploy
  }
  // ...render pages...
}

The rule I wrote at the top of that file afterwards: a post-build nicety must never be able to block a deploy.

Why I didn't notice for two weeks

Being honest about my own process:

• I treated a successful git push as proof the site was updated. It isn't. A push proves the code reached GitHub, nothing more.
• I didn't look at the deployment dashboard, which would have shown a red "Failed" next to every recent commit.
• I had no automated check comparing "what I think is live" with "what is live".

None of these are clever failures. They're habit failures. The bug was small; the blind spot was mine.

What I do now

After a deploy, I verify the live site directly instead of trusting the pipeline. A few quick checks with curl against the production domain:

# 1. Is the chat API alive? (returns ready: true)
curl -s https://www.surajnandan.in/api/chat

# 2. Is the RSS feed being served?
curl -sI https://www.surajnandan.in/rss.xml | head -n 1

# 3. Are generated social images being served?
curl -sI https://www.surajnandan.in/og/blog/<slug>.png | head -n 1

None of these take more than a few seconds, and between them they confirm that the new build is serving, not the old one.

A checklist you can steal

If you deploy anything to a hosted platform, ask these questions:

1. Which of my build steps are essential and which are optional? Optional steps should fail soft.
2. Does my build rely on a postinstall script? Hosted platforms may skip them. Check the install logs.
3. How do I know a deploy actually went out? Have an answer that isn't "the push worked."
4. Do I get notified when a deploy fails? If not, turn it on today. Email or Slack, anything.
5. Does the live site expose a version? Even a build timestamp in the footer lets you spot a frozen deploy at a glance.

The honest lesson

The bug wasn't clever. The failure was that I trusted a green git push as proof of a live site, and I didn't build a habit of checking.

Two weeks is a long time for a site to quietly freeze. It happened to a portfolio, where the cost was a few unseen updates. If the same blind spot existed on something a business depended on, the cost would be much higher. I'd rather learn this lesson on my own site.

Reproduce the failure on your machine

You can simulate the Vercel environment locally. The key is to install dependencies without running postinstall scripts:

rm -rf node_modules
npm ci --ignore-scripts      # skips scripts like Puppeteer's Chrome download
npm run build                # watch the prerender step fail
echo "exit code: $?"

If your build prints a non-zero exit code here, a hosted platform would have failed the deploy for the same reason. Fix it, re-run the same three commands, and confirm exit code: 0.

A rough timeline

When	What I believed	What was true
About Sep 10	"Deploys are working"	Builds had started failing
Sep 10 to 24	"My changes are live"	The live site was frozen on an older build
About Sep 24	n/a	The fallback and fail-soft rule landed, and deploys resumed

The questions I asked afterwards

A post-mortem is only useful if it produces better questions. These are the ones I now ask about any pipeline:

1. What's the cheapest signal that would have told me this broke? A deploy-failed email would have cost nothing.
2. Which steps are essential, which are cosmetic? Mark them in the script.
3. What's different between my machine and the host? Node version, installed browsers, environment variables, install policies, filesystem permissions.
4. If this step fails, what's the safe default? For prerendering, the safe default is "ship without it".

Turn on failure notifications today

Most hosts can notify you on a failed deploy. It takes a minute, and it removes the need to remember to check. Look for deployment notification settings in your project dashboard and route them to email or chat.

A tiny fail-soft pattern for any optional step

# in package.json scripts
"build": "vite build && npm run postbuild || echo 'postbuild failed (non-fatal)'"

The || means the build still exits 0 if the optional step fails, while printing a clear warning. Use it only for steps that are genuinely optional, and make sure the warning is visible in logs.

What this taught me about "green" signals

A green tick answers one narrow question: "did this command succeed?" It does not answer "is the thing I care about true?" The more important the outcome, the more it deserves a direct check of the outcome itself (the live site) instead of an indirect signal (the pipeline).

What's the dumbest reason a deploy has failed for you? Mine is above. Tell me yours in the comments.