linkbruno.com

Posted on Nov 8 Edited on Nov 8

Postmortem: the deploy deleted last week's JavaScript bundles, and cached pages kept asking for them

#postmortem #frontend #caching #deployment

This is a composite postmortem: the details are illustrative, assembled from a failure pattern that turns up in many front-end deploy pipelines. No real company or figures are described.

Summary

After a routine front-end deploy, a portion of visitors saw blank pages or pages without styles for a period after each release. The cause: the deploy deleted the previous build's hashed JavaScript and CSS files, while HTML pages cached at the CDN and in browsers still referenced them. Those references returned 404s.

The site's link checker reported zero broken links throughout, because it only checked <a href> links, never <script src> or <link href>.

Background

The site is a statically built front end. Each build produces assets with content hashes in their filenames:

/assets/app.3f9c1a.js
/assets/app.3f9c1a.css

The HTML references those exact filenames. Hashing allows the assets to be cached aggressively (long max-age, immutable), because a new build produces new filenames.

The HTML itself was cached at the CDN for a short time, and some pages had a longer cache configured years ago for "performance".

The deploy script did three things:

Upload the new build.
Delete everything in /assets/ not in the new build ("keep the bucket clean").
Purge the CDN cache for HTML.

What went wrong

Step 2 ran before step 3, and step 3 didn't cover everything.

Between the delete and the purge, the CDN was still serving old HTML that pointed to app.3f9c1a.js, which had just been deleted. Those requests went to the origin and got 404.

Worse, a handful of templates had a longer HTML cache lifetime that nobody remembered, and the purge pattern didn't match them. Those pages kept serving old HTML long after the deploy.

And browsers that had the old HTML in their own cache (or a tab left open) would request the old chunks the moment the user navigated within the single-page app, because code-split chunks are loaded lazily. For them, clicking a link in the app produced a blank screen or a "Loading chunk failed" error.

Impact

Pages rendered without styles or without interactivity for affected visitors after each deploy.
The single-page app threw chunk-load errors on navigation for users with open tabs.
Error tracking showed spikes of chunk errors after every deploy, which had been dismissed as "normal deploy noise" for months.

Why nobody noticed

The link checker only looked at anchors. Its config crawled <a href> and ignored resource URLs. A page with a missing stylesheet was, as far as it was concerned, a perfectly healthy page.

The deploy smoke test loaded fresh pages. It requested HTML after the purge, so it always got new HTML with new asset names. It could never see the old-HTML-new-assets mismatch.

The errors were transient. For most users, a refresh fixed it. Each individual report looked like a fluke.

The fix

1. Never delete the previous build's assets during deploy. Keep old hashed assets for a retention window longer than your longest HTML cache, plus the time a user might keep a tab open. Since hashed files are immutable and small relative to storage costs, a retention period of a few releases or several weeks is usually cheap. Cleanup becomes a separate scheduled job that only removes assets not referenced by any of the last N builds.

2. Deploy in the right order. Upload new assets first, then switch HTML, then purge. Old assets stay available the whole time.

3. Audit HTML cache headers. Every template got the same short, documented Cache-Control for HTML. The forgotten long cache was removed.

4. Handle chunk-load failures in the app. If a lazy chunk fails to load, reload the page once to fetch fresh HTML:

window.addEventListener("error", (e) => {
  const isChunk = /Loading (CSS )?chunk \d+ failed/i.test(e.message || "");
  if (isChunk && !sessionStorage.getItem("chunk-reload")) {
    sessionStorage.setItem("chunk-reload", "1");
    location.reload();
  }
});

(Frameworks have their own hooks for this; the idea is the same: one guarded reload, never a loop.)

5. Check resource links, not just page links. The link check now extracts script[src], link[href], img[src] and srcset, and reports any 404 asset as a broken link. A quick standalone version:

curl -s https://example.com/ \
 | grep -oE '(src|href)="/assets/[^"]+"' \
 | cut -d'"' -f2 \
 | while read p; do
     code=$(curl -s -o /dev/null -w "%{http_code}" "https://example.com$p")
     [ "$code" != "200" ] && echo "$code $p"
   done

6. Test the mismatch on purpose. The post-deploy check now fetches HTML from the previous release (saved as an artifact) and verifies that every asset it references still returns 200.

How to test for this before it bites

A simple pre-deploy test catches the whole class of problem: save the HTML of a few key pages from production, deploy to staging, then request every script and stylesheet URL referenced in the saved HTML against staging. Any 404 means old cached pages would break after this deploy.

Lessons

A broken link isn't only an <a>. Scripts, stylesheets, images and fonts are links too, and when they break, the whole page breaks.
Caches mean several versions of your site exist at once. Your deploy has to keep all of them working, not just the newest.
"Deploy noise" is a bug report you haven't read yet. Errors that spike after every release are telling you something.

Crawling for broken links is exactly the problem linkbruno.com works on, and this incident is a good reminder to include resource URLs in whatever checker you use.

DEV Community