For three months, every deploy of our customer portal was followed by about an hour of errors in our error tracker reading Failed to fetch dynamically imported module, and by a few calls to support saying the pay button did nothing. The deploys themselves were clean, and anyone who opened the portal after one saw no problem at all.
The portal is a single page application built with Vite. Each route is a separate chunk, loaded when a customer first visits it, and every chunk's file name includes a hash of its contents, so each build has its own names. The deploy step copied the new build into the S3 bucket behind CloudFront and then removed every file that was not part of the new build, to keep the bucket tidy.
A customer who had the portal open during a deploy was still running the previous build in their browser. When they went to pay, their copy asked for the payment chunk under its old name, and that file no longer existed. It got worse than a 404. Our CloudFront distribution was set to answer 403 and 404 errors with index.html and status 200, the usual way of making deep links work in a single page application, so the browser asked for a JavaScript module, received an HTML page, refused it on its MIME type and left the button doing nothing.
We confirmed it in staging by opening the portal, deploying, and clicking pay.
Built assets are now treated as immutable. The deploy uploads new files and removes nothing, and a weekly job deletes assets only once they have been out of every build for thirty days. index.html is served with no-cache and the assets with a year long immutable cache, so a new visit always gets the new build and an old tab can still load the old one. The error response that served index.html is gone, replaced by a CloudFront Function that rewrites only route paths, so a missing asset is now an honest 404. And the router catches a failed chunk load and reloads the page once at the same URL, for anything older than thirty days.
We thought a release replaced the old version. It only replaced the files on our side, and the old version stayed in production for as long as anyone kept a tab open.
– Sergey Shinder
Top comments (0)