DEV Community

Sergey Shinder
Sergey Shinder

Posted on

Our portal releases went out without the new index.html whenever it kept the same size

Our depot portal is a single page app served from an S3 bucket. A build job produces the static files, and a deploy job puts them in the bucket with aws s3 sync. The JavaScript bundles have a content hash in their file names, so a new build gets new bundle files and a new index.html that points at them. Old bundles stay in the bucket for a few releases, so pages already open keep working.

In September we made our build artifacts reproducible. The build job now packs the output with GNU tar using --sort=name, fixed owners and --mtime set to the first of January 1980, so the same commit always gives a tarball with the same checksum. The deploy job extracts it, and tar restores the stored modification times, as it does by default.

Three weeks later a fix for label printing at one depot was deployed twice and was live neither time. The bucket held the new bundle files and the old index.html.

aws s3 sync uploads a local file when the object is missing, when the sizes differ, or when the local file was modified more recently than the object. Every file we deployed now claimed to be from 1980, so the date never won, and only size was left. New bundles had new names and always went up. index.html differs between releases only in the hashes of the bundle names, and the hashes are always the same length, so in most releases it had exactly the same size as the last one and was skipped. Of the nine releases since September, five had changed nothing a user could see. The pipeline had reported every one as deployed.

The tarball is still reproducible. The deploy job extracts it with --touch, so the files carry the time they were unpacked, and index.html and the other files without a hash in their names are copied with aws s3 cp on every deploy whatever sync thinks. The last step fetches index.html from the public address and compares it with the one in the artifact. It has failed once since, on a CDN setting, which is the point of it.

We made the build output identical every time, and the deploy tool decided identical times meant nothing had changed.

– Sergey Shinder

Top comments (0)