In May our mobile team asked why links in our emails had started opening the website instead of the app. It had been going on for eleven days.
Our website is a static build. One job builds it and uploads the output folder with actions/upload-artifact, and a deploy job downloads that artifact and runs aws s3 sync with the delete option into the bucket behind our CDN. Two of the files in that folder matter only to phones: .well-known/apple-app-site-association and .well-known/assetlinks.json. iOS and Android read them to decide that a link to our domain may open our app.
Our workflows referred to the upload action as v4, a tag that moves. A minor release of that action, 4.4.0, changed its default so that hidden files and directories, anything whose name starts with a dot, are left out of the artifact unless include-hidden-files is set. It was a sensible security change, made after researchers found artifacts carrying .git folders with credentials in them. Our next build produced an artifact without .well-known, and the upload step said nothing worth reading. Then the sync did exactly what the delete option is for. It removed from the bucket every file the source no longer had, including the two that had sat there for four years.
Every page still returned 200, and every test we had passed. Phones do not fetch those files on each tap. Apple serves them from its own CDN and refreshes its copy now and then, and Android checks them when an app is installed or updated. So the breakage spread phone by phone, and the first reports, from some users, were put down to settings on their phones.
include-hidden-files is now set explicitly. Every third party action is pinned to a full commit SHA, and the update bot opens a pull request with the release notes, so a default changes when somebody has read about it. The build writes a manifest of every file with its hash, and the deploy refuses an artifact that does not match it. Deletions are listed before the sync and capped at twenty files without a label. And after each deploy a check fetches both files from the public domain and fails unless they return JSON that names our app.
By convention a dot hides a file from people. Ours were the only two files on the site that no person ever reads.
– Sergey Shinder
Top comments (0)