In February we moved the build for our invoicing service from Jenkins to GitHub Actions. The pipeline was rewritten step by step, every test passed, the image deployed, and the first run of monthly invoices went out on the first of March. By lunchtime our Polish and Czech customers were sending us screenshots. Every ł, ś and ř on their invoices was an empty box.
The service renders PDFs with fonts we bundle, because the base image has almost none. The font files live in the repository through Git LFS, so what git stores is a small text pointer and the real file sits on the LFS server. Jenkins had the LFS plugin enabled years ago and fetched the real files on every checkout. The checkout action in GitHub Actions does not fetch LFS content unless you ask it to. Our new pipeline never asked, so it copied four pointer files of about a hundred and thirty bytes each into the image, with .ttf names.
The PDF library tried to load them, failed to parse them, logged a warning and fell back to its built in font, which covers Latin letters without diacritics and nothing else. English and German invoices looked almost right. Our tests ran against a mocked renderer, and the one test that produced a real PDF checked that it was valid and had three pages, which it did.
The image had also shrunk by nine megabytes, and nobody compares that number between builds.
We rebuilt with lfs enabled on the checkout, regenerated the affected invoices and sent them again the same evening. Then the guards. A pipeline step runs git lfs fsck and also searches the workspace for any file that still starts with the LFS pointer header, and fails the build if it finds one. The font loader now refuses to start the service if a bundled font cannot be parsed, instead of falling back. A rendering test produces an invoice containing names in Polish, Czech, Greek and Turkish and compares it with an approved image. And the pipeline fails if the image size changes by more than five percent from the previous release without a label on the pull request saying why.
Moving a pipeline is a translation, and the old system said things we never wrote down. Jenkins had been fetching our fonts for six years, and the only record of it was a checkbox.
– Sergey Shinder
Top comments (0)