Our label service is a Python package that turns a consignment into a carrier label. Every carrier zone has a JSON template, and the package ships them inside itself, under labels/templates. setup.py listed them with package_data and the pattern templates/*.json, which had been right since the first carrier.
In September we added the Highlands and Islands zones for a carrier that prices them separately, and the engineer put the eleven new templates in a subfolder, templates/zones. The tests for all eleven passed in CI. The pipeline built a wheel, the image installed it, and the first Highlands label in production failed with FileNotFoundError. Ninety two parcels for Inverness and Skye waited on the dock until the afternoon.
The pattern was easy to blame once we looked. templates/*.json matches files directly inside templates and nothing in a folder below it, so the wheel held every old template and none of the new ones. setup.py did exactly what it said.
The real question was why CI had been green. Our tests ran with python -m pytest from the root of the checkout, and that puts the current directory first on sys.path. import labels found the folder in the repository, holding every file anyone had committed, before it could find anything installed. For four years we had tested the source tree. The wheel that production ran had never been imported by a single test.
We moved the code under src/, so the package cannot be imported from the repository root at all and the tests have to use an installed copy. CI now builds the wheel once, installs it into a clean virtual environment, runs the suite against that, and puts the same file into the image. A short step lists every file under templates in the repository and fails the build if any of them is missing from the wheel. The package_data pattern now covers subfolders, but the check is what we trust.
A green run tells you about the code your tests imported. Ours imported the repository, and we shipped a package.
– Sergey Shinder
Top comments (0)