Our label service is written in TypeScript, and most of the team develops it on Macs. CI builds and tests it on Linux runners, inside the same image we ship.
On a Monday in October main went red with one error: Cannot find module './DpdClient'. The pull request that caused it had renamed dpdClient.ts to DpdClient.ts, to match our naming convention, and updated the four imports. It had passed review. Three of us cloned main fresh on our Macs and every build was clean, so we decided the runner was at fault. We cleared the build cache, ran it again, and lost the morning.
The rename was never in the repository. The default filesystem on a Mac, APFS, is case insensitive: it keeps the capital letters you type but treats DpdClient.ts and dpdClient.ts as the same file. When Git clones onto a filesystem like that it sets core.ignorecase to true, and then it looks up the path in its index, dpdClient.ts, finds the file under either spelling, and reports nothing changed. The commit carried the four new imports and the old file name. On a Mac './DpdClient' still opened the file. On Linux it pointed at nothing.
The first fix made it worse. A colleague on Linux added DpdClient.ts as a new file, so the repository held both names. Every Linux build passed, and every Mac checkout now had a file Git showed as modified straight after cloning, because two paths in the index were competing for one file on disk.
We removed the duplicate and renamed the file with git mv through a temporary name, which works on any filesystem. Since then a pre-commit hook compares every tracked path with the name the filesystem actually reports and refuses the commit on a difference in case. CI fails any tree that contains two paths differing only in case. And the typecheck developers run before pushing now runs in the Linux image, not on the Mac it happens to be typed on.
CI was the only machine in the building telling the truth that morning, and we spent three hours trying to make it agree with our laptops.
– Sergey Shinder
Top comments (0)