A pull request merged on a Thursday with four failing integration tests. Nobody overrode anything. The merge button was green, branch protection was configured and had been for two years, and the tests had genuinely failed. It took an hour to understand that our required check had not been enforcing anything for five weeks.
The chain is dull and that is the point. Someone added an OS matrix to the integration job. GitHub then reports the job as integration-tests (ubuntu-22.04) and integration-tests (macos-14), not integration-tests. Branch protection still required the old bare name. A required check that never reports leaves the pull request permanently in "Expected — waiting for status", so for two days every PR was unmergeable. The person unblocking the team did the obvious thing under pressure: they unticked the check that was stuck. The matrix jobs still ran, still went red when they should, and nothing on earth stopped you merging past them.
So we had a CI system that looked correct from every angle. Workflows ran. Failures were visible in the checks tab. The only missing piece was the link between a failure and the merge being refused, and that link lived in a settings page nobody reads and nothing tests.
The fix was to stop naming individual jobs in branch protection at all. There is now one job called ci-required, it does nothing but needs: every other job in the workflow, and it runs with if: always() so it is not skipped when something upstream fails. Its whole body checks the needs context and exits non-zero unless every dependency reports success. That job, and only that job, is the required check. Add a new test job, change a matrix, rename something: the required name never moves.
We also wrote twenty lines that call the branch protection API in a scheduled workflow and fail if the required-check list is not exactly ["ci-required"]. It has caught one well-meaning change since.
The broader thing I took from it is that the enforcement layer of a pipeline is configuration, not code, and configuration that nothing verifies drifts until it is decorative. If you cannot point at a test that proves a bad PR gets blocked, you have a CI system that reports, not one that gates.
– Sergey Shinder
Top comments (0)