I found it while doing something else entirely. Our deploy workflow had a job with an if condition checking the branch name, and somewhere in a refactor eight months earlier that condition had been moved onto the wrong step. The build gate checked for main. The deploy step did not. Anyone who could open a branch in that repository could push a commit, watch the pipeline run, and land code in production.
Nobody had done it. That's the only reason I'm writing about it as a near miss rather than an incident. But sixty-odd people had write access, the repository had contractors in it, and the production deploy credentials were a long-lived key sitting in repository-level secrets, available to every workflow run regardless of what triggered it.
The uncomfortable part is that our pipeline looked strict. There were approvals, there was a protected branch, there were required reviews. All of those protected the merge. None of them protected the deploy, because the deploy was a separate mechanism that only appeared to be downstream of the merge. The visual of a linear pipeline in the UI made it feel like a chain of gates when it was really a set of independent jobs sharing a file.
Three changes fixed it and I'd make all three again. First, production credentials moved into an environment with its own protection rules and required reviewers, so the secret is not readable by a workflow run from an arbitrary ref, no matter what the YAML says. Second, deploys stopped triggering on push at all and now trigger only on a tag created from a merge commit, which makes the ref itself the authorization. Third, we added a test to the pipeline test suite: a job that runs the deploy workflow's guard logic against a fake feature branch and fails if it would proceed.
That third one is the habit I want to keep. Pipeline configuration is code with production access, and we were the only code in the company shipping it with no tests and no review standard beyond "the build went green."
Your branch protection protects your branch. Ask separately what protects your credentials.
– Sergey Shinder
Top comments (0)