I came in on a Wednesday to a Slack channel full of people asking why their pull request checks had been queued for fifty minutes. Our runner pool was full. It had been full since about half past ten the night before, and the thing filling it was a single pull request from a colleague who had gone home at six.
We had added an autoformatting job the previous week. It runs the formatter, and if anything changed it commits the result and pushes it back to the branch. It had behaved for five days.
Then somebody opened a branch that contained a file our two formatters disagreed about. One of them wanted a trailing comma in a particular construct and the other removed it. The job ran, changed the file, pushed. The push triggered the workflow. The job ran, changed it back, pushed. There was no fixed point to converge on, so the loop had no reason to stop, and it ran roughly nine hundred times overnight at about two minutes a run.
Nothing alerted, because nothing failed. Every one of those runs was green. Our only signal was the queue depth on everybody else's work the next morning.
The immediate fix was to make the job check rather than fix, failing the build with the diff instead of committing anything. Then the guards. A concurrency group per branch with cancel in progress, so a new push kills the run it supersedes. A condition that skips the workflow entirely when the actor is our own bot account, which would have broken the cycle at the second iteration. And an alert on workflow runs per repository per hour, which is a number none of us had ever looked at.
The part that stayed with me was what the morning would have looked like if we had needed to ship something urgent. Our production deploy pipeline draws from the same runner pool as pull request checks and nightly jobs, first come first served, with no priority of any kind. A colleague's formatting disagreement would have sat in front of a customer facing fix for the better part of an hour, and we would only have found that out while it was happening.
Automation that can change the repository can also start itself, and shared capacity is a dependency of your recovery, not just of your throughput.
– Sergey Shinder
Top comments (0)