I'm Kay, CTO and co-founder at Latchkey. We build managed GitHub Actions runners that repair failures while the job is still running. Before we shipped any of that, we wanted to know what actually breaks CI. Not what people complain about. What the logs say.
So we pulled every failed GitHub Actions run from eight production open source codebases over three months and read them.
156,808 failed runs, across Go, Python, C++, TypeScript and Rust.
The thing that surprised me
Going in, my model of a red build was "someone broke something." Reading a few thousand of these, that model is wrong most of the time.
A package registry 500s. Docker Hub rate limits you. The runner fills its disk on a Gradle cache. Node hits the default heap ceiling and gets SIGKILLed with exit 137. Someone bumps packageManager in package.json and now pnpm isn't on the PATH.
None of those are defects. There is no commit that fixes them. The environment fell over, and the correct response is to fix the environment and run the step again.
And nobody was fixing them. They were being re-run by hand. Someone opens the run, reads a wall of log, decides "yeah, that's flaky," clicks the button, waits eleven minutes. All day, on every team.
That's the actual cost. Not the compute. The interrupt.
The taxonomy
These are the classes we could write a deterministic detector for, which is a much narrower bar than "kinds of failure that exist."
| Class | What it looks like |
|---|---|
| Transient network | Registry timeouts and 5xx across npm, Yarn, pnpm, pip, Go modules, Cargo, Bundler, NuGet. Docker pull rate limits. Git and DNS. API rate limits. Biggest bucket by a distance, and almost all of it is someone else's outage showing up as your red build. |
| Out of memory | Exit 137, kernel OOM kills, plus Node, JVM and Python signatures. Usually not a leak. Usually a default ceiling that stopped being big enough as the codebase grew. |
| Disk full |
no space left on device, inode exhaustion, corrupted caches. Almost always fixed by pruning something rebuildable. |
| Missing tools | Exit 127 and missing shared libraries. A dependency that happens to be on the GitHub-hosted image and is declared nowhere. |
| Package manager drift | pnpm not on PATH, Yarn not matching the pin in package.json. |
| Environment and config | Missing env vars, toolchain mismatches, file handle limits. |
| Flaky tests | Pass on retry with nothing changed. |
| Real defects | Compile errors, type errors, assertions, panics, genuinely failing tests. The build is telling you the truth. |
The hard part is knowing what you're not allowed to fix
This is the whole design problem, and it's where I think most "AI fixes your CI" pitches fall over.
Once you can retry a step, retrying everything is trivially easy and completely disastrous. A tool that reruns a failing test until it goes green isn't a healing tool, it's a mechanism for shipping bugs with a green check on top. Same for --legacy-peer-deps, pip install --no-deps, and || true. The symptom clears. The cause ships.
So the rule we settled on: the system may repair the environment. It may never repair your code, and it may never hide a result.
Compile errors, type errors, assertions, panics and genuine test failures are recognised across a few dozen language and framework signatures, and are never retried and never modified. They fail loudly. That's the point of having a test suite.
The one exception is the flaky-test retry, and it's gated on evidence rather than optimism. A test is only re-run when our own analytics already flagged that exact workflow as one that passes on retry. It runs exactly once. It never edits anything. Without that evidence, a failing test stays red.
There's a related rule I like: if a repeating failure's recent root causes are mostly upstream 5xx and rate limits, we suppress the finding entirely instead of reporting it to you. Someone else's outage is not a problem in your codebase and it shouldn't become your homework.
How it's built
Three layers, cheapest first. Deterministic rules keyed on exit code. Then signature matching against a library of over a hundred known failure shapes. Then, only for failures matching nothing or where a first fix didn't hold, an agent that reads log tails and checked-out files, forms a hypothesis, and picks a fix from the same vetted action set the rules use. It can't run arbitrary commands and can't touch your source.
Most failures never reach layer three. That matters for latency and cost, but mostly for auditability: a deterministic rule can be reviewed, a model's reasoning at 3am cannot.
The rule that caused the most internal argument: when the agent isn't confident, it does nothing and lets the original failure stand. A tool that guesses when it doesn't know is worse than no tool, because then you can't trust the green or the red.
What the corpus gave us
Replaying our detection and repair pipeline against those 156,808 failures:
- 1,300+ runs auto-greened. Transient failures healed and retried in place, zero human touch.
- ~700 fix-PRs opened. Permanent fixes for missing system libraries, setup gaps and too-low timeouts.
- ~$100K per quarter saved, modeled. That one is a model, not a measurement. It assumes a mid-market team on paid runners and prices the engineer interrupt as well as the compute. Take it as a shape, not a number.
Two caveats on the record. This was a retrospective audit against public workflow history, not a live deployment on someone's production org. And the auto-greened count is bounded by what the detectors covered when we ran it, not by what's theoretically fixable. Both should go up.
Where a failure ends up
Four terminal states, and every failure hits exactly one. Healed during the run. Or a pull request with a typed, deterministic edit, labelled verified-at-runtime or proposed. Or handed to your own coding agent over MCP with the root cause, exact failing file and full untruncated logs, secrets stripped. Or explained, with the exact failing job, step and file, which is still a much better morning than a 40,000 line log.
The bit I'd push back on
You should be suspicious of a vendor publishing a study that concludes their product category is necessary. Fair. So here's the version that doesn't sell anything:
Go look at your last month of failed runs. Bucket them by hand into "real defect" and "environment fell over." I'd bet the second pile is bigger than you expect, and that the total human time spent clicking re-run on it is a number your team would be annoyed to see written down.
Whether you fix that with us, with a retry action, or with fifty lines of bash, fix it. The status quo of paying senior engineers to be a retry button is genuinely dumb.
If you want the tooling: Latchkey runs the managed runners this is built into, one line of YAML to switch. We also open-sourced CI Doctor, an MIT-licensed agent skill that does the diagnosis half locally, free, no account.
Top comments (0)