If your continuous integration spend jumped in the last eighteen months and nobody changed the pipeline, the explanation is probably sitting in your editor. AI assistants made producing code cheap. They did nothing for the systems that verify and ship it, and those systems bill by the minute.
This is a practical post about what breaks, in what order, and what to change first.
The arithmetic
CI cost tracks merge volume, not headcount. If merges doubled, expect build spend to roughly double even with an unchanged configuration. That part is expected.
What is not expected is that per-merge duration also degrades. More merges means more contention for runners, more queueing, and — critically — more encounters with tests that fail intermittently. A test failing one run in twenty is background noise at ten merges a day. At forty it fires twice a day, someone re-runs the pipeline, and you have paid twice for the same verification.
So the bill grows faster than the linear expectation, and the loop gets slower at the same time.
What to fix, in order of return
1. Test selection
Running the full suite on every commit is the single largest recoverable cost in most repositories. Map which tests exercise which modules and run only the affected set on pull requests, keeping the full suite for the main branch and a nightly run.
Most language ecosystems have tooling for this now, and even a crude version — directory-based mapping — captures most of the benefit. The reason it does not get done is that it is a week of unglamorous work, not that it is hard.
2. Verify your cache actually helps
Dependency caching is frequently configured once, never measured, and occasionally slower than a cold build because the restore is fetching a large archive over a slow link. Time a cached run against a cold run before assuming the cache is helping. This takes an hour and surprises people regularly.
3. Right-size the runners
Runner class is typically chosen during a debugging session and inherited forever. Compile-heavy jobs may want more CPU; test jobs are often memory-bound and over-provisioned on cores. Measure a representative job at two or three sizes and pick deliberately.
4. Prune the matrix
Every matrix build multiplies cost. Check which combinations of runtime version and OS your customers actually run and which your support policy actually covers. The intersection is usually smaller than the matrix.
5. Shut down idle environments
Non-production environments running twenty-four hours a day for a team that works eight is a scheduled-shutdown problem, not an architecture problem. It is an afternoon of work.
The problem that is not a cost problem
Once the pipeline is efficient, the remaining bottleneck is usually human: pull request review capacity did not scale with generation capacity.
Measure two numbers separately — time from PR opened to first review, and time from first review to merge. If the first dominates, you have a capacity problem, and no amount of pipeline tuning will fix it. Options that actually work:
- Automatically splitting large changes, since review time scales worse than linearly with diff size
- Defining categories of change that do not require two human reviewers, explicitly rather than by convention
- Using model-assisted review for the mechanical pass — style, obvious null handling, missing test coverage — so humans spend attention on design and correctness
- Setting an explicit queue-length threshold that triggers a team conversation, so the constraint is visible rather than ambient
Dependency drift
Worth a specific note because it is new. Generated code adds dependencies more casually than humans do — asked to parse a date, a model reaches for a library rather than writing four lines. It has no opinion about maintenance status, licence compatibility, or whether the package was published last week by an anonymous account.
Enforce this at merge time rather than in an audit: policy on new dependencies with a clear exception path, a software bill of materials generated on every build, and secrets scanning that runs before the commit lands rather than after.
Measuring whether it worked
Track cost per merge and duration per merge as trends. Track PR cycle time split into the two phases above. Track flake rate honestly — count re-runs, not just failures. And keep the four delivery metrics as the spine: deployment frequency, lead time from merge to production, change failure rate, and time to restore.
The longer version, covering platform engineering, engagement models, and how to vet a vendor for this work, is at DevOps Services Company: How to Choose the Right Partner in the AI Era. If you are building the agent-assisted side of this, our agentic workflow development work covers how to scope those loops safely.
Frequently Asked Questions
How do I find which tests to skip on a pull request?
Start with a directory-based mapping from changed paths to test suites — crude but effective. Language-specific tooling for precise coverage-based selection exists in most ecosystems and is worth adopting once the crude version proves the value.
Is model-assisted code review actually useful?
For the mechanical pass, yes: style violations, missing error handling, absent test coverage, obvious null cases. It is not reliable for design judgement or for spotting the subtle logic error, so treat it as a first filter that reduces what a human must attend to, not as a replacement reviewer.
Should flaky tests be quarantined or fixed?
Quarantine to unblock, then fix on a schedule with an owner. Permanent quarantine is how a suite loses its meaning — after a year of accumulation, nobody trusts a failure and the suite stops functioning as a gate.
Does self-hosting runners reduce cost?
Sometimes, but it converts a variable cost into a maintenance obligation. Exhaust configuration wins first; they are cheaper, reversible, and do not add a system someone has to own.
How much CI spend is typically recoverable?
Thirty to fifty percent on a neglected pipeline, mostly from test selection and caching. Ten to fifteen on a well-maintained one. Measure your own baseline before believing any number, including this one.
What is a reasonable pipeline duration target?
Under ten minutes for the feedback a developer actively waits on. Beyond that, people context-switch away and return to failures they have to reconstruct, which degrades change size and recovery time in ways the invoice never shows.
Top comments (0)