Pipeline duration gets treated as a performance metric — nice to improve, competing with other priorities. It is better understood as a hard design constraint, because there is a threshold past which continuous integration stops doing the job it exists to do.
Under roughly ten minutes, a developer opens a pull request, stays in context, and acts on the result. Past twenty, they switch to something else. When they come back, the change is no longer loaded in their head, the fix costs a full re-context, and the tight feedback loop that justified building CI in the first place is gone.
You still have a pipeline. You no longer have continuous integration in any meaningful sense.
Where the minutes actually are
Four things account for most slow pipelines, roughly in order of how much they cost and how rarely they are addressed.
Dependency installation on every run. Still remarkably common. If your pipeline resolves and downloads the full dependency tree on each execution, that is often two to four minutes of pure repetition. Lockfile-keyed caching is a small change with an outsized return.
Serial test execution. Test suites parallelise well and frequently are not parallelised at all, because the suite grew gradually and nobody ever revisited the assumption. Sharding across runners is usually the single largest available win.
Running everything on every change. A one-line documentation edit triggering the full integration suite is waste with a straight face. Test selection based on the dependency graph — running only what the change could plausibly affect — cuts typical run time substantially, with the caveat that the selection logic itself needs to be conservative and tested.
Slow work in the blocking path. Anything genuinely slow — full end-to-end suites, performance benchmarks, container image scanning of large images — does not belong in front of a merge. Move it after merge, run it against a real environment, and page the owning team on failure.
What should actually gate a merge
The instinct is to gate on everything, and it backfires in a specific and predictable way.
Gate on everything and the pipeline becomes slow and flaky. Slow and flaky means engineers learn that red does not necessarily mean broken. Once that is learned, the standard response to a failure is to re-run rather than investigate. At that point the gate is theatre: it delays every change and catches nothing, because failures are dismissed by default.
A defensible policy gates on signals that are fast and deterministic:
- Unit tests
- Linting and formatting
- Type checking
- Dependency and secret scanning
- Contract tests against consumer expectations
Everything slow or inherently flaky runs post-merge against a real environment. This feels like lowering standards and does the opposite — the gate that remains actually means something, so people respond to it.
Flaky tests are the real emergency
Flakiness deserves more urgency than it usually gets, because it is not a nuisance — it is corrosive to the entire mechanism.
One test that fails intermittently teaches the team that failures are sometimes noise. That lesson generalises immediately. Within a few weeks people are re-running failures reflexively rather than reading them, and the day a red build represents a genuine regression, it gets re-run too.
The control that works is automated quarantine. Detect tests that fail non-deterministically across runs on the same commit, automatically move them out of the blocking path, and open an owned ticket. The suite stays green and meaningful while the underlying problem gets fixed properly rather than urgently.
What does not work is asking people to be more diligent about investigating failures. That is a request to spend attention on a signal that has been demonstrated to be unreliable, and it loses to incentives every time.
Deploy and release are different events
The last structural change worth making is separating deployment from release with feature flags.
Once shipping code and activating behaviour are decoupled, deploying stops being a decision that carries risk. Code goes to production dark; the behaviour turns on when someone chooses. Two things follow. Deployment becomes routine enough that continuous deployment is politically achievable in organisations that would otherwise refuse it. And rollback becomes a flag change measured in seconds instead of a redeploy measured in pipeline duration.
The cost is flag debt. Long-lived flags multiply into state combinations nobody can reason about. The control is simple and must be enforced: every flag gets an owner and an expiry date, and a scheduled audit removes anything past it.
A pragmatic order of operations
- Cache dependencies keyed on the lockfile.
- Shard the test suite across runners.
- Move slow and flaky suites out of the blocking path, with quarantine automated.
- Add change-based test selection once the suite is fast enough that correctness of selection is the main risk.
- Introduce feature flags and decouple deploy from release.
Steps one to three usually get a typical pipeline under ten minutes without any architectural change, which is why they come first.
The full treatment — the seven layers of a delivery stack, infrastructure drift, where AI agents genuinely help, costs, and a 90-day roadmap — is at techcirkle.com/blog/devops-automation. We build delivery platforms alongside our custom software development work, which is why the advice here is shaped by systems we have had to operate.
Frequently Asked Questions
Why is ten minutes the threshold for CI feedback?
It approximates how long a developer will wait before switching tasks. Under ten minutes they stay in context and act on the result immediately. Past twenty they move on, and returning to the change costs a full re-context — which is what destroys the tight feedback loop CI exists to provide. The exact number varies by team, but the cliff is real and sits in that range.
Should end-to-end tests block merges?
Generally no. End-to-end suites are slow and inherently more flaky than lower tiers, and putting them in front of a merge is the most common cause of teams learning to re-run until green. Run them post-merge against a real environment with failures paging the owning team. The merge gate should hold only fast, deterministic checks.
How do you detect a flaky test automatically?
Re-run failures against the identical commit. A test that passes on retry with no code change is non-deterministic by definition. CI platforms increasingly support this natively; where they do not, a retry-and-compare job plus a small history store is enough. The important part is that detection triggers automatic quarantine rather than a request for someone to investigate later.
Does test selection risk missing real failures?
Yes, which is why the selection logic must be conservative — err toward running too much — and why it belongs after the basics rather than first. It also pairs naturally with a post-merge full run, so anything selection missed is caught shortly afterward rather than never. Do not adopt it as the first optimisation; adopt it once caching and sharding have already been done.
What is the downside of feature flags?
Flag debt. Flags that are never removed accumulate into combinations of state that nobody can reason about or test, and this compounds quietly. The mitigation is treating every flag as temporary by default: an owner, an expiry date, and a scheduled audit that removes expired ones. The debt is real and manageable; coupling deployment to release is worse.
Top comments (0)