A CI Retry Should Be an Experiment, Not a Reflex

Flaky builds create a difficult kind of pressure. The code may be correct, the failure may disappear on rerun, and the delivery pipeline is blocked right now. Adding a retry feels pragmatic.

Sometimes it is. But a retry can also make the pipeline look healthier while making its signal less trustworthy.

A recent .NET CI investigation reminded me of a useful rule: a retry should change the suspected failure domain and produce evidence. If it does neither, it is probably repeating the same experiment and hoping for a different result.

The first mitigation sounded sensible

The intermittent failure appeared in a Razor-heavy build. The source was valid, local builds were stable, and the error signature changed between hosted CI runs.

The initial mitigation did what many of us would try:

1. Run the build.
2. If it fails, stop build processes.
3. Clean intermediate output.
4. Rerun the same command in the same job.

That approach had a plausible hypothesis behind it: perhaps cached compiler state or stale intermediate files were contaminating the build.

The problem was not the idea. The problem was leaving the idea unmeasured.

When we compared the two attempts, they produced the same failure pattern. Cleaning the workspace and restarting build processes had not crossed the boundary where the fault lived. The retry consumed another full build cycle without adding meaningful evidence.

That creates a second risk. Once a team becomes accustomed to calling a failure “the known flake,” a genuine regression can receive the same label. A retry can quietly turn from mitigation into permission to ignore the gate.

Move the retry to the boundary you suspect

The next version changed the structure of the experiment.

First, the build and test sequence was extracted into one shared CI definition. Both attempts had to run byte-identical steps. Without that, a passing retry might only prove that the second path tested less than the first.

Second, the retry moved to a fresh hosted runner and ran only when the first attempt failed. This changed the machine and process boundary instead of merely repeating work inside the same environment. It also kept the happy path free of retry cost.

Third, the final gate evaluated outcomes explicitly. The primary attempt could pass, or a failed primary could be rescued by a successful second attempt. A cancelled or skipped job was not treated as success.

This matters because conditional CI jobs often report “skipped,” and broad aggregation expressions can accidentally make a required check look healthy.

That design made the retry a more useful second opinion. It still did not explain the failure.

Capture evidence at the point of failure

The important improvement was diagnostic capture.

On failure, the job recorded machine characteristics and an inventory of generated source files. The goal was to distinguish between competing explanations:

• Generated files exist but are incomplete, suggesting truncation or a generation-stage defect.
• No generated files exist, suggesting the build is failing earlier.
• Failures correlate with constrained resources, suggesting concurrency or memory pressure.

The first artifact ruled out the original mental model. No generated files existed, and the errors occurred during parsing. The failure happened before the stage we had been blaming.

That did not prove the root cause, but it narrowed the search. The changing error signature, the size of the Razor project, and the constraints of the hosted runner supported a new hypothesis: parallel build activity was increasing resource pressure and destabilising the parse stage.

The build was then serialised, and worker-process reuse was disabled.

Notice the wording: supported a hypothesis. Not “fixed the compiler.” Not “root cause solved.” The diagnostic capture and fresh-runner fallback stayed in place so later runs could confirm or falsify the theory.

The trade-off is throughput for signal quality

Serial builds are not free. On large runners or well-behaved workloads, reducing parallelism can lengthen feedback time. A fresh-runner retry also consumes extra minutes whenever the first attempt fails.

The costs can still be acceptable when they are bounded:

• The normal successful path does not launch another runner.
• Serialisation is scoped to the unstable suite, not the whole pipeline.
• Both attempts use the same build definition.
• Failure artifacts are retained long enough to compare incidents.
• The mitigation has a stated hypothesis and can be removed when the evidence changes.

The objective is not merely a green badge. It is a gate whose meaning the team can explain.

A practical retry checklist

Before adding a retry to CI, I now want clear answers to five questions:

1. What failure domain do we suspect? Process, workspace, cache, machine, network, or external service?
2. What changes between attempts? If nothing relevant changes, the attempts are not independent.
3. Are the attempts equivalent? A shared definition prevents the retry from becoming a weaker test path.
4. What evidence survives failure? Logs alone may not reveal the pipeline stage or resource conditions.
5. What is the exit criterion? Decide when the retry will be removed, narrowed, or promoted into a permanent resilience policy.

Retries are sometimes the right operational tool. They can protect delivery while a rare platform defect is investigated. But they should buy time and information, not erase uncertainty.

The most useful shift for me was simple: stop treating a retry as a button, and start treating it as an experiment.