DEV Community: Ivan Rossouw

A CI Retry Should Be an Experiment, Not a Reflex

Ivan Rossouw — Tue, 21 Jul 2026 05:37:10 +0000

Flaky builds create a difficult kind of pressure. The code may be correct, the failure may disappear on rerun, and the delivery pipeline is blocked right now. Adding a retry feels pragmatic.

Sometimes it is. But a retry can also make the pipeline look healthier while making its signal less trustworthy.

A recent .NET CI investigation reminded me of a useful rule: a retry should change the suspected failure domain and produce evidence. If it does neither, it is probably repeating the same experiment and hoping for a different result.

The first mitigation sounded sensible

The intermittent failure appeared in a Razor-heavy build. The source was valid, local builds were stable, and the error signature changed between hosted CI runs.

The initial mitigation did what many of us would try:

Run the build.
If it fails, stop build processes.
Clean intermediate output.
Rerun the same command in the same job.

That approach had a plausible hypothesis behind it: perhaps cached compiler state or stale intermediate files were contaminating the build.

The problem was not the idea. The problem was leaving the idea unmeasured.

When we compared the two attempts, they produced the same failure pattern. Cleaning the workspace and restarting build processes had not crossed the boundary where the fault lived. The retry consumed another full build cycle without adding meaningful evidence.

That creates a second risk. Once a team becomes accustomed to calling a failure “the known flake,” a genuine regression can receive the same label. A retry can quietly turn from mitigation into permission to ignore the gate.

Move the retry to the boundary you suspect

The next version changed the structure of the experiment.

First, the build and test sequence was extracted into one shared CI definition. Both attempts had to run byte-identical steps. Without that, a passing retry might only prove that the second path tested less than the first.

Second, the retry moved to a fresh hosted runner and ran only when the first attempt failed. This changed the machine and process boundary instead of merely repeating work inside the same environment. It also kept the happy path free of retry cost.

Third, the final gate evaluated outcomes explicitly. The primary attempt could pass, or a failed primary could be rescued by a successful second attempt. A cancelled or skipped job was not treated as success.

This matters because conditional CI jobs often report “skipped,” and broad aggregation expressions can accidentally make a required check look healthy.

That design made the retry a more useful second opinion. It still did not explain the failure.

Capture evidence at the point of failure

The important improvement was diagnostic capture.

On failure, the job recorded machine characteristics and an inventory of generated source files. The goal was to distinguish between competing explanations:

Generated files exist but are incomplete, suggesting truncation or a generation-stage defect.
No generated files exist, suggesting the build is failing earlier.
Failures correlate with constrained resources, suggesting concurrency or memory pressure.

The first artifact ruled out the original mental model. No generated files existed, and the errors occurred during parsing. The failure happened before the stage we had been blaming.

That did not prove the root cause, but it narrowed the search. The changing error signature, the size of the Razor project, and the constraints of the hosted runner supported a new hypothesis: parallel build activity was increasing resource pressure and destabilising the parse stage.

The build was then serialised, and worker-process reuse was disabled.

Notice the wording: supported a hypothesis. Not “fixed the compiler.” Not “root cause solved.” The diagnostic capture and fresh-runner fallback stayed in place so later runs could confirm or falsify the theory.

The trade-off is throughput for signal quality

Serial builds are not free. On large runners or well-behaved workloads, reducing parallelism can lengthen feedback time. A fresh-runner retry also consumes extra minutes whenever the first attempt fails.

The costs can still be acceptable when they are bounded:

The normal successful path does not launch another runner.
Serialisation is scoped to the unstable suite, not the whole pipeline.
Both attempts use the same build definition.
Failure artifacts are retained long enough to compare incidents.
The mitigation has a stated hypothesis and can be removed when the evidence changes.

The objective is not merely a green badge. It is a gate whose meaning the team can explain.

A practical retry checklist

Before adding a retry to CI, I now want clear answers to five questions:

What failure domain do we suspect? Process, workspace, cache, machine, network, or external service?
What changes between attempts? If nothing relevant changes, the attempts are not independent.
Are the attempts equivalent? A shared definition prevents the retry from becoming a weaker test path.
What evidence survives failure? Logs alone may not reveal the pipeline stage or resource conditions.
What is the exit criterion? Decide when the retry will be removed, narrowed, or promoted into a permanent resilience policy.

Retries are sometimes the right operational tool. They can protect delivery while a rare platform defect is investigated. But they should buy time and information, not erase uncertainty.

The most useful shift for me was simple: stop treating a retry as a button, and start treating it as an experiment.

Your Profile Form Shouldn't Change Your Login

Ivan Rossouw — Mon, 20 Jul 2026 05:33:08 +0000

Generic update endpoints are convenient. A profile page loads a DTO, the user edits a few fields, and the server maps the result back onto the stored record. That pattern works well for a biography, display preferences, or a phone number.

It becomes dangerous when the DTO also contains a security-sensitive identifier such as the sign-in email.

The subtle problem is not validation. A perfectly formatted address can still belong to somebody else. The real question is authority: has the user proved control of the new address, or have they merely typed it into an authenticated form?

A recent change across ASP.NET Core, Blazor, and .NET MAUI reinforced a lesson I keep finding useful: sensitive identity mutations deserve intent-revealing commands, not incidental property mapping.

The hidden authority inside an ordinary field

A login email often does more than label an account. It can determine the username, receive password-reset messages, receive security notifications, and act as a lookup key in related data.

If a general profile update assigns that value directly, the endpoint quietly gains the power to move the account's identity. An authenticated session may be enough to edit a profile, but it should not automatically be enough to redirect account recovery.

This is where "the UI validates it" is not a defence. Clients change. Multiple clients may call the same API. Requests can be constructed without the UI. The invariant has to live on the server.

Separate the command from the profile update

The safer shape is deliberately more explicit.

First, the general profile update ignores the login email. The field may still be returned for display, but it is not writable through that operation. This negative rule matters: there should be no second, easier path around the verified workflow.

Second, a dedicated authenticated command starts the change. It accepts only the proposed address. The server resolves the current account from the authenticated principal rather than accepting an account identifier from the request body. That removes an entire class of "change someone else's value" mistakes.

Third, the server validates the proposal and creates a change token bound to both the account and the new address. It sends that token to the proposed address. Possession of the current session starts the request; possession of the new mailbox completes it.

Fourth, the confirmation target is built from trusted server configuration or an allowlist. A client-supplied origin should not decide where a security link points. Otherwise, a legitimate token can be wrapped in an untrusted destination.

Only the confirmation step performs the authoritative mutation.

Keep projections on the correct side of the boundary

Real systems often mirror an email into profile, contact, search, or role-specific records. Those copies create another temptation: let whichever screen is being edited update every copy, including the identity account.

That reverses the authority.

The identity workflow should own the sign-in address. After confirmation, downstream records may be synchronized from that authoritative result. A projection synchronizer should never reach back and modify the login account. Its job is to follow the decision, not make it.

This distinction also improves failure handling. If a downstream synchronization fails, the system can log the drift and repair it without pretending that the user did not successfully confirm the change.

Whether you choose immediate retry, a durable outbox, or a reconciliation job depends on the consistency needs of the system. The important part is to make the authority boundary explicit.

The trade-off

This design costs more than assigning a property.

There is a dedicated request contract, endpoint, token workflow, email template, confirmation handler, client method, and test surface. Users do not see the new value everywhere until confirmation completes. If data is duplicated, projections can temporarily lag and need an observable repair path.

That complexity is justified because it matches the actual risk. Changing a sign-in identifier is not ordinary CRUD. Treating it as a workflow makes the proof, transition, and failure modes visible.

Tests should pin the boundary

The most valuable tests are often negative:

A blank or malformed proposal is rejected.
The current address is rejected as a no-op.
The account comes from authentication, not a body-supplied identifier.
Starting the request sends a confirmation link but does not change the identity.
The token is tied to the proposed address.
A general profile save cannot change the login email.
A downstream synchronizer cannot write the identity account.
The confirmation URL uses a trusted host.

These tests document who is allowed to change what. That is more durable than testing only that a button eventually displays a success message.

A practical review habit

When reviewing a broad update DTO, stop at every field and ask:

Is possession of this form enough authority to change this value?

For many fields, the answer is yes. For login identifiers, recovery destinations, payout details, privileged roles, and consent states, it often is not.

When the answer is no, remove that property from generic mapping. Give it a dedicated command, derive the subject from authenticated context, require the appropriate proof, and test the paths that must remain impossible.

The goal is not more ceremony. It is making authority visible in the design.

Realtime Is a Hint, Not Your Source of Truth

Ivan Rossouw — Sun, 19 Jul 2026 19:22:20 +0000

A notification badge looks like a small feature. In a distributed application, it can expose a large architectural assumption.

I recently investigated a badge that sometimes stayed stale in an ASP.NET Core and Blazor application running across multiple server instances. The notification itself had been created successfully. Durable state was correct. A separate push path could also complete. Yet the visible count did not always change until the user left the screen and returned.

The bug was not really about arithmetic. It was about assigning too much authority to a realtime transport.

The hidden assumption

The UI initially loaded its count from the server and then relied on live events to remain current. That feels reasonable while developing against one server with a stable connection. In production, several ordinary conditions weaken the assumption:

The client can disconnect briefly.
A load balancer can place the sender and receiver on different instances.
A workflow can create the same durable state without passing through the page that happens to broadcast.
A transport or hub can fail after the database operation has already succeeded.
An idempotent retry can revisit work that should not increment the UI again.

None of these conditions makes realtime technology a poor choice. They simply mean that a realtime message should usually be treated as a hint that fresh state is available, not as the state itself.

A three-layer reliability pattern

The repair became much clearer once the responsibilities were separated into three layers.

First, persist the truth

The durable record must be committed independently of whether any connected client is listening. If the user closes the app, changes networks, or connects to another instance, the correct unread state must still be recoverable later.

This ordering matters. Broadcasting first can produce a convincing UI update for data that subsequently fails to commit. Persisting first means a missed broadcast can be healed; an invented broadcast cannot be trusted.

Second, broadcast from the business chokepoint

The live update originally depended on a narrower UI workflow. Other creation paths could produce the same durable result without emitting the event. Moving the broadcast to the shared dispatch path made the behavior consistent across interactive and background work.

The broadcast was also tied to the outcome of the durable write. A genuinely new record emits a hint. An idempotent replay that finds the work already completed does not. That prevents duplicate increments and makes the event reflect a real state transition rather than merely an attempted command.

Third, reconcile periodically

Even a correctly placed broadcast can be missed. The client therefore refreshes the authoritative count on a bounded interval. The loop is cancelled when the component is disposed, and a transient refresh failure does not kill future attempts.

This is not polling as the primary experience. Realtime still gives the fast path. Polling is the repair path.

The trade-off

Hybrid delivery is not free.

Periodic reconciliation adds read traffic. The interval also defines a maximum period of visible staleness when a realtime event is missed. A very short interval increases load and can waste battery or bandwidth on mobile clients. A long interval lowers cost but leaves stale UI visible for longer.

That makes the interval a product and operational decision, not a magic constant. Ask how damaging stale information is, how expensive the read is, how many active clients exist, and whether the endpoint can use a cheap aggregate or cache.

For a badge, bounded staleness may be acceptable. For a payment confirmation or permission change, the interaction may require an immediate authoritative read instead. The pattern stays the same, but the recovery budget changes.

Failure boundaries matter too. Once durable work has succeeded, a hub failure should not turn the whole operation into a failure. Otherwise a temporary transport problem can encourage retries of already-completed business work. Log the broadcast failure, observe it, and let reconciliation repair the presentation.

Test the contract, not the transport

The most useful tests were not attempts to prove that a network never fails. They pinned the business boundary:

New durable state produces one realtime hint.
Previously completed or deduplicated work produces no new hint.
A failing realtime transport does not invalidate successful durable work.

Client-side tests can add the complementary guarantees: an initial authoritative load occurs, the reconciliation loop refreshes state, transient errors do not stop later ticks, and disposal cancels the loop.

These tests document the intended hierarchy. The database-backed read is authoritative. Realtime reduces latency. Reconciliation restores convergence.

A practical review checklist

When reviewing a live-update feature, I now ask five questions:

What is the source of truth?
Where is the single shared point that knows a real state transition occurred?
Can the live event be safely missed or duplicated?
How and how quickly does the client converge again?
What happens if the transport fails after the business operation succeeds?

If those answers are vague, the feature may look realtime in a demo while remaining fragile in production.

The broader lesson is modest: reliability often comes from combining mechanisms with explicit roles. Durable storage provides truth. Realtime provides speed. Periodic reconciliation provides healing. None needs to pretend it can do the others’ job.

That separation made a small badge easier to trust, and it is a pattern I would reuse for dashboards, counters, presence indicators, job progress, and other derived UI state.