A notification badge looks like a small feature. In a distributed application, it can expose a large architectural assumption.
I recently investigated a badge that sometimes stayed stale in an ASP.NET Core and Blazor application running across multiple server instances. The notification itself had been created successfully. Durable state was correct. A separate push path could also complete. Yet the visible count did not always change until the user left the screen and returned.
The bug was not really about arithmetic. It was about assigning too much authority to a realtime transport.
The hidden assumption
The UI initially loaded its count from the server and then relied on live events to remain current. That feels reasonable while developing against one server with a stable connection. In production, several ordinary conditions weaken the assumption:
- The client can disconnect briefly.
- A load balancer can place the sender and receiver on different instances.
- A workflow can create the same durable state without passing through the page that happens to broadcast.
- A transport or hub can fail after the database operation has already succeeded.
- An idempotent retry can revisit work that should not increment the UI again.
None of these conditions makes realtime technology a poor choice. They simply mean that a realtime message should usually be treated as a hint that fresh state is available, not as the state itself.
A three-layer reliability pattern
The repair became much clearer once the responsibilities were separated into three layers.
First, persist the truth
The durable record must be committed independently of whether any connected client is listening. If the user closes the app, changes networks, or connects to another instance, the correct unread state must still be recoverable later.
This ordering matters. Broadcasting first can produce a convincing UI update for data that subsequently fails to commit. Persisting first means a missed broadcast can be healed; an invented broadcast cannot be trusted.
Second, broadcast from the business chokepoint
The live update originally depended on a narrower UI workflow. Other creation paths could produce the same durable result without emitting the event. Moving the broadcast to the shared dispatch path made the behavior consistent across interactive and background work.
The broadcast was also tied to the outcome of the durable write. A genuinely new record emits a hint. An idempotent replay that finds the work already completed does not. That prevents duplicate increments and makes the event reflect a real state transition rather than merely an attempted command.
Third, reconcile periodically
Even a correctly placed broadcast can be missed. The client therefore refreshes the authoritative count on a bounded interval. The loop is cancelled when the component is disposed, and a transient refresh failure does not kill future attempts.
This is not polling as the primary experience. Realtime still gives the fast path. Polling is the repair path.
The trade-off
Hybrid delivery is not free.
Periodic reconciliation adds read traffic. The interval also defines a maximum period of visible staleness when a realtime event is missed. A very short interval increases load and can waste battery or bandwidth on mobile clients. A long interval lowers cost but leaves stale UI visible for longer.
That makes the interval a product and operational decision, not a magic constant. Ask how damaging stale information is, how expensive the read is, how many active clients exist, and whether the endpoint can use a cheap aggregate or cache.
For a badge, bounded staleness may be acceptable. For a payment confirmation or permission change, the interaction may require an immediate authoritative read instead. The pattern stays the same, but the recovery budget changes.
Failure boundaries matter too. Once durable work has succeeded, a hub failure should not turn the whole operation into a failure. Otherwise a temporary transport problem can encourage retries of already-completed business work. Log the broadcast failure, observe it, and let reconciliation repair the presentation.
Test the contract, not the transport
The most useful tests were not attempts to prove that a network never fails. They pinned the business boundary:
- New durable state produces one realtime hint.
- Previously completed or deduplicated work produces no new hint.
- A failing realtime transport does not invalidate successful durable work.
Client-side tests can add the complementary guarantees: an initial authoritative load occurs, the reconciliation loop refreshes state, transient errors do not stop later ticks, and disposal cancels the loop.
These tests document the intended hierarchy. The database-backed read is authoritative. Realtime reduces latency. Reconciliation restores convergence.
A practical review checklist
When reviewing a live-update feature, I now ask five questions:
- What is the source of truth?
- Where is the single shared point that knows a real state transition occurred?
- Can the live event be safely missed or duplicated?
- How and how quickly does the client converge again?
- What happens if the transport fails after the business operation succeeds?
If those answers are vague, the feature may look realtime in a demo while remaining fragile in production.
The broader lesson is modest: reliability often comes from combining mechanisms with explicit roles. Durable storage provides truth. Realtime provides speed. Periodic reconciliation provides healing. None needs to pretend it can do the others’ job.
That separation made a small badge easier to trust, and it is a pattern I would reuse for dashboards, counters, presence indicators, job progress, and other derived UI state.
Top comments (1)
The three-layer split holds up, but one failure mode walks through all three.
Reconciliation buys you liveness. The client re-reads and eventually matches the store. It says nothing about whether that read is correct. The moment the authoritative read serves a derived value, a stored unread_count or the cheap aggregate you floated as an option, instead of counting the rows themselves, drift is on the table. A write bumps the rows but misses the counter. A partial idempotency key decrements twice. Now reconciliation converges the badge to a wrong number, calmly, on every tick.
That inverts the healing story. A missed hint self-heals next cycle. A drifted aggregate is sticky, and the same loop that repairs transport loss quietly launders the drift, so an intermittently wrong badge becomes a confidently, permanently wrong one.
I'd add a sixth question to the checklist: is the authoritative read itself derived, and if so, what recomputes it from ground state, and how would you notice if it silently diverged? The only cheap catch I've found is recomputing from rows on a sampled fraction of reconciles and comparing to the stored count. By construction the UI and the cache already agree while both being wrong, so nothing else surfaces it.