Short answer: suppress duplicates at the dashboard's state-application boundary with a stable event identity and an atomic first-seen check, while keeping transport retries enabled; page on stale or missing device state, not on a raw duplicate count.
For a fintech fleet streaming terminal health into an incident response dashboard, I would choose at-least-once delivery plus idempotent application before I would promise exactly-once delivery. Reconnects, retries, and fan-out races can all present the same status more than once, but losing a transition from healthy to tamper_detected is the failure that should wake someone. The contract has to preserve that asymmetry.
I've carried the pager through alerts that meant nothing and through the quieter failure where the useful signal never arrived. That history makes me distrust a green dashboard unless I can answer one blunt question: what page fired?
What should realtime duplicate event suppression guarantee for an incident response dashboard?
The guarantee belongs to the state transition, not the socket. Assign every producer event an identity that survives retransmission: tenant, device, producer boot epoch, and a monotonically increasing sequence are a workable tuple when the producer can persist or safely advance them. A consumer then applies that identity once per retention window. Two network deliveries may occur; one operational transition occurs.
Transport settings don't remove this requirement. The W3C WebRTC specification, for example, exposes ordered delivery and retransmission controls for data channels. Those controls describe how messages travel between peers; they don't define the business identity of a terminal status update. A dashboard still needs to decide whether two received payloads represent one observation or two legitimate observations.
Retries happen.
Write the acceptance rule before choosing infrastructure:
- A previously unseen identity may change the materialized device state and emit one downstream notification.
- A repeated identity is acknowledged but cannot change state or notify again.
- A later sequence from the same boot epoch supersedes an earlier sequence.
- A new boot epoch prevents a restarted counter from colliding with old events.
- A gap is observable. It isn't silently rewritten as success.
This is the page-worthy distinction: duplicates test idempotency, while gaps threaten visibility. If device-017 produces sequences 40, 41, 41, and 43, sequence 41 should be applied once and the missing 42 should become a freshness or gap signal. Don't let a high duplicate rate page the responder by itself unless it consumes a defined resource budget or correlates with delayed state.
Read the failure signal before touching the fan-out path
Start a postmortem timeline at the producer identity, then follow it through ingestion, broker or relay, fan-out worker, browser session, and state application. A count at only one layer is ambiguous: two receives with one apply is healthy suppression; one receive with two notifications is a consumer defect; zero recent receives from an active device is a liveness problem. Dashboards flatten those cases into similar-looking charts, which is why I want structured counters and sampled event identities beside the graph. Use four measurements with deliberately different meanings. received_total counts delivery attempts. applied_total counts first-seen identities that reached state. duplicate_total counts acknowledged repeats. sequence_gap_total counts forward jumps within one producer epoch. The invariant to test is received = applied + duplicate + rejected at each consumer boundary, with rejection reasons separated rather than folded into duplicates. This is accounting, not decoration — if the terms don't reconcile, the dashboard can't tell the responder which stage lost custody. Keep cardinality under control by putting tenant and device identifiers in sampled logs or traces, not in an unbounded metric label. The useful alert is usually framed around stale state age, sustained gap rate, queue age, or notification lag. A duplicate counter is diagnostic context.
Count the apply.
One caution: a payload hash is a poor default identity. Two consecutive healthy observations can have identical bodies and still be distinct evidence, while the same logical event may acquire a relay timestamp or serialization difference on retry. Hashing the body answers whether bytes match. It doesn't answer whether the producer intended one event.
Put the idempotency decision beside the state mutation
The safe implementation has one indivisible decision: record the event identity if absent, then allow the winning caller to mutate state. In a single process, a lock can demonstrate the contract. In a multi-worker deployment, the PutIfAbsent operation must be supplied by shared storage with atomic create-if-absent semantics; a local cache alone cannot arbitrate two workers receiving the same retry.
package main
import (
"context"
"fmt"
"sync"
"time"
)
type StatusEvent struct {
TenantID string
DeviceID string
BootID string
Sequence uint64
State string
ObservedAt time.Time
}
type SeenStore interface {
PutIfAbsent(ctx context.Context, key string, expiresAt time.Time) (bool, error)
}
type MemoryStore struct {
mu sync.Mutex
seen map[string]time.Time
}
func (s *MemoryStore) PutIfAbsent(_ context.Context, key string, expiresAt time.Time) (bool, error) {
s.mu.Lock()
defer s.mu.Unlock()
now := time.Now()
if expiry, exists := s.seen[key]; exists && expiry.After(now) {
return false, nil
}
s.seen[key] = expiresAt
return true, nil
}
func eventKey(e StatusEvent) string {
return fmt.Sprintf("%s/%s/%s/%d", e.TenantID, e.DeviceID, e.BootID, e.Sequence)
}
func accept(ctx context.Context, store SeenStore, e StatusEvent, retention time.Duration) (bool, error) {
return store.PutIfAbsent(ctx, eventKey(e), time.Now().Add(retention))
}
The sample is intentionally narrow. Production state application also needs an ordering policy: reject an older sequence from overwriting a newer materialized state, yet retain enough evidence to explain that rejection. If the seen marker and state mutation live in different systems, a crash between them can record an event without applying it. Resolve that with one transactional boundary, or with an inbox record whose unapplied entries are replayed until the state mutation is confirmed. An acknowledgment should follow durable acceptance, not merely parsing.
Retention is an operational parameter, not a magic constant. It must cover the longest retry and replay horizon you intentionally support, plus clock and processing margin; otherwise an old retry can become “new” after expiry. Longer retention increases storage and lookup cost, so estimate it from observed replay age and event volume, then test the chosen bound. I'm not sure what duration fits a given fleet without those distributions, and a round number copied from another system won't resolve that uncertainty.
Prove suppression during deploys, reconnects, and replay
Verification should look like an incident drill. Feed a recorded sequence through the same public ingestion boundary used by devices, disconnect the dashboard consumer after durable receipt but before acknowledgment, reconnect it, and replay an overlapping window. Repeat with two fan-out workers. The expected result is boring: every identity appears in received_total, repeats appear in duplicate_total, each unique identity appears once in applied_total, and the final device state matches the highest accepted sequence.
Then make it adversarial. Send 100 copies of one event concurrently. Deliver sequence 43 before 42. Restart a simulated producer at sequence 1 with a new boot ID. Advance beyond the retention window using a controlled clock. The test should assert both state and notification count, because a dashboard can display the right final state while still paging the same incident repeatedly.
Watch the rollout in two dimensions. First, compare the reconciliation invariant at the old and new consumer paths. Second, track state freshness and sequence gaps by a bounded fleet slice. A canary that merely lowers duplicates may be dropping traffic; lower noise isn't proof of healthier delivery.
Rollback should disable state application on the new path while leaving its observations available for comparison. Keep producer identities stable and don't purge the inbox or seen records during rollback, because erasing that memory turns the next replay into a notification storm. If schema evolution changed the identity tuple, maintain a compatibility reader until the maximum replay horizon has passed.
Stop the rollout if the accounting invariant diverges, final state differs between paths, notification count exceeds unique accepted transitions, or state age breaches the service objective. Fast rollback matters. So does preserving evidence.
Know when this pattern is the wrong tool
TTL-based suppression is not suitable when a legal or financial workflow requires a permanent, auditable uniqueness decision. In that case, use a durable ledger or transactional inbox keyed by the domain command, retain it according to the governing policy, and make the audit trail part of the product record rather than an expiring operational cache. Stick with log compaction or periodic snapshots when consumers only need the latest state and no edge-triggered notification depends on every transition.
The catch is that idempotency doesn't repair missing events, invalid producer epochs, or unauthorized updates. Authentication, schema validation, per-device ordering, gap recovery, and freshness alerts remain separate controls. Nor does every status deserve the same delivery contract: a cosmetic battery-percentage refresh may tolerate coalescing, while a tamper transition may require durable acceptance and explicit acknowledgment.
Choose by the page you are willing to receive. For a live fintech device dashboard, preserve high-consequence transitions, suppress retry noise at the application boundary, and make gaps louder than duplicates.
Top comments (0)