Short answer: test a realtime stock trading watchlist with a deterministic event log, a controllable clock, and an explicit reconnect cursor; never make the assertion wait for a wall-clock delay and hope a notification arrives.
The deciding constraint is recovery, not raw delivery speed. A watchlist can look correct on a clean connection and still duplicate, omit, or misroute an update after a preference change and reconnect. The same failure boundary appears in a healthtech dashboard that streams device status: a disconnected clinician's screen must resume from a known position, while the preference snapshot used for each notification remains explainable.
This is an architecture decision record for that boundary. It treats transport as replaceable and makes reconnect and backfill observable behavior. That's less glamorous than tuning millisecond sleeps. It's also testable.
What invariants keep realtime notification preference testing from becoming flaky?
Start with identities that don't depend on arrival time. Every source event needs a stable event ID, a monotonically comparable position within its stream, and the subject being watched. Every preference update needs its own revision. The delivery decision records both: event 1042 was evaluated against preference revision 7, for example. Those numbers are test fixtures, not timing claims.
Four invariants define the contract:
- An event at or before the acknowledged cursor isn't applied twice to the visible watchlist.
- An event after the cursor is eventually considered after reconnect, even if the live connection dropped first.
- A preference revision has an explicit effective boundary. Tests must say whether an already-created event uses the old revision or the new one.
- Suppression is a recorded decision, not an absent message. Otherwise silence could mean “filtered,” “lost,” or “still in flight.”
That last rule comes from the same operational pressure as email, SMS, and OTP delivery: “nothing arrived” is not a diagnosis. Compliance review also gets easier when the system can distinguish a user's preference from a transport failure without reconstructing intent from logs.
Keep the failure boundaries separate. The source can accept an event while the live transport is disconnected. The preference service can commit revision 8 while a consumer still holds revision 7. The dashboard can acknowledge cursor 1042 and crash before painting it. If one test tries to cover all three with sleep(2), its pass condition is scheduler luck — and a slower CI worker changes the result.
Short sleeps don't fix it.
How should you test realtime notification preferences for a stock trading watchlist?
Drive the system through a fake clock and an append-only test feed, then assert on durable state. A test should advance logical time only when time is part of the business rule, such as a cooldown window. Delivery itself should be triggered by adding an event or reconnecting a session, not by waiting.
Use a compact scenario with a sharp preference boundary. The session watches ALPHA, accepts event 1042, disconnects, changes from revision 7 to revision 8, receives events 1043 and 1044 while offline, and reconnects from cursor 1042. The expected record names which revision evaluated each backfilled event. If the specification hasn't decided that point, I'm not sure a test can decide it honestly; product and compliance owners need to set the effective-boundary rule first.
The test harness should fail with domain evidence. ASSERT_GAP_001: expected 1043 after cursor 1042 is useful. “Timed out after 5000 ms” isn't — it says nothing about whether the event log, cursor, filter, or live adapter broke the contract.
from dataclasses import dataclass
@dataclass(frozen=True)
class Event:
position: int
symbol: str
status: str
@dataclass(frozen=True)
class Decision:
position: int
preference_revision: int
delivered: bool
class DeterministicWatchlist:
def __init__(self):
self.events = []
self.decisions = []
self.cursor = 0
self.preference_revision = 7
self.enabled_symbols = {"ALPHA"}
def append(self, event):
self.events.append(event)
def update_preferences(self, revision, enabled_symbols):
self.preference_revision = revision
self.enabled_symbols = set(enabled_symbols)
def reconnect_and_backfill(self):
pending = [event for event in self.events if event.position > self.cursor]
for event in sorted(pending, key=lambda item: item.position):
self.decisions.append(
Decision(
position=event.position,
preference_revision=self.preference_revision,
delivered=event.symbol in self.enabled_symbols,
)
)
self.cursor = event.position
def test_reconnect_uses_the_committed_preference_revision():
harness = DeterministicWatchlist()
harness.cursor = 1042
harness.update_preferences(revision=8, enabled_symbols={"BETA"})
harness.append(Event(1044, "BETA", "price_changed"))
harness.append(Event(1043, "ALPHA", "price_changed"))
harness.reconnect_and_backfill()
assert harness.decisions == [
Decision(1043, 8, False),
Decision(1044, 8, True),
]
assert harness.cursor == 1044
The code makes one policy choice: preferences committed before reconnect apply to the recovered range. A production design may instead bind each event to the revision effective when the event entered the log. Either policy can be coherent. Mixing them can't. Put the chosen rule in the decision record and test simultaneous-looking operations by ordering commands explicitly, not by racing threads.
Then add property-style cases around the critical path: shuffled input still resolves by position; replay from the same cursor doesn't create a second visible update; an empty backfill changes nothing; disabling a symbol records a suppressed decision; and reconnecting across multiple batches advances the cursor only through the batch actually processed. For a healthtech device dashboard, replace symbol and price status with device ID and connectivity status. The recovery contract stays the same, while authorization and retention rules remain domain-specific.
Compare the recovery shapes before choosing a transport
Transport choice matters, but it can't compensate for an undefined cursor or preference boundary. Compare mechanisms by the recovery behavior the application must own:
| Option | Reconnect and backfill shape | Test consequence | Suitable boundary |
|---|---|---|---|
| Periodic polling | Client asks for changes after a cursor | Invoke the poll directly and inspect the next cursor | Update latency may follow the polling interval |
| One-way event stream | Client reconnects and supplies a resume position | Close the stream deliberately, append events, then reconnect | Server-to-client updates dominate |
| Bidirectional socket | Application defines resume, acknowledgement, and replay messages | Exercise the protocol state machine without real sleeps | Client and server both send frequent messages |
| Peer connection | Application still needs an ordering and recovery contract around its data path | Test signaling and recovery as separate boundaries | Peer-to-peer media or data is a real requirement |
WebRTC is a W3C Recommendation and defines peer connections, but choosing it doesn't define a stock-watchlist cursor, a preference revision, or a backfill policy. Those are application contracts. A test suite should therefore run most recovery cases against the transport-neutral harness, with a smaller adapter suite proving that connect, disconnect, resume, and acknowledgement messages map onto that contract.
The catch is operational weight. A bidirectional or peer-oriented path is not suitable when the browser only consumes modest server updates and the team doesn't need its extra interaction model. Stick with polling when delayed freshness is acceptable and simplicity wins. Choose a one-way stream when downstream delivery fits the workflow. Use a bidirectional transport when the product genuinely needs frequent client-originated messages — not because a demo looked more “realtime.”
No option removes backfill storage. If the product promises recovery after a disconnect, some component must retain enough ordered history to honor that promise, and retention must have a stated limit. Beyond that limit, return a typed “snapshot required” outcome and replace local state from a current snapshot. Don't pretend an expired cursor is an empty result.
The rejected option: assertions based on elapsed time
The rejected design opens a real connection, changes a preference, waits a fixed interval, and checks an inbox or dashboard. It conflates business timing with machine timing. A pass can hide a duplicate, while a failure can reflect nothing more than process scheduling. Repeating the test three times just samples the ambiguity three times.
There is a valid use case for elapsed-time tests: a thin end-to-end probe can verify that a deployed path completes within a broad service objective. Keep that probe out of the correctness suite, give it diagnostic event IDs and cursors, and don't use it to prove ordering, filtering, or exactly-once UI behavior. Those claims belong in deterministic tests below the network edge.
Deployment checks should also expose the decision trail: current connection state, last acknowledged cursor, oldest recoverable cursor, active preference revision, backfill count, duplicate-discard count, and suppression reason. Avoid putting sensitive watchlist contents or health-device payloads into broad logs. Identifiers can be correlated without turning observability into a second data store.
The final decision is plain: model recovery first, run preference tests against a deterministic log and clock, and test the chosen transport as an adapter. A stock trading watchlist and a live healthtech device dashboard differ in policy and risk, but both become flaky when elapsed time substitutes for an explicit event boundary.
Top comments (0)