Short answer: test a stock watchlist's batched realtime delivery with a controlled clock and an explicit event cursor, then prove that reconnect backfill produces the same final state without depending on wall-clock sleeps.
The page says that watchlist updates are late. The on-call view shows quotes arriving now, so the transport looks healthy, yet one symbol has jumped from sequence 4102 to 4105 and another appears twice. This is the awkward alert: the socket is connected, the dashboard is moving, and the user-visible state is still untrustworthy.
Don't start by raising the timeout. Work backward from the page. The earlier signal should have been a growing gap between the highest sequence produced and the highest contiguous sequence applied, split by live delivery and reconnect backfill. A batch-size graph alone can't show that gap; a busy feed can look healthy while omitting one event in the middle.
What should a stock trading watchlist realtime batched event delivery test prove?
It should prove state convergence, not that a callback happened within an arbitrary number of milliseconds. For a fixed input stream, the test needs to establish four properties: events within a batch retain a defined order, a repeated event doesn't change the result, a missing range is requested after reconnect, and live delivery plus backfill yields the same watchlist as uninterrupted delivery.
State wins.
That distinction matters. A test built around time.Sleep(50 * time.Millisecond) asks whether this machine happened to schedule several goroutines quickly enough. A test built around a controlled clock asks whether advancing the batch deadline releases exactly the eligible events. The second question is about system behavior. The first is partly about host load.
I've been paged by both missed jobs and duplicate deliveries. The useful reflex is the same here: assign each event a stable identity, make application idempotent, and keep the cursor tied to contiguous progress rather than the largest number observed. If sequence 4105 arrives before 4103, storing 4105 as the recovery cursor can hide the hole. Store 4102 until 4103 and 4104 have been applied, even if later data is already buffered.
Use a tiny, legible fixture. Three symbols are enough to expose interleaving without turning the assertion into a market-data dump: ACME at sequence 4101, BETA at 4102, and CYAN at 4103. Deliver 4101 and 4103 live, disconnect, then return 4102 and a duplicate 4103 during backfill. The expected state contains each event once and advances the contiguous cursor to 4103. These numbers are test data, not claims about production timing or throughput.
The alert-to-action trace
The page should name the violated user invariant: watchlist_contiguous_lag exceeded its policy window for a watched stream. The first action is to inspect the last contiguous cursor, the highest observed sequence, the age of the oldest gap, and whether the subscriber is in live, disconnected, or backfilling state. Those fields turn “updates are late” into a bounded question.
Now walk backward. The page fired because the oldest unresolved gap aged past a threshold. Before that, an instrumentation event should have recorded the transition from sequence 4102 to 4105, with expected_sequence=4103, observed_sequence=4105, and a stream identifier that doesn't contain account data. On reconnect, another event should record the requested cursor and the cursor committed after replay. Counters for received and applied events are useful, but their difference must be explained by duplicates, buffered gaps, or rejected input; an unexplained remainder is the symptom.
The instrumentation change is small but decisive: measure contiguous progress at the apply boundary. Transport receive time is too early because receipt doesn't prove application, while UI render time is too late because it folds client rendering into delivery health. Record both when they help diagnosis, but page on the boundary owned by the delivery system.
Keep labels bounded. Symbol, account, and cursor values belong in logs or traces with appropriate handling, not as unbounded metric dimensions. Metrics answer whether the failure is widespread; a sampled trace or structured log explains one stream. This split also makes the runbook usable under pressure — first establish scope, then inspect a representative gap.
No guesswork.
Drive batches with a controlled clock
The batching component should depend on a clock interface that the test owns. Production supplies a real timer. The test supplies a manual clock and advances it only after arranging inputs and observers. This makes the release point an explicit action rather than a race against the scheduler.
The following Go sketch keeps the contract narrow. It doesn't model a network or select a transport; it demonstrates the clock boundary and the invariant the surrounding test should assert.
package watchlist
import (
"sort"
"sync"
"time"
)
type Event struct {
ID string
Sequence uint64
Symbol string
Price int64
}
type ManualClock struct {
mu sync.Mutex
now time.Time
}
func NewManualClock(start time.Time) *ManualClock {
return &ManualClock{now: start}
}
func (c *ManualClock) Now() time.Time {
c.mu.Lock()
defer c.mu.Unlock()
return c.now
}
func (c *ManualClock) Advance(d time.Duration) {
c.mu.Lock()
c.now = c.now.Add(d)
c.mu.Unlock()
}
type Batcher struct {
clock *ManualClock
interval time.Duration
openedAt time.Time
pending []Event
}
func NewBatcher(clock *ManualClock, interval time.Duration) *Batcher {
return &Batcher{clock: clock, interval: interval, openedAt: clock.Now()}
}
func (b *Batcher) Add(event Event) {
b.pending = append(b.pending, event)
}
func (b *Batcher) FlushDue() []Event {
if b.clock.Now().Before(b.openedAt.Add(b.interval)) {
return nil
}
out := append([]Event(nil), b.pending...)
sort.SliceStable(out, func(i, j int) bool {
return out[i].Sequence < out[j].Sequence
})
b.pending = b.pending[:0]
b.openedAt = b.clock.Now()
return out
}
A focused test adds two events, calls FlushDue before advancing the clock, and expects no batch. It then advances by the configured interval and expects both events in sequence order. There is no sleep and no eventually loop. The test controls the cause of the transition.
The catch is that a manual clock only removes timing uncertainty inside code that actually uses it. It can't make an end-to-end browser, operating system scheduler, or external service deterministic. Keep a small number of end-to-end checks for wiring and user-visible behavior, but use component tests for exhaustive boundaries such as one tick before the deadline, exactly at the deadline, empty batches, and inputs arriving while a flush is being applied. Your mileage may vary on how many end-to-end cases are useful; the resolving evidence is whether those cases catch integration failures that the component suite cannot represent.
Reconnect and backfill are one state machine
Treating reconnect as transport plumbing is how a green batching test coexists with a bad watchlist. The subscriber has at least three meaningful modes: receiving live events, disconnected, and backfilling from the last contiguous cursor. Live events that arrive during backfill need an explicit policy. They can be buffered until replay catches up, or merged by sequence with replay, but they cannot silently advance the committed cursor across a hole.
Make the apply operation idempotent by event ID and monotonic by sequence per stream. “Idempotent” doesn't mean discarding every lower sequence: an event can be older than the highest observed number and still fill the exact missing slot required for contiguous progress. Apply the missing event once, collapse any now-contiguous buffered range, and commit the new cursor only after the state update succeeds. Then acknowledge progress. That order gives the runbook a concrete recovery point after interruption.
The core test table should vary the failure boundary rather than multiply delays:
| Case | Live input | Backfill input | Required assertion |
|---|---|---|---|
| Clean batch | 4101, 4102, 4103 | none | One ordered application; cursor 4103 |
| Gap before reconnect | 4101, 4103 | 4102 | Final state matches clean batch |
| Duplicate replay | 4101, 4102 | 4102, 4103 | 4102 changes state once; cursor 4103 |
| Live event during replay | 4101, 4103 | 4102 while 4104 arrives | No cursor skips; final cursor 4104 |
Run each case through the same state reducer twice: once as an uninterrupted ordered stream and once with the listed disconnect boundary. Compare the final watchlist, the committed cursor, and the set of applied event IDs. Comparing emitted callback counts alone is weak because an implementation may legitimately coalesce renders while preserving state.
Take the gap case slowly because it catches several plausible mistakes at once. Feed 4101 and assert that the applied cursor becomes 4101; feed 4103 and assert that the displayed state for 4103 has not been committed across the missing slot; mark the connection as lost; ask the recovery component for work and assert that its request starts after 4101; return 4102 followed by a repeated 4103; then assert a single final state transition for each event ID and a cursor of 4103. Run the same fixture again with 4102 arriving live before the disconnect signal is processed. Both schedules must converge. If they do not, the failure belongs to ordering or cursor ownership, not to a vague timeout, and the test output already contains the sequence needed to reproduce it.
Page the gap.
Transport selection sits outside this invariant. A bidirectional peer connection, for example, has a broader standardized model described by the W3C WebRTC Recommendation, but choosing that transport doesn't answer how the application identifies events, commits contiguous progress, or requests replay. Stick with the simplest transport your environment can operate when ordered backfill is an application responsibility; choose a more involved channel only when its communication model is actually required. The state-machine tests should survive that choice.
Set the page threshold from an error budget
The early signal is oldest-gap age, but the paging threshold cannot be copied from the batch interval. A single batch interval is expected waiting, not necessarily failure. Define a delivery objective that includes the allowed batch wait, reconnect detection, backfill, and application, then alert when sustained contiguous lag threatens that objective. Warning telemetry may be tighter; paging should demand action.
This is where I'm not sure a universal number exists. A watchlist used for passive monitoring and one used immediately before an order may carry different risk, and the appropriate threshold needs product policy plus observed recovery distributions. The test suite can prove boundary behavior at any configured duration. It cannot decide the business tolerance.
Test the alert itself with the same controlled clock: create a gap, advance to just before the policy boundary, and assert silence; advance to the boundary and assert one actionable signal; fill the gap and assert recovery. Also verify that a duplicate without a gap never pages and that a disconnected stream enters the documented recovery state rather than being reported as current.
Set this threshold too low and ordinary batching or brief reconnects wake someone who has no action to take. Repeated false positives train the team to distrust the very signal meant to catch missing watchlist events. Set it too high and the first reliable report comes from a user staring at a stale price. The right endpoint of this trace is not “more alerts.” It is one page whose evidence points directly to a cursor gap, a backfill state, and a recovery action.
References
- W3C, “WebRTC: Real-Time Communication in Browsers”: https://www.w3.org/TR/webrtc/
Top comments (0)