Short answer: realtime backpressure handling for testing a shared kanban board should make admission, ordering, and reconnect completion observable, then exercise those boundaries with a bounded Go queue; sleeping for 50 milliseconds is not synchronization.
The page that wakes an on-call engineer is usually a symptom: “presence count is wrong after reconnect.” By then, a shared kanban board for a game studio may have accepted a burst of card moves, broadcast them to a room, and dropped one acknowledgement while a player changed networks. The final board snapshot can look fine. The cursor is where the damage hides.
I work backward from that alert. The earlier signal should be a rising queue depth for one room, followed by rejected admissions and a reconnect replay that exceeds its SLO. That trace tells us whether the problem is capacity, ordering, or presence semantics. It also gives a test something better than a clock to wait on.
What should a realtime backpressure test prove for reconnecting kanban clients?
I put five invariants in the decision record before selecting a transport or queue library:
- Admission is bounded. Once a room queue reaches capacity, the producer pauses or receives a typed rejection; memory cannot grow with client speed.
- Versions for one card are applied in order. Work for another card may run concurrently, but card 7 must not move from version 12 back to version 11.
- Every accepted operation has a correlation ID and a terminal result: applied, rejected, or expired.
- Reconnect returns a snapshot and a server cursor. A client never infers missing messages from its local counter.
- Presence has an explicit lease and expiry policy. A disconnect is not an immediate human-presence fact when mobile networks routinely reconnect.
WebRTC's recommendation describes browser peer-connection behavior; it does not define the application contract for a board queue or a presence lease. That separation is useful. The protocol can carry events while the service owns the ordering and SLO rules.
A deterministic Go harness
The harness pauses consumption, submits ten moves, and checks the capacity boundary directly. No wall-clock delay is involved.
package board
import "testing"
type Operation struct {
CardID string
Version int
Action string
}
type BoundedQueue struct {
capacity int
items []Operation
}
func NewBoundedQueue(capacity int) *BoundedQueue {
return &BoundedQueue{capacity: capacity}
}
func (q *BoundedQueue) Submit(op Operation) bool {
if len(q.items) >= q.capacity {
return false
}
q.items = append(q.items, op)
return true
}
func (q *BoundedQueue) DrainOne() (Operation, bool) {
if len(q.items) == 0 {
return Operation{}, false
}
op := q.items[0]
q.items = q.items[1:]
return op, true
}
func TestAdmissionWithoutSleep(t *testing.T) {
q := NewBoundedQueue(5)
for version := 1; version <= 10; version++ {
accepted := q.Submit(Operation{CardID: "card-7", Version: version, Action: "move"})
want := version <= 5
if accepted != want {
t.Fatalf("version %d: accepted=%v, want %v", version, accepted, want)
}
}
for version := 1; version <= 5; version++ {
op, ok := q.DrainOne()
if !ok || op.Version != version {
t.Fatalf("drain %d: got %+v, ok=%v", version, op, ok)
}
}
}
In a service, the equivalent drain signal may be a channel receive, a broker acknowledgement, or a database transaction boundary. Expose that signal to the test. A bounded condition wait is fine; time.Sleep(50 * time.Millisecond) is a guess about scheduler timing.
I also assert the trace, not just the final board. In one staging run, the cards looked identical after replay, yet the reconnect path compared a server cursor with a client-local version and published the next edit twice. The useful record was: snapshot cursor 18, accepted operation 19, one publication for version 19. That is a concrete failure contract. I would retain the full event sequence in the test output, including the room identifier, lease transition, queue depth before admission, and the exact reason for a rejection, because a single end-state assertion cannot distinguish a correctly rejected move from a move that was accepted, replayed, and silently overwritten. When this trace is attached to an alert, the person on call can reproduce the scheduling order without trying random sleeps or rerunning the test until it happens to pass.
That's the whole test.
Capacity choices and the catch
There are three sensible places to put the bound. Process memory has the shortest path but loses queued work on exit. A durable broker survives a restart and brings acknowledgement and redelivery semantics. A database outbox keeps mutation and publication close together, while polling or change-stream work becomes part of the operating model.
| Boundary | Guarantee | Limitation | Fit |
|---|---|---|---|
| Process memory | Fast pause and resume | One process; queued work is lost on crash | Unit tests and a single-node prototype |
| Durable broker | Restart survival and redelivery | More acknowledgement states to observe | Rooms spread across workers |
| Database outbox | Mutation and publication can share a transaction | Polling or change-stream lag | Boards that need an auditable write log |
The catch is that a durable queue does not make presence accurate by itself. It can preserve a move while a stale lease still says a player is online. Choose the in-process boundary when loss on process exit is acceptable and the room is local; choose a durable boundary when replay after a worker restart is part of the SLO. Stick with a simpler queue when the team cannot operate redelivery metrics, dead-letter handling, and cursor repair.
False positives have a real cost. If the alert fires at a queue depth that normal tournament launches reach for three seconds, on-call will mute it and miss the five-minute buildup that precedes a bad reconnect. Calibrate thresholds from room size, drain rate, and reconnect latency, and alert on sustained violation rather than a single sample.
Instrumentation that closes the loop
Emit room ID, queue depth, admission result, operation version, cursor, and lease age as structured fields. Counters should separate rejected writes from expired leases; otherwise a dashboard can report “healthy traffic” while presence accuracy degrades. Histograms for replay duration and time-to-ack map directly to the reconnect SLO.
For tests, inject a fake clock only for lease expiry and retry policy. Advance it explicitly, trigger the lease timer, and assert the resulting presence event. Keep queue-drain tests event-driven. I'm not sure a single threshold will fit every game mode, so I would start with per-room distributions and revise the policy after observing real reconnect traces.
The operational check is straightforward: when an alert fires, the trace should answer whether admission stopped, which cursor was issued, and when the lease expired. If any of those are missing, adding capacity merely moves the mystery downstream.
Top comments (0)