Realtime Connection Drain Testing: A Shared Kanban Board Without Flaky Timing

Short answer: test a realtime connection drain by advancing a shared Kanban client through observable protocol states, not by sleeping until a timeout. The deciding constraint is trust: a media employee's browser must receive only the board events its scoped token permits, while reconnect and replay must preserve the last applied cursor.

Use a state-based integration test as the main gate, a deterministic clock for leases and backoff, and a small deployed smoke test for the real transport. This split gives each assertion one job. The integration test proves ordering and authorization; the fake clock proves time boundaries; the smoke test proves that the deployed path can connect, drain, and recover.

I've been paged by missed jobs and duplicate deliveries. That history produces a blunt rule for realtime tests — eventually correct isn't enough when a duplicate card move can trigger another notification.

Layer	Advance it with	Assert	Limitation
Client and gateway integration	Named state transitions	Token scope, cursor order, generation fencing	Requires stable observation points
Lease and retry model	A deterministic clock	Exact expiry and backoff boundaries	Does not exercise a transport
Deployed smoke test	Real connections with a bounded deadline	Connect, drain, resume	Scheduling noise makes it a poor exhaustive suite

How should a shared Kanban board test realtime connection drain without flaky timing?

Model the lifecycle explicitly: OPEN, DRAINING, CONNECTING, BACKFILLING, then OPEN again. A drain notice moves the client out of OPEN, so it cannot send a new card mutation on the old connection. Accepted writes settle, the transport closes, the replacement connection presents a narrowly scoped token, and replay begins after the client's last applied cursor. Only a completed replay can make the board interactive and presence trustworthy again.

The assertion should read like a protocol claim: after drain_started, no command uses generation 7; generation 8 asks for events after cursor 42; duplicate cursor 42 changes nothing; event 43 is applied once; a late generation-7 event is ignored. Compare that with sleep for 500 ms, then inspect the board. The sleep says nothing about which transition occurred, and a loaded CI worker can make a correct system look broken.

No sleeps.

Keep a deadline, but use it only as a circuit breaker. When a two-second test deadline expires, print the current phase, connection generation, cursor, pending writes, and last transition. Don't advance the scenario with that deadline. A useful failure is wanted BACKFILLING; got CONNECTING; cursor=42 generation=8, because an operator can connect it to a missing signal. Timed out is barely evidence.

The token belongs in this state machine. A media company's board might contain embargoed stories, ad schedules, and routine production cards in the same workspace. The reconnect test must use a token scoped to the board under test, attempt an allowed update, and separately prove that an event for another board never reaches the reducer. Never put an administrative credential in browser fixtures. If a test client can subscribe to every board, the harness has erased the primary trust boundary it was supposed to verify.

Make the cursor, token scope, and client trust one contract

Connection draining is transport work, but correctness sits above the transport. The application contract needs four independent controls: a monotonically increasing cursor within one board stream, an idempotency key for each client mutation, a connection generation that fences late messages, and authorization evaluated for the new subscription. None can substitute for another. Cursor deduplication handles replayed durable events; generation fencing handles a stale socket; the idempotency key handles a retried outbound move; token scope decides whether the client may see the stream at all. This is where presence causes trouble. Presence is ephemeral and may be conservative during reconnect, while a card move is durable application state. Marking an editor offline for a short interval is less harmful than showing the editor online before missed board events have been applied. The client should therefore publish online only after the scoped subscription is accepted and backfill completes. Your mileage may vary for typing indicators, but don't let a lossy presence channel define the ordering of durable card events. Treat the drain barrier as a real protocol event — not a comment in the test. It completes after every mutation already accepted on the old connection has a terminal acknowledgment and every inbound event already handed to that connection has reached the reducer. At that point, closing generation 7 is safe. If the server cannot expose such a barrier, the test has no deterministic observation that distinguishes a clean drain from an interrupted one.

WebRTC specifies peer-connection and data-channel behavior, but a shared board still owns its application semantics: authorization, event identity, cursor replay, idempotency, and UI readiness. Keep those rules in a transport-independent reducer. Then a browser transport change does not require rewriting the correctness model.

How can protocol events drive a deterministic Go test?

The smallest useful harness is an in-memory event log plus a client reducer. It does not imitate the network. Instead, the test controls delivery order and checks the same invariants used by the production client. The sequence below starts at cursor 41, delivers 42, drains generation 7, reconnects as generation 8, replays duplicate 42 and new event 43, then delivers a late event from the obsolete generation.

Drain. Reconnect. Replay.

package drain

import "fmt"

type Phase string

const (
    Open        Phase = "OPEN"
    Draining    Phase = "DRAINING"
    Connecting  Phase = "CONNECTING"
    Backfilling Phase = "BACKFILLING"
)

type Event struct {
    BoardID string
    Cursor  int
    CardID  string
    Column  string
}

type Client struct {
    boardID   string
    phase     Phase
    generation int
    cursor    int
    cards     map[string]string
}

func NewClient(boardID string, cursor int) *Client {
    return &Client{
        boardID: boardID, phase: Open, generation: 7,
        cursor: cursor, cards: map[string]string{},
    }
}

func (c *Client) BeginDrain() { c.phase = Draining }

func (c *Client) Reconnect(authorizedBoard string) error {
    if authorizedBoard != c.boardID {
        return fmt.Errorf("TOKEN_SCOPE: board %q is not authorized", c.boardID)
    }
    c.generation++
    c.phase = Connecting
    return nil
}

func (c *Client) BeginBackfill() int {
    c.phase = Backfilling
    return c.cursor
}

func (c *Client) Apply(event Event, generation int) error {
    if event.BoardID != c.boardID || generation != c.generation {
        return nil
    }
    if event.Cursor <= c.cursor {
        return nil
    }
    if event.Cursor != c.cursor+1 {
        return fmt.Errorf("CURSOR_GAP: want %d, got %d", c.cursor+1, event.Cursor)
    }
    c.cards[event.CardID] = event.Column
    c.cursor = event.Cursor
    return nil
}

func (c *Client) FinishBackfill() { c.phase = Open }

The reducer intentionally checks board scope before cursor order. An out-of-scope event must not reveal whether another board has a particular cursor. In production, reject the subscription at the authorization boundary as well; the reducer check is defense in depth for a client that handles multiplexed messages.

Drive the scenario with ordinary Go tests and direct method calls. Each action is synchronous here, so there is no timing variable to tune.

package drain

import "testing"

func TestDrainReplaysOnceAndFencesOldConnection(t *testing.T) {
    client := NewClient("newsroom-east", 41)

    if err := client.Apply(Event{
        BoardID: "newsroom-east", Cursor: 42,
        CardID: "story-17", Column: "editing",
    }, 7); err != nil {
        t.Fatal(err)
    }

    client.BeginDrain()
    if err := client.Reconnect("newsroom-east"); err != nil {
        t.Fatal(err)
    }
    if after := client.BeginBackfill(); after != 42 {
        t.Fatalf("resume cursor: want 42, got %d", after)
    }

    events := []struct {
        event      Event
        generation int
    }{
        {Event{"newsroom-east", 42, "story-17", "editing"}, 8},
        {Event{"newsroom-east", 43, "story-17", "scheduled"}, 8},
        {Event{"newsroom-west", 44, "embargo-9", "editing"}, 8},
        {Event{"newsroom-east", 44, "late-card", "editing"}, 7},
    }
    for _, item := range events {
        if err := client.Apply(item.event, item.generation); err != nil {
            t.Fatal(err)
        }
    }

    client.FinishBackfill()
    if client.phase != Open || client.cursor != 43 {
        t.Fatalf("final state: phase=%s cursor=%d", client.phase, client.cursor)
    }
    if got := client.cards["story-17"]; got != "scheduled" {
        t.Fatalf("story-17: want scheduled, got %q", got)
    }
    if _, leaked := client.cards["embargo-9"]; leaked {
        t.Fatal("out-of-scope board event reached the reducer")
    }
}

Add table-driven schedules around this canonical case: start drain before or after event 42; deliver the duplicate before or during backfill; expire presence before or after reconnect; retry the same mutation key on each generation. After every action, assert that cursors never decrease, each mutation has one terminal result, no command uses a draining generation, and OPEN after reconnect implies completed backfill. If schedules are randomized, log the seed so the exact order can be replayed.

Use a fake clock only for local policies such as heartbeat expiry, presence leases, and reconnect backoff. Advance from 29,999 ms to 30,000 ms and drain all work made runnable at that instant. A close acknowledgment is an event, not a duration, so fake time must not manufacture it. I'm not sure a single wall-clock deadline can be defensible across a laptop and a loaded CI runner; named states remove that guess from the pass condition.

Verify a drain during deployment and define rollback first

Before rollout, emit one structured record for each phase change with board ID, anonymous session ID, old and new phase, generation, cursor, and pending mutation count. Do not log bearer tokens, card titles, or user-entered content. Track active connections by phase, drain duration, reconnect attempts, backfill event counts, cursor-gap rejections, duplicate mutation results, and authorization denials. A sustained population in DRAINING or BACKFILLING is actionable; one slow connection is context, not an incident.

Canary the change against a small connection cohort. Start a drain, wait for the old generation count to reach zero through observed state, and confirm that the replacement cohort reaches OPEN only after backfill. During the same run, move a test card with a stable idempotency key and verify exactly one resulting event. Attempt a subscription with a token scoped to a different test board and verify that no board data is delivered. These checks cover loss, duplication, stale delivery, and client trust without putting a stopwatch in the success criteria.

Rollback is a protocol transition too. Stop admitting connections to the new cohort, allow accepted mutations to reach terminal acknowledgments, drain its connections, and direct reconnects to the previous compatible version. Preserve the replay log and cursor contract across both versions. Never roll back by killing sockets first; that turns a controlled drain into an unclassified disconnect and makes ambiguous writes harder to reconcile.

Keep the runbook short: pause the rollout on cursor gaps, rising duplicate mutation results, scope denials for valid test tokens, or a sustained backfill population. Record the last safe generation and deployment version. Resume only after the canonical generation-7-to-8 scenario and the scoped-subscription check pass against the candidate.

Know when this testing strategy is not suitable

A deterministic integration harness cannot prove browser interoperability, middlebox behavior, or real latency distributions. Keep a limited end-to-end browser matrix for those boundaries, using generous deadlines only to stop stuck runs and dumping protocol state on failure. Stick with standards conformance testing when the disputed risk is data-channel behavior rather than the board's replay logic.

The catch is maintenance. Named observation points and a reference state machine become an internal contract, so protocol changes must update both production transitions and tests. Model tests can explore many schedules, but a behavior omitted from both the model and implementation remains invisible. Review the model beside the production state diagram and replay a small set of canonical drain traces through the real adapter.

This architecture may also be unnecessary. If the media workspace treats presence as best effort and always reloads the complete board after reconnect, a durable replay log can be more machinery than the product needs. Choose snapshot reload for that case. Choose cursor-based replay, generation fencing, scoped reconnect tokens, and idempotent mutations when users must keep editing through drains without losing or duplicating card moves.

The release gate is compact: no wall-clock sleep drives progress; a scoped token can see only its board; every accepted mutation reaches one terminal state before close; reconnect resumes after the last applied cursor; duplicates are harmless; gaps are visible; old generations are fenced; and presence becomes trusted only after backfill. Pass those checks, then drain.

References

• https://www.w3.org/TR/webrtc/