How to make a WebSocket market-data collector idempotent after reconnects
By Shakti Tiwari · Engineering note · Not investment advice
Most self-built trading data pipelines look correct in a happy-path demo and then quietly corrupt a backtest the moment the network blips. The culprit is almost always the same: on a WebSocket reconnect, the feed retransmits ticks the client already received, and without a dedup layer those duplicates flow straight into your feature store and your labels. This note shows a small, reproducible pattern — a content-hash plus a per-stream monotonic sequence-id — that makes a collector idempotent: processing the same tick twice produces the same state as processing it once. It is provider-agnostic, has no external dependencies, and is testable without a live exchange connection.
Why idempotency matters more than people think
A trading model is only as honest as the data feeding it. Two silent corruptions destroy more backtests than bad math. First, duplicated ticks from reconnects inflate your sample count and, worse, can leak the future into your labels if a retransmitted tick lands after you have already computed a feature from it. Second, reordered ticks break any time-dependent feature (rolling windows, session markers) and make walk-forward splits unsound. Neither is visible in a 30-second local test. They surface at 3am when the exchange rolls a sequence number or your ISP drops a TCP segment. If your collector is not idempotent, you will discover the corruption only after you have trained on poisoned data.
The failure mode, in detail
A WebSocket stream is not a queue with delivery guarantees. When the socket drops mid-tick-batch and reconnects, common behaviours are: the server resumes from the last sequence it saw (good, but rare without client ack); the server resends a trailing window (duplicates); the client's own retry logic re-subscribes and gets a fresh snapshot that overlaps buffered ticks (duplicates + reordering). Each is benign in isolation and catastrophic in aggregate. A single reconnect during a volatile expiry can inject dozens of duplicate ticks that, once they reach your label computation, shift your target distribution imperceptibly — until the model learns the duplicate, not the market.
The pattern
For each normalized tick, compute a stable content-hash from the fields that define the tick's identity (instrument, timestamp-or-trade-id, price, size, side). Keep a per-stream last_seq monotonic counter. On ingest: (1) if tick.seq <= last_seq[stream] → drop (already seen / out of order). (2) Else compute h = hash(normalized(tick)). If h is in the recent-seen set (bounded LRU) → drop. (3) Else accept, update last_seq and the seen-set, and persist. The content-hash catches semantically identical duplicates even when sequence numbering is unreliable; the sequence-id catches reordered retransmissions. You need both. Relying on sequence alone fails when a provider uses random resume tokens; relying on hash alone fails when two genuinely different ticks hash to a collision (rare but possible with a short hash — use full SHA-256 truncated to 16+ chars, not a 32-bit hash).
Minimal reference implementation
import hashlib
from collections import defaultdict, deque
class IdempotentCollector:
def __init__(self, seen_window: int = 100_000):
self.last_seq = defaultdict(int)
self.seen = defaultdict(lambda: deque(maxlen=seen_window))
def _identity_hash(self, tick) -> str:
key = f"{tick['sym']}|{tick['ts']}|{tick['px']}|{tick['sz']}|{tick['side']}"
return hashlib.sha256(key.encode()).hexdigest()[:16]
def ingest(self, stream: str, tick: dict) -> dict | None:
seq = tick.get("seq", 0)
if seq and seq <= self.last_seq[stream]:
return None # reorder / duplicate by sequence
h = self._identity_hash(tick)
if h in self.seen[stream]:
return None # duplicate by content
self.last_seq[stream] = max(self.last_seq[stream], seq)
self.seen[stream].append(h)
return tick # accepted exactly once
This class is the entire dedup layer. Wrap your existing on_message callback: parse the tick, call ingest, and only persist when it returns a tick (not None).
A worked example
Suppose the feed sends ticks with seq 100, 101, 102. The socket drops after 102. On reconnect the provider resends 101, 102, 103, 104. Tick 101 → seq 101 <= last_seq 102 → drop. Tick 102 → drop. Tick 103 → seq 103 > 102, hash not seen → accept, last_seq = 103. Tick 104 → accept. Net: exactly the new ticks persist, duplicates silently removed. Now imagine the provider sends a snapshot on reconnect including 100–104 again (common with some brokers). The sequence check alone would accept 103 and 104, but the content-hash catches they were already seen — dropped. That is why both checks are necessary.
Failure-injection harness
To prove idempotency you must inject the failures, not hope they don't happen. Timeout: drop the socket mid-batch, reconnect, replay the trailing window. Assert accepted-count == unique-count. Duplicate POST: send the same tick twice in one batch. Assert one accept. Reconnect reorder: deliver ticks 5,6,7, then 4,5,6 again. Assert 4 accepted once, 5/6/7 once. Run N cycles; the collector passes if accepted_total == distinct_ticks for every run. That is the test any idempotency claim must survive.
def test_idempotent_under_reconnect():
c = IdempotentCollector()
accepted = 0
for i in range(1, 101):
if c.ingest("nifty", {"sym":"NIFTY","ts":i,"px":i,"sz":1,"side":"B","seq":i}):
accepted += 1
for i in range(90, 101): # reconnect replay
if c.ingest("nifty", {"sym":"NIFTY","ts":i,"px":i,"sz":1,"side":"B","seq":i}):
accepted += 1
assert accepted == 100, f"expected 100 unique, got {accepted}"
Comparison with queue-based dedup
Some engineers reach for a message queue (Kafka, Redis Streams) and assume "the queue guarantees exactly-once." It does not, unless you build idempotency at the consumer. A queue delivers a message; it does not know whether your feature store already persisted that tick. The content-hash + sequence approach is the consumer-side idempotency that makes the queue safe. Use both: the queue for buffering/backpressure, the hash for correctness.
Production deployment example
A minimal production layout: feed_ws.py (WebSocket client, parses ticks, calls collector.ingest); collector.py (IdempotentCollector); store.py (writes accepted ticks to Postgres / Parquet); monitor.py (exposes duplicate_ratio, reconnect_count to Prometheus). feed_ws.py runs as a supervised process. On crash it reconnects; the collector's seen-set is rebuilt from the last persisted last_seq per stream (read from the store on startup). Even a full restart does not reintroduce duplicates.
Provider-specific behaviour matrix
Different providers reconnect differently. Sequence-resume providers send ticks from the last acked seq — rely on last_seq. Trailing-window resend providers send the last N ticks again — rely on content-hash. Full-snapshot providers send the entire book state — rely on content-hash plus aggressive dedupe. Random resume-token providers send an opaque token with no sequence — rely ONLY on content-hash (sequence is useless). If your provider uses random resume tokens, the sequence check still helps catch reordering within a batch, but the content-hash does the real work. Never depend on a single mechanism.
Edge cases that will bite you
Clock skew: if your tick ts comes from the exchange, great; if you stamp it locally, a clock jump can make last_seq logic misbehave — prefer exchange timestamps. Multi-symbol: never share last_seq across symbols; each stream is independent. Snapshot+deltas: some providers send a full book snapshot then deltas; dedup the deltas against the snapshot's hashes too. Late ticks: a tick arriving 10 seconds late with a higher seq is fine; one with lower seq is dropped. Tune if your feed legitimately sends out-of-order within a small window (allow a small seq slack).
FAQ
Q: Won't the seen-set grow forever? No — it is a bounded deque with maxlen. Old hashes fall off the left end. Size it from your provider's max retransmit window; 100k entries is typically days of tick volume. Q: What if two different ticks hash to the same 16-char prefix? Probability is ~2^-64 per pair — ignore it. If storing the hash as a key, use the full 64-char SHA-256. Q: Should I dedup at the database instead? You can, but dedup at ingest is cheaper (no round-trip) and catches reordering before storage. A unique constraint is a good second line of defence. Q: Does this work for order-book deltas? Yes — normalize the delta into an identity key and hash it. Q: My provider sends a heartbeat/ping frame — does that break anything? No. Heartbeats are not ticks; skip them before ingest.
Monitoring in production
Idempotency is not "set and forget." Watch the duplicate ratio (sent − accepted) / sent — a sudden spike means the provider changed reconnect behaviour or last_seq reset. Watch seen-set churn — if the deque evicts faster than expected, your window is too small. Watch reconnect frequency — it correlates with duplicate bursts. Watch accepted-late — a tick accepted far behind last_seq after a long gap usually means a clock or seq issue. A simple Prometheus counter per stream, scraped every 15s, catches most real-world breakage before it poisons a dataset.
Integration with the rest of the pipeline
Idempotency is the first layer of a clean data pipeline. The second layer is strict time-ordering so your labels never leak. The two together — dedup on ingest, ordering on feature computation — are what let a walk-forward backtest mean something. Persist normalized ticks with their content-hash as the primary key. Downstream models read from the store, never the raw socket. This makes every experiment reproducible and every label auditable.
Limitations
The seen-window is bounded; an extremely long-lived stream can theoretically let a duplicate beyond the window through. Size it from your feed's retransmit window. Provider-specific behaviour still needs per-provider notes. Hash collisions at 16-char truncation are ignorable for tick identity but use the full hash if paranoid. This is an engineering control, not a trading signal — it makes your data trustworthy; it says nothing about whether a model trained on it makes money.
Final word
The pattern is small. The discipline is not. An idempotent collector is the difference between a backtest you can trust and one that quietly lies to you on every reconnect. Build it, test it with injected failures, and never trust a pipeline that has not proved its dedup under replay.
Extended case study: debugging a silent duplicate in production
A real incident shape: your model's walk-forward accuracy drops 4 points between Monday and Tuesday with no code change. You suspect the model. Wrong. You check the duplicate ratio and see it jumped from 0.2% to 3.1% Tuesday at 10:14. The provider had a partial outage and resent a 90-second window. Your old collector (no dedup) appended 4,200 ticks. Those ticks landed in the feature window for the 10:15 decision, shifting the volume and spread features. The model didn't degrade — the data did.
With the idempotent collector, the same outage produces a duplicate ratio spike that triggers an alert, but zero ticks are persisted twice. The 10:15 features are correct. No accuracy drop. The lesson: when live performance moves without a deploy, suspect the data feed before the model. The dedup layer turns an invisible corruption into a visible, alertable metric.
Tuning the seen-window
The seen-window size is a trade-off. Too small and a legitimate late tick (arriving after eviction) is accepted twice — a rare duplicate. Too large and memory grows. The right size comes from your provider's maximum retransmit window: measure the longest replay you have ever observed, add 2x headroom, set that as maxlen. For most equity/options feeds this is seconds of data, so 100k entries (covering millions of ticks) is far more than enough. Revisit only if your provider changes behaviour.
Alternative: server-side sequencing
Some providers emit an opaque server_seq that is strictly monotonic per connection but resets on reconnect. Do not trust it across reconnects — always pair it with the content-hash, which is connection-independent. The hash is your source of truth; the sequence is a fast pre-filter.
Practical walkthrough: shipping this in an afternoon
Step 1 — Copy the IdempotentCollector class into your project. It has zero dependencies beyond the standard library. Step 2 — Find your existing WebSocket on_message handler. Before you persist a tick, call collector.ingest(stream, tick). If it returns None, skip persistence. Step 3 — On startup, read the max seq per stream from your store and seed last_seq so a restart does not re-accept old ticks. Step 4 — Add a duplicate_ratio metric that increments on every received tick and every accepted tick; expose it to your monitoring. Step 5 — Write one test that replays a window and asserts accepted == distinct. Run it in CI.
What good looks like: after a forced reconnect in staging, your duplicate_ratio spikes for a second then returns to near zero, your accepted count equals distinct ticks, and your feature store shows no double rows. If any of those fail, the collector is misconfigured — fix before live.
When you outgrow this pattern
If you move to a distributed setup with multiple collectors writing to the same store, the per-process seen-set is no longer sufficient; use the database unique constraint on tick_hash as the final authority, and keep the in-memory set as a fast pre-filter. The pattern scales: local dedup for speed, global dedup for correctness.
Top comments (0)