Fixing Forex API Duplicate Ticks: A Production-Grade Deduplication Strategy

Quantitative developers frequently encounter a subtle but critical issue: stable API connections and normal data ingestion, yet corrupted candlestick data, skewed technical indicators, and inaccurate volume statistics. Most people attribute these bugs to unstable market sources, but my years of building enterprise Forex data systems prove otherwise.
Nearly all tick data anomalies stem from flawed post-processing logic, not data collection.
Forex real-time tick data is delivered via WebSocket persistent connections. Network flutters, unexpected disconnections, and active subscription resubscription will trigger server-side data backfill. This native compensation mechanism guarantees full data coverage but inevitably produces duplicate tick records. Without standardized deduplication logic, redundant data will pollute subsequent K-line synthesis, indicator computation, and strategy scheduling.
The most common troubleshooting pitfall here is timestamp-only deduplication. This lightweight method works for static low-frequency data but is completely unfit for high-frequency Forex tick scenarios.

Why Timestamp-Only Filtering Breaks Forex Data Accuracy

In early project iterations, I also adopted timestamp-based deduplication to reduce development costs. In actual production verification, however, this approach causes persistent data loss.
Forex markets support ultra-high-frequency price oscillation. Multiple valid quote updates can be generated within the exact same Unix timestamp. If we simply discard all records with repeated timestamps, valid incremental market changes will be mistakenly filtered out.
The final result is incomplete tick snapshots, discontinuous K-line trends, and systematic calculation errors in quantitative strategies.

Source Analysis: How Duplicate Ticks Are Generated During Reconnection

Under steady network conditions, WebSocket market streams maintain ordered, non-repeating output. Duplicate data is purely a side effect of the server’s fault tolerance design.
To avoid data loss during offline periods, the server will actively resynchronize historical tick data once the client reconnects after a disconnection. The core problem occurs when locally persisted ticks are re-pushed by the server during backfill.
Without targeted deduplication rules, repeatedly pushed identical ticks will be written to the database multiple times, resulting in abnormal volume data and biased indicator outputs.

Production Solution: Per-Tick Unique Identification Logic

To resolve misjudgment fundamentally, I replaced single-condition timestamp filtering with a custom unique identification mechanism for every single tick. This scheme precisely distinguishes invalid retransmitted duplicate data from real market fluctuations, adapting to most mainstream Forex API structures. I adopt AllTick API for daily real-time tick access, which works stably with this deduplication framework.
I use two sets of identification strategies based on API field compatibility.
For interfaces that provide exclusive unique fields such as tick_id or quote_id, native unique identifiers are the most accurate deduplication basis:

if tick_id not in cache:
save_data (tick)
cache.add (tick_id)

For general interfaces without built-in unique IDs, I generate composite unique keys using core business dimensions: trading symbol, precise timestamp, and real-time price. This multi-dimensional verification eliminates the defects of single-field judgment:

tick_key = (
data ["symbol"],
data ["timestamp"],
data ["price"]
)
if tick_key not in tick_cache:
tick_cache.add (tick_key)
save_tick (data)

Engineering suggestion: Control the number of combined fields moderately. Too few dimensions cause false filtering; excessive fields bring unnecessary computational overhead.

Dual-Layer Safeguard: Cache Pre-Filtering + Database Constraints

Pure memory cache deduplication cannot handle extreme scenarios such as process restart, program crash, and cache expiration. For 7×24-hour production stability, I implement a dual-layer deduplication architecture combining cache real-time filtering and database persistent constraints.
Standard production processing pipeline:
Receive Tick Stream → Generate Unique Key → Cache Duplicate Check → Filter Redundant Data → Database Persistence
The cache layer undertakes high-frequency real-time deduplication to eliminate duplicate data caused by network jitter and reconnection. The database unique index acts as the final defense line to prevent duplicate writing caused by business logic exceptions:

CREATE UNIQUE INDEX tick_unique
ON forex_tick (symbol, timestamp, price);

Even if the program logic fails momentarily, underlying database constraints can completely block duplicate data deposition and guarantee data purity.

Precise Retransmission Processing: Do Not Blindly Discard Backfill Data

A common engineering mistake is treating all backfilled historical data as invalid duplicates. In active data supplement and batch synchronization scenarios, identical timestamps may carry different valid price quotes.
My production specification adopts full-field matching judgment: only records with consistent symbol, timestamp, and price are defined as duplicate data and filtered. If the timestamp is the same but the price differs, the record represents effective market fluctuation and must be retained.
All deduplication logic is deployed at the front of the business layer to ensure downstream K-line calculation and strategy analysis run on clean tick streams.

WebSocket Deduplication Implementation

The following is a streamlined, production-adaptable WebSocket tick deduplication implementation:

import websocket
import json
cache = set ()
def on_message (ws, message):
data = json.loads (message)
key = (
data ["symbol"],
data ["timestamp"],
data ["price"]
)
if key in cache:
return
cache.add (key)
print (
data ["symbol"],
data ["price"]
)
ws = websocket.WebSocketApp (
"wss://apis.alltick.co/websocket-api/stock-websocket-interface-api/transaction-quote-subscription",
on_message=on_message
)
ws.run_forever ()

This basic implementation covers core deduplication demands. For formal deployment, you need to match cache expiration policies, automatic reconnection mechanisms, and unified timestamp formatting to achieve full-scenario stability.

Key Optimization Details for Production Deployment

Stable Forex data service relies on long-term operational optimization rather than simple function implementation. Two core details determine system performance.
1. Controlled Cache Lifecycle
Unlimited cache accumulation will continuously occupy server memory and degrade program throughput. Setting reasonable expiration rules for cache keys can clean invalid verification data regularly, balancing operational efficiency and deduplication accuracy.
2. Unified Timestamp Standardization
Different Forex APIs return timestamps in second or millisecond precision. Unstandardized time formats will result in inconsistent unique key generation for identical ticks, causing silent failure of deduplication logic. Global time format unification is a prerequisite for stable system operation.

Summary

In enterprise-level financial analysis and quantitative trading systems, data stream stability and accuracy outweigh data acquisition volume.
Duplicate ticks caused by API reconnection and backfill are easy to overlook but severely impact strategy reliability. Abandoning naive timestamp filtering and adopting multi-dimensional unique key verification plus cache-database dual-layer deduplication can effectively eliminate redundant data with low engineering cost.
This set of practices has been fully verified in production environments, significantly reducing data exception rates and improving the robustness of Forex real-time data systems.