Why Your Precious‑Metal Backtest Is Biased: Duplicate Tick Data & How to Fix It

Intro

If you’ve worked on quantitative backtesting, you’ve probably encountered this frustrating scenario: your strategy looks amazing in backtests, but it falls apart once you think about running it against live market data. There are many potential causes, and one easy‑to‑miss culprit is duplicate tick records coming from your real‑time market API.

While building tick‑level backtesting pipelines for precious‑metal instruments, I ran straight into this issue. Small test datasets showed no obvious problems. But as soon as I loaded longer‑horizon historical tick data, weird behaviour popped up: trade counts were artificially high, technical indicators gave skewed outputs, and backtest metrics no longer reflected realistic market conditions.

I spent lots of time reviewing strategy code and tweaking parameters before realizing the bug wasn’t in my trading logic. The problem existed in the data ingestion pipeline. Repeated identical tick entries were saved into my dataset, and the backtest engine treated every duplicate as an actual market fill. For second‑ and minute‑frequency strategies, this data error gets amplified and pollutes your whole evaluation.

What causes duplicate ticks from precious‑metal real‑time APIs?

Tick data streams travel over networks through multiple stages: server sending, network routing, client receiving and parsing. Any disturbance along this path can trigger re‑delivery of already‑received data. Common causes include:

1. Brief network jitter triggers server‑side message retransmission
2. WebSocket disconnect + automatic reconnection makes the server replay cached tick history
3. API internal acknowledgement logic causes identical market messages to arrive multiple times

Precious‑metal tick data updates extremely frequently, so occasional duplicate entries are normal for real‑world market APIs. Skip data cleansing, and your candle generation and strategy backtesting will carry systematic bias.

Pick your deduplication method carefully

There is no one‑size‑fits‑all solution. You need to balance data accuracy and data completeness. Don’t throw away valid real trade records just to remove duplicates.

Approach	Suitable scenario
Unique trade ID check	High‑precision historical backtesting
Time sliding‑window validation	Live real‑time tick stream processing
Multi‑field composite fingerprint check	General market data analysis

⚠️ Common pitfall: Avoid deduplication based purely on full‑field equality.
Two completely independent real‑world trades can coincidentally have the exact same price and volume. Over‑strict filtering will delete legitimate tick samples and ruin your dataset quality.

Practical solution: fingerprint filtering before persistence

My preferred approach is adding a filter layer before writing ticks to storage. Generate a unique fingerprint key for each incoming tick to check whether we’ve already processed this record.

If your API provides trade‑unique IDs, use those for best accuracy. If trade IDs are unavailable, build a composite key from symbol, timestamp, price and volume.

cache = set()

def check_tick(data):
    key = (
        data["symbol"],
        data["timestamp"],
        data["price"],
        data["volume"]
    )

    if key in cache:
        return False

    cache.add(key)
    return True

This snippet filters most exact duplicate ticks without disrupting normal market‑data flow.

Full WebSocket example & engineering notes

When consuming real‑time market data over WebSocket, decouple reception, duplicate validation and persistence steps. Process raw messages first, run fingerprint deduplication, then run downstream business logic.

import websocket
import json

cache = set()

def on_message(ws, message):
    data = json.loads(message)

    key = (
        data.get("symbol"),
        data.get("timestamp"),
        data.get("price")
    )

    if key in cache:
        return

    cache.add(key)

    print("new tick:", data)

ws = websocket.WebSocketApp(
    "wss://api.alltick.co/ws",
    on_message=on_message
)

ws.run_forever()

Things to keep in mind for real projects:

1. Normalize timestamps: Different data sources may use different time units. Mismatched timestamps break fingerprint comparison and create wrong deduplication results.
2. Keep original raw data: Never overwrite source payloads. Run backtesting against cleaned copies, keep raw data for debugging and audit.
3. Expire cache entries: Add TTL for in‑memory cache to prevent memory leaks for long‑running services. For huge tick datasets, switch to high‑performance external cache services.

Wrap‑up

Many quant developers focus heavily on strategy logic, framework building and parameter optimization, while treating data preprocessing as an afterthought.

Duplicate tick messages seem trivial at first glance. But for short‑term trading strategies, small data defects accumulate rapidly, producing the well‑known pain point: great backtest, poor live performance. Bad backtest‑live divergence is not always caused by bad strategy — noisy source market data is often to blame.

Solid foundational work like tick deduplication, timestamp normalization and layered storage saves you countless confusing debugging hours. For my own prototype work, I pull precious‑metal tick feeds from Alltick API and combine it with the preprocessing workflow shown above to reduce distortions caused by data‑source anomalies.

💡 Disclaimer: This is purely technical engineering content, not investment advice.