Do You Use Raw Tick Data Directly From Forex API?

If you have worked on forex backtesting or real‑time market ingestion, you may have run into a confusing situation. You double‑check your K‑line calculation logic and confirm there are no obvious bugs in your code, yet the generated minute‑level candlesticks keep deviating from real‑world market observations.

Many developers spend countless hours tuning strategy formulas while overlooking a hidden root cause: raw tick streams returned by forex API are not guaranteed to arrive in correct chronological order.

The forex market generates quotes non‑stop. Major pairs such as EUR/USD and GBP/USD produce massive tick updates within short time windows. Although API responses contain symbol, price and timestamp fields, raw network payloads cannot be fed into computation directly. Sorting and data cleaning are mandatory pre‑processing steps to deliver reliable candlestick generation and indicator computation.

Why You Must Re‑order Tick Timestamps

Many ingestion services simply persist records in the exact order network packets arrive. Under real‑world network conditions, packet arrival order does not equal the actual sequence of market events.

Here is a practical example:
Order received by application:
10:15:03  1.08625
10:15:01  1.08620
10:15:02  1.08622

The true market event sequence should be 10:15:01 → 10:15:02 → 10:15:03.

Without timestamp correction, candlesticks built from misordered ticks will produce wrong open, high and low values. For short‑horizon market research, timestamp offsets of merely several seconds can completely invalidate your analytical conclusions.
Before persisting records to storage, always re‑sort incoming ticks by timestamp to restore the genuine timeline of market events.

Market Data Ingestion: Polling vs WebSocket

Two mainstream approaches are widely used for real‑time forex quote consumption, each with distinct trade‑offs.

Periodic HTTP polling is easy to implement yet has notable drawbacks. Repeated requests miss large volumes of transient price movements, yield low‑granularity datasets and tend to trigger duplicate deliveries, making it a poor fit for high‑frequency tick collection.

WebSocket persistent connections are far better suited for continuous quote subscription, capturing every single market price update. In my day‑to‑day market pipelines I leverage WebSocket endpoints from AllTick API to pull real‑time tick data and feed it into my pre‑processing workflow.

Essential Cleaning Steps for Tick Datasets

Timestamp re‑ordering is only the starting point. Multiple data‑sanitization tasks should be completed before ticks land in your database.

Remove Duplicate Quote Entries

Network retransmission and API retry logic can deliver identical bid‑ask quotes repeatedly for the same symbol at the same moment. These duplicated records carry zero analytical value; they bloat database storage and degrade computation throughput.

In practice you can combine trading symbol, exact timestamp and bid‑ask prices as composite criteria to detect and discard redundant entries and avoid accumulating useless records.

Mark Anomalous Prices Instead of Blind Deletion

Forex prices shift rapidly. Network jitter and transmission failures occasionally introduce out‑of‑band outliers. A common pitfall is deleting all heavily deviated ticks unconditionally. This practice risks erasing genuine sharp market moves.

A more robust approach is evaluating outliers against neighbouring ticks: whether the price sits within recent normal volatility bounds, whether values revert quickly, and whether the timestamp falls within valid trading hours. Once confirmed as anomalous, keep original records with special markers so you can filter them optionally during later analysis.

Standardize Time Representations

Different forex API providers adopt inconsistent time formats. Some return Unix timestamps, others UTC‑formatted strings, while a few emit exchange‑local timestamps.

A proven engineering pattern: convert all incoming timestamps to UTC upon ingestion. Convert to target timezones later solely for display and business analytics. This pattern eliminates analytical bias stemming from mixed time standards.

Python Example for Real‑Time Tick Consumption

Below is minimal sample code demonstrating WebSocket tick subscription and chronological sorting:

import websocket
import json
tick_data = []
def on_message(ws, message):
    data = json.loads(message)
    tick = {
        "symbol": data.get("symbol"),
        "price": float(data.get("price")),
        "timestamp": data.get("timestamp")
    }
    tick_data.append(tick)
    tick_data.sort(
        key=lambda x: x["timestamp"]
    )
    print(tick_data[-1])
ws = websocket.WebSocketApp(
    "wss://api.alltick.co/ws",
    on_message=on_message
)
ws.run_forever()

This snippet illustrates basic logic. In high‑throughput production systems avoid sorting after every incoming message. Queue ticks first and run batch processing over fixed time windows to improve overall throughput.

Considerations for Long‑Term Tick Storage

When archiving historical market data, do not store only price and timestamp. Preserve complete metadata including symbol, bid price, ask price and volume. Full field sets support multi‑timeframe candlestick construction and enable in‑depth historical market reviews.

For massive tick datasets, partition storage by trading symbol plus date. Partitioning effectively reduces I/O overhead and accelerates query performance.

Final Thoughts

Pulling streaming data via forex API is merely the initial step for a market system. The quality of your analysis is determined by the post‑ingestion processing pipeline.

Seemingly trivial work such as timestamp re‑ordering, deduplication and time‑format normalization directly governs the reliability of backtesting and candlestick computation.
Solid pre‑processing of raw tick data yields much more stable candlestick rendering and historical backtesting. System accuracy does not rely purely on strategy algorithms; data pre‑processing plays an equally critical role.