Handling Overnight Gaps in Forex API Tick Data Across Trading Sessions

#productivity

If you’re building a quant pipeline that ingests tick data from a forex API, you’ve probably run into weird candles at the start of a new trading week. We did. And for a while, it made our backtests look like random noise. In this post, I’ll walk through the problem, the root cause, and the data-processing pattern we now use to keep overnight gaps from corrupting our signals.

The Problem: Inconsistent Backtest Results

We run the same strategy on different time windows, and the results were all over the place. We checked indicators, parameters, and execution logic. Nothing changed. Then we started examining the tick data itself and found something unexpected: the issue appeared exactly at session boundaries. Monday’s first candles were inheriting Friday’s price gap.

The Data Pain Point: Continuous Time vs. Discontinuous Market

Ticks look like simple timestamped rows. But sorting by time isn’t enough. Friday’s New York close and Monday’s Asian open are separated by a long period with no trading. When a new quote arrives Monday, the price can gap due to news or liquidity shifts. If your pipeline just connects the last Friday tick to the first Monday tick, it treats that jump as a regular market move.

Time Period	What Happens to Tick Data
Friday before close	Liquidity drops, tick count falls
Weekend	No valid trading data
Monday after open	New quotes appear, often with a gap

That distortion affects minute candles, and the error grows when you compute moving averages, volatility, or trend indicators.

Our Approach: Normalize Time Before Anything Else

The first thing we do now is unify the timestamp format. Different forex APIs return different time fields—UTC, local exchange time, server time. Mixing them causes session misalignment. We convert every tick to UTC as soon as it arrives. When we later generate candles or run analysis, we convert to the desired timezone. This avoids daylight saving issues and regional differences.

The Fix: Split by Trading Day, Then Build Candles

Our old process was to dump all ticks into one stream and generate candles from the whole thing. That caused cross-session contamination. The new flow looks like this:

Normalize tick timestamps;
Sort by UTC;
Detect date changes;
Tag new trading sessions;
Generate candles per session.

Each session stays isolated. The overnight gap belongs to Monday, not Friday.

Real-Time Ingestion with WebSocket and Time Normalization

For live tick data, we use WebSocket instead of polling a REST endpoint. It’s much better suited to continuous quote streams. We’ve used AllTick’s API as one of our reference implementations, but the key concept is the same regardless of provider.

import websocket
import json
from datetime import datetime, timezone

def on_message(ws, message):
    data = json.loads(message)

    if data.get("symbol") == "EURUSD":
        timestamp = data.get("timestamp")

        utc_time = datetime.fromtimestamp(
            timestamp / 1000,
            tz=timezone.utc
        )

        price = data.get("price")

        print(
            "time:",
            utc_time,
            "price:",
            price
        )

def on_open(ws):
    request = {
        "action": "subscribe",
        "symbol": "EURUSD",
        "type": "tick"
    }

    ws.send(json.dumps(request))

ws = websocket.WebSocketApp(
    "wss://api.alltick.co/ws",
    on_open=on_open,
    on_message=on_message
)

ws.run_forever()

This snippet focuses on timestamp standardization, not price capture. If a tick’s time is wrong, every downstream calculation inherits the error.

Edge Cases to Watch

After implementing this, we noticed several details that often get overlooked:

Weekend gaps need their own logic. The Friday-to-Monday jump is not a normal price move.
Tick density varies by session. European and US hours produce many ticks; Asian hours can be sparse. Short-period indicators on sparse data can be misleading.
Missing ticks need a diagnostic path. If data stops arriving, you need to distinguish low activity from a broken connection.

Final Thoughts

After working with forex tick data for a while, we’ve learned that the biggest sources of strategy failure aren’t complex formulas—they’re weak data foundations. A real-time forex API gives you a stream of prices, but understanding that stream requires attention to market hours, timestamps, and session boundaries. Cross-session concatenation is not just row-append. Normalize time, mark trading days, and detect abnormal gaps, and your backtests and live systems will be far more stable. Overnight gaps are normal in forex. The trick is recognizing them for what they are.

DEV Community