DEV Community

EmilyL
EmilyL

Posted on

How to Handle Real-Time and Historical Data from a Stock Data Interface in One Normalized Pipeline

#api

We have been reviewing brokerage tools and market data feeds for a while, and one pattern keeps coming up. Teams build a live quote dashboard and a separate historical backtest pipeline, only to discover later that the two datasets do not agree. In this post, we will share the normalization approach we use with enterprise financial data analysts.

The Problem

A typical stock data interface returns many data types: ticks, minute bars, daily bars, and so on. Real-time feeds usually contain price, volume, and timestamp. Historical bars contain open, high, low, close, and volume. If each module keeps the raw API shape, field mismatches and timezone drift start appearing as soon as you combine live and historical data.

Our Solution: A Light Transformation Layer

We insert a small conversion step before data enters storage. Every message becomes a consistent object:

market_data = {
    "symbol": "AAPL",
    "price": 225.50,
    "volume": 200,
    "timestamp": "2026-08-14T13:30:00Z"
}
Enter fullscreen mode Exit fullscreen mode

With this shape, live ticks and historical bars can be stored and queried under the same rules. You do not need a custom reader for each data type.

Standardize Time from the Start

Time zone differences are easy to miss. Exchange local time versus UTC can shift every indicator. Our rule is:

  • Normalize all timestamps to UTC at ingestion.
  • Convert to exchange local time only for display or reporting.
  • Keep the same rule for streaming and batch data.

This one decision saves a lot of debugging later.

Connecting Real-Time and Historical Feeds

A typical stock analysis page works like this:

  1. Load a historical window of K-lines.
  2. Subscribe to real-time ticks.
  3. Update the chart continuously.

The trick is ensuring the end of the historical window aligns with the start of the live stream. We pass live data through the same normalization layer first, then into cache or storage. This keeps front-end and strategy code away from raw vendor formats.

We tried this flow with a WebSocket feed for stock ticks, and the normalization looked like this:

import websocket
import json


def on_message(ws, message):
    data = json.loads(message)

    market_data = {
        "symbol": data.get("symbol"),
        "price": data.get("price"),
        "volume": data.get("volume"),
        "timestamp": data.get("timestamp")
    }

    print(market_data)


ws = websocket.WebSocketApp(
    "wss://api.alltick.co/stock/websocket",
    on_message=on_message
)


ws.run_forever()
Enter fullscreen mode Exit fullscreen mode

The focus here is not just receiving data, but enforcing a consistent standard for everything entering the system.

Details Worth Planning Early

Based on our experience, plan for these:

  • Do not bind application fields directly to vendor response keys. Use a mapping layer.
  • Standardize price precision and volume units.
  • Handle reconnect backfill for live streams.
  • Define a merge policy between real-time cache and historical storage.
  • Keep the transformation layer simple and single-purpose.

Summary

Real-time and historical data are not separate concerns. They are different stages of the same data lifecycle. Setting up a shared schema and time rule early makes charting, strategy analysis, and backtesting more reliable. The data interface is just the starting point. The way you govern that data determines the quality of the system.

Top comments (0)