EmilyL

Posted on Aug 19

Optimizing Historical K-Line Queries from a Gold Real-Time API with a Simple Cache

#tutorial

I recently measured the performance of my gold market analysis tool and found a problem: a single backtest run was triggering 950 historical K-line requests and spending about 3.2 seconds just on data retrieval. After I introduced a local caching layer, the same task completed in 0.6 seconds.

That's an 81% improvement, and it came from a relatively simple change in how data flows through the system. Here's a detailed breakdown of the problem, the solution, and the implementation.

The Problem: Redundant Requests for Static Data

In quantitative trading, historical K-lines are accessed constantly. Whether you're calculating moving averages, Bollinger Bands, or running a full strategy backtest, you need to read candles from days or months in the past.

In my first version, every part of the code called the gold real-time API directly when it needed historical data. With a small dataset, that worked fine. But once I ran multiple strategies in parallel or swept through many parameter combinations, the latency became obvious.

The key realization was this: the API wasn't the bottleneck. The repeated network round-trips and JSON parsing were eating up time — and all for data that never changed.

Historical K-lines have a special property: once a candle is closed, it's immutable. Yesterday's 1-minute gold K-line won't change tomorrow. So fetching it repeatedly from a remote server is wasted work.

The Solution: A Two-Step Data Access Pattern

I changed the data access logic to:

Check whether the local cache already covers the requested range.
If data is missing or incomplete, fetch only the missing portion from the API.

This simple pattern eliminated most redundant traffic.

Implementing the Cache Layer

I designed two cache types based on how often the data changes.

Real-Time Data: In-Memory Cache

Real-time quotes change constantly, so I keep them in memory and only retain the most recent tick or price. This provides fast reads without persistence overhead.

Historical Data: Persistent Storage

Historical K-lines are stable, so I persist them to a local file or database. On the next startup, the program loads the cache instead of re-downloading everything.

When I store historical candles, I include these fields:

Field	Purpose
symbol	Identifies the trading instrument
timeframe	Identifies the K-line period
start and end time	Matches the requested range
OHLC data	Used for indicators and backtesting

With this metadata, I can quickly determine whether the cache satisfies a query. If only a few hours are missing, I fetch just that slice and merge it with what's already stored.

Bridging Real-Time Data and Cache

Real-time and historical data shouldn't be treated as isolated systems. If they are, you'll end up with a gap between the latest candle and the historical series.

My approach is to route real-time ticks into the cache first, then build K-lines from those ticks based on the timeframe. When a period closes, I save the completed K-line to persistent storage.

Here's an example using the AllTick API, where I receive tick data over WebSocket and store the latest price in a memory cache:

import websocket
import json
from datetime import datetime


market_cache = {}


def on_message(ws, message):
    data = json.loads(message)

    symbol = data.get("symbol")
    price = data.get("price")
    timestamp = data.get("timestamp")

    market_cache[symbol] = {
        "price": price,
        "timestamp": timestamp,
        "update_time": datetime.now()
    }

    print(symbol, price)


ws = websocket.WebSocketApp(
    "wss://api.alltick.co/ws",
    on_message=on_message
)


ws.run_forever()

Once the latest price is in the cache, subsequent K-line calculations and chart displays can read from memory directly, avoiding additional API calls.

Pitfalls and Maintenance

A cache is not a set-and-forget component.

The gold market operates nearly 24 hours, and real-time data can go stale quickly. So I use different expiration policies for real-time and historical data. Real-time prices expire fast; historical candles can be stored long-term.

Another common mistake is updating the cache by re-downloading an entire range when only a small segment is missing. Instead, fetch just the missing part and merge it. For a long-running system, this difference accumulates significantly.

If you need multiple strategies to read the same data concurrently, consider upgrading to Redis so different processes can share one cached copy. That's the natural next step when the system scales.

Conclusion: Data Flow Matters as Much as API Speed

This optimization changed how I think about performance in market data systems. The API speed is only one factor. How data moves through your application is equally important.

Real-time data provides fresh prices; the cache eliminates redundant reads. Together, they keep backtesting and indicator calculations stable.

If your application queries historical K-lines frequently, treat caching as a core part of the data pipeline — not an optional enhancement. Design it early, and you'll have a much easier time scaling to more symbols and larger datasets later.

DEV Community