👨‍💻 Quant & Backend Engineering Tutorial
⏱️ Read time: 7‑9 min
🏷️ #api #python #quant #websocket #dataintegration
Multi‑asset dashboards sound simple until you deal with fragmented third‑party market APIs. In this post I break down real‑world integration pain points, an API evaluation checklist, a practical data‑normalization architecture, and full runnable Python WebSocket code for your prototypes.
Introduction
If you’ve built tooling for quantitative trading, you have definitely run into this frustrating problem.
You want a real‑time dashboard aggregating forex pairs, spot precious metals, crypto and equities. On paper, it seems straightforward: connect a handful of external APIs and render market quotes.
In reality, most of your development effort goes toward fixing incompatible data schemas rather than implementing core business features.
I learned this lesson while building an internal multi‑asset monitoring dashboard. Below I share real‑world pitfalls, actionable API evaluation criteria, an architectural pattern to isolate third‑party format differences, plus complete Python WebSocket code you can reuse in side projects.
Real‑world pain‑points of multi‑market data integration
My requirement was simple: render EUR/USD, spot gold, Bitcoin and several US stocks within one dashboard view. I assumed wiring a couple data providers would get the job done quickly.
This project ended up relying on three separate vendors: one for forex and precious metals, one dedicated to crypto feeds, and another for stock market quotes.
Each provider implemented custom authentication flows, symbol naming schemes, timestamp formats and JSON payload structures. Most of my engineering cycles were spent writing translation and mapping logic, while core dashboard functionality remained incomplete.
These four issues appear repeatedly across multi‑feed market‑data projects:
1. Inconsistent symbol naming conventions
The same EUR‑USD forex pair can be formatted as EURUSD or EUR/USD. Crypto trading pairs use either BTC‑USDT or BTCUSDT. Stock symbols require exchange suffixes. Passing a symbol string sourced from Provider A to Provider B almost always yields empty responses.
2. Mixed timestamp formats and field data types
Timestamps may arrive in seconds, milliseconds, or raw datetime strings. Price and volume fields can be numeric values or string‑encoded text. Without explicit type casting, time‑series computations and financial arithmetic will trigger unexpected runtime exceptions.
3. Varied streaming behaviour and rate‑limit enforcement
Some data sources offer low‑latency WebSocket push updates; others only support periodic HTTP polling. Heartbeat keep‑alive logic, automatic disconnection recovery and quota rules vary heavily between vendors, making generic reusable client code difficult to implement.
4. Misaligned trading‑hour models for different assets
Cryptocurrencies trade 24/7 all year. Forex markets close during weekends. Equities follow fixed opening hours, lunch breaks and public‑holiday closures. A naive alert such as “trigger warning after 10 seconds with no incoming data” generates floods of false positives during scheduled market downtime and masks genuine production outages.
Checklist for evaluating multi‑asset market‑data APIs
After this difficult integration experience, my primary evaluation metric is clear: how much custom adapter boilerplate will this API force my team to write?
I use this practical checklist for assessment:
- Can multiple asset classes authenticate via one single API token?
- Are subscription request payload structures consistent across different instruments?
- Do streaming real‑time messages follow uniform, predictable field naming?
- Does official documentation contain a complete human‑readable symbol‑code reference table?
More satisfied checklist items equal less manual normalization work for your codebase. That said, no vendor delivers perfectly unified schemas. Even within one provider, stocks, forex and crypto often live behind separate endpoints with minor field‑level differences.
A pragmatic approach: choose an API with solid protocol consistency, then resolve remaining mismatches within your application’s adapter layer.
Rate‑limits and service stability also deserve close attention. Free tiers usually enforce strict quotas; subscribing to large symbol lists will quickly hit throttling limits. Documentation should explicitly cover heartbeat specifications and reconnection behaviour.
For prototyping and side‑project work, I frequently use AllTick API. Forex, precious metals and crypto share one WebSocket endpoint, while US stocks, Hong Kong stocks and A‑shares are served from a second endpoint. The subscription protocol stays consistent on both endpoints, greatly reducing adapter‑layer implementation overhead.
Architecture: create an internal Quote model to isolate external changes
Facing messy third‑party payloads, your most effective defence is straightforward: define your own internal domain model and shield high‑level business logic from external format changes.
I implement a lightweight Quote dataclass containing only fields the application actually requires:
- data‑source identifier
- instrument symbol code
- price
- volume
- millisecond‑precision timestamp
Every raw incoming market tick flows through normalization logic and is converted to this unified Quote object. Upstream modules including quant strategy code, analytics pipelines and dashboard UI components consume exclusively this internal model and have zero awareness of vendor‑specific JSON schemas.
Core benefits
- When switching market‑data providers, you only modify normalization functions; upper‑level business logic remains untouched.
- Adding new market data feeds mostly requires configuration updates, without large‑scale code refactoring.
Thanks to this abstraction pattern, adding Hong Kong and US stock feeds to my earlier dashboard project took roughly half a working day. One rule I always follow: never couple core business logic directly to raw third‑party API responses.
Practical Python implementation: multiple isolated WebSocket feeds in one process
The snippet below is fully test‑ready Python code. A single application process maintains independent WebSocket connections for different market groups. If one connection drops, other market streams keep operating normally.
import asyncio
import json
import os
import uuid
from dataclasses import dataclass
import websockets
BASE = "wss://quote.alltick.co"
FEEDS = {
"multi": { # forex, precious metals, crypto
"path": "/quote-b-ws-api",
"codes": ["EURUSD", "GOLD", "BTCUSDT"],
},
"stock": { # US stocks, HK stocks, A‑shares
"path": "/quote-stock-b-ws-api",
"codes": ["AAPL.US", "700.HK", "600519.SH"],
},
}
heartbeat = {"cmd_id": 22000, "seq_id": 1, "trace": "heartbeat", "data": {}}
@dataclass
class Quote:
feed: str
code: str
price: float
volume: float
ts_ms: int
def normalize(feed, tick):
"""Convert raw vendor tick payload into unified internal Quote model"""
return Quote(
feed=feed,
code=tick["code"],
price=float(tick["price"]),
volume=float(tick["volume"]),
ts_ms=int(tick["tick_time"]),
)
async def run_feed(name, cfg, queue):
token = "Your AllTick_token_here"
uri = f"{BASE}{cfg['path']}?token={token}"
subscribe = {
"cmd_id": 22004,
"seq_id": 1,
"trace": str(uuid.uuid4()),
"data": {"symbol_list": [{"code": c} for c in cfg["codes"]]},
}
while True: # automatic reconnection after disconnect
try:
async with websockets.connect(uri) as ws:
await ws.send(json.dumps(subscribe))
async def beat():
while True:
await asyncio.sleep(10)
await ws.send(json.dumps(heartbeat))
task = asyncio.create_task(beat())
try:
async for raw in ws:
msg = json.loads(raw)
if msg.get("cmd_id") == 22998:
await queue.put(normalize(name, msg["data"]))
finally:
task.cancel()
except (websockets.ConnectionClosed, OSError):
await asyncio.sleep(3)
async def consumer(queue):
while True:
q = await queue.get()
print(q.feed, q.code, q.price, q.ts_ms)
async def main():
queue = asyncio.Queue()
await asyncio.gather(
consumer(queue),
*[run_feed(n, c, queue) for n, c in FEEDS.items()],
)
asyncio.run(main())
⚠️ Critical implementation notes
- Each market group maintains its separate WebSocket connection with independent subscription, heartbeat and reconnection logic. Parsed tick data goes into one shared async queue for unified consumption.
- Resending subscription commands on the same WebSocket session overwrites your active symbol list. When dynamically adding instruments, submit your complete symbol list again.
- Always log raw API responses and validate timestamp units. Incorrect timestamps will break your entire time‑series analysis pipeline.
Additional engineering considerations after data normalization
Converting incoming data into your internal Quote model is not the end of development work. Keep these three practical points in mind.
1. Implement market‑aware alerting, avoid hard‑coded timeouts
Do not apply identical timeout thresholds across every asset class. Use shorter timeout windows for crypto feeds. For forex and equities, combine timeout checks with trading‑calendar metadata. Missing forex data over weekends represents expected market behaviour and should not trigger alerts.
2. Mitigate floating‑point precision drift
Forex quotes commonly use five decimal places, A‑shares use two, while crypto values can extend to eight decimal digits. For financial calculations, use Python’s Decimal type to prevent cumulative floating‑point calculation errors.
3. Normalize historical K‑line / candle data as well
Real‑time quotes stream over WebSocket endpoints, while historical candle data is usually fetched via REST APIs. These two interfaces almost always return incompatible schemas. Transform REST‑retrieved historical records into the identical Quote domain model before persisting to your database. Your backtesting engine and live‑trading components will then reuse the exact same data objects.
Wrap‑up
There is no silver bullet for multi‑asset market‑data integration. Prioritize APIs with clean specifications and comprehensive documentation. Architect your system so vendor‑specific inconsistencies are fully encapsulated inside adapter layers, keeping core business logic loosely‑coupled and maintainable.
Whenever I kick off a new quant project, defining the internal quote domain model is my very first development step. This habit prevents countless hours of rework later in the project lifecycle. Services such as AllTick API, which feature solid protocol consistency, can further cut down the amount of custom adapter‑layer code you need to write.
đź“ť Disclaimer: This article reflects my personal engineering experience. The provided code is intended solely for educational prototyping. Add robust error handling, security hardening and monitoring before deploying to production environments.
Top comments (0)