Building a Zero-API-Key Real-Time Market Data Proxy

w Bush G — Mon, 03 Aug 2026 06:08:59 +0000

Building a Zero-API-Key Real-Time Market Data Proxy

A one-screen market dashboard that pulls live quotes for A-shares, HK, US stocks, commodities, crypto and treasury yields — without a single paid API key.

The problem: market data APIs are expensive

When I started Market Research Cockpit, I priced out the usual suspects:

Polygon.io — $29/mo for real-time US stocks
Finnhub — $49/mo for websockets
Alpha Vantage — 25 calls/day free tier, then $49/mo
EODHD / Tiingo / TwelveData — all $20–80/mo

For a personal research tool that polls every 5 seconds, that's hundreds of dollars a year for data that is, frankly, public. The A-share and HK market data I needed most wasn't even available on these platforms.

So I built the proxy on top of free public endpoints instead. Here's how it works and what I learned.

The data sources

Market	Source	Endpoint
A-shares / HK quotes	Tencent	`qt.gtimg.cn/q=` (GBK)
US index quotes	Tencent + Sina	`hq.sinajs.cn`
VIX	Sina futures	`hq.sinajs.cn/list=hf_VX`
Sector boards / money flow	Tencent + Eastmoney	`proxy.finance.qq.com`, `push2delay.eastmoney.com`
Crypto	Binance / OKX	`/api/v3/ticker/24hr`
Treasury yields	Eastmoney datacenter	`datacenter-web.eastmoney.com`
Earnings calendar	Eastmoney datacenter	`data.eastmoney.com`

All of them are undocumented public JSON endpoints. No keys, no rate-limit contracts — just HTTP.

Architecture

Browser (React 19)
   │  /api/*  (batched every 5s)
   ▼
Node proxy (zero deps beyond iconv-lite)
   │  per-symbol TTL cache (1.5s) + inflight merge
   ▼
Tencent / Sina / Eastmoney / Binance / OKX

The frontend never talks to upstreams directly. The proxy is the single chokepoint: it caches, dedupes, decodes and degrades gracefully.

The 6 engineering decisions that matter

1. Per-symbol caching, not per-request-string caching

The naive approach — cache.set(requestUrl, data) — breaks the moment your frontend's watchlist changes. The dashboard's panels subscribe to different code sets dynamically, so the request string changes every poll and you miss the cache every time, hammering upstream.

The fix: cache per symbol with a 1.5s TTL, and only fetch the symbols that missed:

for (const c of codes.split(",")) {
  const hit = cache.get(`q:${c}`);
  if (hit && Date.now() - hit.ts < 1500) out[c] = hit.data;
  else missing.push(c);
}
// fetch only `missing`, in chunks of 60

2. Inflight merging — kill the thundering herd

When a watchlist of 300 codes comes in and the cache is cold, 300 concurrent requests would hit upstream in one burst. The cached() helper stores the in-flight promise in the cache slot:

async function cached(key, ttl, fn) {
  const hit = cache.get(key);
  if (hit?.data && now - hit.ts < ttl) return hit.data;
  if (hit?.inflight) return hit.inflight; // share the same promise
  const inflight = fn().then(data => {
    cacheSet(key, { ts: now, data, inflight: null, ttl });
    return data;
  }).catch(e => {
    if (hit?.data !== undefined) return hit.data; // stale-while-error
    throw e;
  });
  cacheSet(key, { ts: hit?.ts || 0, data: hit?.data, inflight, ttl });
  return inflight;
}

Cold start = one upstream request per missing symbol, not N. And if upstream dies mid-flight, callers get the last good value instead of an error — the dashboard keeps rendering stale data instead of flashing red.

3. GBK decoding and referer spoofing

Two Chinese upstreams have old-school quirks:

Tencent returns GBK-encoded text. fetch gives you UTF-8 garbage unless you decode the buffer manually with iconv.
Sina returns 403 without a Referer header pointing at finance.sina.com.cn. The same URL, same UA, different referer — different result.

Both took me a debugging session each. The proxy normalizes both so the frontend only ever sees clean UTF-8 JSON.

4. The curl fallback

Node's fetch (undici) has a different TLS fingerprint than a browser. Some upstreams (CNBC especially) intermittently reset connections to undici while serving curl fine. The fix is brutally simple — a dual channel:

async function fetchTextAny(url, opts) {
  try { return await fetchText(url, opts); }
  catch { return curlText(url, opts); } // execFile("curl", ...)
}

fetchTextAny is now the default for every upstream call. Zero config, zero dependencies beyond the system curl.

5. Chunking — long URLs get rejected

Batch endpoints like Tencent's q=code1,code2,... reject URLs with too many symbols. 60 per chunk keeps every request under the limit while staying parallel:

const chunks = [];
for (let i = 0; i < missing.length; i += 60) chunks.push(missing.slice(i, i + 60));
const texts = await Promise.all(chunks.map(c => fetchText(`https://qt.gtimg.cn/q=${c.join(",")}`)));

6. Defensive caching: bounded LRU, no prototype pollution

The cache is a Map capped at 2000 entries with a 60s sweeper, so a user-typed garbage key can't grow it unboundedly.
Response objects are built on Object.create(null) — a symbol named __proto__ from upstream can't pollute the prototype chain.
Every cache write does cache.delete(key) before cache.set(key, ...) to refresh insertion order — that's what makes the eviction a real LRU.

What the frontend gets

The React app just calls /api/quotes?codes=sh600519,sz000001,... every 5 seconds. The proxy decides what's cached, what's stale, and what upstream to hit. The result:

Zero API keys — clone, npm install, npm run dev, done
~0 upstream cost — one public request per symbol per 1.5s at most
Resilient — stale-while-error + curl fallback + multi-source (Binance → OKX) keeps the screen alive through upstream outages
~400KB gzipped bundle for the whole cockpit (React 19 + Vite 7 + Tailwind)

Lessons learned

Public endpoints are undocumented contracts — pin the exact URL shape in one place, because they will break. My proxy centralizes every upstream call behind one function so a format change is a one-line fix.
Test against the real upstream, not mock data. GBK encoding and referer requirements only show up against production.
Rate limits still apply, they're just invisible. The 429 you don't see is the connection that silently hangs. Treat every upstream as hostile: timeout everything (AbortController, 8s default), cache aggressively, degrade gracefully.

Try it

Live demo: mrd.hermes.cc.cd
Repo: github.com/theBigGavin/marketingdashboard

One screen: A-shares / HK / US, commodities, crypto, treasury yields, sector heat, money flow, industry-chain watchlists, and a 7×24 news flash. Plus an Android TV shell with D-pad navigation if you want it on a wall screen.

If you've built something similar — or hit a quirk in one of these upstreams — I'd love to hear about it in the comments.

DEV Community: w Bush G

Building a Zero-API-Key Real-Time Market Data Proxy

Building a Zero-API-Key Real-Time Market Data Proxy

The problem: market data APIs are expensive

The data sources

Architecture

The 6 engineering decisions that matter

1. Per-symbol caching, not per-request-string caching

2. Inflight merging — kill the thundering herd

3. GBK decoding and referer spoofing

4. The curl fallback

5. Chunking — long URLs get rejected

6. Defensive caching: bounded LRU, no prototype pollution

What the frontend gets

Lessons learned

Try it