Aggregating real-time financial data at scale looks simple from the outside — hit an API, cache it, display it — until you actually try to do it reliably for 10,000+ assets across crypto, stocks, forex, and commodities. Some lessons from building that pipeline for Vextor Capital.
Source attribution is not optional
Every data point needs a clear chain back to its origin. We pull from CoinGecko, Polygon.io, Alpha Vantage, and FRED, and each asset page has to carry the source, the timestamp, and the update cadence. This isn't just a compliance nicety — it's what lets you debug a bad number fast. If a price looks wrong, you need to know in seconds whether it's your pipeline or the upstream provider.
Rate limits shape your architecture more than you'd think
Free and mid-tier financial APIs have rate limits that don't scale with the number of assets you want to track. The naive approach (one request per asset per refresh) falls over immediately at scale. What actually works:
- Batch requests wherever the provider supports it
- Stagger refresh intervals by asset volatility (crypto needs tighter refresh than, say, government bonds)
- Cache aggressively with a TTL matched to how fast that asset class actually moves, not a single global TTL for everything
Delayed vs. real-time is a UX decision, not just a technical one
Not every asset needs to be real-time, and pretending otherwise wastes API budget. Making the delay explicit and visible on every page (rather than hiding it) turned out to matter more for user trust than shaving seconds off latency.
Revalidation methodology has to be public
If you're going to claim "real-time" or "live" data anywhere, you need a public methodology page describing exactly how often each asset class is revalidated and from where. Without that, "real-time" is just a marketing claim nobody can verify — and it's the first thing a skeptical user (or a journalist) will ask about.
None of this is groundbreaking distributed-systems theory, but the constraints are different from a typical SaaS backend: correctness and provenance matter more than raw throughput, because the cost of being wrong is a user making a financial decision on bad data.
Full methodology write-up: https://vextorcapital.com/methodology
Top comments (5)
Hi, Twelve Data team here. If you'd like to test our API for your next article, we'd be happy to provide you with complimentary access. We offer real-time market data and provide a variety of solutions to help you work with high volumes of assets. Just reach out and let us know!
Thanks for the offer! We build our own attribution and sourcing pipeline for the site (IRS/Fed/CoinGecko-style primary sources with timestamps), so we're cautious about mixing in third-party feeds without full transparency on methodology. Appreciate you reaching out though.
Happy to walk you through our data sources, methodology, coverage anything else, so you can assess whether Twelve Data fits your requirements, and if you're up for it we can always set up short call to discuss this in more detail.
Appreciate the offer, but we're not looking to change our data sourcing setup right now. Will reach out if that changes!
Appreciate the offer, but our pipeline is set for now so I'll pass on the call. If we ever hit a coverage gap our internal sourcing can't fill, Twelve Data will be on the shortlist — thanks for following up.