Every week, publicly listed stocks in the US have their off-exchange ("dark pool") trading volume disclosed per symbol, per venue, under a market-transparency rule that predates most people's interest in market data APIs. The catch: it's published as raw, paginated compliance data, not a friendly product. Nobody looks at it directly.
Dark Pool Pulse turns that into a signal: for each ticker, this week's off-exchange volume compared against its own trailing average over prior weeks - a stock trading 3x its own normal volume is a far more useful signal than an absolute threshold, which punishes small-caps and ignores mega-caps equally. Only tickers that clear both a minimum volume floor and a minimum spike ratio come back; empty runs are never billed.
Getting clean data out of the underlying transparency API took a few real gotchas.
The interesting engineering bits
Partition-key gotchas. The API enforces that any sort operation must filter ALL its partition keys with an exact-match comparator, or you get a 400 - though the error message thankfully tells you exactly which keys are missing. Once you know that, filtering instead of sorting sidesteps the whole problem.
Comparator naming isn't obvious. "greater or equal" isn't spelled the way you'd guess - it's a short enum name you basically have to discover by trial. A handful of curl calls with different guesses got there faster than reading the docs end to end.
JSON vs CSV by header, not URL param. Same endpoint, same body - send an Accept: application/json header and you get typed JSON back instead of a quoted-CSV text blob you'd otherwise have to parse by hand.
Packaged it as an Apify actor (pay-per-result): Dark Pool Pulse - https://apify.com/0xgollum/dark-pool-pulse
Always curious what data-anomaly patterns other people are scanning for - drop them in the comments.
Top comments (0)