Twice the workers, 14% faster, three broken calls: what a 110-call map taught me about API latency

A founder asks "so where are our holders — Korea or the US?" and the honest answer takes 40 seconds to compute. Not because the maths is hard. Because the answer is spread across ~110 API calls, and each one takes about two seconds.

I built Holder Atlas to draw that answer as one world map: type a token, and the countries its top holders reach exchanges from fill in bubble by bubble, read from Nansen's exchange entity labels and a curated exchange → country table. Code is at github.com/edycutjong/holderatlas. This post is about the part that didn't work: the afternoon I doubled the concurrency to make it faster, got 14 %, and got three broken calls for it.

Where 110 calls come from

A map is one atlas() function calling five Nansen endpoints. The first two are cheap and fixed — search/general to resolve the contract (0 credits), tgm/holders twice for the top-100 population and the exchange-custody subset (5 credits each). The cost is in the per-wallet part:

per wallet   = tgm/transfers (CEX-only, newest, 1 cr)
             → transaction-with-token-transfer-lookup (1 cr)
             → "🏦 Upbit: Deposit"
country      = exchanges.json[entity]        # Upbit → KR · Coinbase → US · Binance → global (never placed)

Up to 52 wallets get examined (12 exchange-custody + 40 people), two calls each. Here is the tail of a real cold run for PEPE, pasted from the CLI's --explain output:

117 credits · 110 calls (0 cached) · 57.3s · atlas 241f4d6ce145

calls:
  search/general                             1 calls     0 cr  0 cached  580 ms avg live
  tgm/holders                                2 calls    10 cr  0 cached  1025 ms avg live
  tgm/transfers                             78 calls    78 cr  0 cached  1695 ms avg live
  transaction-with-token-transfer-lookup    29 calls    29 cr  0 cached  2007 ms avg live

107 of the 110 calls average 1.7–2.0 s each. Serially that would be three and a half minutes; with the 4-wide pool the client ships with, under a 5-requests-per-second bucket, the benchmark across four tokens came out at cold p50 40.3 s, p95 59.0 s (warm p50 7 ms — every call is cached for 24 h and a warm map costs 0 credits).

Forty seconds is a long time to watch a map fill. So the obvious idea: widen the pool.

The experiment: 8-wide under 10 rps

Same bench script (npm run bench), same four tokens, one cold run each, DEFAULT_CONCURRENCY 4 → 8 and the rate bucket 5 → 10 rps. Nansen's documented cap is 300 requests per minute, so 10 rps is fine for one map in isolation. Results, side by side with the defaults:

	4-wide / 5 rps (2026-09-18)	8-wide / 10 rps (2026-09-22)
cold p50	40.3 s	34.5 s
cold p95	59.0 s	58.1 s
failed calls	0 / 444	3 / 441
warm hash = cold hash	4 / 4 tokens	3 / 4 tokens

p50 improved by 14 %. p95 did not move — DEGEN on Base took 59.0 s and then 58.1 s. And the wider run produced three failed lookups where the narrow one had none, all on DEGEN, which is also why its warm hash no longer matched its cold hash: a failed lookup isn't cached, so the second run made different calls and got a different picture.

I reverted it. The defaults stay 4-wide / 5 rps, and the comment in the client now says why:

// Nansen's cap is 300/min. 5 rps keeps one ~110-call atlas inside it with headroom for a second visitor; the rolling
// 300/min window is the hard stop for a third. Measured 2026-09-22 (docs/BENCH.md): 8-wide under 10 rps only moved
// cold p50 40.3 → 34.5 s and introduced 3 failed calls in 441 — per-call latency, not the pool, is the ceiling.
this.limiter = new RateLimiter(opts.rps ?? 5, opts.rpm ?? 300);

The mental model I had wrong

I was treating the map's latency as a throughput problem: N calls, W workers, finish in N/W × per-call time. Double W, halve the time.

That model has two holes when the per-call time is 2 s and the server is the one doing the work.

First, the long pole isn't average-shaped. A map's wall-clock is dominated by its slowest handful of calls — tgm/transfers with a to_address filter over a year of a high-volume token is the slow one, and it stays slow no matter how many siblings run beside it. Widening the pool compresses the middle of the distribution and leaves the tail exactly where it was. That is what "p50 −14 %, p95 flat" means.

Second, concurrency isn't free on the server side either. The client only reports a call as failed after an 8 s timeout and one retry (which also covers 429 and 5xx), so each of those three failures held a worker for up to ~17 s before giving up — and the same lookups had succeeded at 4-wide four days earlier. The bench doesn't record which of timeout/429/5xx each one was, so I won't claim it. What it does record is that the extra width partly paid for itself in failures.

The honest statement is in the DX report I wrote for Nansen alongside the repo: latency is the product's ceiling, and it lives in the per-call time, not the client. The only thing that would make this a five-second experience is a batch endpoint — N transaction hashes in, N labelled transfers out. Nansen has counterparties/batch; a transfer-lookup/batch would do it. That is a wish, not a workaround.

What did survive the experiment: the rolling 300-per-minute window. The old client only had a per-second bucket, which would happily let two concurrent visitors blow through Nansen's documented per-minute cap. The experiment introduced a second sliding window, and it's kept as a hard stop:

/** Two sliding windows: at most `rps` requests per rolling second AND at most `rpm` per rolling minute. */
class RateLimiter {
  private timestamps: number[] = [];
  constructor(private rps: number, private rpm: number) {}
  async take(): Promise<void> {
    for (;;) {
      const now = Date.now();
      this.timestamps = this.timestamps.filter((t) => now - t < 60_000);
      const lastSecond = this.timestamps.filter((t) => now - t < 1000);
      if (lastSecond.length < this.rps && this.timestamps.length < this.rpm) {
        this.timestamps.push(now);
        return;
      }
      // wait for whichever window is full to open by one slot
      const wait = lastSecond.length >= this.rps ? 1000 - (now - lastSecond[0]) : 60_000 - (now - this.timestamps[0]);
      await new Promise((r) => setTimeout(r, wait + 5));
    }
  }
}

The other thing the same bench run caught: 🏦 is not "exchange"

WLFI's attributable share went from 55.2 % to 3.3 % between the two runs. My first reaction was that the wider pool had broken something. It hadn't. The wallet holding 52 % of the analysed supply — Upbit's internal wallet — had a newer exchange touch by the second run: "🏦 Blockchain.com: Deposit". Blockchain.com wasn't in my table yet, so the biggest wallet on the map fell into "entity not in the table" and the number collapsed. Live data moved; the atlas hash said so (warm hash = cold: yes in both runs, but a different hash on each day).

That was the second time the bank emoji bit me. Nansen puts 🏦 on exchange entities — "🏦 Binance 14", "🏦 Upbit: Deposit", "🤖 🏦 Coinbase" — which is exactly the field this whole product hangs on. But it also puts 🏦 on DEX pools, staking contracts, bridges and aggregators: "🤖 🏦 Uniswap: V3 USD1-WLFI … Liquidity Pool", "🤖 🏦 PancakeSwap: CAKE Staking Pool", "🤖 🏦 OKX: DEX Aggregator". And the label_type: "exchange" filter on tgm/holders returns those too. On CAKE, 5 of the 12 "exchange custody" holders were staking pools.

If you take 🏦 at face value, a swap into a Uniswap pool reads as "this person used an exchange whose country I don't know" — grey, but the wrong kind of grey — and a staking contract gets counted as exchange custody in the denominator. The fix is to read the name, and to let a real exchange always win:

/**
 * Nansen puts 🏦 on DEX pools, staking contracts and bridges too ("🤖 🏦 Uniswap: PoolManager V4", "🤖 🏦 PancakeSwap: CAKE
 * Staking Pool"). A 🏦 label that names no exchange in the table AND reads as a pool/contract is not
 * an exchange trace at all; the caller treats such a transfer as "no exchange trace" instead of "entity not in the table".
 * A table exchange always wins ("🏦 Binance: Bridge" is still Binance).
 */
export function isDexOrContractEntity(label: string | null | undefined): boolean {
  if (!label || !label.includes(EXCHANGE_MARK)) return false;
  if (exchangeOf(entityKey(label))) return false;
  return STRUCTURAL_TAG.test(label);
}

And in the engine, a "custody" holder that turns out to be a pool leaves the denominator entirely rather than being placed as global:

if (contractLabel) {
  // the "exchange" holder is a pool / staking / bridge contract wearing Nansen's 🏦: structural, out of the denominator
  row.kind = "structural";
  row.share = 0;
  row.bucket = "untraced";
  emit({ type: "reclass", row, reason: `lookup: ${contractLabel} is a pool/contract, not exchange custody` });
} else if (!label && onlyDexParties(transfers)) {
  // the exchange-filtered transfer was a swap or a stake (Nansen marks DEX pools 🏦 too): no exchange trace, not a table gap
  row.bucket = "untraced";
}

Every label spelling seen live — "Upbit", "Upbit : Link Wallet", "Upbit: Internal Wallet", "Coinbase Prime: Custody Wallet", the ones with invisible U+200B prefixes — is pinned in labels.test.ts. 220 tests, 100 % coverage on the core package, plus 24,000 generated property cases that check things like "global is never attributed" and "structural never enters the denominator".

What it still doesn't do

• Solana can't be named. The transfer lookup has no Solana support, so Solana tokens show 0 % placed with a banner saying which field is missing.
• Supply-weighted means whales decide. 86 % of LINK's analysed supply is one 2017 team wallet with no exchange trace, so LINK reads 4 % placed; the by-wallet share (44 %) prints beside it for exactly this reason.
• Countries are exchange jurisdictions, not people. Coinbase → US, Revolut → GB by licence. Binance, OKX, Bybit are global and stay grey on purpose — 47 % of PEPE's analysed supply is that grey bar.
• It's still 40 seconds cold. That is the point of this post.

Try it on a token you care about — holderatlas.edycu.dev — and open the provenance drawer to see every one of the ~110 calls with its latency. The repo is github.com/edycutjong/holderatlas; the benchmark that produced every number above is docs/BENCH.md, and the DX report with all nine frictions is docs/DX-REPORT.md. Built for Nansen's Meridian buildathon.