1,000 users, 1 API call: the fan-out pattern that saved my rate limit

If every visitor's browser polls your API, your usage grows with your audience. Mine doesn't: whether one person or a thousand open the app, the upstream API gets a single call.

The app is this live airspace visualizer, a 3D globe that shows live air traffic, with new aircraft positions every 15 seconds. If you wonder how the aircraft keep flying smoothly between updates, I explained it in this other article.

The data comes from a third-party API with a rate limit per account, not per user. If each browser called it directly, a hundred open tabs would mean a hundred calls every 15 seconds, and I'd hit the limit long before the app got any traffic. Besides, the API key can't live in the frontend.

One poller, many clients

The solution is a small proxy between the browsers and the API. It has a single poller that asks the API for new data every 15 seconds and sends the result to every connected browser over Server-Sent Events (SSE). The number of users only changes how many connections the proxy keeps open, not how many requests reach the API.

Each browser opens a connection to /api/states. The proxy sets the SSE headers, keeps the response open and hands it to the poller. When the browser closes the connection, the poller forgets about it.

  app.get('/api/states', (req, res) => {
    res.setHeader('Content-Type', 'text/event-stream')
    res.setHeader('Cache-Control', 'no-cache')
    res.flushHeaders()

    const client_id = poller.register(res)
    // The browser closed the tab or lost the connection
    req.on('close', () => poller.unregister(client_id))
  })

On every tick, the poller makes one request and writes the result to every connection it has. An SSE message is just text, so writing to an open response is enough. If the request fails, the clients get an error event instead and the poller tries again on the next tick. There's also a lock so two requests don't overlap when the API is slow, and a timeout so one that never answers can't hold that lock forever.

  private broadcast(event: string, data: unknown) {
    // An SSE message: event name, data and a blank line
    for (const res of this.clients.values()) {
      res.write(`event: ${event}\ndata: ${JSON.stringify(data)}\n\n`)
    }
  }

  private async dispatcher() {
    // The previous request is still running: skip this tick
    if (this.is_fetching) return
    this.is_fetching = true
    try {
      // One request, no matter how many clients there are
      const data = await this.get_data_with_timeout()
      this.latest_data = data
      this.latest_data_at = Date.now()
      this.broadcast('success', data)
    } catch (error) {
      this.broadcast('error', { message: this.error_message(error) })
    } finally {
      this.is_fetching = false
    }
  }

The poller only runs while someone is watching. The first client to connect starts it. Since the latest result is cached, anyone who connects later gets it straight away instead of waiting up to 15 seconds for the next tick. If the cache is more than a minute old, it's thrown away rather than served stale.

  register(res: express.Response) {
    const client_id = randomUUID()
    this.clients.set(client_id, res)

    // Older than a minute (or nothing yet): discard it
    const age = Date.now() - (this.latest_data_at ?? 0)
    if (age > MAX_DATA_AGE_MS) this.latest_data = null
    // The new client doesn't wait for the next tick
    if (this.latest_data !== null) this.write_sse(res, 'success', this.latest_data)

    // First client: start polling
    if (this.interval_id === null) {
      void this.dispatcher()
      this.interval_id = setInterval(() => void this.dispatcher(), this.time_ms)
    }
    return client_id
  }

When the last client disconnects, the interval is cleared and the proxy stops calling the API until someone comes back.

  unregister(client_id: string) {
    this.clients.delete(client_id)
    // Last one out turns off the lights
    if (this.clients.size === 0 && this.interval_id !== null) {
      clearInterval(this.interval_id)
      this.interval_id = null
    }
  }

Why SSE and not WebSockets

WebSockets would work too, but they solve a problem I don't have. Here the data only goes one way, from the server to the browser, and that's exactly what SSE is for. It works over plain HTTP, so it goes through proxies and load balancers without extra setup, and the browser reconnects on its own if the connection drops. On the client it's just a few lines:

  const event_source = new EventSource(api_url('states'))
  event_source.addEventListener('success', (event) => {
    layer.update_aircrafts(JSON.parse(event.data))
  })

With WebSockets I'd have to handle the protocol upgrade, the reconnection and a message format myself, for a channel where the browser never sends anything.

Disconnecting when no one is watching

There's one case left. If someone opens the app and leaves it in a background tab, the SSE connection stays open. If they're the only user, the poller keeps calling the API every 15 seconds for nobody.

To avoid it, the frontend listens to the visibilitychange event. When the tab is hidden it starts a one-minute timer, and when the timer runs out the stream is paused. If the user comes back before that, the timer is cancelled and nothing happens.

  const INACTIVE_DISCONNECT_MS = 60_000

  const handle_visibility_change = () => {

const handle_visibility_change = () => {
  clearTimeout(timeout)
  if (document.hidden) {
    // Hidden tab: pause the stream after a minute
    timeout = setTimeout(() => set_stream_paused(true), INACTIVE_DISCONNECT_MS)
  } else {
    set_stream_paused(false)
  }
}
document.addEventListener('visibilitychange', handle_visibility_change)

typescript

Pausing the stream closes the EventSource. To the proxy that's just another client leaving, so if it was the last one, polling stops. When the user comes back, a new connection opens and gets the cached data if it's still fresh, or starts the poller again if it isn't. I also add a "Paused" prefix to the page title, so the tab bar shows why the data isn't moving.

useEffect(() => {
  if (stream_paused) return
  const event_source = new EventSource(api_url('states'))
  // ...
  // Closing it unregisters this client in the proxy
  return () => event_source.close()
}, [stream_paused])

Conclusion

This pattern works for any project that needs regular updates from an API you don't control: prices, weather, scores, the status of a deploy. If the data is the same for everyone, there's no reason for each user to ask for it. One poller, a cache and a list of open connections are enough, and the cost stays the same with one user or a thousand.

It has its limits: it only works when everyone sees the same data, and the connections live in the memory of one process, so if you run several instances, each one has its own poller. For a project like this one, it's a good trade-off.

You can see it live at airspace.juanesroig.com. The code is on GitHub: the frontend in airspace-visualization and the proxy in opensky-proxy.