Eshaan Agrawal

Posted on Jul 3

The Cache Was Working. And Still Causing Duplicate API Calls

#webdev #performance #opensource #typescript

The Cache Was Working. And Still Causing Duplicate API Calls.

The cache wasn't broken.
Every response was being stored correctly.
Every cache hit was returning instantly.

And yet — three concurrent requests for the same username were hitting GitHub three times.

This is the story of a thundering herd bug I found in CommitPulse — a high-performance Next.js API that transforms raw GitHub contribution data into 3D isometric SVG streak badges. 106 stars, actively used, real traffic hitting the GitHub API on every badge render and dashboard load.

The cache worked perfectly for sequential requests. Concurrent ones were a different story.

What is CommitPulse

Quick context before the bug.

CommitPulse takes a GitHub username, fetches contribution data from GitHub's GraphQL API, and renders it as a premium SVG badge. The kind you embed in a README. Every time someone loads a page with your badge, a request fires. If your README gets popular, that's a lot of requests — and GitHub's API has rate limits.

So CommitPulse has a cache layer. Fetch once, store the result, return it for subsequent requests. Makes sense. The problem was what happened before the first response came back.

How I Found It

I was reading through lib/github.ts tracing the contribution fetch path end-to-end.

The cache check looked like this — simplified, but this is the pattern across contribution, profile, and repo fetches:

// lib/github.ts:254-258
const cached = await contributionsCache.get(key)
if (cached) return cached

const res = await fetchWithRetry(GITHUB_GRAPHQL_URL, options)
// ... process response
contributionsCache.set(key, result)
return result

Check cache. Miss. Fetch from GitHub. Store result. Return.

That's correct for a single request. But read it again thinking about two requests arriving at the same time for the same username.

Request A arrives. Checks cache. Miss. Starts fetching from GitHub.

Request B arrives 5ms later. Checks cache. Miss — because Request A hasn't finished yet, so nothing is stored. Starts fetching from GitHub.

Request C arrives 10ms later. Same thing. Third GitHub API call for identical data.

All three requests check the cache, all three see a miss, all three independently hit GitHub. The first one to finish populates the cache. But by then, the other two have already sent their requests.

The cache only helps after a response is complete. While the first request is still in-flight, every concurrent request for the same key goes straight to GitHub.

Why This Actually Matters

For a badge that's embedded in one README and gets 50 views a day, this is invisible. Fine.

But CommitPulse serves public badges. A popular README can get thousands of views in a short burst — a Hacker News post, a GitHub trending spike, a deploy cold start where the cache is empty. In those moments, concurrent requests for the same username fire simultaneously, and without in-flight deduplication, every one of them hits GitHub independently.

GitHub's GraphQL API has rate limits. Authenticated requests get 5,000 points per hour. A complex contribution query costs points. During a cold start with high traffic, this amplifies GitHub API usage proportionally to the number of concurrent requests — not proportionally to the number of unique usernames.

This is the thundering herd problem. A cache miss under load triggers a flood of identical upstream calls, each one racing to populate the cache with the same data.

The Fix: In-Flight Request Deduplication

The solution is to track pending requests separately from completed cache entries.

When a cold miss happens and a fetch starts, store the pending Promise in a Map keyed the same way as the cache. Any concurrent request that misses the cache checks the in-flight map next — if there's already a pending request for that key, return the same Promise instead of starting a new one.

Once the request settles (success or failure), remove it from the in-flight map.

Here's the implementation for the contributions fetch path:

// lib/github.ts — after PR #1589

// in-flight request maps, one per data type
const contributionsInFlight = new Map<string, Promise<GitHubContributions>>()
const profileInFlight = new Map<string, Promise<GitHubProfile>>()
const reposInFlight = new Map<string, Promise<GitHubRepo[]>>()

async function fetchGitHubContributions(
  username: string,
  year: number,
  bypassCache = false
): Promise<GitHubContributions> {
  const key = `${username}:${year}`

  // 1. check completed cache first
  if (!bypassCache) {
    const cached = await contributionsCache.get(key)
    if (cached) return cached
  }

  // 2. check in-flight map — return existing promise if one is pending
  if (!bypassCache && contributionsInFlight.has(key)) {
    return contributionsInFlight.get(key)!
  }

  // 3. cold miss — start the fetch, register it in the in-flight map
  const request = fetchWithRetry(GITHUB_GRAPHQL_URL, options)
    .then(async (res) => {
      const result = await processResponse(res)
      contributionsCache.set(key, result)
      return result
    })
    .finally(() => {
      contributionsInFlight.delete(key)  // clean up after settling
    })

  contributionsInFlight.set(key, request)
  return request
}

Three things happening here:

First, check the completed cache (unchanged from before). Second, check the in-flight map — if a request is already pending for this key, return that same Promise. All concurrent callers now share one pending request. Third, on a genuine cold miss, create the fetch Promise, register it in the in-flight map, and remove it in finally once it settles.

The bypassCache / refresh=true path intentionally skips both the cache and the in-flight map. An explicit refresh should always be a real fresh request — that's the whole point of a force refresh.

The Repro Test

The clearest way to verify this class of bug is to delay the mock fetch slightly and fire concurrent requests:

// before the fix — this test would fail (fetch called 3 times)
it('deduplicates concurrent cold cache misses', async () => {
  let fetchCount = 0

  vi.mocked(fetch).mockImplementation(async () => {
    fetchCount++
    await new Promise(r => setTimeout(r, 50))  // simulate network delay
    return mockGraphQLResponse()
  })

  // fire three concurrent requests for the same username
  await Promise.all([
    fetchGitHubContributions('octocat', 2026),
    fetchGitHubContributions('octocat', 2026),
    fetchGitHubContributions('octocat', 2026),
  ])

  // should only have hit GitHub once
  expect(fetchCount).toBe(1)
})

Without deduplication: fetchCount is 3. All three concurrent requests miss the cache before any of them finish.

With deduplication: fetchCount is 1. Requests B and C see the pending Promise from Request A and return it directly. GitHub gets one call.

I added this regression test for all three data paths — contributions, profile, and repos — so the behavior is pinned going forward.

The Broader Pattern

This is a specific instance of a pattern called request coalescing — combining multiple identical pending requests into one.

It comes up whenever you have:

A cache that only holds completed responses
High concurrency or bursty traffic
An upstream resource with rate limits or cost (GitHub API, a database, a paid third-party service)

The cache alone isn't enough. You need deduplication at the in-flight level too.

The in-flight Map approach is the standard fix. It's small — a Map per data type, a few lines of check-and-register logic — and it slots cleanly into existing cache code without changing the API surface or the completed-cache behavior.

If you're building anything with a cache layer that sits in front of a rate-limited API — check whether your in-flight requests are deduplicated. If they're not, a cold start under load will hit your upstream harder than you expect.

While I Was There — Three More Bugs

Since I was deep in lib/github.ts already, I kept reading. Found three more things worth fixing in the same file:

PR #855 — getHeaders() always built an Authorization: bearer ${process.env.GITHUB_PAT || process.env.GITHUB_TOKEN} header. When neither variable was set, outgoing requests got Authorization: bearer undefined. Not a valid credential, and the only feedback was a confusing 401 from GitHub. Fixed it to resolve the token first, validate it, and throw a clear config error before making any request.

PR #1131 — fetchWithRetry added an abort event listener to the caller's AbortSignal on every attempt but never removed it after the request settled. On successful requests, retries, and network failures, stale listener references accumulated on long-lived signals. Fixed it with a named callback and a finally cleanup.

PR #1244 — GitHub REST helpers were inserting raw usernames directly into URL paths: `/users/${username}`. A username containing /, ?, or # reshapes the URL instead of being treated as one path segment. Fixed it with encodeURIComponent(username) before building REST URLs, with regression tests asserting / becomes %2F.

Four bugs. One file. Same pattern across all of them — code that works fine for the normal happy path but quietly misbehaves under edge cases or load.

DEV Community