Don't Burn Free Tokens Twice: Cache Streaming Chat Responses in a Service Worker

Two weeks ago, I watched a demo chat app consume nearly six million tokens in a single afternoon without producing a single new answer. Testers were refreshing the same form, re-clicking the same prompt, and triggering the same streaming call again and again. The server responded with 200 every time, so nobody noticed until the quota chart started bending in the wrong direction. That failure made me ask why an identical streaming prompt should ever reach a model twice.

The symptom: 200 responses, zero new information

The network tab showed several identical POST requests to /api/chat, each carrying the same JSON body and the same headers. Every response started from scratch, emitted the same token sequence, and held the connection open for roughly the same duration. The frontend treated every response as fresh because the status was always 200, and the user never saw a difference between a repeated request and a genuinely new one. In a free tier with a large but finite token pool, that behavior is expensive.

The fix: cache the completed stream, not the prompt text

A service worker can intercept those repeated POST requests and return the previously completed response before the request ever leaves the browser. This does not change the model call when the prompt is new; it only short-circuits exact repeats. The key is to include the full request body in the cache fingerprint, not just the pathname, because two prompts can share a URL but have very different payloads.

// sw.js
const CACHE_NAME = 'chat-cache-v1';
const MAX_CACHE_ENTRIES = 40;
const CACHE_ROOT = '/__chat_cache/';

self.addEventListener('install', () => self.skipWaiting());

self.addEventListener('fetch', (event) => {
  const request = event.request;
  const url = new URL(request.url);
  if (request.method !== 'POST' || !url.pathname.endsWith('/chat')) return;
  event.respondWith(handleChatRequest(event));
});

async function handleChatRequest(event) {
  const request = event.request;
  const key = await cacheKeyFor(request);
  const cache = await caches.open(CACHE_NAME);

  const cached = await cache.match(key);
  if (cached) {
    const headers = new Headers(cached.headers);
    headers.set('X-Chat-Cache', 'HIT');
    return new Response(cached.body, {
      status: cached.status,
      statusText: cached.statusText,
      headers,
    });
  }

  const response = await fetch(request);
  if (!response.ok || response.status === 206) return response;

  const headers = new Headers(response.headers);
  headers.set('X-Chat-Cache', 'MISS');
  const cacheable = new Response(response.body, {
    status: response.status,
    statusText: response.statusText,
    headers,
  });

  event.waitUntil((async () => {
    const target = await caches.open(CACHE_NAME);
    await target.put(key, cacheable.clone());
    await trimCache(target);
  })());

  return cacheable;
}

async function cacheKeyFor(request) {
  const body = await request.clone().text();
  const hash = await sha256(request.url + '|' + body);
  return new URL(CACHE_ROOT + hash, self.location.origin).href;
}

async function sha256(text) {
  const bytes = new TextEncoder().encode(text);
  const digest = await crypto.subtle.digest('SHA-256', bytes);
  return [...new Uint8Array(digest)]
    .map((byte) => byte.toString(16).padStart(2, '0'))
    .join('');
}

async function trimCache(cache) {
  const keys = await cache.keys();
  while (keys.length > MAX_CACHE_ENTRIES) {
    await cache.delete(keys.shift());
  }
}

Why a cloned Response body can stream and cache at the same time

response.body is a ReadableStream. Creating a new Response with it and then calling cacheable.clone() tees the stream, so one branch goes to the client while the browser drains the other into Cache Storage. The client still receives chunks as they arrive; the cache only fills in the background. That means the cached hit on the next identical request is not a synthetic preview, but the same completed stream the first caller received.

Fingerprint the body, or you will cache the wrong conversation

• Hash request.url plus the full JSON body to avoid collisions between different prompts on the same endpoint.
• Include only content-affecting headers such as accept and content-type; do not put volatile authentication headers into the cache key.
• Use crypto.subtle.digest because it returns a cryptographic fingerprint without shipping a third-party hash library.

Accessibility boundary: an instant cache hit must not disappear from the screen reader

A cached response arrives so fast that a loading announcement can be missed. If the UI fires status.textContent = 'Streaming...' and then immediately replaces it with the answer, a screen reader may never speak the transition. So I set the status region explicitly based on the X-Chat-Cache response header.

const response = await fetch('/api/chat', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify(message),
});

const cacheState = response.headers.get('X-Chat-Cache');
const status = document.getElementById('stream-status');
status.textContent = cacheState === 'HIT'
  ? 'Cached response loaded'
  : 'Streaming new response';
status.setAttribute('role', 'status');

That tiny status update turns an invisible performance trick into a state change that keyboard and screen-reader users can hear. Without it, a cached answer can look like the page just jumped, which is exactly the kind of silent state change accessibility audits keep catching.

Where MonkeyCode fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I built the demo as a static application, and I needed two things: a hosting target that stayed free during testing and a model quota large enough to expose repetition costs. MonkeyCode currently lists 30 million free tokens and a free server tier in its project materials. I used the free server to host the page and service worker, then pointed the demo at an upstream endpoint through an environment variable. The worker never cares whose API is behind the URL.

# .env.example
AI_UPSTREAM_URL=https://your-model-provider.example/v1/chat
NODE_ENV=production

Before you rely on any free tier for production, re-check the project page for current quota and server details. Free availability changes, and a debugging demo should not become a hard dependency on something you have not verified today.

Limits and who should not use this

• First-time prompts still consume tokens, so this helps only when users repeat identical requests.
• Cache Storage is not infinite; service worker storage is usually capped at a percentage of disk space, so trim aggressively.
• Do not cache answers that contain personal data, account information, or anything controlled by Cache-Control: no-store.
• Long streams increase first-call latency before the cache entry is available; the client still sees the original stream, but the cache is not usable until the response completes.
• The fingerprint hash adds a tiny CPU cost on each request, usually negligible next to the model call.

Test plan

1. Run the demo locally with npx serve . and confirm the service worker registers in Chrome DevTools under Application > Service Workers.
2. Open the Network panel, submit one prompt, and note the model request and X-Chat-Cache: MISS in the response.
3. Submit the identical prompt again, and confirm the network entry is served from the service worker with X-Chat-Cache: HIT; no new model request should appear.
4. Open Application > Cache Storage > chat-cache-v1 and verify one cached response.
5. From the console, run caches.delete('chat-cache-v1') to clear the cache and repeat the request to verify the model is called again.
6. Test with VoiceOver on macOS and NVDA on Windows: after a cached hit, the status region should announce Cached response loaded.

Free model access is never free from constraints; it is free within a budget. If your users can repeat a prompt and your frontend re-bills that prompt every time, the budget shrinks without producing anything new. A service worker cache, when paired with an explicit status announcement and aggressive trimming, stops charging twice for the same interaction. If you want to try the pattern against a generous free tier, deploy the demo on MonkeyCode's free server, watch the cache-hit counter, and see which repeated prompt you catch first.