DEV Community: babycat

Streaming Tokens Need a Frame Budget: Keeping Chat Input Responsive at 60 FPS

babycat — Thu, 20 Aug 2026 01:00:57 +0000

I asked a free model to write a 2,000-word essay inside a streaming chat widget, and the input box froze so badly that my cursor stopped blinking. The network tab showed a healthy connection; the model was generating tokens faster than my renderer could paint them, and the whole interface paid the price.

That experiment ran on MonkeyCode, an open-source AI assistant that offers free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free tier made the test trivial to set up, but the performance problem it exposed is universal: every streaming token is a DOM write, and DOM writes are not free.

This post is a rendering retrospective, not a network debugging story. I measured the frame drops, found three bottlenecks, fixed them in order, and ended up with a chat interface that stays responsive even when the model is outputting a hundred tokens per second.

The 12 FPS Symptom

The first sign of trouble was the caret. I typed a follow-up message while the model was still streaming, and the caret moved in visible jumps instead of a smooth glide. Then the input started dropping keystrokes entirely, and I knew the main thread was saturated.

I attached a requestAnimationFrame loop to measure frame times, and the numbers confirmed the diagnosis. The median frame time during streaming was 83 milliseconds, which works out to roughly 12 frames per second. A healthy interface needs 16.7 milliseconds per frame to hit 60 FPS; I was five times over budget.

The key measurement was key-to-render latency: the time between a keystroke and the character appearing on screen. At rest, it was 8 milliseconds. During streaming, it ballooned to 240 milliseconds, which is the difference between typing feeling instant and typing feeling like remote desktop software.

Bottleneck 1: One DOM Write per Token

The naive render loop appends each token as it arrives, and each appendChild invalidates the layout of the entire chat container.

// The naive approach: one DOM write per token
async function naiveRender(tokens) {
  for (const token of tokens) {
    const span = document.createElement("span");
    span.textContent = token;
    container.appendChild(span);
    // Each appendChild forces a synchronous layout
  }
}

This looks harmless, but it has a quadratic cost. Every new token forces the browser to recalculate the layout of every previous token in the container, and as the conversation grows, each append gets more expensive. A 500-token response is not 500 layout calculations; it is 125,000 of them.

Fix 1: Batch With requestAnimationFrame

The first fix is almost embarrassingly simple: accumulate tokens in a buffer, then flush them all in a single DOM write on the next animation frame.

// Batched approach: one DOM write per frame
let tokenBuffer = [];
let rafId = null;

function scheduleFlush() {
  if (rafId) return;
  rafId = requestAnimationFrame(() => {
    const fragment = document.createDocumentFragment();
    for (const token of tokenBuffer) {
      const span = document.createElement("span");
      span.textContent = token;
      fragment.appendChild(span);
    }
    container.appendChild(fragment);
    tokenBuffer = [];
    rafId = null;
  });
}

function renderToken(token) {
  tokenBuffer.push(token);
  scheduleFlush();
}

The median frame time dropped from 83 milliseconds to 21 milliseconds, and key-to-render latency fell to 45 milliseconds. The browser was going to paint all those DOM changes on the next frame anyway; batching them just eliminated the redundant layout calculations in between.

Bottleneck 2: The Chat Log Grows Without Bound

Batching fixed the per-token cost, but the chat log itself kept growing. Every new token appended to the container made the next layout calculation more expensive, and after a few thousand tokens, even batched appends started to hurt.

The fix was virtualization: render only the tokens that are actually visible in the viewport, and recycle DOM nodes as the user scrolls.

// Virtualized chat log: render only visible tokens
class VirtualizedChatLog {
  constructor(viewport, rowHeight = 20) {
    this.viewport = viewport;
    this.rowHeight = rowHeight;
    this.tokens = [];
    this.container = document.createElement("div");
    this.container.style.position = "relative";
    viewport.appendChild(this.container);
  }

  appendToken(token) {
    this.tokens.push(token);
    this.render();
  }

  render() {
    const scrollTop = this.viewport.scrollTop;
    const height = this.viewport.clientHeight;
    const start = Math.floor(scrollTop / this.rowHeight);
    const end = Math.ceil((scrollTop + height) / this.rowHeight);

    const fragment = document.createDocumentFragment();
    for (let i = start; i < Math.min(end, this.tokens.length); i++) {
      const div = document.createElement("div");
      div.textContent = this.tokens[i];
      div.style.position = "absolute";
      div.style.top = `${i * this.rowHeight}px`;
      div.style.left = "0";
      div.style.right = "0";
      fragment.appendChild(div);
    }

    this.container.innerHTML = "";
    this.container.appendChild(fragment);
    this.container.style.height = `${this.tokens.length * this.rowHeight}px`;
  }
}

Virtualization keeps the DOM size proportional to the viewport, not the conversation length. A 5,000-token conversation renders the same number of nodes as a 50-token one, and the frame time stays flat.

The Measured Results

Here is the before-and-after table from the same 2,000-word generation:

Strategy	Median frame time	Key-to-render latency	DOM nodes at 5,000 tokens
One DOM write per token	83 ms	240 ms	5,000+
rAF batching	21 ms	45 ms	5,000+
Batching + virtualization	16 ms	16 ms	~40

The combination of batching and virtualization brought the interface from 12 FPS to a solid 60 FPS, and typing during generation now feels the same as typing at rest.

The Accessibility Angle: Live Regions Need Throttling

Batching fixes the renderer, but it creates a new problem for screen reader users. If every token triggers a live region update, NVDA or VoiceOver will try to announce a stream of word fragments, which is useless and exhausting.

The fix is to throttle live region updates to a fixed interval, separate from the render batching:

// Throttled live region: announce progress, not tokens
const liveRegion = document.getElementById("chat-status");
let lastAnnouncement = 0;

function announceProgress() {
  const now = performance.now();
  if (now - lastAnnouncement < 500) return;
  liveRegion.textContent = `Generated ${tokenCount} characters so far`;
  lastAnnouncement = now;
}

This gives screen reader users a progress update every 500 milliseconds instead of a token-by-token narration, which is both informative and tolerable. The final completion announcement still fires exactly once, when the stream ends.

Limitations and Who Should Skip This

The rAF batching strategy assumes the model generates tokens at a steady pace. If your model emits in bursts—50 tokens at once, then a 2-second pause—the batching may add artificial latency between bursts, and a timer-based flush (every 100 milliseconds) would feel snappier.

Virtualization is overkill for short conversations. If your chat interface rarely exceeds a few hundred tokens, batching alone will get you to 60 FPS, and the complexity of virtual scrolling is not worth it. It also adds complexity around focus management and text selection, which can break keyboard navigation if implemented carelessly.

This post is about rendering performance, not network recovery. If your stream drops mid-sentence, no amount of batching will save you; you need a resume protocol, which is a separate problem entirely.

The Frame Budget Is the Contract

A streaming chat interface makes a promise to the user: you can read the response as it arrives, and you can keep typing while it does. That promise is broken the moment the main thread saturates, and the fix is not a faster model or a bigger server—it is a frame budget on the client.

The next time you build a streaming interface, measure the frame time before you measure the network latency. The model will always be faster than the renderer eventually, and the renderer is the side you can actually control.

If you want to reproduce this experiment, point any streaming chat widget at MonkeyCode's free model endpoint and watch the performance panel while a long response streams in. The free server and free model access make the test cost nothing, and the frame meter will tell you exactly where your renderer stands.

localhost Was Fine, Production Wasn't: A CORS Debugging Retrospective for a Deployed AI Chat Widget

babycat — Wed, 19 Aug 2026 18:44:47 +0000

Last week I built a small streaming chat widget against MonkeyCode's free model tier, tested it locally until the responses felt instant, and deployed the static build to the free server option. The first browser test on the deployed URL failed before a single token arrived, and the console showed a CORS error I had never seen during development. The same request from curl streamed perfectly, which made the browser failure feel personal.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

This is a debugging retrospective, not a product review, because the bug was entirely mine. I assumed localhost and production were the same environment, and they were not. The fix took about twenty minutes once I stopped blaming the model and started reading the preflight request.

The symptom: a frozen chat with a 200 in the network tab

The widget looked fine: input field, send button, streaming output area. I clicked send on the deployed URL, and nothing appeared. DevTools showed the POST to the model API returning 200, yet the browser refused to hand the response to my JavaScript. The console said the request had been blocked by CORS policy, and my code sat in the streaming state forever because I only handled HTTP errors, not fetch rejections.

Why did curl work? Because curl does not enforce CORS; the browser does. Think of CORS as the browser's bouncer: the server may have answered the request, but the browser checks the guest list before handing the response to your JavaScript. That is the first lesson of this retrospective: a successful server response is not the same thing as a response the browser will let you read. My widget had two states, streaming and done, and the missing third state, error, is exactly where the bug lived.

The preflight I forgot

My fetch sent an Authorization header with the model API key, which makes the request non-simple and triggers a preflight. The browser first sent an OPTIONS request asking permission:

OPTIONS /v1/chat
Origin: https://my-widget.example.com
Access-Control-Request-Method: POST
Access-Control-Request-Headers: authorization, content-type

The server answered the OPTIONS with a status the browser could accept, but the response carried no Access-Control-Allow-Origin header. That single missing header is enough for the browser to abort the whole exchange, even though the actual POST would have succeeded. The network tab showed both requests, but I had only been looking at the POST.

The painful part was that localhost worked perfectly. Why? Because I had configured a Vite dev proxy, which made every request same-origin and skipped CORS entirely. The deployed static site had no proxy, so the browser talked directly to the model API and hit the wall. The environment matrix explains the whole story:

Environment	Request path	Result
localhost + Vite proxy	same-origin `/api/chat`	works
deployed static site	cross-origin model API	blocked by CORS
deployed site + server proxy	same-origin `/api/chat`	works
`curl`	direct model API	works, no CORS enforced

The fix: a same-origin proxy that streams

I considered three options and picked the one that did not depend on the model provider changing anything:

Option	Works?	Notes
Ask the provider to add CORS headers	Not guaranteed	Free tiers rarely expose CORS configuration
Remove the `Authorization` header	No	The endpoint requires a key
Proxy through the same origin that serves the static files	Yes	Full control, key stays server-side

The proxy is a small Node server that serves the static build and forwards /api/chat to the model API. The browser only ever talks to its own origin, so CORS disappears:

// server.js — static files + streaming proxy on the same origin
import http from "node:http";
import { readFile } from "node:fs/promises";

const MODEL_URL = "https://model-api.example.com/v1/chat";

http
  .createServer(async (req, res) => {
    if (req.url.startsWith("/api/chat") && req.method === "POST") {
      let body = "";
      for await (const chunk of req) body += chunk;

      const upstream = await fetch(MODEL_URL, {
        method: "POST",
        headers: {
          "Content-Type": "application/json",
          Authorization: `Bearer ${process.env.API_KEY}`,
        },
        body,
      });

      res.writeHead(upstream.status, {
        "Content-Type": upstream.headers.get("content-type") ?? "application/json",
      });

      for await (const chunk of upstream.body) res.write(chunk);
      res.end();
      return;
    }

    const file = req.url === "/" ? "index.html" : req.url.slice(1);
    try {
      const data = await readFile(`dist/${file}`);
      res.writeHead(200, { "Content-Type": "text/html" });
      res.end(data);
    } catch {
      res.writeHead(404);
      res.end("Not found");
    }
  })
  .listen(process.env.PORT || 3000);

The client change is even smaller: point the fetch at the relative URL and drop the key from the browser entirely.

const res = await fetch("/api/chat", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ messages }),
});

A side benefit I did not expect: the API key stopped leaking into the browser bundle, which the old code was doing without me noticing. That alone was worth the refactor, because a leaked key in a public bundle is a credential incident waiting for a scraper.

Reusable debugging techniques

The specific failure was CORS, but the process transfers to any environment difference between localhost and production:

Reproduce with the same client. curl proves the server works, not that the browser can read the response. Test with the exact headers your app sends.
Read the preflight, not just the POST. The OPTIONS request and its response headers tell you exactly what the browser asked for and what the server refused.
Enumerate environment differences. Proxy, origin, TLS, base URL, headers — write them down before you start guessing.
Add a CORS smoke test to your QA matrix. A tiny page that fetches with the same headers catches this in minutes.
Make the UI surface the failure. A silent CORS block leaves users staring at a frozen chat, so treat fetch rejections as first-class states.

The accessibility angle you cannot skip

A blocked request is invisible to the user until the UI says something, so my widget needed a real error state with a clear announcement and a way forward. The state machine should have at least four states, and each one needs a visible UI and an announcement:

State	UI	Announcement
`idle`	input enabled, button ready	—
`streaming`	partial text, stop button	"Response started"
`error`	alert box, retry button	"Request failed"
`cancelled`	input enabled	"Stopped"

{error && (
  <div role="alert" className="chat-error">
    <p>The request failed: {error.message}</p>
    <button type="button" onClick={retry}>Retry</button>
  </div>
)}

When the fetch rejects, move focus to the retry button, announce the failure through the alert region, and keep the input usable so the user can edit the prompt. If your UI cannot represent error, it will represent it as a frozen streaming, which is exactly what happened to me.

Limitations and who should not use this approach

This fix assumes you control the server that serves the static files, so if your host only serves static assets and cannot run a proxy, you need a separate proxy host or a provider that sends the right CORS headers. Check whether the model API already allows your origin before building anything, because if it does, the proxy is unnecessary complexity. And remember that a proxy is not a security boundary by itself: it keeps the key out of the browser, but you still need proper auth on the server side.

If you want to reproduce this failure cheaply, MonkeyCode is open source, and its free model tier (currently a 10M-token allowance) plus free server option give you a place to deploy the same widget and watch the preflight yourself. The bug is the point: it is a five-minute fix once you actually see the OPTIONS request, and now you know where to look.

A 30M-Token Free Tier Needs a Token Ledger, Not Just a Budget Bar

babycat — Wed, 19 Aug 2026 00:29:48 +0000

Last week I linked a source-review agent to a free model tier and watched two numbers disagree: the client budget bar said 9.2M tokens remained, the provider dashboard said 24.7M, and the server still returned 429 budget_exceeded on the first retry. I spent the first hour blaming a quota throttle, because free tiers are supposed to break in exactly that way. The real defect was much closer to home. A streaming response was double-counted in my token ledger, and the budget bar was narrating a lie to screen-reader users.

The lesson from that afternoon is not that free AI is unreliable. It is that a large free grant makes accounting bugs invisible for much longer. A 30M-token pool will happily consume millions of tokens before a client-side double count crosses the failure boundary. By then the bug has touched every retry, every cache entry, and every accessibility announcement. I needed a token ledger, not another progress bar.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's open-source project with its free model access and the free server path described in the operator materials. Those materials list a 30M-token pool and a no-cost server option; I treated those claims as ceilings to verify, not as product guarantees.

Start with the symptom table, not the provider

The conversation this week kept repeating a pattern. A request completed with HTTP 200, the reader finished, and the UI announced 79% of token budget remaining. Then a retry for the same prompt failed with budget_exceeded. The server's final usage said total tokens were 4.1M, while my ledger said 2.4M. Nothing about that failure pointed to a bug in the chunk reader except the shape of the numbers.

Source	Token count	What it claimed
Client ledger	2,410,388	9.2M remaining
Server usage	4,108,942	actual total
Dashboard after retry	26,115,400	quota remaining before failure

The gap between client and server was roughly the size of the cached prompt tokens plus one completion pass. That clue meant my retry path was counting prompt tokens twice, while the streaming path was counting completion tokens using character length instead of tokenizer output. I should have stopped trusting the client and started instrumenting the wire.

Build a replay harness before you touch the live UI

My first mistake was inspecting the production code in place. The second was assuming a free server would behave like a hosted endpoint with consistent final usage. When a serverless or free server path truncates the SSE stream, the last data: chunk can be missing. You then recalculate the budget from an estimate, and the drift becomes permanent. That is exactly when a free grant hides the failure, because the next 20M tokens run on top of the corrupted number.

A small replay harness makes the accounting observable. It replays the same JSON request body, reads the SSE stream, and records only the final usage when the provider sends one.

async function replayConversation(requestBody, endpoint, key) {
  const controller = new AbortController();
  const timeout = setTimeout(() => controller.abort(), 30_000);

  const response = await fetch(endpoint, {
    method: 'POST',
    headers: {
      'content-type': 'application/json',
      authorization: `Bearer ${key}`,
    },
    body: JSON.stringify(requestBody),
    signal: controller.signal,
  });

  if (!response.ok) {
    throw new Error(`HTTP ${response.status}: ${await response.text()}`);
  }

  const reader = response.body.getReader();
  const decoder = new TextDecoder();
  let buffer = '';
  let finalUsage = null;
  let completionChars = 0;

  while (true) {
    const { value, done } = await reader.read();
    if (done) break;
    buffer += decoder.decode(value, { stream: true });

    let boundary = buffer.indexOf('\n\n');
    while (boundary !== -1) {
      const eventText = buffer.slice(0, boundary);
      buffer = buffer.slice(boundary + 2);

      for (const line of eventText.split('\n')) {
        if (!line.startsWith('data:')) continue;
        const json = line.slice(5).trim();
        if (!json || json === '[DONE]') continue;

        const chunk = JSON.parse(json);
        const delta = chunk.choices?.[0]?.delta?.content ?? '';
        completionChars += delta.length; // not a token count
        if (chunk.usage) finalUsage = chunk.usage; // authoritative when present
      }

      boundary = buffer.indexOf('\n\n');
    }
  }

  clearTimeout(timeout);
  return { completionChars, finalUsage };
}

Notice that completionChars is deliberately not used as the billing number. Character length and token length differ, so using delta.length to update a token budget creates drift that grows with every streamed response. The harness records it only as a cross-check.

Reconcile the ledger from one source of truth

The fix in the real UI came down to a state machine that accepts exactly one authoritative input: the final usage object. If the final usage is missing, the ledger is marked unverified, and the UI stops making replacement claims until a verified value arrives.

function getNextLedger({ previousLedger, finalUsage, failedAttempts }) {
  if (finalUsage && Number.isFinite(finalUsage.total_tokens)) {
    return {
      status: 'verified',
      used: previousLedger.used + finalUsage.total_tokens,
      source: 'server-usage',
    };
  }

  return {
    status: 'unverified',
    used: previousLedger.used,
    source: failedAttempts > 0 ? 'retry-pending' : 'stream-truncated',
    failedAttempts: failedAttempts + 1,
  };
}

This changed the behavior in three places that users actually notice. A failed retry no longer says budget exceeded with a confident number. The budget bar no longer decreases from an estimate, because an estimate is never allowed to replace the server number. The screen-reader announcement now says token usage is unverified after a truncated response instead of reading a false percentage.

Make the budget boundary accessible, not just visible

The original budget bar used aria-valuenow with a number that was already wrong. When the server returned a corrected usage on the next request, the accessible name changed from 79% remaining to 43% remaining with no explanation. A user listening to that announcement assumes the model consumed 36% in one request. What actually happened is that the earlier number was fabricated.

A better pattern is to announce the transition, not the raw refresh. I added a visually hidden live region with aria-live='polite' and updated it only when the ledger status changed.

<p class='sr-only' aria-live='polite'>
  Token budget status: <span id='budget-status'>verified</span>.
</p>

The visible meter can keep its role, but it should render as a progress bar only when the status is verified. In the unverified state it becomes plain text: Token usage is still being reconciled. If the next stream ends with a final usage, the screen reader then hears one clear transition from unverified to verified, not two competing numbers.

Where the free server made this harder

A free server option changes the failure surface. A cold start can delay the first token, a proxy can drop the final usage chunk, and a retry can consume another prompt pass before the UI knows the first attempt failed. Each of those conditions makes client-side token estimation more tempting, because the server number is absent right when the user is watching the budget bar. A token ledger that refuses to estimate is boring, but it remains honest when the network is not.

The operator-supplied MonkeyCode materials still describe a 30M-token pool and a free server option. I did not need that whole pool to find the bug; the defect appeared after a few million tokens because my retry path was broken, not because the quota was small. If you use a free model access path, capture the final usage before you render any remaining-balance claim.

Who should not copy this approach

Do not use a client token ledger for billing, quotas, or anything involving money or hard limits. Tokenizer equivalence is provider-specific, and a prompt that is cached can count differently than the same prompt after a cache miss. If your free server path can be restarted mid-stream, the final usage may belong to a partial generation. Treat the ledger as a UI reconciliation tool, not as a source of record.

The working rule I settled on is short: when a number can be absent, mark the state unverified; when a number exists, show the server value; when a retry masks the absence, block the next estimate. A free 30M-token grant makes arithmetic errors cheap to hide and expensive to debug. The harness is what makes them cheap to find again.

If you try the MonkeyCode free endpoint or free server path, run the replay harness before you publish a budget bar. The product is the open-source project and the no-cost server; the debugging lesson is that the next quota failure may be your own token math, not the quota.

Don't Burn Free Tokens Twice: Cache Streaming Chat Responses in a Service Worker

babycat — Tue, 18 Aug 2026 18:23:15 +0000

Two weeks ago, I watched a demo chat app consume nearly six million tokens in a single afternoon without producing a single new answer. Testers were refreshing the same form, re-clicking the same prompt, and triggering the same streaming call again and again. The server responded with 200 every time, so nobody noticed until the quota chart started bending in the wrong direction. That failure made me ask why an identical streaming prompt should ever reach a model twice.

The symptom: 200 responses, zero new information

The network tab showed several identical POST requests to /api/chat, each carrying the same JSON body and the same headers. Every response started from scratch, emitted the same token sequence, and held the connection open for roughly the same duration. The frontend treated every response as fresh because the status was always 200, and the user never saw a difference between a repeated request and a genuinely new one. In a free tier with a large but finite token pool, that behavior is expensive.

The fix: cache the completed stream, not the prompt text

A service worker can intercept those repeated POST requests and return the previously completed response before the request ever leaves the browser. This does not change the model call when the prompt is new; it only short-circuits exact repeats. The key is to include the full request body in the cache fingerprint, not just the pathname, because two prompts can share a URL but have very different payloads.

// sw.js
const CACHE_NAME = 'chat-cache-v1';
const MAX_CACHE_ENTRIES = 40;
const CACHE_ROOT = '/__chat_cache/';

self.addEventListener('install', () => self.skipWaiting());

self.addEventListener('fetch', (event) => {
  const request = event.request;
  const url = new URL(request.url);
  if (request.method !== 'POST' || !url.pathname.endsWith('/chat')) return;
  event.respondWith(handleChatRequest(event));
});

async function handleChatRequest(event) {
  const request = event.request;
  const key = await cacheKeyFor(request);
  const cache = await caches.open(CACHE_NAME);

  const cached = await cache.match(key);
  if (cached) {
    const headers = new Headers(cached.headers);
    headers.set('X-Chat-Cache', 'HIT');
    return new Response(cached.body, {
      status: cached.status,
      statusText: cached.statusText,
      headers,
    });
  }

  const response = await fetch(request);
  if (!response.ok || response.status === 206) return response;

  const headers = new Headers(response.headers);
  headers.set('X-Chat-Cache', 'MISS');
  const cacheable = new Response(response.body, {
    status: response.status,
    statusText: response.statusText,
    headers,
  });

  event.waitUntil((async () => {
    const target = await caches.open(CACHE_NAME);
    await target.put(key, cacheable.clone());
    await trimCache(target);
  })());

  return cacheable;
}

async function cacheKeyFor(request) {
  const body = await request.clone().text();
  const hash = await sha256(request.url + '|' + body);
  return new URL(CACHE_ROOT + hash, self.location.origin).href;
}

async function sha256(text) {
  const bytes = new TextEncoder().encode(text);
  const digest = await crypto.subtle.digest('SHA-256', bytes);
  return [...new Uint8Array(digest)]
    .map((byte) => byte.toString(16).padStart(2, '0'))
    .join('');
}

async function trimCache(cache) {
  const keys = await cache.keys();
  while (keys.length > MAX_CACHE_ENTRIES) {
    await cache.delete(keys.shift());
  }
}

Why a cloned Response body can stream and cache at the same time

response.body is a ReadableStream. Creating a new Response with it and then calling cacheable.clone() tees the stream, so one branch goes to the client while the browser drains the other into Cache Storage. The client still receives chunks as they arrive; the cache only fills in the background. That means the cached hit on the next identical request is not a synthetic preview, but the same completed stream the first caller received.

Fingerprint the body, or you will cache the wrong conversation

Hash request.url plus the full JSON body to avoid collisions between different prompts on the same endpoint.
Include only content-affecting headers such as accept and content-type; do not put volatile authentication headers into the cache key.
Use crypto.subtle.digest because it returns a cryptographic fingerprint without shipping a third-party hash library.

Accessibility boundary: an instant cache hit must not disappear from the screen reader

A cached response arrives so fast that a loading announcement can be missed. If the UI fires status.textContent = 'Streaming...' and then immediately replaces it with the answer, a screen reader may never speak the transition. So I set the status region explicitly based on the X-Chat-Cache response header.

const response = await fetch('/api/chat', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify(message),
});

const cacheState = response.headers.get('X-Chat-Cache');
const status = document.getElementById('stream-status');
status.textContent = cacheState === 'HIT'
  ? 'Cached response loaded'
  : 'Streaming new response';
status.setAttribute('role', 'status');

That tiny status update turns an invisible performance trick into a state change that keyboard and screen-reader users can hear. Without it, a cached answer can look like the page just jumped, which is exactly the kind of silent state change accessibility audits keep catching.

Where MonkeyCode fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I built the demo as a static application, and I needed two things: a hosting target that stayed free during testing and a model quota large enough to expose repetition costs. MonkeyCode currently lists 30 million free tokens and a free server tier in its project materials. I used the free server to host the page and service worker, then pointed the demo at an upstream endpoint through an environment variable. The worker never cares whose API is behind the URL.

# .env.example
AI_UPSTREAM_URL=https://your-model-provider.example/v1/chat
NODE_ENV=production

Before you rely on any free tier for production, re-check the project page for current quota and server details. Free availability changes, and a debugging demo should not become a hard dependency on something you have not verified today.

Limits and who should not use this

First-time prompts still consume tokens, so this helps only when users repeat identical requests.
Cache Storage is not infinite; service worker storage is usually capped at a percentage of disk space, so trim aggressively.
Do not cache answers that contain personal data, account information, or anything controlled by Cache-Control: no-store.
Long streams increase first-call latency before the cache entry is available; the client still sees the original stream, but the cache is not usable until the response completes.
The fingerprint hash adds a tiny CPU cost on each request, usually negligible next to the model call.

Test plan

Run the demo locally with npx serve . and confirm the service worker registers in Chrome DevTools under Application > Service Workers.
Open the Network panel, submit one prompt, and note the model request and X-Chat-Cache: MISS in the response.
Submit the identical prompt again, and confirm the network entry is served from the service worker with X-Chat-Cache: HIT; no new model request should appear.
Open Application > Cache Storage > chat-cache-v1 and verify one cached response.
From the console, run caches.delete('chat-cache-v1') to clear the cache and repeat the request to verify the model is called again.
Test with VoiceOver on macOS and NVDA on Windows: after a cached hit, the status region should announce Cached response loaded.

Free model access is never free from constraints; it is free within a budget. If your users can repeat a prompt and your frontend re-bills that prompt every time, the budget shrinks without producing anything new. A service worker cache, when paired with an explicit status announcement and aggressive trimming, stops charging twice for the same interaction. If you want to try the pattern against a generous free tier, deploy the demo on MonkeyCode's free server, watch the cache-hit counter, and see which repeated prompt you catch first.

A Free Model Returned Markdown That NVDA Read as One Endless Sentence: Normalizing LLM Output Before It Reaches the Live Region

babycat — Tue, 18 Aug 2026 12:14:18 +0000

Liquid syntax error: Unknown tag 'endraw'

The Network Tab Said 200, but the Streaming Chat Never Finished: A Response-Frame Debugging Retrospective

babycat — Mon, 17 Aug 2026 23:29:55 +0000

Some streaming failures announce themselves with a loud network error, but this one stayed quiet. The browser showed a 200, the server logs completed, and every unit test passed, yet the chat kept announcing that it was still thinking long after the last visible token arrived. The final token looked half-finished, and the assistive-tech status region never reached a completed state. That combination pointed away from the prompt and toward the way the stream was being terminated.

To reproduce the failure without spending a paid route, I pointed a small harness at MonkeyCode's free model access and ran the proxy on the free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free path mattered less for cost and more for control: I could inspect raw response frames without an SDK hiding the transport details.

Start with the raw frames, not the framework

Language model SDKs often smooth over streaming quirks, which is useful in production but dangerous when you are debugging a protocol-level failure. I removed the SDK and sent the same prompt with curl, then wrote the response body to a file so I could inspect the exact frame boundaries.

curl -N 'https://your-endpoint.example/v1/stream?prompt=Explain%20focus%20management%20in%20one%20paragraph.' -o stream.log
tail -c 300 stream.log | xxd

I expected the file to end with data: [DONE] followed by two newlines. Instead, the tail ended with data: "focus mov and then nothing: no closing quote, no terminal event, and no newline. The transport had closed exactly when the upstream stream was interrupted.

Separate the HTTP status from the stream contract

A 200 status only proves that the server sent headers and started a response. It does not prove that a chunked stream will end cleanly. A 200 is like a restaurant confirming your order before the kitchen loses power; the request was accepted, but the meal never arrived.

In this failure pattern, the endpoint or an intermediary proxy began sending chunked data, then the connection dropped before a terminal frame appeared. The browser exposed this as a normal close of a ReadableStream, so the client treated the response as finished. My UI then waited forever for a [DONE] frame that could never arrive.

What the browser showed	What the raw body showed	Likely cause
200 and a stuck spinner	last chunk ends mid-token, no terminal frame	upstream closed before sending `[DONE]`
200 and apparently complete text	terminal frame present but data split oddly	client parsed before a frame boundary arrived
failed request	connection reset	network or proxy refused the stream

Make the stream close the source of truth

The fix is to stop treating [DONE] as the only valid finish signal. The client should also emit a terminal state when the stream closes or goes idle, and it should preserve any partial chunk instead of throwing it away.

const decoder = new TextDecoder();
const CR = String.fromCharCode(13);
const LF = String.fromCharCode(10);
const CRLF = CR + LF;

function watchSseStream(response, { onEvent, onDone, idleMs = 12000 } = {}) {
  if (!response.body) {
    throw new Error('ReadableStream responses are required for this harness.');
  }

  const reader = response.body.getReader();
  let buffer = '';
  let idleTimer;
  let finished = false;

  function finish(reason) {
    if (finished) return;
    finished = true;
    clearTimeout(idleTimer);

    emitCompleteFrames();

    const leftover = buffer.trim();
    if (leftover) {
      onEvent({ data: leftover, partial: true });
    }

    onDone({ reason, hadPartialFrame: leftover.length > 0 });
  }

  function resetIdle() {
    clearTimeout(idleTimer);
    idleTimer = setTimeout(() => finish('idle-timeout'), idleMs);
  }

  function emitCompleteFrames() {
    const separator = LF + LF;

    while (buffer.includes(separator)) {
      const boundary = buffer.indexOf(separator);
      const frame = buffer.slice(0, boundary);
      buffer = buffer.slice(boundary + separator.length);

      const data = frame
        .split(LF)
        .filter((line) => line.startsWith('data:'))
        .map((line) => line.slice(5).trimStart())
        .join(LF);

      if (data && data !== '[DONE]') {
        onEvent({ data, partial: false });
      }
    }
  }

  function pump({ done, value }) {
    if (done) return finish('stream-closed');

    buffer += decoder.decode(value, { stream: true }).split(CRLF).join(LF);
    emitCompleteFrames();
    resetIdle();

    return reader.read().then(pump);
  }

  resetIdle();
  return reader.read().then(pump).catch(() => finish('stream-error'));
}

This snippet is a harness, not a complete production parser. It deliberately treats an orderly close or idle timeout as terminal, and it surfaces a partial final frame as real information rather than waiting for a separator that will never arrive.

Keep screen-reader announcements out of the token loop

The browser-side fix is not only about showing text. If every token writes directly into a live region, VoiceOver and NVDA can turn one response into dozens of interruptions. The status region should announce a stable state, not a stream.

<section id='chat-status' aria-live='polite' aria-atomic='true'>
  Waiting for response.
</section>

const status = document.getElementById('chat-status');
let lastAnnouncement = '';

function announce(state) {
  if (lastAnnouncement === state) return;
  lastAnnouncement = state;

  status.textContent = '';
  requestAnimationFrame(() => {
    status.textContent = state;
  });
}

watchSseStream(response, {
  onEvent({ data }) {
    appendToken(data);
  },
  onDone({ reason, hadPartialFrame }) {
    announce(
      hadPartialFrame
        ? 'Response stopped before completion.'
        : reason === 'idle-timeout'
          ? 'Response stopped after a long silence.'
          : 'Response complete.'
    );
  },
});

The screen reader now hears one outcome instead of every token. That distinction matters most when the stream fails, because the user needs to know that the response is incomplete and can retry.

Test the failure path, not just the success path

A happy-path test proves very little when the bug only appears after a dropped connection. I added a small matrix around the exact transition that failed.

Run curl -N before trusting any SDK or UI abstraction.
Throttle the connection to 50 ms latency and cut it off at about one third of the expected stream.
Confirm that the live region announces a single stable outcome in Chrome with NVDA and Safari with VoiceOver.
Capture the raw tail with tail -c 300 stream.log | xxd and record whether the final frame arrived.

Limits of this harness

It cannot restore tokens the server never sent. If the upstream model stops mid-thought, the client can only tell the user it stopped.
The idle timeout needs tuning. Set it too low and slow reasoning gets cut; set it too high and a dead connection still feels unresponsive.
A free server and shared model endpoint may restart or drop a long-lived session, so keep conversation state outside process memory.
This is not a replacement for official SDK retry behavior or durable server-side logging in regulated systems.

The harness is most useful when you can point it at a free endpoint and inspect the frames before adding UX on top. MonkeyCode's free model access and free server option makes that cheap to do, but the underlying lesson applies to any streaming provider: treat a 200 as the beginning of the stream contract, not the end.

A Free Token Pool Won't Reveal Dropped Fields; a Request Fingerprint Harness Will

babycat — Mon, 17 Aug 2026 04:12:47 +0000

The most expensive bug in a free model integration is not the token bill. It is the field that disappears between your frontend and the upstream provider while the stream still returns 200. You only notice it later, when the model ignores a tool schema, answers without the context you attached, or returns a final chunk that has no way to reconnect to the request that started it.

When I test a provider gateway, I prefer to use a low-stakes free tier so I can send a lot of small, deliberately malformed requests without treating every probe as a cost decision. MonkeyCode's current operator materials describe an open-source project with a free server option and a 30,000,000-token pool. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I treat those availability claims as things to recheck in the current documentation, not as a permanent quota, a model list, or a quality benchmark.

The failure usually starts in the middle of the hop. Your UI sends a tidy JSON body with messages, tools, and a nested metadata object. Your relay forwards that body through middleware, then to an upstream host. Somewhere in the chain a library keeps the fields it recognizes and silently omits the fields it does not. The upstream still streams a response, so both sides appear healthy. The relay behaves like a switchboard operator who passes the main call through but loses the second line; the conversation continues, and the missing party is only discovered much later.

What you see	What it usually means
A 200 stream that plays but ignores your tools	The relay or a body parser dropped the `tools` array before forwarding
A 200 stream with the right text but no session reference	The nested `metadata` object was renamed, omitted, or replaced by a default
A 400 from your relay with a blank body	The error handler caught the upstream reason and returned a generic response
A 502 after you press cancel	The proxy tied client disconnect to upstream cancellation while the UI never announced the new state

A successful HTTP status is not enough evidence that the payload stayed intact. The test you need is a fingerprint that describes the shape of the request without dumping tokens, keys, or user text into a log.

The following harness returns a deterministic list of paths and types, not values. It deliberately avoids logging strings longer than their length, because a free-tier debugging experiment should not become an accidental data leak.

export function shapeOf(value, path = '$', out = []) {
  if (Array.isArray(value)) {
    out.push(`${path}:array(${value.length})`);
    value.slice(0, 1).forEach((item, index) => shapeOf(item, `${path}[${index}]`, out));
  } else if (value && typeof value === 'object') {
    for (const [key, item] of Object.entries(value)) {
      const kind = typeof item;
      out.push(`${path}.${key}:${kind}${kind === 'string' ? `(${item.length})` : ''}`);
      if (kind === 'object') shapeOf(item, `${path}.${key}`, out);
    }
  } else {
    out.push(`${path}:${typeof value}`);
  }
  return out;
}

Run that function on the browser side before serialization, then again inside your relay after the body parser runs, and you get a diffable record of which nested objects survive the hop. A minimal pass-through relay is enough to expose the drop.

app.post('/chat', async (req, res) => {
  console.log('client->proxy', shapeOf(req.body));
  const controller = new AbortController();
  req.on('close', () => controller.abort());

  const upstream = await fetch(UPSTREAM_URL, {
    method: 'POST',
    headers: { 'content-type': 'application/json', authorization: `Bearer ${UPSTREAM_KEY}` },
    body: JSON.stringify(req.body),
    signal: controller.signal
  });

  res.status(upstream.status);
  for (const [key, value] of upstream.headers) {
    if (!['content-encoding', 'transfer-encoding'].includes(key.toLowerCase())) {
      res.setHeader(key, value);
    }
  }
  res.flushHeaders();

  for await (const chunk of upstream.body) {
    res.write(chunk);
  }
  res.end();
});

On the client, keep the announcement path separate from the data path. A live region should report the transition from connecting to streaming to error or cancelled, because a screen-reader user otherwise experiences the same silent loss as the dropped field but in the interface layer.

const controller = new AbortController();
const status = document.querySelector('[aria-live=polite]');

status.textContent = 'connecting';
const res = await fetch('/chat', {
  method: 'POST',
  headers: { 'content-type': 'application/json' },
  body: JSON.stringify(message),
  signal: controller.signal
});

if (!res.ok) {
  status.textContent = `error ${res.status}: ${await res.text()}`;
  throw new Error(`stream failed with ${res.status}`);
}

status.textContent = 'streaming';
const reader = res.body.getReader();
const decoder = new TextDecoder();
while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  append(decoder.decode(value, { stream: true }));
}

The free server option matters here because a fingerprint harness is noisy. You need many short requests to find the middleware version, body size, or nested field that disappears. A free token pool is the reasonable place for that noise, provided you still treat the provider as a black box and re-read its current limits before relying on them.

This approach has clear limits. A shape check is not schema validation. A field can have the right type and still mean the wrong thing, or survive as an empty array that did not exist in the original request. The function above is intentionally shallow; production versions need redaction rules, allowlists for top-level keys, and a cap on recursive depth. If you already have typed OpenAPI contracts and server-side validation, a manual fingerprint logger is worse than a contract test. If you are not proxying streaming output, a straightforward end-to-end request test will catch more with less code.

Do not use this harness as proof that a free integration is production-ready. Use it as a way to make the next silent 200 fail loudly at the exact point where a nested object stops looking like the thing you sent.

Free AI Quotas Break Silently When Your Proxy Swallows Retry-After

babycat — Sun, 16 Aug 2026 21:44:39 +0000

Free AI endpoints usually do not fail because the model ran out of capability. They fail because the quota, rate limit, or upstream gateway returned a status your browser code never learned to read. The fix is not a more polished error toast; it is a small relay that forwards the operational metadata you need to turn a silent limit into an actionable state.

This week's AI feed has been occupied by watermarking and agent tool gates, but the failure that shows up in free-model demos is less dramatic and more common. A 429 or 402 from an upstream provider reaches the browser as a generic 500 from the developer's own server, so the user hears "Something went wrong" and cannot distinguish between "wait and retry," "you have spent the allowance," and "the connection actually died."

MonkeyCode positions itself as an open-source option with a free model allowance, currently stated as 30 million tokens, and a free server path you can use to lab-test this exact relay. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Treat the token figure as a claim to confirm in your own dashboard rather than a permanent contract; the implementation below is provider-agnostic.

The relay owns the failure contract

Once you add your own server between the browser and the model, you own the part that usually damages the failure contract. The browser fetch can only inspect the response you give it. If your relay catches the upstream 429, logs it, and returns res.status(500).json({ error: 'Upstream failed' }), you have deleted the information that would have told the client to wait.

app.post('/api/generate', async (req, res) => {
  const upstream = await fetch(process.env.MODEL_URL, {
    method: 'POST',
    headers: { 'content-type': 'application/json' },
    body: JSON.stringify(req.body),
  });

  const retryAfter = upstream.headers.get('retry-after');
  res.status(upstream.status);
  if (retryAfter) res.set('retry-after', retryAfter);

  for await (const chunk of upstream.body) {
    res.write(chunk);
  }
  res.end();
});

The important line is res.status(upstream.status), not the stream forwarding. When your server accidentally collapses every upstream status into 500, the frontend can only guess, and guessing produces the worst kind of retry button: one that offers a fresh request while the quota is still exhausted.

Map the status to a state, not a message

On the client, model the response as a state machine before you write any DOM updates. A rate limit is not an error; it is a temporary state with a clock. A spent allowance is terminal for the session. A transport failure is a different problem entirely.

type ChatState =
  | { name: 'streaming'; controller: AbortController }
  | { name: 'rateLimited'; retryAt: number }
  | { name: 'quotaExhausted' }
  | { name: 'failed'; message: string };

function handleResponse(res: Response): ChatState | null {
  if (res.status === 429) {
    const seconds = Number(res.headers.get('retry-after') ?? 0);
    return { name: 'rateLimited', retryAt: Date.now() + seconds * 1000 };
  }
  if (res.status === 402) return { name: 'quotaExhausted' };
  if (!res.ok) return { name: 'failed', message: `HTTP ${res.status}` };
  return null;
}

A screen-reader user should not have to infer this from a red border or a toast that disappears. Place a single aria-live="polite" region in the DOM and never replace it. When the state changes, write a sentence, not a status name: "Rate limit reached. You can try again in 18 seconds." For a spent allowance, say "Free allowance used. Start a new session or switch the model route." That is the difference between a recoverable wait and an opaque failure.

Keep focus separate from announcement

Do not move focus to the error message just because it feels helpful. Focus movement is a navigation action; a live region is an update. For terminal states such as quota exhaustion, you may move focus to the retry or settings button, but do so once and only after the announcement has a chance to run. Keyboard users should not be yanked away from the transcript while a stream is still settling.

When a retry starts, call controller.abort() on the previous stream and replace the transcript writer. A free endpoint will sometimes deliver several chunks after the client has already moved on. If you share one reader variable across prompts, you can end up with old tokens appearing after the new prompt, and the screen reader will read a transcript that no longer matches the user's intent.

Test the four states without spending a paid key

Using the free server option rather than calling the model directly from the browser has two practical benefits here. You keep credentials out of frontend code, and you create a single place to standardize upstream status codes from any provider. That is what makes the retry logic testable instead of being a special case buried in the UI.

Run four upstream conditions against the same interface: a successful stream, a 429 with a valid Retry-After, a 429 without Retry-After, and a terminal allowance response such as 402. In each case, test with keyboard only, then with VoiceOver on macOS and NVDA or Narrator on Windows. Record the exact announced string and where focus rests. If the screen reader repeats the same "Something went wrong" phrase for all four states, the relay or the client mapping is wrong.

Limits of the approach

This design does not make a free allowance larger, and it cannot help if the upstream provider hides its status under a load balancer, or if the free server itself introduces a separate rate limit you have not designed for. If Retry-After is missing, show an estimated countdown but label it as an estimate. Never let the UI imply that a user can skip a wait the provider has not disclosed.

The valuable work is not the retry button. It is preserving the upstream's operational answer long enough for a keyboard or screen-reader user to act on it. Run the matrix once against a free endpoint such as the MonkeyCode free tier and free server path, and you will catch the quota bug before a paying user does.

AI Agent Tool Calls Need an Accessible Approval Gate, Not Just a Confirmation Dialog

babycat — Sun, 16 Aug 2026 09:09:38 +0000

When an AI agent can trigger a real action—sending an email, deleting a document, calling an HTTP endpoint—the confirmation dialog stops being a convenience and becomes the only boundary between the model's suggestion and the user's consent. If that boundary is not operable by keyboard, announced to a screen reader, and recoverable after a failure, you have not really asked for consent; you have merely placed a styled button in front of an opaque transition. The pattern I want to show here is a small accessible approval gate that treats every tool call as an asynchronous state transition, and you can test it against a free model endpoint without standing up a paid backend.

Start with the failure that usually ships. A custom modal appears when the agent wants to use a tool, the user presses Enter on "Approve," and the modal closes immediately. For a sighted mouse user the next few seconds might feel fine because the button they are watching changes into a progress indicator. For a keyboard or screen reader user, however, focus often jumps back to the document body, the live region was never populated, and the only feedback is silence while the model streams a tool call that may fail later. If the request fails after the dialog is gone, the user has no obvious way to retry or inspect what was sent. The confirmation was rendered as a one-shot visual gate, not as a state machine with an announced destination.

A better mental model is a lockbox with a receipt. When the agent proposes a tool call, the gate opens a reviewing state and stores the proposed action in an accessible list. Approving is not the end of the interaction; it is the transition into an executing state, which can end in success, failure, or cancellation. Each transition must move focus to the element that represents the new state and use an aria-live region to say what just happened. If you do not announce the destination, a screen reader user hears a click and then nothing, which is the accessibility equivalent of handing someone a key and walking away.

The state table below is the first artifact worth copying into your design notes, because it forces you to decide where focus goes and what is announced before you write any JavaScript.

State	Focus target	Announcement
idle	trigger button	—
reviewing	dialog heading	"Reviewing: send_email with 2 recipients"
approving	progress region	"Approving tool call"
executing	status log	"Executing send_email"
success	status log	"Email sent"
failure	retry button	"Failed to run tool. Retry button available"
cancelled	trigger button	"Tool call cancelled"

The code below strips that table down to a minimal browser implementation. It is deliberately unopinionated about your agent framework; it only cares that a proposed tool call arrives as a named action and an arguments object. The gate owns its own state, keeps focus where the next decision has to be made, and exposes the same transition methods whether a model streams a tool call proposal or a test harness feeds one in manually.

class ToolApprovalGate {
  constructor({ trigger, dialog, liveRegion, onExecute, log }) {
    this.state = "idle";
    this.trigger = trigger;
    this.dialog = dialog;
    this.liveRegion = liveRegion;
    this.onExecute = onExecute;
    this.log = log;
    this.lastProposal = null;
  }

  async open(proposal) {
    this.lastProposal = proposal;
    this.state = "reviewing";
    this.dialog.hidden = false;
    this.dialog.querySelector("h2").textContent =
      `Reviewing: ${proposal.name}`;
    this.dialog.querySelector("[data-args]").textContent =
      JSON.stringify(proposal.args);
    this.dialog.querySelector("[data-approve]").focus();
    this.announce("Reviewing: " + proposal.name);
  }

  approve() {
    if (this.state !== "reviewing") return;
    this.state = "approving";
    this.announce("Approving tool call");
    this.dialog.hidden = true;
    this.log.focus();
    this.run();
  }

  async run() {
    this.state = "executing";
    this.announce("Executing " + this.lastProposal.name);
    try {
      const result = await this.onExecute(this.lastProposal);
      this.state = "success";
      this.announce(result.label || this.lastProposal.name + " completed");
    } catch (error) {
      this.state = "failure";
      this.announce("Failed to run tool. Retry button available");
      this.log.nextElementSibling.focus(); // retry button
    }
  }

  cancel() {
    if (this.state !== "reviewing") return;
    this.state = "cancelled";
    this.dialog.hidden = true;
    this.trigger.focus();
    this.announce("Tool call cancelled");
  }

  announce(message) {
    this.liveRegion.textContent = "";
    requestAnimationFrame(() => { this.liveRegion.textContent = message; });
  }
}

The important detail is not the class name or the exact selector; it is that approve() can fail after the dialog has closed. That is why the failure path moves focus to a visible retry button instead of reopening the original dialog, and why every state change writes to a single live region rather than relying on the browser's default focus ring. If you wire onExecute to a real model call, you can trigger that failure deterministically by aborting the fetch after approval, which is exactly the kind of regression a free endpoint makes cheap to run repeatedly.

To build a minimal reproduction, have a free model endpoint propose a tool call from a one-line prompt such as "Show me a tool call for sending a short confirmation email." Feed the parsed proposal into the gate, then host the static page on a free server so you can test from a real phone and screen reader rather than only from localhost. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's operator advertises a free model access path with a 30-million-token allowance and a free server option, which is enough for the repeated aborted-request and retry loops this test requires; the same harness works with any endpoint that returns a tool-call proposal. Keep the harness separate from the agent itself, because you want to test the consent boundary in isolation.

Your QA matrix should cover at least four transitions: the user approves with Enter and the call succeeds, the user approves and the call fails, the user presses Escape during review, and the user cancels after the live region has already announced one state. Record the browser, operating system, and assistive technology versions along with the exact transition that failed, because screen reader timing differs enough that a passing Chrome and NVDA run does not guarantee the same behavior in Safari and VoiceOver. The most common bugs you will find are focus landing on a hidden element after the dialog closes, an announcement being swallowed because the live region was updated before it became visible, and the retry button being present in the DOM but unreachable by keyboard.

This pattern is not a security boundary. It makes approval legible and recoverable, but it does not validate the tool call, restrict the agent's permissions, or remove the need for server-side authorization. You should not lean on this gate for destructive actions where the model could misstate its own intent, and you should not present it as a substitute for an audit log. It is most useful for the middle layer of agent interactions where the risk is confusion rather than catastrophe, and where the confirmation dialog would otherwise be the only thing standing between a streamed suggestion and a real effect.

If you want to test that middle layer without first negotiating a paid model key or a cloud function, a free endpoint and a free static server are enough to expose the worst focus and announcement bugs in an afternoon. The gate itself is deliberately small; the value comes from treating approval as a state machine that can fail after the user says yes, and from testing that path with the same seriousness you would give to a backend error.

A Free Model Endpoint Loses Its Value When a Page Refresh Destroys the Chat

babycat — Sun, 16 Aug 2026 02:34:42 +0000

Free model access and a free server do not save you from the oldest browser failure: a refresh or crash that erases the conversation and spends tokens on a second attempt behind your user's back. The fix is not a fancier loading spinner; it is a tiny resumable state machine that treats the tab as unreliable.

The recent interest in watermarked model output and agent tool gatekeepers is really a question about observable behavior, and an interface that says 'streaming' while a refresh destroys the transcript is just as opaque.

A free hosted chat can feel like a public kitchen with a very generous ticket. The model streams food quickly, the server keeps the door open, and everyone is welcome. But if the only copy of the order lives in the waiter's pocket, one dropped tray means the meal disappears and the kitchen has already used the ingredients. In a browser, the dropped tray is a tab refresh, a mobile browser eviction, or a laptop sleep.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's project is described by its operator as an open source option with free model access and a free server. The operator states that the current allowance is 30 million free tokens. I am not going to invent model names, quotas, hardware, or permanence beyond that, because the useful part of this article does not depend on those details. What matters is that you can host a small HTML file on the free server and point it at a streaming model endpoint without first building a whole backend.

The demo below is a single file that keeps the conversation in localStorage, announces each state change in a polite live region, and handles one difficult case that many free-tier demos ignore: the user refreshes while the model is still streaming.

<!doctype html>
<html lang='en'>
<head>
<meta charset='utf-8'>
<title>Resumable streaming chat</title>
</head>
<body>
<main>
  <h1>Resumable chat</h1>
  <p id='status' aria-live='polite'></p>
  <div id='messages' aria-label='Conversation'></div>
  <form id='composer'>
    <label for='prompt'>Your message</label>
    <textarea id='prompt' required></textarea>
    <button type='submit'>Send</button>
    <button type='button' id='cancel'>Cancel</button>
  </form>
</main>
<script>
const MODEL_URL = 'https://your-monkeycode-endpoint.example/v1/stream';

const status = document.getElementById('status');
const messages = document.getElementById('messages');
const prompt = document.getElementById('prompt');
const cancel = document.getElementById('cancel');

const STORE_KEY = 'babychat:session';
let state = 'idle';
let controller = null;

const say = (text) => {
  status.textContent = '';
  requestAnimationFrame(() => { status.textContent = text; });
};

function render(conversation) {
  messages.textContent = '';
  for (const item of conversation) {
    const p = document.createElement('p');
    p.textContent = `${item.role}: ${item.text}`;
    messages.append(p);
  }
}

function save(conversation, inflight) {
  localStorage.setItem(STORE_KEY, JSON.stringify({ conversation, inflight }));
}

function load() {
  try {
    const raw = localStorage.getItem(STORE_KEY);
    if (!raw) return { conversation: [], inflight: null };
    return JSON.parse(raw);
  } catch {
    return { conversation: [], inflight: null };
  }
}

function restore() {
  const saved = load();
  if (!saved.conversation.length) return;
  render(saved.conversation);
  if (saved.inflight) {
    say('Restored an interrupted response. Retry if you want to continue.');
  } else {
    say(`Restored ${saved.conversation.length} previous messages.`);
  }
}

async function streamResponse(conversation) {
  controller = new AbortController();
  state = 'sending';
  say('Sending message to model.');
  const response = await fetch(MODEL_URL, {
    method: 'POST',
    headers: { 'content-type': 'application/json' },
    body: JSON.stringify({ conversation }),
    signal: controller.signal
  });

  if (!response.ok || !response.body) throw new Error(`HTTP ${response.status}`);
  const reader = response.body.getReader();
  const decoder = new TextDecoder();
  let partial = '';
  conversation.push({ role: 'assistant', text: '' });
  state = 'streaming';
  say('Model is responding.');

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    partial += decoder.decode(value, { stream: true });
    conversation[conversation.length - 1].text = partial;
    render(conversation);
    save(conversation, { role: 'assistant', text: partial });
  }
  state = 'idle';
  save(conversation, null);
  say('Response complete.');
}

document.getElementById('composer').addEventListener('submit', async (event) => {
  event.preventDefault();
  const text = prompt.value.trim();
  if (!text) return;
  prompt.value = '';
  const conversation = load().conversation;
  conversation.push({ role: 'user', text });
  render(conversation);
  save(conversation, { role: 'user', text });
  try {
    await streamResponse(conversation);
  } catch (error) {
    if (error.name === 'AbortError') {
      state = 'cancelled';
      save(conversation, conversation[conversation.length - 1]);
      say('Cancelled. The partial response is still visible.');
      return;
    }
    state = 'error';
    save(conversation, conversation[conversation.length - 1]);
    say(`Error: ${error.message}. Retry with the visible conversation.`);
  } finally {
    controller = null;
  }
});

cancel.addEventListener('click', () => {
  controller?.abort();
  cancel.disabled = true;
  setTimeout(() => { cancel.disabled = false; }, 200);
});

window.addEventListener('beforeunload', () => {
  const conversation = load().conversation;
  if (state === 'sending' || state === 'streaming') {
    save(conversation, conversation[conversation.length - 1]);
  }
});

restore();
</script>
</body>
</html>

The critical design choice is not the fetch call; it is that every state change has a text announcement and a stable conversation record. When the user sends a message, the assistant message object exists before the first token arrives, so the page always has a place to write partial text. When the response fails or is cancelled, the saved record keeps that partial text instead of throwing it away. That turns an invisible token spend into a visible, retryable state.

A polite live region is enough for progress updates because the user should keep typing and reading without focus being stolen. If you add an Escape key handler, keep the same rule: a cancellation announcement is polite, while a blocking error may deserve focus. The form stays keyboard-operable, and the visible conversation is built from plain DOM elements rather than an unlabeled list or canvas.

Here is the basic QA matrix to use with a browser and a screen reader.

| Scenario | Expected behavior |
| Refresh during streaming | User prompt and assistant partial remain visible; the live region says an interrupted response was restored. |
| Cancel button while streaming | Fetch aborts, partial remains, no completion message is announced. |
| Network drops mid-response | Error state announces the failure and leaves the partial for manual retry. |
| Screen reader running | Progress changes are announced once per state transition, not once per token. |

Limitations of this approach are easy to state. localStorage is limited to roughly five megabytes and is tied to one browser, so it cannot replace server-side session storage for a real product. The free server may restart, and the operator-supplied token allowance can be consumed by the original attempt even when the user never sees the completion. If the model endpoint changes its stream events or refuses partial state, you still need to adapt the parser. This harness deliberately avoids automatic retry; any retry you add should send the retained conversation only after the user approves it, because the partial assistant turn has already consumed tokens.

You should not use this as a production chat system for authenticated, regulated, or multi-user synchronized data, and it will not solve rate-limit or abuse control on the server. It is a browser-side probe for the exact failure that makes a generous free allowance feel unreliable.

If you have access to the free server, host the file and interrupt a response on purpose; the conversation should survive the refresh and the live region should tell you exactly what happened.

A Free Token Pool Still Needs a Screen-Reader-Visible Budget Boundary

babycat — Sat, 15 Aug 2026 19:50:59 +0000

A free token allowance is a limit, not a progress bar. The failure you want to catch early in a browser-based AI tool is not the first request, but the moment a long-running loop has spent the whole pool while the screen still says “streaming.” The fix is to put a small boundary guard in front of fetch and make that boundary visible to a keyboard or screen-reader user at the three transitions that matter: before a send is allowed, after usage is recorded, and after a rate limit resets.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The availability details used below—free model access, a free server route, and an advertised 30 million token allowance—are operator-supplied and worth rechecking against the current project page before you build anything durable on top of them. The technique itself is also transferable to any quota-based endpoint.

The painful version of this failure starts when you paste a long prompt into a small test harness and press Enter. The response streams, but the usage object is hidden inside JSON that the UI never reads. After a few runs, the server starts returning 429, the free server route stops being generous, and the person using the tool has no idea whether the budget is exhausted or the request simply failed. That ambiguity is worse when the user cannot see a small token counter buried under the transcript. A screen reader will announce “send” and then hear silence, or it will announce every incremental stream update but never the boundary that already closed.

A guard belongs around the request path rather than inside the chat bubble. I model only four phases: idle, running, blocked, and retry_after. The guard does not predict the exact vendor tokenizer. It estimates the pending spend before the request leaves the page, then corrects that estimate with the actual usage.total_tokens returned by the model. The boundary is therefore a preflight decision, not a post-hoc log line. For a keyboard user, focus remains on the prompt input when a send is rejected. For a screen-reader user, the status region announces the blocked condition assertively.

function makeBudgetGuard({ limit, live }) {
  let spent = 0;
  let pending = 0;

  function say(message, priority = 'polite') {
    live.textContent = '';
    window.setTimeout(() => {
      live.textContent = message;
      live.setAttribute('aria-live', priority);
    }, 50);
  }

  function estimate(prompt) {
    if (!prompt) return 0;
    return Math.max(1, Math.ceil(prompt.length / 4));
  }

  function canSend(prompt) {
    pending = estimate(prompt);
    const projected = spent + pending;
    if (projected > limit) {
      say(`Send blocked. Estimated ${pending} tokens would take the total to ${projected}, above the ${limit} token boundary. Reduce the prompt and try again.`, 'assertive');
      return false;
    }
    say(`Preparing to send. ${pending} tokens estimated; ${limit - projected} remain.`, 'polite');
    return true;
  }

  function record(usage = {}) {
    const measured = usage.total_tokens ?? usage.completion_tokens ?? 0;
    if (measured <= 0) return;
    spent += measured;
    say(`Recorded ${measured} tokens. New total is ${spent}; ${Math.max(0, limit - spent)} remain.`, 'polite');
  }

  function retryAfter(seconds) {
    say(`Rate limited. Next send should wait ${seconds} seconds.`, 'assertive');
    window.setTimeout(() => say('The guarded send is clear to try again.', 'polite'), seconds * 1000);
  }

  return { canSend, record, retryAfter, spent: () => spent, pending: () => pending };
}

The estimate here is deliberately crude: roughly one token for every four characters. That is not tokenizer truth, and you should say so in the interface rather than presenting it as precise accounting. After the response arrives, the reported usage replaces the estimate. The important property is that the guard can block a send before it happens, not that it guesses perfectly. If you want greater accuracy, you can replace the character heuristic with a local tokenizer or simply run fewer preflight checks and rely on the first response to inform the next decision.

Pair the guard with a status region that is already in the DOM before the first announcement. The region should not be created lazily on the first error because some screen readers do not reliably announce content inside a newly inserted live region. I keep it as a role='status' element paired with an input and button. Polite progress updates stream through it, while blocked and rate-limited transitions switch to assertive so they interrupt the current reading.

<input id='prompt' type='text' placeholder='Paste a prompt long enough to move the budget' />
<button id='send' type='button'>Send</button>
<div id='budget-status' role='status' aria-live='polite'></div>

const status = document.getElementById('budget-status');
const prompt = document.getElementById('prompt');
const sendButton = document.getElementById('send');
const guard = makeBudgetGuard({ limit: 30_000_000, live: status });
const controller = new AbortController();

sendButton.addEventListener('click', async () => {
  const text = prompt.value.trim();
  if (!guard.canSend(text)) return;

  sendButton.disabled = true;
  try {
    const response = await fetch('/agent/complete', {
      method: 'POST',
      signal: controller.signal,
      body: JSON.stringify({ prompt: text })
    });

    if (response.status === 429) {
      const retryAfterSeconds = Number(response.headers.get('retry-after') || 8);
      guard.retryAfter(retryAfterSeconds);
      return;
    }

    const data = await response.json();
    guard.record(data.usage);
  } catch (error) {
    if (error.name === 'AbortError') {
      status.textContent = 'Send cancelled.';
      return;
    }
    status.textContent = 'The request failed before usage could be recorded. The token total has not changed.';
  } finally {
    sendButton.disabled = false;
  }
});

I use the free server route as a place to host the same boundary as a small read model rather than keeping the total only inside a single browser tab. That matters because a browser reload can lose the accumulated spent value unless you persist it. A tiny server endpoint that stores the current total and returns it after each request gives every client the same boundary. It does not make the server data durable by itself: if the free server restarts or the process is recycled, the in-memory total can reset, so treat persistence as a separate requirement before you claim any conformance.

app.get('/budget', (req, res) => {
  res.json({ spent, limit, remaining: limit - spent, pending });
});

The QA pass should test the transitions that actually fail, not just the happy path. I run three probes. First, with a keyboard, tab to the input and send a prompt that is too long; focus must remain on the input, the send button must not activate, and the status region must announce the blocked reason without requiring a mouse hover. Second, with the screen reader running, allow a short prompt and confirm that the polite progress message does not read over the model stream every token. Third, force a 429 and confirm that the assertive retry message announces the wait once, while repeated Enter presses do not create a pile of overlapping announcements. I test that combination in at least two browsers because aria-live can behave differently with rapid DOM replacement.

This approach is useful when a token pool can be burned by loops, when the interface has no visible spend meter, or when the person operating it cannot infer rate-limit state from a raw network tab. It is less useful for one-off prompts, for systems that already have a trusted vendor usage endpoint, and for any budget where the cost must be exact before spending. In that stricter case, the client-side estimate is a nice UX signal but it cannot be the source of truth. Keep the boundary in the UI, but let the authoritative accounting stay with the service that reports actual token counts after each response.

The guard is deliberately small enough to rehearse against any quota-based endpoint. If you already have MonkeyCode's free model access and free server route, that combination is a cheap sandbox for exercising the failure path before you wire it into an interface that spends real budget. Add the guard early, announce the boundary, and do not let a generous free allocation turn into a silent loop that stops without saying why.

When the Free Token Bucket Is Empty, Tell the User Before They Ask

babycat — Sat, 15 Aug 2026 13:38:43 +0000

The first useful lesson a free model endpoint teaches you is not how well the model writes; it is how the endpoint behaves when the boundary is hit. A free tier is a production environment in one specific sense: it can reject you, throttle you, and expire at an inconvenient moment, and if your interface collapses all of those states into a single vague error, you have replaced a precise system message with a mystery. This article shows how to turn that boundary into an accessible, announced state by running a small budget gate on a free server, because the failure path is exactly where most streaming AI demos stop being keyboard-operable or screen-reader-legible.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free model access, the free server option, and the 30,000,000-token allowance I mention are operator-supplied availability claims. I did not treat the token figure as a benchmark or a promise, and the workflow below works with any budgeted model endpoint.

Current AI discussions are full of gatekeepers for agent tools and debates about watermarking, but those conversations tend to focus on what the model should be allowed to do or how its output can be recognized later. For a front-end developer, a more immediate problem is what happens when a request simply stops because the allowance is gone. The HTTP response may arrive as a 402, a 429 with a retry-after header, a provider-specific JSON payload, or a mixed format that a generic fetch wrapper was never taught to classify. The user, meanwhile, sees a spinner, then a button that says Retry, and then the same spinner again.

Think of an allowance the way a car treats a low-fuel warning. When the fuel is low, the driver gets a dedicated state with a clear decision: refuel soon or choose a nearer station. A chat interface should do the same. It needs to distinguish retryable throttling from terminal allowance exhaustion from a real upstream outage, because the correct next action is different in each case. One state might mean wait a few seconds, another might mean shorten the prompt, and another might mean check credentials or come back later.

MonkeyCode is an open-source project, but the part that matters here is the combination of a free model endpoint and a free server deployment path. I used that combination as a cheap way to provoke a real quota failure without spending my own infrastructure budget. If you already have a budgeted endpoint, you can place the same gateway in front of it.

Build the budget gate on the free server

Rather than teaching the client every proprietary error schema, put a normalizing gateway in front of the model request. The gateway passes through successful responses without inventing new meaning. When a request fails, it converts the upstream status and body into a small problem object the client can render as one announced state.

export default {
  async fetch(request) {
    const auth = request.headers.get('authorization') ?? '';
    const payload = await request.text();

    const upstream = await fetch('https://model-gateway.example/v1/chat', {
      method: request.method,
      headers: {
        'content-type': 'application/json',
        authorization: auth,
      },
      body: payload,
    });

    const retryAfter = upstream.headers.get('retry-after');
    const raw = await upstream.text().catch(() => '');

    if (!upstream.ok) {
      return Response.json({
        reason: classifyUpstream(upstream.status, raw),
        status: upstream.status,
        retryAfterMs: retryAfter ? Number(retryAfter) * 1000 : null,
      }, { status: 502 });
    }

    return new Response(raw, {
      status: upstream.status,
      headers: {
        'content-type': upstream.headers.get('content-type') ?? 'application/json',
      },
    });
  },
};

This is a Worker-style handler, so it can run in many free server offerings. The endpoint placeholder is deliberately not a real MonkeyCode path. Keep the gateway server-side, not in the browser, because the browser should never hold long-lived model credentials. The classifyUpstream function is the part you own: treat a 402, or a 429 whose body mentions balance, billing, quota, or insufficient tokens, as allowance_exhausted; treat a 429 with a retry-after header as rate_limited; treat 401 or 403 as unauthorized so you can explain a credential problem instead of asking someone to wait for a token bucket that is not the cause.

Make the client announce once, not constantly

On the client, do not announce every streaming chunk. Reserve live regions for state transitions. Keep an aria-live region mounted in the DOM from the start, then change only its text. Many screen readers will miss a live region if you change its role after insertion, so choose role=status for ordinary updates and switch the message, not the role, for urgent failures. For an allowance_exhausted transition, set the text to something specific, such as: The model allowance is empty. Shorten your prompt or retry after fifteen minutes. The retry control should be a real button in the tab order immediately after the status message, not a clickable span, so keyboard users do not have to search for it.

const region = document.querySelector('#stream-status');
const retry = document.querySelector('#retry-button');

function announce(state, retryAfterMs) {
  const message = {
    throttled: 'The request is rate-limited. Wait a moment and try again.',
    allowance_exhausted: `The model allowance is empty. Shorten your prompt${retryAfterMs ? ` or retry after ${Math.ceil(retryAfterMs / 60000)} minutes` : ''}.`,
    unavailable: 'The model service is not reachable. Your prompt was not sent.',
    unauthorized: 'The request could not be authorized. Check the server token.',
  }[state];
  region.textContent = message ?? '';
}

const result = { reason: 'allowance_exhausted', retryAfterMs: 120000 };
announce(result.reason, result.retryAfterMs);
retry.textContent = result.retryAfterMs ? 'Retry when ready' : 'Retry now';

Under the hood, model the request as explicit states: idle, waiting_for_tokens, streaming, throttled, allowance_exhausted, unavailable, retrying, and canceled. Each transition gets exactly one announcement. That keeps a screen reader from reading the full stream twice, and it keeps cancel and retry deterministic. When the user activates Retry, move to waiting_for_tokens, clear the old message, and announce again only when the next state actually changes.

What to test before you trust it

Run the failure path with NVDA and VoiceOver, then repeat it with a keyboard alone and with a pointer alone. The transition worth checking is this one: start a stream, activate Cancel, send a second request that hits allowance_exhausted, then verify that focus did not jump into a modal, the status message was read exactly once, and the Retry button is reachable with Tab. On iOS and older WebKit, test whether an assertive live region causes a focus change; usually you want aria-live=assertive without moving focus. Also open the browser console with network throttling enabled and confirm that the gateway returned the problem object instead of a blank caught promise.

The gateway does not create more allowance, does not make a free endpoint production-grade, and can be ignored by a client that never renders the problem object. Treat the free server route as a controlled space for prototypes and failure drills. Do not push personal messages, long-lived secrets, or high-stakes business traffic through it without checking the provider terms. If you need billing, per-user quotas, audit logs, or guaranteed latency, a managed API gateway is the correct tool rather than a small worker.

The next time you try a free model, do not measure how quickly it streams a successful answer first. Instead, force the boundary: exhaust the allowance, cancel mid-stream, hit a throttle, and watch how the screen reader and Tab key interpret each transition. If you have MonkeyCode's free server option available, host this gateway there and make the first version fail loudly on purpose. That is the fastest way to learn whether your interface treats an empty token bucket as a real state or just another generic error. Share the reproduction with your browser, operating system, and assistive technology versions, plus the exact transition that failed.