Mock Streams Pass, Real Streams Fail: Test Your Accessible Streaming UI Against a Real Model

#ai #a11y #webdev #javascript

My streaming chat UI passed every unit test. Mock chunks arrived at a polite, steady rhythm; the loading state announced itself; the cancel button moved focus back to the composer. Then I pointed it at a real model and everything got worse in ways my mocks never reproduced: a burst of twelve tokens in one animation frame, a 40-second silence mid-answer, an error that arrived after a partial sentence, and a retry that silently reused a stale abort signal.

The uncomfortable lesson: mock streams test your state machine's happy timing, not its failure timing. Real models produce the timing that actually breaks focus management and screen-reader announcements. This post shows a small, runnable harness I use to rehearse those failure timings against a live model — and how I keep that rehearsal free.

The interaction failure, concretely

Here's the screen-reader experience that motivated this work. With a mock:

Submit → aria-live announces "Assistant is responding…"
Chunks render into an aria-live="off" region (intentionally, so we don't spam announcements)
Completion → "Response complete. 4 paragraphs."

With a real model on a cold connection:

Submit → silence for 9 seconds (pre-stream latency my mock never had)
First chunk arrives as a burst — three paragraphs at once — and my throttled "still responding" announcement fired after text was already visible
Stream errors mid-sentence; my error handler replaced the partial text, so a sighted user saw the error but a screen-reader user lost the partial answer entirely — it was removed from the DOM before anything announced it

None of these are bugs a mock with setInterval(chunk, 50) will ever catch.

Why rehearse against a real model (and do it for free)

The failure timings that matter — pre-stream latency, token burstiness, mid-stream errors, slow-final-token — are properties of the model and network, not your code. So your test environment needs a live model endpoint, but it doesn't need a fast or expensive one. In fact, a slow free model is a better test fixture: latency amplifies every race in your UI.

I use MonkeyCode's free model access for this, running the harness against its free server option so the whole rehearsal costs nothing and lives outside my production stack. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The specific reason it fits this workflow is narrow and honest: I need any real streaming endpoint to stress failure timing, and a free one means I can leave the harness running in CI-adjacent manual checks without budget anxiety. I am deliberately not claiming anything about speed, quotas, or which models are available — for failure-timing tests, slower is fine, and you should verify current availability yourself before depending on it.

The state table, early

Everything below hangs off this transition table. If your UI can't name these states, the harness will find that out quickly:

State	Trigger	Visual	Announcement	Focus
`idle`	—	composer enabled	—	composer
`connecting`	submit	composer disabled, status visible	"Contacting assistant…"	stays in composer
`streaming`	first chunk	partial text, stop button	"Assistant is responding…"	unchanged
`stalled`	no chunk for 8s	subtle spinner + elapsed note	(re-announce once) "Still waiting…"	unchanged
`cancelling`	stop pressed	—	"Stopping…"	unchanged
`cancelled`	abort confirmed	partial text kept, retry offered	"Response stopped. Partial answer kept."	moves to retry button
`errored`	error event / non-200	partial text kept, error + retry	"Response failed. Partial answer kept above the error."	moves to retry button
`complete`	stream end	full text	"Response complete."	stays in composer

The two rows mocks never exercise well are connecting (real pre-stream latency) and stalled (real mid-stream silence). The two most important accessibility decisions: never delete partial text on error/cancel, and only move focus on user-initiated endings (cancel/error), never on complete.

The runnable harness

Single-file demo. Save as index.html, serve with any static server, and point ENDPOINT at a real streaming model. The harness deliberately instruments timing so you can see burstiness and stalls, not just correctness.

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Real-stream a11y harness</title>
<style>
  :root { font-family: system-ui, sans-serif; }
  #log { border: 1px solid #999; padding: .5rem; min-height: 8rem; white-space: pre-wrap; }
  #status { margin: .5rem 0; font-weight: 600; }
  .err { color: #8b0000; }
  #timing { font-size: .85rem; color: #444; }
  .visually-hidden { position:absolute; width:1px; height:1px; overflow:hidden; clip-path: inset(50%); }
  button { margin-right: .5rem; }
</style>
</head>
<body>
  <h1>Streaming harness</h1>

  <!-- Polite announcer: state changes only, never token content -->
  <div id="announcer" class="visually-hidden" aria-live="polite"></div>

  <form id="composer">
    <label for="q">Prompt</label>
    <input id="q" name="q" autocomplete="off" required>
    <button id="send" type="submit">Send</button>
    <button id="stop" type="button" disabled>Stop</button>
    <button id="retry" type="button" disabled>Retry</button>
  </form>

  <p id="status" role="status"></p>
  <p id="timing" aria-hidden="true"></p>
  <div id="log" aria-live="off" aria-label="Assistant response"></div>
  <p id="error" class="err" role="alert" hidden></p>

<script type="module">
// ---- typed state machine ----
const S = {
  idle:       { submit: 'connecting' },
  connecting: { chunk: 'streaming', error: 'errored', timeout: 'stalled' },
  streaming:  { chunk: 'streaming', done: 'complete', error: 'errored', cancel: 'cancelling', timeout: 'stalled' },
  stalled:    { chunk: 'streaming', done: 'complete', error: 'errored', cancel: 'cancelling' },
  cancelling: { aborted: 'cancelled', error: 'errored' },
  cancelled:  { retry: 'connecting', reset: 'idle' },
  errored:    { retry: 'connecting', reset: 'idle' },
  complete:   { reset: 'idle' },
};

let state = 'idle';
let controller = null;
let stallTimer = null;
let t0 = 0, lastChunk = 0, chunkCount = 0, burstMax = 0;

const $ = (id) => document.getElementById(id);
const say = (msg) => { $('announcer').textContent = ''; requestAnimationFrame(() => $('announcer').textContent = msg); };

function transition(event) {
  const next = S[state]?.[event];
  if (!next) { console.warn(`ignored: ${event} in ${state}`); return; }
  state = next;
  render();
}

function render() {
  $('send').disabled  = !['idle', 'cancelled', 'errored', 'complete'].includes(state);
  $('stop').disabled  = !['connecting', 'streaming', 'stalled'].includes(state);
  $('retry').disabled = !['cancelled', 'errored'].includes(state);

  const messages = {
    connecting: 'Contacting assistant…',
    streaming:  'Assistant is responding…',
    stalled:    'Still waiting for the model…',
    cancelling: 'Stopping…',
    cancelled:  'Response stopped. Partial answer kept.',
    errored:    'Response failed. Partial answer kept above the error.',
    complete:   'Response complete.',
    idle:       '',
  };
  $('status').textContent = messages[state] || '';
  if (messages[state]) say(messages[state]);

  // Move focus ONLY on user-initiated endings.
  if (state === 'cancelled' || state === 'errored') $('retry').focus();
}

function armStallTimer() {
  clearTimeout(stallTimer);
  stallTimer = setTimeout(() => transition('timeout'), 8000);
}

async function run(prompt) {
  controller = new AbortController();
  t0 = performance.now(); lastChunk = t0; chunkCount = 0; burstMax = 0;
  $('log').textContent = ''; $('error').hidden = true;
  transition('submit');
  armStallTimer();

  const ENDPOINT = '/api/stream'; // point at your real streaming model endpoint

  try {
    const res = await fetch(ENDPOINT, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ prompt }),
      signal: controller.signal,
    });
    if (!res.ok || !res.body) throw new Error(`HTTP ${res.status}`);

    const reader = res.body.getReader();
    const decoder = new TextDecoder();
    let burst = 0;

    for (;;) {
      const { done, value } = await reader.read();
      if (done) break;
      const now = performance.now();
      // burst = bytes arriving within 16ms of previous chunk
      burst = (now - lastChunk < 16) ? burst + value.byteLength : value.byteLength;
      burstMax = Math.max(burstMax, burst);
      lastChunk = now; chunkCount++;

      armStallTimer(); // any chunk resets the stall detector
      transition('chunk');
      $('log').textContent += decoder.decode(value, { stream: true });
    }

    clearTimeout(stallTimer);
    transition('done');
  } catch (e) {
    clearTimeout(stallTimer);
    if (e.name === 'AbortError') { transition('aborted'); }
    else {
      $('error').textContent = `Stream error: ${e.message}`;
      $('error').hidden = false;
      transition('error');
    }
  } finally {
    $('timing').textContent =
      `chunks: ${chunkCount} · max burst: ${burstMax}B · first chunk at: ${Math.round(lastChunk - t0)}ms (approx)`;
  }
}

$('composer').addEventListener('submit', (e) => {
  e.preventDefault();
  if (state === 'idle') run($('q').value);
});
$('stop').addEventListener('click', () => { transition('cancel'); controller?.abort(); });
$('retry').addEventListener('click', () => { transition('retry'); run($('q').value); });

render();
</script>
</body>
</html>

Three details worth stealing even if you ignore the rest:

say() clears then re-sets the live region. Repeated identical messages ("Still waiting…" twice) are not re-announced by most screen readers unless the region's content actually changes.
The stall timer is armed on every chunk, including the wait for the first one. "Connecting" and "stalled" are the same user experience at different timestamps — one detector covers both.
bursts are measured, not assumed. If max burst routinely exceeds what your layout can paint without jank, you have a rendering problem to fix, not a test to skip.

What the real model exposed that mocks didn't

Running this against a live free endpoint, three concrete fixes came out:

Pre-stream latency of 6–12s on cold starts. My mock started streaming in 100ms, so the connecting state had never been visible for more than a blink. Real behavior: users pressed Send twice. Fix: the state machine ignores submit in non-idle states (see the guard in the form handler) — but the harness is what proved the guard was actually reachable.
A mid-stream stall of 40s followed by normal completion. Without the stalled state, a screen-reader user had nine seconds of announced "responding" and then nothing, with no way to distinguish "thinking" from "dead."
An error after 80% of the answer. Keeping the partial text (rather than replacing it with the error) meant the error case still delivered most of the value, and the role="alert" error appeared below the kept content so reading order stayed sane.

Manual QA matrix

The harness tells you timings; humans still verify announcements. I run this matrix before calling a streaming UI done:

Scenario	Chrome + NVDA (Win)	Safari + VoiceOver (macOS)	Firefox + NVDA
Slow connect (throttle: offline→online)	✓/✗	✓/✗	✓/✗
Mid-stream stall > 8s
Stop during burst
Error after partial text
Retry after error (fresh AbortController)
Keyboard-only: full flow, visible focus

One regression this matrix caught: on VoiceOver, focusing the retry button on error worked, but the role="alert" fired after focus moved, so the announcement interrupted the button's label. Reordering (update DOM, announce, then focus on next frame) fixed it.

Limitations and who shouldn't use this

A free endpoint is a fixture, not a production dependency. Don't ship against it, don't benchmark against it, and don't assume availability, quotas, or specific models — verify current terms yourself. Its only job here is producing realistic timing.
This tests your UI, not the model. Output quality, safety, and latency SLAs are out of scope.
The harness is intentionally manual. If you need CI, record real timings once and replay them through a deterministic mock — but only after a real-model pass has told you which timings exist.
If your chat UI doesn't keep partial text, doesn't have a stop button, or announces every token, this harness will just confirm a redesign is needed before testing is useful.

If you want to try this, the cheapest path is exactly what I did: point the harness at any free real endpoint — MonkeyCode's free server option worked for me — and let a slow model do the adversarial testing your mocks were too polite to do. If you run it and hit a transition that breaks differently for you, tell me your browser, OS, assistive tech versions, and the exact state transition that failed — that's the bug report I actually want.