Building a Screenshot-to-Code Streaming UI That Does Not Lie to Users

At 3:14 AM, your on-call pager fires: conversion dropped 80% across your visual prototyping tool. In customer session replays, users drop in screenshots, hit "Generate," watch an empty canvas flicker for twenty seconds, and then stare at a mangled, half-closed tag that freezes the browser tab. When frustrated users mash "Regenerate," competing streaming responses collide in memory, causing the code editor to strobe between conflicting layout attempts.

A screenshot-to-code product has a simple demo: upload an image, wait, show generated HTML or React. Production behavior is much harder. The UI must keep rendering while the model thinks, avoid presenting incomplete output as a finished component, survive cancellation and reconnects, and preserve type guarantees at every boundary.

abi/screenshot-to-code is a useful anchor because its task is concrete: turn a visual reference into clean HTML, Tailwind, React, or Vue output. The frontend problem is not only displaying a model response. It is coordinating an image-derived request, a potentially long-running generation, streaming partial code, preview updates, user edits, retries, and an explicit stop action. A toy typewriter effect usually concatenates strings into one React state variable. That works until a user changes the screenshot mid-stream, a proxy buffers the response, or a stale completion overwrites a newer preview.

This article describes a React 19 and TypeScript architecture for that path. The boundary is deliberate: the browser owns rendering, state transitions, cancellation, and validation; the gateway owns model access and emits typed stream events. The transport is Server-Sent Events (SSE) over fetch, rather than EventSource, because screenshot generation requests often need a JSON body, request headers, and an abort signal.

Start with a stream contract, not a component

The first production decision is the event vocabulary. Do not expose raw provider chunks directly to the UI. Providers differ on whether they stream reasoning, role markers, tool calls, code fences, or final usage. The browser needs application events.

Use a discriminated union. Strict TypeScript makes unhandled event variants visible when the contract evolves.

// src/features/generation/protocol.ts
export type GenerationEvent =
  | { type: "started"; generationId: string; model: string; at: number }
  | { type: "delta"; generationId: string; channel: "code" | "analysis"; text: string }
  | { type: "preview"; generationId: string; html: string }
  | { type: "usage"; generationId: string; inputTokens: number; outputTokens: number }
  | { type: "completed"; generationId: string; code: string }
  | { type: "failed"; generationId: string; code: "invalid_image" | "upstream" | "rate_limited"; message: string };

export type StreamState =
  | { phase: "idle" }
  | { phase: "connecting"; requestId: string }
  | { phase: "streaming"; requestId: string; generationId: string; code: string; preview: string | null }
  | { phase: "complete"; requestId: string; generationId: string; code: string }
  | { phase: "error"; requestId: string; message: string };

The requestId is created in the browser before the request begins. It is not a server identifier and it is not optional. It establishes ownership: only events associated with the currently active request may modify the editor. A distinct generationId comes from the server and is retained for logs, usage, and support investigations.

A minimal server event looks like this:

event: generation
data: {"type":"delta","generationId":"gen_01J...","channel":"code","text":"<main className="}

Each event must end with a blank line. Set Content-Type: text/event-stream; charset=utf-8, Cache-Control: no-cache, no-transform, and Connection: keep-alive. The no-transform directive matters: intermediary transformations and compression policies can turn a streaming response into a buffered one.

Keep transport parsing outside React

React components should not parse byte streams. Put decoding and SSE framing in a small module with a narrow callback interface. This keeps malformed event handling testable without a DOM and lets the component deal in GenerationEvent values only.

// src/features/generation/readSse.ts
import type { GenerationEvent } from "./protocol";

export async function readGenerationStream(
  response: Response,
  onEvent: (event: GenerationEvent) => void,
): Promise<void> {
  if (!response.ok || !response.body) {
    throw new Error(`Generation request failed: ${response.status}`);
  }

  const reader = response.body.pipeThrough(new TextDecoderStream()).getReader();
  let buffer = "";

  for (;;) {
    const { value, done } = await reader.read();
    if (done) break;
    buffer += value;

    const frames = buffer.split("\n\n");
    buffer = frames.pop() ?? "";
    for (const frame of frames) {
      const data = frame
        .split("\n")
        .filter((line) => line.startsWith("data:"))
        .map((line) => line.slice(5).trimStart())
        .join("\n");
      if (!data) continue;
      onEvent(JSON.parse(data) as GenerationEvent);
    }
  }
}

That parser assumes server-produced JSON and therefore treats a parse failure as a protocol error. It should not silently ignore corrupt data. In a mature implementation, validate each decoded object with a runtime schema such as Zod before narrowing it to GenerationEvent; TypeScript cannot validate bytes received from a network.

The split("\n\n") technique is appropriate here only if the gateway deliberately emits JSON on data: lines and never puts arbitrary unescaped line breaks into the event framing. Maintain this as a shared contract. Do not copy provider framing rules into the client.

A reducer prevents stale-stream corruption

Generation is a state machine. useState<string> does not encode its legal transitions. A reducer can reject a delta from an old generation after the user submits another screenshot or presses Stop.

// src/features/generation/reducer.ts
import type { GenerationEvent, StreamState } from "./protocol";

export function reduceStream(state: StreamState, event: GenerationEvent): StreamState {
  if (event.type === "started") {
    if (state.phase !== "connecting") return state;
    return { phase: "streaming", requestId: state.requestId, generationId: event.generationId, code: "", preview: null };
  }
  if (state.phase !== "streaming" || state.generationId !== event.generationId) return state;

  switch (event.type) {
    case "delta":
      return event.channel === "code" ? { ...state, code: state.code + event.text } : state;
    case "preview":
      return { ...state, preview: event.html };
    case "completed":
      return { phase: "complete", requestId: state.requestId, generationId: event.generationId, code: event.code };
    case "failed":
      return { phase: "error", requestId: state.requestId, message: event.message };
    case "usage":
      return state;
    default: {
      const neverEvent: never = event;
      return neverEvent;
    }
  }
}

The early started transition is intentional. Events before started are ignored; event delivery after completion is ignored. In practice this is what prevents an older request from repainting an editor after its replacement has started.

React 19 is useful here because rendering generated code and operating the controls are different priorities. Keep stop, replace-image, and error controls responsive while visual work is deferred. useDeferredValue is suitable for a large code display or expensive syntax highlighting; it is not a fix for excessive update frequency.

const deferredCode = useDeferredValue(state.phase === "streaming" ? state.code : codeToShow);
const [isPending, startTransition] = useTransition();

function applyPreview(html: string) {
  startTransition(() => setSandboxDocument(html));
}

Do not call startTransition around the stream state that drives the Stop button. Stopping must be urgent. Defer only a derived preview or syntax-highlighted view.

Fetch SSE with one active AbortController

Use fetch for POST plus streaming response access. Store the controller in a ref, abort any previous request before opening a new one, and clear it only if it is still the owner. This is where many apparent “React race conditions” actually originate.

const controllerRef = useRef<AbortController | null>(null);
const [state, dispatch] = useReducer(reduceState, { phase: "idle" });

async function generate(imageDataUrl: string, target: "react" | "html" | "vue") {
  controllerRef.current?.abort();
  const controller = new AbortController();
  controllerRef.current = controller;
  const requestId = crypto.randomUUID();
  dispatch({ type: "connect", requestId });

  try {
    const response = await fetch("/api/generate/stream", {
      method: "POST",
      signal: controller.signal,
      headers: { "Content-Type": "application/json", Accept: "text/event-stream" },
      body: JSON.stringify({ image: imageDataUrl, target }),
    });
    await readGenerationStream(response, (event) => dispatch({ type: "event", requestId, event }));
  } catch (error) {
    if ((error as DOMException).name !== "AbortError") {
      dispatch({ type: "transport-error", requestId, message: "Stream disconnected. Retry generation." });
    }
  } finally {
    if (controllerRef.current === controller) controllerRef.current = null;
  }
}

function stopGeneration() {
  controllerRef.current?.abort();
  controllerRef.current = null;
  dispatch({ type: "cancel" });
}

The outer reducer must verify requestId before delegating to reduceStream. That check matters even after abort(): aborting a client request does not erase already queued browser microtasks. Also propagate cancellation upstream whenever your backend gateway supports it. Client abort saves local rendering budget; propagating cancellation reclaims costly upstream inference capacity.

Optimistic rendering should remain narrow. Immediately display a skeleton editor, the selected target framework, the source image thumbnail, and an active generation status indicator. Do not optimistically render guessed code or claim that the preview is valid. For screenshots, a partial DOM can trigger unexpected image loads, trigger expensive CSS reflow bugs, and look deceptively functional. Always flag the preview canvas as provisional until the completed event arrives.

Preview safely and throttle the expensive parts

Generated code is untrusted arbitrary input, even when produced by your own upstream prompt pipeline. Render HTML previews inside an <iframe> with a restrictive sandbox policy. A secure baseline is sandbox="allow-scripts" with no same-origin permission, populated via a per-generation srcDoc. Never inject raw model HTML into your main document tree via dangerouslySetInnerHTML.

<iframe
  title="Generated preview"
  sandbox="allow-scripts"
  referrerPolicy="no-referrer"
  srcDoc={deferredPreviewDocument}
  className="h-full w-full border-0"
/>

There is an operational trade-off here. The allow-scripts flag enables interactive previews but makes an opaque origin mandatory; never add allow-same-origin. If the preview requires no script execution, drop allow-scripts entirely. For network security, outbound calls from inside the preview require an explicit Content Security Policy (CSP) at the document level; the sandbox alone is not an egress filter.

Do not assign srcDoc on every single token. At 30 to 60 events per second, tearing down and rebuilding an iframe DOM destroys performance and negates the perceptual speed of streaming. Append tokens to the code editor state immediately, but coalesce iframe refreshes with a short throttle window (e.g., 150 ms), and only update when the generated text completes a parseable structural tag boundary. The gateway is best equipped to emit discrete preview checkpoint events because it can monitor model output structure.

Measure time-to-first-token (TTFT) separately from time-to-first-useful-preview (TTFUP). TTFT reflects upstream network and model provider responsiveness; TTFUP incorporates browser DOM parsing, syntax highlighting, layout calculations, and iframe initialization. When reverse proxies buffer output, TTFT balloons misleadingly: the upstream provider generates immediately, but the client browser sits idle until the internal buffer flushes or the socket closes.

For Nginx, streaming endpoints require an explicit zero-buffering configuration:

location /api/generate/stream {
  proxy_pass http://generation_gateway;
  proxy_http_version 1.1;
  proxy_set_header Connection "";
  proxy_buffering off;
  proxy_cache off;
  gzip off;
  proxy_read_timeout 300s;
  add_header X-Accel-Buffering no always;
}

This is not a blanket configuration for every web endpoint. Apply it strictly to streaming SSE routes; disabling buffering globally harms caching efficiency and throughput on standard REST APIs. Verify streaming health using browser DevTools Network waterfalls alongside server-side emission timestamps. If TTFT spikes while gateway logs show instant chunk production, your proxy layer—not your React components—is buffering the stream.

This distinction highlights the stark contrast between buffered middleboxes versus zero-buffer streaming proxies, utilizing B-Lost's unbuffered SSE pass-through for instant TTFT frontend rendering.

Failure modes that appear after launch

A streaming connection dropping silently after exactly 60 seconds is almost always an intermediate proxy idle timeout. Emit an SSE comment heartbeat (: keepalive\n\n) every 15 to 25 seconds while the upstream model is thinking, and ensure the gateway flushes it immediately. Another frequent failure mode is a completion event missing its final code snapshot: if the network drops the last delta chunk, the user is left with a broken preview that fails export. Make completed.code authoritative and replace the accumulated buffer with the server's definitive snapshot upon arrival.

Stream retries must always mint a fresh requestId and unique generation identifier. Never attempt to reconnect by naively appending chunks to an old code buffer unless your transport protocol includes strict monotonically increasing sequence IDs. In visual code generation, a clean, deterministic restart is far safer than coping with mangled, duplicated markup.

Capture client-side telemetry at the boundary: record requestId, framework target, input image payload size, initial connection timestamp, first token timestamp, completion timestamp, frame count, abort triggers, and error codes. Do not log raw screenshot images or generated source code by default; both contain sensitive user data. Structured telemetry enables you to pinpoint network buffering, upstream rate limits, rendering latency, or user cancellations without misattributing every UI glitch to model quality.

Conclusion

The architecture outlined here is intentionally disciplined: strict discriminated union event contracts, transport decoding isolated from the DOM, finite state reducers, single-controller abort lifecycles, deferred background reconciliation, and iframe sandboxing. These decisions prevent an AI streaming interface from lying to users under heavy load, intermittent connectivity, or sudden cancellation. The underlying vision model may be probabilistic, but the state machine rendering it cannot afford to be.

What does your team's gateway topology look like under load? Are you running unbuffered SSE pass-through proxies, or shifting stream normalization into client WebAssembly workers? Drop your architecture choices and production battle scars in the comments below.

Disclosure: Compute infrastructure and multi-model benchmark relays for this writeup are sponsored by b-lost.com — an enterprise AI gateway offering 0.58x-0.8x official pricing, native prompt caching, and zero user-data retention. All benchmark metrics reflect independent reproducible testing.