Building SpecDecode: a visual lab for speculative decoding

Speculative decoding is simple to summarize and difficult to teach. “A small model guesses and a large model checks” does not communicate the batch verification, the accepted prefix, the first-mismatch rule, or why the serving layer matters. SpecDecode was built to make those mechanics understandable by watching one trace unfold.

The central question behind the app was: can developers understand a target-model verification pass before they read a paper or configure a backend? The answer guided the product toward a focused interaction instead of a generic chat UI.

The critical boundary: educational simulation vs. native speculation

The app makes one distinction explicit everywhere it matters. The built-in trace is an educational simulation. It does not present two standard LLM API calls as native speculative decoding.

Native speculative decoding happens in the inference server: a draft model proposes a sequence; the target model verifies that sequence efficiently; the accepted prefix becomes output; processing resumes from the first mismatch. LangChain is useful around that system as a provider abstraction and orchestration layer, but is not the serving implementation itself.

This is why the UI uses “Educational simulation” and “Illustrative results” badges. Accuracy in the explanation is more useful than a flashy but ambiguous speedup claim.

Translating the algorithm into an interface

The core SpeculativeDecoderDemo is a compact state machine. Its single stage progresses through the phases that matter to a learner: prompt, draft batch, target verification, acceptance, mismatch, and final prefix.

const drafted = Math.min(demoTokens.length, Math.max(0, stage - 1));
const verified = Math.min(drafted, Math.max(0, stage - 8));

The demo uses a deterministic batch so every replay tells the same story:

export const demoTokens = ["A", "hash", "table", "is", "a", "data", "structure"];
export const accepted = [true, true, true, true, true, true, false];

The draft lane first emits all seven tokens. The target lane then resolves each chip: green tokens enter the accepted-output rail, while the last chip becomes a visible mismatch. Play, Pause, Step, and Replay all operate on the same state rather than duplicating animation logic.

Framer Motion gives chips and Agent Lab nodes a deliberate entry/active transition. CSS handles the supporting cursor, signal, and pulse effects. This keeps motion meaningful rather than decorative.

Architecture prepared for real streaming

The current UI is static, but its model is ready for a backend that streams semantic events. lib/speculation.ts defines a small contract:

export type SpeculationEvent =
  | { type: "draft"; tokens: string[] }
  | { type: "verify"; tokens: string[]; accepted: boolean[] }
  | { type: "complete"; metrics: SpeculationMetrics };

export interface SpeculativeBackend {
  supportsNativeSpeculation: boolean;
  generate(options: SpeculativeGenerationOptions): AsyncGenerator<SpeculationEvent>;
}

That contract makes the intended next step clear:

An SSE route could turn the async generator into browser events. The visualizer stays provider-agnostic because it only cares about draft tokens, verification outcomes, and completion metrics. supportsNativeSpeculation allows a backend to say “false” truthfully instead of silently faking support.

Keeping LangChain integration small

Provider configuration is centralized in createChatModel. It supports OpenAI, OpenAI-compatible endpoints, Anthropic, Gemini, Ollama, and LM Studio through the relevant LangChain adapters.

if (config.provider === "anthropic") {
  return new ChatAnthropic({ apiKey: config.apiKey, model: config.model, streaming });
}
if (config.provider === "ollama") {
  return new ChatOllama({ baseUrl: config.baseURL, model: config.model, streaming });
}
return new ChatOpenAI({
  apiKey: config.apiKey,
  model: config.model,
  streaming,
  configuration: config.baseURL ? { baseURL: config.baseURL } : undefined,
});

The factory is intentionally not an elaborate provider layer. It is one dependable seam for request validation, server-side credentials, and future capability discovery. API keys should be used only in the current request and never be persisted in local storage, URLs, or source files.

Agent latency: the other teaching problem

Speculation matters for agents because agents repeatedly generate text between tool calls. The Agent Lab models Think → Calculator → Think → Answer and compares conceptual normal and speculative timing.

The calculator duration does not shrink. This is the lesson:

Speculative decoding does not make tools faster. It can reduce the time a model spends generating output between tool calls.

That distinction prevents an overly broad claim about agent acceleration. It also provides a useful template for real instrumentation: measure model time and tool time separately.

Visual design decisions

The visual language borrows from lab instruments rather than sci-fi dashboards. Graphite indicates target work; pale signal blue indicates drafting; verification green indicates agreement; copper marks the mismatch and conceptual boundary. The token verification rail is the memorable element. Everything surrounding it stays intentionally quiet, so a viewer can infer the algorithm from the movement.

The responsive design stacks model lanes on small screens and lets token streams scroll horizontally rather than wrapping into a misleading order.

A responsible next iteration

The best next feature is a native backend integration—not more interface surface.

1. Add a SpeculativeBackend implementation for a serving runtime that explicitly supports draft + target verification.
2. Add a Next.js streaming route and emit the existing event contract.
3. Record actual timestamps and show Measured results only for those runs.
4. Retain Illustrative results for concept-only interactions.
5. Test acceptance-prefix logic and streamed event ordering.

SpecDecode’s value comes from its clarity about both possibilities and constraints. It demonstrates why speculative decoding can reduce expensive generation steps without pretending that every provider call—or every agent workflow—gets the same speedup.

Code & more: https://www.dailybuild.xyz/project/221-specdecode