DEV Community: Borui Cai

How to Search Cursor Chat History Across Workspaces

Borui Cai — Wed, 09 Sep 2026 02:08:55 +0000

Originally published on Extracurricular AI.

You remember fixing the authentication bug. You cannot remember which project contained the conversation.

cursor-history searches conversation text in supported local Cursor stores across discovered workspaces. It can read history created before you installed it, without a capture hook, embeddings, or an API key.

Start with a phrase you remember

npm install -g cursor-history
cursor-history list --all
cursor-history search "connection pool"

Run this on the machine where the history is stored, using Node.js 20.x or 22.x–26.x. The CLI searches parsed message content using case-insensitive text matching. It does not semantically infer that “database saturation” means “connection pool.”

Try a distinctive substring if a broad query returns too much:

cursor-history search "ECONNRESET" -n 30
cursor-history search "oauth callback"
cursor-history search "refreshToken"

Search returns up to 10 matching sessions by default. -n 30 raises the limit; -n 0 returns all matching sessions. list --all removes the listing limit; it is not a switch that enables global search. Search covers discovered workspaces by default unless you select a workspace.

Open the session that actually matched

For interactive use, copy the session index displayed in the result. If it is 12:

cursor-history show 12
cursor-history export 12

The number refers to the underlying session list within the same data roots and workspace scope. It is not the search result's ordinal position. New sessions can also change list indices, so use exact UUIDs for saved commands.

With jq installed, retrieve those UUIDs directly:

cursor-history search "authentication" --json   | jq -r '.results[].sessionId'

Copy a returned ID into these commands, replacing the placeholder:

session_id='PASTE_RETURNED_SESSION_UUID'
cursor-history show "$session_id"
cursor-history export "$session_id" --format json --output ./conversation.json

The JSON export contains the available session representation. It may include source details and inferred timestamps; it is not proof of a complete original conversation.

Narrow the workspace when you know it

Keep the same scope on the follow-up read:

cursor-history --workspace /absolute/path/to/project search "authentication"
cursor-history --workspace /absolute/path/to/project show 1

Here 1 means session 1 in that workspace's list. Do not reuse a global index in a workspace-scoped read. A workspace filter also limits which conversation sources may be opened; it is more than a display filter.

If you get no useful matches

Check	Why it matters
Try a shorter phrase or exact error token	Text search does not paraphrase the query
Run `cursor-history list --all --json`	Inspect discovered sessions and any diagnostics
Check which machine and user account owns the files	Local search does not fetch SSH, cloud, or another user's history
Check custom roots or WSL paths	Composer data and `~/.cursor` can live in different environments
Review source status	Missing, partial, or ambiguous sources can affect what is searchable

For an explicit environment, set both roots to the intended directories:

CURSOR_DATA_PATH='/absolute/path/to/Cursor/User/workspaceStorage' CURSOR_STORE_ROOT='/absolute/path/to/.cursor' cursor-history search "authentication"

The platform guide explains these selectors. An empty result means no match was returned from the readable selected sources; it does not prove that a chat was deleted.

When to use Cursor's own commands

To continue an Agent CLI conversation, start with Cursor's native session commands. This workflow is useful when you need to locate text across local conversations and then inspect or export the result.

The MCP companion can let an assistant perform the search. As checked on September 8, 2026, MCP 0.3.1 uses reader 0.18.0. The packages release independently, so update the MCP package when upgrading the server.

If the search finds something useful, keep cursor-history on GitHub. For moved projects, continue with the recovery guide.

Implementation references: search traversal, text matcher, CLI JSON formatter.

How to Rewind Codex Conversations and Files Together

Borui Cai — Fri, 21 Aug 2026 12:07:53 +0000

Codex can move a conversation back to an earlier prompt. The missing half is the workspace: the old conversation can return while the newer files remain on disk.

This guide shows how to add synchronized conversation-and-file rewind to Codex CLI now. The first section is the practical setup. The rest is a Rust-level look at the snapshot design: how it covers Git-tracked files, files directly edited by the agent, and a bounded set of recent shell-made changes without taking an unbounded snapshot of the entire workspace.

Disclosure: I built codex-rewind, the project used in this article. It is an unofficial distribution of OpenAI Codex CLI and is not affiliated with or supported by OpenAI. I am describing the implementation and its limits, not claiming that it is the best agent for every workflow.

The two-minute setup

Install codex-rewind from npm:

npm install -g codex-rewind

The executable is codexr, so it can live beside the official codex command rather than overwriting it. Start a new tracked session with:

codexr --enable file_snapshots

Use Codex normally. When an experiment goes in the wrong direction, run:

/rewind    # choose an earlier prompt; restore the conversation and its files
/redo      # undo that rewind and return to the pre-restore state

The important word is together. Restoring only the files leaves the model with a conversation that describes edits which no longer exist. Restoring only the conversation leaves the model reasoning about an older world while newer files remain on disk. The selected turn is the shared coordinate for both histories.

To enable tracking for future new sessions, add this to the existing Codex config:

# ~/.codex/config.toml
[features]
file_snapshots = true

Tracking is session-scoped. A session starts with snapshot tracking for its whole life, or it has none. Turning the feature on after the mistake cannot create snapshots for earlier turns.

A small test outside Git

You can test the behavior in an ordinary directory; no repository is required:

mkdir rewind-demo
cd rewind-demo
printf 'an uncommitted idea\n' > local-notes.md
codexr --enable file_snapshots

Ask the agent to delete local-notes.md, then use /rewind and select the prompt before the deletion. The file and the conversation should return to that point. /redo should move both forward again.

That example is deliberately not a Git trick. A useful agent-level undo buffer has to work for an untracked note, a generated document, or a directory that was never a repository.

The actual failure mode: two histories drift apart

There are two relevant states in an agent session:

The transcript that determines what the model believes has happened.
The workspace that contains the effects of what actually happened.

Codex CLI users have asked for a unified restore in open issues #9203 and #11626. The latter describes the gap precisely: conversation rewind exists, but the code changes after the selected point remain in the working tree.

Git is still the right tool for durable, reviewed project history. It is not a complete substitute for a per-turn safety net:

A never-added file has no committed version to recover.
Agent work increasingly includes notes, documents and data outside repositories.
Committing or stashing every exploratory turn mixes disposable agent history with intentional project history.
Shell commands and MCP tools can change files without producing a structured edit record that the model can later replay in reverse.

The target is therefore not “replace Git.” It is “make every restore point carry a coherent conversation state and a reversible workspace state.”

Boundary one: own the snapshot storage, not the repository

codex-rewind stores content-addressed blobs, manifests and per-thread references under Codex's own home directory:

~/.codex/file_snapshots/
├── blobs/
├── manifests/
└── refs/

The snapshot crate has no dependency on Codex's other crates and does not invoke Git or write to .git. Restore writes workspace files from its own store. The Git-tracked partition may read the index to obtain project-owned file paths, but snapshot objects, refs, commits and index mutations never enter the user's repository.

That separation is about ownership and compatibility. Session history belongs to the agent; repository history belongs to the user and their tools.

It also keeps the official Codex session format untouched. The existing rollout file gets no new fields or event types. A sidecar maps Codex turn IDs to snapshot manifest IDs:

Codex rollout:       ... turn_id = "turn-42" ...
Snapshot sidecar:        "turn-42" -> "manifest-b3..."

As a result, the same conversation can be opened by codex or codexr without conversion. That is format compatibility, not magical snapshot coverage: turns run in the official build do not create sidecar snapshots, so reopening that session in codexr cannot retroactively make those turns file-rewindable.

Boundary two: track a bounded union, not the whole tree

Separating storage from .git does not answer how many workspace files should be observed. That is a different problem.

On the repository used to develop this subsystem, a full subtree walk saw 70,609 files and about 116 GB. Most of that was build output, logs and caches that would not normally enter Git. The bounded tracked union was about 6,096 files and 59 MB.

Those numbers do not argue that Git is bad. They demonstrate why capture scope cannot be “everything below the working directory.” A build can keep inflating that tree even when the actual project has not changed.

The implementation unions three partitions:

Partition	What it covers	What bounds it
Git-tracked	Files the project already identifies as source	The project index
Agent-touched	Files directly edited by the agent in this session	The agent's recorded actions
Recently modified residue	Shell/MCP changes outside the first two sets	100 extra files, 16 MiB each, with churn directories skipped

In Rust, the ordering is load-bearing. Recent files are selected last so their small budget is not wasted on paths the first two partitions already cover:

pub fn tracked_files(
    roots: &[PathBuf],
    already_known: impl IntoIterator<Item = PathBuf>,
    include_hidden: bool,
) -> BTreeSet<PathBuf> {
    let ignores: Vec<Gitignore> = roots.iter().map(|root| load_ignore(root)).collect();

    // Partition 1 in this function: files the agent already touched.
    let mut files: BTreeSet<PathBuf> = already_known.into_iter().collect();

    // Partition 2: project-owned paths read from the Git index.
    for (root, ignore) in roots.iter().zip(&ignores) {
        files.extend(git_tracked_files(root, ignore));
    }

    // Partition 3: bounded residue not already covered above.
    for (root, ignore) in roots.iter().zip(&ignores) {
        files.extend(recent_files(root, ignore, include_hidden, &files));
    }
    files
}

The names “partition 1” and “partition 2” in that annotated excerpt describe program order; conceptually I usually list Git-tracked first because it is easier to explain. The result is a set union either way.

Only the recency partition needs arbitrary hard limits:

pub const RECENT_MAX_FILE_BYTES: u64 = 16 * 1024 * 1024;
pub const RECENT_LIMIT: usize = 100;

// After filtering ignored, covered, oversized and churn-directory entries:
candidates.sort_by_key(|(modified, _)| std::cmp::Reverse(*modified));
candidates.truncate(RECENT_LIMIT);

The other two partitions are bounded by meaning rather than a numeric cutoff. A Git-tracked file is part of the project even if it is large. An agent-touched file is relevant because the session deliberately changed it. Silently dropping the 101st agent edit would make the most important partition unreliable.

The 116 GB measurement is the observation that produced those engineering trade-offs: use all three partitions to broaden coverage, but never let ambient build churn decide the size of every checkpoint.

A separate ignore file is a feature, not duplicated configuration

Rewind and version control answer different policy questions:

.gitignore: should this path enter shared repository history?
.codexsnapignore: may the rewind system capture and later touch this path?

Those sets should be allowed to overlap, but neither should imply the other. For example, local-notes.md may be intentionally excluded from Git while still being valuable enough to rewind. A database, credential file or generated cache may be excluded from snapshots regardless of the Git policy.

# .codexsnapignore
.env
data/*.sqlite
tmp/

The matcher uses familiar Gitignore syntax but loads only the rewind-specific file:

pub const SNAPSHOT_IGNORE_FILENAME: &str = ".codexsnapignore";

pub fn load_ignore(root: &Path) -> Gitignore {
    let mut builder = GitignoreBuilder::new(root);
    builder.add(root.join(SNAPSHOT_IGNORE_FILENAME));
    builder.build().unwrap_or_else(|_| Gitignore::empty())
}

pub fn is_ignored(ignore: &Gitignore, path: &Path) -> bool {
    ignore.matched_path_or_any_parents(path, false).is_ignore()
}

The rule is symmetric: an ignored path is never captured, restored or deleted by a restore. Applying it only during capture would create a dangerous asymmetry where an excluded file could still be removed later.

Safe restore requires evidence, then an escape hatch

A bounded snapshot cannot honestly claim that every absent path should be deleted. Absence may mean “did not exist,” or it may mean “outside the tracked set.” Confusing the two is how a restore becomes a deletion bug.

The restore planner therefore deletes only from witnessed history: the subsystem must have observed enough of a path's lifecycle to know that the target checkpoint says it was absent. It does not infer deletion from a partial directory listing.

Before applying any restore, it also checkpoints the current state. /redo restores that safety manifest. This changes the cost of a mistaken choice from lost work to an extra round trip.

There are still real limitations:

A file no checkpoint ever saw cannot be restored.
A file outside the Git index, changed only by a shell command and pushed out of the 100-file recent window may be missed.
Hidden files are skipped by default unless directly edited; .git remains excluded from capture.
Rewind cannot undo remote side effects such as a pushed branch, sent request or database operation.
Two concurrent sessions can overwrite each other's workspace changes; this is not a merge system.

Stating those gaps is more useful than calling any snapshot system “complete.”

Why the core and app-server boundary matters

A feature implemented only in a terminal UI solves one surface and creates the next compatibility problem. Codex also has Desktop and IDE clients, so the reusable unit must live below any one interface.

Capture lives in the Rust core where turn boundaries and tool execution are visible. Restore is exposed through the app-server protocol. The protocol field is intentionally small:

/// Also restore tracked workspace files to their state at the fork point.
/// A safety checkpoint is recorded first, so the restore is reversible.
#[experimental("thread/fork.restoreFiles")]
#[serde(default, skip_serializing_if = "std::ops::Not::not")]
pub restore_files: bool,

codex-rewind exposes the CLI/TUI user flow today. Desktop and IDE clients do not ship this implementation, but the core store and protocol path are structured so another Codex surface can reuse the same capture and restore semantics instead of inventing another checkpoint format.

This cross-project, cross-surface concern is easy to miss in a local UI patch. It is also why “it works in my terminal” is not enough evidence that a rewind architecture fits Codex as a whole.

How the mechanism compares with Claude Code and OpenCode

This is a comparison of boundaries, not a product leaderboard.

Claude Code's checkpoint documentation says it tracks changes made through its file-editing tools, but not files modified by Bash commands. That is a clean and predictable boundary. codex-rewind keeps an equivalent agent-touched set, then adds Git-tracked paths and the bounded recency residue to catch more shell and MCP changes. The benefit is broader write-path coverage; the cost is more scope machinery and an explicit bounded-coverage gap.

OpenCode's snapshot documentation says it uses an internal Git repository and warns that large repositories or many submodules can cause slow indexing and significant disk use. codex-rewind instead uses its own content-addressed store and a bounded tracked set. The benefit is no Git repository dependency and less coupling to ambient tree size; the cost is a custom store, custom garbage collection and no Git-native interoperability.

System	Tracking boundary	Storage approach	Main trade-off relevant here
Claude Code	File-editing tools	Session checkpointing separate from Git	Bash-made changes are not tracked
OpenCode	Agent operations / project snapshot	Internal Git repository	Docs warn about indexing and disk use on large repos/submodules
codex-rewind	Git-tracked + agent-touched + bounded recent residue	Independent content-addressed sidecar	Broader but still deliberately incomplete coverage

The specific reason to try codex-rewind is therefore narrow: you use Codex, need conversation-and-file rewind now, want coverage beyond direct edit tools, and do not want the undo history written into the project's Git state.

Quick answers to the compatibility questions

Does codex-rewind replace the official `codex` command?

No. It installs a separate codexr executable, so both can exist on the same machine. It is an unofficial distribution built from an upstream Codex baseline, not an extension loaded into the official binary.

Does rewind require a Git repository?

No. In a repository, the subsystem reads the index as one source of project file paths. In a non-repository directory that partition is empty, while agent-touched and recently modified files still provide coverage. It never needs commits or stashes and does not write Git state.

Is the conversation format merely “compatible,” or actually unchanged?

It is unchanged. The implementation adds no field or event to Codex's conversation rollout. File history lives in a separate sidecar keyed by existing turn IDs. The practical caveat is coverage, not serialization: the official binary can read the conversation but does not produce snapshot entries for the turns it runs.

Does the same feature work across projects and Codex surfaces?

The capture policy is based on session workspace roots, not one repository layout, so the same subsystem handles repositories and ordinary directories across projects. The user-facing implementation here is CLI/TUI. Its Rust core and app-server restore API are reusable by Desktop and IDE clients, but those official surfaces have not adopted this subsystem.

Is it a complete backup system?

No. It is a bounded, session-level undo buffer. Keep using Git and normal backups for durable history. The three partitions increase useful coverage without promising a full-tree snapshot, and the limits section above defines the remaining gaps.

Compatibility and the upstream path

The package follows upstream Codex releases and installs alongside the official CLI. It deliberately shares the normal ~/.codex directory so login, configuration and conversation history can carry across. If you alternate executables inside one conversation, remember that only codexr writes file snapshots; finish a tracked conversation in the executable that started it, or use a separate CODEX_HOME if you prefer total isolation.

As of August 21, 2026, the official Codex contribution guide says that external code contributions and pull requests are not accepted. It asks for issues, root-cause analysis and design discussion instead. This project is therefore a usable bridge and a concrete design experiment, not a promise that an upstream PR will be merged.

If you want to inspect or try it:

The fastest path remains:

npm install -g codex-rewind
codexr --enable file_snapshots

If you test it, the feedback I care about most is not “does undo sound useful?” It is where the tracking boundary surprises you: which real agent-made file change escaped the three partitions, or which captured path you believe should have stayed outside the rewind system.

AI-assistance disclosure: I used an AI coding assistant to help structure and edit this article. I reviewed the technical claims and code excerpts against the project source and the linked documentation before publication.

Speculative Decoding, Illustrated: Why Generating 7 Tokens Can Cost the Same as 1

Borui Cai — Wed, 05 Aug 2026 02:27:56 +0000

This is a plain-language, illustrated explainer. You don't need an inference background to read it: every idea arrives as an everyday metaphor first (a small model that guesses, a big model that grades; a warehouse, a truck, and a workshop) and as math second — and every formula that does appear gets a plain-English translation right next to it. In a few places I deliberately oversimplify to keep the intuition front and center; footnoted rigor is what the papers in the reference list are for. If you've bounced off the original papers, this is the on-ramp.

🔑 The one sentence that unlocks everything

Why is LLM inference slow, and why does speculative decoding make it fast? It comes down to one sentence: making the matrices bigger does add more multiplications for the GPU — but no matter how big the matrix is, one matrix multiplication loads the weights exactly once. More compute does not mean more loading. And in LLM inference, loading (moving weights from GPU memory into the compute units) is where nearly all the time goes, while compute is close to free. So processing a few extra tokens costs almost no extra wall-clock time. This is not an algorithmic trick — it is a physical fact of GPU hardware. Every bit of speedup speculative decoding delivers is this one hardware dividend being cashed in.

A concrete example: the sentence "the cat sat on the red mat" is a matrix. Each token is first turned into a row of numbers (its embedding, say d dimensions). Stack the 7 rows and you get a 7 × d matrix X:

        ┌                                ┐
  the   │  0.12  -0.83   0.05   0.47  …  │
  cat   │ -0.31   0.22   0.68  -0.10  …  │
  sat   │  0.55   0.09  -0.42   0.33  …  │
  on    │ -0.07   0.61   0.18  -0.25  …  │
  the   │  0.12  -0.83   0.05   0.47  …  │
  red   │  0.29  -0.14   0.52   0.40  …  │
  mat   │  0.44   0.37  -0.20   0.08  …  │
        └                                ┘
            = matrix X  (7 rows × d columns)

Rows = tokens (7 here). A longer sentence means more rows and more multiplications — but multiplying this entire block X by the weight matrix W loads W from memory exactly once. That is what "more compute, not more loading" looks like in the flesh. (Easter egg: the two "the" rows are identical — same token, same embedding row, before attention gets involved.)

Context: This is the technical deep-dive in a two-part series. Part 1 argued that speculative decoding is one of the most underrated skill niches for LLM inference roles in 2025–2026, and introduced the core idea ("a small model guesses, a big model grades"). This part covers the full math (including the rejection-sampling proof), the main research directions of the past two years, and a list of real interview questions.

How to read this: Section 1 is the foundation. Section 2 is the math (formulas, but every one gets a plain-English explanation). Section 3 maps the research landscape (use it as a study roadmap). Section 4 is a framing insight that stands out in interviews. Section 5 is the learning path and interview question bank.

1. Why LLM Inference Is Slow: Two Orthogonal Bottlenecks

To understand why speculative decoding works, you first have to see clearly why inference is slow at all. This section doesn't touch the algorithm yet, but it is the foundation for everything after — and explaining it well in an interview instantly separates you from candidates who merely memorized the algorithm.

1.1 Bottleneck A: The Serial Dependency of Autoregressive Generation

Mainstream LLMs are causal Transformers [15] that generate autoregressively:

P (x_{1}, x_{2}, \dots, x_{n}) = i = 1 \prod n P (x_{i} ∣ x_{< i})

In plain English: every token needs all previous tokens as input. Generating 100 tokens means 100 serial model calls.

This serial nature lives at the algorithm level and cannot be bypassed without changing the modeling paradigm — which is exactly where more radical approaches like Lookahead Decoding and diffusion language models come from.

1.2 Bottleneck B: Memory-Bandwidth Bound (the important one)

This one is counterintuitive and a favorite interview topic: the bottleneck of LLM inference is not GPU compute (FLOPs) — it is memory bandwidth.

GPU memory is hierarchical:

Tier	Capacity	Bandwidth	Role
HBM (device memory)	~80 GB	~3 TB/s	Where model weights live
L2 cache	~50 MB	~10 TB/s	Staging area
SRAM (on-chip)	~100 KB	~30 TB/s	Where matrix multiplies actually happen

GPU compute cores can only work out of SRAM. On every forward pass, all model weights must stream from HBM into SRAM to participate in the computation — this is dictated by GPU physics; no software layer can skip it.

Put a number on it: a 70B model in FP16 is roughly 140 GB of weights. At an H100's ~3 TB/s of HBM bandwidth, the floor for a single forward pass is about 47 ms — the physical lower bound on per-token latency.

But the more important fact is this: that 140 GB transfer cost barely changes with how many tokens you process. One token: load everything once. Seven tokens: still load everything once. Processing a few extra positions adds almost nothing to the time of that forward pass.

Why is "a few extra positions" nearly free? The core operation of a forward pass is the matrix multiply Y = X · W: input X has shape [N, d] (N positions, d dimensions each), weights W have shape [d, d']. Once the GPU has moved W from HBM into SRAM, it multiplies that same copy of W against all N rows of X.

To be precise: going from 1 row to 7 rows does increase the GPU's FLOPs — six extra rows of multiply-accumulate is not zero work. The key is that one matrix multiplication loads W exactly once, and the loading volume does not depend on the number of rows. Since LLM inference is memory-bound — compute is wildly abundant and bandwidth is the constraint — the extra compute hides inside the weight-loading time: the compute cores were sitting idle waiting for W to arrive anyway, so multiplying a few more rows costs almost no extra wall-clock time. "Extra compute is free" doesn't mean compute literally costs nothing — it is a direct consequence of the GPU's compute ≫ bandwidth characteristic.

1.3 Putting the Two Bottlenecks Together: The Core Pain

Bottleneck A (algorithm): N serial forward passes are mandatory.
Bottleneck B (hardware): each forward pass pays a fixed loading cost but produces only 1 token.

Together, they are the essence of slow inference: paying "move 140 GB of weights" per pass and getting exactly 1 token back is a terrible exchange rate.

The elegance of speculative decoding is that it uses B to partially defeat A — since extra tokens are free, find a way for one forward pass to verify many tokens. Hold on to this and every paper in the field snaps into place.

1.4 The Key Clarification: What Does One Forward Pass Actually Produce?

Time to puncture a very common misconception — getting this right is the key to everything that follows.

Most people think one Transformer forward pass computes 1 token. It doesn't.

💡 The key mechanism: one forward pass = one weight load + one matrix multiplication, and that single matrix multiplication computes every input position at once. Loading and computing are welded together — there is no such operation as "load once, then compute N separate times." "Computing N times" does not physically exist. What exists is "1 forward pass," "2 forward passes," … and each forward pass is an inseparable bundle of load once + compute all current positions.

Transformers [15] use causal self-attention: given an input sequence of length N, one forward pass outputs a "next-token distribution" at all N positions simultaneously. Every input position i gets a distribution P(x_{i+1} | x≤ᵢ). This is precisely why Transformers train so much faster than RNNs — all positions compute in parallel.

So why does ordinary decoding look like "1 token per pass"? Because:

Position 0 predicts "what comes after input token 1" → that's input token 2, which we already have
Position 1 predicts "what comes after input token 2" → that's input token 3, which we already have
…
Position N−1 predicts "what comes after the last input token" → the only genuinely new information

So the predictions at the first N−1 positions are entirely wasted — they predict input we've already seen. Ordinary decoding keeps only the last position's output, hence "one new token per pass" from the outside.

Speculative decoding exploits exactly this waste: let a small model pre-fill those "future positions" with guesses, and suddenly all N distributions from the big model's single forward pass become meaningful —

the first γ distributions verify the corresponding draft tokens ("do I, the big model, agree with the small model's guess?")
the last distribution yields a free bonus token (if all γ drafts were accepted)

Put ordinary and speculative decoding side by side and it becomes crystal clear (both producing 7 tokens):

Decoding	Forward passes	Weight loads	What those matrix multiplies compute
Ordinary	7	7 (≈ 7 × 47 ms)	Pass k computes k positions but keeps only the last one; the rest is wasted
Speculative	1	1 (≈ 47 ms)	One pass computes 7 positions, all of them used to verify drafts

Here is the counterintuitive part: ordinary decoding's 7th forward pass also computes 7 positions — it's just that the first 6 predict "input we already know" and get thrown away. So the total matrix-multiply work of the two approaches is roughly the same. The real difference isn't how much you compute — it's how many times you load: ordinary decoding loads the weights 7 times; speculative decoding compresses that to 1. And since loading dominates latency (~47 ms per load), compressing 7 loads into 1 is compressing the time itself.

Within that single verification pass, the costs are also wildly asymmetric:

Stage	Who	What	Cost
Verification forward	Big model (GPU)	One forward pass, one weight load, distributions at all positions in parallel	The main cost (~47 ms)
Accept/reject decisions	Algorithm layer (CPU or a tiny GPU kernel)	Table lookups over a batch of distributions + corrected rejection sampling	Practically free

⚠️ A common wrong formulation is "multiple verifications inside one forward pass" or "multiple forward passes." Both are imprecise. The forward pass is 1, the load is 1, and verification is many distributions from a single pass being consumed in parallel by the outer algorithm. It is a free dividend of the Transformer's causal structure.

Internalize this mechanism and the proof, the research landscape, and the interview questions in the next sections all become intuitive.

2. The Core Algorithm

2.1 The Procedure

Let:

Target model M_p with distribution p(· | context) (big, slow — say 70B)
Draft model M_q with distribution q(· | context), far smaller than M_p (small, fast — say 1B)

Each round of speculative decoding has 4 steps [1, 2]:

Draft: M_q autoregressively generates γ candidate tokens x₁, …, x_γ (typically γ = 4–7)
Parallel verification: feed all γ tokens plus the prefix to M_p in one forward pass, obtaining the conditional distributions p(· | x<ᵢ) at every position
Corrected rejection sampling: decide each xᵢ left to right —
- accept with probability min(1, p(xᵢ)/q(xᵢ))
- otherwise reject, resample a replacement from the corrected distribution p′(x) ∝ (p(x) − q(x))₊, and discard all subsequent drafts (where (·)₊ = max(0, ·))
Bonus token: if all γ are accepted, sample one extra token from p(· | x≤_γ) — that distribution was computed in the pass anyway, so it's free

2.2 The Key Question: Why Is the Output Exactly Equivalent to Sampling from M_p?

This is the most magical and most important property of speculative decoding: it is not "approximately lossless" — it is mathematically exact. Deeply counterintuitive, yet the proof is short. Deriving it live in an interview instantly separates "read the paper" from "understands it."

Goal: show the probability of outputting any token x is exactly p(x).

There are two paths to outputting x:

Path 1 (x is accepted):

Pr (output = x, accept) = q (x) \cdot min! (1, \frac{p ( x )}{q ( x )}) = min (p (x), q (x))

Path 2 (rejected, then x is resampled from the corrected distribution):

Pr (output = x, reject) = Pr (reject) \cdot \frac{( p ( x ) - q ( x )) +}{\sum _{y} ( p ( y ) - q ( y )) +}

Here we need one key identity:

y \sum (p (y) - q (y))_{+} = y \sum [p (y) - min (p (y), q (y))] = 1 - y \sum min (p (y), q (y)) = Pr (reject)

That is: the normalizing constant of the corrected distribution equals the rejection probability exactly. The two cancel:

Pr (output = x, reject) = (p (x) - q (x))_{+}

Combine both paths:

Pr (output = x) = min (p (x), q (x)) + (p (x) - q (x))_{+}

Check both cases:

If p(x) ≥ q(x): min = q(x), (p−q)₊ = p(x) − q(x), sum = p(x) ✓
If p(x) < q(x): min = p(x), (p−q)₊ = 0, sum = p(x) ✓

The output follows p exactly. ∎

Why this proof is beautiful: the normalizing constant Σ_y (p(y)−q(y))₊ would normally require summing over the entire vocabulary (tens of thousands of tokens). Through this identity, it cancels against the rejection probability automatically — the algorithm never has to compute it explicitly, yet the result is exactly correct. That is the elegance of corrected rejection sampling.

💡 Interview tip: learn these five lines cold and derive them live. Instantly upgrades you from "has read about it" to "understands it."

2.3 Efficiency: Acceptance Rate Sets the Speedup Ceiling

Define the acceptance rate:

α = E_{x \sim q} [min (1, p (x) / q (x))] = 1 - TV (p, q)

where TV is total variation distance. Intuitively, α measures the overlap between the target and draft distributions — the more alike they are, the higher the acceptance, the bigger the speedup.

Expected tokens per round [1]:

ParseError: KaTeX parse error: Expected 'EOF', got '#' at position 13: \mathbb{E}[#̲\text{tokens}] …

With draft-to-target cost ratio c (typically c ∈ [0.02, 0.1]), the per-round speedup is:

S (γ) = \frac{1 - α ^{γ + 1}}{( 1 - α ) ( 1 + γ c )}

Bottom line: as c → 0 with optimal γ, the speedup approaches 1 / (1 − α). vLLM reports up to 2.8× in production [3]; EAGLE-family methods reach 3–4×.

3. The Research Landscape (5 Subfields)

Once vanilla speculative decoding was established, the past two years produced several active branches. These are exactly the tech stacks that inference teams hire for.

3.1 Self-Speculation: Medusa & the EAGLE Family

Vanilla requires deploying a separate small model — real ops cost and memory overhead. Self-speculation lets the target model draft for itself:

Medusa [4]: attach K independent prediction heads after the target model's last layer, predicting tokens 1 through K ahead. Shares the backbone features; cheap to train, simple to deploy
EAGLE [5]: the core insight is that token-level drafting uncertainty mostly originates at the feature level, so speculate in hidden-state space (predict the next hidden state, then decode it to a token). Acceptance rates significantly beat token-level methods
EAGLE-2 / EAGLE-3 [6]: introduce the dynamic draft tree — current SOTA on academic benchmarks, integrated into vLLM [7]

EAGLE-family speedups are typically 3–4×, currently the mainstream production choice.

3.2 Token-Tree Verification: SpecInfer

Vanilla verifies one linear draft per round; a rejection at position 1 discards everything after it — wasteful.

SpecInfer [8] has the draft model generate multiple candidate branches organized as a tree, and the target model verifies the whole tree in one pass via custom tree attention, substantially raising expected accepted tokens. Later work (EAGLE-2 and others) adds dynamic depth adjustment.

💡 Note: SpecInfer was published at ASPLOS — a systems conference, not an ML one. That fact alone reflects the true nature of this field (see Section 4).

3.3 Multi-Token Prediction (MTP)

MTP [9, 10] takes a different route but is often discussed alongside spec decoding.

The idea is to change the training objective so the model predicts k future tokens at every position:

L MTP = - \sum i = 1^{N} j = 1 \sum k lo g P_{j} (x_{i + j} ∣ x_{\leq i})

MTP delivers two kinds of value:

High-quality drafts for spec decoding (a form of self-speculation)
Improved representation quality in the base model (empirically shown in Meta's paper) — a side benefit no other acceleration method offers

DeepSeek-V3 adopted MTP at scale [10]; it is one of the most watched directions in industry.

💡 Advanced insight: MTP and spec decoding attack different bottlenecks — spec decoding borrows the hardware dividend (Bottleneck B) to relieve the algorithmic one (A); MTP attacks A directly. If future hardware (compute-in-memory, etc.) erases B, spec decoding's dividend shrinks dramatically — but MTP still works. Articulating this distinction is a senior-level signal in interviews.

3.4 Lookahead Decoding

Lookahead Decoding [11] needs no draft model at all: view autoregressive decoding as solving a nonlinear system, update multiple positions in parallel with Jacobi iteration, and converge to output identical to standard autoregressive decoding.

Pro: zero auxiliary models, zero training.
Con: speedups usually below spec decoding (1.5–2×).

A good fit for resource-constrained deployments or teams that can't maintain an extra draft model.

3.5 Online Learning & Production Deployment

Online Speculative Decoding [12] observes that a draft model's acceptance rate drifts across query distributions — a draft trained on general text can lose half its acceptance rate on code-heavy production traffic. The fix: continuously fine-tune the draft model on live traffic during serving.

On the engineering side, vLLM Speculators [7] productizes spec-decoding training and deployment, supports mainstream variants like EAGLE-3, and is used in large production systems including Amazon Rufus and LinkedIn AI [13].

4. The Framing That Matters: This Is Software Engineering, Not Algorithmic Novelty

Understanding this makes you sound like someone who truly gets the field, not someone who just read the papers.

Step back and look at the whole area:

The model itself never changes. Vanilla spec decoding uses the target model exactly as-is — zero parameter updates, zero architecture changes
Asymptotic complexity never changes. Generating N tokens is still O(N) serial steps worst-case; only the constant factor drops severalfold
The core math is classical corrected rejection sampling — textbook sampling theory, not a new invention
The physical source of the speedup is a GPU-architecture dividend — the enormous gap between HBM bandwidth and SRAM speed opened a memory-bound window

Which implies:

It is tied to a hardware generation. If compute-in-memory (PIM/CIM) hardware erases the memory wall, the dividend shrinks dramatically
It is closer to a systems contribution. SpecInfer went to ASPLOS; vLLM went to SOSP [14] — the venues tell the story
Its algorithmic purity is limited — no new learning paradigm, no new representational capacity, no new objective

None of this diminishes its value — quite the opposite. It showcases what engineering insight does for LLM deployment: spotting wasted hardware capability and cashing it in with a clever scheme.

If you're job-hunting, this is great news. You don't need top-venue publications or deep ML theory. You need:

Solid systems knowledge (GPU architecture, memory hierarchy, concurrency)
The ability to read real code (navigate vLLM internals)
Hands-on projects with numbers (measured speedups, acceptance-rate distributions)

This is precisely the window of opportunity for strong engineers without elite-school or research pedigrees.

5. Learning Path & Interview Question Bank

5.1 Must-Read Papers (in order)

#	Paper	Focus
1	Leviathan et al. (2023) [1]	Algorithm 1 + appendix proof
2	Chen et al. (2023) [2]	Read against #1; two independent concurrent inventions
3	Cai et al. (2024) Medusa [4]	Method section
4	Li et al. (2024) EAGLE [5]	The motivation for feature-level speculation
5	Fu et al. (2024) Lookahead [11]	The Jacobi-iteration intuition

Advanced: MTP [9], the DeepSeek-V3 report [10], SpecInfer [8], EAGLE-3 [6].

5.2 Hands-On Projects (where the real differentiation happens)

Do at least one:

Run a spec-decoding demo with HuggingFace transformers: model.generate(..., assistant_model=...) — a few lines of code
Measure the acceptance rate α: swap draft models of different sizes (1B vs 3B) and watch α move
Read the vLLM spec-decoding source: github.com/vllm-project/vllm
Plot the acceptance-position distribution: which tokens get rejected? (Named entities and numbers get rejected often; "the/a/of" sail through)

One sentence in an interview — "I benchmarked the X + Y model pair, got a Z× speedup at α = …, and found that gets rejected most" — and you've pulled ahead of the pack.

5.3 Interview Question Bank

Warm-up (asked in every interview):

Explain speculative decoding in one sentence
Where does the speedup come from?
Why is the output exactly lossless? Derive the proof live
Define the acceptance rate α and what drives it

Mid-level (where most candidates get filtered):

If Transformers are parallel, why can an LLM only output 1 token per pass? (Key: each position predicts P(· | prefixᵢ))
How is the KV cache handled in spec decoding? How do rejected drafts get "rolled back" from the cache?
How do you choose the draft length γ? Why isn't bigger always better? Give the E[#tokens] formula and discuss
When does spec decoding fail to accelerate? (Large batches, low α, a draft model that's too heavy relative to the target — extremely common question)
What are Medusa's and EAGLE's core innovations, and what problems do they solve?

Senior (what separates top candidates):

The fundamental difference between spec decoding and MTP? When to use which?
If the workload becomes compute-bound (large batch, PIM hardware), does spec decoding still matter?
What does token-tree verification buy over single-branch verification?
What's the tension between speculative decoding and continuous batching?

Open-ended design:

Design a production spec-decoding system for a 70B model at 1000 QPS: draft selection, dynamic γ, memory budget
Acceptance rate is stuck at 50% — diagnose (tokenizer mismatch / domain drift / temperature / model-family mismatch)

5.4 Tricks for Interns and New Grads

Use the three key terms unprompted: memory-bound, acceptance rate α, corrected rejection sampling. Interviewers hear them and know you're real
Derive, don't recite: producing min(p,q) + (p−q)₊ = p live raises your level instantly
Volunteer the limitations: "In high-batch serving the gains shrink, because large batches are no longer memory-bound" — interviewers love hearing this
Nail spec decoding vs MTP: "one borrows the memory-bound dividend to attack Bottleneck A; the other attacks A directly" — top-candidate signal

Closing

Speculative decoding is not ML mysticism. It is a clear, learnable engineering optimization. The whole field fits in one sentence:

Paying "move 140 GB of weights" for 1 token is a terrible deal; let a small model guess several tokens, have the big model grade them all in one load, and use corrected rejection sampling to guarantee the output is exactly lossless.

Hold that thread and every variant — Medusa, EAGLE, SpecInfer, MTP, Lookahead — is just a different branch of the same tree.

I hope this pair of articles helps anyone preparing for LLM inference roles. This field genuinely isn't that hard — the key is building projects with measurable numbers.

And if you don't have an elite-school or research pedigree: this is a fully viable path. Companies desperately need engineers who understand inference optimization, and the barrier to entry here is far lower than "training large models." Work through these two articles carefully, build one or two small projects with real measurements, and you're already ahead of most candidates.

References

Academic Papers

[1] Leviathan, Y., Kalman, M., & Matias, Y. (2023). Fast Inference from Transformers via Speculative Decoding. Proceedings of the 40th International Conference on Machine Learning (ICML 2023). arXiv:2211.17192.

[2] Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., & Jumper, J. (2023). Accelerating Large Language Model Decoding with Speculative Sampling. arXiv preprint arXiv:2302.01318.

[4] Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J. D., Chen, D., & Dao, T. (2024). Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. ICML 2024. arXiv:2401.10774.

[5] Li, Y., Wei, F., Zhang, C., & Zhang, H. (2024). EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. ICML 2024.

[6] Li, Y., Wei, F., Zhang, C., & Zhang, H. (2024). EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees. arXiv preprint arXiv:2406.16858.

[8] Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Wang, Z., Wong, R. Y. Y., et al. (2024). SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification. ASPLOS 2024.

[9] Gloeckle, F., Idrissi, B. Y., Rozière, B., Lopez-Paz, D., & Synnaeve, G. (2024). Better & Faster Large Language Models via Multi-token Prediction. ICML 2024.

[10] DeepSeek-AI. (2024). DeepSeek-V3 Technical Report. arXiv preprint arXiv:2412.19437.

[11] Fu, Y., Bailis, P., Stoica, I., & Zhang, H. (2024). Break the Sequential Dependency of LLM Inference Using Lookahead Decoding. ICML 2024.

[12] Liu, X., Hu, L., Bailis, P., Stoica, I., Deng, Z., Cheung, A., & Zhang, H. (2024). Online Speculative Decoding. ICML 2024.

[14] Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023.

[15] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention Is All You Need. NeurIPS 2017. arXiv:1706.03762.

Industry Blogs

[3] vLLM Team. (2024-10-17). How Speculative Decoding Boosts vLLM Performance by up to 2.8x. https://blog.vllm.ai/2024/10/17/spec-decode.html

[7] vLLM Team. (2025-12-13). Diving into speculative decoding training support for vLLM with Speculators v0.3.0. https://blog.vllm.ai/2025/12/13/speculators-v030.html

[13] vLLM Team. (2025-01-10). vLLM 2024 Retrospective and 2025 Vision. https://blog.vllm.ai/2025/01/10/vllm-2024-wrapped-2025-vision.html

If this helped, the best way to support the series is to share it with someone prepping for inference-engineering interviews. Questions and corrections welcome.