Dead Reckoning: Resumable gRPC Server-Streaming for LLM Inference

A technical deep-dive into a thin protocol layer that turns gRPC's
server-streaming into a resumable transport — so a dropped LLM generation
continues from the last confirmed token instead of restarting from zero.

---

The problem: gRPC streaming has no checkpointing

Server-streaming gRPC is the natural fit for LLM token delivery: the server
pushes a stream of GenerateResponse messages, each carrying a token. It works
beautifully — until the stream drops. A flaky network, a proxy timeout, a
browser tab that reloads, a process that restarts. When that happens, the
client has lost every token after the last one it processed, and the only
recovery is to re-run the whole inference.

For an LLM that is uniquely bad:

1. It's expensive. You re-pay the full inference cost for tokens you already received.
2. It's slow. The user waits through the entire time-to-first-token again.
3. It's non-deterministic. Sampling means the "retry" produces a different story. The user watches the text restart and diverge.

HTTP solved the analogous problem for static resources in 1999 with Range
headers. gRPC has no equivalent for streams.

The insight: this is a checkpointing problem, not a retry problem

The key realization is that a dropped stream doesn't need to be re-done — it
needs to be resumed. If the server journals every token it produces, then a
reconnecting client only needs to say "where did I stop?" and the server can
replay from that point. That is checkpointing.

• Retry = re-run from scratch (expensive, slow, divergent).
• Checkpoint = replay from the last confirmed position (cheap, instant, bit-identical after the resume point).

gRPC already gives us the primitive we need — server-streaming with ordered
messages — but no checkpointing. So we add a thin protocol layer on top: a
monotonic token_index in every response, a resume_from_token in the
request, a server-side token journal, and periodic ACKs.

The protocol

syntax = "proto3";

package inference;

service InferenceService {
  rpc Generate(GenerateRequest) returns (stream GenerateResponse);
  rpc Ack(AckRequest) returns (AckResponse);
}

message GenerateRequest {
  string prompt = 1;
  string session_id = 2;
  int32 resume_from_token = 3;
}

message GenerateResponse {
  string token = 1;
  int32 token_index = 2;
  bool is_final = 3;
}

message AckRequest {
  string session_id = 1;
  int32 acked_token_index = 2;
}

message AckResponse {
  string session_id = 1;
  int32 acked_token_index = 2;
  int32 journaled_token_count = 3;
  int32 first_retained_index = 4;
}

Three moving parts, all on top of stock gRPC:

1. The cursor. token_index is a monotonic position in the stream. The client tracks the last one it received — its "dead reckoning" position.
2. The journal. The server stores every token it produces, keyed by session_id. On resume, it replays from the requested index with the original indices, so the client's stream is gapless and de-duplicated.
3. The ACK. During long streams the client periodically confirms its watermark with Ack. The server records it and trims journaled tokens the client no longer needs — keeping memory bounded for very long generations.

Architecture

 Browser (React + Vite)
      │  WebSocket (JSON)
      ▼
 Bridge (Express + ws) ──gRPC──▶ gRPC Server ──OpenAI-compatible──▶ LLM
      │                              │
      │                              ├── journal.ts (tokens + TTL + ACK trim)
      │                              └── handler.ts (Generate + Ack)
      └── forwards token + journal_state back to browser

The bridge exists because browsers cannot speak gRPC natively. It
translates WebSocket JSON into gRPC calls and forwards the stream back. This
keeps the protocol demonstration honest: the gRPC server is the real thing, and
the bridge is just a transport shim.

The full stack is TypeScript/Node.js across an npm-workspace monorepo:

• server/ — gRPC server (@grpc/grpc-js + @grpc/proto-loader), token journal, OpenAI-compatible wrapper, Express+WS bridge
• client/ — standalone gRPC CLI client for terminal testing
• web/ — React 18 + Vite, raw CSS
• benchmark/ — retry-vs-resume harness

The journal: the heart of it

interface JournalEntry {
  tokens: string[];      // sliding window; absolute index of tokens[i] = base_index + i
  base_index: number;    // advances as ACK-trimming drops the head
  acked_index: number;   // highest token the client confirmed via Ack
  created_at: number;
  last_accessed: number; // drives TTL eviction
}

Two memory-management mechanisms:

• ACK-aware trimming. When the client confirms it has rendered token N, the server drops everything at-or-below N. A 10,000-token generation only retains the un-ACKed tail.
• TTL eviction. Idle sessions (default 10 minutes, configurable) are swept every 60 seconds.

The journal never re-runs the LLM. It only stores and replays tokens that
were already generated. That's the whole point.

The Generate handler: resume vs. fresh

function generateHandler(journal, prompts, call): void {
  const { prompt, sessionId, resumeFromToken } = call.request;

  if (resumeFromToken > 0 && journal.has(sessionId)) {
    // 1. Replay journaled tokens from resumeFromToken with ORIGINAL indices.
    await replayJournal(journal, call, sessionId, resumeFromToken);
    // 2. Continue streaming fresh tokens from the LLM at lastIndex + 1.
    const endIndex = await streamFresh(
      journal, call, sessionId, effectivePrompt, journal.lastIndex(sessionId) + 1,
    );
    await finish(call, endIndex);
  } else {
    // Fresh inference: journal every token as it arrives.
    prompts.set(sessionId, prompt);
    const endIndex = await streamFresh(journal, call, sessionId, prompt, 0);
    await finish(call, endIndex);
  }
}

The subtle detail: on resume, the continuation starts at
journal.lastIndex(sessionId) + 1, not at resumeFromToken. That's because
the journal may have grown past the drop point (the bridge keeps the stream
alive after the browser's WebSocket closes). The replay covers the gap, then
fresh inference picks up exactly where the journal ends — so indices are always
contiguous.

The bridge: surviving the "Kill Stream" moment

The demo's "Kill Stream" button closes the browser's WebSocket abruptly. The
bridge keeps the gRPC stream alive (that's what lets the journal keep growing),
but it must not crash when it tries to send to a closed socket:

function sendToBrowser(socket: WebSocket, message: unknown): void {
  if (socket.readyState === WebSocket.OPEN) {
    socket.send(JSON.stringify(message));
  }
}

That one guard is the difference between a demo that survives a dropped client
and one that crashes on it.

The CLI client: dead reckoning in the terminal

stream.on("data", (response) => {
  if (response.isFinal) return;
  lastIndex = response.tokenIndex;
  process.stdout.write(response.token);   // inline, no newlines
  maybeAck();                              // periodic Ack
});
// on end/error:
// summary: session="demo" last_token_index=41 (resume with --resume-from 42)

The client tracks its cursor, ACKs periodically so the server can trim, and on
a drop reports the exact index to resume from. --resume-from K+1 after a drop
at token K gives a gapless stream.

The benchmark: honest numbers from a real LLM

The benchmark boots a real Dead Reckoning server and drives actual Generate
calls against the configured endpoint. For each drop point it simulates
kill-then-resume and compares retry (fresh session from 0) vs resume
(reconnect the same session from D).

Representative run against a real endpoint:

drop after	strategy	reconnect→1st token	new inference tokens
50	retry	13,326 ms	431
50	resume	5 ms	454
200	retry	13,919 ms	383
200	resume	7 ms	442
400	retry	10,116 ms	411
400	resume	8,117 ms	380 (−31)

The decisive, user-visible number is reconnect→first token: with retry the
client waits 10–14 seconds of dead air for the LLM to re-generate from scratch;
with resume the server replays the journaled gap from memory in 5–7 ms.

The inference-token column is honestly noisy — because sampling is
non-deterministic, a retry can be longer or shorter than the original. That
noise is the point of benchmarking against a real LLM. With a deterministic
backend (temperature: 0), resume's saving is exactly the drop size D tokens.

Prior art

No mainstream gRPC implementation — gRPC-core, grpc-web, or connect-rpc —
provides resumable server-streaming. Their retry policies don't apply to
server-streaming calls; channel reconnect restores the ability to make new
calls but doesn't resume an interrupted stream. The full review is in
docs/prior-art.md.

What's next (and where you can help)

The protocol design is deliberately thin and portable. The biggest production gaps, in order:

1. Persistent journal — swap the in-memory map for a WAL/SQLite/Redis store so sessions survive restarts.
2. A Cancel RPC — actually stop a generation (vs. dropping the client).
3. Multi-replica shared journal — resume from any server.
4. gRPC-web / connect-rpc clients — prove the protocol works without the bridge.

The repo is open source. Clone it, point .env at any OpenAI-compatible endpoint, and watch a 500-word story get killed mid-sentence and resume from the exact token.

Code & more: https://www.dailybuild.xyz/project/235-dead-reckoning