Most model calls end one way.
You send a request. You get an answer. Done.
Long answers are starting to break that shape.
On October 13, 2026, Google released Gemini 4 Argon. The launch post credits an "industry-leading 1M unified input-output token limit via the Gemini API" and asks developers to read up on "the new continuation protocol." The Gemini API release notes list gemini-4-argon as generally available the same day.
That protocol is the part I keep thinking about.
One request no longer has to hold the whole answer. Google's developer guide for Argon says a single request caps out at 65,536 tokens (32,768 for non-streaming direct HTTP). Past that, the API returns finish_reason=CONTINUATION and an encrypted continuation_token that contains "a hash of the original prompt and the generation state up to that point." You send it back, and the model keeps going.
The same guide is very direct about what not to do:
Don't append partial model output into
contentsor send a follow-up prompt asking the model to continue.
It also says "don't use prompt_token_count to trigger context compaction in agent harnesses."
Google's long decode page (last updated October 13), where the servers run the 64K hops for you, makes the same point: "total_input_tokens is the sum of every hop's input, not the size of your context."
The same day, Vercel's AI SDK merged PR #23263, adding gemini-4-argon with automatic continuation. It exposes initialPromptTokenCount "so context-window checks don't use the summed input tokens."
Different teams. Same warning.
A long answer is now a loop. Your token math has to know that.
So let's build a tiny version of the client side of that loop.
By the end, you'll run one command:
npx tsx continuation.ts
And watch a continuation loop get through a thinking-only segment, skip a compaction that would have fired for no reason, survive a dropped stream without duplicating text, and resume after the caller's cap runs out.
This is my small model of the idea. It is not how Gemini or the Google Gen AI SDK works inside. The model is fake, one word counts as one token, and the 40-token segment size and 1,000-token window are example inputs so the demo fits on a screen. Argon's real numbers are in the docs linked above.
Table of Contents
- The Problem
- Step 1: Requests, Checkpoints, and Usage
- Step 2: A Fake Model That Decodes in Segments
- Step 3: The Loop
- Step 4: Count the Context, Not the Bill
- Step 5: The Loops People Write First
- Step 6: Run It
- Where It Breaks Down
- The Bigger Idea
1. The Problem
A continuation protocol turns one answer into several requests.
That sounds like plumbing. It breaks four things your code used to get for free.
The request has to stay identical. The token carries a hash of the original prompt, and the guide says to resend that prompt every time.
Text can arrive before it is safe. A stream can drop halfway through a segment.
A segment can be all thinking. Empty text does not mean done.
Usage adds up in a strange way. Each continuation re-sends the prompt plus everything decoded so far, so summed input tokens grow much faster than the context does.
None of those are model bugs. They are loop bugs.
The model gives you a checkpoint. Your loop decides what counts as committed.
2. Step 1: Requests, Checkpoints, and Usage
Save the following TypeScript blocks in order as continuation.ts.
// tiny-continuation-loop: finish a long decode without lying about tokens.
// Fake model, fake tokens (1 word = 1 token). No API key. No network.
import { createHash } from "node:crypto";
type Request = { contents: string; systemInstruction: string; tools: string[] };
type Finish = "CONTINUATION" | "STOP" | "MAX_TOKENS";
type Usage = { prompt: number; text: number; thoughts: number };
type Checkpoint = { text: string; finish: Finish; token?: string; usage: Usage };
const count = (s: string) => (s.trim() ? s.trim().split(/\s+/).length : 0);
const hash = (r: Request) => createHash("sha256").update(JSON.stringify(r)).digest("hex");
const pack = (o: object) => Buffer.from(JSON.stringify(o)).toString("base64url");
const unpack = (t: string) => JSON.parse(Buffer.from(t, "base64url").toString());
const log = (s: string) => console.log(s);
A Request is the three things the guide says to resend every time: contents, system instruction, and tools.
A Checkpoint is what arrives when a request ends: text, a finish reason, maybe a token, and usage. The guide says the token and usage come back "at the end of each request."
My token is plain base64 with a hash inside. Real tokens are encrypted. The idea we need is the hash.
3. Step 2: A Fake Model That Decodes in Segments
// Each request decodes at most `perRequest` tokens, thinking first, then text.
// Text streams in chunks. The checkpoint (token + usage) arrives at the end.
class FakeModel {
requests = 0;
private dropAt = -1;
constructor(
private perRequest: number,
private thoughts: number,
private answer: string[],
) {}
dropNextStreamAfter(chunks: number): void {
this.dropAt = chunks;
}
async *stream(req: Request, maxOutput: number, token?: string): AsyncGenerator<string, Checkpoint> {
this.requests += 1;
let pos = 0;
if (token) {
const t = unpack(token);
if (t.h !== hash(req)) throw new Error("400: continuation token does not match this request");
pos = t.pos;
}
const total = this.thoughts + this.answer.length;
const end = Math.min(pos + this.perRequest, total, maxOutput);
const thoughts = Math.max(0, Math.min(end, this.thoughts) - pos);
const words = this.answer.slice(Math.max(0, pos - this.thoughts), Math.max(0, end - this.thoughts));
for (let i = 0, n = 0; i < words.length; i += 10, n += 1) {
if (n === this.dropAt) {
this.dropAt = -1;
throw new Error("stream dropped");
}
yield words.slice(i, i + 10).join(" ");
}
const prompt = count(req.contents) + count(req.systemInstruction) + pos; // re-prefill
const finish: Finish = end >= total ? "STOP" : end >= maxOutput ? "MAX_TOKENS" : "CONTINUATION";
const next = finish === "STOP" ? undefined : pack({ h: hash(req), pos: end });
return { text: words.join(" "), finish, token: next, usage: { prompt, text: words.length, thoughts } };
}
}
The fake decodes thinking first, then text, at most perRequest tokens at a time.
Three rules are borrowed from the docs on purpose.
A token whose hash does not match the request gets a 400. That is my stand-in for "the token encodes the original prompt."
The prompt count includes pos, because every continuation re-prefills what was already decoded.
If the caller's maxOutput runs out first, the finish reason is MAX_TOKENS and a token still comes back.
dropNextStreamAfter() exists only so the demo can kill a stream on purpose.
4. Step 3: The Loop
This is the most important function in the file.
type Run = {
status: "done" | "raise-cap" | "failed";
parts: string[];
token?: string;
segments: number;
billed: Usage; // sum of every request: what you pay for
initialPrompt: number; // what you sent once
error?: string;
};
// The loop. Same request every time. Only the token changes.
async function longDecode(model: FakeModel, req: Request, maxOutput: number, prev?: Run): Promise<Run> {
const frozen: Request = structuredClone(req);
const run: Run = prev
? { ...prev, parts: [...prev.parts], billed: { ...prev.billed }, status: "done" }
: {
status: "done",
parts: [],
segments: 0,
billed: { prompt: 0, text: 0, thoughts: 0 },
initialPrompt: count(frozen.contents) + count(frozen.systemInstruction),
};
let token = prev?.token;
let retries = 0;
while (true) {
const buffer: string[] = []; // uncommitted until the checkpoint arrives
let cp: Checkpoint;
try {
const it = model.stream(frozen, maxOutput, token);
let step = await it.next();
while (!step.done) {
buffer.push(step.value);
step = await it.next();
}
cp = step.value;
} catch (e) {
const msg = (e as Error).message;
if (msg === "stream dropped" && retries < 2) {
retries += 1;
log(` drop discarded ${count(buffer.join(" "))} uncommitted tokens, retry from the same token`);
continue;
}
return { ...run, status: "failed", error: msg };
}
retries = 0;
run.segments += 1;
run.parts.push(cp.text); // commit only now
run.billed.prompt += cp.usage.prompt;
run.billed.text += cp.usage.text;
run.billed.thoughts += cp.usage.thoughts;
const note = cp.usage.text === 0 ? " (thinking only, keep going)" : "";
log(` seg ${run.segments} ${cp.finish.padEnd(12)} prompt ${String(cp.usage.prompt).padStart(3)} thoughts ${String(cp.usage.thoughts).padStart(2)} text ${String(cp.usage.text).padStart(2)}${note}`);
if (cp.finish === "CONTINUATION" && cp.token) {
token = cp.token;
continue;
}
if (cp.finish === "MAX_TOKENS" && cp.token) return { ...run, status: "raise-cap", token: cp.token };
return { ...run, status: "done", token: undefined };
}
}
The important part is the sequence:
- Freeze the request once. Every segment sends the same one.
- Stream text into a
buffer, not into the answer. - Only when the checkpoint arrives, commit the buffer and add the usage.
-
CONTINUATIONwith a token: keep going, even if the text was empty. -
MAX_TOKENSwith a token: stop and hand back the token so the caller can raise the cap. - A dropped stream: throw away the buffer and retry from the same token.
That last rule is the one I like most. The previous token is still valid, so the retry starts from the last committed checkpoint. Nothing uncommitted ever reached the answer.
prev lets the caller resume with a higher cap. The Vercel PR describes the same move: pass the token back "with a higher maxOutputTokens to resume."
5. Step 4: Count the Context, Not the Bill
Two tiny functions.
// What actually sits in the context window after the decode.
const contextSize = (r: Run) => r.initialPrompt + r.billed.text + r.billed.thoughts;
const shouldCompact = (tokens: number, window: number) => tokens >= window * 0.8;
run.billed.prompt is the sum of every request's input. That is closer to what you pay for. It is not what sits in the window.
What sits in the window is the prompt you sent once, plus the thoughts and text that came back.
The guide points harnesses at initial_prompt_token_count for exactly this, and says to wait until the decode finishes before compacting. The 80% threshold is an example.
6. Step 5: The Loops People Write First
Two loops I can picture myself writing on a tired afternoon.
// The two loops people write first.
async function naiveDecode(model: FakeModel, req: Request, maxOutput: number, mode: "ask-to-continue" | "keep-partials"): Promise<string> {
let r = req;
let token: string | undefined;
let text = "";
for (let attempt = 0; attempt < 8; attempt += 1) {
try {
const it = model.stream(r, maxOutput, token);
let step = await it.next();
while (!step.done) {
text += (text ? " " : "") + step.value; // straight into the answer
step = await it.next();
}
if (step.value.finish === "STOP") return text;
token = step.value.token;
if (mode === "ask-to-continue") r = { ...req, contents: `${req.contents} ${text} Please continue.` };
} catch (e) {
log(` naive error: ${(e as Error).message}`);
if (mode === "ask-to-continue") return text;
}
}
return text;
}
ask-to-continue does the thing the guide says not to do. It pastes the partial output into the prompt and adds "Please continue." That changes the request, so the token stops matching.
keep-partials streams chunks straight into the answer. When a stream drops, it retries from the same token and appends the segment again.
Both look reasonable. Both only break on long outputs.
Then the demos:
const files = Array.from({ length: 150 }, (_, i) => `src/billing/file${i}.ts`).join(" ");
const req: Request = {
contents: `Plan a safe migration for these files: ${files}`,
systemInstruction: "You are a careful refactoring agent.",
tools: ["read_file", "run_tests"],
};
const answer = (
"Move the shared money helpers into packages/money first and keep the old imports as re-exports. " +
"Then migrate invoices, refunds, and tax in that order, because refunds import invoices and tax imports both. " +
"Run the contract tests after each package, not at the end. " +
"Keep the feature flag off until the refund snapshots match byte for byte. " +
"Delete the re-exports only after one full release with zero imports from the old path. " +
"If any step fails, stop, revert that package alone, and report which test broke and why."
).split(" ");
const WINDOW = 1000; // example context window for the demo, not Argon's
const model = () => new FakeModel(40, 60, answer);
const joined = (r: Run) => r.parts.filter(Boolean).join(" ");
const same = (text: string) => (text === answer.join(" ") ? "matches answer" : "WRONG");
async function main(): Promise<void> {
log("=== Demo 1: the loop, through a thinking-only segment ===");
const r1 = await longDecode(model(), req, 400);
log(` done ${r1.segments} segments, ${count(joined(r1))} text tokens, ${same(joined(r1))}`);
log(` billed prompt (sum of requests) ${r1.billed.prompt}, compact? ${shouldCompact(r1.billed.prompt, WINDOW)}`);
log(` actual context ${r1.initialPrompt} + ${r1.billed.thoughts} + ${r1.billed.text} = ${contextSize(r1)}, compact? ${shouldCompact(contextSize(r1), WINDOW)}`);
log("\n=== Demo 2: 'please continue' changes the request ===");
const n2 = await naiveDecode(model(), req, 400, "ask-to-continue");
log(` naive stopped with ${count(n2)} of ${answer.length} text tokens`);
log("\n=== Demo 3: the stream drops mid-segment ===");
const [m3, m3b] = [model(), model()];
m3.dropNextStreamAfter(2);
m3b.dropNextStreamAfter(2);
const n3 = await naiveDecode(m3, req, 400, "keep-partials");
log(` naive ${count(n3)} text tokens for an ${answer.length}-token answer`);
const r3 = await longDecode(m3b, req, 400);
log(` loop ${count(joined(r3))} text tokens, ${same(joined(r3))}, ${m3b.requests} requests`);
log("\n=== Demo 4: the caller's cap runs out first ===");
const m4 = model();
const r4 = await longDecode(m4, req, 100);
log(` status ${r4.status} after ${count(joined(r4))} text tokens, token kept`);
const r4b = await longDecode(m4, req, 200, r4);
log(` resume ${r4b.status}, ${count(joined(r4b))} text tokens, ${same(joined(r4b))}`);
}
main();
The prompt is a list of 150 made-up file paths. The answer is a short migration plan I wrote by hand. Both are example inputs.
7. Step 6: Run It
Install and run:
npm install
npx tsx continuation.ts
Example output (example inputs only):
=== Demo 1: the loop, through a thinking-only segment ===
seg 1 CONTINUATION prompt 163 thoughts 40 text 0 (thinking only, keep going)
seg 2 CONTINUATION prompt 203 thoughts 20 text 20
seg 3 CONTINUATION prompt 243 thoughts 0 text 40
seg 4 STOP prompt 283 thoughts 0 text 27
done 4 segments, 87 text tokens, matches answer
billed prompt (sum of requests) 892, compact? true
actual context 163 + 60 + 87 = 310, compact? false
=== Demo 2: 'please continue' changes the request ===
naive error: 400: continuation token does not match this request
naive stopped with 0 of 87 text tokens
=== Demo 3: the stream drops mid-segment ===
naive error: stream dropped
naive 107 text tokens for an 87-token answer
seg 1 CONTINUATION prompt 163 thoughts 40 text 0 (thinking only, keep going)
seg 2 CONTINUATION prompt 203 thoughts 20 text 20
drop discarded 20 uncommitted tokens, retry from the same token
seg 3 CONTINUATION prompt 243 thoughts 0 text 40
seg 4 STOP prompt 283 thoughts 0 text 27
loop 87 text tokens, matches answer, 5 requests
=== Demo 4: the caller's cap runs out first ===
seg 1 CONTINUATION prompt 163 thoughts 40 text 0 (thinking only, keep going)
seg 2 CONTINUATION prompt 203 thoughts 20 text 20
seg 3 MAX_TOKENS prompt 243 thoughts 0 text 20
status raise-cap after 40 text tokens, token kept
seg 4 CONTINUATION prompt 263 thoughts 0 text 40
seg 5 STOP prompt 303 thoughts 0 text 7
resume done, 87 text tokens, matches answer
Demo 1 is the happy path, with one trap in it.
Segment 1 is all thinking. Zero text. A loop that treats empty text as done quits right there. This one keeps going and gets the full answer in four segments.
Then the counting. Summed prompt tokens are 892, past the example 80% line on a 1,000-token window. The real context is 310. Compacting there throws away good context for nothing.
Demo 2 is "please continue." One thinking-only segment, an edited prompt, and the next request is rejected. Zero text.
Demo 3 is a dropped stream. The naive loop ends with 107 tokens for an 87-token answer. The real loop discards 20 uncommitted tokens, retries from the same token, and matches exactly, for one extra request.
Demo 4 is the cap. At 100 tokens the loop stops with raise-cap and keeps the token. A second call with a cap of 200 picks up where it left off.
That is the whole feature.
8. Where It Breaks Down
The fake is too polite. Real segments are thousands of tokens, and real failures are not labelled "stream dropped." Your retry rule needs real error codes.
A retry is not free. In Demo 3 the dropped segment gets decoded twice. I don't know how a real dropped stream is billed, so check your usage reports, not my fake.
The fake ignores caching. Google's long decode docs describe a 95% implicit cache discount on repeated hop input. The 892 here is a token count, not a price.
Time is missing. The guide lists a 20 minute limit per request. A real loop needs timeouts, cancellation, and visible progress.
The SDK may already do this. The Google Gen AI SDK follows continuation_token automatically by default, and Vercel's AI SDK now does too. Write your own loop only if you need your own retries. Either way, check which token count your harness reads.
Committing late costs latency. You can stream to the screen and still commit at the checkpoint, as long as you can take the uncommitted part back.
9. The Bigger Idea
request (frozen)
└→ segment
├→ stream into buffer
├→ dropped? discard buffer, retry same token
└→ checkpoint arrives
├→ commit buffer, add usage
├→ CONTINUATION + token: next segment
├→ MAX_TOKENS + token: raise the cap, resume
└→ STOP: join parts, then check context
context = initial prompt + thoughts + text
bill = every request's input + output
The model provides the reasoning.
The token provides the resume point.
The frozen request provides the match.
The buffer provides the honesty.
The counter provides the truth about the window.
This part is prediction, not history.
I think long decode quietly changes what an "LLM call" means in agent harnesses. It used to be one round trip. Now it can be a small state machine with its own retries and its own accounting.
Google and Vercel both putting the same counting warning in their docs on day one is the right instinct. It makes the trap visible.
The rest is on us.
A long answer is a loop. Treat the checkpoint as the truth.
Software should explain itself.
I'm building Helix around this idea: every change should come with the why. When a model spends a long decode planning a migration across hundreds of files, the plan, the evidence, and the reasons should not disappear when the session ends.
What critical engineering knowledge is your team losing right now?
Top comments (0)