o1's Context Window and the Reasoning Tokens Eating Into It

#ai #llm #machinelearning #programming

o1 has a 200,000-token context window and a 100,000-token cap on a single response. The second number looks absurdly generous until you learn what shares it: the reasoning the model does before answering is counted against that same 100,000, and you never see a token of it.

The two numbers

OpenAI documents context length and maximum output tokens per model on its models reference. For the o-series at the time of writing:

o1 — 200,000-token context window, 100,000 max output tokens.
o1-mini — 128,000-token context window, 65,536 max output tokens.
o1-preview, the original September 2024 release — 128,000-token context window, 32,768 max output tokens.

These are per-snapshot figures and the o-series has moved quickly. Read the current values off the models page for the exact model string you send; the mechanism below is what does not change.

The relationship between the two is the same as on any other model: the window is the total, and input plus output must fit inside it. What is different is what “output” contains.

The third consumer nobody budgeted for

A reasoning model does work before it answers. That work is generated as tokens, in the model’s own space, and OpenAI does not return it — the API exposes only a count. In the usage object it appears here:

"usage": {
  "prompt_tokens": 1420,
  "completion_tokens": 4680,
  "total_tokens": 6100,
  "completion_tokens_details": {
    "reasoning_tokens": 4224
  }
}

Read those numbers carefully, because the containment is the whole point. completion_tokens is 4,680. Of that, reasoning_tokens is 4,224. The visible answer is the difference: 456 tokens. Reasoning tokens are not an additional line item alongside the completion — they are inside it. They count against your max_completion_tokens, they count against the context window, and they are billed at the output rate, which is the subject of reasoning-token billing.

This is also why the parameter was renamed. On GPT-4o, max_tokens bounded the thing you got back. On o1 there are two different things that could mean, so the parameter is max_completion_tokens and it bounds the sum. Passing the old max_tokens to a reasoning model is rejected rather than reinterpreted.

It is worth being precise about what “you never see them” covers, because two different things get conflated. The tokens are not returned in the response — there is no field carrying the reasoning text, only the count. Separately, OpenAI has at times exposed a human-readable summary of the reasoning as an opt-in, which is a generated description of the process rather than the tokens themselves. A summary does not let you audit the 4,224 tokens above; it is a different artefact with its own cost. Neither changes the accounting.

The arithmetic, worked

Take a request with a large retrieved context — say a 40,000-token document plus a 2,000-token instruction — against o1 with a 200,000 window.

context window                  200,000
  prompt (document + system)   − 42,000
                               ─────────
  available for completion      158,000

but the per-response cap is    100,000
so the binding limit is        100,000

within that 100,000:
  reasoning tokens              variable, model-chosen
  visible answer                whatever is left

observed on one request:
  reasoning_tokens               12,400
  visible answer                    900
  completion_tokens              13,300   (= 12,400 + 900)

Now set max_completion_tokens: 1000 because you only want a 900-token answer and you are being careful with cost. The model spends the first 1,000 tokens of its budget reasoning, hits the ceiling, and the request ends. You get back a response with finish_reason: "length", an empty content, and a bill for 1,000 output tokens. Nothing failed loudly. You paid for reasoning that produced nothing.

The general shape is:

max_completion_tokens  ≥  expected_reasoning  +  desired_answer

where expected_reasoning is not under your control and varies
per prompt — hard problems reason longer.

OpenAI’s reasoning guide addresses this directly by recommending a substantial reserve — the documented advice has been to leave on the order of 25,000 tokens of headroom for reasoning while you are learning how a given prompt behaves, then tune down once you have seen real reasoning_tokens values for your traffic. The reserve is not money spent: unused budget is not billed, only generated tokens are. A large max_completion_tokens on a reasoning model is a safety margin, not a cost.

The empty-response failure

This is the failure worth recognising on sight, because it does not look like a limit problem:

{
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": "" },
      "finish_reason": "length"
    }
  ],
  "usage": {
    "completion_tokens": 1000,
    "completion_tokens_details": { "reasoning_tokens": 1000 }
  }
}

Empty content, HTTP 200, non-zero cost. Every token of the budget went into reasoning and the model never reached the answer. If your client treats empty content as “the model had nothing to say” and retries with the same parameters, it will do the same thing again at the same price. The check is one line: when finish_reason is "length" and reasoning_tokens is close to completion_tokens, the budget was the problem, and the fix is a larger one rather than a retry.

Budgeting the window

Three practical consequences follow from reasoning and answer sharing one budget.

Reasoning tokens do not persist between turns. They are discarded from the conversation once the response completes, so a multi-turn chat does not accumulate them in its history — you are not slowly filling the window with invisible text. You pay for them once, per turn.
Longer prompts squeeze reasoning. Prompt tokens and completion tokens share the 200,000. A 190,000-token prompt leaves 10,000 for everything the model does next, reasoning included, and no value of max_completion_tokens can conjure more.
Where the effort is tunable, it is the real lever. The o-series exposes a reasoning-effort setting whose low, medium and high values change how many reasoning tokens the model spends. Lowering it reduces both latency and bill on tasks that do not need deep deliberation, which is a better instrument than clamping max_completion_tokens and hoping.

The message structure differs too — the o-series took the developer role in place of system, and the earliest models supported neither. That is covered in o1 and system message support, and the reasoning behind the rename in the developer role.

One last asymmetry, because it changes how a long context should be filled. On a non-reasoning model, a very long prompt is mostly a latency and price problem: prefill is a single parallel pass and its cost is predictable. On a reasoning model the prompt also sets how much there is to reason about, and reasoning token counts tend to rise with the size and ambiguity of what was given. So padding the window with marginally relevant retrieved documents is doubly expensive here — you pay for the input tokens, and then you pay again, at the output rate, for the model working through material that did not need to be there. Retrieval precision is a cost control on these models in a way it is not on GPT-4o, whose window arithmetic is set out in what 128K buys you.