Gemini's Thinking Budget Parameter

#ai #machinelearning #llm #programming

Gemini’s thinking models spend tokens working through a problem before they answer. thinkingBudget caps that spend. The consequence people arrive here for is that the cap and your answer draw on the same allowance, so a badly chosen pair of numbers returns a response with no text in it.

Where the field sits

It is nested two levels inside generationConfig, which is easy to get wrong because most sampling settings are one level in:

{
  "contents": [
    { "role": "user", "parts": [{ "text": "Reconcile these two ledgers and list every discrepancy." }] }
  ],
  "generationConfig": {
    "maxOutputTokens": 8192,
    "thinkingConfig": {
      "thinkingBudget": 4096,
      "includeThoughts": true
    }
  }
}

A misplaced thinkingBudget at the top of generationConfig is generally ignored rather than rejected, so the symptom is a model that keeps thinking as much as it likes and a setting that appears to do nothing. Check the nesting first.

The values, including the two special ones

thinkingBudget is a token count, and two values do not mean what a plain count would, per Google’s thinking documentation:

-1 — dynamic thinking. The model decides how much to spend based on the difficulty of the request. This is the default behaviour for the 2.5 models when you set nothing.
0 — thinking disabled, on models that permit it. Gemini 2.5 Flash accepts zero; Gemini 2.5 Pro does not, and enforces a non-zero minimum instead. Sending zero to a model that requires thinking is an INVALID_ARGUMENT, not a silent clamp.
A positive integer — an upper bound. It is not a target: the model may spend less, and often does on easy prompts.

The permitted range differs by model, and the difference between the Flash and Pro rules is the one that produces errors in practice. Flash allows the budget to be turned off; Pro has a floor above zero and a ceiling of its own.

The exact minimum and maximum budget per model are documented in Google’s thinking guide and have changed across the 2.5 series. Read the range for the model you are calling rather than hard-coding a value that was valid for a preview build—an out-of-range budget fails the request outright.

The split that produces an empty answer

This is the mechanism worth carrying away. Thinking tokens are output tokens. They are generated by the model, they are billed at the output rate, and—critically—they are drawn from the same maxOutputTokens allowance as the answer.

The response reports them separately so you can see the split:

"usageMetadata": {
  "promptTokenCount": 1204,
  "thoughtsTokenCount": 3871,
  "candidatesTokenCount": 402,
  "totalTokenCount": 5477
}

thoughtsTokenCount is the reasoning spend; candidatesTokenCount is the visible answer. Now trace what happens when the allowance is tight:

maxOutputTokens = 1024
thinkingBudget  = 1024

  Model thinks. Thinking consumes output allowance.
  After 1024 thinking tokens the output allowance is exhausted.

Response:
  usageMetadata.thoughtsTokenCount    = 1024
  usageMetadata.candidatesTokenCount  = 0
  candidates[0].finishReason          = "MAX_TOKENS"
  candidates[0].content               = absent, or parts with no text

You are billed 1024 output tokens for an empty response.

That is the whole of the “Gemini returned nothing and charged me for it” report. It is not a bug and it is not a safety block— the finishReason is MAX_TOKENS, which distinguishes it cleanly from the block-related values. The fix is arithmetic: maxOutputTokens must exceed thinkingBudget by at least the length of the answer you want.

A workable rule, stated as an assumption rather than a measurement: size the answer first, then add the budget on top. maxOutputTokens = expected_answer_tokens + thinkingBudget, with headroom, and treat a run of MAX_TOKENS responses with candidatesTokenCount near zero as the signal that the sum is wrong. Because dynamic thinking (-1) has no bound you can predict, pair an unbounded budget with a generous ceiling or with a fixed budget—not with a tight one.

Seeing what it thought

includeThoughts: true adds thought summaries to the response. They arrive as ordinary parts flagged with thought: true, interleaved before the answer parts:

"parts": [
  { "thought": true, "text": "Both ledgers use different date conventions; normalise first..." },
  { "text": "Three discrepancies were found. 1) Invoice 4471 appears twice..." }
]

Two things about this. First, they are summaries, not the raw reasoning trace—you are seeing a condensed account of what the model did, not its internal tokens verbatim. Second, and this is the part that breaks display code, any part with thought: true must be filtered out of what you show as the answer. An SDK helper that concatenates all text parts will splice the reasoning into the middle of the response, which reads as the model talking to itself in production.

answer = "".join(
    part.text
    for part in response.candidates[0].content.parts
    if part.text and not getattr(part, "thought", False)
)

If you are building a multi-turn tool loop, there is a further consideration: thought signatures returned with a response are meant to be sent back with the following turn so the model can resume its own reasoning across a tool call. Dropping them when you rebuild the conversation history is a quiet quality regression rather than an error.

What the budget does to latency

Reasoning tokens are generated one at a time, exactly like answer tokens, at roughly the same rate. That single fact determines the latency behaviour and it is worth stating as arithmetic rather than as an impression:

Perceived time to first visible token
  = prefill time
  + (thinking tokens spent / tokens per second)

A budget of 8,192 thinking tokens, at a generation rate of a few
hundred tokens per second, is therefore tens of seconds of silence
before the first character of the answer appears.

The consequence for streaming is the one people are unprepared for. Streaming a thinking model does not give you an early first token: it gives you the prefill wait, then the whole thinking period with nothing on the wire, then the answer streaming normally. The perceived latency win that makes streaming worthwhile for a non-reasoning model is largely cancelled by the budget.

Two things address it, and they are different in kind. includeThoughts gives you something to display during the silence—thought summaries arrive as they are produced, so a “working on it” surface can show real content rather than a spinner. And an explicit budget converts an unbounded wait into a bounded one, which is what makes a timeout defensible: with dynamic thinking you cannot state a worst case, and with a fixed budget you can compute one.

The infrastructure consequence follows from the same silence. An idle timeout anywhere between your process and the API—a load balancer, a serverless execution limit, a default HTTP client setting—can cut a connection during the thinking period, because from the outside a working reasoning model and a hung request look identical. Timeouts sized against non-reasoning traffic are a common cause of requests that fail at a suspiciously consistent number of seconds.

Choosing a budget

Zero, where the task is retrieval or formatting. Classification, extraction, rewriting and JSON shaping do not benefit from reasoning tokens and pay for them at the output rate. On Flash, turning thinking off is the largest single cost reduction available for these workloads.
A fixed budget, where latency has a ceiling. Dynamic thinking makes worst-case latency unbounded from your side. A hard budget converts an unpredictable tail into a predictable one.
Dynamic, where difficulty varies and correctness is the product. Mixed-difficulty traffic is exactly what dynamic thinking is for—an easy request spends little.
Watch the ratio, not the total. thoughtsTokenCount divided by candidatesTokenCount, per request type, tells you where you are buying reasoning you do not need. It is in every response already.

Reasoning-token accounting is the one place cross-provider cost tracking usually goes wrong, because each API reports it under a different name—Gemini in usageMetadata.thoughtsTokenCount, and other providers under their own reasoning-token fields, all billed as output. If you route the same feature across more than one, a gateway that normalises usage reporting means your cost-per-request figure does not silently exclude the tokens you are actually spending most on.