Gemini spent 341 tokens thinking. LangChain reported five.

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

If you bill customers from LangChain's token counts and your users talk to Gemini, check your numbers. On a reasoning call they are wrong for certain. On a streamed call they are wrong whatever the model.

Sentry told me a Gemini call used 46 tokens. The call used 387. The other 341 were thinking tokens — billed by Google, buried inside total_tokens, and named by no field LangChain returned.

The shape is always the same: a value computed correctly, then dropped on the way out. Nothing throws. Nothing logs. The number that reaches your dashboard is usually a plausible number, which is exactly why this survived. I found it four times, across two packages and both of the channels LangChain reports usage on.

Pull requests: #11423 and #11425 Issues: #11422 and #11424

One note on the numbers below: Gemini's thinking budget is not deterministic, so exact counts shift between calls. What never shifts is that a gap opens the moment thinking is on. Every figure I quote comes from a run I screenshotted, and where two figures come from different calls I say so.

Project Overview

@langchain/google-genai is the LangChain integration for Google's Gemini models. If you use ChatGoogleGenerativeAI in a TypeScript app, this is the package you installed.

Token counts are not decoration in that stack. They are how you bill customers, enforce quotas, fill dashboards, and notice when a prompt change quietly triples your spend. LangChain exposes them two ways: usage_metadata on the message, and llmOutput for the callback system that observability tools hook into.

Gemini 2.5 and 3 models think before answering, and they charge for it. The API reports that separately as thoughtsTokenCount. That number is the whole story here.

Bug Fix or Performance Improvement

What is broken

Ask gemini-2.5-flash a question that requires reasoning, with thinking enabled, and read usage_metadata:

{ "input_tokens": 41, "output_tokens": 5, "total_tokens": 387 }

41 plus 5 is 46. The object says 387. The 341-token difference is named by no field in the object.

You do not need to know anything about LangChain to see the problem. The object contradicts itself.

Any cost tracker reading output_tokens sees 5 for a call that produced 346 output tokens.

Before. The probe asserts input + output == total and raises when it fails. This is a silent bug forced to announce itself — the issue title is the entire defect.

Root cause

convertUsageMetadata in libs/providers/langchain-google-genai/src/utils/common.ts builds the result from four fields:

const output: UsageMetadata = {
  input_tokens: usageMetadata?.promptTokenCount ?? 0,
  output_tokens: usageMetadata?.candidatesTokenCount ?? 0,
  total_tokens: usageMetadata?.totalTokenCount ?? 0,
};

thoughtsTokenCount is never read. It does not appear anywhere in the package.

The reason it was missed is worth knowing. This package still depends on the legacy @google/generative-ai SDK, whose UsageMetadata interface declares only four fields. thoughtsTokenCount is not one of them. TypeScript never complained, because you cannot read a property the type says does not exist.

It arrives at runtime anyway. This is the payload from a direct call to the same API with the same prompt — a separate call, so its thinking count is its own:

{
  "promptTokenCount": 41,
  "candidatesTokenCount": 5,
  "totalTokenCount": 427,
  "promptTokensDetails": [{ "modality": "TEXT", "tokenCount": 41 }],
  "thoughtsTokenCount": 381,
  "serviceTier": "standard"
}

Three of those fields are undeclared, including the one that matters. The type was the blindfold.

Why the obvious fix is worse than no fix

Reading thoughtsTokenCount in convertUsageMetadata seems like the whole job. It is not, and stopping there breaks streaming.

Gemini reports usage cumulatively. Every chunk carries the running total for the whole request. The provider already knows this, which is why _streamResponseChunks converts input_tokens, output_tokens and total_tokens into per-chunk deltas after convertUsageMetadata returns. But it leaves nested detail fields cumulative, and mergeOutputTokenDetails in @langchain/core sums reasoning across chunks.

So a three-chunk stream carrying cumulative thoughts of 100, 250 and 381 does this:

100 + 250 + 381 = 731

The true value is 381. The converter-only fix reports 731. It does not merely fail to fix the bug, it creates a 92% overcount, which is the same double-counting class as #8266 — an issue that sat open for over a year describing exactly this failure.

There is a test in the PR that fails if you patch only the converter.

Code

Both fixes are open pull requests against langchain-ai/langchainjs:

• #11423 — reasoning tokens dropped in @langchain/google-genai. +284 / -8 across 5 files, with 10 new unit tests.
• #11425 — llmOutput token usage wrong on both streaming paths, touching @langchain/core and @langchain/google-genai. +343 / -26, with 7 new unit tests.

They close #11422 and #11424, both of which I filed with reproductions.

These are open pull requests against langchain-ai/langchainjs. #11425 has already drawn review from another contributor; #11423 is open and unreviewed as I write this. Not a fork, not a patch in my own repo — if they merge, the fix reaches every consumer of @langchain/google-genai. It is one of several packages in LangChain JS that reach the Gemini Developer API, and the only one that gets reasoning tokens wrong.

The changes are walked through below.

My Improvements

Three changes, all in #11423.

1. Read the field. Defensively, since the legacy type does not declare it:

const thoughtsTokenCount = getThoughtsTokenCount(usageMetadata);
const candidatesTokenCount = usageMetadata?.candidatesTokenCount ?? 0;

const output: UsageMetadata = {
  input_tokens: usageMetadata?.promptTokenCount ?? 0,
  // Gemini reports reasoning separately from candidatesTokenCount, while
  // output_tokens is defined as the sum of all output token types.
  output_tokens: candidatesTokenCount + thoughtsTokenCount,
  total_tokens: usageMetadata?.totalTokenCount ?? 0,
};

if (thoughtsTokenCount) {
  output.output_token_details ??= {};
  output.output_token_details.reasoning = thoughtsTokenCount;
}

@langchain/google-common and @langchain/google - which back the Vertex packages and also reach the Gemini Developer API - already read thoughtsTokenCount this way. @langchain/google-genai, the standalone Developer-API package built on the legacy SDK, is the one that does not. So this is two packages agreeing with a third, not new behaviour being proposed.

2. Convert reasoning to a delta while streaming, so concatenating chunks does not sum the running totals.

3. An unrelated bug in the same function. While reading it I found this:

if (model === "gemini-3-pro-preview") {
  const over200k = Math.max(0, usageMetadata?.promptTokenCount ?? 0 - 200000);

?? binds looser than -, so 0 - 200000 evaluates first. The line actually reads promptTokenCount ?? -200000, which for any real prompt is Math.max(0, promptTokenCount). On gemini-3-pro-preview, a 500-token prompt reported 500 tokens of overage past 200k. The next line has the same problem. The model gate is probably why nobody had hit it yet.

The tests

Ten unit tests, no network required. Revert the fix and eight of them fail with the exact wrong numbers — the other two are regression guards that have to keep passing either way:

AssertionError: expected undefined to be 381
AssertionError: expected 5 to be 386
AssertionError: expected 46 to be 427
AssertionError: expected 500 to be undefined
AssertionError: expected 250000 to be 50000

Failed Tests 8
Test Files  2 failed (2)

expected 46 to be 427 is the same arithmetic failure my assertion caught on a live call and Sentry carried, reproduced offline with no API key.

The same arithmetic failure the probe caught on a live call, reproduced offline with no API key. Revert the fix and these eight come back.

Full package suite passes: 103 tests, no type errors, oxlint and oxfmt clean.

The same bug, twice more in the callback path

Once I knew the shape, I went looking. Two more, both in what the callback system reads — and, as it turns out, one I missed.

Two. _generate returns llmOutput: { estimatedTokenUsage: tokenUsage } on its streaming branch, where tokenUsage is declared {} at the top of the branch and returned at the bottom without ever being written to. Wrong key too. The non-streaming path of the same class returns tokenUsage.

Three. _streamIterator in @langchain/core reassigns llmOutput on every chunk, so the surviving value is the last chunk's usage. Correct for providers that report cumulative totals on a final chunk. Wrong for providers emitting deltas, where the survivor is the last delta.

Measured across all three call paths, with #11423 already applied so only the llmOutput bug is visible:

call path	what the callback receives	true usage
invoke()	{ 41, 366, 407 }	407
invoke() with streaming: true	estimatedTokenUsage: {}	401
.stream()	{ 0, 3, 3 }	458

Three tokens reported for a 458-token call. That one is worse than the empty object, because an empty object looks broken and gets noticed. 3 renders fine on a dashboard.

Both are fixed in #11425.

Best Use of Sentry

Sentry is why I found instances two and three. Not as a monitoring tool, but because instrumenting the code forces one question: what does the observability layer actually read?

The answer for LangChain in JavaScript is llmOutput.tokenUsage, with an Anthropic-shaped llmOutput.usage fallback and nothing after that. No usage_metadata fallback, unlike the Python SDK, which checks the message first. So I went to look at what llmOutput contained, and it was wrong in two different ways on two different paths.

Before and after

I built a probe that makes the same call three ways, tagged the runs before and after, and pointed it at a local patched build.

One trace, one prompt. generate_content is Google's own SDK; the two chat spans are LangChain measuring the same work. The correct count and the wrong count, side by side.

Same span, same prompt, same response. The output count goes from 5 to 371, and every total downstream follows it:

Before: 41 in, 5 out, 46 total — and look at the Cost line. Sentry priced the call from the undercounted tokens. Every number downstream inherits it: cost, context utilization, budget alerts.

After: 41 in, 371 out, 412 total. Same prompt, same response, correct bill.

That is the difference between a missing field and wrong billing, and Sentry is the one saying it.

The response, by the way, was $0.05. Three characters — and on that call, 366 of the 371 output tokens were the model thinking.

Making a silent bug visible

Token miscounting throws nothing. So the probe asserts the arithmetic and raises an error when it fails.

The wrong accounting and the ground truth, six milliseconds apart — LangChain reporting 46 while the raw SDK reports 381 thinking tokens on the same prompt.

After the fix, nothing to report. The assertion that fired on every reasoning call now never fires.

What Seer made of it

I pointed Seer at the issue without telling it anything about the bug — no source access to my fork, no hint about thoughtsTokenCount. All it had was the error and the breadcrumbs.

It got the root cause right:

LangChain's Gemini integration excluded thoughtsTokenCount from output_tokens, causing input+output ≠ total when the model uses extended thinking.

And the mechanism, unprompted: that output_tokens was being set to candidatesTokenCount alone, that totalTokenCount includes thinking tokens, and that the gap should equal thoughtsTokenCount from the raw response. Its reproduction steps are the ones I had written by hand a day earlier.

Two things it got wrong, and they are the interesting part. It named the function responseToUsageMetadata; the real one is convertUsageMetadata. And its evidence pointer — gemini.ts L1235–L1274 — is not the broken code at all. That file is @langchain/google-common, the sibling package, and those exact lines read:

const thoughtsTokenCount = usageMetadata.thoughtsTokenCount ?? 0;
const output_tokens = candidatesTokenCount + thoughtsTokenCount;

That is the correct implementation. It is what my fix makes @langchain/google-genai do.

So Seer reasoned to the right diagnosis and then cited the one file in the repo that already does it properly. Which is what you would expect from a tool working backwards from telemetry rather than from the repository: the breadcrumbs told it what was wrong, and nothing in them could tell it which of several similarly-named packages the call had gone through.

That is still the argument for making a silent bug loud. Give an RCA tool one contradiction to hold onto and it can walk back to the cause from a stack trace and eight console lines. Just check where it points before you believe it.

The agent view

The same run, with the span's Output tab open. Because recordInputs and recordOutputs are on, Sentry captured the model's actual answer — $0.05 — and sits it in the same panel as the token line that says the call cost 46 tokens. A three-character response and a four-hundred-token bill, in one panel, disagreeing with each other. That juxtaposition is what makes the undercount legible as a cost rather than a stray metric.

One thing worth knowing

Sentry's automatic AI instrumentation patches modules as they load, so it only works if Sentry.init() runs before the AI package is evaluated. Under ESM every static import is hoisted, so calling init() in the same file that imports LangChain is already too late. Sentry documents the fix - put init in its own file and run node --import ./instrument.mjs app.mjs - and their own CI exercises exactly that for these integrations.

I was running through tsx, and even with --import the loader hooks did not take for me: my first runs produced only http.client spans and no gen_ai spans at all, and I nearly wrote the wrong conclusion from it. Attaching the instrumentation by hand is the reliable escape hatch:

const handler = Sentry.createLangChainCallbackHandler({
  recordInputs: true,
  recordOutputs: true,
});
const ai = Sentry.instrumentGoogleGenAIClient(new GoogleGenAI({ apiKey }));

If your AI spans are missing on ESM, check that you are loading Sentry via --import before concluding anything - and if you are running through a custom loader like tsx, try attaching by hand before blaming the SDK.

Sentry also filtered the console breadcrumbs carrying the token JSON, marking them [Filtered]. Its scrubber saw the word "token" and did what it should.

Best Use of Google AI

Two models agreeing is not evidence — both can be wrong the same way. So the ground truth in this work is Google's own SDK, called directly.

@google/genai types thoughtsTokenCount properly. The legacy @google/generative-ai SDK that LangChain still depends on does not declare it at all. That gap is why nobody caught it, and it is also how I proved it: run the same prompt through both, and read what each one admits to.

The raw SDK's object closes its own arithmetic. LangChain's does not:

raw @google/genai   41 prompt + 5 candidates + 0 tool-use + 381 thoughts = 427 = totalTokenCount   OK
LangChain           41 input  + 5 output     +         (missing)         = 46  != 387              FAIL

That is what establishes thinking tokens as a separate bucket rather than something folded into candidatesTokenCount. Without it, "output_tokens should include reasoning" is an opinion about API semantics. With it, it is arithmetic, and the PR stops being a proposal and becomes a correction.

The controlled version

Thinking budgets are not deterministic, so a single pair of calls proves less than it looks. I ran three prompts twice each — thinking on, thinking off — through both SDKs:

prompt          thinking  raw thoughts  LC in  LC out  LC total  gap
bat and ball    on        523           41     5       396       350
bat and ball    off       absent        41     5       46        0
primary colour  on        42            12     1       55        42
primary colour  off       absent        12     1       13        0
sequence        on        477           31     3       460       426
sequence        off       absent        31     2       33        0

thinking ON  : 3/3 runs have an unaccounted gap
thinking OFF : 0/3 runs have an unaccounted gap

With thinking off, thoughtsTokenCount is absent and LangChain's arithmetic closes exactly. Turn thinking on and the gap opens in every run here. That is 3/3 on short text-only prompts against gemini-2.5-flash - enough to show the gate is thinking, not enough to characterise every call shape.

Which turned out to matter, because there is a fifth.

The one I found too late

While fact-checking this post I ran the same comparison with thinking off and a tool attached:

prompt 14 + candidates 101 = 115,  total 150,  gap 35
toolUsePromptTokenCount: 35

Thinking fully off, one tool attached. The gap is still there, and applying my fix does not close it.

totalTokenCount also includes toolUsePromptTokenCount, and convertUsageMetadata does not read that field either. So the arithmetic breaks on any grounded call, with thinking fully disabled - and my fix does not close it, because it only adds thoughts. Same shape, fifth occurrence, and it is in neither PR.

It is not a one-line addition. @langchain/core's InputTokenDetails has slots for cache and modalities but none for tool-use tokens, and @langchain/google-common declares the field without mapping it either. That is an upstream conversation, not a patch to slip into a review that is already open.

So the honest scope of what I fixed: reasoning tokens, and the llmOutput paths. Tool-use tokens have the same defect and are still open.

The raw SDK call also runs as a span in the same trace as the LangChain call, so the correct count and the wrong count sit side by side in one view.

What I would take from this

The bug was not hard. One field, plus a delta, plus a pair of parentheses.

Finding it was hard, and only because nothing was broken in a way anything noticed. No exception, no log line, no failing test. Wrong numbers on a dashboard look exactly like right numbers.

Here is how well this shape hides. A few minutes after I opened the second PR, @jackjin1997 pointed out another copy of the last-chunk logic in _generateUncached — a second path in a file I had just finished reading, looking for exactly this. They had a reproduction: two chunks of {4, 6, 10} and {0, 3, 3} arriving at handleLLMEnd as {0, 3, 3} instead of {4, 9, 13}. They were right. It went in the same day with their case as a regression test. Their review is here.

I had the shape in my head, I was actively hunting it, and I still walked past one.

So the question I am still sitting with: how do you catch a wrong number when nothing throws and the number is plausible? Error rates will not show it. Tests will not show it — you would have to already suspect the field to assert on it. The only thing that worked here was asking what the observability layer actually reads, and then checking whether the answer was true.

Links

• Fix 1: #11423 — reasoning tokens, streaming delta, precedence
• Fix 2: #11425 — llmOutput on both streaming paths
• Issues: #11422, #11424

Environment: @langchain/google-genai 2.3.0, @langchain/core 1.2.9, @sentry/node 10.70.0, gemini-2.5-flash, Node 24.