DEV Community

Cover image for Two AI debuggers found the same bug. Both stopped one function short.
Hassan Imam
Hassan Imam Subscriber

Posted on

Two AI debuggers found the same bug. Both stopped one function short.

#devchallenge #bugsmash #python #opensource

Summer Bug Smash: Clear the Lineup 🐛🛹

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

ChatGoogleGenerativeAI computed your Gemini token counts correctly, then threw
half of them away. Any cost tracker built on LangChain's callback system read
zero tokens on every call.

I reproduced it on the current release, fixed it, and shipped
PR #1954 to
langchain-google.

The interesting part: I pointed two AI debugging tools at the same code — Google's
Gemini CLI and Sentry's Seer. Both correctly identified the same function. Seer
even independently validated the trickiest design decision in the patch. Neither
of them found the second half of the fix — and without it, the bug is still there.

Project Overview

langchain-google-genai is
the official LangChain integration for Google's Gemini models — the package you
install to use ChatGoogleGenerativeAI in a LangChain app. It lives in the
langchain-google monorepo alongside the Vertex AI and community packages, and
ships roughly every couple of weeks.

Token counts are not a nice-to-have in that stack. They are how you bill
customers, enforce per-tenant quotas, populate observability dashboards, and
notice that a prompt change quietly tripled your spend. LangChain exposes them in
two places:

  • AIMessage.usage_metadata — the modern, per-message standard
  • LLMResult.llm_output["token_usage"] — the older convention that the callback system reads, and that every other chat integration still populates

Callbacks are what production apps hook into, because they fire for every call
without touching business logic. Which is exactly what made this bug expensive:
the counts were right, the callbacks read zero, and nothing raised an error.

Bug Fix or Performance Improvement

The bug

Issue #957 reports
that BaseCallbackHandler.on_llm_end sees no token usage for
ChatGoogleGenerativeAI. The issue was closed with no linked PR and no fix commit,
and the package was rewritten onto the unified google-genai SDK in 4.0.0 — so my
first job was to find out whether the bug still existed at all. In other words: a
bug the tracker considered resolved, in a package that had since been rewritten
from the ground up.

It does. Here it is on langchain-google-genai 4.3.4, the current release:

class TokenProbe(BaseCallbackHandler):
    def on_llm_end(self, response: LLMResult, **kwargs) -> None:
        print("llm_output:", response.llm_output)
        msg = response.generations[0][0].message
        print("message.usage_metadata:", msg.usage_metadata)

llm = ChatGoogleGenerativeAI(model="gemini-2.5-flash", callbacks=[TokenProbe()])
llm.generate([[HumanMessage(content="Say hello in 3 words.")]])
Enter fullscreen mode Exit fullscreen mode 
llm_output            : {}
message.usage_metadata: {'input_tokens': 8, 'output_tokens': 592, 'total_tokens': 600}
Enter fullscreen mode Exit fullscreen mode

The counts exist. They're just not where callback-based token tracking looks for
them. Every get_openai_callback-style cost tracker, every LangSmith-style
accounting layer reading llm_output["token_usage"], reports zero for Gemini.

Root cause

It's in libs/genai/langchain_google_genai/chat_models.py, in
_response_to_result. The function builds llm_output at line 1298:

llm_output = (
    {"prompt_feedback": response.prompt_feedback.model_dump()}
    if response.prompt_feedback
    else {}
)
Enter fullscreen mode Exit fullscreen mode

Then it computes the token counts into lc_usage, attaches them to the message
at line 1404:

message.usage_metadata = lc_usage
Enter fullscreen mode Exit fullscreen mode

…and returns at line 1456:

return ChatResult(generations=generations, llm_output=llm_output)
Enter fullscreen mode Exit fullscreen mode

llm_output is never touched in between. grep confirms it: the identifier
appears at exactly two lines in the entire file — where it's created and where
it's returned.

The package migrated to the modern usage_metadata standard on the message and
left the llm_output["token_usage"] convention — which every other LangChain chat
integration still populates, and which the callback API still reads — empty.

Why the obvious fix is incomplete

This is the part both AI tools missed, and it's the reason a one-line patch
doesn't work.

Populating llm_output inside _response_to_result fixes the callback path.
It does not fix the value you get back from generate(). Here's why, from
langchain_core/language_models/chat_models.py:

flattened_outputs = [
    LLMResult(generations=[res.generations], llm_output=res.llm_output)
    for res in results
]
llm_output = self._combine_llm_outputs([res.llm_output for res in results])
output = LLMResult(generations=generations, llm_output=llm_output)
if run_managers:
    for manager, flattened_output in zip(run_managers, flattened_outputs):
        manager.on_llm_end(flattened_output)   # <- gets res.llm_output DIRECTLY
return output                                  # <- built via _combine_llm_outputs
Enter fullscreen mode Exit fullscreen mode

Two different objects. on_llm_end receives flattened_output, built from
res.llm_output directly. The return value is built through
_combine_llm_outputs — whose BaseChatModel default is:

def _combine_llm_outputs(self, _llm_outputs, /) -> dict[str, Any]:
    return {}
Enter fullscreen mode Exit fullscreen mode

ChatGoogleGenerativeAI never overrode it. So patch only
_response_to_result and result.llm_output is still {}.

My reproduction printed both paths, which is the only reason I noticed.

Code

Two changes. First, mirror the counts into llm_output using the same key
convention as the other integrations:

if lc_usage is not None:
    llm_output["token_usage"] = {
        "prompt_tokens": lc_usage["input_tokens"],
        "completion_tokens": lc_usage["output_tokens"],
        "total_tokens": lc_usage["total_tokens"],
    }
    llm_output["usage_metadata"] = lc_usage
if response.model_version:
    llm_output["model_name"] = response.model_version
Enter fullscreen mode Exit fullscreen mode

Note it sources from lc_usage, not the raw response. lc_usage has already had
prev_usage subtracted, so on the streaming path each chunk carries its own delta
— consistent with how message.usage_metadata already behaves.

Second, the override that makes the return value work, mirroring
langchain-openai:

def _combine_llm_outputs(self, llm_outputs, /) -> dict[str, Any]:
    combined: dict[str, Any] = {}
    token_usage: dict[str, int] = {}
    for llm_output in llm_outputs:
        if not llm_output:
            continue
        for key, value in (llm_output.get("token_usage") or {}).items():
            if value is not None:
                token_usage[key] = token_usage.get(key, 0) + value
        if "model_name" in llm_output:
            combined.setdefault("model_name", llm_output["model_name"])
    if token_usage:
        combined["token_usage"] = token_usage
    return combined
Enter fullscreen mode Exit fullscreen mode

After:

llm_output : {'token_usage': {'prompt_tokens': 8, 'completion_tokens': 775,
              'total_tokens': 783},
              'usage_metadata': {...}, 'model_name': 'gemini-2.5-flash'}
RESULT.llm_output : {'model_name': 'gemini-2.5-flash', 'token_usage': {...}}
Enter fullscreen mode Exit fullscreen mode

The PR diff showing both changes — the token_usage block in _response_to_result and the _combine_llm_outputs override

Deliberately out of scope: streaming aggregation. _response_to_result is
shared with the streaming path, and the new llm_output carries the same
per-chunk delta as message.usage_metadata, so the two stay consistent. Changing
how chunks aggregate is a separate problem with real double-counting risk
(see langchainjs #8266) and deserves its own PR.

My Improvements

Tests and CI

Four unit tests covering both sites plus the no-usage-metadata edge case. They
assert the invariant that matters — that llm_output agrees with the message:

assert result.llm_output["token_usage"]["total_tokens"] == usage_metadata["total_tokens"]
Enter fullscreen mode Exit fullscreen mode

Full suite: 377 passing, no regressions. ruff, ruff format and mypy
clean across all 35 source files. CI green on Python 3.10, 3.11, 3.12, 3.13
and 3.14.

GitHub CI checks all passing: make lint on Python 3.10 and 3.14, make test on 3.10 through 3.14

Pull request 1954 on langchain-ai/langchain-google, 149 additions and 1 deletion across 2 files

What I took away

Two AI tools, different vendors, different architectures, different access —
Seer had my actual repository — reached the same correct-but-partial answer. Both
found the function where the tokens should be written. Neither traced what
happens to llm_output after _response_to_result returns.

That's not a knock on either. Both were genuinely useful, and Seer's streaming
observation was sharper than I expected. But the gap is instructive: they answered
the question as posed — why is llm_output empty? — and stopped. They didn't
ask the follow-up: does fixing that actually make the value reach the caller?

The reproduction script is what caught it, because it printed both paths and only
one of them changed.

Best Use of Sentry

I ran the instrumented probe against the unpatched fork and against the patched
one, tagging each run before / after.

Error monitoring. A cost-tracker assertion failing on the empty llm_output,
captured across eight runs. Each event carries the package versions and a context
block naming the suspect file and function, so the issue page alone tells you
where to look.

Sentry issues list showing TokenAccountingError with 8 events, culprit gemini-token-probe-before

Sentry issue detail showing tags for langchain-core 1.6.0 and langchain-google-genai 4.3.4, plus the issue 957 context block naming the suspect file and function

Tracing. The gen_ai.chat span, before and after:

before after
Tokens 8 in + 29 out = 37 total 8 in + 741 out = 749 total
Issues on trace 1 0

Sentry trace before the fix: gen_ai.chat span showing 8 in, 29 out, 37 total tokens, with 1 issue on the trace

Sentry trace after the fix: gen_ai.chat span showing 8 in, 741 out, 749 total tokens, with 0 issues on the trace

Sentry error list filtered to environment after, showing no issues match the search

Agent monitoring and conversation traces. Because the probe ran with
include_prompts=True, Sentry captured the full exchange on the gen_ai.chat
span — prompt, completion, model, latency, cost and token breakdown.

Sentry gen_ai.chat span Output tab showing the response

This one span makes the cost argument better than any explanation of mine. The
prompt was "Say hello in 3 words." The response was "Hello, my friend."
Three words. It cost 749 tokens, because gemini-2.5-flash spent 736 of
them reasoning before answering:

'output_token_details': {'reasoning': 736}
Enter fullscreen mode Exit fullscreen mode

That is exactly the kind of spend you cannot eyeball from the response text, and
exactly what token accounting exists to surface. Before this fix, a
BaseCallbackHandler-based cost tracker watching that call reported zero tokens
and zero cost.

And here's where I have to be honest about something.

I expected to write "Sentry showed zero Gemini tokens." It would have been a
great line. It's also false, and my own traces prove it — the token counts are
correct in both runs.

I checked the SDK source before publishing. sentry_sdk/integrations/langchain.py
resolves tokens like this:

possible_names = ("usage", "token_usage", "usage_metadata")

message = _get_value(obj, "message")      # checked FIRST
if message is not None:
    for name in possible_names: ...

llm_output = _get_value(obj, "llm_output")   # only if message had nothing
Enter fullscreen mode Exit fullscreen mode

Sentry checks the message before llm_output, and falls back to
_extract_tokens_from_generations when the LLMResult has nothing. In other
words: Sentry ships a defensive workaround for exactly this class of bug. It
reports Gemini tokens correctly despite the defect, because it doesn't trust
llm_output to be populated.

That's a better story than the one I expected to tell. The fix doesn't repair
Sentry's numbers — it removes the need for the workaround.

Logs. Structured logs recording llm_output on every on_llm_end, empty
before and fully populated after.

Sentry structured logs under environment before, showing on_llm_end llm_output equals an empty dict

Sentry structured logs under environment after, showing on_llm_end llm_output populated with token_usage, usage_metadata and model_name

Seer. Sentry's AI debugger, connected to my fork. It read chat_models.py
straight from the repo, identified _response_to_result, and produced a
two-step plan. Step 2 said to use the delta-adjusted lc_usage rather than the
raw response values, so streaming chunks stay correct — independently arriving at
the same conclusion I'd flagged for reviewers in the PR description.

Its cited evidence covers chat_models.py L1292–L1489. _combine_llm_outputs
lives at L3370. It never got there.

Sentry Seer root cause analysis and solution plan, showing it read chat_models.py from the fork and identified _response_to_result

One practical note for anyone instrumenting LangChain: it swallows exceptions
raised inside callbacks. My first probe asserted inside on_llm_end and nothing
ever reached Sentry. Assert on the returned LLMResult instead.

Best Use of Google AI

Gemini CLI as an agentic bisect. I ran @google/gemini-cli v0.56.0 with
gemini-2.5-flash against the unpatched tree, so the diagnosis couldn't be
contaminated by my fix:

gemini -p "<prompt>" --model gemini-2.5-flash --skip-trust
Enter fullscreen mode Exit fullscreen mode

It found _response_to_result, cited lines 1298–1302 for the initialization,
1404 for message.usage_metadata = lc_usage, and 1456 for the return, then
proposed token_usage / usage_metadata / model_name with the correct key
names — essentially the patch I'd written, at the same insertion point.

It did not mention _combine_llm_outputs.

Gemini API as a correctness oracle. Self-consistency isn't enough — matching
numbers could both be wrong. So I validated the patched llm_output against the
raw google-genai SDK's own prompt_token_count for the same prompt, which is
deterministic for a fixed model:

prompt                                                SDK in   LC in  match   sums
Say hello in 3 words.                                      8       8   PASS   PASS
Name one primary color. Answer with a single word.        12      12   PASS   PASS
Reply with exactly the word: ok                            8       8   PASS   PASS

RESULT: 3/3 passed
Enter fullscreen mode Exit fullscreen mode

Links

Environment: langchain-google-genai 4.3.4, langchain-core 1.6.0,
sentry-sdk 2.68.0, gemini-2.5-flash, Python 3.13.

Top comments (0)