DEV Community: Hassan Imam

Two AI debuggers found the same bug. Both stopped one function short.

Hassan Imam — Sat, 22 Aug 2026 02:31:33 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

ChatGoogleGenerativeAI computed your Gemini token counts correctly, then threw
half of them away. Any cost tracker built on LangChain's callback system read
zero tokens on every call.

I reproduced it on the current release, fixed it, and shipped
PR #1954 to
langchain-google.

The interesting part: I pointed two AI debugging tools at the same code — Google's
Gemini CLI and Sentry's Seer. Both correctly identified the same function. Seer
even independently validated the trickiest design decision in the patch. Neither
of them found the second half of the fix — and without it, the bug is still there.

Project Overview

langchain-google-genai is
the official LangChain integration for Google's Gemini models — the package you
install to use ChatGoogleGenerativeAI in a LangChain app. It lives in the
langchain-google monorepo alongside the Vertex AI and community packages, and
ships roughly every couple of weeks.

Token counts are not a nice-to-have in that stack. They are how you bill
customers, enforce per-tenant quotas, populate observability dashboards, and
notice that a prompt change quietly tripled your spend. LangChain exposes them in
two places:

AIMessage.usage_metadata — the modern, per-message standard
LLMResult.llm_output["token_usage"] — the older convention that the callback system reads, and that every other chat integration still populates

Callbacks are what production apps hook into, because they fire for every call
without touching business logic. Which is exactly what made this bug expensive:
the counts were right, the callbacks read zero, and nothing raised an error.

Bug Fix or Performance Improvement

The bug

Issue #957 reports
that BaseCallbackHandler.on_llm_end sees no token usage for
ChatGoogleGenerativeAI. The issue was closed with no linked PR and no fix commit,
and the package was rewritten onto the unified google-genai SDK in 4.0.0 — so my
first job was to find out whether the bug still existed at all. In other words: a
bug the tracker considered resolved, in a package that had since been rewritten
from the ground up.

It does. Here it is on langchain-google-genai 4.3.4, the current release:

class TokenProbe(BaseCallbackHandler):
    def on_llm_end(self, response: LLMResult, **kwargs) -> None:
        print("llm_output:", response.llm_output)
        msg = response.generations[0][0].message
        print("message.usage_metadata:", msg.usage_metadata)

llm = ChatGoogleGenerativeAI(model="gemini-2.5-flash", callbacks=[TokenProbe()])
llm.generate([[HumanMessage(content="Say hello in 3 words.")]])

llm_output            : {}
message.usage_metadata: {'input_tokens': 8, 'output_tokens': 592, 'total_tokens': 600}

The counts exist. They're just not where callback-based token tracking looks for
them. Every get_openai_callback-style cost tracker, every LangSmith-style
accounting layer reading llm_output["token_usage"], reports zero for Gemini.

Root cause

It's in libs/genai/langchain_google_genai/chat_models.py, in
_response_to_result. The function builds llm_output at line 1298:

llm_output = (
    {"prompt_feedback": response.prompt_feedback.model_dump()}
    if response.prompt_feedback
    else {}
)

Then it computes the token counts into lc_usage, attaches them to the message
at line 1404:

message.usage_metadata = lc_usage

…and returns at line 1456:

return ChatResult(generations=generations, llm_output=llm_output)

llm_output is never touched in between. grep confirms it: the identifier
appears at exactly two lines in the entire file — where it's created and where
it's returned.

The package migrated to the modern usage_metadata standard on the message and
left the llm_output["token_usage"] convention — which every other LangChain chat
integration still populates, and which the callback API still reads — empty.

Why the obvious fix is incomplete

This is the part both AI tools missed, and it's the reason a one-line patch
doesn't work.

Populating llm_output inside _response_to_result fixes the callback path.
It does not fix the value you get back from generate(). Here's why, from
langchain_core/language_models/chat_models.py:

flattened_outputs = [
    LLMResult(generations=[res.generations], llm_output=res.llm_output)
    for res in results
]
llm_output = self._combine_llm_outputs([res.llm_output for res in results])
output = LLMResult(generations=generations, llm_output=llm_output)
if run_managers:
    for manager, flattened_output in zip(run_managers, flattened_outputs):
        manager.on_llm_end(flattened_output)   # <- gets res.llm_output DIRECTLY
return output                                  # <- built via _combine_llm_outputs

Two different objects. on_llm_end receives flattened_output, built from
res.llm_output directly. The return value is built through
_combine_llm_outputs — whose BaseChatModel default is:

def _combine_llm_outputs(self, _llm_outputs, /) -> dict[str, Any]:
    return {}

ChatGoogleGenerativeAI never overrode it. So patch only
_response_to_result and result.llm_output is still {}.

My reproduction printed both paths, which is the only reason I noticed.

Code

Two changes. First, mirror the counts into llm_output using the same key
convention as the other integrations:

if lc_usage is not None:
    llm_output["token_usage"] = {
        "prompt_tokens": lc_usage["input_tokens"],
        "completion_tokens": lc_usage["output_tokens"],
        "total_tokens": lc_usage["total_tokens"],
    }
    llm_output["usage_metadata"] = lc_usage
if response.model_version:
    llm_output["model_name"] = response.model_version

Note it sources from lc_usage, not the raw response. lc_usage has already had
prev_usage subtracted, so on the streaming path each chunk carries its own delta
— consistent with how message.usage_metadata already behaves.

Second, the override that makes the return value work, mirroring
langchain-openai:

def _combine_llm_outputs(self, llm_outputs, /) -> dict[str, Any]:
    combined: dict[str, Any] = {}
    token_usage: dict[str, int] = {}
    for llm_output in llm_outputs:
        if not llm_output:
            continue
        for key, value in (llm_output.get("token_usage") or {}).items():
            if value is not None:
                token_usage[key] = token_usage.get(key, 0) + value
        if "model_name" in llm_output:
            combined.setdefault("model_name", llm_output["model_name"])
    if token_usage:
        combined["token_usage"] = token_usage
    return combined

After:

llm_output : {'token_usage': {'prompt_tokens': 8, 'completion_tokens': 775,
              'total_tokens': 783},
              'usage_metadata': {...}, 'model_name': 'gemini-2.5-flash'}
RESULT.llm_output : {'model_name': 'gemini-2.5-flash', 'token_usage': {...}}

Deliberately out of scope: streaming aggregation. _response_to_result is
shared with the streaming path, and the new llm_output carries the same
per-chunk delta as message.usage_metadata, so the two stay consistent. Changing
how chunks aggregate is a separate problem with real double-counting risk
(see langchainjs #8266) and deserves its own PR.

My Improvements

Tests and CI

Four unit tests covering both sites plus the no-usage-metadata edge case. They
assert the invariant that matters — that llm_output agrees with the message:

assert result.llm_output["token_usage"]["total_tokens"] == usage_metadata["total_tokens"]

Full suite: 377 passing, no regressions. ruff, ruff format and mypy
clean across all 35 source files. CI green on Python 3.10, 3.11, 3.12, 3.13
and 3.14.

What I took away

Two AI tools, different vendors, different architectures, different access —
Seer had my actual repository — reached the same correct-but-partial answer. Both
found the function where the tokens should be written. Neither traced what
happens to llm_output after _response_to_result returns.

That's not a knock on either. Both were genuinely useful, and Seer's streaming
observation was sharper than I expected. But the gap is instructive: they answered
the question as posed — why is llm_output empty? — and stopped. They didn't
ask the follow-up: does fixing that actually make the value reach the caller?

The reproduction script is what caught it, because it printed both paths and only
one of them changed.

Best Use of Sentry

I ran the instrumented probe against the unpatched fork and against the patched
one, tagging each run before / after.

Error monitoring. A cost-tracker assertion failing on the empty llm_output,
captured across eight runs. Each event carries the package versions and a context
block naming the suspect file and function, so the issue page alone tells you
where to look.

Tracing. The gen_ai.chat span, before and after:

	before	after
Tokens	8 in + 29 out = 37 total	8 in + 741 out = 749 total
Issues on trace	1	0

Agent monitoring and conversation traces. Because the probe ran with
include_prompts=True, Sentry captured the full exchange on the gen_ai.chat
span — prompt, completion, model, latency, cost and token breakdown.

This one span makes the cost argument better than any explanation of mine. The
prompt was "Say hello in 3 words." The response was "Hello, my friend."
Three words. It cost 749 tokens, because gemini-2.5-flash spent 736 of
them reasoning before answering:

'output_token_details': {'reasoning': 736}

That is exactly the kind of spend you cannot eyeball from the response text, and
exactly what token accounting exists to surface. Before this fix, a
BaseCallbackHandler-based cost tracker watching that call reported zero tokens
and zero cost.

And here's where I have to be honest about something.

I expected to write "Sentry showed zero Gemini tokens." It would have been a
great line. It's also false, and my own traces prove it — the token counts are
correct in both runs.

I checked the SDK source before publishing. sentry_sdk/integrations/langchain.py
resolves tokens like this:

possible_names = ("usage", "token_usage", "usage_metadata")

message = _get_value(obj, "message")      # checked FIRST
if message is not None:
    for name in possible_names: ...

llm_output = _get_value(obj, "llm_output")   # only if message had nothing

Sentry checks the message before llm_output, and falls back to
_extract_tokens_from_generations when the LLMResult has nothing. In other
words: Sentry ships a defensive workaround for exactly this class of bug. It
reports Gemini tokens correctly despite the defect, because it doesn't trust
llm_output to be populated.

That's a better story than the one I expected to tell. The fix doesn't repair
Sentry's numbers — it removes the need for the workaround.

Logs. Structured logs recording llm_output on every on_llm_end, empty
before and fully populated after.

Seer. Sentry's AI debugger, connected to my fork. It read chat_models.py
straight from the repo, identified _response_to_result, and produced a
two-step plan. Step 2 said to use the delta-adjusted lc_usage rather than the
raw response values, so streaming chunks stay correct — independently arriving at
the same conclusion I'd flagged for reviewers in the PR description.

Its cited evidence covers chat_models.py L1292–L1489. _combine_llm_outputs
lives at L3370. It never got there.

One practical note for anyone instrumenting LangChain: it swallows exceptions
raised inside callbacks. My first probe asserted inside on_llm_end and nothing
ever reached Sentry. Assert on the returned LLMResult instead.

Best Use of Google AI

Gemini CLI as an agentic bisect. I ran @google/gemini-cli v0.56.0 with
gemini-2.5-flash against the unpatched tree, so the diagnosis couldn't be
contaminated by my fix:

gemini -p "<prompt>" --model gemini-2.5-flash --skip-trust

It found _response_to_result, cited lines 1298–1302 for the initialization,
1404 for message.usage_metadata = lc_usage, and 1456 for the return, then
proposed token_usage / usage_metadata / model_name with the correct key
names — essentially the patch I'd written, at the same insertion point.

It did not mention _combine_llm_outputs.

Gemini API as a correctness oracle. Self-consistency isn't enough — matching
numbers could both be wrong. So I validated the patched llm_output against the
raw google-genai SDK's own prompt_token_count for the same prompt, which is
deterministic for a fixed model:

prompt                                                SDK in   LC in  match   sums
Say hello in 3 words.                                      8       8   PASS   PASS
Name one primary color. Answer with a single word.        12      12   PASS   PASS
Reply with exactly the word: ok                            8       8   PASS   PASS

RESULT: 3/3 passed

bugsmash

Hassan Imam — Wed, 19 Aug 2026 02:15:04 +0000

AccessLens — A persistent on-device visual interpreter for the blind

Hassan Imam — Sun, 24 May 2026 06:46:43 +0000

This is a submission for the Gemma 4 Challenge: Build with Gemma 4

What I Built

AccessLens is an Android app that turns a Pixel 8 worn on a lanyard into a persistent visual interpreter for blind and low-vision users. Rear camera forward, bone-conduction headphones in, the phone describes the world — and remembers.

The problem with existing visual-assist apps (Be My Eyes, Seeing AI, Envision) is that they are screen-bound, stateless, and cloud-bound. A blind person navigates by sound; an app that needs you to hold up a phone, tap a screen, and wait on a datacenter interrupts that signal stream. AccessLens is different on three axes:

Worn, not held: Two physical buttons drive everything. Volume Up → read text in front of me, verbatim. Volume Down → describe this room with memory from earlier today and recent days. A gyroscope-based SettleTrigger also fires a description automatically when the user stops walking.
Persistent memory across days/weeks: Every gesture writes a SessionEvent to a SQLCipher database. A nightly Gemma 4 worker compresses each day into a DailySummary; Sundays roll into a WeeklyMemory. LONG-press prompts splice that history into the Gemma call, so the model has a world model of this specific apartment, this specific day.
100% on-device: No image, audio, embedding, or location leaves the phone. SQLCipher + Android KeyStore (AES-256-GCM wrapping a SecureRandom DB key) protect everything at rest. A SelfTest on first launch opens a probe DB with the wrong key and asserts the read fails before the app reports encryption healthy.

Face recognition uses MediaPipe FaceLandmarker to produce a 192-dim L2-normalized landmark vector per enrolled person. At identify time, cosine-similar matches inject only the names into the Gemma prompt — Gemma never sees a face crop or an embedding, code-review-verified.

Three gestures, three target latencies (Pixel 8, Tensor G3): SINGLE ≤14 s end-to-end, DOUBLE scales with text length, LONG adds memory retrieval. Voice fillers ("I'm looking…", "Still looking…") cover the prefill gap so the user hears acoustic progress, not dead air. Everything runs with airplane mode on after the model is pushed once.

Demo

Code

hassaninnovate / AccessLens

A blind person's lanyard, powered by Gemma 4 E2B on a Pixel 8. 100% on-device visual assistant with persistent memory and face recognition.

AccessLens

An always-on, on-device visual interpreter for blind and low-vision users — built for the DEV.to "Build with Gemma 4" challenge.

Pitch

A phone worn on a lanyard becomes the user's "eyes." The rear camera is always on; the gyroscope watches for motion. When the user stops walking, AccessLens describes what's in front of them. When a friend whose face has been enrolled walks into frame, the phone says their name. When the user wants to read what's in front of them, they press Volume Up; for a richer description of the room, Volume Down. Bluetooth bone-conduction headphones carry the audio — the user's ears stay free for the world.

What separates AccessLens from existing apps like Be My Eyes, Seeing AI, and Envision is persistent on-device memory + 100% on-device inference. Existing tools are stateless and cloud-bound. AccessLens runs Gemma 4 E2B locally via LiteRT-LM, encrypts…

View on GitHub

Apache 2.0. The repo includes the full Kotlin/Compose source, the encryption self-test, the nightly compression WorkManager job, and a README documenting which file enforces each of the six privacy invariants.

Reference implementation that taught me the LiteRT-LM API: google-ai-edge/gallery — adapted patterns are cited inline in inference/LiteRtLmRuntime.kt.

How I Used Gemma 4

Model Chosen: Gemma 4 E2B (litert-community/gemma-4-E2B-it-litert-lm, ~2.59 GB int4), loaded once at service start via LiteRT-LM 0.12.0 with Backend.GPU() for the vision adapter.

This model was the perfect fit for AccessLens for three core reasons:

Multimodal in one model, on-device: Image input goes in as Content.ImageBytes, text as Content.Text, in that order (per the Gallery's "for accurate last token" comment), all through one Engine.generate call. No separate vision encoder + decoder to stitch, no second model to keep resident. That fits the latency budget and the memory budget on Pixel-class 8 GB RAM.
E2B is the smallest competent multimodal Gemma 4: It fits in RAM alongside MediaPipe FaceLandmarker, a CameraX pipeline, and the Compose UI without OOM-ing on a Pixel 8. I prototyped against E4B (the brief's "quality path") and measured the latency lift on one-sentence scene descriptions — not worth doubling the prefill cost for a use case where the user is waiting in real time, lanyard-mounted, with no screen feedback. The architecture is parametric on the model path (InferenceRuntime.load(modelPath, Modality)), so a future LONG-press branch could swap to E4B in one line. I documented the tradeoff in the README and shipped E2B for all three gestures.
Gemma is the only practical way to do nightly memory compression on-device: The 03:00 CompressionWorker calls Gemma in JSON mode to compress the day's SessionEvent rows into a single DailySummary, and on Sundays into a WeeklyMemory. That's a real LLM task — extracting persistent facts, deduplicating recurring observations, distinguishing "the blue mug is mine" from "I saw a blue mug today" — and it has to happen without a network. E2B handles it in under a minute per day on Tensor G3 while the phone is on the charger.

Production fixes discovered during implementation:

The LiteRT-LM Android artifact must be 0.12.0 or later — 0.11.0 fails vision init inside vision_litert_compiled_model_executor.cc:273 on Tensor G3.
AndroidManifest needs <uses-native-library> declarations for libOpenCL.so, libOpenCL-car.so, libOpenCL-pixel.so (all android:required="false"). Without them, Android 12+ silently denies GPU OpenCL access and the vision backend fails to initialize. Documented at ai.google.dev/edge/litert-lm/android.

The thing I'm proudest of: when you uninstall AccessLens, the KeyStore wrapping key is destroyed with it. The encrypted DB on disk becomes cryptographically unrecoverable. The user can throw the phone away and their memories — kitchen layout, friends' faces, places they've been — go with it. That's what on-device privacy is supposed to mean, and Gemma 4 + LiteRT-LM made it possible without compromising the assistant on quality.

AccessLens — a blind person's lanyard, powered by Gemma 4 on-device

Hassan Imam — Sat, 23 May 2026 17:27:35 +0000

This is a submission for the Gemma 4 Challenge: Build with Gemma 4

What I Built

Worn, not held. Two physical buttons drive everything. Volume Up → read text in front of me, verbatim. Volume Down → describe this room with memory from earlier today and recent days. A gyroscope-based SettleTrigger also fires a description automatically when the user stops walking.
Persistent memory across days/weeks. Every gesture writes a SessionEvent to a SQLCipher database. A nightly Gemma 4 worker compresses each day into a DailySummary; Sundays roll into a WeeklyMemory. LONG-press prompts splice that history into the Gemma call, so the model has a world model of this specific apartment, this specific day.
100% on-device. No image, audio, embedding, or location leaves the phone. SQLCipher + Android KeyStore (AES-256-GCM wrapping a SecureRandom DB key) protect everything at rest. A SelfTest on first launch opens a probe DB with the wrong key and asserts the read fails before the app reports encryption healthy.

Demo

Code

hassaninnovate / AccessLens

A blind person's lanyard, powered by Gemma 4 E2B on a Pixel 8. 100% on-device visual assistant with persistent memory and face recognition.

AccessLens

An always-on, on-device visual interpreter for blind and low-vision users — built for the DEV.to "Build with Gemma 4" challenge.

Pitch

View on GitHub

Reference implementation that taught me the LiteRT-LM API: google-ai-edge/gallery — adapted patterns are cited inline in inference/LiteRtLmRuntime.kt.

How I Used Gemma 4

Model: Gemma 4 E2B (litert-community/gemma-4-E2B-it-litert-lm, ~2.59 GB int4), loaded once at service start via LiteRT-LM 0.12.0 with Backend.GPU() for the vision adapter. Three reasons E2B was the right fit:

Multimodal in one model, on-device. Image input goes in as Content.ImageBytes, text as Content.Text, in that order (per the Gallery's "for accurate last token" comment), all through one Engine.generate call. No separate vision encoder + decoder to stitch, no second model to keep resident. That fits the latency budget and the memory budget on Pixel-class 8 GB RAM.
E2B is the smallest competent multimodal Gemma 4. It fits in RAM alongside MediaPipe FaceLandmarker, a CameraX pipeline, and the Compose UI without OOM-ing on a Pixel 8. I prototyped against E4B (the brief's "quality path") and measured the latency lift on one-sentence scene descriptions — not worth doubling the prefill cost for a use case where the user is waiting in real time, lanyard-mounted, with no screen feedback. The architecture is parametric on the model path (InferenceRuntime.load(modelPath, Modality)), so a future LONG-press branch could swap to E4B in one line. I documented the tradeoff in the README and shipped E2B for all three gestures.
Gemma is the only practical way to do nightly memory compression on-device. The 03:00 CompressionWorker calls Gemma in JSON mode to compress the day's SessionEvent rows into a single DailySummary, and on Sundays into a WeeklyMemory. That's a real LLM task — extracting persistent facts, deduplicating recurring observations, distinguishing "the blue mug is mine" from "I saw a blue mug today" — and it has to happen without a network. E2B handles it in under a minute per day on Tensor G3 while the phone is on the charger.

Two production fixes the brief didn't cover, in case they help someone else:

The LiteRT-LM Android artifact must be 0.12.0 or later — 0.11.0 fails vision init inside vision_litert_compiled_model_executor.cc:273 on Tensor G3.
AndroidManifest needs <uses-native-library> declarations for libOpenCL.so, libOpenCL-car.so, libOpenCL-pixel.so (all android:required="false"). Without them, Android 12+ silently denies GPU OpenCL access and the vision backend fails to initialize. Documented at ai.google.dev/edge/litert-lm/android.

DEV Community: Hassan Imam

Two AI debuggers found the same bug. Both stopped one function short.

Project Overview

Bug Fix or Performance Improvement

The bug

Root cause

Why the obvious fix is incomplete

Code

My Improvements

Tests and CI

What I took away

Best Use of Sentry

Best Use of Google AI

Links

bugsmash

AccessLens — A persistent on-device visual interpreter for the blind

What I Built

Demo

Code

hassaninnovate / AccessLens

A blind person's lanyard, powered by Gemma 4 E2B on a Pixel 8. 100% on-device visual assistant with persistent memory and face recognition.

AccessLens

Pitch

How I Used Gemma 4

Production fixes discovered during implementation:

AccessLens — a blind person's lanyard, powered by Gemma 4 on-device

What I Built

Demo

Code

hassaninnovate / AccessLens

A blind person's lanyard, powered by Gemma 4 E2B on a Pixel 8. 100% on-device visual assistant with persistent memory and face recognition.

AccessLens

Pitch

How I Used Gemma 4