DEV Community: Aina Zulfiqar

The grant money already exists. My AI kept inventing foundations to spend it on

Aina Zulfiqar — Mon, 07 Sep 2026 06:57:48 +0000

This is a submission for Weekend Challenge: Generosity Edition

What I Built

The money exists. A small NGO just cannot find it.

That is what a generosity problem looks like at the small end. The giving has already happened — foundations with open, rolling, unclaimed programmes, sitting there — and it is spread across a few thousand pages nobody has time to read. After the 2025–26 collapse of USAID funding, organisations that had one funder now need six, and the people doing that searching are the same people running the programme: a director who is also the grant writer, working evenings. Generosity is not the scarce thing here. Attention is.

So an AI grant finder is an obvious idea. It is also a dangerous one, because the failure mode is not "unhelpful." A three-person NGO that spends a week writing an application against a deadline that never existed has lost a week it cannot get back, and it will not find out until it submits. The tool would have taken the one thing that was actually scarce.

FundFinderAI is the response, in one sentence: it searches the live web for currently open grants that fit your NGO, and then it refuses to trust its own model about any of them. Every application URL Gemini produces is independently fetched before you see it, and the card tells you what happened when we tried.

The interesting part is not that it searches. It is everything the app does to establish that the search actually happened and that the result actually exists.

Demo

Live: fundfinder-ai.vercel.app — describe an NGO, get grants, open a drafted Letter of Inquiry.

Give it 30–120 seconds. It is running ten to thirty real Google searches and then fetching every URL that comes back, and the page shows you the clock while it does. Paste this in if you would rather not invent an NGO:

NGO name: Kisumu STEM Girls Collective
Location: Kisumu, Kenya
Mission: We run after-school robotics and coding clubs for girls aged 12-17 in Kisumu, Kenya, and train their teachers to keep the clubs running.
Focus areas: STEM education, girls education, teacher training
Annual budget size: Under $250,000 annually
Target population: Adolescent girls in low-income neighbourhoods of Kisumu

You will not get my results. They vary run to run — which is itself the point, and is in Limitations.

A real, unmodified run against the deployed app, for a girls' STEM education NGO in Kisumu, Kenya. Gemini ran 26 Google searches and returned five grants, sorted so the ones that survived checking lead:

✓ verified     Ambassador's Special Self-Help Fund      U.S. Embassy Kenya
✓ verified     Seed Grants                              The Pollination Project
✓ verified     Overseas Aid Trust grants                Blackfriars (BOAT)
  unverified   Grassroots Human Security Projects       Embassy of Japan in Kenya
⚠ didn't       Hansen Family Foundation Grant           Hansen Family Foundation
  resolve

The bottom row is the product working — and the grounding panel on that same page lists hansenfamilyfoundation.org among the pages it retrieved. Google's index has that funder. The live site does not answer. Grounding retrieved it and the URL is still dead — which is the entire argument for checking rather than trusting, in one row.

Ask for a Letter of Inquiry against a grant in that state and the app says so in red before you write a word: we could not reach this funder's page, so we cannot confirm this grant — or this funder — exists. Drafting a warm, professional letter to an organisation that may not exist is the failure this whole project is against, and the button that does it should not be silent about it.

An earlier run caught a harder one: au-eu-youthlab.com, a confident, plausible, entirely non-existent domain. DNS does not resolve it. It never reached the user.

When the link does check out, a second Gemini call drafts the letter — carrying its own standing caution, because a first draft is not a submission:

You do not have to take the screenshots on faith

This is a post about plausible output not being proof, so the central claim — that the app searches rather than remembers — ships as a runnable check. It talks to the Gemini API and nothing else: not my server, not my code paths. About thirty seconds:

git clone https://github.com/ainazulfiqar99acc/fundfinder-ai && cd fundfinder-ai
npm install
cp .env.local.example .env.local   # then put your key in it
npm run check-grounding

It prints which models actually honour googleSearch and exits non-zero if the one this app is configured with did not search. The table it produces is further down.

Code

ainazulfiqar99acc / fundfinder-ai

Grant radar for small NGOs: Gemini Search grounding finds currently-open grants, then every application URL is independently checked before you see it.

FundFinderAI

Live: https://fundfinder-ai.vercel.app

A grant radar for small NGOs. Describe your organisation once; Gemini searches the live web for currently open grants that fit, and drafts a tailored Letter of Inquiry for any of them.

Built for the DEV Weekend Challenge: Generosity Edition.

Why the verification matters more than the search

An AI grant finder that invents a grant is worse than no grant finder. A three-person NGO that spends a week preparing an application against a deadline that never existed has lost something it cannot get back.

So the model is not trusted about whether a grant is reachable. Every applicationUrl it produces is independently fetched before you ever see it and the result is shown on the card:

Badge	Meaning
`✓ Link verified`	The page loads.
`→ Funder site`	The exact page the model named was gone; the funder's own site is live, so the card links

…

View on GitHub

Four files carry the argument:

src/lib/verify-link.ts — the gate. The asymmetry between broken and unverified is the whole design, and the comments say why.
scripts/check-grounding.mjs — the four-model grounding check above.
src/app/api/match-grants/route.ts — the grounded call, the prompt rule that every funder must have appeared in a search result, and verified-first ranking.
src/types/index.ts — four link states, with the reasoning for each written into the type.

How I Built It

The trap: a model can decline to search and never say so

The premise is live Search grounding. The app was configured with gemini-3.1-pro-preview and passed tools: [{ googleSearch: {} }], and it returned six well-formed, plausible grants in sixty seconds. It looked like it worked.

It was not searching at all.

The response carried no groundingMetadata — the candidate had only content, finishReason, index. No error, no warning, no refusal. The model accepted a tool it does not run and answered from memory, in exactly the shape I asked for. The tell was in the output, not the API: that run invented an application URL.

This generalises further than one model. Sending one identical grounded prompt to four models:

model                       time  chunks  queries  grounded
------------------------------------------------------------
gemini-2.5-flash             17s       0        8  YES
gemini-flash-latest          11s       0        0  NO  <-- ignored the tool
gemini-3.1-pro-preview       38s       0        0  NO  <-- ignored the tool
gemini-pro-latest            27s       0        0  NO  <-- ignored the tool

Three of four accept googleSearch and never call it. Any of them would have produced a working, convincing, ungrounded app.

That check is committed as npm run check-grounding. It talks to the Gemini API and nothing else — not my server, not my code paths — and exits non-zero if the model the app is configured with did not actually search. If the premise of this project ever stops being true, that command fails.

The second trap, which was mine

My first version of that check asked whether groundingChunks came back non-empty. By that measure gemini-2.5-flash was also ungrounded, and I nearly changed the model a second time chasing it.

Look at the chunks column above: it is zero for the model that is grounded. Grounding chunks attach citations to prose, and this prompt demands bare JSON with no prose in it, so there is nothing for a citation to attach to. The signal that grounding ran is webSearchQueries — the actual searches issued.

That mistake was live in the UI too. The app showed a red "Gemini may have answered from its training data" warning whenever chunks were empty, which meant it accused itself of hallucinating on runs where it had correctly searched eight times. The fix was to report the searches, which is also just better: the panel now shows the literal queries Gemini ran, which is far more meaningful to a user than a list of opaque vertexaisearch redirect URLs.

Turning recall up, and watching precision fall over

With grounding correctly on, a search for the Kisumu NGO came back with zero grants — "no currently open grants were found with high confidence" — while the search panel showed ten real Google searches. The model had looked and then declined to commit.

My first instinct was that the gate makes conservatism unnecessary: let the model offer plausible funders and let verification sort them out. So I told it that returning an empty list was a failure, and that a funder whose exact call it could not confirm still belonged in the results.

That worked, and it was a bad trade. Results went from zero to six — and five of the six links were dead. Not guessed paths on real funders, which the fallback handles: dead domains. Told to reach a number, the model had begun assembling organisations that sound exactly like real grantmakers and do not exist.

The rule that fixed it draws the line at the search result rather than at confidence:

Every single one must be an organisation that actually appeared in your search results. If you did not see the funder in a search result, it does not go in the list, however plausible it sounds. Three real funders is a good answer; six with two invented ones is a bad answer, because the reader cannot tell which is which.

Plus the same rule for the URL, which is the part most easily fabricated: use the address as it appeared, and if you did not see one, use the funder's homepage — a short address is far likelier to be real than a guessed /grants/apply-2026.

Same profile, after: five grants, three verified, one inconclusive, one caught. A later run on the same profile came back four for four. The gate's job is to catch what slips through, not to license the model to guess — and the better the rule got, the less often the gate had anything to do.

The bug in the gate itself

Worth admitting, because it is the same class of error as everything above. The funder-site fallback — the thing that rescues a real funder whose deep path the model guessed — was very nearly dead code. checkLink tried HEAD first and returned immediately on any settled verdict, and a 404 is a settled verdict, so the ordinary case (a well-behaved server answering HEAD with 404) returned broken and never reached the root probe below it. The fallback only ever fired for servers that reject HEAD outright.

The feature worked in the demo, was described accurately in the README, and was unreachable for the exact case it was written for. The fix is one condition — let broken fall through instead of returning — and it is why real funders now surface as their homepage instead of a dead end.

The gate, and why its verdicts are deliberately lopsided

src/lib/verify-link.ts fetches every URL the model produced. What matters is what it refuses to conclude:

broken only on positive evidence the page is not there: a malformed URL, a DNS failure, or a 404/410 from the server itself.
unverified for everything ambiguous — timeouts, 403s from bot protection, rate limits, TLS quirks.

A funder's firewall blocking a robot is not evidence the grant is fake. Marking a real grant broken would be the same category of harm as inventing one, pointed the other way, so the check is built to be unsure out loud rather than confidently wrong.

Two smaller things fell out of real runs. The model sometimes copies an opaque vertexaisearch.../grounding-api-redirect/ URL straight out of its own grounding metadata into applicationUrl, so those get followed and replaced with wherever they land. And a dead deep path on a live domain turned out to be the single most common failure — real funder, guessed page — so the check retries the origin and offers the funder's own site under a distinct badge.

Honest note on the failures shown here

Every failure in this post came out of a live, unmodified run against the deployed app. I did not corrupt a fixture or hand-write a bad response to make the gate look busy. The model produced au-eu-youthlab.com and ned.org/apply-for-a-grant/ on its own, and both are exactly the kind of thing that would have cost somebody a week.

Limitations, plainly

Reachability is not truth. A page that loads is not proof the grant is open, or that the deadline and eligibility on the card are correct. The gate raises the floor; it does not certify a grant. Every card says to confirm on the funder's page, and it means it.
Eligibility and match reasoning are the model's reading, unverified by anything.
The LOI is a first draft, not a submission. It is there to get past a blank page.
No real NGO has used this. It is tested against profiles I wrote.
Grounded results are narrower. Turning grounding on cut a six-grant answer to three-to-five. That is the honest number, and the six were partly fiction.
It is slow. 30–120 seconds per search, because 10–30 real Google searches happen and then every URL returned is fetched. That is the cost of not guessing, but it is a real cost and I have not hidden it behind a fake progress bar.
Results vary between identical runs. The same profile returns different funders each time, and the caught-link rate moves with it. There is no seed to pin.

Prize Categories

Best Use of Google AI.

Gemini is used twice, and the interesting engineering is in refusing to trust it both times.

Search grounding is treated as a claim to be checked, not a setting to be enabled. Three of four models tested accept googleSearch and silently never run it, producing a fully convincing ungrounded app. npm run check-grounding is a committed, runnable test of that premise that exits non-zero when it fails. The search call passes two Google tools — googleSearch to retrieve, and urlContext so the model reads candidate funder pages rather than only their snippets — and the app then re-fetches every URL itself anyway. A tool the model may have invoked is not evidence that it did.
webSearchQueries, not groundingChunks, is the proof grounding ran — a distinction that a bare-JSON prompt makes load-bearing, and getting it wrong made a correct configuration look broken.
The model's output is externally verified before display. Every URL is fetched; verdicts are asymmetric so ambiguity never becomes a false accusation; dead deep paths fall back to the funder's live site with the invented URL shown struck through.
The second call is deliberately unglamorous. LOI drafting needs no grounding, so it uses the stronger model with no tools — the two calls are configured for what they actually need rather than uniformly.

Remove Gemini and there is no product. Remove the verification and there is something worse than no product: a confident list of grants, some of which do not exist, handed to people who cannot afford to find out the hard way.

Gemini spent 341 tokens thinking. LangChain reported five.

Aina Zulfiqar — Thu, 20 Aug 2026 21:09:47 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

If you bill customers from LangChain's token counts and your users talk to Gemini, check your numbers. On a reasoning call they are wrong for certain. On a streamed call they are wrong whatever the model.

Sentry told me a Gemini call used 46 tokens. The call used 387. The other 341 were thinking tokens — billed by Google, buried inside total_tokens, and named by no field LangChain returned.

The shape is always the same: a value computed correctly, then dropped on the way out. Nothing throws. Nothing logs. The number that reaches your dashboard is usually a plausible number, which is exactly why this survived. I found it four times, across two packages and both of the channels LangChain reports usage on.

Pull requests: #11423 and #11425 Issues: #11422 and #11424

One note on the numbers below: Gemini's thinking budget is not deterministic, so exact counts shift between calls. What never shifts is that a gap opens the moment thinking is on. Every figure I quote comes from a run I screenshotted, and where two figures come from different calls I say so.

Project Overview

@langchain/google-genai is the LangChain integration for Google's Gemini models. If you use ChatGoogleGenerativeAI in a TypeScript app, this is the package you installed.

Token counts are not decoration in that stack. They are how you bill customers, enforce quotas, fill dashboards, and notice when a prompt change quietly triples your spend. LangChain exposes them two ways: usage_metadata on the message, and llmOutput for the callback system that observability tools hook into.

Gemini 2.5 and 3 models think before answering, and they charge for it. The API reports that separately as thoughtsTokenCount. That number is the whole story here.

Bug Fix or Performance Improvement

What is broken

Ask gemini-2.5-flash a question that requires reasoning, with thinking enabled, and read usage_metadata:

{ "input_tokens": 41, "output_tokens": 5, "total_tokens": 387 }

41 plus 5 is 46. The object says 387. The 341-token difference is named by no field in the object.

You do not need to know anything about LangChain to see the problem. The object contradicts itself.

Any cost tracker reading output_tokens sees 5 for a call that produced 346 output tokens.

Before. The probe asserts input + output == total and raises when it fails. This is a silent bug forced to announce itself — the issue title is the entire defect.

Root cause

convertUsageMetadata in libs/providers/langchain-google-genai/src/utils/common.ts builds the result from four fields:

const output: UsageMetadata = {
  input_tokens: usageMetadata?.promptTokenCount ?? 0,
  output_tokens: usageMetadata?.candidatesTokenCount ?? 0,
  total_tokens: usageMetadata?.totalTokenCount ?? 0,
};

thoughtsTokenCount is never read. It does not appear anywhere in the package.

The reason it was missed is worth knowing. This package still depends on the legacy @google/generative-ai SDK, whose UsageMetadata interface declares only four fields. thoughtsTokenCount is not one of them. TypeScript never complained, because you cannot read a property the type says does not exist.

It arrives at runtime anyway. This is the payload from a direct call to the same API with the same prompt — a separate call, so its thinking count is its own:

{
  "promptTokenCount": 41,
  "candidatesTokenCount": 5,
  "totalTokenCount": 427,
  "promptTokensDetails": [{ "modality": "TEXT", "tokenCount": 41 }],
  "thoughtsTokenCount": 381,
  "serviceTier": "standard"
}

Three of those fields are undeclared, including the one that matters. The type was the blindfold.

Why the obvious fix is worse than no fix

Reading thoughtsTokenCount in convertUsageMetadata seems like the whole job. It is not, and stopping there breaks streaming.

Gemini reports usage cumulatively. Every chunk carries the running total for the whole request. The provider already knows this, which is why _streamResponseChunks converts input_tokens, output_tokens and total_tokens into per-chunk deltas after convertUsageMetadata returns. But it leaves nested detail fields cumulative, and mergeOutputTokenDetails in @langchain/core sums reasoning across chunks.

So a three-chunk stream carrying cumulative thoughts of 100, 250 and 381 does this:

100 + 250 + 381 = 731

The true value is 381. The converter-only fix reports 731. It does not merely fail to fix the bug, it creates a 92% overcount, which is the same double-counting class as #8266 — an issue that sat open for over a year describing exactly this failure.

There is a test in the PR that fails if you patch only the converter.

Code

Both fixes are open pull requests against langchain-ai/langchainjs:

#11423 — reasoning tokens dropped in @langchain/google-genai. +284 / -8 across 5 files, with 10 new unit tests.
#11425 — llmOutput token usage wrong on both streaming paths, touching @langchain/core and @langchain/google-genai. +343 / -26, with 7 new unit tests.

They close #11422 and #11424, both of which I filed with reproductions.

These are open pull requests against langchain-ai/langchainjs. #11425 has already drawn review from another contributor; #11423 is open and unreviewed as I write this. Not a fork, not a patch in my own repo — if they merge, the fix reaches every consumer of @langchain/google-genai. It is one of several packages in LangChain JS that reach the Gemini Developer API, and the only one that gets reasoning tokens wrong.

The changes are walked through below.

My Improvements

Three changes, all in #11423.

1. Read the field. Defensively, since the legacy type does not declare it:

const thoughtsTokenCount = getThoughtsTokenCount(usageMetadata);
const candidatesTokenCount = usageMetadata?.candidatesTokenCount ?? 0;

const output: UsageMetadata = {
  input_tokens: usageMetadata?.promptTokenCount ?? 0,
  // Gemini reports reasoning separately from candidatesTokenCount, while
  // output_tokens is defined as the sum of all output token types.
  output_tokens: candidatesTokenCount + thoughtsTokenCount,
  total_tokens: usageMetadata?.totalTokenCount ?? 0,
};

if (thoughtsTokenCount) {
  output.output_token_details ??= {};
  output.output_token_details.reasoning = thoughtsTokenCount;
}

@langchain/google-common and @langchain/google - which back the Vertex packages and also reach the Gemini Developer API - already read thoughtsTokenCount this way. @langchain/google-genai, the standalone Developer-API package built on the legacy SDK, is the one that does not. So this is two packages agreeing with a third, not new behaviour being proposed.

2. Convert reasoning to a delta while streaming, so concatenating chunks does not sum the running totals.

3. An unrelated bug in the same function. While reading it I found this:

if (model === "gemini-3-pro-preview") {
  const over200k = Math.max(0, usageMetadata?.promptTokenCount ?? 0 - 200000);

?? binds looser than -, so 0 - 200000 evaluates first. The line actually reads promptTokenCount ?? -200000, which for any real prompt is Math.max(0, promptTokenCount). On gemini-3-pro-preview, a 500-token prompt reported 500 tokens of overage past 200k. The next line has the same problem. The model gate is probably why nobody had hit it yet.

The tests

Ten unit tests, no network required. Revert the fix and eight of them fail with the exact wrong numbers — the other two are regression guards that have to keep passing either way:

AssertionError: expected undefined to be 381
AssertionError: expected 5 to be 386
AssertionError: expected 46 to be 427
AssertionError: expected 500 to be undefined
AssertionError: expected 250000 to be 50000

Failed Tests 8
Test Files  2 failed (2)

expected 46 to be 427 is the same arithmetic failure my assertion caught on a live call and Sentry carried, reproduced offline with no API key.

The same arithmetic failure the probe caught on a live call, reproduced offline with no API key. Revert the fix and these eight come back.

Full package suite passes: 103 tests, no type errors, oxlint and oxfmt clean.

The same bug, twice more in the callback path

Once I knew the shape, I went looking. Two more, both in what the callback system reads — and, as it turns out, one I missed.

Two. _generate returns llmOutput: { estimatedTokenUsage: tokenUsage } on its streaming branch, where tokenUsage is declared {} at the top of the branch and returned at the bottom without ever being written to. Wrong key too. The non-streaming path of the same class returns tokenUsage.

Three. _streamIterator in @langchain/core reassigns llmOutput on every chunk, so the surviving value is the last chunk's usage. Correct for providers that report cumulative totals on a final chunk. Wrong for providers emitting deltas, where the survivor is the last delta.

Measured across all three call paths, with #11423 already applied so only the llmOutput bug is visible:

call path	what the callback receives	true usage
`invoke()`	`{ 41, 366, 407 }`	407
`invoke()` with `streaming: true`	`estimatedTokenUsage: {}`	401
`.stream()`	`{ 0, 3, 3 }`	458

Three tokens reported for a 458-token call. That one is worse than the empty object, because an empty object looks broken and gets noticed. 3 renders fine on a dashboard.

Both are fixed in #11425.

Best Use of Sentry

Sentry is why I found instances two and three. Not as a monitoring tool, but because instrumenting the code forces one question: what does the observability layer actually read?

The answer for LangChain in JavaScript is llmOutput.tokenUsage, with an Anthropic-shaped llmOutput.usage fallback and nothing after that. No usage_metadata fallback, unlike the Python SDK, which checks the message first. So I went to look at what llmOutput contained, and it was wrong in two different ways on two different paths.

Before and after

I built a probe that makes the same call three ways, tagged the runs before and after, and pointed it at a local patched build.

One trace, one prompt. generate_content is Google's own SDK; the two chat spans are LangChain measuring the same work. The correct count and the wrong count, side by side.

Same span, same prompt, same response. The output count goes from 5 to 371, and every total downstream follows it:

Before: 41 in, 5 out, 46 total — and look at the Cost line. Sentry priced the call from the undercounted tokens. Every number downstream inherits it: cost, context utilization, budget alerts.

After: 41 in, 371 out, 412 total. Same prompt, same response, correct bill.

That is the difference between a missing field and wrong billing, and Sentry is the one saying it.

The response, by the way, was $0.05. Three characters — and on that call, 366 of the 371 output tokens were the model thinking.

Making a silent bug visible

Token miscounting throws nothing. So the probe asserts the arithmetic and raises an error when it fails.

The wrong accounting and the ground truth, six milliseconds apart — LangChain reporting 46 while the raw SDK reports 381 thinking tokens on the same prompt.

After the fix, nothing to report. The assertion that fired on every reasoning call now never fires.

What Seer made of it

I pointed Seer at the issue without telling it anything about the bug — no source access to my fork, no hint about thoughtsTokenCount. All it had was the error and the breadcrumbs.

It got the root cause right:

LangChain's Gemini integration excluded thoughtsTokenCount from output_tokens, causing input+output ≠ total when the model uses extended thinking.

And the mechanism, unprompted: that output_tokens was being set to candidatesTokenCount alone, that totalTokenCount includes thinking tokens, and that the gap should equal thoughtsTokenCount from the raw response. Its reproduction steps are the ones I had written by hand a day earlier.

Two things it got wrong, and they are the interesting part. It named the function responseToUsageMetadata; the real one is convertUsageMetadata. And its evidence pointer — gemini.ts L1235–L1274 — is not the broken code at all. That file is @langchain/google-common, the sibling package, and those exact lines read:

const thoughtsTokenCount = usageMetadata.thoughtsTokenCount ?? 0;
const output_tokens = candidatesTokenCount + thoughtsTokenCount;

That is the correct implementation. It is what my fix makes @langchain/google-genai do.

So Seer reasoned to the right diagnosis and then cited the one file in the repo that already does it properly. Which is what you would expect from a tool working backwards from telemetry rather than from the repository: the breadcrumbs told it what was wrong, and nothing in them could tell it which of several similarly-named packages the call had gone through.

That is still the argument for making a silent bug loud. Give an RCA tool one contradiction to hold onto and it can walk back to the cause from a stack trace and eight console lines. Just check where it points before you believe it.

The agent view

The same run, with the span's Output tab open. Because recordInputs and recordOutputs are on, Sentry captured the model's actual answer — $0.05 — and sits it in the same panel as the token line that says the call cost 46 tokens. A three-character response and a four-hundred-token bill, in one panel, disagreeing with each other. That juxtaposition is what makes the undercount legible as a cost rather than a stray metric.

One thing worth knowing

Sentry's automatic AI instrumentation patches modules as they load, so it only works if Sentry.init() runs before the AI package is evaluated. Under ESM every static import is hoisted, so calling init() in the same file that imports LangChain is already too late. Sentry documents the fix - put init in its own file and run node --import ./instrument.mjs app.mjs - and their own CI exercises exactly that for these integrations.

I was running through tsx, and even with --import the loader hooks did not take for me: my first runs produced only http.client spans and no gen_ai spans at all, and I nearly wrote the wrong conclusion from it. Attaching the instrumentation by hand is the reliable escape hatch:

const handler = Sentry.createLangChainCallbackHandler({
  recordInputs: true,
  recordOutputs: true,
});
const ai = Sentry.instrumentGoogleGenAIClient(new GoogleGenAI({ apiKey }));

If your AI spans are missing on ESM, check that you are loading Sentry via --import before concluding anything - and if you are running through a custom loader like tsx, try attaching by hand before blaming the SDK.

Sentry also filtered the console breadcrumbs carrying the token JSON, marking them [Filtered]. Its scrubber saw the word "token" and did what it should.

Best Use of Google AI

Two models agreeing is not evidence — both can be wrong the same way. So the ground truth in this work is Google's own SDK, called directly.

@google/genai types thoughtsTokenCount properly. The legacy @google/generative-ai SDK that LangChain still depends on does not declare it at all. That gap is why nobody caught it, and it is also how I proved it: run the same prompt through both, and read what each one admits to.

The raw SDK's object closes its own arithmetic. LangChain's does not:

raw @google/genai   41 prompt + 5 candidates + 0 tool-use + 381 thoughts = 427 = totalTokenCount   OK
LangChain           41 input  + 5 output     +         (missing)         = 46  != 387              FAIL

That is what establishes thinking tokens as a separate bucket rather than something folded into candidatesTokenCount. Without it, "output_tokens should include reasoning" is an opinion about API semantics. With it, it is arithmetic, and the PR stops being a proposal and becomes a correction.

The controlled version

Thinking budgets are not deterministic, so a single pair of calls proves less than it looks. I ran three prompts twice each — thinking on, thinking off — through both SDKs:

prompt          thinking  raw thoughts  LC in  LC out  LC total  gap
bat and ball    on        523           41     5       396       350
bat and ball    off       absent        41     5       46        0
primary colour  on        42            12     1       55        42
primary colour  off       absent        12     1       13        0
sequence        on        477           31     3       460       426
sequence        off       absent        31     2       33        0

thinking ON  : 3/3 runs have an unaccounted gap
thinking OFF : 0/3 runs have an unaccounted gap

With thinking off, thoughtsTokenCount is absent and LangChain's arithmetic closes exactly. Turn thinking on and the gap opens in every run here. That is 3/3 on short text-only prompts against gemini-2.5-flash - enough to show the gate is thinking, not enough to characterise every call shape.

Which turned out to matter, because there is a fifth.

The one I found too late

While fact-checking this post I ran the same comparison with thinking off and a tool attached:

prompt 14 + candidates 101 = 115,  total 150,  gap 35
toolUsePromptTokenCount: 35

Thinking fully off, one tool attached. The gap is still there, and applying my fix does not close it.

totalTokenCount also includes toolUsePromptTokenCount, and convertUsageMetadata does not read that field either. So the arithmetic breaks on any grounded call, with thinking fully disabled - and my fix does not close it, because it only adds thoughts. Same shape, fifth occurrence, and it is in neither PR.

It is not a one-line addition. @langchain/core's InputTokenDetails has slots for cache and modalities but none for tool-use tokens, and @langchain/google-common declares the field without mapping it either. That is an upstream conversation, not a patch to slip into a review that is already open.

So the honest scope of what I fixed: reasoning tokens, and the llmOutput paths. Tool-use tokens have the same defect and are still open.

The raw SDK call also runs as a span in the same trace as the LangChain call, so the correct count and the wrong count sit side by side in one view.

What I would take from this

The bug was not hard. One field, plus a delta, plus a pair of parentheses.

Finding it was hard, and only because nothing was broken in a way anything noticed. No exception, no log line, no failing test. Wrong numbers on a dashboard look exactly like right numbers.

Here is how well this shape hides. A few minutes after I opened the second PR, @jackjin1997 pointed out another copy of the last-chunk logic in _generateUncached — a second path in a file I had just finished reading, looking for exactly this. They had a reproduction: two chunks of {4, 6, 10} and {0, 3, 3} arriving at handleLLMEnd as {0, 3, 3} instead of {4, 9, 13}. They were right. It went in the same day with their case as a regression test. Their review is here.

I had the shape in my head, I was actively hunting it, and I still walked past one.

So the question I am still sitting with: how do you catch a wrong number when nothing throws and the number is plausible? Error rates will not show it. Tests will not show it — you would have to already suspect the field to assert on it. The only thing that worked here was asking what the observability layer actually reads, and then checking whether the answer was true.

DEV Community: Aina Zulfiqar

The grant money already exists. My AI kept inventing foundations to spend it on

What I Built

Demo

You do not have to take the screenshots on faith

Code

ainazulfiqar99acc / fundfinder-ai

Grant radar for small NGOs: Gemini Search grounding finds currently-open grants, then every application URL is independently checked before you see it.

FundFinderAI

Why the verification matters more than the search

How I Built It

The trap: a model can decline to search and never say so

The second trap, which was mine

Turning recall up, and watching precision fall over

The bug in the gate itself

The gate, and why its verdicts are deliberately lopsided

Honest note on the failures shown here

Limitations, plainly

Prize Categories

Gemini spent 341 tokens thinking. LangChain reported five.

Project Overview

Bug Fix or Performance Improvement

What is broken

Root cause

Why the obvious fix is worse than no fix

Code

My Improvements

The tests

The same bug, twice more in the callback path

Best Use of Sentry

Before and after

Making a silent bug visible

What Seer made of it

The agent view

One thing worth knowing

Best Use of Google AI

The controlled version

The one I found too late

What I would take from this

Links