DEV Community: Asuran

The LangGraph agent trace that quietly lost its input and output

Asuran — Tue, 11 Aug 2026 16:44:40 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

Sentry's JavaScript SDK can auto instrument LangGraph so every agent run shows up in the AI Agents view with its input and its output attached to a gen_ai.invoke_agent span. That is the whole reason you reach for it: when an agent misbehaves in production you open the trace and read what went in and what came out. Issue #19628 is about the case where that view is quietly empty and the fix makes the agent trace carry its data no matter how the graph state is shaped.

Bug Fix or Performance Improvement

LangGraph lets you define the graph state two ways. The common tutorials use MessagesAnnotation, where the state is a list of chat messages under a messages key. But you can also define a custom state annotation with your own keys, which is normal once an agent tracks more than a chat log:

const CustomState = Annotation.Root({
  idea: Annotation(),
  expanded: Annotation(),
  validated: Annotation(),
});

const graph = new StateGraph(CustomState)
  .addNode("expand", expandNode)
  .addEdge(START, "expand")
  .addEdge("expand", END);

Sentry.instrumentStateGraph(graph, { recordInputs: true, recordOutputs: true });
const compiled = graph.compile({ name: "my_agent" });
const result = await compiled.invoke({ idea: "test idea" });

The instrumentation read the input as args[0].messages and the output as result.messages. For a custom state there is no messages key, so the input attribute came out as an empty array and the output attribute was never set. No error and no warning. The span looked healthy in the UI with nothing inside it, which is the worst kind of observability bug because you only notice when you already need the data and it is not there.

Code

The defect is in packages/server-utils/src/ai/langgraph. The input read collapsed a missing key into an empty array:

const inputMessages =
  args.length > 0 ? ((args[0] as { messages?: LangChainMessage[] } | null)?.messages ?? []) : [];

and the output helper bailed out the moment there was no messages array:

const outputMessages = resultObj?.messages;
if (!outputMessages || !Array.isArray(outputMessages)) {
  return; // records nothing for a custom state
}

The fix keeps the MessagesAnnotation path exactly as it was. When there is no messages array, it serializes the whole state and records it, wrapped as a single role and content message so the attribute stays a valid chat array:

// input, in instrumentCompiledGraphInvoke
} else if (inputState && typeof inputState === "object") {
  span.setAttributes({
    [GEN_AI_INPUT_MESSAGES]: stringify([{ role: "user", content: stringify(inputState) }]),
  });
}

// output, in setResponseAttributes
if (result && typeof result === "object") {
  span.setAttribute(GEN_AI_RESPONSE_TEXT, stringify([{ role: "assistant", content: stringify(result) }]));
}

Wrapping the state into a role and content message is not a random choice. The same package already wraps raw string prompts that way in extractLLMRequestAttributes "to align with the chat schema used elsewhere", so this follows an existing convention rather than inventing a new shape. A bare object in gen_ai.input.messages would risk the UI failing to render it, which would trade one silent empty for another. Serialization uses the SDK's own circular safe stringify so an unusual state object cannot throw inside the span callback.

My Improvements

I added three tests in the package's own vitest suite. One drives a graph on a custom state annotation and asserts the input and output state land on the span. One is a regression test that a MessagesAnnotation graph still records its real messages and its response delta. One asserts a null input (the resume case) records nothing rather than a misleading empty array. To prove the tests actually catch the bug, I ran them against the unfixed source and they fail with expected [] to have a length of 1 but got +0, which is the empty input the issue describes. With the fix they pass.

Everything is green on the repo's own gates. The full @sentry/server-utils suite goes from 335 passing to 338 passing with the three added tests and zero failures. oxlint --type-aware is clean, oxfmt --check is clean and tsc is clean on the changed source and the changed test. The change is about 14 lines of real source across two files.

I kept the diff focused and matched the maintainer's stated direction. On the issue, a Sentry maintainer wrote that they would "have a go at serializing the entire state here", so serializing the whole state is exactly the approach taken.

Best Use of Sentry

This is a fix to Sentry's own JavaScript SDK, in the LangGraph auto instrumentation that powers the AI Agents product. The value is in what the trace now shows. Before the fix a custom state agent produced an invoke_agent span whose input and output were empty, so the AI Agents view told you an agent ran and nothing about what it did. After the fix the same span carries the input state and the full output state, so the trace is useful for the exact debugging session it exists for. It also stops emitting a misleading empty array on a resume, which reads as real but blank data.

Best Use of Google AI

I reproduced the bug end to end with real Google Gemini, because a unit test with a mock LLM proves the code path but not the product story. The reproduction builds a LangGraph StateGraph on a custom state annotation Annotation.Root({ topic, poem }) where one node calls Gemini through @langchain/google-genai ChatGoogleGenerativeAI (model gemini-flash-latest) to write a short poem into the state. The graph is instrumented with the local build of the SDK and a real Sentry client with tracing on, then the invoke_agent span is read straight off the client.

Same real agent run, two SDK builds. Against the unfixed build Gemini returned a real poem and the graph state held it, yet the invoke_agent span reported gen_ai.input.messages as [] and had no gen_ai.response.text at all. Against the fixed build the input state showed up as [{"role":"user","content":"{\"topic\":\"the sea\"}"}] and the response text carried the whole output state including the Gemini generated poem. Google Gemini produced the content and Sentry is where the missing content reappears once the SDK reads the state correctly.

A note on how this was built

AI assistance (Claude, Anthropic) was used in developing this change. The design, review and verification were done by the author. Verified locally before submitting: the full @sentry/server-utils vitest suite (338 passed, up from 335, zero failures), the new tests failing on the unfixed source to prove they catch the bug, oxlint --type-aware clean, oxfmt --check clean, tsc clean on the changed files, plus a real Google Gemini LangGraph run showing the span input and output empty before the fix and populated after.

Nine green tests and a parser that never worked once

Asuran — Tue, 11 Aug 2026 11:51:29 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

Two bugs, one story. A parser that failed on every real call, plus the test that certified it as working for two review rounds. The second one is the one worth reading about.

What the code does

I was writing adapters for a protocol indexer. Each adapter has a Receipt parser: it receives the on-chain state changes from one transaction and produces a structured outcome, "this account swapped 100 of token A for 98 of token B". Downstream consumers trust that outcome. If the parser picks the wrong movement out of the transaction, the system reports a swap that never happened, with no error anywhere.

The tests ran against live mainnet, which is where this starts.

Bug one: the test that could not fail

The end-to-end test for one adapter looked roughly like this:

if (halted) {
  expect(index).toBe(1)          // the call reverted on chain, fine
} else {
  expect(result).toEqual(...)    // the call succeeded, check the payload
}

The reasoning was sensible. This is a live chain, the contract call might legitimately revert depending on pool state, so the test tolerates a revert instead of failing the build on someone else's liquidity.

Read it again. There is no input that makes this test fail. Success gets checked. Failure gets accepted. The assertion has no opinion.

Nine cases against Monad mainnet, all green, through two rounds of review. What they were actually reporting was my own parser throwing on every single call. The halt the test tolerated was never the chain rejecting anything. It was my code.

The reviewer did not catch it. I did not catch it. The suite was green, which is what we both looked at.

Bug two: the parser underneath

With the tolerance removed the real bug surfaced immediately, a one-liner of the worst kind:

const leg = transfers.find(t => /* right endpoints, right amount, not our own token */)

.find() returns the first match. The predicate described a shape that any ERC-20 transfer of the right size could satisfy. So in a transaction carrying two structurally identical movements, the parser reported whichever came first in the log.

Two ways that goes wrong. I reproduced both:

Put a decoy transfer with the same shape ahead of the real one and the decoy is reported as the outcome asset.
Duplicate the real movement and two genuine transfers collapse into one outcome, while coverage still looks complete.

Neither raises anything. You get a plausible, wrong answer.

The fix is a rule rather than a patch: filter the candidates, then require exactly one. Zero keeps the existing missing-evidence error. Two or more throws and names the count. Ambiguity becomes a loud failure instead of a coin flip, which is the only correct behavior when the data genuinely cannot distinguish two movements.

Why I care more about the test than the parser

The parser bug is ordinary. .find() where you meant "the only one" is a mistake every codebase has. Once you see it you fix it in a minute.

The test bug is the interesting one, because it consumed the evidence I was using to judge the parser. A green suite is what I hand a reviewer. A branch in the assertion means the suite cannot tell the happy path from the exact failure the code has, so "green before submit" becomes a sentence nobody checked.

Three things I now do because of it.

Any if (failure) {...} else {...} in a test is unverified until probed. So is any comment excusing a failure mode. When a test says a failure is acceptable because external state might reject the call, go find out which one actually happened. Dump the raw trace once, then assert the single real behavior and let it fail loudly if the world changes.

Grep for first-match selection. .find(, [0], .at(0), .shift(). For each one, ask what stops a second, equally valid item from matching. If the predicate cannot name something fixed, it is guessing.

Prove the new assertion fails without the fix. Be careful how you undo it. This one bit me a second time. To show an assertion catches the bug you revert the source and run the suite. I used git checkout -- <file> to undo the experiment, but that file also held the round's real work, so my new assertions vanished and the next run went green for an entirely different reason. Caught by diffing the working tree, not by the suite. So: toggle only the file the experiment needs, never one carrying the change, then capture the failing output to a file the moment it fails, because you cannot reproduce it once the state is restored.

The shape of both bugs

Neither of these was a crash. Nothing threw, nothing logged, no monitor fired. One produced a wrong answer that looked like a right answer. The other produced a green test that looked like a passing test.

That is the class of bug I have started looking for first, because it is the class that survives review. A crash gets fixed the day it happens. A confident wrong answer ships, then it gets built on.

Written with AI assistance (Claude, Anthropic). The bugs, the reproductions and the review history are from my own work on a real open source adapter. Both defects were reproduced from both directions, the decoy and the duplicate, before the fix was submitted. The corrected assertions were confirmed failing against the unpatched parser.

The sparring partner was reading my config file

Asuran — Tue, 11 Aug 2026 11:51:27 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

I was tuning an agent for a simulation competition. Thirty in-game days, two players, one shared commodity market. Plant crops, hire hands, buy livestock, sell into a market whose prices move with supply. Whoever ends with more coins wins.

The agent has about forty knobs. How much cash to keep before buying an animal, how many hands to hire on each day, when to buy more land. Tuning it means picking a knob, moving it, then finding out whether the agent got better.

Which needs an opponent. So I generated one from my own agent: same file, frozen configuration, saved as bench/rivalsim.py. Sixteen seeds, sixteen games, count the wins. Standard practice.

The result that should have scared me

I lowered animal_reserve from 400 to 150, so the agent would keep buying livestock instead of hoarding cash, bought eight cows instead of five and pushed the crew to twelve hands.

16 out of 16. Plus 40,000 coins of margin.

Every previous experiment had moved the number by two or three thousand. This one moved it by forty. I had been fiddling with the same parameters for days and suddenly one change swept the board.

I shipped it to the live ladder. Then I shipped a second variation on the same idea.

Both lost.

The knob was wired to both players

My agent reads its overrides from an environment variable. One line near the top:

_ov = _os.environ.get("KAGG_P")   # tuning overrides, parsed into the config

The sparring partner was generated by copying the agent source. It copied that line too.

So when I set animal_reserve=150 and ran the match, the environment variable was set for the whole process. Both agents read it. My opponent also stopped keeping cash, also bought livestock it could not feed, also starved its herd.

I was not measuring a better strategy. I was measuring an opponent I had quietly sabotaged, using the same lever I thought I was pulling on myself.

Here is the same configuration re-measured with the override hook stripped out of the partner:

config	wins	margin
baseline	11/16	+1,953
the "breakthrough"	0/16	-15,314
the variation I shipped after it	1/16	-13,044

A 16-0 win became 0-16. The margin swung by 55,000 coins. The change was one of the worst regressions I had produced and it presented as the best result of the project.

Why it read as a triumph

This is the part worth generalising, because the bug is not specific to farming simulations.

Sabotaging your opponent and improving yourself move the metric in the same direction. Win rate cannot tell them apart. A benchmark exists to hold everything constant except the thing under test. A shared configuration channel breaks that invariant without leaving a trace. No exception, no warning, no failing assertion. Just a number that got better.

The magnitude was the tell and I read it as the reward. Forty thousand against a usual two or three thousand should have been the moment I stopped and asked what else changed. Instead the size of the jump is what convinced me it was real. That is exactly backwards. I would not have said so out loud before it happened to me.

What I changed

The generator strips the hook. The script that produces a sparring partner now cuts the override block out of the copy, so the partner is a fixed strategy rather than a mirror of whatever I am currently testing.

A partner has to prove it is inert. Run it against an unrelated third agent, once with the environment variable set to something deliberately destructive and once without. Byte-identical scores or the partner is not frozen. Mine now returns 155,875 either way. That check takes about a minute and it is the cheapest thing in the pipeline.

One opponent is not a benchmark. Even with an inert partner, a single opponent rewards whatever exploits that opponent. A later build went 16 out of 16 against one partner and then sat around 27th of the ~30 opponents it met on the real ladder. So the gate became a seven-agent round robin with the seating permuted, 144 games each. Two identical agents score about 42% there with 16% ties, which gives me a floor: anything not clearly above that is noise.

Treat a great result as a measurement bug until proven otherwise. Not as false, just as unverified in a specific direction. The question is no longer "did this help" but "what else did this touch".

The line I keep coming back to

A sparring partner that reads the same configuration as the candidate is not a sparring partner. It is a mirror. A mirror always tells you that you are winning.

If you benchmark anything against a copy of itself (common in agent work: self-play, A/B harnesses, model comparisons) go and check what your two sides share. Environment variables, config files, a singleton, a cache, a random seed. Anything the candidate can reach that the opponent can also reach is a channel where your experiment leaks into your control.

Two bad submissions went out before I found mine.

Written with AI assistance (Claude, Anthropic). The bug, the diagnosis and the numbers above are from my own work and were re-measured before publishing: every figure in the tables comes from a captured 16 seed run, plus the inert-partner check is reproducible with one command.

A documented prop that silently stacked every component at the origin

Asuran — Tue, 11 Aug 2026 11:43:35 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

tscircuit lets you describe a printed circuit board in React. You write <resistor name="R1" footprint="0805" /> and the renderer produces real manufacturing output: component placement, copper, silkscreen, a netlist, Gerbers. tscircuit/core is the renderer at the centre of that, so a placement bug there ships into every board anyone exports.

I found this one by building a board, not by reading the source. That distinction matters. I come back to it at the end.

Bug Fix or Performance Improvement

I was laying out an LED matrix and reached for pcbLayout={{ matchAdapt: true }}, a documented prop. Every component landed on top of every other component at the group origin.

Not an error. Not a warning. Four components, one coordinate, a board that looks plausible until you notice all the footprints are in the same place.

The cause is four lines of dispatch in Group.ts:

if (pcbLayoutMode === "grid") {
  this._doInitialPcbLayoutGrid()
} else if (pcbLayoutMode === "pack") {
  this._doInitialPcbLayoutPack()
} else if (pcbLayoutMode === "flex") {
  this._doInitialPcbLayoutFlex()
}
// "match-adapt" falls off the end

Group._getPcbLayoutMode() happily returns "match-adapt". The dispatch handles grid, pack and flex. So the mode is valid on the way in, no branch claims it, no layout runs, so every component keeps the default position it was born with. Schematic match-adapt works fine, which is what makes it convincing: the same prop name does something real on the other view.

That is the shape of bug I have learned to trust. Not a crash you can grep for in a log, but a documented input that quietly produces geometry a fab house would happily build wrong.

Code

PR: tscircuit/core#3137, which fixes issue #3136. +76 lines, no deletions.

Six lines of source:

} else if (pcbLayoutMode === "match-adapt") {
  // There is no dedicated PCB match-adapt layout. Without this branch the
  // mode falls through and every component stays on the group origin,
  // producing overlapping footprints. Fall back to packing so the board is
  // still laid out. Schematic match-adapt is handled separately.
  this._doInitialPcbLayoutPack()
}

Plus a regression test and a PCB snapshot.

My Improvements

The test asserts the bug, not the feature. It would be easy to write a test that renders the board and checks it looks right. That passes for the wrong reasons. Instead it asserts the two things that were actually false before:

// Components must be laid out, not collapsed onto a single origin point.
const distinctCenters = new Set(
  pcbComponents.map((c) => `${c.center.x.toFixed(2)},${c.center.y.toFixed(2)}`),
)
expect(distinctCenters.size).toBe(pcbComponents.length)

// Packing keeps footprints apart, so there should be no overlap DRC errors.
const overlapErrors = (circuit.getCircuitJson() as any[]).filter(
  (el) =>
    el.type === "pcb_courtyard_overlap_error" ||
    el.type === "pcb_footprint_overlap_error",
)
expect(overlapErrors).toHaveLength(0)

Four distinct centres and zero overlap errors. Both fail on the old code, both for the real reason. The PCB snapshot comes along so a reviewer can see the board rather than trust the assertions.

I said what the fix is not. Match-adapt on the PCB side does not exist in this repo. My branch routes it to the packing layout, which lays the board out correctly. I put that in the PR body as a limitation rather than presenting it as a match-adapt implementation. A maintainer might want the real algorithm instead. Either way the current behavior, silent overlap on a documented prop, is not defensible. The fallback is at least honest about being a fallback.

Green before submit, on their gates not mine. CI runs format-check, a smoke test and ten test shards. All green on the PR.

What building a board taught me that reading code did not

This is the part I would actually tell someone.

I had spent a stretch scanning this codebase for bugs by reading it. I filed several. Two maintainers pushed back on the batch. The substance of it was: these are nitpicks in the wrong places, they are not user issues, go build a board.

That was fair and it was useful. So I built boards. An LED matrix, a Bluetooth speaker front end. Within one afternoon of actually using the library the way a user does, I hit a documented prop that silently destroys the layout, worth more than everything the reading pass produced. Reading finds code that looks wrong. Using finds behavior that is wrong.

The bug was in a file I had already read. I did not see it, because a dispatch handling three of four cases reads like a dispatch. It only becomes a bug when you type the fourth case into a real board and watch the components pile up.

Written with AI assistance (Claude, Anthropic). The board that surfaced the bug, the diagnosis, the fix and the tests are mine, verified before submitting: the repo's own test suite plus format checks green locally and in CI across ten shards, the new assertions confirmed failing against unpatched code, plus the issue checked live for an existing fix PR before opening mine.

Sentry's trace said gemini-flash-latest. Google served gemini-3.6-flash.

Asuran — Tue, 11 Aug 2026 11:43:00 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

The project is Sentry's own Python SDK, getsentry/sentry-python. Its google_genai integration is what turns a Gemini call into a gen_ai.chat span, so your AI monitoring dashboard has token counts, finish reasons, cost and the model that answered. It ships in the same SDK as the error monitoring, which means for a lot of teams it is the only record of what their agent actually did in production.

I went in to fix a five month old issue about two missing span attributes. What I found was worse than missing.

Bug Fix or Performance Improvement

Issue #5812 says streaming responses never capture response_id or model_version. It is exact and it is correct. accumulate_streaming_response in google_genai/streaming.py declares both locals, reads text, finish reasons, tool calls and five token counts out of the chunks, then hands the two declared locals to the span without ever reading them:

usage_data = None
response_id = None          # declared
model = None                # declared

for chunk in chunks:
    ...                     # everything else: read from the chunk
                            # response_id and model: never touched

accumulated_response = AccumulatedResponse(
    ...
    id=response_id,          # always None
    model=model,             # always None
)

The non-streaming path in the same integration reads both off the response, so the two halves of one integration disagree.

Before writing a line of fix I wanted to know the true blast radius, so I built a differential probe. It sends identical content through the real integration twice, once as one whole GenerateContentResponse and once as a chunk sequence, then diffs the gen_ai.* attributes that land on the chat span. Six response shapes: plain text, reasoning plus cached tokens, a tool call, a MAX_TOKENS finish, metadata only on the final chunk, then no usage metadata at all.

SUMMARY: attributes broken on the streaming path
  gen_ai.response.id                 in 6/6 cases
  gen_ai.response.model              in 6/6 cases

Nothing else diverged. Text, finish reasons, tool calls and all five token counts match the non-streaming path in every shape. AsyncModels.generate_content_stream is affected identically. So this is one narrow defect, not a broken integration, which is exactly what you want to know before touching anything.

Then Sentry showed me the real bug

I wired my Sentry DSN in, ran real Gemini traffic through the unfixed code, then opened the span. gen_ai.response.model was there. My own bug report appeared to be wrong.

It was not. I dumped the eleven attributes the SDK actually put on the wire and gen_ai.response.model was not among them. Sentry was filling the row in from gen_ai.request.model, the model I asked for.

For gemini-2.5-flash you cannot see the difference, because ask and served are the same string. So I ran it again against gemini-flash-latest, an alias, with the source reverted to the parent commit. Same prompt, same commit, one streaming and one not:

Span	`gen_ai.response.id`	`gen_ai.response.model`
streaming (the bug)	no row at all	`gemini-flash-latest`
non-streaming (control)	`1_l6armtIJOKqfkPnai9yQM`	`gemini-3.6-flash`

Google served gemini-3.6-flash. The streaming trace says gemini-flash-latest.

Same prompt, same commit, side by side in Sentry. Left, the streaming span reports gemini-flash-latest, the alias I asked for. Right, the non-streaming control reports gemini-3.6-flash, the model Google actually served. The streaming span also has no response.id row at all.

That reframes the whole issue. A blank field is honest, you know you have nothing. A field silently backfilled from the request is a confident wrong answer. gen_ai.response.model exists precisely to record the case where those two differ. Every production streaming span, which is to say every user-facing chat, attributes the generation to whatever alias the caller happened to type.

gen_ai.response.id has no fallback, so it is simply gone. It is the only handle that ties a Sentry span to a specific Gemini response, which is what you need when you take a bad generation to Google.

Code

The fix: getsentry/sentry-python, branch fix/google-genai-streaming-response-metadata, commit 5b4c291.

Six lines of source, inside the loop that was already walking every chunk:

        # Gemini repeats the response id and the served model version on every
        # chunk, but keep the last non-None value so a chunk that omits either
        # one does not discard it.
        response_id = getattr(chunk, "response_id", None) or response_id
        model = getattr(chunk, "model_version", None) or model

Plus 85 lines of test. Two assertions added to the existing streaming test, plus one new async streaming test where only the first chunk carries the metadata.

The streaming span before and after the fix. gen_ai.response.id goes from absent to recorded, response.model now carries the served value instead of leaning on the UI fallback.

Per Sentry's CONTRIBUTING.md a contribution needs maintainer agreement on the issue before a PR exists. A bot enforces that. Two earlier PRs on this exact issue were auto-closed for skipping that step, neither reviewed on merit. So I posted the verified finding on #5812 and asked to pick it up. The PR follows once a maintainer answers. The branch above is the full change either way.

My Improvements

Last non-None, not first. Real traffic decided this. I measured live Gemini streams and every chunk carries both fields: 4 of 4 chunks on gemini-2.5-flash, 8 of 8 on gemini-flash-latest. But this repo's own streaming fixture puts them on the first chunk only. So first-wins passes the tests while last-wins-without-a-guard would let a metadata-free chunk clear a good value. Last non-None satisfies both, matching the convention the anthropic integration in the same SDK already uses (model = event.message.model or model).

Prove the test catches it. A new assertion that passes alongside a fix proves nothing. I reverted the source, kept the tests, then ran again:

5 failed, 68 passed
FAILED test_streaming_generate_content[True-True]  - KeyError: 'gen_ai.response.id'
FAILED test_streaming_generate_content[True-False] - KeyError: 'gen_ai.response.id'
FAILED test_streaming_generate_content[False-True] - KeyError: 'gen_ai.response.id'
FAILED test_streaming_generate_content[False-False]- KeyError: 'gen_ai.response.id'
FAILED test_async_streaming_response_id_and_model  - KeyError: 'gen_ai.response.id'

Why 510 passing tests missed it, which I think is the interesting part. Two reasons, neither of them carelessness.

The fixture at test_google_genai.py:743 already carries "responseId": "response-id-stream-123" and "modelVersion": "gemini-1.5-flash". The data was on the way in. No streaming test asserted either attribute reached the span. The only assertions on those two constants in the whole 4,760 line file are in the non-streaming test. The suite tested the door, not the delivery.

Then a hardening commit made the dead code look deliberate. a66281a, "Guard against None response ID and response model", rewrote the consumer:

-    if accumulated_response.get("id"):
-        span.set_data(SPANDATA.GEN_AI_RESPONSE_ID, accumulated_response["id"])
+    response_id = accumulated_response.get("id")
+    if response_id is not None:
+        set_on_span(SPANDATA.GEN_AI_RESPONSE_ID, response_id)

That is a careful None guard on a value that is always None. It reads as intentional, so the producer never got a second look. A guard is not a test.

What I deliberately did not report. element_wise_usage_max takes an independent per-field maximum across chunks, so a stream whose candidate count dips on the final chunk reports output_tokens=9, total_tokens=19 where the final chunk said 5 and 15. I measured it and it is real. But the code says what it is doing and why: "Use last possible chunk, in case of interruption, and gracefully handle missing intermediate tokens by taking maximum". That is a stated design choice about interrupted streams, not an oversight. Filing it as a bug would have been me not reading.

Gates, all captured:

Gate	Result
`pytest tests/integrations/google_genai/`	511 passed, 1 skipped (510 before, one new test)
new assertions, source reverted	5 failed, all `KeyError: 'gen_ai.response.id'`
`ruff check` and `ruff format --check`	clean
`uv run --group typing mypy sentry_sdk`	28 errors before, 28 after, diff identical
the differential probe	6/6 shapes now report parity
real Gemini through a real DSN	both attributes present on the streaming span

The 28 mypy errors are pre-existing on master. I diffed the output rather than claiming zero, because "mypy passes" would not have been true.

Best Use of Sentry

Sentry is not decoration on this one. It is where the second, worse bug was visible.

AI agent monitoring is the surface. The integration emits gen_ai.chat spans that Sentry renders as first-class AI spans with the model, a token breakdown, cost and context utilisation, plus an Agent Timeline tab beside the waterfall. That panel is what showed me gen_ai.response.model populated on a span where the SDK had sent no such attribute. A local test could not have told me that, because the fallback lives on Sentry's side, not in the SDK.

To be exact about what I verified: I know the eleven attributes the SDK sent and I know what the UI displayed. That the fallback is gen_ai.request.model is an inference from those two facts plus the alias run, not from reading Sentry's server code.

Distributed tracing gave me the control. Each run is one transaction with the gen_ai.chat span nested under http.server and the outbound POST generativelanguage.googleapis.com under that. Running streaming and non-streaming in separate transactions in the same environment made the non-streaming span a control I could put beside the broken one. Same prompt, same commit, one attribute panel each. The bug is not an argument at that point, it is a diff.

Environments carried before and after. bugsmash-before, bugsmash-after and bugsmash-unfixed-alias, one release string per run, so the two states coexist in one project and neither screenshot depends on my memory of which build was live.

Everything is real traffic. Real Gemini API, real DSN, EU region ingest. The evidence script sends both runs and reads the attributes off the transport, which is worth a note for anyone else instrumenting this: stream_gen_ai_spans defaults to True, so gen_ai.* spans leave as their own span envelope items rather than inside the transaction payload. My first capture read before_send_transaction and found zero gen_ai spans. If your AI spans look missing, that is why.

Best Use of Google AI

Google AI is where the ground truth came from. It changed the fix.

Gemini's own traffic settled the accumulation rule. The repo's fixture says the metadata is on the first chunk. That is a guess. I ran real streams and counted: response_id and model_version on 4 of 4 chunks for gemini-2.5-flash, 8 of 8 for gemini-flash-latest, one distinct value each, stable across the stream. Fixtures tell you what someone assumed. The API tells you what happens.

A Gemini alias produced the smoking gun. The wrong-model bug is invisible on a pinned model id. It only appears when the served model differs from the requested one, which is what gemini-flash-latest does: I asked for the alias and model_version came back gemini-3.6-flash. That is not a contrived case, it is the documented purpose of a -latest alias. It is exactly the case gen_ai.response.model exists to record. Without a real Google-native key there is no way to discover that, because a fixture would have whatever value I typed into it.

Longer generations made the stream multi-chunk. A short prompt returns a single chunk, which hides any accumulation bug. Asking Gemini for a 120 word paragraph gave 4 to 8 chunks per run, so the last-non-None rule was tested against real chunk boundaries rather than a two-element list I invented.

What I would tell the next person

Read what your instrumentation sends, not what the dashboard shows you. The dashboard is allowed to be helpful. Helpful is not the same as accurate. A row that is populated from a plausible neighbour looks exactly like a row that is right, so it passes every eyeball review you throw at it.

And when a five month old issue has a careful None guard sitting on top of it, check whether the value can ever be anything else.

Written with AI assistance (Claude, Anthropic). The bug hunt, the probe design, the fix and every number above are mine and were verified before publishing: the differential probe over six response shapes, the google_genai suite at 511 passed and 1 skipped, the same suite at 5 failed with the source reverted, ruff clean, mypy diffed against master rather than claimed clean, plus both Sentry runs captured from real Gemini traffic against a real DSN. Every screenshot is an unedited capture of my own project.