This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.
Project Overview
The project is Sentry's own Python SDK, getsentry/sentry-python. Its google_genai integration is what turns a Gemini call into a gen_ai.chat span, so your AI monitoring dashboard has token counts, finish reasons, cost and the model that answered. It ships in the same SDK as the error monitoring, which means for a lot of teams it is the only record of what their agent actually did in production.
I went in to fix a five month old issue about two missing span attributes. What I found was worse than missing.
Bug Fix or Performance Improvement
Issue #5812 says streaming responses never capture response_id or model_version. It is exact and it is correct. accumulate_streaming_response in google_genai/streaming.py declares both locals, reads text, finish reasons, tool calls and five token counts out of the chunks, then hands the two declared locals to the span without ever reading them:
usage_data = None
response_id = None # declared
model = None # declared
for chunk in chunks:
... # everything else: read from the chunk
# response_id and model: never touched
accumulated_response = AccumulatedResponse(
...
id=response_id, # always None
model=model, # always None
)
The non-streaming path in the same integration reads both off the response, so the two halves of one integration disagree.
Before writing a line of fix I wanted to know the true blast radius, so I built a differential probe. It sends identical content through the real integration twice, once as one whole GenerateContentResponse and once as a chunk sequence, then diffs the gen_ai.* attributes that land on the chat span. Six response shapes: plain text, reasoning plus cached tokens, a tool call, a MAX_TOKENS finish, metadata only on the final chunk, then no usage metadata at all.
SUMMARY: attributes broken on the streaming path
gen_ai.response.id in 6/6 cases
gen_ai.response.model in 6/6 cases
Nothing else diverged. Text, finish reasons, tool calls and all five token counts match the non-streaming path in every shape. AsyncModels.generate_content_stream is affected identically. So this is one narrow defect, not a broken integration, which is exactly what you want to know before touching anything.
Then Sentry showed me the real bug
I wired my Sentry DSN in, ran real Gemini traffic through the unfixed code, then opened the span. gen_ai.response.model was there. My own bug report appeared to be wrong.
It was not. I dumped the eleven attributes the SDK actually put on the wire and gen_ai.response.model was not among them. Sentry was filling the row in from gen_ai.request.model, the model I asked for.
For gemini-2.5-flash you cannot see the difference, because ask and served are the same string. So I ran it again against gemini-flash-latest, an alias, with the source reverted to the parent commit. Same prompt, same commit, one streaming and one not:
| Span | gen_ai.response.id |
gen_ai.response.model |
|---|---|---|
| streaming (the bug) | no row at all | gemini-flash-latest |
| non-streaming (control) | 1_l6armtIJOKqfkPnai9yQM |
gemini-3.6-flash |
Google served gemini-3.6-flash. The streaming trace says gemini-flash-latest.
Same prompt, same commit, side by side in Sentry. Left, the streaming span reports gemini-flash-latest, the alias I asked for. Right, the non-streaming control reports gemini-3.6-flash, the model Google actually served. The streaming span also has no response.id row at all.
That reframes the whole issue. A blank field is honest, you know you have nothing. A field silently backfilled from the request is a confident wrong answer. gen_ai.response.model exists precisely to record the case where those two differ. Every production streaming span, which is to say every user-facing chat, attributes the generation to whatever alias the caller happened to type.
gen_ai.response.id has no fallback, so it is simply gone. It is the only handle that ties a Sentry span to a specific Gemini response, which is what you need when you take a bad generation to Google.
Code
The fix: getsentry/sentry-python, branch fix/google-genai-streaming-response-metadata, commit 5b4c291.
Six lines of source, inside the loop that was already walking every chunk:
# Gemini repeats the response id and the served model version on every
# chunk, but keep the last non-None value so a chunk that omits either
# one does not discard it.
response_id = getattr(chunk, "response_id", None) or response_id
model = getattr(chunk, "model_version", None) or model
Plus 85 lines of test. Two assertions added to the existing streaming test, plus one new async streaming test where only the first chunk carries the metadata.
The streaming span before and after the fix. gen_ai.response.id goes from absent to recorded, response.model now carries the served value instead of leaning on the UI fallback.
Per Sentry's CONTRIBUTING.md a contribution needs maintainer agreement on the issue before a PR exists. A bot enforces that. Two earlier PRs on this exact issue were auto-closed for skipping that step, neither reviewed on merit. So I posted the verified finding on #5812 and asked to pick it up. The PR follows once a maintainer answers. The branch above is the full change either way.
My Improvements
Last non-None, not first. Real traffic decided this. I measured live Gemini streams and every chunk carries both fields: 4 of 4 chunks on gemini-2.5-flash, 8 of 8 on gemini-flash-latest. But this repo's own streaming fixture puts them on the first chunk only. So first-wins passes the tests while last-wins-without-a-guard would let a metadata-free chunk clear a good value. Last non-None satisfies both, matching the convention the anthropic integration in the same SDK already uses (model = event.message.model or model).
Prove the test catches it. A new assertion that passes alongside a fix proves nothing. I reverted the source, kept the tests, then ran again:
5 failed, 68 passed
FAILED test_streaming_generate_content[True-True] - KeyError: 'gen_ai.response.id'
FAILED test_streaming_generate_content[True-False] - KeyError: 'gen_ai.response.id'
FAILED test_streaming_generate_content[False-True] - KeyError: 'gen_ai.response.id'
FAILED test_streaming_generate_content[False-False]- KeyError: 'gen_ai.response.id'
FAILED test_async_streaming_response_id_and_model - KeyError: 'gen_ai.response.id'
Why 510 passing tests missed it, which I think is the interesting part. Two reasons, neither of them carelessness.
The fixture at test_google_genai.py:743 already carries "responseId": "response-id-stream-123" and "modelVersion": "gemini-1.5-flash". The data was on the way in. No streaming test asserted either attribute reached the span. The only assertions on those two constants in the whole 4,760 line file are in the non-streaming test. The suite tested the door, not the delivery.
Then a hardening commit made the dead code look deliberate. a66281a, "Guard against None response ID and response model", rewrote the consumer:
- if accumulated_response.get("id"):
- span.set_data(SPANDATA.GEN_AI_RESPONSE_ID, accumulated_response["id"])
+ response_id = accumulated_response.get("id")
+ if response_id is not None:
+ set_on_span(SPANDATA.GEN_AI_RESPONSE_ID, response_id)
That is a careful None guard on a value that is always None. It reads as intentional, so the producer never got a second look. A guard is not a test.
What I deliberately did not report. element_wise_usage_max takes an independent per-field maximum across chunks, so a stream whose candidate count dips on the final chunk reports output_tokens=9, total_tokens=19 where the final chunk said 5 and 15. I measured it and it is real. But the code says what it is doing and why: "Use last possible chunk, in case of interruption, and gracefully handle missing intermediate tokens by taking maximum". That is a stated design choice about interrupted streams, not an oversight. Filing it as a bug would have been me not reading.
Gates, all captured:
| Gate | Result |
|---|---|
pytest tests/integrations/google_genai/ |
511 passed, 1 skipped (510 before, one new test) |
| new assertions, source reverted | 5 failed, all KeyError: 'gen_ai.response.id'
|
ruff check and ruff format --check
|
clean |
uv run --group typing mypy sentry_sdk |
28 errors before, 28 after, diff identical |
| the differential probe | 6/6 shapes now report parity |
| real Gemini through a real DSN | both attributes present on the streaming span |
The 28 mypy errors are pre-existing on master. I diffed the output rather than claiming zero, because "mypy passes" would not have been true.
Best Use of Sentry
Sentry is not decoration on this one. It is where the second, worse bug was visible.
AI agent monitoring is the surface. The integration emits gen_ai.chat spans that Sentry renders as first-class AI spans with the model, a token breakdown, cost and context utilisation, plus an Agent Timeline tab beside the waterfall. That panel is what showed me gen_ai.response.model populated on a span where the SDK had sent no such attribute. A local test could not have told me that, because the fallback lives on Sentry's side, not in the SDK.
To be exact about what I verified: I know the eleven attributes the SDK sent and I know what the UI displayed. That the fallback is gen_ai.request.model is an inference from those two facts plus the alias run, not from reading Sentry's server code.
Distributed tracing gave me the control. Each run is one transaction with the gen_ai.chat span nested under http.server and the outbound POST generativelanguage.googleapis.com under that. Running streaming and non-streaming in separate transactions in the same environment made the non-streaming span a control I could put beside the broken one. Same prompt, same commit, one attribute panel each. The bug is not an argument at that point, it is a diff.
Environments carried before and after. bugsmash-before, bugsmash-after and bugsmash-unfixed-alias, one release string per run, so the two states coexist in one project and neither screenshot depends on my memory of which build was live.
Everything is real traffic. Real Gemini API, real DSN, EU region ingest. The evidence script sends both runs and reads the attributes off the transport, which is worth a note for anyone else instrumenting this: stream_gen_ai_spans defaults to True, so gen_ai.* spans leave as their own span envelope items rather than inside the transaction payload. My first capture read before_send_transaction and found zero gen_ai spans. If your AI spans look missing, that is why.
Best Use of Google AI
Google AI is where the ground truth came from. It changed the fix.
Gemini's own traffic settled the accumulation rule. The repo's fixture says the metadata is on the first chunk. That is a guess. I ran real streams and counted: response_id and model_version on 4 of 4 chunks for gemini-2.5-flash, 8 of 8 for gemini-flash-latest, one distinct value each, stable across the stream. Fixtures tell you what someone assumed. The API tells you what happens.
A Gemini alias produced the smoking gun. The wrong-model bug is invisible on a pinned model id. It only appears when the served model differs from the requested one, which is what gemini-flash-latest does: I asked for the alias and model_version came back gemini-3.6-flash. That is not a contrived case, it is the documented purpose of a -latest alias. It is exactly the case gen_ai.response.model exists to record. Without a real Google-native key there is no way to discover that, because a fixture would have whatever value I typed into it.
Longer generations made the stream multi-chunk. A short prompt returns a single chunk, which hides any accumulation bug. Asking Gemini for a 120 word paragraph gave 4 to 8 chunks per run, so the last-non-None rule was tested against real chunk boundaries rather than a two-element list I invented.
What I would tell the next person
Read what your instrumentation sends, not what the dashboard shows you. The dashboard is allowed to be helpful. Helpful is not the same as accurate. A row that is populated from a plausible neighbour looks exactly like a row that is right, so it passes every eyeball review you throw at it.
And when a five month old issue has a careful None guard sitting on top of it, check whether the value can ever be anything else.
Written with AI assistance (Claude, Anthropic). The bug hunt, the probe design, the fix and every number above are mine and were verified before publishing: the differential probe over six response shapes, the google_genai suite at 511 passed and 1 skipped, the same suite at 5 failed with the source reverted, ruff clean, mypy diffed against master rather than claimed clean, plus both Sentry runs captured from real Gemini traffic against a real DSN. Every screenshot is an unedited capture of my own project.
Top comments (0)