I build small LLM agents. Not the impressive kind you see in demos, just
practical little things that answer support questions or dig through some
...
For further actions, you may consider blocking this person and/or reporting abuse
Keeping the raw payload so old data can be replayed through a better translator later is the decision I'd steal. It turns your normalization from something you have to get right on day one into something you can be wrong about and fix, which is the only realistic plan given three dialects that keep moving. The blank-instead-of-a-guessed-price call is the same instinct, and both are the kind of thing that only shows up when one person is the whole roadmap.
The translation-layer section is the part I'd underline, so I pulled the bundles before saying anything —
npm packon ai@7.0.30, @ai-sdk/otel@1.0.30 and ai@6.0.229, then read the shipped dist. Three things that might save your parser some pain.Your "mixes old and new in the same span" is real, and I can hand you the exact shape. In ai@6.0.229 the core still emits spans itself (
getTracerappears 9×), and 4 of its 34 attribute objects put vendorai.*right next togen_ai.system— which is the pre-rename name ofgen_ai.provider.name(0 occurrences in v6). One span, three vintages:{ai.prompt.messages, gen_ai.request.model, gen_ai.system}. You weren't imagining it.But the ground moved under that parser in v7. ai@7.0.30 has zero opentelemetry deps, and zero occurrences of
opentelemetry,getTracer,startSpanorgen_ai.in its bundle. Telemetry became a callback interface (registerTelemetry(),onStart/onStepStart) and span emission moved out to@ai-sdk/otel. Consequence for Otterscope: an agent that upgrades to v7 and doesn't add that package stops sending traces — no error, no empty span, the runs just never arrive, and it's indistinguishable from "nothing ran." (That's read from the bundle rather than measured on a live v7 agent — but there's no tracer left in core to measure.)The dialect is now a user choice, not a version fact.
@ai-sdk/otel@1.0.30exports two builders:OpenTelemetry(30gen_ai.*, no pre-rename keys) andLegacyOpenTelemetry(59ai.*+ 14gen_ai.*, still shippinggen_ai.system, 2 of its 11 attribute objects still mixed). Same SDK version, two shapes on the wire, depending on which class the user registered.Why single-attribute sniffing loses: 13
gen_ai.*keys show up in both builders —gen_ai.request.modelamong them. Only two are unambiguous:gen_ai.provider.name⇒ new builder,gen_ai.system⇒ Legacy. Key on those two and the guess goes away.Keeping the raw payload for replay is the decision that ages well here — it's what lets you fix the mapping after the dialect shifts under you again.
I went and pulled the current versions rather than take this, and it holds, with one trap in the detection rule.
ai@7.0.99, shipped dist: zero getTracer, zero gen_ai.* keys, zero @opentelemetry imports. registerTelemetry is present. So the silent failure you describe is current behaviour and not a 7.0.30 artifact. Upgrade to v7 without adding @ai-sdk/otel and there is nothing left in core to emit, which makes "no runs arrived" and "nothing ran" the same state on my side.
@ai-sdk/otel@1.0.99 still exports exactly the two classes. Your two-key rule survives: all 9 gen_ai.provider.name writes sit inside the OpenTelemetry class, both bare gen_ai.system writes sit inside LegacyOpenTelemetry.
The trap is that the new builder writes gen_ai.system_instructions four times. Substring match on gen_ai.system and every new-builder span gets filed as Legacy. It has to be an exact key comparison. That one would have cost me an afternoon, so thank you.
The privacy point is the one I feel most, since agent traces are basically raw customer messages and shipping them to a hosted dashboard is a non-starter for support work. The four-stateful-service tax for a few thousand calls a day is real too. When you kept it tiny, did you still capture per-step token and tool-call spans, or trim down to just the run timeline?
Kept them. The run timeline is the top level, not the whole thing. Open a step and you get the actual messages that went in and came back, with the token count and cost sitting on that call.
Trimming to the timeline was tempting and would have removed the reason I built it. "Nine seconds and forty cents" is only actionable once you can see which of three tool calls burned it, and the run-level total tells you a bad run happened without telling you where.
On the four-service tax, the thing that makes it cheap is not sampling the detail away. A few thousand calls a day is a small table. The scale that justifies a separate stateful stack is well above where most people asking for this actually are, which is most of the argument.
the my-laptop-scale framing is the most honest thing said in the observability space in a while - the forklift-for-groceries problem is real. the feature that would make this a daily driver follows from your own sentence: "you tweak a prompt, ship it, and you cannot tell if you made it better or worse". a side-by-side of two run groups - same inputs, before and after a prompt version - beats any dashboard at that job: token deltas, tool-call counts, which runs diverged and at what step. does otterscope group runs by prompt or config version yet, or is that roadmap territory?
Both, and the code answers it better than I would from memory, so I went and read it.
The dimension exists. gen_ai.prompt.name and gen_ai.prompt.version get normalised into one prompt identity on the run, so "support-reply v3", and the run carries it aggregated. The run filter takes a prompt param, and the compare view calls the runs and stats endpoints once per side with whatever filter each side is given. So your side-by-side already works: support-reply v2 on the left, v3 on the right, token and cost totals per side. Time windows were just the example I used in the post, not the only axis. OpenInference's llm.prompt_template.name feeds the same field.
Three honest gaps. The match is a substring against a display string rather than a real version dimension, so v3 also matches v30, and reply matches both. There is no config version at all, only the prompt. And on the Vercel AI SDK you get none of this, because that dialect never maps a prompt identity, which is a bug rather than roadmap and I'll fix it.
The part you actually asked for that is not built: which runs diverged and at what step. Compare is aggregate per side, so it can tell you v3 got worse and not where. That is the roadmap item, and you've picked the right one.
I hit the same wall building small agent setups of my own. The agent's summary of what it did and what its diff or logs actually show can drift apart fast, and the summary is always the more flattering version. What convinced me observability was worth the setup cost was catching a case where the agent reported success but the actual tool call had silently failed and it just moved on. Does your tool surface that kind of mismatch directly, or is it still on the human to cross reference the log against the claim?
Not automatically, no. What you get is both halves in one place: the tool call with its real return or error, and the model's account of it, inside the same run instead of across two log files. The cross-reference is still yours to do.
The gap is real and your example is the clearest argument for closing it. The place it belongs is an eval check stamped onto the run, because that is already where results land next to production traffic rather than in a side dataset, and "claimed success while a step errored" is a check that needs no judge model and no labelled set. Structural, cheap, and it would have caught your case on the run itself.
Great write-up. The translation layer handling all three OTel dialects is the part that would quietly save people hours of debugging and most wouldn't even notice it's there.
If your AI agents kept a diary, what would surprise you most (snacks included)?