TL;DR — I instrumented a deterministic, content-blind AI workflow enforcer with the emerging OpenTelemetry GenAI semantic conventions. It technically worked, but pinched in six specific places — each one exposing the same hidden assumption: that a GenAI system is, by definition, a model invoker. This post catalogs the six frictions, shows the design I shipped instead, and follows one friction into a live semantic-conventions discussion.
What you'll get out of it
- A concrete map of where the GenAI conventions assume a model sits at the center of every span
- A reusable pattern for instrumenting orchestration / governance / policy layers that don't call models
- Why "refused" must not be encoded as "errored," and how to fix that in one line
- How a friction log — not a formal proposal — can still move a standard
Contents
- The system under instrumentation
- The six frictions (at a glance)
- The frictions in detail
- What I shipped instead: own the namespace
- The pattern underneath
- Where one friction ended up
- Takeaways
The system under instrumentation
First, a disambiguation, because "LLM router" is an overloaded term.
Most projects with that name are model-selection routers — they pick between a cheap and an expensive model based on query complexity or cost. This system is not that. It's a topology-enforcement layer: given structured metadata about an interaction, it answers exactly one question —
Is this workflow transition permitted?
— independent of model choice or content. It validates workflow topology at load time, enforces invocation limits, and returns one of three terminal decisions: PROCEED, REFUSE, or PAUSE.
The defining property, for instrumentation purposes: it is content-blind by design. It never sees a prompt, never calls a provider, never generates a token. It observes metadata and structure — nothing else.
The experiment wrapped two operations in OTel spans:
| Operation | What it does |
|---|---|
validate_workflow_config() |
Static structural validation, before anything runs |
WorkflowEngine.evaluate() |
The per-transition permission decision |
The rule I set myself: use the existing GenAI conventions everywhere they fit, and write down every place they don't. That log of mismatches is the heart of this post.
The six frictions (at a glance)
| # | Friction | The hidden assumption it exposes |
|---|---|---|
| 1 | No operation name for validation | Observability begins only when inference begins |
| 2 |
provider.name assumes a provider |
Every GenAI span calls a model |
| 3 | No vocabulary for topology | The unit of interest is a conversation, not a graph position |
| 4 | A refusal reads as an error | The only non-success outcome is failure |
| 5 | Content observability is the default | Instrumentation means watching prompts and completions |
| 6 | No trace-parenting for standalone runs | Workflow spans are children of a model span |
Read together, they all say one thing: the conventions model a GenAI system as a model invoker. The detail sections below walk each one; skip to What I shipped instead if you just want the design.
The frictions in detail
Friction 1 — There's no operation name for validation
What broke. The current gen_ai.operation.name values all describe runtime inference: chat, generate_content, invoke_agent, retrieval, execute_tool. None describe static structural validation, so the only accurate value was a non-standard one like validate_workflow_config.
Why it matters. The conventions treat GenAI observability as beginning when model activity begins. But for deterministic orchestration systems, much of the real safety work happens before execution starts. The spec has no vocabulary for that phase at all.
Friction 2 — gen_ai.provider.name assumes a provider exists
What broke. The system has no provider. It doesn't call OpenAI, Anthropic, Gemini, or a local inference server, so the attribute is simply omitted — leaving the spans technically incomplete relative to the model.
Why it matters. That's fine when every GenAI span lives inside a model invocation. But it presumes the category of system in scope: model invokers. Orchestration, governance, and policy-enforcement layers that sit adjacent to models have no home here — LangGraph's graph-execution layer, Temporal-style workflow runtimes wrapping LLM steps, or guardrail frameworks like NeMo Guardrails that gate execution without generating anything.
Friction 3 — Workflow topology has no vocabulary
What broke. A meaningful evaluation span needs three facts: the current container, the requested transition target, and the terminal decision. No gen_ai.* attribute represents any of them. The nearest neighbor, gen_ai.conversation.id, describes conversational identity — not structural position in a graph.
Why it matters. Using conversation semantics for topology would produce actively misleading telemetry. This is the clearest point where the model shows its conversational bias: the spec can describe who is talking, but not where in the graph you are.
Friction 4 — A refusal is not an error
What broke. The system emits typed structural refusal reasons — UNKNOWN_CONTAINER, INVALID_TRANSITION, MAX_INVOCATION_EXCEEDED, CONTAINER_REENTRY_BLOCKED. The obvious attribute, error.type, describes failures of execution: timeouts, API errors, exceptions.
Why it matters. A refusal is the opposite of a failure — the system is working correctly when it refuses an invalid transition. Recording enforcement outcomes as errors poisons every downstream error-rate dashboard. "The system failed" and "the system said no, on purpose" are fundamentally different, and the conventions can't tell them apart.
Friction 5 — Content observability is treated as the default
What broke. The conventions center on input/output messages, prompts, completions, tool calls, and retrieval context. This system observes none of those, by design. Its stance is structural, not semantic: it traces topology, transition legality, and enforcement outcomes while staying blind to user content.
Why it matters. That stance is a privacy and an architecture decision. But the spec implicitly treats content observability as the natural shape of GenAI instrumentation, with no notion of a system whose entire telemetry story is structural.
Friction 6 — Standalone evaluations have no trace-parenting model
What broke. Every evaluation produced an isolated trace (parent_id: null), because there was no upstream model-invocation span to inherit context from — and the conventions imply workflow spans exist as children of model or agent spans.
Why it matters. The result is observability fragmentation: config validation and the evaluations that depend on it become disconnected islands instead of one structural lifecycle. There's no guidance for relating these spans when no model runtime sits above them.
What I shipped instead: own the namespace
The conclusion wasn't "force the fit." It was: don't borrow gen_ai.* at all. The shipped instrumentation defines its own documented, versioned convention surface under a wfrouter.* namespace (schema version 1.0.0), on four design rules.
The four design rules
1. Zero hard dependency. The engine never imports the instrumentation; the instrumentation imports the engine. If opentelemetry-api isn't installed, every function degrades to a no-op and behavior is unchanged.
2. The engine stays pure. Nothing is instrumented inside evaluate(). The wrapper traces it from the outside, so the engine stays deterministic, stateless, and trivially unit-testable with no telemetry in the way.
3. Own the namespace, own the versioning. Every attribute lives under wfrouter.*. When the still-evolving gen_ai.* conventions change, nothing moves underneath this schema. If interop ever justifies it, gen_ai.* attributes can be emitted alongside, additively.
4. A refusal sets span status OK. All three terminal states — PROCEED, REFUSE, PAUSE — are valid, successful outcomes. Status ERROR is reserved for genuine failures: a config that fails validation, or an unexpected exception. That's Friction 4, resolved in code.
The convention surface
Three low-cardinality span names — the container name is always an attribute, never part of a span name:
wfrouter.evaluate
wfrouter.validate_config
wfrouter.analyze_topology
Attributes on the wfrouter.evaluate span:
| Group | Attributes |
|---|---|
| Input |
wfrouter.container, wfrouter.requested_action, wfrouter.previous_state, wfrouter.invocation_depth
|
| Decision |
wfrouter.decision, wfrouter.refused, wfrouter.reason_code
|
| Correlation |
wfrouter.correlation_id — app-level, kept distinct from the OTel trace ID so you can join on either |
Two deliberate cardinality decisions worth stealing:
- The raw transition history is not emitted — it's unbounded. Only
wfrouter.transition_history.lengthis recorded. - Topology issues become span events (
wfrouter.topology_issue), each carrying a severity and a stable, greppable code — not one attribute per issue.
And two metric instruments, both with bounded attribute sets so aggregation stays safe:
| Instrument | Type | Attributes |
|---|---|---|
wfrouter.evaluations |
counter |
decision, reason_code, container
|
wfrouter.invocation_depth |
histogram | container |
That first counter gives you things like "REFUSE rate by container" for free.
Using it
pip install "llm-workflow-router[otel]"
from router.engine import WorkflowEngine
from router.observability.otel import ObservableEngine
engine = ObservableEngine(WorkflowEngine(cfg))
result = engine.evaluate(metadata)
ObservableEngine is a drop-in for WorkflowEngine at any call site that wants telemetry. The pure engine underneath never changes.
The pattern underneath
Individually, each friction is small. Together they point at one thing:
The current GenAI conventions model AI systems primarily as systems of generation — prompts in, completions out, tools called, context retrieved.
That's a valid model for a large slice of the ecosystem. But a whole class of systems already exists that participates in AI execution pipelines without generating anything: orchestration engines, workflow runtimes, safety gates, validation layers, execution governors. They need observability too, and their semantics are structural, not conversational. Today, each of them has to do what I did — step outside the spec and own a namespace.
A friction log is valuable precisely because it makes a spec's hidden assumptions visible. This one suggests the conventions may eventually need to separate three things now blurred together:
- Inference semantics
- Structural orchestration semantics
- Governance / enforcement semantics
Where one friction ended up
Here's the part I didn't expect.
I left the friction log as a comment on an open semantic-conventions discussion about agentic authorization — a proposal introducing producer-emitted decision signals like gen_ai.agent.trust_score and gen_ai.agent.drift_score.
A practitioner in that thread, building the producer side in production, surfaced a failure mode you only see once a system is running: a producer's scoring function isn't fixed. Models get re-fit, thresholds move, feature sets change. The emitted score then shifts for a reason that has nothing to do with the agent's behavior — and a consumer correlating scores against outcomes can't tell whether the agent changed or the measurement changed.
That clicked against my frictions, and I framed it the only way I could:
A signal is only worth anything if you can tell when the ruler itself changed length.
The shape the discussion converged on is a minimal provenance pattern — an opaque, producer-scoped method token next to each score:
gen_ai.agent.trust_score.method
gen_ai.agent.drift_score.method
gen_ai.agent.scan_verdict.method
Consumers never parse the token; they compare it for equality only. Same token → scores stay comparable. Changed token → the measurement function was replaced, so reset your baseline instead of chasing a phantom drift event. One disciplined pattern instead of three special cases — and it generalizes to any derived, evolving signal emitted for downstream correlation.
An observation from an odd little friction log, sharpened by someone hitting the same wall from the production side, is now part of a live convention discussion. That's standards work at its best: a practitioner names a blind spot, the principle generalizes, and the fix is smaller than anyone expected.
Takeaways
- Instrument the weird system you have. Edge cases are exactly what an in-development spec needs. A mismatch is information — write it down.
-
Refusals are not errors — encode that in span status. Keep enforcement outcomes out of your error semantics from day one.
REFUSEwith statusOKis one line of code, and it saves every error-rate dashboard downstream. - When a spec doesn't fit, own a namespace instead of forcing the fit. A small, documented, versioned surface is more honest and more stable than misusing someone else's vocabulary — and it leaves room to emit standard attributes alongside later.
- Guard your cardinality on purpose. No container names in span names, history length not the raw list, bounded metric attribute sets. Decisions, not defaults.
- Friction logs are a real contribution path. You don't need to author a proposal to move a standard. Sometimes you just need to describe, precisely, where the shoe pinched.
The GenAI semantic conventions are still marked experimental — which makes this the cheapest moment to give this kind of feedback. If you're instrumenting anything AI-adjacent that isn't a model invoker, go log your frictions. The spec gets better every time someone maps its edges.
Top comments (0)