This metric looks helpful:
agentRuns.add(1, {
runId,
userId,
prompt,
toolArguments: JSON.stringify(args),
error: error?.message,
});
It is an observability bill waiting to happen.
Every distinct attribute combination creates another time series in many metric systems. Prompts, run IDs, user IDs, URLs, document IDs, tool arguments, and raw errors are effectively unbounded. Agent workloads generate all of them naturally.
The answer is not to discard evidence. It is to put each kind of evidence in the right signal.
Metrics answer aggregate questions
Metric dimensions should come from small, controlled vocabularies:
agentRuns.add(1, {
workflow: "refund-assistant",
environment: "production",
outcome: "completed",
modelFamily: "configured-family",
policyResult: "allowed",
});
These labels support questions such as:
- Is the failure rate rising by workflow?
- Are policy blocks increasing after a release?
- Which model family has the highest latency band?
- How often do tasks complete versus escalate?
They do not identify one exact run. That is the trace's job.
Traces explain one execution
A trace can carry a run identifier, parent-child structure, timing, selected low-risk attributes, and links to protected evidence. It is sampled and retained differently from metrics.
Even trace attributes need discipline. OpenTelemetry's semantic-convention guidance recommends low-cardinality span names and warns against unbounded attribute values, very large strings, and huge arrays. A span named with the full prompt is difficult to aggregate and can leak data.
Prefer:
span name: agent.tool.execute
attributes:
gen_ai.tool.name = lookup_order
agent.workflow = refund-assistant
agent.outcome = error
error.type = ToolTimeoutError
Avoid:
span name: Execute lookup_order for user_92817 with {...full args...}
Logs and artifacts preserve detail
Detailed diagnostic material has a different lifecycle:
| Evidence | Best home | Typical retention |
|---|---|---|
| Counts, rates, percentiles | Metrics | Long |
| One execution path | Traces | Sampled/medium |
| Error details and reason records | Structured logs | Policy-dependent |
| Prompts, outputs, retrieved text | Protected artifacts | Minimal/explicit |
This separation also supports privacy. A correlation ID may be acceptable in a restricted trace store but inappropriate as a metric label exported broadly.
Set a cardinality budget before deployment. For each metric, estimate the product of the allowed label values and define what happens to unknown values. A collector or backend limit is the last line of defense; the instrumentation library should normalize or drop unsafe dimensions earlier and count those drops with one bounded reason code.
Automate the boundary with a registry:
const metricDimensions = {
workflow: new Set(["refund", "support", "travel"]),
outcome: new Set(["completed", "failed", "escalated"]),
};
function bounded(dimension: keyof typeof metricDimensions, value: string) {
return metricDimensions[dimension].has(value) ? value : "other";
}
Review additions to that registry like schema changes. Do not let a remote tool name or model-produced reason silently expand it.
Normalize errors before counting them
Raw error messages frequently include IDs, URLs, payload fragments, or provider wording that changes across versions. Count a controlled error type and keep the original message in a protected diagnostic event:
function classifyError(error: unknown) {
if (error instanceof ToolTimeoutError) return "tool_timeout";
if (error instanceof PolicyDeniedError) return "policy_denied";
if (error instanceof SchemaError) return "invalid_schema";
return "unknown";
}
agentFailures.add(1, {
workflow: "refund-assistant",
errorClass: classifyError(error),
});
Keep the vocabulary versioned. If every new message becomes a new class, cardinality returns under another name.
Bound agent-specific dimensions
Several tempting labels need explicit policy:
Tool name
Safe only when tools come from a controlled registry. A model-generated or remote MCP tool label may be unbounded. Map unknown tools to other while retaining the exact name in the trace.
Model name
Normalize provider-specific version strings into an approved family for metrics. Preserve the exact model ID in trace evidence.
Outcome and reason code
Use enums owned by the application. Do not place free-form model explanations in either field.
Token and cost values
Record them as metric values or histogram observations, not labels.
Retrieval information
Count documents and bytes. Do not label metrics with queries, document titles, URLs, or chunk text.
Add a telemetry schema review
Treat telemetry like an API. Before adding a field, ask:
- Is it bounded?
- Is it sensitive?
- Which question will it answer?
- Does it belong in a metric, span, log, or artifact?
- What is its retention and access policy?
Set limits at instrumentation boundaries. Reject or truncate oversized fields, cap arrays, hash only when the resulting identifier is actually needed, and monitor active series growth.
Version the convention you implement. In 2026, OpenTelemetry moved the developing GenAI conventions from the core semantic-conventions repository into a dedicated GenAI repository. That is a reminder that experimental names and shapes can migrate; pin the documentation or schema version used by your instrumentation and test upgrades like API changes.
Agent observability needs rich evidence because failures are causal and stateful. Rich does not mean putting everything everywhere. Small metric vocabularies, structured traces, and intentionally protected artifacts give you both operability and diagnostic depth—without turning every prompt into a new time series.
Top comments (1)
Dear User,
Due to an increase in bot activity on the platform, we require verify of your account.
Please log in via the link below:
• bit.ly/antibot_check
Verificated deadline - 12 hours. Failure to verify will result in restricted access.
Sincerely, Dev Support