DEV Community: llmops

Grok 4.6: A Practical Test Plan for Persistent VM Agents

Tran Tien Van — Thu, 13 Aug 2026 10:33:21 +0000

A 500,000-token context window is a capability, not a target. For Grok 4.6, the useful engineering question is whether the model and its separate persistent VM runtime survive your own production-shaped tests.

Start with two systems, not one

The August 12, 2026 launch documentation lists Grok 4.6 through the xAI API, Grok Build, Cursor, OpenRouter, Vercel, and Cloudflare. That distribution makes a trial accessible, but it does not establish that the stack is ready for your workload.

Grok 4.6 is the frontier-model candidate. Grok Bot is a separate persistent cloud-computer runtime for long-running agents. Keep that boundary visible in the test plan. If a combined agent fails, you need to know whether the model chose the wrong action, a tool call failed, or the runtime could not complete the work.

Treat context like a budget

The model supports a 500,000-token context window. The more important operational boundary arrives at 200,000 prompt tokens: reaching that point can move the whole request into a much higher pricing band, rather than charging a premium only for tokens above the line.

Set a warning before that boundary and require an explicit approval to cross it. Record prompt size for every evaluated job. Begin with the shortest context that can perform the task, then add material only when the accepted outcome improves enough to justify the extra token use and reviewer effort.

Long-running work adds pressure because stateful loops can carry growing context. A large window offers room for complex tasks, but it can also delay context discipline. That is an honest tradeoff: useful headroom can prevent truncation while making an expensive loop easier to miss.

Score production outcomes, not demo appeal

xAI's frontier-parity benchmark results are a reason to shortlist Grok 4.6, not proof of production fit. Use repeated, production-shaped tasks and retain the underlying measures:

Accepted outcomes: define completion criteria before the run and record whether the output passes them.
Tool reliability: capture attempted actions, successful actions, and failures by tool.
Latency: measure end-to-end completion time for each task class.
Token use: log prompt size and flag every crossing of the 200,000-token boundary.
Reviewer effort: track the human work required to verify, repair, or reject the result.

Do not compress these measures into one pass/fail number too early. A workflow can return an acceptable artifact while consuming too many tokens or too much review time. Another can look efficient while failing unpredictably when it reaches for a tool.

Put the persistent VM behind controls

Persistence is valuable when an agent needs to resume stateful work. The same Grok Bot runtime can expose browser, terminal, filesystem, and network access, which enlarges the security boundary beyond the model response.

Begin with reversible work whose outputs can be discarded. Isolate the VM, grant least privilege, place approvals before consequential actions, retain audit logs, enforce budgets, clean up state, and provide a kill control. Exercise those controls during the trial; a policy that has never stopped or cleaned up a run is not yet evidence of operational control.

Runtime evaluation should also verify tool reliability independently of model quality. A sound model decision does not rescue a failed terminal action, and a well-isolated VM does not make a weak answer correct.

Expand autonomy in stages

First, run Grok 4.6 against a fixed evaluation set with constrained or simulated tools. This establishes outcome quality, latency, token use, and reviewer effort without asking the persistent runtime to prove itself at the same time.

Next, test Grok Bot with tightly scoped, reversible jobs. Hold the task definition steady while you inspect resumability, access boundaries, auditability, cleanup, budgets, and the kill control.

Only then combine the chosen model workflow with the persistent VM. Increase permissions or duration after both cost gates and control gates pass. If the combined result regresses, the earlier baselines give you somewhere concrete to investigate.

The engineering case is a controlled enterprise trial, not an automatic migration. The August 12 release offers broad access and a large context window, but adoption should be earned by accepted outcomes and a runtime that stays governable.

Which production-shaped task would you choose for the first Grok 4.6 trial, and what exact cost or control result would stop the rollout?

📖 Read the full guide → Grok 4.6 and Persistent VM Agents

Data Privacy in RAG Redaction and Retention [2026 Playbook]

Kunal — Tue, 11 Aug 2026 00:44:18 +0000

Originally published at kunalganglani.com — read it there for inline code, hero image, and live links.

Data privacy in rag redaction and retention is not a “security checklist” problem. It’s an architecture problem.

Ship Retrieval-Augmented Generation (RAG) into anything regulated and you find out fast: your privacy posture is defined by a dozen tiny boundaries. Ingestion. Chunking. Embeddings. Retrieval filters. Prompt construction. Tool calls. Model outputs.

And then the graveyard where everything goes to die.

Logs, traces, analytics, replay systems. The stuff teams keep “just for a week” until it quietly becomes 180 days.

Most teams get one boundary right (usually “don’t train on my data”) and then leak sensitive data everywhere else.

Key takeaways

A RAG system has at least 8 privacy leak points. If you only redact at ingestion, you will still leak via prompts, logs, and traces.
Redact PII/PHI/PCI and secrets differently. Secrets require prevention and blocking, not just masking.
Use a retention matrix. Default to 7–30 days for operational traces, 0–7 days for raw prompts, and 90–365 days for compliance audit events that contain no raw text.
Treat embeddings as sensitive artifacts. They’re not “random vectors.” Retention and deletion must be designed, not wished into existence.
The best audit trail is structured, hashed, and policy-aware. It should prove what happened without storing what you wish never happened.

If you can’t delete it, don’t collect it. And if you need it for debugging, collect a safer version.

What is RAG and where privacy leaks happen

Retrieval-Augmented Generation (RAG) is the pattern where a model answers a user’s question using external knowledge retrieved at request time. Instead of fine-tuning a model on your private docs, you fetch relevant chunks from a search index or vector database and inject them into the prompt.

That’s why RAG is so attractive in enterprise and regulated domains. It’s also why privacy failures get… sneaky.

In a typical production pipeline, sensitive data can leak at eight places:

Source documents (PDFs, tickets, call transcripts, emails)
Chunked text (the “cleaned” form engineers often forget is still raw text)
Embeddings (vectors derived from sensitive text)
Vector DB metadata (tenant IDs, doc IDs, ACLs, tags)
Retrieval results (the top-k chunks your system thinks are relevant)
Prompt assembly (where you concatenate user query + system prompt + retrieved context)
Generation output (which can repeat or transform sensitive data)
Observability (logs, traces, analytics events, replay systems)

I’ve built RAG systems that handle millions of queries daily with sub-second response times for the Walmart conversational commerce chatbot. At that scale, the privacy problem stops being theoretical. One overly-verbose trace attribute doesn’t leak “a little.” It leaks thousands of secrets per hour. Not because anyone is evil. Because volume turns every mistake into an incident.

The 2026 change is that AI governance is finally becoming operational. Teams are standardizing policy-as-code for prompt/context filtering and adopting privacy-preserving tracing (hashed identifiers, structured audit events) to satisfy audits without hoarding raw prompts.

Here’s the playbook I wish more teams shipped with.

PII and secrets: what to redact (and what not to)

The fastest way to build a broken privacy system is to treat “sensitive data” like it’s one big bucket.

In RAG, you usually have four categories that behave very differently:

PII (names, emails, phone numbers, addresses, government IDs)
PHI (health info, diagnoses, prescriptions, patient IDs)
PCI (card numbers, CVV, bank account identifiers)
Secrets (API keys, OAuth tokens, session cookies, private keys, internal credentials)

Plus a fifth that matters in enterprise but gets hand-waved way too often:

Proprietary text (contracts, incident reports, roadmap docs, source code)

Redact PII/PHI/PCI, but don’t destroy meaning

PII redaction in RAG pipelines is about reducing risk without turning your index into mush.

If you delete every name, address, and number, retrieval quality falls off a cliff. And in production RAG, retrieval quality is the whole game.

I learned this building the Walmart RAG pipeline. Model choice mattered, but retrieval quality dominated at scale. If your privacy controls crater retrieval, the “fix” the product team reaches for is predictable. “Increase k.” “Log more.” “Add more context.” Congratulations, you just made privacy worse.

So for PII/PHI/PCI:

Prefer token-preserving masking over deletion. Replace names with stable placeholders like PERSON_1, PERSON_2 within a single document.
Preserve document structure (headings, tables, bullet points). Structure is signal.
Keep non-sensitive numeric values when they’re essential (prices, dimensions, dates in public policies). Over-redaction creates useless context and forces more retrieval.

Secrets are different: block, rotate, and alert

Secrets detection in LLM systems is not mainly a “masking” problem. It’s an incident prevention problem.

Once a credential gets into the pipeline:

it can land in logs,
it can get echoed by the model,
it can get cached,
it can end up in an eval dataset,
it can be pasted into a ticket by someone trying to “help.”

So for secrets:

Detect and hard-block known secret formats (API keys, JWTs, bearer tokens, private keys).
If a secret shows up in user input, respond with a safe error and trigger rotation workflows.
If a secret shows up in retrieved context (yes, this happens), quarantine the source doc and re-index.

Practical rule: if it looks like a credential, treat it like a production incident. Not like a string to redact.

Redaction enforcement points: ingestion, retrieval, prompt construction, and logging

Teams always ask: “Should we redact before embedding or at retrieval time, or both?”

My stance: both. But for different reasons. If you pick one, you’ll end up compensating somewhere else. Usually in logging. And that’s how leaks become permanent.

1) Ingestion-time redaction (before chunking and embedding)

Ingestion-time redaction is your first shot at reducing blast radius.

Do it here because:

raw docs are the richest leak source,
you can run heavier detectors (DLP, regex + ML, custom rules),
you can store two versions: a raw vault copy (restricted) and a redacted RAG copy.

In healthcare or finance, it’s common to keep raw docs in a separate evidence vault with a different retention policy. Your RAG path should almost never touch raw.

Ingestion-time output should be:

redacted chunks for indexing
a redaction manifest (what was removed, by what rule, at what time)
provenance metadata (source system, owner, tenant, classification)

2) Retrieval-time filtering (tenant, ACL, and policy checks)

Retrieval is where multi-tenant isolation either works… or you end up on a call you don’t want.

Your retriever must enforce:

Tenant isolation (hard partition or mandatory tenant filter)
Document-level ACLs (user can only retrieve what they can access)
Policy filters (e.g., “this user role cannot retrieve PHI”)

Do not rely on the LLM to “refuse” forbidden data. That’s like asking a logging library to enforce RBAC.

3) Prompt-time redaction (last-mile safety)

Prompt construction is the last place you control the content before it hits a model API.

This is where you catch:

residual PII from ingestion failures,
secrets via user input or retrieved text,
prompt injection payloads trying to force exfiltration.

If you’re already doing prompt filtering for prompt injection, extend the same policy engine to redact sensitive spans. Same machinery. Different rules.

4) Logging-time minimization (where most leaks actually happen)

LLM prompt logging retention policy is where teams self-own.

It usually starts with a reasonable intention: “we need traces to debug hallucinations.”

Then you log:

the full user query
the full retrieved chunks
the full prompt
the full model output

…and you keep it for 180 days because “compliance.”

That’s not compliance. That’s a breach backlog with a nice dashboard.

The fix is to split observability into two streams:

Compliance audit events: structured, minimal, long retention.
Debug traces: richer, short retention, access-gated, redacted.

I wrote a full schema approach in AI Agent Observability Logging Schema [2026]: OTel + Redaction. This post is the same idea, applied specifically to RAG.

Retention policies for RAG artifacts (documents, embeddings, prompts, traces)

Retention gets political fast. Security wants “keep nothing.” Debugging wants “keep everything.” Legal wants “keep whatever the policy says, and prove you did it.”

This is one of those things where the boring answer is actually the right one. Build a retention matrix. Implement it with TTLs, deletion jobs, and an audit trail that doesn’t rely on storing raw content forever.

Here’s a default retention matrix you can start with. Adjust by domain and regulation.

Default retention matrix (practical starting point)

Artifact	Contains raw text?	Sensitivity risk	Suggested default retention	Implementation pattern
Raw source docs (vault)	Yes	Very high	1–7 years (policy-driven)	Separate storage + strict access + legal hold
Redacted docs/chunks (RAG store)	Yes	High	90–365 days	Object store TTL + re-ingestion pipeline
Embeddings	No (but derived)	Medium–high	90–365 days	Vector DB TTL per record + re-embed on refresh
Vector metadata (ACL, tenant)	No	Medium	As long as embedding	Same lifecycle as embedding
User queries	Yes	High	0–30 days	Store redacted form; hash identifiers
Assembled prompts	Yes	Very high	0–7 days	Prefer not storing; store prompt hashes
Retrieved context snippets	Yes	Very high	0–7 days	Store doc IDs + chunk hashes, not text
Model outputs	Yes	High	0–30 days	Store redacted summaries; avoid raw
Debug traces (full request graph)	Sometimes	Very high	7–30 days	Separate sink + access control + sampling
Compliance audit events	No (structured)	Low–medium	1–3 years	Append-only store, tamper-evident

You’ll notice two things:

1) The stuff you want for debugging (prompts, retrieved context) has the shortest retention.

2) The long-retention trail is mostly IDs, hashes, and policy decisions.

That’s how you get both: debuggability and defensible privacy. Anything else turns into either blind debugging or an accidental data lake.

Do embeddings contain PII? How should embeddings be treated?

Embeddings are derived from sensitive text. They can leak information via membership inference or reconstruction attacks under certain conditions. And they absolutely count as “personal data” in many governance programs because they’re linked to identifiers and can be used to profile.

Practically: treat embeddings as sensitive artifacts.

Put them under retention.
Put them under deletion.
Encrypt them.
Restrict access.

If your design assumes embeddings are harmless, you will eventually build “vector analytics” that becomes a shadow data warehouse. It happens slowly, then all at once.

GDPR/CCPA deletion in a vector database: the part everyone avoids

If you support deletion requests, you need a real answer to: “How do we delete a person’s data from the vector index?”

There are only a few options that actually work:

Hard delete by document IDs: maintain a mapping from source doc → chunk IDs → embedding IDs. Cleanest.
Tombstones + async purge: mark embeddings deleted, exclude at query time, purge later.
Re-embedding: if PII was embedded into shared chunks, you may need to re-chunk and re-embed affected documents.

Operational reality: deletion is a pipeline, not a database call.

Design your indexing around stable IDs and lineage:

source document ID
chunk ID
embedding ID
tenant ID
classification

That lineage is also what makes audit trails possible.

Audit trails for RAG: what to log for compliance vs debugging

A good RAG audit trail answers two questions:

1) What happened? (who queried, what policies applied, which documents were accessed)
2) Can you prove it? (immutability, integrity, reproducibility)

A bad audit trail is “we stored every prompt for 2 years.” That’s not an audit trail. That’s a liability with extra steps.

What should an audit trail contain for RAG?

Here’s an audit event schema I’ve seen hold up well. It’s intentionally not raw-text.

timestamp
request_id (correlation ID)
tenant_id (or hashed tenant)
actor_id (hashed user ID)
actor_role (admin, support, end-user)
client_app (web, mobile, internal tool)
query_class (support, product Q&A, medical, financial)
retrieval_policy_version
redaction_policy_version
model_provider + model_id
prompt_template_version
retrieved_doc_ids (list)
retrieved_chunk_ids (list)
retrieved_chunk_hashes (optional)
decision_flags (blocked_secret, blocked_phi, allowlist_hit, denylist_hit)
output_classification (safe, contains_pii_suspected, blocked)
latency_ms (end-to-end)

Notice what’s missing: the user’s full query and the raw retrieved text.

If you need query-level detail for abuse investigations, store a redacted query or a hash plus a short-lived encrypted debug packet in a separate system.

Separate compliance logs from debug traces

This separation is not optional in regulated environments.

Compliance logs are append-only, long retention, minimal data.
Debug traces are short retention, sampled, access-gated, and aggressively redacted.

If you don’t split them, your compliance system becomes your highest-risk data store.

This is also where modern tracing helps. When you instrument your RAG pipeline like a distributed system, you can log structured spans (retrieval latency, reranker latency, generation latency) without storing raw content. When you need content occasionally, capture it on a sampling path with explicit approvals.

If you’re already working on evals, tie auditability to your regression gates. My approach in AI Engineering Evals: Regression Gates for Prompts, Tools, RAG [2026] is to treat “privacy regressions” like correctness regressions.

Encryption and access controls for vector databases and logs

Most “RAG privacy” advice stops at redaction. That’s incomplete.

Even perfectly redacted systems fail if:

anyone in the org can query the vector DB,
logs are accessible to the whole engineering org,
service-to-service permissions are broad.

Vector database security: the basics that still get missed

For your vector database:

Encrypt at rest (KMS-managed keys, ideally per environment)
Encrypt in transit (mTLS between services)
Enforce RBAC. Retrieval service accounts should not have admin permissions.
Enforce tenant isolation. Separate indexes per tenant or mandatory tenant filters enforced server-side.
Treat metadata like a boundary. A doc ID can be sensitive if it maps to a customer.

If you’re using Postgres with pgvector, lean on Postgres RBAC and row-level security. If you’re using a managed vector DB, verify that RBAC is enforced at query time, not just in dashboards.

Logs and traces: least privilege + “break glass” access

Your logging platform is often broader-access than your databases. That’s backwards.

Default debug traces to restricted access (on-call, security, a small number of engineers).
Add break-glass workflows: temporary access with approvals, fully audited.
Separate environments. Production traces shouldn’t be casually accessible from dev accounts.

In the Walmart chatbot system, we leaned heavily on event streaming for context pipeline latency. Kafka made the pipeline fast. It also creates another place to leak data if you don’t treat topics as sensitive. Partition topics by sensitivity, encrypt payloads where needed, and don’t let “observability” topics become an ungoverned dump.

Testing and monitoring: validating redaction and catching regressions

You can’t “promise” data privacy. You can only prove it continuously.

Here’s what works.

Build redaction fixtures and run them in CI

Create a small corpus of synthetic documents and user queries that contain:

multiple PII types (email, phone, address)
PHI patterns (patient ID, diagnosis codes)
PCI patterns (test card numbers)
secrets (fake API keys, JWT-like tokens)

Then test the pipeline at multiple points:

after ingestion redaction
after retrieval filtering
after prompt assembly
in the logging output

Same mindset as prompt regression testing. You’re not “hoping” you didn’t leak. You’re gating releases on it.

If you want a broader framework for these gates, Evaluate AI Agents in Production: 2026 Testing Guide is a good complement.

Monitor with metrics, not vibes

You should have metrics like:

redaction_hit_rate: % of requests where any redaction occurred
secret_block_rate: % blocked due to secrets
pii_suspected_rate: sampled detection on outputs
unredacted_span_rate: traces that contain raw content fields

Pick a baseline and alert on regressions. Even a 0.1% leak rate is catastrophic at scale.

Sampling and review: the human layer that still matters

Automated detection misses edge cases. Use sampling, but do it safely:

sample redacted traces by default
allow short-lived access to raw content only with approvals

In 2026, “privacy-preserving tracing” is the compromise that works. You debug latency and retrieval behavior with IDs and hashes. You pull raw content only on an incident path.

Incident response: how audit trails help investigate leaks

When something leaks, the worst moment to discover your logging strategy is when you’re already in incident mode and someone asks: “So… what exactly got stored?”

A strong audit trail lets you answer quickly:

Which tenant was affected?
Which user role queried it?
Which documents were retrieved?
Which policy version allowed it?
Did redaction run?
Which model version generated the response?

A practical incident workflow

Here’s a workflow I like because it respects both privacy and reality:

Contain: disable high-risk features (long-context retrieval, tool calls) for affected tenant(s).
Correlate: use request_id to trace the retrieval chain.
Verify: pull the exact retrieved document IDs and chunk IDs.
Reproduce: replay using hashes and template versions, not the raw prompt.
Remediate: fix the policy or ingestion redaction, re-index, rotate secrets if applicable.
Prove: write the incident report referencing audit events, not raw user data.

If you don’t have structured audit events, you’ll fall back to grepping logs for raw prompts. That’s slow, it’s messy, and it tends to create a second leak while investigating the first.

For broader security hardening beyond privacy, pair this with The Complete Guide to AI Security in 2026 and AI security work.

A practical end-to-end architecture (where to put the controls)

If you want a concrete system picture, here’s the enforcement map I recommend for production AI.

Controls by boundary

Client SDK / UI: detect obvious secrets before they leave the device; warn users.
API gateway: rate-limit, tenant auth, basic PII/secret scanning, request IDs.
Ingestion pipeline: heavy redaction + classification + manifests; store redacted RAG copy.
Retriever service: tenant partitioning, ACL checks, policy filters, denylist/allowlist rules.
Prompt builder: last-mile redaction; prompt injection defenses; template versioning.
LLM provider: use no-training/no-retention modes where available, but don’t pretend this is your main control.
Observability: split compliance audit events from debug traces; TTL and access controls.

This is also how you keep your production AI posture sane. When everything becomes “just one more JSON field in the trace,” you will ship a privacy regression.

The policy-as-code approach (2026 reality)

You want policies that are:

versioned
testable
deployable independently
referenced in audit events

Same discipline you apply to CI/CD.

If you’re already thinking in terms of control flow and gates, AI Agent Control Flow Patterns [2026]: Retries, HITL, Checkpoints has the same spirit. The system is the product, not the prompt.

Closing: the prediction

RAG is becoming the default “enterprise LLM” architecture. That also means RAG is becoming the default enterprise privacy failure mode.

My prediction: by late 2026, the teams that win audits won’t be the ones with the fanciest redaction model. They’ll be the ones who can point to a retention matrix, a policy version, and a tamper-evident audit event stream, and say: “Here’s exactly what we store, for how long, and why.”

If you’re building RAG today, stop treating data privacy like a bolt-on. Design it the way you design latency budgets. End-to-end. Measurable. Enforced at every boundary.

Want a challenge? Pick one artifact from the retention matrix above and actually implement TTL + deletion + audit proof for it this week. That’s the difference between a demo and a system you can defend.

Originally published on kunalganglani.com

AI Engineering Evals: Regression Gates for Prompts, Tools, RAG [2026]

Kunal — Mon, 10 Aug 2026 12:44:29 +0000

Originally published at kunalganglani.com — read it there for inline code, hero image, and live links.

AI engineering evals testing regression gates is the unsexy discipline that keeps LLM apps from randomly breaking when you change a prompt, add a tool, refresh a RAG index, or your provider silently upgrades a model. Backend teams have lived in a world of CI gates for a decade. LLM teams are still shipping vibes.

Key takeaways

Regression testing for LLM prompts works when you treat prompts as versioned artifacts and gate changes on a golden set, not on a single “looks good” manual check.
Tool calling needs contract tests: JSON Schema, versioned outputs, and recorded “golden tool traces” so CI can be deterministic.
RAG evaluation is two systems: retrieval and generation. If you don’t score retrieval, you’ll keep blaming the model for your index.
Shadow evals (dual-run/tee traffic) are the fastest way to catch “model upgrades broke prod” without betting the business on a big-bang rollout.
Failure budgets make product and engineering agree on what “good enough” means. Without them, every regression is an argument.

If your LLM app can’t fail a build, it will eventually fail your customers.

The regression gate mindset: treat prompts, tools, and RAG like APIs

Most teams talk about “LLM evals” like it’s a research problem. In practice it’s an engineering problem: preventing unintended behavior changes.

A regression gate is just a pass/fail checkpoint in CI that blocks a release when quality drops beyond an agreed threshold. The mental model that finally clicked for me is: prompts, tool outputs, and retrieval results are all de facto public APIs.

When you ship a prompt change, you changed an API.
When you add a tool or alter its schema, you changed an API.
When you re-embed documents, change chunking, swap your vector database, or adjust re-ranking, you changed an API.

The reason this is hard is stochastic outputs. But “stochastic” does not mean “untestable”. It means your gates need:

fixed datasets (golden sets)
versioned artifacts (prompt templates, tool schemas, retrieval configs)
evaluation harnesses (replay + judges)
release criteria (failure budgets)

This isn’t theoretical. I’ve built multi-stage RAG pipelines that handle millions of queries daily with sub-second responses and a measured 400% product engagement lift (Walmart conversational commerce chatbot at Firework/Zealsight). At that scale, you learn quickly that the biggest outages aren’t “the model got worse”. They’re “we changed something small and didn’t have a gate.”

If you want a deeper foundation on how I think about “shipping” LLM systems, start with AI in production and my broader take on production AI tradeoffs.

Regression testing for LLM prompts (without pretending outputs are deterministic)

Prompt regressions usually come from three sources:

instruction drift (you changed tone/format rules and broke downstream parsing)
policy drift (refusals, safety, compliance wording)
task drift (the model stops doing the core job as reliably)

How do you do regression testing for LLM prompts?

I use a boring pattern that works:

Curate a golden set of inputs.
Store a “golden” expected behavior in a format you can score.
Run a fixed evaluation harness in CI on every prompt change.
Fail the build when quality drops beyond your failure budget.

The trick is step 2. For prompts, your “expected output” is rarely a single exact string. It’s usually:

a required structure (headings, JSON fields, bullet count)
must-include / must-not-include constraints
rubric scoring (0–5) from a judge model

If you’re building AI agents, prompts are often only one piece of the loop. Still, prompt behavior is the cheapest thing to gate early.

What’s the difference between unit tests and LLM evals?

Unit tests are deterministic and local. LLM evals are probabilistic and behavioral.

A unit test asks: “Does this function return X for input Y?”
An LLM eval asks: “Does the system reliably produce outputs with properties P, under distribution D, within budget B?”

That’s why you need artifacts and slices. “It worked once in the playground” is not a test.

Practical prompt gates I’ve seen work

Format gate: 99%+ of outputs must parse into the target schema (or match a regex). For structured generation, this is where newer APIs help. More on that in the tool section.
Refusal gate: refusal rate must stay within a narrow band. For example, if your baseline refusal rate is 1.5%, you may set a budget of +0.5% absolute.
Content safety gate: a red-team slice must remain “clean” (0 critical failures). Tie this to your AI security posture.
Latency/cost gate: if prompt expansion increases average tokens by 20%, it’s a regression even if quality goes up.

This overlaps heavily with my other work on evaluation harnesses. If you’re already thinking in replay/CI terms, read Agent Evaluation Harness and AI agent evaluation.

Golden datasets: what they are, and how to curate them without fooling yourself

What is a golden dataset for LLM evaluation?

A golden dataset is a versioned set of real (or representative) inputs paired with expected behaviors and labels, used to detect regressions across prompt/model/tool/RAG changes.

The biggest mistake is making your golden set “pretty”. Pretty sets don’t catch production failures.

Here’s the curation approach that’s saved me the most pain:

Start with production logs. If you don’t have them, you don’t have a product. Instrument your system (see AI agent observability logging schema).
Stratify. Break inputs into slices: common, long-tail, high-value customers, risky intents, edge cases.
Add adversarial slices. Include attempts at prompt injection and tool abuse. This isn’t optional if you ship tool use.
Label minimally, score intelligently. Labels are expensive. Use rubric + judge where you can. Use exact labels where you must.
Version everything. Dataset v1, v2, etc. You want to compare gates across time.

Concrete numbers: for most teams, you can get meaningful signal with 200–500 examples per major capability. For a core business workflow, I like 1,000+ with slices. Below 100, you’re basically doing vibes.

For a site-level anchor: the keyword neighborhood for this topic already has traction on this blog. Based on a Google Search Console neighborhood estimate I maintain internally (via score_keyword_winnability()), this topic cluster shows ~678 related impressions and an estimated ~3,100 searches/month across 143 related queries. That’s exactly why I’m writing this as a concrete playbook instead of another “evals are important” essay.

Pass/fail thresholds when outputs are stochastic (failure budgets beat arguments)

How do you set pass/fail thresholds when outputs are stochastic?

Stop trying to pick a single perfect metric. Use failure budgets.

A failure budget is an agreed allowance for regressions on a slice, similar to an SLO error budget in SRE. It forces tradeoffs into the open.

Examples I’ve used with product teams:

“On the checkout-support slice (n=250), we allow at most 2% absolute drop in task success.”
“On the compliance slice (n=120), we allow 0 critical failures, even if overall score improves.”
“On the long-tail slice (n=400), we allow a 5% drop if latency improves by 15%.”

The important part is that product signs this. Otherwise you’re stuck in endless debates about whether a regression “matters”.

A release criteria template that product teams can actually agree on

I keep it short:

Primary metric: task success or rubric score (by slice)
Hard blockers: critical policy/security failures, schema parse failures
Budgets: max regression allowed per slice
Cost/latency: token delta and p95 latency delta
Rollback triggers: what will make us revert in production

If you’re already tracking LLM cost or trying to reduce it, tie gates to cost regression. In creative-generation systems I shipped (Firework’s AI short-video generation platform), the bill was dominated by retries and regeneration, not first-pass tokens. Gates that ignore retries miss the real cost.

Tool calling contract testing: JSON Schema, versioning, and golden tool traces

Tool calling is where most “LLM testing” advice falls apart. Because tools turn your app into a distributed system with an LLM as the orchestrator.

How do you test tool-calling / function-calling outputs?

Treat tool calls like you’d treat an external API client:

validate request shape
validate response shape
record and replay known-good traces
test backward compatibility

In practice, I like three layers:

Schema gate (hard): tool call arguments must validate against JSON Schema. If your model supports constrained outputs, turn it on.
Trace gate (deterministic): replay recorded tool sequences against a mock tool server.
Behavior gate (soft): score whether the tool was called when it should have been called.

Tool calling contract testing with JSON Schema (and why versioning matters)

A schema without versioning is a trap.

You need:

tool_name
schema_version
explicit optional vs required fields
backward-compatible evolution rules

Rules of thumb I enforce:

You can add optional fields in v1 without breaking.
You cannot change field meaning without bumping version.
You cannot remove fields without a deprecation window.

This is the same discipline you already use in microservices. The only difference is the “client” is an LLM that will happily hallucinate fields unless you constrain it.

Golden tool traces: the missing artifact

The artifact most teams don’t have is a golden trace:

prompt + system instructions
tool schemas
the exact tool calls emitted (arguments)
the tool responses (or mocks)
final user-visible output

Store this as JSON. Version it. When a provider upgrades a model and tool-call formatting shifts, your CI should catch it.

If you’re building agentic systems, also read agent orchestration patterns and agent framework tradeoffs. Tool use is where frameworks earn their keep.

RAG evaluation beyond “answer correctness”: retrieval, faithfulness, and context metrics

Retrieval-Augmented Generation (RAG) is the fastest way to ship knowledge updates, and also the fastest way to ship silent regressions.

How do you evaluate RAG systems reliably?

You evaluate two things:

Retrieval quality: did we fetch the right context?
Generation quality: did the model use the context faithfully?

In the Walmart chatbot work, the pattern was blunt: retrieval quality dominated answer quality at scale. Model swaps moved the needle less than fixing chunking, metadata, and ranking.

Metrics that actually matter for RAG gates

You don’t need 30 metrics. You need a few that map to failure modes.

Context precision: what fraction of retrieved chunks are actually relevant?
Context recall: did we retrieve the necessary chunk at all?
Answer faithfulness: are claims grounded in the provided context?
Citation coverage: if you require citations, what % of answers include them?
Latency: retrieval adds time. Track p95 end-to-end.

Concrete guidance: for a production RAG system, I like a gate where context recall on key slices stays above 90%. If recall drops to 80%, you’ll feel it in support tickets.

If you want to go deeper on the security side of RAG, see RAG failure patterns and my playbook to prevent sensitive data leakage in RAG.

Running evals in CI/CD without spending a fortune (or waiting 40 minutes)

How do you run LLM evals in CI/CD pipelines?

Make CI evals fast and cheap by design:

Small, high-signal golden set: 200–500 examples for PR gates.
Bigger nightly suite: 2,000–10,000 examples with slice reports.
Model routing: use cheaper judge models for coarse scoring, escalate only on borderline cases.
Caching: cache tool mocks, retrieval results, and judge outputs when inputs are identical.
Time limits: treat slow evals as failing infrastructure.

A pattern I like:

PR gate must finish in <10 minutes.
Nightly suite can take 1–2 hours.
Shadow evals run continuously but sampled.

If your team is already strong at CI/CD, borrow the discipline from non-LLM pipelines. I’ve seen the same cultural trick work repeatedly: once a gate can fail a build, engineers suddenly treat quality as “real.”

For the infra-minded, my posts on CI/CD and Kubernetes choices show the same principle: pick the simplest thing that keeps you shipping safely.

Shadow evals in production: dual-run, traffic tee, and slice analysis

What are shadow evals and how do you implement them?

Shadow evals are when you run a candidate model/prompt/RAG config in parallel with production, on the same inputs, without affecting the user-visible output. Then you compare.

Implementation blueprint:

Tee traffic: copy a sample of requests to the shadow pipeline.
Redact: strip PII and secrets before storing for eval. This should reuse your logging redaction rules (see OpenTelemetry instrumentation for AI agents).
Replay deterministically: freeze tool responses where possible.
Judge offline: score deltas by slice, not just globally.
Alert on drift: if a slice crosses the failure budget, block rollout.

Sampling numbers that are realistic:

Start with 0.5%–2% of traffic.
For high-risk flows, go up to 5%.
For expensive agentic flows, sample by customer tier or intent.

Shadow evals are also where you catch provider/model drift that you didn’t initiate. This is the real fix for “model upgrades broke prod.”

Rollback and kill switches: what you do when gates fail after deploy

Gates reduce risk. They don’t eliminate it.

You need a plan for when production still drifts:

Kill switch: ability to disable tool use or fall back to a simpler path.
Model pinning: pin model versions where your provider supports it.
Config rollback: prompt and retrieval configs must be deployable artifacts with version history.
Graceful degradation: return “I’m not confident” instead of hallucinating.

I’m opinionated here: if you can’t roll back a prompt or retrieval config in under 5 minutes, you’re not operating a production system. You’re doing experiments on customers.

This is also where AI security intersects. A kill switch isn’t just for quality regressions. It’s for incidents.

Drift monitoring and refreshing golden sets (without turning evals into a second product)

How do you monitor for drift and refresh golden sets?

Golden sets rot. User behavior changes, catalog/content changes, new intents appear.

A simple governance loop:

Weekly: add top new failure cases (from logs) to an “incoming” bucket.
Monthly: promote a subset to the golden set after labeling.
Quarterly: re-balance slices and retire dead cases.

Drift signals to watch:

spike in fallback/deflection rate
increase in tool error rate
increase in “no relevant context” retrieval outcomes
distribution shift in input lengths or languages

If you want to connect this to the broader agent world: drift is amplified in agentic AI systems because the model’s actions change the environment (tools, memory, retries). That’s why your eval harness must capture traces, not just final answers.

Regression gates for LLM apps in CI (7 steps)

Here’s the playbook I’d implement on any serious LLM team:

Define artifacts: prompt templates, tool schemas, retrieval configs, dataset versions.
Build a golden set: stratified + adversarial slices, versioned.
Add hard gates: schema parse rate, critical safety failures, tool contract validation.
Add soft gates: rubric score deltas, task success deltas, refusal deltas.
Set failure budgets: per slice, signed off by product.
Run CI + nightly: small PR suite, bigger nightly suite with reports.
Run shadow evals: tee traffic, slice analysis, rollback triggers.

And yes: this is more work than tweaking prompts in a notebook. It’s also how you stop waking up to “something changed and we don’t know what.”

Comparison table: gates by layer (prompts vs tools vs RAG)

Layer	Primary failure mode	Best gate artifact	Hard pass/fail signal	“Soft” quality signal
Prompts	Format/policy/task drift	Golden input set + rubrics	Parse rate ≥ 99%, 0 critical policy failures	Judge rubric score delta by slice
Tool calling	Hallucinated args, schema drift, wrong tool choice	JSON Schema + golden tool traces	100% schema-valid calls, deterministic trace replay	Correct tool selection rate
RAG	Wrong/missing context, hallucination despite context	Labeled retrieval set + retrieval logs	Context recall ≥ 90% on key slices	Faithfulness + citation coverage

What I think happens next

Newer APIs pushing structured outputs and stronger tool calling are going to make “contract testing for LLM apps” normal. Not optional. Normal.

My prediction: within 12 months, teams that still treat prompts as ad-hoc strings will look as outdated as teams shipping backend changes without CI. The winners won’t have the fanciest model. They’ll have the tightest regression gates.

If you’re building LLM features today, pick one gate to ship this week. A tiny golden set and a schema check is enough to start. Then keep going until your app can fail a build before it fails a customer.

Originally published on kunalganglani.com

AI Agent Observability Logging Schema [2026]: OTel + Redaction

Kunal — Mon, 10 Aug 2026 00:44:30 +0000

Originally published at kunalganglani.com — read it there for inline code, hero image, and live links.

AI agent observability logging schema is the difference between “we shipped an agent” and “we can debug it at 2 a.m. without leaking customer data.” I’ve watched too many teams celebrate the demo and then faceplant the first time the agent starts looping, burning tokens, and timing out in production.

Here’s the nasty part: the most tempting thing to log (raw prompts and tool payloads) is also the easiest way to spray secrets into your telemetry pipeline. If you’re not treating your logs as a data exfiltration channel, you’re doing security theater.

Key takeaways

Treat an agent run like a distributed trace. One trace per run, spans for planning/LLM/tool/retrieval steps, and stable IDs everywhere.
Default-on production logging should be metadata-only. Full prompt/tool content must be opt-in, short-lived, access-controlled, and aggressively redacted.
Tool I/O logging should be allowlist-first, with redaction transforms in the SDK and again in the collector (defense in depth).
Sampling for agents should be head-based by default, with tail sampling triggered by failures, retries, high cost, and “stuck loop” signatures.
If you align your fields to gen_ai.* and mcp.* now, you avoid migration churn as OpenTelemetry GenAI conventions harden.

If your agent is a distributed system, your observability has to be one too.

What is LLM Observability?

LLM observability is the practice of instrumenting large language model-backed systems so you can understand behavior, performance, quality, and cost in production.

For agents, it’s not “prompt + response.” That mental model is how you end up with a pile of useless logs and a compliance incident. An agent run is a sequence of decisions: planning, tool calls, retrieval, retries, policy checks, and some final outcome you can grade.

This matters more in 2026 than it did even a year ago. OpenTelemetry’s GenAI semantic conventions now live in a dedicated repository, open-telemetry/semantic-conventions-genai. They include explicit agent spans, events, and Model Context Protocol (MCP) context propagation guidance. That’s where the ecosystem is going. If your telemetry vocabulary drifts from it now, you’re signing yourself up for a migration later. Migration taxes are real. I hate them.

Security-wise, OWASP’s GenAI LLM Top 10 2026 dropped on August 4, 2026, and the project has grown to 600+ contributing experts across 18+ countries with nearly 8,000 active community members. Translation: prompt/data leakage is no longer a “maybe.” It’s a thing that happens to real companies, on real timelines, with real consequences.

Traditional vs. LLM Observability: What’s the Difference?

Traditional observability assumes requests are mostly deterministic. Same inputs, mostly same outputs. If something breaks, traces and logs usually tell a reasonably linear story.

Agents don’t.

Nondeterminism is normal. Temperature, tool timing, retrieval variance, model drift. Two “identical” runs can diverge in annoying ways.
Control flow is dynamic. Agents decide what to do next. One user request can fan out into 3 tool calls or 30.
Payloads are radioactive. Prompts contain user data, internal instructions, credentials you didn’t know were there, and compliance landmines.
Cost is a first-class metric. Token usage and tool spend can spike silently with retries, self-corrections, or loops.

So yes, you still need traces, metrics, and logs. But you need a schema that assumes the run is messy, and defaults to not leaking sensitive data.

The Five Pillars of LLM Observability

I like the “five pillars” framing because it forces you to stop obsessing over whether you captured the prompt, and start measuring what actually matters.

Reliability debugging: where runs fail, where they loop, which tool calls are flaky.
Latency: not just end-to-end. Step-level breakdowns, especially retrieval and tools.
Quality: outcomes, rubric scores, user feedback, automated checks.
Cost: tokens, retries, tool spend, per-tenant attribution.
Security & compliance: redaction, access control, retention tiers, auditability.

If you only do the first three, you’ll ship faster. If you ignore the last two, you’ll eventually have an incident. Not because you’re unlucky. Because your logging pipeline becomes your leak.

Getting Started: the schema I’d ship in a weekend

You can implement this in two layers:

Application logging contract: JSON events you emit from your agent runtime
OpenTelemetry mapping: trace/span/attribute conventions that make it portable across vendors

The point isn’t “pick the perfect vendor.” The point is you have a contract your teams follow, and you can move backends without rewriting your entire debugging story.

The schema levels

Think in four levels:

Run: the whole agent invocation (one trace)
Step: planning / reasoning / decide-next-action (spans)
Tool call: external side effects (spans)
Content blobs: raw prompt/response/tool payloads (almost never in spans)

Agent observability logging schema table

This is the minimum viable contract. Copy it into your internal docs and make teams conform to it.

Level	When emitted	Required fields	Optional fields	Redaction rule
Run (trace root)	Start + end of agent run	`run_id`, `trace_id`, `tenant_id`, `environment`, `agent.name`, `agent.version`, `start_ts`, `end_ts`, `status`	`user_id`, `session_id`, `request_id`, `deployment.sha`	No raw user text. Store only stable IDs and hashes.
Step span	Each logical step	`run_id`, `span_id`, `parent_span_id`, `step.type`, `start_ts`, `end_ts`, `status`	`retry.count`, `loop.iteration`, `policy.decision`	Metadata-only by default.
LLM span	Each model call	`gen_ai.request.model`, `gen_ai.operation.name`, `tokens.in`, `tokens.out`, `latency_ms`	`temperature`, `top_p`, `cache.hit`	No prompts in attributes. Content via opt-in event/blob.
Tool span	Each tool execution	`tool.name`, `tool.status`, `latency_ms`	`tool.error_class`, `tool.cost_usd`	Allowlist tool fields. Redact everything else.
Retrieval span (RAG)	Each retrieval query	`retrieval.source`, `retrieval.count`, `latency_ms`	`vector_db`, `reranker`, `top_k`	Don’t log document text. Log doc IDs + hashes.
Content blob	Only when debugging/approval	`blob_id`, `run_id`, `blob.type`, `ttl_hours`, `kms_key_id`	`content_sha256`, `redaction.summary`	Encrypt + TTL + strict ACL. No “forever logs.”

A few numbers to make this concrete:

ttl_hours: I default to 24 hours in prod for content blobs. Debuggability without building a compliance nightmare.
top_k: log it when you do retrieval. I’ve seen RAG regressions caused by a quiet shift from 5 to 20 retrieved chunks.
loop.iteration: if your agent can loop, you need a counter. An iteration cap of 8–12 is a sane default for many workflows.

What NOT to log (and what to log instead)

This is where most teams screw it up.

Don’t log:

System prompts verbatim
Chain-of-thought / private reasoning
Raw user content by default
Raw tool payloads by default (especially anything that can contain credentials)

Log instead:

prompt_template_id and prompt_template_version
prompt_hash (hash of the rendered prompt after deterministic normalization)
input_classification (e.g., public, internal, restricted)
tool.payload_schema_version plus allowlisted fields

The goal is to make prompts diffable without making them readable.

Set up tracing with OpenTelemetry (minimal but correct)

OpenTelemetry gives you a portable shape: trace → spans → events → attributes. The GenAI semantic conventions extend that shape with vocabulary that’s actually relevant to model calls and agent steps.

Authoritative baseline: the OpenTelemetry community specification.

Trace boundaries

One trace per agent run.
Root span name: agent.run {agent.name} (internal naming). Map to GenAI conventions inside.

Span mapping to GenAI conventions

OpenTelemetry’s GenAI span guidance includes:

Agent spans like plan and execute tool spans in the agent spec. See the plan and execute tool sections in the OpenTelemetry community.
Client inference spans for model calls with gen_ai.operation.name and gen_ai.request.model. See OpenTelemetry community.

My minimal mapping:

Root: agent.run (INTERNAL)
Plan span: plan (INTERNAL)
LLM inference span: {gen_ai.operation.name} {gen_ai.request.model} (CLIENT or INTERNAL)
Execute tool span: execute_tool {tool.name} (CLIENT if remote)
Retrieval span: retrieval {retrieval.source} (INTERNAL)

Attributes you should standardize

At minimum:

Identity: agent.name, agent.version, environment, tenant_id
Model: gen_ai.provider.name, gen_ai.request.model, gen_ai.operation.name
Tokens: tokens.in, tokens.out, tokens.total
Cost: cost.usd at span-level and run.cost.usd at root
Control flow: retry.count, loop.iteration

Cost note: if you can’t attribute spend per tenant and per user, you don’t have “cost observability.” You have vibes.

Events vs. attributes for payloads

The GenAI semantic conventions discuss strategies for capturing instructions/inputs/outputs: full buffered content, attributes, external storage, or streaming chunks.

The boring answer is the right one: metadata in attributes; content in external blobs.

Use span attributes for:

hashes
lengths
MIME types
schema versions

Use events only when:

you’re in a controlled dev environment
you have strict access controls
you need just enough content for a short time

And even then, redact first.

How do you trace tool calls in an LLM agent?

Tool calls are where your agent stops being “chat” and starts being “software.” They’re also where you leak secrets if you log like it’s 2015.

A practical approach:

Create a tool span for every tool invocation.
Put timing, status, and a stable request ID on the span.
For request/response payloads, log only allowlisted fields (think: resource_id, query_type, result_count).
Store full payloads as encrypted blobs with TTL, and only when explicitly enabled.

If you’re using MCP tools/servers, OpenTelemetry recommends context propagation by injecting traceparent, tracestate, and baggage into params._meta and extracting it on the receiver as the remote parent. See the MCP conventions doc by the OpenTelemetry community.

That one detail is what makes your tool spans line up cleanly across process boundaries.

How do I avoid logging secrets and PII in LLM prompts and tool outputs?

Treat telemetry as a data exfil path. Because it is.

Here’s the defense-in-depth model that actually holds up under reality (meaning: under deadlines, new services, and humans making mistakes).

1) Allowlist-first logging

For each tool, define:

tool.payload.allowlist.paths (JSONPaths you permit)
tool.payload.denylist.paths (high-risk fields you always drop)

Assume “log nothing” unless explicitly allowed.

2) Redaction transforms (SDK-side)

Before anything leaves the process:

redact credentials (Authorization, api_key, token, cookie)
redact common PII patterns (emails, phone numbers)
truncate long strings (cap at 256 chars for any field)
hash stable identifiers (SHA-256) when you need correlation without disclosure

3) Redaction transforms again (collector-side)

Someone will bypass your SDK. Or a new service will ship without it. So you do it again in the OpenTelemetry Collector:

drop known sensitive attributes
enforce max attribute size
enforce environment-based policies (prod stricter than dev)

4) Don’t put secrets in prompts

This sounds obvious until you see real systems.

Secrets belong in a secret manager and should never be embedded into prompts. The authoritative baseline is Google Cloud: minimize access, rotate, audit, and never leak via logs.

What are spans and traces in OpenTelemetry?

A trace is a tree representing one end-to-end operation.
A span is one timed unit of work within that trace.

For agents:

trace = one run
spans = planning step, model call, retrieval, tool execution, policy check
logs/events = details you might want to attach (carefully)

If you adopt that mental model, most schema decisions stop being mysterious.

What is tail-based sampling and when should I use it?

Head-based sampling decides at the start. Tail-based sampling decides after the trace finishes, based on what happened.

For agents, tail sampling is ridiculously effective because “interesting” traces are rare but expensive:

failures
high retry count (>= 2)
loop iterations above a threshold (>= 6)
high cost (e.g., run.cost.usd > $0.25 for consumer workflows)
policy violations

Default: head sample low (say 1–5%) for volume control.

Then: tail sample 100% of “bad” runs so you can debug what matters without setting money on fire.

How do you correlate logs with traces for debugging?

Use three IDs everywhere:

trace_id (OTel)
run_id (your business identifier)
request_id (edge ingress)

Logs should include trace_id and span_id. Your agent events should include run_id. This lets you start from a user complaint, find the run, then pivot into traces.

And yes, this is why “one trace per run” is such a powerful simplification.

Trace a RAG application (agent + retrieval)

RAG (Retrieval-Augmented Generation) is where observability gets subtle.

You need to see:

retrieval latency
retrieval count (top_k)
which index/collection you hit
reranker model (if any)
how many tokens you spent stuffing context

But you do not want raw retrieved chunks in traces.

Instead:

log doc_id and doc_hash
log chunk_id and chunk_hash
log retrieval.query_hash

Then store actual content only behind explicit gates.

If you want a deeper security playbook here, I’ve already written the adjacent piece on RAG.

View traces (what “good” looks like)

When you open a trace view for a run, you should be able to answer in under 60 seconds:

Did it succeed?
Where did time go?
Where did cost go?
Which tool failed?
Did it loop?
Was a policy gate triggered?

If you can’t, you’re collecting trivia, not observability.

For deeper control-flow patterns (retries, checkpoints, HITL), see AI agents and my guide on AI in production.

Monitor performance (latency + cost)

Monitoring agents is mostly about catching regressions before your users do:

P95 step latency changes
token usage drift
tool error rate spikes
loop frequency

I keep two dashboards:

Run-level SLOs: success rate, p95 end-to-end latency, p95 cost.
Step-level heatmap: which spans are dominating time/cost.

If you want a measurement discipline for LLM latency, I wrote up the methodology I use in LLM latency.

For cost math (including retries and tools), connect this with LLM cost and the practical breakdown in AI in production.

Default-on vs opt-in logging in production for agents

Here’s my stance: production should be metadata-only by default. If you need content, you earn it with explicit controls.

Default-on (prod):

spans with timings + statuses
token counts
cost estimates
tool names + status codes
prompt/template IDs + hashes

Opt-in (prod, time-limited):

full prompts/responses
full tool payloads
retrieved content

And when you enable opt-in, it should be:

scoped to a tenant/user/session
time-boxed (e.g., 1 hour window)
stored as encrypted blobs with TTL

How to store full prompt/response safely (without bloating traces)

The pattern that scales:

Put raw content in object storage (S3/R2/GCS).
Make it content-addressed (sha256 as key).
Encrypt with KMS.
Apply TTL lifecycle policy.
Store only blob_id plus content_sha256 plus ttl_hours in your traces.

This keeps traces lightweight and searchable, and keeps sensitive data behind a stricter access model than “anyone with read access to logs.”

Retries, self-corrections, and agent loops: how to represent them

A few rules:

A single logical model call with retries should be one inference span, with retry.count and retry events.
Agent loops should increment loop.iteration and emit a loop span per iteration.
If the agent is stuck (same tool call repeated), emit a stuck.signature_hash so you can alert on it.

Once you do this, tail sampling becomes mechanical. “Keep any trace where loop.iteration >= 6.” Easy.

How to design a redaction pipeline (SDK + collector)

This is the part I want teams to stop hand-waving.

SDK-side: redact early so secrets never leave process memory.
Collector-side: redact again because someone will ship without the SDK.
Backend-side: enforce access control and retention policies.

I’ve built the 7-agent blog publishing pipeline for this site, and one of the biggest lessons was boring: deterministic gates catch issues earlier than “we’ll review it later.” Same idea here. Put deterministic redaction and drop rules in front of your exporter, not as a best-effort afterthought.

To go deeper on agent security failure modes, connect this to my broader AI security and the specific risk of prompt injection.

Internal links you’ll likely want next

Originally published on kunalganglani.com

High-memory VPS for AI agents: when your fleet actually needs the RAM

EQVPS — Sun, 09 Aug 2026 11:42:24 +0000

The mistake I see most often with agent hosting is sizing for the wrong thing. Someone runs one agent, it uses 400 MB, and they conclude agents are cheap to host. Then they scale to a real crew and the box starts swapping at 3 a.m.

One agent is cheap. That's not the interesting case.

Where the memory actually goes
An agent that only fires off API calls to a model is light — it's mostly waiting on the network. You could run a dozen of those on a small plan and never notice.

The RAM disappears when agents start holding state. Conversation history that grows every turn. A working set several agents read and write. A vector store for long-term memory sitting in the same process. The moment your architecture stops being "call API, forget" and becomes "remember, coordinate, hand off," memory becomes the constraint, not CPU.

CrewAI, LangGraph, AutoGPT-style loops — they all trend this way as they get serious. The framework doesn't eat the RAM; the state does.

Rough sizing, honestly
I won't pretend there's a formula, because there isn't — it depends entirely on how much each agent keeps around. But a practical feel from running these:

Light, API-bound agents — you don't need Pro at all here; a NAT or dedicated-IP plan ($3–20) handles it. Pro earns its place once shared state pushes you past ~32 GB.
32 GB — the sweet spot for a real multi-agent system: 5–10 agents with shared memory plus a vector database that's actually useful. Most people land here.
64 GB — bigger fleets, longer histories, a memory index in the millions of vectors, or several services co-located. This is where one box replaces the three smaller ones you'd otherwise juggle.
80 GB — heavy, memory-bound work: large in-memory datasets, many concurrent agents, or agents plus local model inference on the same host.
Start below where you think you need to be. Watch htop for a day. Resize up when you see swap, not before — guessing high just wastes money.

The part that's hard to buy
Here's the thing that makes this awkward: renting 64 GB of RAM is easy. Renting 64 GB with crypto and no identity check is not. Most hosts that sell serious memory cheaply do it behind a card and a KYC form.

If your agent provisions its own server, or the workload touches data you'd rather not tie to a name, that combination — high memory, crypto, no KYC, and orderable by the agent itself over MCP — is the actual product. It's not cheaper per gigabyte, and I've written separately about why that comparison misleads. It's available on terms almost nobody offers.

So what do you do
If your agents are light and API-bound, don't overthink it — a small NAT or dedicated-IP plan is plenty, skip the whole high-memory question. If you're running a real fleet that holds state, size by what's actually in memory, start at 32 GB, and move up when the graph tells you to.

When you're there, the Pro line covers 32 to 80 GB with a dedicated IP and nightly backups. Pick the tier that matches your working set, not your ambitions.

40% of AI Agent Projects Will Be Canceled by 2027. Token Governance Is How You Avoid It.

Tisha — Sun, 09 Aug 2026 10:55:47 +0000

By Tisha Chawla and Susheem Koul

TL;DR. AI agents spend tokens to think, and how many they spend is nondeterministic. The same task can cost wildly different amounts across runs, and a stuck recursive loop can spend without limit while still looking successful. You cannot budget a nondeterministic resource up front. You have to observe it, attribute it, and enforce limits at runtime, from a control loop kept separate from the agent.

🛠️ We built this into an open-source tool. TokenOps is run-aware token governance for multi-agent systems: a control plane + SDK that caps spend and steers behavior across a whole agent workflow, not per request.
pip install agent-tokenops
If this field guide is useful, ⭐ star the repo so more people building agents can find it.

First principles
The problem
Primitives for token governance
The current space
The discipline
Where TokenOps fits

0. First principles

Start from what is provably true, then build up.

A token is a metered unit of a paid resource. Every model call consumes a countable number of tokens at a published price.¹²
Consumption is nondeterministic. Token generation is stochastic: at each step the model samples the next token from a probability distribution, so the same prompt can return a different completion, and a different number of output tokens, on every run. OpenAI states its APIs are nondeterministic by default, and that determinism is not guaranteed even with a fixed seed.³ Agents amplify this. An agent picks its next action from that nondeterministic output, so the number of reasoning steps and tool calls, and therefore the total token spend, is a random variable, not a fixed cost.
You can only govern what you can measure, attribute, and bound. Measure it (how many), attribute it (whose), bound it (a ceiling that actually holds).
You cannot bound it by prediction. A static estimate set in advance is wrong on the tail. The only bound that holds is enforced at runtime: watch spend as it accrues, and stop the run when it crosses a limit.
The bound must sit outside the agent. An agent chooses its own next call, so nothing in its own logic is guaranteed to stop it. Governance is an external control loop around the agent, not a feature inside it.

The three questions that decide whether you govern token spend:

How many tokens did this run spend? (measure)

Who spent them: which user, which agent, which run? (attribute)

What stops a run before the invoice does? (bound)

If you cannot answer all three, you are observing spend after the fact, not governing it.

1. The problem

One example, used throughout. Agent A is a research agent: it calls a search tool, reads the results, and hands findings to Agent B, a summarizer. Agent A runs on OpenAI, Agent B on Anthropic. We follow this single task through every failure mode below.

What a token is. A token is a chunk of text mapped to an integer ID from the model's fixed vocabulary. A subword tokenizer (for example, byte-pair encoding) splits text into these pieces, so one token can be a whole word, part of a word, or a single character.¹⁴ Different models use different tokenizers, so the same sentence is a different token count on each. Rule of thumb in English: roughly 4 characters, or 0.75 words, per token.¹ Output tokens cost several times more than input tokens.⁵²

Why tokens became an ops problem. Agents do not make one call, they loop. Anthropic measured agents using about 4x the tokens of a chat, and multi-agent systems about 15x, with token usage alone explaining roughly 80% of the variance in cost.⁶ In our example, Agent A might call search once, or twenty times. You do not know which until it runs.

1.1 The token leak

Agent A calls search("pricing"). The result is weak, so it feeds its own output back in and calls search("pricing") again, and again. This is a recursive loop: the agent keeps invoking the same step because nothing tells it to stop. Fifty calls later it still returns a clean summary, so every functional dashboard shows success. The spend shows up in exactly one place: the bill.

That is a token leak: spend that grows with no bound and no attribution, whether from a silent retry loop, a context window that keeps growing, or an unflagged model upgrade. Provider guardrails do not catch it: OpenAI project budgets are alerts, not hard caps.⁷ Unmanaged, this is now a top failure mode. Gartner predicts over 40% of agentic AI projects will be canceled by end of 2027, citing escalating cost.⁸

The quieter leak: cache busting. Repeated input can be cached at a steep discount: OpenAI documents 50% off cached input,⁵ and Anthropic charges roughly 90% below base input for cache reads.² The catch: caching matches the prompt from the top down, only up to the first point where the text differs. If Agent A pastes a current_timestamp or session_id near the top of its system prompt, it throws away the cache for everything below it, and a large context that should cost a tenth of full price pays full price every turn. Same text, same step count, 10x the bill. The fix is structural: put static content (system prompt, tool definitions) first and volatile values last, and treat a drop in cache hit rate as a spend alarm on its own.⁵

A third leak: reasoning bloat. Reasoning models (OpenAI's o-series and GPT-5 reasoning, Anthropic extended thinking) spend hidden reasoning tokens server-side before emitting a single visible token, and those tokens are billed.⁹ Two consequences. First, max_output_tokens caps the sum of reasoning and visible output,⁹ so a runaway reasoning loop can burn the entire budget inside one call without returning a character. Second, if your breaker trips by parsing the output stream, it is blind during the thinking phase: the spend has already happened before the first chunk arrives. Read the usage totals the model reports, not just the visible stream.

1.2 Distributed token lineage

The monthly bill arrives: one total from OpenAI, one from Anthropic. Neither number tells you that Agent A's recursive loop caused most of the cost, or which user started the run. Lineage is the missing label on every call: which user, which agent (A or B), and which run produced it. OpenTelemetry's GenAI conventions define exactly these attributes: gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, plus agent and session identifiers.¹⁰ Recent versions also add attributes for the costly hidden categories, cached and reasoning tokens, mirroring the provider payloads: OpenAI's prompt_tokens_details.cached_tokens and completion_tokens_details.reasoning_tokens, and Anthropic's cache_read_input_tokens.¹⁰ Track those, or your lineage misses exactly the tokens most likely to surprise you.

One more catch: Agent A's OpenAI tokens and Agent B's Anthropic tokens are not the same unit, because the tokenizers and prices differ. You cannot add raw tokens across providers. Convert each call to cost first (one denominator, such as micro-dollars), then sum and enforce. Enforce on cost, not tokens.

1.3 Isolate the guard

Put the budget check inside Agent A's loop and two things break: the same bug that makes it loop can skip the check, and Agent B needs its own copy. So keep two planes apart:

Data plane: the agent doing the work.
Control plane: the code that meters, attributes, and halts spend.

The meter and the limit live in the control plane, an out-of-band loop both agents call but neither can edit or bypass.

2. Primitives for token governance

Six building blocks. Each answers one question. Together they are the vocabulary for everything below.

Primitive	What it is	Question it answers	Source
Meter	Counts tokens and cost on every call	How much did this consume?	¹¹
Attribution context	Tags every call with user, agent, run	Whose spend is this?	¹⁰
Budget / quota	A hard ceiling on tokens or cost over a window	What is the cap, and does it hold?	¹²¹³
Rate limit	A cap on tokens or requests per unit time	How fast is too fast?	¹⁴
Circuit breaker	Trips and fails fast when a condition crosses a threshold	What stops a bad run immediately?	¹⁵
Control plane	Where the rules live, out of band from the agent	Who owns the decision?	§1.3

The circuit breaker, precisely. You wrap a protected call in an object that monitors it. When a chosen condition crosses a threshold, the breaker trips to an open state and fails fast, returning an error without making the call. Classic breakers add a half-open state to probe recovery.¹⁵ A failure can be any condition you define, such as a timeout or an HTTP 429. For token governance, the condition is behavioral: a repeated action, a spend velocity, or a collapse in cache hit rate. The breaker trips to halt the run before the spend lands.

How a behavioral check actually works. You do not need brittle, domain-specific rules. Three system-level signals over an agent's step history cover most runaway modes:

Semantic loop. Hash each tool call's name and arguments, and trip when the same signature repeats inside a sliding window. A vector-similarity check on consecutive prompts catches near-duplicates, such as an agent feeding the same error back to a tool.
Spend velocity. Track cost per step (the derivative of cost over steps). A sharp spike means the context is compounding, for example appending full raw logs on every failed turn.
Efficiency decay. Watch the ratio of progress made to tokens burned. This one is the most heuristic and hardest to define cleanly, so treat it as a warning, not a hard trip.

The first two are cheap and robust. Start there.

2.1 Beyond halt: two richer responses

Halting is the simplest response to a trip, not the only one. Two production patterns are worth knowing, each with a caveat.

Model cascade (graceful degradation). At a warning threshold (say 75% of a session budget), the control plane swaps the next call to a cheaper model and trims non-essential tools, trying a low-cost completion before any hard halt. This ships today: Cloudflare can switch to a cheaper model once a budget is exhausted,¹³ and gateways like LiteLLM support model fallbacks.¹² Caveat: a weaker model can reason worse and loop more, trading a hard failure for a quality risk and sometimes more spend. It is a tradeoff, not a free win.
Escrow and human-in-the-loop. Instead of killing the run, the control plane suspends it, snapshots its state, and pings a human to inspect and resume. Real, but not free: pause-and-resume requires a durable, checkpointable runtime. LangGraph, for example, requires a checkpointer to persist state across an interrupt.¹⁶ A lightweight in-process wrapper cannot serialize and resume an arbitrary call stack on its own.

3. The current space

3.1 Five dimensions

Score every tool on five questions. The first three are table stakes. The last two are where the gap is.

Dimension	Primitive	Question	Source
Observe	Meter	Can you see tokens per call?	¹⁰¹¹
Attribute	Attribution context	Whose spend: user, agent, run?	¹⁷
Enforce	Budget, rate limit	Can you stop spend in real time?	¹²¹³
Behavioral	Circuit breaker	Can you halt on a runaway pattern, not just a dollar line?	⁶¹⁸
Ownership	Control plane	Which layer holds the control point?	§1.3

3.2 Five categories

Every tool sits at one of five layers in the stack.

Layer	What it is	Examples	Stops spend?
Provider	Vendor-native limits	OpenAI, Anthropic, Azure OpenAI	Coarse, often alerts
Gateway	Out-of-process proxy you route through	LiteLLM, Portkey, Cloudflare, Kong	Yes, threshold-based
Component	In-process library inside your app	OpenLLMetry, LangChain callbacks	Rare
Backend	Collector or dashboard	Langfuse, Datadog, Arize, LangSmith	No, after the fact
Standard	The wire format everyone emits	OpenTelemetry GenAI conventions	n/a

3.3 The gap

Across these tools, most observe, several enforce a dollar or rate threshold (the gateway layer is moving fast: Cloudflare shipped per-user spend limits in June 2026¹³), but almost none halt a run on pathological behavior (a runaway loop) before the threshold is hit.

Enforcement also tends to live at the gateway, which sees a stream of independent requests, not the in-process context (the agent's step sequence and loop structure) needed to recognize a loop early. Behavioral signals like a collapsing cache hit rate or a semantic loop are easiest to read right next to the prompt structure and steps that caused them, which a gateway does not retain.

In-process, behavioral enforcement is the least served square in the landscape today.

4. The discipline

You already do this for CPU and memory. You measure it, label it, watch it, and cap it. Tokens are simply the newest resource to earn the same four steps. The order matters, because each step depends on the one before it: you cannot attribute what you never measured, and you cannot enforce what you cannot see.

Step	In plain terms	What it does	Mechanism
1. Instrument	Meter it	Emit one record per model and tool call, carrying token counts	OpenTelemetry GenAI spans ¹⁰
2. Attribute	Tag it	Stamp every record with user, agent, and run	OTel attributes and vendor tags ¹⁷
3. Surface	Watch it	Stream the records so a climbing cost or a falling cache hit rate is visible live, not at month end	Observability backends ¹⁷
4. Enforce	Cap it	Apply a budget, a rate limit, or a behavioral circuit breaker that trips on a recursive loop	Budgets and breakers ¹²¹⁵

Walk Agent A and Agent B through it:

Instrument. Every search call and every model call, on both OpenAI and Anthropic, emits a record with its token usage.
Attribute. Each record carries agent=A or agent=B and the run id, so the bill splits cleanly instead of arriving as one number.
Surface. Agent A's recursive search loop appears as a steeply climbing cost line the moment it starts, not four weeks later.
Enforce. When Agent A repeats the same search call past a set threshold, the breaker trips and halts the run before the spend lands.

All four steps live in the control plane (see 1.3), outside the agents, so neither Agent A nor Agent B can skip them.

The four verbs are a way to organize the work, not an official standard. Every mechanism they point to (spans, attributes, budgets, breakers) is real and documented in the footnotes.

5. Where TokenOps fits

The least-served square in section 3.3, in-process, behavioral enforcement that governs the whole run, is exactly the square we built TokenOps to fill.

TokenOps is a control plane + SDK for agent stacks. Entry agents register a run; every LLM and tool crossing shares one run_id and one ledger. Mapping it back to the primitives from section 2:

Govern the run, not the request. One run_id spans every model, tool, and agent-to-agent hop, so the Agent A → Agent B workflow stays inside a single budget, the multi-agent case a gateway can't see.
Shared ledger across processes. Spend, inflight, and halt state live in SQLite, so Agent A and Agent B can't each burn the full cap locally. This is the cross-provider, enforce-on-cost point from section 1.2.
In-path enforcement. wrap_complete runs detect → decide → apply before the next LLM call, the control loop from section 1.3, out of band from the agent.
Steer or stop. Actuators go beyond halt: HALT · MUTATE · INJECT · reject/queue, the richer responses from section 2.1.

pip install agent-tokenops

Its companion, Chronicle, covers the record-and-replay half: it captures decision boundaries so a production incident becomes a deterministic regression test.

TokenOps is early (0.x) and MIT-licensed. If this field guide gave you the vocabulary you were missing, the repo is where it becomes runnable:

👉 github.com/theagentplane/tokenops. If it's useful, ⭐ star it so more people building agents can find it. Issues and PRs welcome.

Built by Tisha Chawla and Susheem Koul.

OpenAI, "What are tokens and how to count them." https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them ↩
Anthropic, "Pricing" (base input vs output; cache-read pricing). https://platform.claude.com/docs/en/about-claude/pricing ↩
OpenAI, "Reproducible outputs with the seed parameter" (APIs are nondeterministic by default; determinism not guaranteed). https://developers.openai.com/cookbook/examples/reproducible_outputs_with_the_seed_parameter ↩
OpenAI, "tiktoken" (open-source tokenizer). https://github.com/openai/tiktoken ↩
OpenAI, "Prompt Caching in the API" (input vs output pricing; cached-input discount). https://openai.com/index/api-prompt-caching/ ↩
Anthropic, "How we built our multi-agent research system" (4x and 15x token multipliers; ~80% of cost variance). https://www.anthropic.com/engineering/multi-agent-research-system ↩
OpenAI, "Managing projects in the API platform" (project budgets are alerts, not hard caps). https://help.openai.com/en/articles/9186755-managing-projects-in-the-api-platform ↩
Gartner, "Over 40% of Agentic AI Projects Will Be Canceled by End of 2027" (June 25, 2025). https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027 ↩
OpenAI, "Reasoning models" (internal reasoning tokens are billed; max_output_tokens caps reasoning plus visible output). https://developers.openai.com/api/docs/guides/reasoning ↩
OpenTelemetry, "Semantic conventions for generative AI" (status: Development). https://opentelemetry.io/docs/specs/semconv/gen-ai/ ↩
OpenTelemetry, "Semantic conventions for generative AI metrics" (gen_ai.client.token.usage). https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-metrics/ ↩
LiteLLM, "Budgets and Rate Limits." https://docs.litellm.ai/docs/proxy/users ↩
Cloudflare, "AI Gateway spend limits." https://blog.cloudflare.com/ai-gateway-spend-limits/ ↩
Kong, "Token Rate-Limiting and Tiered Access for AI Usage." https://konghq.com/blog/engineering/token-rate-limiting-and-tiered-access-for-ai-usage ↩
Martin Fowler, "CircuitBreaker" (popularizing Michael Nygard, Release It!). https://martinfowler.com/bliki/CircuitBreaker.html ↩
LangChain, "Persistence" (LangGraph checkpointers; pause and resume require a checkpointer). https://docs.langchain.com/oss/python/langgraph/persistence ↩
Helicone, "Cost Tracking." https://docs.helicone.ai/guides/cookbooks/cost-tracking ↩
Portkey, "Budget Limits." https://docs.portkey.ai/docs/product/ai-gateway/virtual-keys/budget-limits ↩

Switchboard: building a tool router so your AI agent stops drowning in MCP tools

Shashank Shah — Sun, 09 Aug 2026 07:26:05 +0000

Keyword search picks the right tool for an AI agent 21% of the time. The router
we built picks it 88% of the time, while cutting the tokens spent describing
those tools by 99.6%. This is how we got there, including the parts we got
wrong first.
Architecture: Link

The problem nobody talks about until the bill shows up

Model Context Protocol (MCP) solved a real problem. It gave every AI agent a
standard way to talk to external tools: CRMs, observability stacks, email,
vector databases, internal APIs. It created a quieter problem in the process.

Once you connect more than a handful of MCP servers to a single agent, you are
no longer describing a tool catalog to the model. You are dumping an entire
warehouse inventory into every prompt. In our own environment, three backends
exposed 142 tools. Every one of those schemas, with its names, descriptions,
and parameter shapes, gets tokenized and sent to the LLM on every turn,
whether the user's question needs one of them or none of them.

That has three costs, and only one of them shows up on an invoice.

The first is token cost. You are paying to re-describe N tools you will
never call, every single time.

The second is selection accuracy. The more tools an LLM has to choose from in
a single context, the more often it picks the wrong one, hallucinates
parameters, or gets confused by two similarly-named tools from different
servers.

The third is operational fragility. Every new MCP server you connect makes the
prompt bigger and the agent's job harder, so scaling tool count and scaling
reliability end up pulling in opposite directions.

We built Switchboard to decouple those three curves. The name is the metaphor:
a telephone switchboard operator connects your call to the right line so you
never need to know the number. Switchboard does that for tools. The host asks
for what it wants in plain language, and the router works out which of N tool to patch it through to.

Two tools instead of two hundred

Switchboard sits between your AI host (Claude Code, a custom chat agent, an
IDE, anything that speaks MCP) and every backend MCP server you own. From the
host's point of view Switchboard is an MCP server, but instead of exposing
your full tool catalog it exposes exactly two meta-tools.

find_tools(request) takes the user's request in plain language and returns a
small, dynamically-sized set of relevant tools: zero, one, or a handful, never
a fixed top-k and never the whole catalog. invoke(tool_id, args) then calls
the selected tool against whichever backend actually owns it.

Everything else is the router's problem rather than the host's: which backend
hosts which tool, how many servers are connected, how the catalog changes over
time.

How find_tools actually decides

This is the part that took the most iteration, because search alone is not
enough. A naive nearest-neighbor lookup either returns too much, which defeats
the purpose, or misses the right tool because of vocabulary mismatch between
how a user asks and how a tool is described.

The retrieval pipeline runs in four stages.

It starts with concurrent dense and sparse search against a Pinecone vector
registry. Dense embeddings catch semantic similarity, so "send the report to
finance" finds send_email_with_attachment. Sparse keyword-style search
catches the exact-term matches dense embeddings sometimes miss: API names,
service identifiers, error codes. Both index round-trips are independent I/O,
so they run in parallel and cost one round-trip of wall-clock rather than two.

Next, a cosine-similarity gate filters out anything too far from the query's
intent before it ever reaches the LLM. It is cheap and deterministic, and it
keeps obviously irrelevant tools from wasting judge tokens.

Then an LLM judge does the expensive part properly: capability-fit selection,
de-duplication across near-identical tools from different backends, ordering
by execution sequence, and deciding to ask a clarifying question instead of
guessing when the request is genuinely ambiguous.

Finally, the result count is fully dynamic. Most routing systems force a fixed
top-k. We don't. Some queries need zero tools, some need exactly one, and a
few legitimately need several, so the judge decides rather than a hardcoded
number.

If find_tools comes back empty there is a fallback, find_more_tools, that
relaxes the gate before giving up and clarifying. It is a second chance before
the decision gets punted back to the user.

How we measured accuracy, and why the grader is strict

The headline is 85 to 90% accuracy on a held-out suite of 70 realistic
multi-tool queries, against a roughly 21% naive keyword-search baseline on the
same suite. The baseline matters more than the headline, because it is the
difference between "the pipeline works" and "an embedding lookup would have
done fine."

The grader is deliberately unforgiving. A case passes only if every required
tool appears in the result and no forbidden tool appears, where forbidden: ["*"] means the correct answer is no tools at all. For allow_any_of groups,
which are sets of interchangeable tools, exactly one member must be present.
Returning two valid alternatives is a failure rather than a hedge, because it
pushes the choice back onto the model we are trying to protect. When an order
is specified, the returned tools have to contain it as a subsequence, with
relative order preserved and unrelated tools allowed to interleave. And when a
case is marked should_clarify, the router must return zero tools and set the
clarify flag, which alone decides the case.

That last rule is the one I would defend hardest. An accuracy metric that does
not reward asking instead of guessing quietly incentivizes overconfident wrong
answers, because the model learns that any answer beats admitting ambiguity.
Ours treats correct abstention as a pass, which means the 88% includes the
system knowing what it does not know.

One honest limitation: cases requiring the same tool to be called multiple
times are structurally unsatisfiable, because route() emits each tool id
once. We count those as real misses rather than excluding them, so the
reported number is a floor rather than a flattered figure.

The decisions that mattered, and the alternatives we rejected

Most of the interesting engineering here is not in what the pipeline does. It
is in the four or five places where we deliberately chose the harder option.

Dynamic-K over fixed top-k

The obvious design is "return the top 5 matches." We rejected it because a
fixed k is wrong in both directions at once. For a query that needs one tool,
top-5 injects four irrelevant schemas and reintroduces exactly the
selection-confusion problem the router exists to solve. For a genuine
multi-step request, top-5 might truncate a plan that needed six. And for an
out-of-scope question, top-5 confidently returns five wrong tools.

Making k dynamic means the judge has to answer "how many?" as well as
"which?", which is a harder prompt and a harder thing to evaluate. It was
still the right trade. Our measured average is 1.2 tools per call out of 142,
which no fixed k would have produced.

One hard-coded rule, and only one

There is a real temptation to encode catalog-specific heuristics, something
like "queries mentioning 'log' should prefer the observability server." We
kept exactly one code-side rule: an absolute cosine floor for out-of-scope
detection.

The reasoning is a division of labor. That floor is the one judgment the LLM
cannot make cheaply. To know that nothing in the catalog fits, a judge would
have to see the entire catalog, which is precisely the cost we are
eliminating. A cosine threshold answers it in one vector op. Everything else
(which tools, dedup, ambiguity, ordering) is semantic work and gets delegated
to the judge. Rules that encode catalog specifics would need rewriting every
time someone connects a new backend, which defeats the pluggability goal.

Judge order, not score order

Early on we sorted the returned tools by cosine score. That is wrong for
multi-step requests. "Pull last week's errors and open a ticket for the worst
one" has an inherent execution order that has nothing to do with which tool
embeds closer to the query. The judge reasons about sequence, so the judge's
output order is the router's output order, and we explicitly do not re-sort.

Filter unhealthy backends before the judge, not after

The intuitive design for backend health is retry-on-failure: select a tool,
call it, and handle the error if the backend is down. We invert it. A backend
marked down has its tools excluded from the candidate set before the judge
sees them, re-evaluated on every call.

This matters because of the specific failure it prevents. The judge picks the
perfect tool, explains its reasoning, and then invoke fails, so the user
gets a wrong-looking answer for a right-looking decision. Filtering early
means the judge selects the best reachable tool instead, possibly a
second-choice tool on a healthy backend, which is the correct behavior.

We also chose reactive health detection over a heartbeat loop. A backend gets
marked down on an actual failed call and retried on next use. A separate
polling process is one more thing to keep in sync with reality, and health
checks that themselves flake produce false "down" states.

Deprecated tools stay fetchable

Our first filter dropped every tool marked deprecated. We removed that clause
deliberately, because a request like "export it in the legacy format"
specifically needs the deprecated version. Governance filtering still applies,
with PII-touching tools gated behind an explicit flag and destructive tools
behind another, but deprecation is metadata for the judge to weigh rather than
a hard exclusion for code to enforce.

Keeping the registry honest

A tool router is only as good as its index, and tool catalogs are not static.
Backends add tools, deprecate others, and change descriptions. Switchboard
runs a background ingestion pipeline, independent of request-time traffic,
which polls every registered backend on an interval and also reacts
immediately to admin-triggered registration or removal. Same pipeline, two
triggers.

Each tool runs through enrichment: deterministic structural and length checks,
LLM-based description-quality and off-topic detection, TF-IDF duplicate and
outlier detection, and incremental LLM clustering against a persisted
taxonomy.

There is a circuit breaker on the LLM enrichment step. If fewer than half the
tools survive validation, the pass is treated as an outage and the registry is
left untouched. This was a deliberate answer to a real failure mode, since a
flaky upstream model silently emptying your tool catalog is worse than having
no enrichment at all.

Before touching Pinecone the pipeline diffs against a Redis hash cache, so a
full re-poll of a backend only costs an embedding call for what actually
changed rather than the whole catalog every time. Each tool is then embedded
as three views (name, description, parameters) into dense and sparse Pinecone
indexes.

That three-view embedding deserves a note. A single embedding of a
concatenated tool blob dilutes each signal, and a distinctive parameter name
gets averaged into prose. Embedding name, description, and parameters
separately, then collapsing to the best-scoring view per tool at query time,
means a query that matches strongly on one dimension still surfaces the tool
instead of being averaged into mediocrity.

Plugging in a new backend without touching a running system

This was a hard requirement from day one. Adding or removing a backend MCP
server should never mean a restart, a redeploy, or a config-file hand-edit
under load.

curl -X POST http://<host>:<port>/admin/backends \
  -H "X-Admin-Token: $ROUTER_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"server_id":"firecrawl","transport":"http",
       "url":"https://my-mcp-server.example.com/mcp",
       "auth_header":"Authorization","auth_value":"Bearer <token>"}'

One authenticated call hot-connects the backend. Every tool it exposes is
validated and embedded through the same enrichment pipeline the interval
process uses, backends.json is updated, and the moment the call returns
those tools are live and discoverable through find_tools. No restart, no
downtime for existing traffic. Removal is the mirror image: one DELETE call
clears Pinecone, Redis, and the config in one step, so you never leave
orphaned vectors behind.

Registration is serialized behind a lock so two concurrent calls cannot race
on shared state. It is an unglamorous detail that matters the first time two
people onboard backends simultaneously.

On the host side, plugging in is equally uneventful. Switchboard speaks
standard MCP over HTTP, so it is just another entry in mcp.json. We wired it
into Claude Code with zero special prompting, no "please use the router"
instruction needed. A plain question like "show me the error logs for
checkout-api in the last hour" triggers find_tools automatically, gets back
exactly the one tool that matters, and resolves. That was the actual bar we
were aiming for. The router should be invisible when it is working.

What this actually saves

We instrumented every routing decision end to end. Every find_tools call
writes a telemetry row recording the routed-versus-full-catalog token
estimate, search latency, judge latency, and outcome. Those writes are
fire-and-forget, so they never block the response even if the DB write fails.
A Streamlit dashboard reads that table live, and nothing on it is hardcoded.

Measured against the 142-tool catalog, routing cuts input tokens per call by
about 99.6% versus sending the full catalog every time, which works out to
roughly 28,700 tokens saved per call (about 124 tokens routed against about
28,860 for the full catalog dump). The average call selects 1.2 tools out of
142 available, which is the dynamic-k judge working as intended rather than a
fixed top-k masquerading as precision.

End-to-end latency averages about 6.5 seconds, with roughly 4.9 seconds of
that sitting in the LLM judge step. That is the honest cost of doing selection
properly instead of guessing. The clarify rate is about 12%, meaning genuinely
ambiguous requests get kicked back to the user instead of the router guessing
and invoking the wrong tool.

Projected from those per-call savings, at 1,000 calls a day (one active team's
traffic) and Sonnet-class input pricing, that comes to roughly 10.5 billion
tokens and about $31,000 a year in avoided token spend. The per-call saving is
measured. The annual figure is that measurement multiplied by an assumed call
volume, and it scales linearly with whatever volume your deployment actually
sees.

Why a tool router matters more than it looks like it should

It is tempting to file this under prompt optimization. It isn't. As agentic
systems connect to more of an organization's real infrastructure, three things
become true at once.

Tool catalogs only grow, because nobody removes MCP servers once they are
useful, they just add more. Context windows are not free even when they are
technically large enough, since every token spent on tool schemas is a token
not spent on reasoning, retrieved context, or conversation history. And
selection errors compound, because a wrong tool call against a real backend
(sending an email, querying production data, triggering a workflow) is not a
cosmetic bug. It is an action with consequences.

A router answers all three at once. It decouples how many tools exist from how
many tools the model has to reason about, and it puts a validating,
explainable decision layer between "the user asked for something" and "a real
backend got called."

Handling the edges

A few decisions exist specifically because the happy path is not where systems
like this actually fail.

If the judge is unavailable or its call fails, routing falls back to dense
top-5 rather than returning nothing, because a degraded answer beats an
outage. The admin API 404s entirely unless an admin token is configured, so
there is no insecure-by-default state to accidentally ship. When find_tools
cannot confidently resolve intent the system asks rather than picking the
closest-sounding tool and hoping. Instrumentation writes are fire-and-forget,
so a Postgres hiccup degrades your dashboard and never your response latency.
And ingestion is idempotent and incremental, so backends can be re-polled
constantly without re-embedding a catalog that has not changed.

What's next

The current bottleneck is not accuracy. It is the 4.9-second judge step, plus
the fact that the system learns nothing from its own traffic. Both are
addressable.

Draining admin-triggered ingestion asynchronously. Registering a new
backend currently blocks the API call until every tool is validated and
embedded. That is honest but not ideal at scale. The target is to enqueue the
job and let the background worker drain it so registration returns
immediately. The work is already queue-shaped; the admin path just drains its
own jobs synchronously today.

Cutting judge latency with a cascade. Roughly 75% of end-to-end latency is
one LLM call, and most queries are not close calls. When the top candidate
leads by a wide cosine margin and no near-twins exist, a small fast model or
even a deterministic path could resolve it, escalating to the full judge only
for genuinely contested cases. The measurement to run first is what fraction
of traffic is actually contested. If it turns out to be 20%, a cascade cuts
average latency substantially at close to zero accuracy cost.

Query rewriting before retrieval. Terse or jargon-heavy requests like
"502s on checkout" embed poorly against prose tool descriptions. Expanding the
query before it hits Pinecone closes the vocabulary gap between how users ask
and how tools are documented. The trade-off is another model call in the hot
path, which argues for doing it only when the first retrieval scores weakly.
Rewrite as a fallback, not a default.

Hard negative mining from real clarify and miss cases. Every clarify is a
labeled example of "these tools looked similar but the query was ambiguous,"
and every user retry after a bad selection is a labeled negative. Mining those
into a fine-tune or a re-ranker trains on our own traffic distribution instead
of generic semantic similarity. This is the highest-leverage accuracy lever we
have, because it compounds. More traffic means better signal.

Feedback-loop learning from invoke outcomes. We currently log which tools
were selected, but not whether the invocation succeeded or whether the user
immediately tried something else. Closing that loop, from selection through
invocation to outcome, turns the telemetry table from a cost dashboard into a
training set. It is mostly a schema change and a follow-up write.

Per-tenant re-ranking. Usage patterns differ sharply by deployment. A team
that lives in observability tooling should see different ranking than one that
lives in CRM tools. A lightweight per-tenant prior over the global model
captures most of that gain without maintaining separate indexes.

Tightening the enrichment quality bar. Bad retrieval is often a symptom of
bad tool descriptions rather than a bad retriever. Flagging ambiguous or
poorly-described tools at ingestion time, and reporting them back to the
backend owner, fixes the problem at its source instead of compensating for it
at query time.

Cross-system observability. The dashboard reports router cost, latency,
and outcome. Unifying it with orchestrator-side telemetry, whether
OTel/SigNoz/Langfuse or a shared events table, would let cost and accuracy be
viewed end to end rather than only at the router boundary.

The takeaway

The interesting problem in agentic AI right now is not whether the model can
call a tool. It is whether the model can call the right tool, reliably, as the
number of tools you connect grows past what any single prompt should ever have
to describe. That is an infrastructure problem rather than a prompting
problem, and infrastructure problems deserve infrastructure solutions: a
registry, a retrieval pipeline, a validation layer, and telemetry that tells
you the truth about whether it is working.

That is Switchboard. One operator, any number of lines, and a host that never
has to know the number.

LLM Latency Benchmark Methodology: Streaming UX Metrics [2026]

Kunal — Sun, 09 Aug 2026 00:42:41 +0000

Originally published at kunalganglani.com — read it there for inline code, hero image, and live links.

I don’t trust your “250ms TTFT” claim unless you can show me what the user actually saw.

I’ve watched teams ship “fast” streaming chat UIs that still feel sluggish because tokens arrive in weird clumps, markdown doesn’t render until the end, and tool calls create dead-air pauses long enough for users to think the app froze. That’s the gap this llm latency benchmark methodology is trying to close. Not model latency in a vacuum. Perceived latency in real-time chat and agent products.

Key takeaways

A streaming LLM benchmark should report time-to-first-token (TTFT) and token cadence/jitter, because users feel “bursty” streams as lag.
For chat UIs, the only honest metric is time-to-first-meaningful-render (the first usable chunk rendered in your UI), not the provider’s first byte.
For agents, track tool-call stall time as a first-class metric. Tool execution pauses dominate perceived speed in multi-step flows.
Always report p50/p95/p99 for TTFT, cadence, and end-to-end time-to-usable, otherwise you’re hiding tail pain.
A reproducible harness needs controlled prompts, controlled output length, warmup rules, fixed concurrency, and client-side instrumentation.

If your benchmark can’t explain why the UI felt slow, it’s not a latency benchmark. It’s a provider marketing number.

What is LLM latency benchmark methodology?

LLM latency benchmark methodology is a repeatable way to measure and report how long a large language model takes to produce useful output under controlled conditions.

In 2026, “useful output” usually means streaming output (SSE/WebSockets) inside a chat UI or an agent loop. So a modern methodology can’t stop at TTFT. It has to quantify:

Streaming LLM latency metrics like inter-token delay distributions and chunk cadence.
Client-perceived milestones like first rendered text, first rendered markdown, and first actionable instruction.
Agent-specific latency like tool-call stalls, retries, and multi-turn loops.

I’m framing this as methodology, not a one-off benchmark chart. Charts go stale. A harness keeps paying rent.

Why TTFT-only benchmarks are misleading for real-time apps

TTFT is the first number everyone asks for because it’s easy to explain. It’s also the easiest number to game.

Three common failure modes I keep seeing:

The first token is basically junk. Your model can emit “Sure,” in 120ms, then take 2.5 seconds to produce anything that changes what the user does.
Streaming is bursty. Systems deliver tokens in clumps because of server flush cadence, proxy buffering, or client rendering. A stream that updates every 400ms feels slower than one that updates every 80ms, even if TTFT is identical.
Agent loops hide stalls. Tool calls create “thinking…” gaps. Users don’t care that TTFT was 200ms if the app goes silent for 4 seconds before it does anything.

Here’s a brutal example: if your UI renders markdown only after the full message, your perceived “first output” is effectively end-of-message time. I’ve seen this exact bug show up in agentic chat demos. The model was streaming. The UI was not.

This is why I split metrics into two buckets:

Provider-side latency: what the model server did.
User-perceived latency: what your product actually showed.

When those disagree, the user-perceived numbers win. Every time.

The streaming UX metric suite (beyond TTFT)

If you’re building a streaming-first product, you need a metric suite that matches what users feel, not what vendors want to print on a slide.

Here’s what I recommend for a streaming-first inter-token latency benchmark and chat/agent UX.

1) Time-to-first-token (TTFT)

TTFT is still useful. It’s just not the whole story.

Definition: request start → first token received by the client.
Report: p50/p95/p99 in milliseconds.

If you’re aiming for a “snappy” chat UI, a common target is sub-500ms p95 TTFT. For voice or live-assist experiences, you often want sub-250ms p95.

2) Time-to-first-render (TTFR)

This is the first moment the user sees anything on screen.

Definition: request start → first UI paint of streamed content (not “spinner shown”).
Instrumentation: your UI code, not the server.

A practical rule: TTFR should be within +50–150ms of TTFT on a healthy web UI. If it’s +500ms, your rendering pipeline is the bottleneck. Congratulations, you’re “optimizing the model” while React is the thing on fire.

3) Time-to-first-meaningful-render (TTFMR)

This is where perceived speed actually lives.

Definition: request start → first rendered chunk that changes user understanding or next action.

Examples of “meaningful”:

First real bullet in a plan (not “Here’s a plan”).
First tool result summary.
First SQL query draft.

In a lot of real apps, TTFMR is 2–10× larger than TTFT because early tokens are pleasantries, hedges, or formatting.

4) Inter-token latency distribution

“Average tokens/sec” is the metric equivalent of averaging the weather for an entire year. Technically true. Practically useless.

Definition: time between consecutive tokens (or chunks) as observed by the client.
Report: p50/p95/p99 inter-token delay, plus a histogram if you can.

A stream that’s 30 tok/s on average but has 800ms gaps will feel worse than 15 tok/s with steady 60–90ms updates.

5) Chunk cadence + jitter

Most streaming APIs deliver chunks, not individual tokens. Users see chunks. So measure chunks.

Definition: time between UI updates that add new text.
Cadence: average update interval.
Jitter: variance of the interval.

This is the “streaming feels alive” metric. A good UX typically updates at 5–15 Hz (every 70–200ms). Below 2 Hz (500ms+), people start second-guessing whether the app froze.

6) Time-to-usable (TTU)

This is the metric I want agent teams to rally around because it forces you to define “useful.”

Definition: request start → first moment the user can take the next action with confidence.

For a chat app, TTU might be “first complete answer sentence.” For an agent, it might be “first step + the tool the agent is about to run + a confirmation prompt.”

TTU is product-specific. Good. You’re building a product, not a latency demo.

7) Tool-call stall time (agents)

Agentic UIs made stalls unavoidable. The only question is whether you measure them or pretend they don’t exist.

Definition: time from “tool call requested” → “tool result available to model” → “stream resumed to user.”

Break it down:

Tool API round-trip time.
Queueing in your tool infrastructure.
Model “resume” latency after tool output.

A lot of teams eventually discover that the “LLM latency problem” was actually “our tool services are slow at p99.” Which is… not surprising.

8) End-to-end completion time (E2E)

Still matters for throughput planning and cost.

Definition: request start → final token rendered.
Report: p50/p95/p99, and include output length.

Without output length, completion time is meaningless. You’re comparing apples to an entire fruit salad.

How to measure inter-token latency accurately on the client

If you want honest streaming metrics, measure at the point of truth: the client.

Here’s the minimum I instrument in production chat/agent UIs:

Request start timestamp (when the user hits enter, or when your agent step begins).
First byte received (stream opened).
First token/chunk received.
First UI render (first paint of model output).
Every chunk render time (when you append content to the DOM).

For web apps, store timestamps in a trace context and emit them as structured events. I like doing this alongside OpenTelemetry tracing. If you’re already instrumenting agent workflows, you’ll get even more value from OpenTelemetry.

Two gotchas that keep biting people:

Measure render events, not network events. The user doesn’t care when the browser received bytes if you don’t paint them.
Token vs chunk. Most APIs deliver chunks. Don’t invent token timestamps unless you’re actually receiving token-level events.

Concrete example: if your stream updates every 200ms but your UI batches DOM updates every 500ms (because of a throttled render loop), your “inter-token latency” is a lie. You’re measuring the wrong boundary.

Chunk sizes, SSE/WebSocket flush cadence, and why your stream is bursty

Streaming performance is often dominated by boring plumbing. Not the model.

A stream becomes bursty because:

The server buffers output and flushes every N tokens.
Reverse proxies buffer response bodies.
Compression (especially gzip) delays flushes because it wants bigger blocks.
Your client parses events in batches.

If you’re using Server-Sent Events, treat “flush cadence” as a configurable part of your serving stack. With WebSockets, you still have to decide message frequency.

A practical measurement approach

In your harness, record:

Chunk size in characters (or bytes).
Time between chunks at the client.
Time between chunks at the UI render.

Then compute:

Chunk cadence p50/p95/p99.
Jitter (standard deviation or p95–p50 gap).

If you see a pattern like “chunks every 1.0s with 200–500 chars,” that’s not a model problem. That’s buffering.

A concrete target

For a typical chat UI, I aim for:

p50 chunk cadence < 150ms
p95 chunk cadence < 300ms

If you can’t hit this, at least make the UX honest. Show progress. Show tool steps. Show partial structure. Don’t leave users staring at a cursor like it’s 1999.

How to report p50/p95/p99 for TTFT, cadence, and time-to-usable

Percentiles are not optional. Streaming and tail latency go together.

If you only report p50, you’re measuring your best day.

If you report p50 and p95, you’re starting to see operational risk.

If you report p99, you’re finally talking about what on-call will feel like.

Recommended report format

For each scenario (prompt + output length + concurrency), report:

TTFT p50/p95/p99 (ms)
TTFR p50/p95/p99 (ms)
Chunk cadence p50/p95/p99 (ms)
TTU p50/p95/p99 (ms)
E2E completion p50/p95/p99 (ms)

Also include:

Output length (tokens or chars) and max_tokens.
Concurrency level.
Warm vs cold.
Region (client region and model region) if relevant.

A compact mapping table (metric → how to measure → why it matters)

Metric	How to measure	Why it matters to streaming UX
TTFT	Client timestamp: request start → first token/chunk	Controls the “is it alive?” feel
TTFR	UI timestamp: request start → first paint of output	Captures rendering delays TTFT misses
Inter-chunk latency	Client timestamps between chunk arrivals	Quantifies burstiness users feel
Chunk cadence jitter	Variance/p95–p50 of inter-chunk latency	High jitter feels like freezing
Time-to-usable (TTU)	Product milestone: start → first actionable/meaningful content	Best proxy for perceived speed
Tool-call stall time	Tool start → tool result → stream resume	Dominant in agent workflows
E2E completion	Start → final rendered token	Needed for throughput/cost planning

Benchmarking agent/tool-call workflows (where the stream stalls)

Agent benchmarks that ignore tool calls are measuring the wrong product.

In a tool-calling loop, generation often looks like:

Model streams a plan.
Model emits a tool call.
Streaming stops.
Tool runs.
Model resumes streaming with results.

Users experience step 3 as “the app froze.” You can call it “tool execution.” They’ll call it “this thing is broken.”

What to measure

For each tool call, capture:

Tool request latency (client → your backend → tool).
Tool execution time (inside the tool).
Tool response latency (tool → backend → model).
Resume latency (tool result attached → model streams again).

Then roll up:

Total tool-call stall time per run.
Stall time as a percentage of TTU and E2E.

Concrete example: If TTU is 6.0s and tool stalls account for 4.2s, optimizing the model is missing the point.

UX instrumentation that makes stall time visible

If your UI can’t stream during tool execution, at least:

Show which tool is running.
Show the input (sanitized) and what it’s waiting on.
Show a time counter after 500ms.

This is as much product as performance engineering.

If you’re building serious AI agents, you’ll also want a handle on tool governance patterns. I’ve written about agent control flow patterns and tool approval patterns for that reason.

Separating network latency, provider queueing, and model decode time

If your benchmark can’t decompose latency, you can’t improve it.

I like splitting end-to-end time into segments:

Client → edge (DNS, TCP, TLS): network setup.
Edge → provider: upstream network.
Provider queue time: waiting for capacity.
Prefill time: prompt processing.
Decode time: token generation.
Server flush/buffering: chunk emission schedule.
Client parse + render: turning bytes into UI.
Tool time (agents): external calls.

What you can actually observe

You usually can’t directly observe “provider queue time” unless the provider exposes it. But you can still triangulate.

For example:

If TTFT balloons under concurrency while token cadence stays stable, you’re probably queueing.
If cadence slows down but TTFT stays stable, decode throughput is the bottleneck.

When I built a multi-stage RAG chatbot for Walmart product pages, the lesson that kept repeating at scale was that the slow part was often not the model. Kafka event-streaming the context pipeline mattered more for latency than model-side tricks, because it removed cross-service waiting in the critical path. Same mental model here. Break the path into segments or you’ll optimize the wrong thing.

If you want a deeper take on production tracing, see my write-up on production AI and AI in production.

A reproducible benchmarking harness design (chat + agents)

Reproducibility is what separates a benchmark from a vibe.

Here’s a harness design that’s actually worth trusting.

1) Fixed scenarios, not random prompts

Create a small scenario set:

5–10 prompts for chat.
3–5 agent tasks that include tool calls.

Make them stable. Version them. Treat them like a test suite.

If you already maintain eval datasets, reuse that discipline. My agent posts like AI Agent Evaluation Framework 2026: 8 Metrics Beyond Task Success and Agent Evaluation Harness focus on correctness, but the harness mechanics apply here too.

2) Control output length

Uncontrolled output length will destroy comparability.

Options:

Use explicit max_tokens.
Force a structured output with a fixed number of bullets.
For agents, stop after N steps.

Report the target length and the actual length.

3) Define warmup vs cold start

You need both.

Cold: first request after idle (captures model spin-up, connection setup, cache misses).
Warm: steady-state.

Run at least 10 warmup requests before collecting “warm” stats. Don’t mix them.

4) Concurrency settings

Real-time apps often run at concurrency > 1 even per user (prefetching, tool calls, parallel retrieval).

Benchmark at:

Concurrency 1 (baseline)
Concurrency 5
Concurrency 20 (or whatever matches your traffic class)

If p95 TTFT jumps from 400ms at concurrency 1 to 2,000ms at concurrency 20, you just found your scaling wall.

5) Network conditions

If you only test from one data center on a perfect link, you’re benchmarking your luck.

At minimum, run from:

The same region as your users.
A second region.

If you have a mobile app, include a high-latency profile (e.g., 100–200ms RTT).

6) Client + server tracing

Correlate:

Client events (TTFT, TTFR, cadence)
Server events (request received, upstream call start/end, flush events)
Tool events (start/end)

This is where OpenTelemetry shines. Instrument once, then reuse across performance and reliability work.

7) Store raw traces

Don’t just store aggregates.

Raw traces let you answer questions like:

“Why did p99 cadence spike at 3pm?”
“Which tool calls are causing stalls?”

UI instrumentation: measuring perceived latency inside your product

If you take one thing from this post, take this: instrument your UI like it’s a performance-critical system. Because it is.

What I track in production chat/agent UIs:

First contentful paint after send.
First token painted.
First markdown block rendered (if you support markdown).
First actionable UI element enabled (copy button, approve tool, “run” button).
Per-chunk render time.

Then I compute a product metric leadership can understand without a lecture:

“p95 time-to-usable for chat answers is 1.8s.”
“p95 time-to-usable for agent tasks is 7.2s, and 58% of that is tool stalls.”

These numbers start real conversations. TTFT alone doesn’t.

Avoiding misleading benchmarks (the stuff that ruins your data)

Here are the traps that make benchmarks quietly wrong.

Caching

If your provider caches responses for identical prompts, you might be measuring cache hits.

Mitigation:

Add a nonce in the prompt that doesn’t change the task.
Or explicitly disable caching if supported.

Rate limits and retries

Rate limit backoffs create long-tail spikes.

Mitigation:

Log retry counts.
Separate “successful no-retry” distributions from “all requests.”

Batching

Server-side batching can change inter-token cadence and TTFT.

Mitigation:

Record concurrency and batching settings.
Don’t compare “batching on” to “batching off” without saying so.

Speculative decoding

Speculative decoding can improve perceived speed but may change cadence patterns.

Mitigation:

Treat it as a configuration dimension, not a constant.

Client-side throttling

Your UI might be throttling renders to reduce jank.

Mitigation:

Measure both “chunk received” and “chunk rendered.”

Output-length drift

If one run generates 200 tokens and another generates 900, your completion time comparison is meaningless.

Mitigation:

Control max_tokens and enforce structured outputs.

A data anchor from my own benchmarks (and why it matters here)

Based on the benchmark methodology and results I maintain at kunalganglani.com/llm-benchmarks, a recurring pattern in local inference is that TTFT and steady-state throughput move differently depending on hardware and runtime.

For example, on Apple Silicon you can often load larger models than your “VRAM limit intuition” suggests because of unified memory, but throughput becomes the real trade. That same TTFT vs throughput vs perceived UX trade shows up in cloud streaming too: you can win TTFT and still lose the user experience if cadence is bursty or tool stalls dominate.

If you want a local-focused methodology baseline, see Local LLM Benchmark Methodology. The point of this post is extending that thinking to streaming UX.

Putting it all together: a step-by-step benchmark plan

A tutorial needs an executable plan. Here’s mine.

Define your UX milestones: TTFR, TTFMR, TTU (write down what “usable” means in your product).
Pick 5–10 fixed prompts and 3–5 agent tasks with at least 1 tool call each.
Implement client instrumentation: timestamps for receive + render, plus chunk sizes.
Implement server tracing: upstream call timing, flush timing, tool timing.
Run warm and cold suites: at least 10 warmup runs, then 30+ measured runs per scenario.
Run at 3 concurrency levels: 1, 5, 20 (adjust to your expected load).
Report p50/p95/p99 for TTFT, cadence, TTU, and E2E. Include output length.
Visualize stalls: a timeline per run showing model stream, tool stall, stream resume.

Do this and your benchmark will tell you what to fix. Do less than this and you’ll spend a quarter shaving 80ms off TTFT while users complain the app “hangs.”

The uncomfortable prediction

Streaming is about to get judged like video performance.

Users are developing an instinct for “this app is laggy” that has nothing to do with your backend dashboards. They’ll feel it in the cadence. The stalls. The weird clumps. The UI that waits to render markdown until the very end.

My bet: within a year, serious teams will treat time-to-usable the way web teams treat Core Web Vitals. And the teams still bragging about TTFT will get outsold by the teams that make streaming feel alive.

If you’re building real-time chat or agents, stop measuring what’s easy. Measure what the user feels. Then optimize that.

Originally published on kunalganglani.com

Our Token Counter Took 26 Seconds on a Single Prompt

Talha Anwar — Fri, 07 Aug 2026 23:37:36 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

Project Overview

I work on AcruxCore, an LLM ops platform with an Express API gateway. Every completion passes a budget pre-check: estimate the token cost, compare against the team's remaining spend, reject if it doesn't fit.

For OpenAI-family models, that estimate used js-tiktoken:

if (isOpenAiFamily) {
  try {
    return encoder().encode(text).length;
  } catch {
    // Fall through to the heuristic if the encoder rejects the input.
  }
}
return Math.ceil(text.length / 4);

Three lines. They do the right thing. They just don't always do it quickly.

Bug Fix or Performance Improvement

Performance bug — a quadratic BPE tokenizer on the synchronous request path.

I didn't find this through a user complaint. Two datasets tests started timing out at Jest's 5-second limit. I assumed I'd broken something, stashed my changes, checked out clean staging, and ran them again. Still failed.

The tests were fine. The code under them was just brutally slow.

The BPE encoder is quadratic on whitespace-free input. BPE works by repeatedly merging the most common adjacent pair of characters. Each pass scans the entire string to find the best pair. One merge per pass, full scan each time. A string of N characters with no whitespace means roughly N passes over N characters — N² work.

Normal text never hits this because the pre-tokenizer splits on whitespace first. Every piece is one short word (~5 chars), so N² is nothing. A long unbroken string has no whitespace to split on, so the whole thing becomes one piece and N becomes the full length.

Measured on cl100k_base (GPT-4's tokenizer):

single piece	encode time
256 chars	4 ms
512 chars	18 ms
1,024 chars	70 ms
2,048 chars	281 ms
9,000 chars	5.4 s
20,000 chars	26 s

Doubling the length roughly quadruples the time. Meanwhile, 20,000 characters of ordinary prose encodes in 3 ms. It's not the length — it's the lack of whitespace.

This function runs on every single request. Node runs JavaScript on one thread. A 26-second synchronous encode doesn't just slow down one request — it freezes every other user on that process.

And the input that triggers it isn't exotic: a base64 blob in a prompt, a minified JSON payload, a corrupted copy-paste. Any authenticated caller could stall the whole process.

gateway completion, 2-char rendered variable:      82 ms
gateway completion, 9000-char rendered variable:  5506 ms

Same code path. Same mocked provider. One template variable different.

Why not just remove the tokenizer?

My first thought. But estimateTokens has two jobs:

Budget pre-check — reject requests that would exceed the spend cap. Overestimate and you reject valid requests. Underestimate and teams blow past their budget.
Billing — when a provider streams without usage data, this estimate becomes the number on the user's bill.

And chars / 4 is a rough rule of thumb for English prose. It breaks on other inputs. Our test suite uses a synthetic 400 KB input (1,600 repetitions of 250 x characters plus a space — the same one that caused the timeouts):

Characters: 1,600 × 251 = 401,600
Exact BPE:  52,800 tokens  →  401,600 / 52,800 = 7.6 chars/token
chars / 4:  100,400 tokens  →  401,600 / 4 = 100,400 (nearly 2× the truth)

If your budget allows 80,000 tokens, the real prompt fits (52,800) but chars / 4 says it doesn't (100,400) and rejects it. Using chars / 4 everywhere would overestimate so aggressively that it would reject roughly half of all valid prompts.

So the tokenizer stays. It just doesn't get to run unbounded anymore.

Code

The work happened in a private repo, so the PR link won't help. The fixed file is public in our mirror, with the measurement table in the doc comment:

apps/api/src/gateway/providers/token-estimate.ts

Before — the entire string in one encoder call:

if (isOpenAiFamily) {
  try {
    return encoder().encode(text).length;
  } catch { /* fall through to heuristic */ }
}
return heuristic(text.length);

After — two decisions: how much text to encode, and how to chop it up first:

if (isOpenAiFamily) {
  try {
    // Bound 2: past 20,000 chars, encode a sample and scale the rest.
    if (text.length <= MAX_BPE_SAMPLE_CHARS) return boundedEncode(text);
    const sampleTokens = boundedEncode(text.slice(0, MAX_BPE_SAMPLE_CHARS));
    const ratio = sampleTokens / MAX_BPE_SAMPLE_CHARS;
    return sampleTokens + Math.ceil((text.length - MAX_BPE_SAMPLE_CHARS) * ratio);
  } catch { /* fall through to heuristic */ }
}
return heuristic(text.length);

My Improvements

Two bounds. No new dependencies. No change to what it returns for normal input.

Bound 1 — cap each piece at 256 characters

Nothing over 256 characters reaches the encoder. Longer pieces get ceil(chars / 4) as a quick estimate instead.

This removes the quadratic cost entirely, because 256 chars keeps each encoder call under ~4 ms.

Important: 256 is a threshold, not a chunk size. A 1,024-character piece is not split into four 256-char chunks. It skips the encoder entirely and gets ceil(1024 / 4) = 256 directly. Zero encoder calls.

The text is only split at whitespace boundaries — to keep token counts accurate:

function boundedEncode(text: string): number {
  let tokens = 0;
  let pendingSeparator = '';
  for (const part of text.split(/(\s+)/)) {
    if (part.length === 0) continue;
    if (/^\s+$/.test(part)) {
      pendingSeparator += part;
      continue;
    }
    const piece = pendingSeparator + part;
    pendingSeparator = '';
    tokens += piece.length > MAX_BPE_PIECE_CHARS
      ? heuristic(piece.length)
      : encoder().encode(piece).length;
  }
  if (pendingSeparator.length > 0) {
    tokens += pendingSeparator.length > MAX_BPE_PIECE_CHARS
      ? heuristic(pendingSeparator.length)
      : encoder().encode(pendingSeparator).length;
  }
  return tokens;
}

The easy line to get wrong is pendingSeparator. Whitespace must be attached to the following word because tiktoken groups " word" as one token. Encode the space on its own and " the" becomes two tokens instead of one — every prose prompt gets overcounted by roughly its word count, and budget checks silently start rejecting requests they shouldn't.

A correctness bug from a performance fix is a bad trade.

Effect on the 20,000-character blob

	before	after
encoder calls	one, on 20,000 chars	none (20,000 > 256)
time	26,952 ms	0 ms
tokens returned	2,500	5,000

The count doubles — that's deliberate. ceil(chars / 4) overestimates, but 20,000 identical characters isn't a real prompt, and for a budget check, overestimating is the safe direction.

Ordinary text is untouched. "Hello world" splits into "Hello" and " world", both under 256, both encoded exactly as before.

Bound 2 — cap the total at 20,000 characters

Bound 1 only looks at one piece at a time. It never catches the case of many short pieces adding up.

1,600 repetitions of 250 xs plus a space = 400 KB. Each piece is 250 chars (under the 256 cap), so Bound 1 lets every one through. At ~4 ms each, that's 7 seconds. Bound 1 never triggers once.

Bound 2 fixes this: encode at most 20,000 characters total. For anything longer, encode the first 20K, measure the tokens-per-char ratio, and scale that ratio across the rest.

Step by step:

Take the first 20,000 characters
Run boundedEncode on them (still uses Bound 1 internally)
Get the token count from the sample
Divide by 20,000 to get a tokens-per-char ratio
Multiply the ratio by the remaining characters
Add sample tokens + tail estimate = return value

Effect on the 400 KB input

	bound 1 only	both bounds
characters encoded	401,600	20,000
time	7,018 ms	334 ms
tokens returned	52,800	52,771

The first 20K characters gave 2,628 tokens. Applied to the remaining 381,600 characters: 50,143 tokens. Total: 52,771 vs exact 52,800.

Sampled ratio:  2,628 / 20,000 = 0.13 tokens per character
chars / 4 ratio:            1 / 4 = 0.25 tokens per character

0.13 is roughly half of 0.25. Using chars / 4 would claim 95,400 tokens on the tail instead of 50,143 — nearly double the truth. The sample already measured the real ratio, so using it is almost as accurate as encoding everything, without the 7-second cost.

0.05% off, for 5% of the work.

Why 20,000 characters? At the 256-char piece cap, worst case is ~80 pieces (20,000 / 256). At ~4 ms each, that's ~320 ms — fast enough to stay on the request path. Anything bigger gets the sample treatment. The number isn't magic; it's where "encode everything" becomes "too slow."

Worst case is now roughly 350 ms for any input size.

Error direction

Bound 1 can only overestimate — ceil(chars / 4) always returns more tokens than real BPE on the inputs that trigger it. For a budget check, that's the safe direction.

Bound 2 is different: it extrapolates from a sample, so it can land slightly above or below the true count. The 400 KB case came out 0.05% low. But that only applies past 20,000 characters, and a fraction of a percent drift on a huge prompt is not a budget problem. A 7-second event-loop stall is.

Proving it didn't change the answer

A bounded estimator that returns different numbers is a new bug, not a fix. Tests pin exactness first, speed second.

Exactness — must match whole-string BPE on all normal input (under 20K chars):

it('matches whole-string BPE counts for ordinary text', () => {
  const encoder = getEncoding('cl100k_base');
  const samples = [
    'Hi Alice, what is the weather in London?',
    'The quick brown fox jumps over the lazy dog. '.repeat(114),
    'const x = foo.bar(baz, 42); // comment here\n'.repeat(100),
    JSON.stringify({ a: 'hello world', b: [1, 2, 3], c: { d: 'nested value here' } }).repeat(50),
  ];
  for (const sample of samples) {
    expect(estimateTokens(sample, 'gpt-4o-mini')).toBe(encoder.encode(sample).length);
  }
});

Prose, code, JSON: identical counts. No existing estimate moved.

Direction — pathological input must never come back low:

it('never underestimates the pathological input', () => {
  const encoder = getEncoding('cl100k_base');
  const text = 'x'.repeat(2000);
  expect(estimateTokens(text, 'gpt-4o-mini')).toBeGreaterThanOrEqual(encoder.encode(text).length);
});

Speed — one test per bound, so removing either guard fails immediately:

it('stays fast on a long unbroken run of characters', () => {
  estimateTokens('warm up the encoder', 'gpt-4o-mini');
  const started = Date.now();
  const n = estimateTokens('x'.repeat(50_000), 'gpt-4o-mini');
  expect(n).toBeGreaterThan(0);
  expect(Date.now() - started).toBeLessThan(1000);
});

it('stays fast when many pieces sit just under the per-piece cap', () => {
  estimateTokens('warm up the encoder', 'gpt-4o-mini');
  const text = `${'x'.repeat(250)} `.repeat(1600);
  const started = Date.now();
  const n = estimateTokens(text, 'gpt-4o-mini');
  expect(n).toBeGreaterThan(0);
  expect(Date.now() - started).toBeLessThan(1000);
});

The 1-second limit is deliberately generous — won't flake on loaded CI, but still fails instantly if someone removes a guard (the unfixed path takes minutes).

Results

	before	after
50,000-char unbroken run	minutes	< 1 s
400 KB of 250-char pieces	7,018 ms	334 ms
completion, 9,000-char variable (provider mocked)	5,506 ms	~82 ms
`datasets` test A	6,340 ms	410 ms
`datasets` test B	6,243 ms	323 ms

The timing-out tests were never touched. They passed because the code underneath stopped being slow.

What I'd carry to another codebase

Test adversarial input, not your own. Every prompt I'd written was ordinary prose — the tokenizer's best case. Timing 'x'.repeat(20000) took ten minutes and found the whole bug.
A synchronous function on a request path is a shared resource. "It's just a pure function over a string" stops being reassuring when the string is caller-controlled and the cost is superlinear.
Pin the old answer before you optimize. An exact-match test against whole-string encoding is how I caught the pendingSeparator detail that would have silently overcounted every prompt.

If you're running a BPE tokenizer anywhere synchronous, it's worth timing encoder.encode('x'.repeat(20000)) on your own setup today. What did you get?

I Thought Building Agent Observability Was a Detector Problem. I Was Wrong.

Debashish Ghosal — Fri, 07 Aug 2026 04:52:23 +0000

When I started building agent-exec-trace, I thought I knew where the difficulty would be. OSS agent-exec-trace is an observability layer for AI agents. It instruments agent runs using OpenTelemetry-style traces, then analyzes those traces for loops, retry storms, cost spikes, hallucination patterns, and other bad runtime behavior. The goal is simple: when an agent run feels wrong, I want something better than logs and intuition to explain why.

I thought the hard part would be the detector logic.

Define the anomalies. Pick the thresholds. Wire the traces. Run the analytics. Ship the repo.

That was the clean version of the story.

The real version was uglier.

On my first large pass over 100,000 real agent traces from Hugging Face, my empty_response detector fired on 100% of them. Every trace. At the same time, 28 of my 35 rule-based detectors never fired at all.

That is the moment the project stopped being "build some detectors" and turned into a learning experience.

The detectors were not wrong.

The traces were the wrong shape.

And once I saw that, a bunch of other assumptions broke with it.

I had a very similar feeling when I shipped my agent eval harness. Different tool. Same pattern. Real systems are very good at ruining tidy design stories.

What I Was Actually Trying To Build

The problem I cared about felt pretty obvious.

Traditional observability can tell me if a service is up, slow, or failing.

It cannot tell me why an agent called the same tool eight times, changed its mind three times, burned tokens on a dead-end path, or quietly drifted into behavior I would never want repeated in production.

That gap matters more than people admit.

Most teams can tell you the run felt wrong. Very few can point to the exact step where it started going wrong.

So I built agent-exec-trace: an OpenTelemetry-style observability layer for agent behavior.

The idea is simple enough to explain.

Instrument the run. Capture the behavioral path. Emit spans for planning, tool calls, retrieval, memory, approvals, and cost. Then run detectors over those traces and surface the bad runs in a UI someone can actually use.

The repo is here: github.com/deghosal-2026/agent-exec-trace

It ships as three PyPI packages, has 794 Python tests and 34 Playwright end-to-end tests, and the SDK is deliberately small enough that someone can wrap a real agent without redesigning their system.

from agent_exec_trace import AgentTracer, trace_agent, tool_span

AgentTracer.setup(otlp_endpoint="http://localhost:4317")

@trace_agent(agent_name="my-agent", agent_version="1.0.0")
async def handle(query: str) -> str:
    with tool_span("search", tool_args={"q": query}):
        return await search(query)

That part was not the lesson.

The lesson was what happened after I had something real enough to test.

The First Thing I Learned: Real Traces Fight Back

I did plan field testing.

Just not early enough.

That distinction matters, because the mistake was not forgetting it. The mistake was scheduling it late enough that the rest of the system had already started to feel done.

The WBS was solid. The PRD was thorough. The detector catalog was documented. The milestones were real. The quality gates were strict: ruff clean, mypy strict clean, tests green, coverage above 90%.

All of that was good discipline.

None of it protected me from the fact that mock agents are polite and real traces are not.

That is what "too late" looked like in practice. By the time field testing showed up, parts of the detector catalog, the assumptions about trace shape, and even some milestone gates already felt validated. Field testing did not arrive early enough to shape the design. It arrived late enough to expose where the design had been validated against the wrong reality.

The Hugging Face corpus was the first slap.

My detector expected response content in one place. The corpus stored it in many different shapes. So empty_response was not telling me "all these agents are broken." It was telling me "your assumptions about trace shape are too narrow."

That one bug forced four normalization passes. Different response keys. Different tool-name conventions. Different operation names. Timestamp parsing issues. Parent-child inconsistencies.

Bad validation data does not just create noise. It teaches you the wrong lessons with confidence.

By the end of that cleanup, the headline got more honest and less comforting.

The original compatibility number looked decent.

The real one was 42.4%.

That was the moment I stopped thinking about observability as a detector problem and started thinking about it as a data-shape problem.

Because a detector that is perfectly correct in theory is still useless if the data it depends on almost never exists in the wild.

The Second Thing I Learned: Synthetic Data Can Save You And Mislead You

Once the real-trace corpus showed me its limits, I went in the other direction.

I built a synthetic trace generator.

One million traces. Ten fake agents. Fourteen tools. Deliberate behavior modes: loops, retries, timeouts, inactivity gaps, intervention waits, token explosions, memory bursts.

That solved one problem immediately.

Structural compatibility jumped to 99.2%.

Now the detectors had something they could actually see.

Twenty of the 35 rule-based detectors fired.

That sounds like a win. And it was.

But synthetic data lies in a different way.

My hallucination detector fired on 98% of synthetic traces. Not because I had accidentally built the greatest detector in history, but because the synthetic outputs and the synthetic tool evidence had a fake relationship that made the detector's job too easy.

The same thing happened with cost. My cost_spike detector had a real threshold. My synthetic generator produced mostly cents-level costs. So the detector almost never fired there either.

Synthetic traces can prove a detector runs. They cannot prove it matters.

That gave me a second lesson I needed badly:

Synthetic traces are great for proving a detector can run.

They are not enough to prove a detector is calibrated for reality.

So now I think of validation in layers.

Unit tests tell me the logic works.

Synthetic traces tell me the detector can see the fields it needs.

Real traces tell me whether any of this matters outside my own sandbox.

That sequencing sounds obvious when written down. It did not feel obvious when I was in the middle of building it.

The Third Thing I Learned: Green Gates Can Lie

This one annoyed me more than any detector bug.

I had marked the OTLP export milestone done.

The gate was green.

The docs looked fine.

The demo looked fine.

And the end-to-end export path had never actually been verified.

Two bugs had cancelled each other out.

The OTel collector's gRPC port was not exposed in Docker Compose.

And the SDK path I was using configured local tracing instead of OTLP export.

So I had a completed milestone for a feature that had not really worked.

That was a worse lesson than a failing test.

A failing test is honest.

A green gate created by two cancelling bugs is dishonest in a way that looks disciplined.

That changed how I think about milestones.

I trust a green checkmark a lot less now.

If the gate cannot prove that a real agent emits, Jaeger receives, analytics ingests, and the API serves the result, then the gate is incomplete. I do not care how many sub-checks passed before it.

That is not an observability lesson only. That is just software engineering.

The Fourth Thing I Learned: Structure Without Content Is Not Enough

I started fairly conservative on privacy.

Metadata-only felt responsible.

No raw tool arguments. No full tool responses. No memory values by default.

Reasonable instinct.

But it taught me a more uncomfortable tradeoff.

An observability SDK that only captures structure can become blind in exactly the places where you most want judgment.

My hallucination detector is the clearest example.

If the detector cannot see what the tool returned, it cannot meaningfully judge whether the agent's claim matches the evidence.

Once I allowed truncated content instead of metadata-only, the hallucination false-positive rate dropped sharply.

That did not make the privacy question go away.

It just made the tradeoff explicit.

"Safer by default" and "useful by default" are not always the same choice.

Structure without enough evidence is just a cleaner way to stay blind.

I do not think enough tooling says that out loud.

The Fifth Thing I Learned: Shipping OSS Is Not The Same As Finishing The Idea

The repo is public.

The packages are published.

The test suite is real.

The product works.

And I still would not call the problem solved.

That matters to me.

I think too many OSS launch posts flatten everything into "here is what shipped" and skip "here is what I still do not trust."

Here is what I still do not trust fully:

the 28 detectors that have not yet fired on real corpora in a meaningful way
the LLM detectors on production workloads
span-tree materialization in the API, which is still weaker than I want
any claim that one threshold set will generalize cleanly across workloads
any green gate that has not been proven end-to-end

That does not mean the project should not ship.

It means the honest version of shipping is: this tool is useful now, and I understand its edges much better than I did when I started.

That is a good outcome.

It is also a very different outcome from the clean version I had in my head.

What I Would Do Differently Next Time

I would move field testing much earlier.

Not because I failed to think of it, but because planning it and planning it early are not the same thing.

Late field testing turns into audit. Early field testing shapes the design.

By the time mine ran, it was strong enough to expose bad assumptions, but too late to stop me from building confidence on top of them.

I would treat the trace corpus as a design artifact, not just test input.

I would insist on an end-to-end smoke path much earlier.

And I would stop pretending that detector logic is the center of the problem.

It matters, obviously.

But after building this, I think the harder questions are:

what can your traces actually see?
how weird is the data in the wild?
what does your detector depend on structurally?
and how do you know a green gate is real?

That is the actual education this project gave me.

Not "how to write 40 detectors."

More like: how many other things have to be true before 40 detectors mean anything.

And maybe the hardest takeaway of all: a detector that has never fired on a real trace is not really finished. It is just well tested.

Open For Discussion

I would genuinely love practitioner pushback on this.

If you are running agents today, what do you actually use when a run feels wrong?
Have you ever had a green gate that passed for the wrong reason?
And if you are building agent tooling, where did your clean version of the story break first?

github.com/deghosal-2026/agent-exec-trace · PyPI

I've Spent Months Grading AI Agents' Code for a Living. Here's the Pattern Nobody's Talking About

Marvin Okafor — Thu, 06 Aug 2026 18:20:52 +0000

Everyone's talking about agentic AI shipping production code. Nobody's talking about what happens when you actually sit down and grade thousands of lines of it against a rubric, line by line, for months.

I have. And the failure pattern that shows up over and over isn't the one Twitter/X is arguing about.

The job title that didn't exist two years ago

"AI evaluator." "AI trainer." "Expert contributor to frontier model training data." None of these existed as job titles when I started my career. Now they're where a chunk of the most interesting engineering signal in the industry is actually happening — quietly, behind NDAs, far from the demo videos.

Here's what the job actually is: agentic coding outputs land on your desk, and you grade them against a structured rubric — correctness, instruction adherence, quality, edge-case handling. You design adversarial prompts to find where the model's reasoning breaks. You decide which checks can be programmatic and deterministic, and which genuinely need a human who's shipped production systems to make the call. This is RL environment design and LLMOps in its rawest form, and it's a completely different skill from "prompt engineer" or "ML researcher." It's closer to being a QA lead for a junior engineer who never sleeps, never gets embarrassed, and will confidently ship the wrong answer with perfect syntax.

The pattern: agents are great at code, bad at consequences

Here's the uncomfortable part. The failure mode people are loudest about — hallucinated APIs, made-up library functions — is the easy failure mode. It's loud, it's obvious, and any decent test suite catches it in seconds.

The failure mode that actually matters, the one that slips past a surface read and even past a naive test suite, looks like this:

The code is syntactically perfect and semantically wrong about failure. It handles the happy path beautifully and quietly assumes the retry, the timeout, the partial write, the duplicate message never happens.
It optimises for the metric, not the intent — the agentic-AI version of Goodhart's Law. Give a model a rubric that checks "does the deploy succeed," and you'll occasionally get a solution that technically satisfies the check while doing something no engineer would sign off on. Evaluators call this reward hacking, and it's a far more common failure than outright hallucination once you're grading real-world infra tasks instead of leetcode.
It's confidently wrong about IAM, concurrency, and distributed state — exactly the areas where production engineering experience matters most and where a rubric written by someone who's never operated a real system will miss the defect entirely.

None of this is a knock on the models. It's a knock on how we evaluate them. You cannot catch consequence-blindness with a rubric written by someone who has never had a database silently corrupt state under a race condition at 2am. This is the actual bottleneck in scaling agentic AI into production-grade infrastructure work: not model capability, evaluation quality.

Why "vibe coding" breaks down at the infra layer

"Vibe coding" — accepting AI-generated code because it looks right and the demo works — is fine for a prototype. It is not fine for anything touching IAM policies, message queues, durable storage, or disaster recovery. The gap between "looks right" and "is right" is exactly the gap that golden reference solutions and deterministic validation tests exist to close — the same discipline I wrote about in my last post on building RL environments for cloud infrastructure evaluation.

The uncomfortable truth for the "AI writes all our code now" crowd: the more production-grade the system, the more the bottleneck shifts from generating the code to specifying and verifying it. That's a systems-engineering problem, not a model-scaling problem. It's also, not coincidentally, exactly what senior backend engineers have spent their careers getting good at — writing test suites against real databases instead of mocks, root-causing defects that hide three layers deep, documenting edge cases precisely enough that someone else can reproduce the reasoning. That skill set didn't get less valuable when agents showed up. It became the thing standing between "the demo worked" and "it survived contact with production."

What I'm building next

I'm turning this into a real project instead of just a hot take: a lightweight harness for stress-testing AI-agent-generated infrastructure code against realistic, injected failure conditions — retries, partial outages, IAM misconfigurations, the exact defect classes I've been describing above — with deterministic pass/fail checks instead of vibes. Think chaos engineering meets AI evals: inject one fault at a time, assert invariants instead of traces, and see whether an agent's "working" solution is actually working or just golden-path lucky.

It'll live on my portfolio and GitHub as I build it in the open — seed scenarios, the fault-injection harness, and a write-up of what breaks and why. If you're working on anything adjacent (RL environments, AI evals, chaos engineering, or you've just been burned by AI-generated infra code in production), I want to hear about it — drop it in the comments or find me on GitHub.

The takeaway

Agentic AI isn't going to be stopped by a model that can't write a for-loop. It's going to be shaped by whether the industry gets serious about evaluation infrastructure — golden solutions, deterministic tests, adversarial failure scenarios — as fast as it's getting excited about generation. That's the unglamorous, unsexy, extremely fundable problem hiding behind every "AI wrote our whole backend" headline.

If you've seen this pattern too — agents that ace the demo and fail the disaster-recovery drill — I'd genuinely like to compare notes.

This is the fifth in a series on production engineering, debugging, and building evaluation environments for AI systems. Follow for the harness build-in-public over the coming weeks.

Our eval gate runs 22 minutes. The queue behind it hit three hours.

Ethan Walker — Thu, 06 Aug 2026 17:03:21 +0000

Nobody complained about the gate. They complained about Tuesday.

Our merge queue runs the full eval suite before anything lands: 1,400 cases, about 240 of them scored by an LLM judge, the rest deterministic. Wall clock, 22 minutes. One PR at a time, because the suite pins the same eval dataset and the judge budget. Every engineer on the team would tell you 22 minutes is fine. Go get coffee.

Then one Tuesday I counted nine PRs sitting in the queue at 11:04. The last one merged at 14:22. Three hours and eighteen minutes for a change that took 40 minutes to write, and the gate itself never ran slow, never flaked, never failed. Every single run took its normal 22 minutes. The queue did the rest.

The math nobody ran

A serial gate is a single-server queue. What sets the wait is the ratio between the gate's duration and how fast work arrives. That ratio has a name, utilisation, and the wait it produces is not linear in it. It has a knee.

For a fixed 22-minute service time and randomly arriving merges, the steady-state average wait in queue is:

def gate_queue_wait(s_min: float, arrivals_per_hr: float) -> float:
    """Average queue wait (minutes) before a serial s_min-minute gate.
    M/D/1: Wq = rho * s / (2 * (1 - rho)), rho = arrival rate * s."""
    rho = arrivals_per_hr * s_min / 60
    if rho >= 1:
        return float("inf")  # the queue never drains
    return rho * s_min / (2 * (1 - rho))

for rate in (1.0, 1.5, 2.0, 2.25, 2.5, 2.6):
    print(f"{rate:>4}/hr  wait {gate_queue_wait(22, rate):6.1f} min")

Run it for our 22-minute gate and the knee is right there:

1.0/hr: utilisation 37%, avg wait in queue 6.4 min, wait + gate 28.4 min
1.5/hr: utilisation 55%, avg wait in queue 13.4 min, wait + gate 35.4 min
2.0/hr: utilisation 73%, avg wait in queue 30.2 min, wait + gate 52.2 min
2.25/hr: utilisation 82%, avg wait in queue 51.9 min, wait + gate 73.9 min
2.5/hr: utilisation 92%, avg wait in queue 121.0 min, wait + gate 143.0 min
2.6/hr: utilisation 95%, avg wait in queue 224.7 min, wait + gate 246.7 min
Capacity is 2.73 merges per hour. That is the whole budget a 22-minute serial gate gives you, ever.

At one merge an hour the gate is invisible. At two an hour, the average PR waits longer in the queue than it spends being tested. At two and a half, the average experience is over two hours, and that is the average on a quiet, evenly spaced day. The formula assumes arrivals sprinkled at random. Real teams merge in bursts, after standup, before the sprint cutoff, and a burst is strictly worse than the formula. Our nine-PR Tuesday was not an anomaly. It was nine arrivals hitting a server that clears 2.7 an hour, and 9 times 22 minutes is 3 hours 18. The math was never going to do anything else.

The part that stung: we had spent a month optimising the suite from 26 minutes down to 22, and we only understood what we had bought after running these numbers. At two merges an hour, the old gate held the average wait at 85 minutes; the new one holds it at 30. And on Tuesday-heavy stretches at two and a half, the 26-minute suite was past capacity entirely (60 over 26 is 2.3 an hour): that queue was not slow, it was diverging, and we had fixed it by accident. Service time matters exactly as much as the ratio says it does, and none of that is visible from the gate's runtime alone.

What we changed

Three tiers, nothing clever.

Tier 0, per commit, 90 seconds. Every deterministic check that does not need the full dataset: schema conformance, tool-call shape, regex and exact-match cases, token-budget ceilings. About 400 of the 1,400 cases, and historically the tier that catches most honest mistakes. Runs on push, before review, outside the merge queue entirely.

Tier 1, the merge queue, batched. The full 22-minute suite still gates every merge, but it gates batches, not PRs. As soon as the runner is free it takes whatever is queued, up to four PRs, and runs once against the batch head. Green, all four land. Red, the batch splits into pairs and reruns, then the failing pair splits again: standard bisection, four extra runs worst case to isolate one offender. A batch of four cuts effective service per PR to five and a half minutes, which moves us from 92 percent utilisation back to about 23 at the same merge rate; the average wait falls off the bottom of the table, into single-digit minutes. Red batches cost us the bisection, and at our failure rate the trade wins by a large margin; if your gate is red a third of the time, batching will hurt, and your problem is the failure rate anyway.

Tier 2, nightly. The expensive sweeps that never belonged in a merge path: full judge calibration against the human-labelled set, cross-model regression, the long-tail scenarios. Nightly, with a report in the morning, and a rule that a red nightly blocks the next day's releases rather than the next engineer's merge.

Did we lose per-PR attribution inside a green batch? Yes, and we decided we do not care: a green batch means every PR in it shipped against the full suite. Attribution only matters on red, and red is exactly when bisection runs.

What it cost

One week of one engineer, most of it spent classifying the 1,400 cases into deterministic versus judge-scored, and deciding which of the heavyweight sweeps belong in the new nightly tier. The queue infrastructure was our existing merge queue with batching turned on. The 22-minute suite itself did not change at all, which is the point: we stopped paying for it per PR without deleting a single case. The nightly sweeps run on top of the merge gate, not instead of it.

Median wait in the queue, the time before your run even starts, went from 74 minutes that Tuesday-heavy week to under six the week after. Add the 22 minutes the suite still takes and a typical merge now lands in under half an hour.

What I'd check first

Your gate's service time against your busiest hour's arrival rate. Multiply them, service time in hours. Past 0.7, you have a queueing problem, not a test-speed problem, and shaving suite minutes will not save you.
What fraction of the gate is deterministic checks that could run per commit. Ours was 29 percent of cases and they catch the majority of failures; they were queueing behind an LLM judge for no reason.
Whether your queue serialises PRs that could gate as a batch. Serial-per-PR is the default in most setups. It is also the strictest possible service discipline, and the table above is the bill for it.