DEV Community

t49qnsx7qt-kpanks
t49qnsx7qt-kpanks

Posted on

agents fail 70-95% of the time in production. here's why the fix isn't observability.

agents fail 70-95% of the time in production. here's why the fix isn't observability.

Fiddler's latest data puts AI agent production failure rates between 70-95%. the number gets worse when you chain agents: if each one succeeds 70% of the time, a three-agent sequence succeeds 34% of the time. most teams look at that and think "we need better monitoring."

monitoring isn't the problem.

the teams i've watched get stuck in production aren't flying blind — they have dashboards, traces, latency numbers. what they don't have is a record that answers the question their compliance team, their customers, or their incident reviewer will eventually ask: what decision did the agent make, why, and who authorized it to make that call?

observability tells you an agent failed. audit tells you what it tried to do before it failed, what policy it was operating under, and whether that policy was even the right one to apply.

those are different problems.

the three-agent math is a governance problem, not a reliability problem

a 34% success rate across a three-agent chain isn't just an engineering failure. it's a governance failure. when the chain breaks, you need to know:

  • which agent in the chain made the first bad call
  • what data it was operating on at that moment
  • whether that decision was within its authorized scope
  • what the downstream agents did with the bad output before anyone noticed

without that, you're doing forensics on traces — matching timestamps and log lines manually, hoping the failure left enough breadcrumbs. that works once. it doesn't work when your prod system runs 10,000 agent executions a day.

the teams that move past pilot phase aren't the ones with the best observability stack. they're the ones that built a decision log before they needed it.

what a decision log actually looks like

it's not a trace. a trace records what happened at the infrastructure layer — function calls, latency, token counts. a decision log records what the agent chose and why:

agent_id: agent_checkout_v2
timestamp: 2026-05-12T14:32:11Z
action: initiate_purchase
authorized_by: policy_v4_spend_under_500
amount: $312.00
prior_balance_check: passed
anomaly_flag: none
human_approval_required: false (below threshold)
Enter fullscreen mode Exit fullscreen mode

that's auditable. that's what a compliance reviewer can read and sign off on. traces can't produce that because they're not structured around decisions — they're structured around execution.

the governance gap LangChain's own data confirms

LangChain's State of Agent Engineering 2026 report shows 89% of their users have implemented observability. only 20% have mature governance models. the gap isn't awareness — it's that teams assume observability closes the governance gap. it doesn't.

governance is about authorization, scope, and accountability. observability is about performance and debugging. you need both, in that order.

what actually moves agents from pilot to production

the orgs i've seen clear the pilot-to-production gate share one pattern: they defined the audit surface before they scaled. not after the first incident. before.

that means:

  • every agent has a declared authorization scope (what it can and can't do)
  • every decision is logged against the policy it was operating under
  • human-in-the-loop triggers are wired in at the scope boundary, not bolted on after failures

this isn't a lot of engineering. it's mostly configuration and discipline. but it has to happen before the 10,000-execution day, not on the day after your first production incident.

the $997 question

BizSuite AI Audit is a 48-hour audit report that maps your agent deployment against these governance requirements. it's a wedge — not a platform. the point is to find out, fast, what your actual governance exposure is before you're answering questions from a regulator or a customer.

if you're running agents in production and haven't done a governance pass, that's where i'd start: https://getbizsuite.com/ai-audit

the 70-95% failure rate isn't a reason to slow down agent deployments. it's a reason to get the governance layer right before you scale.

Top comments (0)