DEV Community: Arleen Kaur

Why Your AI Agents Are Failing: The Routing Problem Nobody Is Solving

Arleen Kaur — Tue, 16 Jun 2026 12:46:29 +0000

AI Disclosure: This post was written with AI assistance and has been reviewed and approved for publication by the Linksoft Technologies team.

Everyone's racing to deploy AI agents. Speed creates the illusion of progress, but it doesn't guarantee advantage. The real cost shows up later — in how the system behaves under load.

Read those three numbers together. Almost every enterprise is running AI. Most say cost efficiency is a top priority. And almost none have built the AI agent architecture layer that would actually solve it. That's the defining infrastructure gap of this moment.

The conversation in most strategy decks is still stuck in the wrong place: which model to pick, which vendor to trust, build or buy. Surface-level. Symptom-chasing. Completely missing the structural problem underneath.

Companies running AI at real scale aren't running better models. They're running better systems around models. That's the difference most teams still miss and it usually shows up in the budget later.

The Instinct That's Costing You

When organizations get serious about AI, the instinct makes sense. Use the most capable model available. It reasons best, handles ambiguity best, writes best. So you build your first agent on GPT-4 or Claude Opus or whatever tops the benchmark table and it works. Impressively, even.

Then you try to scale it. That's where the math gets uncomfortable.

Large frontier models are built for complexity. But most tasks in any real-world AI pipeline aren't complex. They're repetitive, narrow, and structurally simple. When you route everything through a hundred-billion-parameter model, you're paying for capability you don't need, latency you don't want, and token counts that scale linearly with volume.

Google Research's work on Switch Transformers documented up to 7x gains in pre-training efficiency with the same compute, proving these aren't theoretical. The question is whether your orchestration layer is built to capture them.

Sequoia Capital's analysis points to a $500B annual revenue gap where infrastructure investment dramatically exceeds realized returns. Getting model routing wrong isn't just an efficiency concern. At scale, it turns into a margin problem.

The Architecture Is the Problem

The default approach produces a flat pipeline: one input, one large model, one output, repeat. No routing. No complexity awareness. Every task treated identically regardless of what it needs.

In a proof of concept this works fine. At scale, the cost problem stops being abstract and by then the architecture is already too embedded to change easily.

The pilot looks fine. Production is where things start to break and that's the trap most scaling teams walk into.

What Is Model Routing in AI and Why Does It Matter?

Model routing is the orchestration layer that decides which AI model handles which task. It sends complex, ambiguous requests to large frontier models and simple, repetitive ones to smaller, faster, cheaper models.

Without it, every task gets routed to the same model regardless of what it actually needs. You pay frontier-model prices for work a fraction of the cost could handle equally well.

At scale, that's not an efficiency gap. It's a margin problem. Model routing is what closes it by matching compute to complexity the same way a hospital matches patient complexity to the right tier of care, rather than routing every case to the senior specialist.

What the Fix Actually Looks Like

Think of it like triage in a hospital. You don't route every patient with a minor injury to your most senior specialist. You have a system that matches people to the right level of care, reserving specialist time for cases where their expertise is genuinely irreplaceable.

Your large model's compute is the specialist's time. The orchestration layer is the triage system. Without it, you have queues, waste, and costs that don't hold at scale.

"The key isn't just about choosing the cheapest option, but about finding the right recipe of tools and services that aligns with your workload patterns."
-- Google Cloud

How to Design Efficient AI Agent Architectures for Enterprises

Efficient enterprise AI agent architecture is built in tiers:

Tier 1 Lightweight model: Handles narrow, high-volume, structurally simple tasks
Tier 2 Mid-tier model: Handles moderate reasoning and mixed-complexity requests
Tier 3 Frontier model: Reserved for genuinely complex or high-stakes cases only

Each tier has defined cost, latency, and quality thresholds. On top of this sits an observability layer that tracks which tasks are going where, at what cost, and with what outcomes, so routing decisions can be continuously calibrated rather than set once and forgotten.

The organizations that reduce AI agent orchestration costs at scale aren't running better models. They're running better systems around models, with architecture that matches spend to need at every step.

Why Most Teams Haven't Built This Yet

There are really two reasons and neither has anything to do with a lack of skill.

Reason 1 The early pain isn't visible.
When you're running a proof of concept, the cost difference between a large model and a small one feels abstract. It only becomes obvious at scale, when the budget impact is undeniable and the system is already too embedded to change easily.

Reason 2 Tiered orchestration is genuinely harder to build.
A single model pointed at a task is simple. An orchestration layer that correctly classifies tasks, routes them, handles edge cases, and maintains consistency across multiple models is a serious systems problem. It's the kind that takes six to eighteen months to build properly.

The Agent Reality Check

Let's be direct: the hype cycle has significantly outpaced the deployment reality. Most of what organizations have built and called "agents" are, on close inspection, sophisticated chatbots with tool access bolted on. They fail in three specific, predictable ways and all three are architectural problems, not model quality problems.

This is precisely why now is the right moment to pivot. The infrastructure including Kubernetes, LangGraph, sandboxed execution environments, and proper observability tooling exists and is maturing. Companies that start building now will be early-to-mid players, not laggards doing emergency re-architecture two years from now.

NVIDIA defines agentic systems as "autonomous, long-running agents that reason, plan and act across complex, multi-step workflows," a definition that highlights how far most current implementations still have to go. This isn't a reason to pull back but a signal to treat this like a real systems problem.

What You Should Actually Be Tracking

Tracking the right metrics requires an AI oversight framework that connects routing decisions to business outcomes, not just benchmark scores.

Most AI business cases get approved on model performance benchmarks, which is the wrong number to optimize for. The real cost including container orchestration, workflow state management, sandboxed execution, observability tooling, and routing model maintenance rarely makes it into the same deck. So the ROI gap isn't surprising. The real cost was never fully accounted for in the first place.

McKinsey estimates generative AI could add $2.6T to $4.4T annually to the global economy, with total productivity impact reaching $7.9T. The cost of getting system design wrong will scale right alongside the opportunity, not independently of it.

Three metrics worth tracking instead of benchmark scores:

Cost per automated task: should decline as volume grows. Flat or rising cost signals wrong-tier routing
Routing accuracy rate: target above 92% of tasks correctly classified by complexity. Mis-routing routine tasks to frontier models is where budget leaks
Escalation override rate: target below 8% of auto-routed decisions manually corrected. A high rate signals the routing model needs recalibration, not more reviewers

Q&A: What Engineering and Architecture Teams Actually Ask

What's the difference between model routing and prompt routing?
Prompt routing selects between different prompts or instructions for the same model. Model routing selects between different models entirely based on task complexity. The distinction matters at scale: prompt routing doesn't reduce compute costs because you're still running the same model. Model routing does, by matching task complexity to appropriately sized infrastructure.

How do you classify task complexity reliably enough to route it?
Start with a lightweight classification model, often a fine-tuned smaller model trained on your own task distribution. The classification step itself costs almost nothing relative to the savings from correct routing. Track misroutes (tasks sent to the wrong tier) the same way you'd track model errors: as a calibration signal, not a failure.

What happens when a task is misclassified and routed to the wrong tier?
A task routed down (sent to a smaller model than it needs) produces a lower-quality output, detectable via output scoring or human review flags. A task routed up (sent to a larger model than needed) just costs more than necessary. Build fallback logic: if the lower-tier model's confidence score falls below a threshold, escalate automatically.

Does tiered routing work for LLM-based agents, or just classification tasks?
It works for both. For agents, the routing decision happens at the task-dispatch layer before any tool calls are made. Simple deterministic sub-tasks like formatting, extraction, and lookup go to lightweight models. Multi-step reasoning chains or ambiguous open-ended tasks go to frontier models. The orchestration layer manages the handoff.

How long does it realistically take to build a proper routing layer?
Six to eighteen months for a production-grade system, depending on the number of task types, the variance in your data distribution, and how mature your observability infrastructure is. The first version is always simpler. The hard part is continuous calibration: keeping routing decisions accurate as your task mix shifts over time.

Three Verdicts, One Principle

01 Single-model stacks are not production architectures.
Routing every task to the same frontier model has no cost-efficiency mechanism, no complexity awareness, and no path to economic viability at scale. Without an AI oversight framework to govern routing decisions, better models only delay the budget problem. They don't solve it.

02 Routing is required and it can't be an afterthought.
Bolted on after the fact, tiered orchestration requires re-architecting systems already embedded in production. The organizations building it now are the ones who won't be explaining budget overruns to their CFO eighteen months from now.

03 The infrastructure is where the advantage actually sits.
Kubernetes, LangGraph, sandboxed execution, observability tooling, feedback-integrated recalibration. These aren't operational add-ons. The organizations with structural AI advantages aren't running the most powerful models. They're the ones who figured out that the game is about using the right model for each task and built the systems to make that happen.

"Enterprises that build intelligent orchestration into their AI systems early will run dramatically more automations per dollar of cloud spend. The competitive advantage in agentic AI is not a better model. It is a better system."

That's not an AI strategy. It's a systems design strategy, applied to AI. And that distinction is where most of the real value is going to be created.

Everything else works right up until it hits a budget ceiling.

About the Author:
Arleen Kaur writes about enterprise AI, system architecture, and the gap between AI pilots and production systems at Linksoft Technologies, a custom software development company.

Sources referenced:
Sequoia Capital -- $500B AI infrastructure revenue gap analysis
McKinsey -- Generative AI economic impact ($2.6T to $4.4T annually)
NVIDIA -- Agentic AI system definition

Human in the Loop AI as a Production Requirement: Why Control Architecture Determines Enterprise AI Success

Arleen Kaur — Mon, 15 Jun 2026 18:10:16 +0000

AI Disclosure: This post was written with AI assistance and has been reviewed and approved for publication by the Linksoft Technologies team.

88% of enterprises are running AI. Only 4% are generating meaningful returns. The gap isn't the model it's everything built around it.

Here's a number that should make every engineering leader uncomfortable:

95% of enterprise AI pilots deliver zero measurable ROI. Not low ROI. Not disappointing ROI. Zero.
McKinsey Global AI Survey, 2025

Read that again. Not in under-resourced startups. Across the enterprise, across industries, after years of investment and board-level attention.

And the conversation in most strategy decks stays exactly where it's been for three years: better models, faster inference, which LLM to pick, whether to build or buy. Symptom-chasing. Completely missing the structural problem underneath.

The companies generating returns aren't running better models. They're running better systems and that starts before the output layer. The routing problem is where most architectures break first long before human oversight even becomes relevant.

The Adoption Numbers Tell a Story Nobody Wants to Read

The gap between 88% adoption and 4% meaningful returns isn't a model quality problem. GPT-4, Claude, Gemini these are not the bottleneck.

The bottleneck is organizational design: how the AI is deployed, what governs it, and what happens when it gets something wrong.

The dominant failure pattern, documented consistently across McKinsey's 2025 State of AI report and the Partnership on AI's Enterprise Landscape research, is this: organizations insert AI into existing workflows without redesigning those workflows first. AI inherits broken processes and accelerates them. Garbage in, faster garbage out.

55% of high-performing AI organizations redesign workflows around AI before deploying. Among the broader population, that figure is 20%. That 35-point gap in process redesign explains most of the performance differential.

Why the Architecture Is the Problem, Not the Algorithm

AI models are probabilistic systems. They output confidence scores that measure certainty not correctness. A model can be 94% confident and completely wrong, not because it's a bad model but because the input falls outside its training distribution. And here's what makes this dangerous in production: the model has no mechanism to know this.

The error propagates downstream, silently, until something breaks visibly.

In enterprise environments, three things compound this that simply don't exist in a controlled pilot: data that changes constantly, decisions that can't be reversed, and legacy infrastructure never designed for AI.

The standard autonomous architecture is:

Input → Model → Output → Action

No monitoring. No feedback. No correction layer.

In a controlled pilot, this works. In live production with financial and legal consequences, it fails not immediately, but inevitably.

64% of organizations stall at the scaling stage because of infrastructure debt a clean pilot environment never exposed.
The pilot succeeded. The production environment is not the pilot.

What Is Human-in-the-Loop and Why It's Not Enough on Its Own

Human-in-the-loop (HITL) places a human reviewer between an AI's output and the action it triggers. It creates an intervention point and satisfies regulatory mandates like EU AI Act Article 14, which requires human oversight for high-risk AI in employment, credit, healthcare, and critical infrastructure.
It's structurally necessary. But at production scale, HITL as currently implemented fails in three specific, predictable ways.

Failure 1 — Automation bias

Review interfaces present cases structured around the model's interpretation. Reviewers are evaluating a pre-framed answer, not the situation itself. Research is consistent: humans default to confirming AI outputs rather than questioning their premise. HITL looks like independent oversight. Functionally, it's a rubber stamp at velocity.

Failure 2 — Volume collapse

Human attention doesn't scale with decision throughput. As queues grow, reviewers apply faster heuristics to clear them effectively re-automating the decisions HITL was supposed to oversee. No amount of reviewer training changes this. It's an architectural constraint, not a personnel problem.

Failure 3 — The feedback loop nobody owns

A consistent 30% override rate on a specific case type means the model is wrong in that domain with high regularity. The correct response is structural: recalibrate the threshold, retrain the model, redesign the rule. The observed response, almost universally, is to absorb the overhead and move on. The feedback loop exists in the architecture it just doesn't operate in practice.

What a Closed-Loop Control System Actually Looks Like

The organizations generating real AI returns have built something structurally different. Whether they've named it this way or not, they've built closed-loop control systems architectures where uncertainty is managed rather than ignored, and where the system improves continuously from its own operational data.

Here's what that architecture looks like in practice:

1. Input & Confidence Scoring
Raw data enters. The model produces output and a calibrated confidence score. Uncertainty is highest here the system acknowledges this rather than suppressing it.

2. Decision Routing by Confidence + Risk Tier
High confidence + low risk → Auto-execute
Medium confidence or moderate risk → Human review
Low confidence or high risk → Hold / escalate.

3. Bounded, Auditable Action
Every decision executed with defined ownership. Confidence score, routing decision, and reviewer action all logged not just the outcome.

4. Outcome Tracking + Feedback Loop
Human corrections flow into retraining pipelines. Override patterns trigger threshold recalibration not queue management.

5. Drift Detection
Performance monitored continuously. Detected degradation triggers automatic adjustment before it causes outcome failure. The loop closes.

This isn't theoretical. It's the architecture every organization generating meaningful AI returns has built — most just haven't named it as a design principle.

AI Use Case Risk Profiles: Controls Scale With Consequence

Not every AI decision carries the same risk. The control requirements should match the consequence level not the model's confidence alone.

Fraud Detection: What the Two Architectures Actually Produce

Abstract architecture becomes concrete when you trace it through a real use case. Fraud detection exposes every failure mode at once.

In the standard pipeline deployment: a transaction is scored. High score triggers auto-block. Low score passes. No monitoring. No outcome tracking. No feedback.

Within weeks, two things happen: false positives accumulate silently, and fraudsters adapt to patterns the model wasn't trained on — novel attack vectors get low confidence scores and pass through undetected.

Both failures are architectural. A better model delays them. The same failures recur.

In a closed-loop system, novel attack vectors get flagged for human review based on low confidence, not auto-blocked or auto-passed. Override rates by fraud type feed back into threshold calibration. The model gets smarter because the system does.

The same logic applies to credit decisioning, insurance triage, HR screening anywhere AI handles high volume with variable exception rates.

What Architecture Is Needed to Scale AI Across an Enterprise

Scaling AI beyond a single use case requires four architectural layers most organizations lack:

A shared data and integration platform to avoid rebuilding pipelines for every new use case
Standardised confidence thresholding and routing logic configurable per use case, not hardcoded
An MLOps layer with model versioning, drift monitoring, and automated retraining triggers
An audit and governance layer that logs decisions with full context not just outcomes

Without these, every AI initiative stays one-off. With them, each deployment compounds the previous investment.

The Real Cost Is Never in the Deck That Gets Approved

Most AI business cases get approved on model performance which is the wrong number to optimize for.

The real cost infrastructure overhaul, compute, drift monitoring, retraining pipelines, and people who actually understand what they're reviewing rarely makes it into the same deck. So the ROI gap isn't surprising. The investment was undercounted from the start.

Then there's the people problem.

60% of organizations say AI literacy is their biggest scaling barrier. The humans assigned to oversee AI decisions often can't tell when something has gone wrong. Oversight exists on paper. In practice, it has no teeth.

Three metrics worth tracking instead of accuracy:

Q&A: What Engineers and Leaders Actually Ask

What's the difference between human-in-the-loop and human-on-the-loop?
HITL places a human between the model output and the action they must approve before anything executes. Human-on-the-loop means the system acts autonomously but a human monitors and can intervene. HITL gives stronger control; human-on-the-loop scales better but requires reliable drift detection to catch errors before they compound.

How do you set confidence thresholds without ground truth data?
Start with domain expert judgment for initial tiers, then calibrate empirically. Track override rates per confidence band if reviewers override 40% of "high confidence" decisions in a specific case type, the threshold is miscalibrated for that domain. Use those rates as recalibration signals, not anecdotes.

At what volume does HITL break down?
There's no universal number — it depends on decision complexity, reviewer expertise, and queue management. The signal to watch is reviewer throughput under load: when average review time drops sharply as queues grow, reviewers are heuristically clearing cases rather than genuinely evaluating them. That's the architectural ceiling.

Does closed-loop control require a full MLOps platform?
No. You can start with lightweight instrumentation: log confidence scores and outcomes, track override rates manually, and run threshold reviews quarterly. The architecture matters more than the tooling. A spreadsheet tracking overrides by case type is more valuable than a sophisticated platform that nobody queries.

How does EU AI Act Article 14 map to this architecture?
Article 14 mandates human oversight capability for high-risk AI systems the ability to understand, monitor, and intervene in AI outputs. A closed-loop system with tiered routing and full decision logging satisfies this structurally. A HITL layer bolted onto an autonomous pipeline satisfies it formally but often not functionally, because the override signals aren't acted on.

Three Verdicts

01 — Autonomous AI is not a production architecture.
The failure is structural, not algorithmic. A model operating without thresholding, routing, monitoring, and feedback has no mechanism for self-correction. Better models delay the failure. They don't prevent it.

02 — Human-in-the-loop is required, but it can't be the endpoint.
HITL provides accountability and an intervention point before errors propagate. At scale, it fails under automation bias, volume pressure, and the absence of feedback integration. Treating it as a permanent solution builds systems constrained by human bandwidth — not systems that improve.

03 — Closed-loop control is the engineering requirement.
Confidence thresholding, risk-tiered routing, structured escalation, continuous monitoring, feedback-integrated retraining, and drift detection. These are not operational add-ons. They are the product.

Every organization that has generated meaningful AI returns has, in practice, built this. Most haven't recognized it as the design principle it is. The ones who do are the 4%.

Everything else is a pilot waiting to fail.

About the Author:
Arleen Kaur writes about enterprise AI, system architecture, and the gap between AI pilots and production systems at Linksoft Technologies, a custom software development company.

Sources referenced:
McKinsey Global AI Survey, 2025
Partnership on AI — Enterprise Landscape Research
EU AI Act, Article 14 (Human Oversight Requirements)
Princeton / Georgia Tech GEO Study — Aggarwal et al., ACM KDD 2024

Related reading on linksft.com: