DEV Community

Cover image for Your Agent Harness Is Paying LLM Prices for Classifier Decisions
Jayesh Tankariya
Jayesh Tankariya

Posted on Originally published at Medium

Your Agent Harness Is Paying LLM Prices for Classifier Decisions

#python #aiagents #langgraph #langchain

Your Agent Harness Is Paying LLM Prices for Classifier Decisions

Keep your LangGraph router. Let a decision model take the calls it's sure about, and prove the savings on your own traffic before you change anything.

The 60-Second Version

  • The Problem: Modern AI agents make dozens of small decisions on every run: which specialist agent handles this task, which tool to invoke, which model tier to use, or where to route next. Most teams make each of those decisions with a full frontier LLM call—paying LLM latency (800ms–3,000ms+) and LLM token prices.
  • The Reality: These decisions almost never generate new text. They select one option from a known, discrete list. They are classification problems masquerading as generative prompts.
  • The Approach: Put a fast System 1 decision model (TypeSafe's Jev) in front of the router you already have. When Jev is confident, it decides in ~100ms for a fraction of a cent. When it isn't, or if the API is ever unreachable, your existing LLM router decides exactly as it does today.
  • The Universal Scope: This isn't built for a single niche. It applies to any agentic architecture in any industry that uses LLM calls for control flow.
  • The Proof: A non-invasive shadow mode runs alongside your router without touching production behavior, measures agreement against your existing router, and recommends an optimal confidence threshold on your own live traffic.
  • To try it: pip install langgraph-jev-router and add one decorator (@jev_first) to your router function.

1. The Decisions Hiding Inside Your Agent Harness

The AI engineering community has settled on a fundamental architectural principle: a raw foundation model is not an agent; an agent is a model plus a harness.

The harness is everything surrounding the model: memory, state schemas, guardrails, tool bindings, and the control-flow graph that orchestrates what happens next.

Look closely at the control flow in a typical LangGraph workflow. Between an incoming user request and the final synthesized response, the graph evaluates multiple decision points:

  • Which specialist sub-agent should handle this intent? (Billing, Technical Support, Triage, DevOps, Sales)
  • Does this task require a heavyweight frontier model (Gemini 1.5 Pro / GPT-4o) or a lightweight worker (Flash / Haiku / 4o-mini)?
  • Which tool or MCP server should be called?
  • Does this input violate safety policies or require escalation?

Almost none of these choices produce free-form text. They evaluate input context and pick one discrete item from a known set of routes.

As analyzed in The Router Is the Harness, the decision layer of an agent is fundamentally a classification problem, yet production systems routinely pay generative LLM prices for it: roughly $0.000006 per decision for a typed classifier versus ~$0.02 for a frontier LLM call.

                           [ Incoming Request ]
                                    │
                                    ▼
                      ┌───────────────────────────┐
                      │   TypeSafe Jev Fast-Path  │
                      │   (~100ms System 1 reflex)│
                      └─────────────┬─────────────┘
                                    │
                 ┌──────────────────┴──────────────────┐
                 │                                     │
      Confidence ≥ Threshold                Confidence < Threshold
         (Clear intent)                     (Ambiguous / Edge case)
                 │                                     │
                 ▼                                     ▼
      ┌──────────────────────┐              ┌──────────────────────┐
      │ Direct Route Dispatch│              │ Existing LLM Router  │
      │ ⚡ ~100ms / No LLM   │              │ 🤖 Trusted Fallback  │
      └──────────────────────┘              └──────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Even when context caching shrinks token costs, the primary operational penalties remain: latency, jitter, and failure modes. A generative model requires multiple token-generation steps and is vulnerable to prompt drift, formatting hallucinations, rate limits (429), and upstream API outages. A typed classifier evaluates the input in a single pass, responds deterministically in ~100ms, and cannot be prompt-injected into an invalid route.

2. Why Not Just Replace Your LLM Router with a Classifier?

Because you would be trading a router you have painstakingly tuned, tested, and trust for a model you haven't validated on your data.

TypeSafe's Jev is a purpose-built decision model, but enterprise engineering teams should not blindly trust third-party confidence scores or unverified benchmarks with mission-critical routing.

The battle-tested architectural pattern is confidence-gated routing:

  • The classifier handles the high-confidence, unambiguous queries (the 60%–80% of routine traffic).
  • The existing LLM router acts as the authoritative fallback whenever the classifier is uncertain or ambiguous.

That pattern is exactly what langgraph-jev-router implements as a drop-in library for LangGraph.

Adopting it cannot make routing worse than it is today. If Jev has any doubt, or if the network blips, your trusted router executes exactly as it does right now.

3. Universal Applicability: An Agent Primitive, Not an Industry Tool

A common misunderstanding is asking: "Which industry is this built for?"

langgraph-jev-router is not an industry-specific application. It is an infrastructure primitive for agentic harnesses, akin to an HTTP connection pool or an edge-caching layer.

It applies to any agentic system in any domain where an LLM is currently used to decide control flow:

  • Customer Experience & Support: Routing conversations to billing, account management, tier-2 technical triage, or human agents.
  • Enterprise DevOps & SRE Agents: Dispatching incident alerts to database, networking, infrastructure, or application specialists.
  • Developer Tools & Coding Agents: Choosing between code generation, unit test creation, linting, or documentation modules.
  • Financial Services & FinTech: Categorizing transactions, compliance disputes, fraud verification, or credit requests.
  • Healthcare & Clinical Workflows: Triaging intake requests to cardiology, billing, appointment scheduling, or emergency escalation.
  • Industrial & Manufacturing Operations: Routing plant floor safety reports to mechanical, electrical, chemical, or safety officers.

The industry benchmarks we publish (manufacturing, supply chain, healthcare, telecom) are not boundaries of the tool—they are empirical stress tests on messy, real-world domain vocabularies to prove that the architecture holds up under enterprise conditions.

4. One Decorator, and Your Router Becomes the Safety Net

Integrating langgraph-jev-router requires adding a single decorator to your existing routing function.

from typing import Literal
from langchain_core.messages import HumanMessage, AIMessage
from langgraph.graph import StateGraph, START, END, MessagesState
from langgraph_jev_router import jev_first, attach_routes

# 1. Define your routing options with clear docstrings/descriptions
ROUTES = {
    "billing": "Invoices, credit card charges, payment failures, and refund requests.",
    "technical": "System errors, stack traces, API timeouts, and login failures.",
    "account": "Password resets, team invites, SSO, and user profile management.",
}

# 2. Add one decorator to your existing router
@jev_first(
    routes=ROUTES,
    threshold=0.75,  # Only take over when 75%+ confident
)
def route_ticket(state: MessagesState) -> Literal["billing", "technical", "account"]:
    # ── YOUR EXISTING LLM ROUTER (Unchanged, acts as 100% safety fallback) ──
    print("🤖 Ambiguous or low confidence: Falling back to LLM router...")
    # Your standard OpenAI / Gemini / Claude call runs here
    return "technical"

# 3. Build your LangGraph as usual
builder = StateGraph(MessagesState)

builder.add_node("billing", lambda s: {"messages": [AIMessage("Handling Billing")]})
builder.add_node("technical", lambda s: {"messages": [AIMessage("Handling Tech Issue")]})
builder.add_node("account", lambda s: {"messages": [AIMessage("Handling Account")]})

builder.add_conditional_edges(START, route_ticket)
for node in ["billing", "technical", "account"]:
    builder.add_edge(node, END)

# 4. Attach routes and compile
# attach_routes reads node docstrings or route dicts as descriptions
graph = attach_routes(builder).compile()
Enter fullscreen mode Exit fullscreen mode

When an incoming message arrives:

  1. Jev analyzes the request against the route descriptions.
  2. If confidence $\ge 0.75$, Jev dispatches directly to the destination node in ~100ms without invoking your LLM router.
  3. If confidence $< 0.75$, your existing LLM router function executes normally.

What if you route using prebuilt supervisors?

Many LangGraph developers use prebuilt abstractions like create_supervisor from langgraph-supervisor, where the routing function is compiled internally and cannot be decorated directly.

For these graphs, langgraph-jev-router provides install_shortcut:

from langgraph_jev_router import install_shortcut

# Intercept before the prebuilt supervisor node
handle = install_shortcut(workflow, before="supervisor")
app = workflow.compile()
Enter fullscreen mode Exit fullscreen mode

When Jev is confident, the request skips the supervisor's LLM call entirely and routes directly to the specialist worker node. When uncertain, control passes to the supervisor node unaltered.

5. Engineering Safety Defaults & The "Loop Trap"

Most of the engineering in langgraph-jev-router went into handling non-ideal edge cases gracefully:

Situation Runtime Behavior
Jev is confident ($\ge \text{threshold}$) Jev dispatches route; LLM router call is bypassed
Jev is below threshold Existing LLM router decides; zero disruption
Network timeout, 500 error, or no API key Existing LLM router decides; one warning logged; graph never crashes
Jev predicts route not in valid list Fallback to LLM router
Intermediate agent/tool spoke last LLM router decides (preserves multi-turn reasoning)
Terminal state (END) Never chosen by Jev unless explicitly opted in
Identical input arrives again In-memory cache returns route instantly (0ms)
Kill switch: JEV_ROUTER_DISABLED=1 Jev is completely disabled globally; 100% LLM execution

The "Loop Trap" Bug in Cyclic Graphs

One of the most critical bugs uncovered during testing occurs in cyclic multi-agent graphs:

[Router] ──> [Worker Agent] ──> [Tool Execution] ──> [Router] ──> ...
Enter fullscreen mode Exit fullscreen mode

In a loop-shaped graph, a worker agent finishes an action and hands control back to the router. A naive fast-path router inspecting only the original user request will see the initial prompt again and re-route right back to the same worker agent—creating an infinite execution loop.

At that stage in the conversation, the real question is no longer "Who should handle this request?" but rather "Is the task complete, or does it require another iteration?" Answering that requires analyzing the full conversation history—a task suited for the reasoning LLM.

To prevent this, langgraph-jev-router automatically inspects message sender history: Jev only activates on fresh user requests. Any internal loop or agent-to-agent handoff automatically routes to your full LLM.

6. Prove It on Your Traffic Before You Switch: Shadow Mode

Engineering leaders should never adopt routing changes based on vendor claims. You need to prove the behavior on your own production traffic.

langgraph-jev-router includes built-in Shadow Mode. In shadow mode, your existing LLM router makes 100% of the live routing decisions, while Jev executes asynchronously in parallel to observe, record, and evaluate agreement:

# Enable zero-risk shadow observation
export JEV_ROUTER_SHADOW=1
export JEV_ROUTER_LOG=jev_decisions.jsonl

# Run your normal traffic for a few days...
Enter fullscreen mode Exit fullscreen mode

Then generate an automated agreement and savings report using the CLI:

jev-router report jev_decisions.jsonl --llm-cost-per-call 0.002
Enter fullscreen mode Exit fullscreen mode

Output:

======================================================================
SHADOW MODE REPORT: 425 decisions compared
(Your existing router decided all traffic; Jev observed in parallel)
======================================================================
  Threshold | Jev Autonomous % | Agreement with LLM Router
     0.60   |   82.4% (n=350)  |   94.1%
     0.70   |   74.2% (n=315)  |   96.8%  <-- Recommended Threshold
     0.80   |   68.5% (n=291)  |   98.2%
     0.90   |   58.1% (n=247)  |   99.1%
----------------------------------------------------------------------
Recommendation: Set threshold=0.70 to eliminate ~74% of routing LLM calls
while maintaining 96.8% agreement with your current router.
Enter fullscreen mode Exit fullscreen mode

(Note: Agreement is measured against your current LLM router. When disagreement occurs, manual inspection often reveals the classifier chose the correct category while the LLM drifted.)

7. Empirical Validation: The Industrial Safety Benchmark

To measure langgraph-jev-router against real-world data containing specialized technical jargon, we built an open-source evaluation suite: jev-router-benchmark.

The Dataset

We evaluated 425 real industrial plant accident reports from the IHM Stefanini Industrial Safety and Health Analytics Database (CC0 1.0 Universal Public Domain). Incidents involve complex operational hazards: molten metal splashes, hydraulic press pinch points, high-voltage flashovers, and chemical inhalation.

We benchmarked a standard Google Gemini Flash router against langgraph-jev-router across 50 real manufacturing incidents.

Results at Default Threshold (0.70)

Metric Traditional LLM Router Jev-First (langgraph-jev-router) System Impact
LLM Calls Required 50 (100%) 13 (26%) -74.0% LLM calls eliminated
Autonomous Jev Decisions 0 (0%) 37 (74%) 37 calls routed via fast reflex
Decision Latency 305 ms (baseline) / 25s+ (API queue) 116 ms (TypeSafe Jev) Sub-120ms deterministic speed
Total Routing Cost $0.00750 $0.00290 61.3% net financial reduction

The Executive Trade-off: Threshold Calibration Curve

How does threshold selection affect safety and LLM call reduction? We executed a multi-threshold sweep from 0.60 to 0.95 across the incident dataset:

Confidence Threshold Calls Avoided (%) Autonomous Jev Decisions LLM Fallback Calls Agreement with Ground Truth Avg Latency
0.60 78.0% 39 / 50 11 / 50 56.0% 183 ms
0.70 74.0% 37 / 50 13 / 50 58.0% 82 ms
0.80 72.0% 36 / 50 14 / 50 60.0% 88 ms
0.85 64.0% 32 / 50 18 / 50 66.0% 112 ms
0.90 60.0% 30 / 50 20 / 50 70.0% 125 ms
0.95 56.0% 28 / 50 22 / 50 72.0% 137 ms

Architectural Takeaway:
Even if you configure an ultra-conservative threshold of 0.95, Jev still eliminates 56% of routing LLM calls at an average latency of 137ms. Engineering teams have a predictable knob to tune cost savings against accuracy preservation.

8. Where This Fits in the Ecosystem

Different tools address model routing at different layers of the AI stack:

  • Gateway Layer (e.g. LiteLLM Proxy): Dispatches requests across model providers or cost tiers before requests reach the application code.
  • Middleware Layer (e.g. LangChain create_agent middleware): Filters tool calls or model tiers during agent initialization.
  • Graph Orchestration Layer (langgraph-jev-router): Operates inside the state graph at conditional edges, supervisors, and handoff nodes, with zero-risk fallback to your application's custom LLM logic.

9. For Engineering Leaders: 5 Questions Before Approving a Pilot

If you are a CTO, VP of Engineering, or Principal Architect evaluating this architecture, here is the operational reality:

  1. What changes on Day 1? Nothing. With JEV_ROUTER_SHADOW=1, Jev runs asynchronously in parallel. Your current LLM router continues to make 100% of routing decisions.
  2. What is the rollback mechanism? Set JEV_ROUTER_DISABLED=1 in your environment variables and restart. The library completely disengages with zero code changes.
  3. What data leaves our perimeter? Only the newest user prompt and your route candidate names/descriptions are sent to TypeSafe's API endpoint. Message history is never logged or stored by default.
  4. Can vendor model updates silently drift our routing? No. The model version (jev-1-preview) is explicitly pinned by default.
  5. How do we quantify success? The CLI report provides exact numbers: LLM calls bypassed, fallback rates, agreement percentage, and net cost reduction.

10. What's Next in This Series

Empirical claims require open benchmarks. In the upcoming articles of this series, we examine full benchmark evaluations across diverse operational domains:

  1. Manufacturing (Published): Industrial plant safety incident classification (GitHub Benchmark).
  2. Supply Chain & Logistics: Real-time carrier exception and shipment routing.
  3. Healthcare Operations: Patient clinical intake and urgency triage.
  4. Telecommunications: Customer service intent dispatch and billing escalations.

All benchmark code, raw dataset references, and calibration scripts are publicly available and reproducible.

🛠️ Get Started

Install the library:

pip install langgraph-jev-router
Enter fullscreen mode Exit fullscreen mode

Verify your setup:

export TYPESAFE_API_KEY="your_api_key"
jev-router check
Enter fullscreen mode Exit fullscreen mode

Note: langgraph-jev-router is an independent open-source project released under the MIT license. Not affiliated with LangChain, Inc. or TypeSafe AI.

Top comments (0)