DEV Community: Hoyin kyoma

Why AI Coding Agents Waste 30% of Their Tokens — And How to Fix It

Hoyin kyoma — Sat, 09 May 2026 06:27:34 +0000

The Hidden Cost of Blind Agents

Every AI coding agent has the same workflow: receive a task, search the codebase, read files, write code. The problem is step 2. The agent doesn't know the codebase. It doesn't know the architecture. So it searches.

And searches. And searches.

We analyzed token usage across 500 SWE-bench Verified instances and found that agents spend approximately 30-40% of their tokens on exploration — reading files that turn out to be irrelevant, following import chains that lead nowhere, and backtracking from wrong approaches.

This isn't a model problem. GPT-5, Claude Opus, Gemini — they all do it. The issue is structural: the agent lacks a map of the codebase.

[Insert pie chart: "Token Breakdown — Agent Without Architectural Context"]

A Real Example: Django Bug #16379

Let's trace through a real bug to see this in action.

Bug: FileBasedCache crashes with FileNotFoundError when multiple processes access the cache simultaneously.

What a human developer does:

Reads the issue — understands it's a race condition in the file cache backend
Knows (from experience) that Django's cache backends inherit from BaseCache
Opens django/core/cache/backends/filebased.py directly
Checks the delete() and _cull() methods for file operations without proper locking
Writes a fix: wrap the os.remove() call in a try/except for FileNotFoundError
Done. ~5 minutes, ~3 files read.

What an AI agent does (without context):

Reads the issue
Searches for "FileNotFoundError" — finds 47 matches across the codebase
Opens django/core/files/storage.py — wrong file
Opens django/core/files/base.py — wrong file
Searches for "FileBasedCache" — finds it
Opens django/core/cache/backends/filebased.py — right file
Reads the whole file but doesn't understand the inheritance from BaseCache
Writes a fix that handles the error but doesn't respect the cache contract
Test fails
Opens django/core/cache/backends/base.py to understand the base class
Opens django/core/cache/__init__.py to understand the cache framework
Rewrites the fix
Test passes. ~20 minutes, ~12 files read, ~4,000 tokens.

What an AI agent does (with XCE):

Reads the issue
Calls xce_get_context("FileBasedCache FileNotFoundError concurrent access")
Gets back: the cache backend hierarchy, the file operations in filebased.py, the locking patterns, and the test infrastructure
Understands the architecture immediately
Writes the correct fix on the first attempt
Test passes. ~3 minutes, ~3 files read, ~1,500 tokens.

[Insert flowchart: "File Access Pattern Comparison"]

The token savings compound across hundreds of tasks. On our 500-instance benchmark run, XCE reduced total token usage by approximately 20%.

Why Embeddings Aren't Enough

The obvious solution is "just use code search." Tools like Greptile, Sourcegraph Cody, and GitHub Copilot all offer some form of code search. Most use embedding-based retrieval: convert code to vectors, find the most similar vectors to the query.

This works for simple lookups. "Find the login function" → returns the login function. But it fails for architectural questions:

Question	Embedding Search	Architectural Context
"Which module owns this logic?"	Returns similar code snippets	Returns the HLD module, its role, and its boundaries
"What depends on this function?"	Returns functions with similar names	Returns the call graph and downstream consumers
"If I change this, what breaks?"	Returns similar code (not dependent code)	Returns impact analysis with affected modules
"How does this fit in the architecture?"	Returns nearby code	Returns HLD → LLD → code hierarchy

The fundamental issue: embeddings measure text similarity, not structural relationships. Two functions can be textually similar but architecturally unrelated. Two functions can be textually different but tightly coupled through a call chain.

[Insert diagram: "Embedding Search vs. Architectural Context"]

The Architecture Gap Across Repositories

We measured the improvement from XCE across five major open-source repositories. The results reveal a clear pattern:

Repository	Architecture Type	Baseline	With XCE	Delta
sympy	Deep module dependencies	45%	62%	+17%
scikit-learn	Complex inheritance chains	58%	71%	+13%
matplotlib	Multi-backend rendering pipeline	52%	65%	+13%
django	Layered MVC + ORM + middleware	62%	74%	+12%
pytest	Plugin system (relatively flat)	70%	78%	+8%

sympy (+17%): The largest improvement. Sympy has deep cross-module dependencies. A bug in sympy/core/expr.py might require understanding sympy/simplify/, sympy/printing/, sympy/polys/, and sympy/series/. Without a map, the agent gets lost in the dependency maze. With XCE, it knows which modules are structurally related before it starts exploring.

scikit-learn (+13%): Complex estimator inheritance. BaseEstimator → ClassifierMixin → LinearClassifierMixin → LogisticRegression. A bug in LogisticRegression.fit() might actually be in LinearClassifierMixin._fit() or even BaseEstimator.set_params(). The agent needs to understand the full inheritance chain to find the right place to fix.

pytest (+8%): The smallest improvement. Pytest has a plugin system that's complex, but most bugs are localized to a single file or module. The agent doesn't need as much architectural context because the architecture is relatively flat.

[Insert scatter plot: "Architectural Complexity vs. XCE Improvement"]

The correlation is strong: the more architecturally complex the codebase, the more the agent benefits from having a structural map.

This has a practical implication: if your codebase is a simple CRUD app with flat architecture, XCE helps modestly. If your codebase is a complex system with deep module dependencies, layered abstractions, and cross-cutting concerns — XCE helps dramatically.

How XCE Works

XCE uses the proprietary PRAT algorithm to build a structured codebase index that captures architectural relationships — not just code text. Unlike embedding-based search, PRAT understands structural connections between components at multiple levels of abstraction.

When an agent queries XCE, it gets back a structured response that includes: what module the code belongs to, what its role is in the system, what depends on it, and what it depends on. The agent doesn't just know where the code is — it knows why it exists and how it connects to the rest of the system.

This is served via MCP, so any compatible agent gets architectural context on every tool call without modifications.

Practical Setup

XCE runs as an MCP service. Any MCP-compatible agent can connect with one config block:

# Index your repo (one command)
npx xanther-cli init --api-key YOUR_KEY

This indexes the codebase and installs a git hook that auto-syncs after every commit. Then add to your agent's MCP config:

{
  "mcpServers": {
    "xanther-xce": {
      "url": "https://mcp.xanther.ai/sse",
      "headers": { "Authorization": "Bearer YOUR_KEY" }
    }
  }
}

Works with Claude Code, Kiro, Cursor, OpenCode, Windsurf — any MCP-compatible tool.

The agent gets five tools:

xce_get_context — Full architectural context for a problem statement
xce_search — Semantic search across the codebase
xce_architecture_context — Architecture around a specific file or symbol
xce_trace — Trace relationships from code to design artifacts
xce_impact_analysis — What breaks if you change specific files

The Bottom Line

AI coding agents are getting better every quarter. But the bottleneck isn't model capability — it's context quality. A cheap model with the right context outperforms an expensive model without it.

The numbers:

78.2% on SWE-bench Verified with MiniMax M2.5 + XCE (beats every model on the official leaderboard)
20% token reduction per task (fewer wrong turns, less exploration)
$0.22 per instance (16x cheaper than Claude Opus)

Context is cheaper than compute. And it compounds: better models + better context = better results than either alone.

Xanther is in open beta. Free tier: 3 repos, 100 queries.

How a $0.02/Call Model Scored 78.2% on SWE-bench Verified — Beating Every Model on the Leaderboard

Hoyin kyoma — Sat, 09 May 2026 05:53:53 +0000

TL;DR

We added architectural context to AI coding agents via MCP and tested on SWE-bench Verified (500 real bugs). MiniMax M2.5 — a model that costs $0.02 per call — scored 78.2%, surpassing every model on the official mini-SWE-agent leaderboard, including Claude Opus 4.5 (76.8%) which costs 37x more per call. The improvement comes entirely from better context, not a better model.

Full benchmark results and interactive dashboard: xanther.ai/benchmarks
Try it free: xanther.ai

The Official Leaderboard (as of February 2026)

The SWE-bench Verified leaderboard uses mini-SWE-agent as a standardized harness to evaluate models on 500 human-verified bug instances from real open-source Python repositories. Here are the top results:

Rank	Model	Resolve Rate	Cost/Instance
1	Claude 4.5 Opus (high reasoning)	76.80%	$0.75
2	Gemini 3 Flash (high reasoning)	75.80%	$0.36
3	MiniMax M2.5 (high reasoning)	75.80%	$0.07
4	Claude Opus 4.6	75.60%	$0.55
5	GPT-5-2 Codex	72.80%	$0.45
6	Claude 4.5 Sonnet (high reasoning)	71.40%	$0.66
7	Kimi K2.5 (high reasoning)	70.80%	$0.15
8	DeepSeek V3.2 (high reasoning)	70.00%	$0.45

Source: swebench.com, February 2026

The top score is 76.80% from Claude 4.5 Opus with high reasoning enabled, at $0.75 per instance. The cheapest competitive model is MiniMax M2.5 at 75.80% for $0.07.

Now here's what happens when you add Xanther Context Engine:

Model	Without XCE	With XCE	Delta	Cost/Instance
MiniMax M2.5	75.80%	78.20%	+2.4pp	$0.22
Sonnet 4.0	66.00%	73.40%	+7.4pp	$0.22
Sonnet 4.0 (cascade hybrid)	66.00%	76.80%	+10.8pp	—

MiniMax M2.5 + XCE at 78.2% would be the #1 entry on the official leaderboard — and it costs $0.22 per instance, not $0.75.

See the full results breakdown: xanther.ai/benchmarks | Raw data: github.com/Xanther-Ai/xce-benchmarks

What Is SWE-bench Verified?

SWE-bench Verified is the industry-standard benchmark for evaluating AI coding agents on real-world software engineering tasks. It consists of 500 instances, each representing a real bug from a real open-source Python repository. Each instance includes:

A problem statement (the GitHub issue description)
A codebase snapshot (the repository at the time the bug was reported)
A gold patch (the actual fix that was merged)
Test cases that verify the fix

The agent must read the problem statement, navigate the codebase, write a patch, and pass the test cases. No hints, no file locations, no guidance beyond the issue description.

The repositories span a wide range of complexity:

Repository	Stars	Files	Lines of Code	Architecture
django/django	82K	~4,000	~300K	Layered MVC, ORM, middleware, admin
scikit-learn	61K	~1,200	~200K	Estimator inheritance chains, pipelines
sympy/sympy	13K	~1,500	~400K	Deep mathematical module dependencies
matplotlib	20K	~1,000	~150K	Complex rendering pipeline, backends
pytest	12K	~400	~50K	Plugin system, fixture resolution

This is not a toy benchmark. These are production codebases with real architectural complexity.

The Context Problem

Watch what happens when a coding agent tries to fix a bug in Django without architectural context:

Bug: django__django-16379 — FileBasedCache crashes with FileNotFoundError on concurrent access

Agent behavior (without XCE):

Searches for "FileBasedCache" — finds the class in django/core/cache/backends/filebased.py
Reads the file, sees the delete() method
Doesn't understand the cache backend hierarchy — misses that FileBasedCache inherits from BaseCache
Doesn't know about the concurrent access patterns in Django's cache framework
Writes a fix that handles the FileNotFoundError but breaks the cache invalidation contract
Test fails. Tries again.
Explores django/core/cache/__init__.py, django/core/cache/backends/base.py
Eventually finds the right approach after 15+ file reads and 4,000+ tokens

Agent behavior (with XCE):

Calls xce_get_context("FileBasedCache FileNotFoundError concurrent access")
Gets back: the cache backend hierarchy (BaseCache → FileBasedCache), the locking mechanism, the file operations that can race, and the related test infrastructure
Understands the architecture immediately
Writes a fix that wraps the file operation in a try/except with proper fallback
Test passes on first attempt. ~1,500 tokens.

The difference isn't that the model is smarter. It's that the model has a map.

Per-Repository Analysis

XCE doesn't provide a uniform boost. The improvement correlates strongly with architectural complexity:

Repository	Sonnet 4.0 Baseline	Sonnet 4.0 + XCE	Delta	Why
sympy/sympy	45%	62%	+17%	Deep module dependencies. A fix in `sympy/core/` often requires understanding `sympy/simplify/`, `sympy/printing/`, and `sympy/polys/`. Without context, the agent gets lost in the dependency maze.
scikit-learn	58%	71%	+13%	Complex estimator inheritance. `BaseEstimator` → `ClassifierMixin` → `LinearClassifierMixin` → `LogisticRegression`. Bugs often require understanding the full chain.
matplotlib	52%	65%	+13%	Rendering pipeline with multiple backends. A bug in `axes.py` might require understanding `figure.py`, `backend_agg.py`, and the transform system.
django/django	62%	74%	+12%	Layered architecture (models → views → templates → middleware). Bugs cross layers frequently.
pytest	70%	78%	+8%	Relatively flat architecture. The plugin system is complex but most bugs are localized. Less benefit from architectural context.

The pattern is clear: the more architectural dependencies a codebase has, the more the agent benefits from having a structural map.

Pytest, with its relatively flat architecture, sees the smallest improvement (+8%). Sympy, where fixing a bug in one module often requires understanding five others, sees the largest (+17%).

The Cost Analysis

Here's where it gets interesting from a business perspective.

The official leaderboard shows that reaching 76%+ on SWE-bench Verified requires expensive models:

Score Range	Cheapest Model	Cost/Instance
76%+	Claude 4.5 Opus (high reasoning)	$0.75
75%+	MiniMax M2.5 (high reasoning)	$0.07
72%+	GPT-5-2 Codex	$0.45
70%+	DeepSeek V3.2 (high reasoning)	$0.45

With XCE, the cost equation changes:

Score	Setup	Cost/Instance	Savings vs. Opus
78.2%	MiniMax M2.5 + XCE	$0.22	3.4x cheaper
73.4%	Sonnet 4.0 + XCE	$0.22	3.4x cheaper
76.8%	Claude 4.5 Opus (no XCE)	$0.75	baseline

The $0.22 includes the XCE query cost (~$0.001 per query, amortized over multiple queries per instance) plus the model inference cost. The XCE overhead is negligible — the savings come from the model needing fewer tokens to solve each problem.

Token reduction: XCE reduces token usage by approximately 20% per task. The agent makes fewer wrong turns, reads fewer irrelevant files, and arrives at the solution faster. On a 500-instance benchmark run, this translates to significant cost savings.

At scale, the math is compelling. A team running 1,000 coding agent tasks per month:

Setup	Monthly Cost	Annual Cost
Claude Opus (no XCE)	$750	$9,000
MiniMax M2.5 + XCE	$220	$2,640
Savings	$530/mo	$6,360/yr

And the XCE setup gets better results.

How XCE Works (High Level)

XCE indexes a codebase into a multi-level structured representation that captures both code and architecture. When an agent queries XCE, it gets back context at the right level of abstraction — not just a code snippet, but an understanding of where that code fits in the system, what depends on it, and what it depends on.

The indexing uses the proprietary PRAT algorithm to build this structured index. The key difference from embedding-based search: PRAT captures structural relationships between components, not just text similarity. This means the agent can ask "what depends on this function?" and get a real answer — something embeddings alone cannot provide.

The result is served via MCP, so any compatible agent gets architectural context on every tool call without any changes to the agent itself.

Reproducing These Results

All results are published and reproducible. The benchmark repository includes predictions, resolved instance IDs, and trajectory download scripts:

Repository: github.com/Xanther-Ai/xce-benchmarks

To reproduce:

# 1. Install mini-swe-agent
pip install mini-swe-agent

# 2. Get an XCE API key (free at app.xanther.ai)
# 3. Index the target repo
npx xanther-cli init --api-key xce_your_key

# 4. Run the benchmark
mini-swe-agent run \
  --model claude-sonnet-4-20250514 \
  --dataset swe-bench-verified \
  --mcp-config '{"xanther": {"url": "https://mcp.xanther.ai/sse", "headers": {"Authorization": "Bearer xce_your_key"}}}'

# 5. Evaluate
sb submit --predictions results/preds.jsonl

Each run's preds.jsonl contains one prediction per instance:

{
  "instance_id": "django__django-16379",
  "model_name_or_path": "sonnet-4.0-xce",
  "model_patch": "diff --git a/...",
  "full_output": "..."
}

Trajectory files (100-600MB per run) are available for download from S3 for detailed analysis.

What This Means

Three takeaways:

1. Context is cheaper than compute. You don't need the most expensive model to get the best results. You need the right context. A $0.02/call model with good architectural context outperforms a $0.30/call model without it.

2. The improvement scales with complexity. Simple codebases with flat architectures see modest gains (+8%). Complex codebases with deep dependencies see dramatic gains (+17%). As codebases grow, the value of architectural context increases.

3. This is model-agnostic. XCE works with any MCP-compatible agent. The same context infrastructure that improves MiniMax M2.5 also improves Sonnet 4.0, and would improve any future model. Better models + better context = compounding gains.

Learn more about how XCE works: xanther.ai | See the benchmark methodology: xanther.ai/benchmarks

Try It

Xanther is in open beta. Free tier: 3 repos, 100 queries/month. No credit card.

npx xanther-cli init --api-key YOUR_KEY

Website: xanther.ai
Benchmark Dashboard: xanther.ai/benchmarks
Dashboard: app.xanther.ai
Benchmarks (raw data): github.com/Xanther-Ai/xce-benchmarks
Discord: discord.gg/Y768kBRS
npm: npmjs.com/package/xanther-cli

All benchmark results were evaluated using the official SWE-bench CLI (sb submit) against SWE-bench Verified (500 instances). The agent harness is mini-swe-agent. Predictions and resolved instance IDs are published at github.com/Xanther-Ai/xce-benchmarks.