DEV Community

Cover image for We Ran 736 Tests Across 23 Models to See Why Agents Fail in Production
ARPA Hellenic Logical Systems profile image Ross Peili
Ross Peili for ARPA Hellenic Logical Systems

Posted on

We Ran 736 Tests Across 23 Models to See Why Agents Fail in Production

#ai #programming #python #machinelearning

The default architecture for autonomous AI agents in 2026 relies on a simple premise: give a frontier language model a tool execution sandbox (a bash shell or a Python REPL), feed it high-level objectives, and let it synthesize scripts on the fly. If the script throws a traceback, feed stderr back into the context window and let the model self-correct.

In simple demos, this looks effective. In production data pipelines—extracting base64 email attachments, cross-referencing corporate registry filings, calculating compound engineering units, or parsing financial transactions—this pattern degrades quickly.

When agents write ad-hoc Python to perform routine tasks, they encounter severe runtime impedance:

  1. The MIME and byte-parsing trap: Models consistently botch multipart MIME decoding, character encodings, and binary file writes when writing raw scripts, triggering recursive repair loops.
  2. Context poisoning: Each failed script attempt dumps stack traces, unformatted payloads, and apologetic re-explanations into the prompt context, degrading the model's reasoning window until it hits a terminal failure floor.
  3. Run-to-run non-determinism: The same prompt executed across identical seeds yields erratic execution paths, making CI/CD testing and regression gating impossible.
  4. Token exhaustion: Script synthesis burns thousands of output tokens per sub-task, consuming daily operational budgets within hours.

To measure these failure modes with empirical rigor, we built Skillware-Bench (GitHub Repository), an open-source evaluation harness that puts 23 language models through a 16-layer progressive ablation matrix across 736 live experimental test conditions.

We contrasted two fundamental execution paradigms:

  • Vanilla Baseline (Unassisted Code Generation): The model writes raw Python code inside a 30-second timeout sandbox, inspects error feedback, and attempts to repair its code.
  • Skillware Equipped (Governed Skill Execution): The model invokes pre-compiled, typed, deterministic Python skill bundles governed by immutable contracts (manifest.yaml), deterministic runtime effects (skill.py), and JIT schema validation.

Here is what 736 live trials across 11 model families revealed.

Grand Aggregate Results: 23 Models Across 736 Conditions

The benchmark evaluates 23 models spanning sovereign open weights (Qwen, Gemma, Llama, Nemotron, GLM, Hunyuan) and commercial frontier APIs (Claude, GPT, Gemini, Grok, DeepSeek).

Tasks are organized into four real-world enterprise crucibles across four progressive difficulty tiers:

  1. Compliance / PII Masking: Parsing nested records, stripping GDPR/HIPAA identifiers, and preventing prompt injection leaks.
  2. Office Mailbox Automation: IMAP traversal, query filtering, and binary MIME attachment extraction.
  3. Corporate Entity Verification: Resolving company registration numbers, officer registries, and insolvency records via statutory APIs.
  4. Multi-Skill Chained Pipelines: Sequential multi-step DAG execution combining PII masking, entity verification, and budget gating under strict token limits. 
+----------------------------------------------------------------------------------------------------+
| Metric                     | Vanilla Baseline (Raw Code) | Skillware Equipped (Governed Skills)     |
+----------------------------------------------------------------------------------------------------+
| Total Evaluated Conditions | 368 conditions              | 368 conditions (736 live runs total)    |
| Overall Fleet Pass Rate    | 300 / 368 (81.5%)           | 337 / 368 (91.6%) [+10.1% overall lift] |
| Total Tokens Consumed      | 3,087,129 tokens            | 1,500,967 tokens [-51.4% reduction]     |
| Cumulative Wall-Clock Time | 6,914.7 seconds (115.2 min) | 4,331.4 seconds (72.2 min) [1.60x speed]|
| Workspace Scripts Debugged | 464 failed repair scripts   | 0 scripts [100% debugging eliminated]    |
+----------------------------------------------------------------------------------------------------+
Enter fullscreen mode Exit fullscreen mode

Figure 25: Capability Compression Heatmap
Figure 25. Capability compression heatmap across all 23 evaluated models and 4 enterprise crucibles (736 live experimental conditions). Across the fleet, Skillware produces a +10.1% grand aggregate pass lift (from 81.5% to 91.6%). Uplift is heavily concentrated in complex multi-step workflows: Office Mail achieves a fleet lift of +22.8% (68.5% to 91.3%), and Companies House records +12.0% (75.0% to 87.0%). Meta Llama 4 Maverick gains +56.2% (jumping from 37.5% to 93.8%), effectively closing the capability gap with frontier models.

Finding 1: The Parameter Paradox

The most striking empirical outcome of the benchmark is what we term the Parameter Paradox:

A compact sovereign model (16B to 35B parameters) equipped with deterministic, typed skills consistently matches or outperforms unassisted 500B+ frontier models running raw code generation.

When frontier models like Claude Sonnet 5 or GPT-6 Luna operate without governed skills, they write ad-hoc Python to interact with OS primitives, parse headers, and decode binary buffers. They fail on edge cases, syntax nuances, and subprocess timeouts.

In contrast, compact sovereign models using Skillware bypass code synthesis entirely. The model only needs to generate a clean, typed JSON invocation matching the skill manifest. The actual execution runs in compiled, offline-tested Python on the host.

Figure 5: The Parameter Paradox
Figure 5. The empirical parameter paradox across three decades of parameter scale (8B to 550B+). An equipped Qwen 3.8 Flash (16B) or Gemma 4 (31B) achieves a perfect 16/16 (100.0%) pass rate, outscoring unequipped frontier models Claude Sonnet 5 and GPT-6 Luna (both 14/16, 87.5%) at roughly 1/30th the operational cost.

+-------------------------------------------------------------------------------------------------------+
| Model + Execution Architecture        | Pass Rate | Total Tokens | Wall Time | vs. Vanilla Baseline   |
+-------------------------------------------------------------------------------------------------------+
| Qwen 3.6 35B A3B + Skillware          | 16 / 16   | 69,603       | 224.0s    | +2 tasks vs Claude 5   |
| Google Gemma 4 31B IT + Skillware     | 16 / 16   | 43,033       | 123.7s    | +2 tasks vs Grok 4.7   |
| Qwen 3.8 Flash + Skillware            | 16 / 16   | 75,924       | 333.7s    | +1 task vs DeepSeek v4 |
| Meta Llama 4 Maverick + Skillware     | 15 / 16   | 54,862       | 197.9s    | +1 task vs Claude 5    |
| Anthropic Claude Sonnet 5 (Vanilla)   | 14 / 16   | 122,790      | 235.0s    | 43.3% more tokens      |
| xAI Grok 4.7 (Vanilla)                | 14 / 16   | 207,605      | 204.4s    | 79.3% more tokens      |
| DeepSeek v4 Pro (Vanilla)             | 15 / 16   | 151,103      | 561.9s    | 49.8% more tokens      |
+-------------------------------------------------------------------------------------------------------+
Enter fullscreen mode Exit fullscreen mode

Decoupling intelligence from execution means you no longer need a 500B parameter model just to reliably download an email attachment or format a table.

Figure 18: Equip-Any-Model Pass Rate Uplift
Figure 18. Pass rate trajectory under Skillware equipment across 14 representative models. Sovereign and compact workhorses achieve the steepest improvements when decoupled from raw code generation, with Meta Llama 4 Maverick jumping from 37.5% to 93.8% to achieve parity with frontier models.

Finding 2: The Debugging Hazard Curve and 464 Script Failures

Why does raw code synthesis fail so reliably in multi-turn environments?

In Vanilla mode, unassisted models across the fleet triggered 464 exploratory script debugging attempts. When a script failed on turn 1, the model attempted to fix it. We tracked the recovery rate across consecutive debug iterations:

Figure 22: The Debugging Hazard Curve
Figure 22. The debugging hazard curve across 368 unassisted vanilla conditions. While initial debugging yields high recovery rates (85% to 100% pass rate across attempts 1 to 4), continuing to debug past 7 attempts precipitates a steep collapse: pass rates plunge to 50% (attempt 8), 33% (attempt 9), and reach a 0% terminal failure floor at 12+ attempts as accumulated stack trace context poisoning exhausts model reasoning capacity.

The data exposes a dangerous cognitive trap:

  • Attempts 1 to 4: The model often catches simple typos, missing imports, or path errors (85% to 100% recovery).
  • Attempt 7: Recovery drops to 57%.
  • Attempt 8: Recovery drops to 50%.
  • Attempt 9: Recovery drops to 33%.
  • Attempt 12+: 0% recovery. The model enters an infinite loop of apologetic re-explanations, trying variations of broken code until hitting context or turn limits.

By contrast, Skillware eliminated 100% of workspace script debugging loops (from 464 to exactly 0). Because the tool handles the byte decoding and error states deterministically in host memory, the agent succeeds on attempt 0.

Figure 31: The 464 Script Retry Mass Distribution
Figure 31. The script retry mass distribution across 368 conditions per configuration. In unassisted vanilla execution, 151 conditions (41.0%) fail their initial attempt and enter script debugging loops, producing 464 total exploratory script retries with a long tail extending up to 13 attempts per task. Skillware-equipped agents execute with zero script synthesis overhead (0 script retries, 100% single-shot execution at attempt 0).

Finding 3: The MIME Trap and the Forensic Protocol Trace

To see how this looks in practice, examine the protocol trace of Office Mail Tier 2 (searching an inbox and extracting a binary agenda PDF from an email attachment):

Figure 19: Protocol Trace of One Call
Figure 19. Forensic message-level protocol trace of Office Mail Tier 2. In Vanilla mode, unassisted models attempt to synthesize ad-hoc Python scripts to interact with IMAP and decode binary MIME attachments; syntax errors, buffer truncations, and subprocess crashes trigger recursive debugging loops (averaging 9.2 turns, 4.7 script attempts, 41.8s latency, and 28,041 tokens). Equipped with Skillware, execution passes through host-side schema verification directly into deterministic JIT tool calls, completing the task cleanly in 3.7 turns, 0 script attempts, 11.7s latency, and 6,194 tokens (-77.9% token expenditure).

In Vanilla mode:

  1. The model writes a script importing imaplib and email.
  2. It hits a buffer truncation or decodes base64 payload into a string instead of raw bytes, corrupting the PDF header (%PDF-1.4).
  3. An error is returned: PDF syntax error: corrupt header.
  4. The model tries regex splitting on boundary strings.
  5. The regex fails on folded headers.
  6. The model tries writing temporary files with open(path, 'w') instead of 'wb', raising TypeError.
  7. Average cost for this single task: 9.2 conversational turns, 4.7 script attempts, 41.8 seconds of latency, and 28,041 tokens.

In Skillware Equipped mode:

  1. The model invokes office/gmail_handler with action search_messages.
  2. The skill returns structured JSON with message IDs.
  3. The model invokes action download_attachment with the attachment ID.
  4. Host-side Python streams the exact bytes directly to disk with cryptographic SHA-256 verification.
  5. Average cost: 3.7 turns, 0 script attempts, 11.7 seconds, and 6,194 tokens.

Finding 4: Token Ledgers and Context Excision

Where did the 1,586,162 saved tokens go? We decomposed the token ledger for each successful tool invocation:

Figure 20: Token Ledger Breakdown per Tool Call
Figure 20. Token ledger decomposition per successful tool call during Office Mail attachment extraction. Unassisted Vanilla execution incurs a severe token tax (5,520 tokens per call), driven by verbose schema and documentation prefill (1,840 tokens), monolithic tool preambles (620 tokens), subprocess retry overhead (940 tokens), and traceback repair loops (780 tokens). Skillware preserves task reasoning (1,340 tokens) while excising redundant documentation down to compact typed schemas (220 tokens, -88%) and eliminating code repair loops entirely (0 tokens, -100%), yielding a 70.6% reduction in per-call token expenditure (1,620 tokens total).

Figure 29: The Token Invoice
Figure 29. Itemized token invoice per tool invocation during attachment extraction. The vanilla schema overhead alone (2,310 tokens) exceeds the entire equipped execution turn (1,620 tokens).

Long-Horizon Context Bloat

When running continuous agent sessions (e.g. customer support, code refactoring, infrastructure monitoring), context accretion determines operational survival.

Extrapolating the 23-model telemetry over hundreds of turns demonstrates why unconstrained script synthesis crashes long-running agents:

Figure 24: Long-Horizon Context Bloat
Figure 24. Long-horizon context bloat trajectory over conversational turns. Unassisted vanilla code generation accretes context at ~1,050 tokens per turn as recursive failed script tracebacks and raw stdout/stderr dumps accumulate, precipitating context exhaustion (128k window breach) near turn 107. Equipped with Skillware, pre-compiled manifests and deterministic JIT execution eliminate script debugging bloat, accreting only ~58 tokens per turn and sustaining a 500-turn enterprise session at just 31.5k tokens (75.4% below the window ceiling).

Finding 5: Upfront Preflight Gates vs. Cascading Failure

In unconstrained environments, models frequently emit malformed parameters or violate API contracts. In Vanilla execution, 11.8% of emitted tool calls fail preflight validation, triggering 231 replanning cycles and 57 terminal aborts.

Figure 33: 368-Layer Call Survival Flow
Figure 33. Tool invocation survival flow normalized across 1,000 emitted calls per arm. In unassisted vanilla execution, 11.8% of emitted calls fail preflight schema validation (118 invalid calls), and while 231 calls recover through replanning loops, 57 calls abort completely. In contrast, Skillware preflight gates enforce typing upfront (99.6% valid), resolving edge cases via fast typed reflexes (52 repaired) to deliver 941 clean first-pass calls (94.1% vs 71.2% vanilla) with only 7 terminal aborts.

Finding 6: The Latency Continuum and Variance Contraction

Autonomous agent governance spans distinct clock domains:

  • Cryptographic hash chaining: 50 microseconds
  • Three-stage AST security firewall: 1.2 milliseconds
  • Hermetic sandbox simulation: 15 milliseconds
  • Deliberative model inference: 0.8s to 3.5s per turn

Figure 9: The Four-Scale Continuum of Latency
Figure 9. The four-scale continuum of execution latency (>24,000x dynamic range). Offloading security, verification, and simulation from model prompt context into sub-millisecond host micro-compute eliminates prompt token rent, enabling a -55.6% latency compression.

Figure 11: Latency Anatomy
Figure 11. Latency anatomy of a complex reference workflow (Office Mail Tier 2). JIT context excision shrinks prefill latency from 4.1s to 0.7s, compact model routing shortens decode from 6.2s to 3.4s, and deterministic host validation eliminates speculative error loops: end-to-end median latency falls from 14.2s to 6.3s (-55.6%). Time to first token (TTFT) collapses from 1.9s to 0.4s.

Run-to-Run Determinism

In enterprise testing, variance is a liability. Under 40 seeded replays with identical seeds:

  • Vanilla code synthesis: Mean score 61.0, standard deviation sigma = 4.1.
  • Skillware equipped: Mean score 88.2, standard deviation sigma = 0.8 (5.1x tighter dispersion).

Figure 14: Run-to-Run Determinism
Figure 14. Run-to-run determinism and variance contraction across 40 seeded replays under identical seeds (8B compact model). Skillware contracts run-to-run dispersion by 5.1x tighter, establishing the execution predictability required for enterprise regression testing and CI/CD release gating.

Finding 7: Enterprise Economics & Budget Survival

What does this mean for operating costs?

Figure 12: Cost-Capability Frontier
Figure 12. The cost-capability frontier across eight representative models. The Skillware frontier dominates the Vanilla fleet everywhere, shifting up (higher accuracy) and to the left (lower operational token cost per task).

When task volume scales, the compound effect of script debugging taxes, context bloat, and frontier API pricing creates a massive cost divergence:

Figure 15: Cumulative Spend and Break-Even
Figure 15. Cumulative enterprise spend against scaling task volume (log-log scale). Unassisted Frontier Vanilla APIs burn $15.58/task due to recursive multi-script debugging loops. Equipping models with Skillware compresses spend: Frontier APIs drop to $4.92/task (-68%), while local workhorses equipped with open-source Skillware Core achieve $0.42/task (-97.3% spend reduction).

Under a fixed daily operational ceiling ($100/day):

Figure 35: Token Budget Survival (Kaplan-Meier)
Figure 35. Token budget survival analysis under an empirical $100/day operating ceiling (Kaplan-Meier survival curve across 500 simulated enterprise agent deployments). Unassisted vanilla deployments exhaust the daily budget prematurely, with 50% of instances failing after only 3.7 tasks. Skillware delivers a median survival horizon of 238 tasks—a 64x throughput expansion before budget depletion.

How Skillware Works Architecturally

Skillware does not use a heavyweight daemon or external orchestrator. It is a lightweight, dependency-free Python framework that decomposes agent know-how into five first-class primitives:

skills/<category>/<skill_name>/
├── manifest.yaml        # Contract: typed parameters, permissions, outputs, constitution
├── skill.py             # Effect: deterministic, pre-compiled Python execution logic
├── instructions.md      # Directive: standard operating procedures & progressive disclosure
├── test_skill.py        # Assurance: hermetic pytest fixtures & invariants
└── card.json            # Presentation: UI card schema & inspector metadata
Enter fullscreen mode Exit fullscreen mode

Figure 2: Anatomy of a Skill Bundle
Figure 2. Anatomy of a Skillware skill bundle, exemplified by finance/uk_companies_house_handler. Know-how is decomposed into five first-class primitives rather than monolithic prompts.

Code Comparison: The Two Approaches

The Vanilla Approach (Fragile Script Generation) 

# The model receives bash or Python REPL tool
# System prompt: "Write Python scripts to inspect company registration 00002065"

# Turn 1: Model synthesizes ad-hoc request script
script = """
import urllib.request, json
req = urllib.request.Request('https://api.company-information.service.gov.uk/company/00002065')
# Forgot Basic Auth encoding...
resp = urllib.request.urlopen(req)
"""
# Output: HTTPError 401 Unauthorized

# Turn 2: Model attempts base64 auth repair
# Turn 3: Model tries handling pagination with while True loop
# Turn 4: Subprocess timeout or memory exhaustion
# Result: 4 turns, 8,200 tokens burned, failure likely
Enter fullscreen mode Exit fullscreen mode

The Skillware Approach (Governed Typed Skill) 

from skillware import SkillContext

# Initialize host session with progressive disclosure
ctx = SkillContext(skills=["finance/uk_companies_house_handler"])

# The model emits a typed JSON tool call; host executes deterministic Effect
result = ctx.execute("finance/uk_companies_house_handler", {
    "action": "resolve_company",
    "company_number": "00002065"
})

if result["status"] == "ready":
    print(result["company"]["company_name"])
    # Execution completes in 1 turn, 0 script attempts, exact schema guarantees
Enter fullscreen mode Exit fullscreen mode

Exploring the Benchmark Data

All raw telemetry, execution traces, harness scripts, and interactive charts are public and reproducible:

If you are building autonomous agents in production, stop forcing language models to write disposable Python scripts for routine work. Decouple model reasoning from deterministic execution.

Top comments (1)

Collapse
 
fintechbear profile image
Ekin Yegül •

I agree with the direction. Coming from payment systems, I want routine parsing, validation and execution to go through tested, versioned code. Having an agent rewrite those parts on every run adds uncertainty I would rather avoid.

A few questions from that perspective:

  1. What reusable libraries and tooling were available to the vanilla baseline? I’m curious how much of the improvement comes from reusing tested implementations, and how much comes from Skillware’s contracts and runtime validation.

  2. Could you share a breakdown of the remaining failures? In particular, were there cases where a call passed schema validation but selected the wrong action or supplied a valid yet incorrect parameter? That boundary between a technically valid call and the intended business operation interests me.

  3. How do the contracts handle a timeout after an external system has already performed an action? In payments, the next retry needs to distinguish “failed” from “completed, but the response was lost.” I’d be interested in how you represent that uncertainty and prevent duplicate effects.

  4. Have you measured the engineering effort required to build and maintain the skill bundles alongside the runtime savings, particularly when upstream APIs change?

These are the details I would look at when applying this approach to a payment workflow, where a reported failure can still have moved money.