Test Agent Tool-Boundary Failures on a Free Tier Before Production

The alert that started this

Two weeks ago our coding-agent workers started returning 200s with empty diffs. No error logs. Queue depth fine. Latency fine. The SLO dashboard was green while every single job silently produced nothing.

The root cause was boring: one MCP tool endpoint started returning {"error": "rate_limited"} with HTTP 200, and our agent loop treated any 200 as a successful tool call. The model then hallucinated around the missing output instead of failing the job. Nobody had ever tested what the agent does when a tool lies to it.

This post is the drill I should have run before that incident: a local fault-injection harness for agent tool-boundary failures, runnable on free infrastructure so it costs nothing to keep in CI.

Why a free tier is the right place for this

Failure drills against paid production model endpoints have three problems: they burn quota, they pollute production telemetry, and people quietly disable them when the invoice arrives. A drill you can't afford to run is a drill you don't have.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode currently offers free model access and a free server option, which is enough for this workload because the drill's cost driver is number of tool calls under fault injection, not model quality. You are testing your orchestration code — timeouts, retries, validation, circuit breaking — not the model's reasoning. A smaller free model is arguably better here: if your harness handles a weaker model's messier tool-call formatting, it will handle a stronger model's. If you're evaluating free options for this kind of pre-production gate, the free tier at monkeycode.dev is one place to start; the harness below is provider-agnostic, so swap the endpoint for whatever you already run.

Topology

┌────────────┐   tool calls   ┌──────────────────┐
│ Agent loop │ ─────────────▶ │ Fault-injecting  │ ──▶ real tool (echo/fs mock)
│ (harness)  │ ◀───────────── │ tool proxy       │
└─────┬──────┘                └──────────────────┘
      │ model API
      ▼
┌──────────────────┐
│ Free model       │
│ endpoint         │
└──────────────────┘

The proxy sits between the agent and its tools and injects declared faults. Nothing touches production.

Declared workload

• 1 agent loop, 3 tools: read_file, write_file, run_tests
• 40 jobs per fault mode, each job = one small edit-and-test task
• Model: whatever the free tier serves; temperature 0
• Timeout budget: 30s per tool call, 5 min per job
• Success criterion: no job reports success without verified tool output

The fault proxy

# fault_proxy.py — sits between agent loop and tool implementations
import random, time, json

FAULTS = {
    "none":            lambda r: r,
    "http200_error":   lambda r: {"error": "rate_limited"},        # the incident
    "empty_body":      lambda r: {},
    "slow":            lambda r: (time.sleep(45), r)[1],             # exceeds 30s timeout
    "truncated_json":  lambda r: json.dumps(r)[:12],                 # unparseable
    "wrong_schema":    lambda r: {"result": None, "extra": "x"},
}

class ToolProxy:
    def __init__(self, tools, fault="none", seed=0):
        self.tools, self.fault = tools, fault
        random.seed(seed)

    def call(self, name, args):
        raw = self.tools[name](**args)
        return FAULTS[self.fault](raw)

The harness assertions (the part that matters)

The drill is not "run the agent with broken tools." The drill is asserting that your orchestration layer detects each failure mode:

# drill.py — expected behavior under each fault
EXPECT = {
    "http200_error":  "job_failed_tool_error",    # must NOT be treated as success
    "empty_body":     "job_failed_validation",
    "slow":           "job_failed_timeout",
    "truncated_json": "job_failed_parse",
    "wrong_schema":   "job_failed_validation",
}

def run_drill(agent, fault, jobs=40):
    outcomes = {"verified_success": 0, "silent_success": 0, "correct_failure": 0}
    for _ in range(jobs):
        result = agent.run_job(fault=fault)
        if result.ok and result.tool_outputs_verified:
            outcomes["verified_success"] += 1
        elif result.ok:                      # 200 + garbage = the incident class
            outcomes["silent_success"] += 1
        elif result.failure_class == EXPECT.get(fault):
            outcomes["correct_failure"] += 1
    return outcomes

The gate is one line: silent_success must be 0 under every fault mode. Everything else is a tuning detail.

The validation shim that fixed our incident

def call_tool_verified(proxy, name, args, timeout=30):
    try:
        resp = proxy.call(name, args)          # raises on timeout
    except TimeoutError:
        return ToolResult.fail("timeout")
    if isinstance(resp, str):
        try:
            resp = json.loads(resp)
        except json.JSONDecodeError:
            return ToolResult.fail("parse")
    if "error" in resp:
        return ToolResult.fail("tool_error", detail=resp["error"])
    if not SCHEMAS[name].is_valid(resp):
        return ToolResult.fail("validation")
    return ToolResult.ok(resp)

Four checks: timeout, parse, error-field, schema. That is the entire fix. The drill exists to prove the shim stays wired in.

Telemetry fields to emit per job

Log these so a regression in CI is debuggable:

job_id, fault_mode, tool_name, latency_ms, http_status,
parse_ok, schema_ok, error_field_present, outcome_class, verified

outcome_class=silent_success with http_status=200 is the exact signature of our original incident.

Expected output (labeled as expected, run locally to confirm)

fault=http200_error  verified=0  silent=0  correct_failure=40  PASS
fault=empty_body     verified=0  silent=0  correct_failure=40  PASS
fault=slow           verified=0  silent=0  correct_failure=40  PASS
fault=truncated_json verified=0  silent=0  correct_failure=40  PASS
fault=wrong_schema   verified=0  silent=0  correct_failure=40  PASS
fault=none           verified=40 silent=0  correct_failure=0   PASS

Your numbers will differ with a different free model — especially under truncated_json, where weaker models sometimes retry creatively. That variance is fine; the only non-negotiable column is silent=0.

Rollback and cleanup

• The drill runs against mocks and a free endpoint; rollback = delete the CI job, nothing else exists in production.
• Keep the validation shim and the drill in the same repo and same PR process. A shim without its drill rots.
• If you point this at a shared free server, add a concurrency cap of 1–2; free tiers are shared capacity and a fault-injection loop is exactly the kind of workload that gets you rate-limited.

Limitations and who should not do this

• A free model is not a stand-in for your production model's behavior. This drill validates orchestration, not output quality. Quality evals need your real model and real eval data.
• Free tiers change. Pin the harness so a provider-side change fails loudly in CI rather than silently skipping the drill.
• If your agent's tools have side effects on shared systems (deploys, ticket writes), mock them — fault injection against real side-effecting tools is how drills become incidents.
• Don't run this as your only safety net. Admission control on queue age and deadline slack (which I've written about before) stops overload; this drill stops silent wrongness. You need both.

The operational takeaway

Our incident wasn't a model failure. It was a missing four-line validation shim that nobody tested because testing it "cost money." Moving the drill onto free infrastructure removed the excuse. The question I'd ask any team running agents in production: under a tool that returns HTTP 200 with an error body, does your pipeline produce zero silent successes — and can you prove it in CI this week?