DEV Community

Emery Yang
Emery Yang

Posted on

Bound the Tool Loop: A 90-Minute Agent Spike

#testing #python #ai #agents

Unbounded agent retries are a control defect, not extra effort. They waste rounds and hide a missing contract. A 90-minute spike can prove a loop cap works.

This article is a labeled spike protocol only. No live production run is claimed here.

The hypothesis

Write one hypothesis down before any model prompt. Keep it falsifiable and narrow.

If an agent retries tools without a hard round cap and a JSON contract, it emits a retry storm instead of a mergeable diff.

Ship-or-kill evidence must answer that sentence. Extra metrics are out of scope.

Why this spike exists

Public threads keep circling agent loops this week. Most takes stay qualitative and anecdotal. This spike stays mechanical and time-boxed.

Loop control fails in three observable ways:

  1. The agent repeats one tool with identical arguments.
  2. The agent treats malformed JSON as a retry signal.
  3. The agent calls a second, unrequested tool after failure.

None of those need a larger model. They need a budget and a contract.

Time box: 90 minutes

Do not extend the clock for polish. Kill the spike if the harness is incomplete.

Minute Work
0–10 Write hypothesis and kill criteria
10–25 Freeze one tool payload as a contract
25–55 Implement loop cap and duplicate detection
55–75 Add the ship-or-kill log format
75–90 Run four fixtures; record a verdict

A laptop runtime is enough for this protocol. A free remote workspace is enough too.

Fail closed as a state machine

Fail closed is a state machine, not a slogan. Name every terminal state.

After N tool rounds, the harness must stop. After a schema miss, the harness must stop. After a duplicate (name, args) pair, the harness must stop.

The agent does not receive a bonus retry comment. The log records KILL with a reason code.

Keep the code list tiny:

  • LOOP_CAP
  • SCHEMA_MISS
  • DUP_CALL
  • EMPTY_DIFF
  • SHIP

Five codes only. No free-text excuses in the verdict.

Artifact: loop-cap harness

The artifact is a small Python harness. It wraps a fake agent and a fake tool. Swap the fake agent later. Do not swap the contracts.

Label: this is an unexecuted example. Treat it as a fixture, not a benchmark.

# loop_cap_spike.py
# Unexecuted example for a 90-minute spike.
from __future__ import annotations

import json
import hashlib
from dataclasses import dataclass, field
from typing import Any, Callable

MAX_ROUNDS = 4
SHIP_CODES = {"SHIP"}

@dataclass
class SpikeLog:
    events: list[dict[str, Any]] = field(default_factory=list)

    def record(self, code: str, detail: str) -> None:
        self.events.append({"code": code, "detail": detail})

    def verdict(self) -> str:
        if not self.events:
            return "KILL"
        last = self.events[-1]["code"]
        return "SHIP" if last in SHIP_CODES else "KILL"


def fingerprint(name: str, args: dict[str, Any]) -> str:
    blob = json.dumps({"name": name, "args": args}, sort_keys=True)
    return hashlib.sha256(blob.encode()).hexdigest()[:16]


def validate_tool_payload(payload: Any) -> str | None:
    if not isinstance(payload, dict):
        return "SCHEMA_MISS"
    required = {"ok", "stdout", "exit_code"}
    if set(payload.keys()) != required:
        return "SCHEMA_MISS"
    if not isinstance(payload["ok"], bool):
        return "SCHEMA_MISS"
    if not isinstance(payload["stdout"], str):
        return "SCHEMA_MISS"
    if not isinstance(payload["exit_code"], int):
        return "SCHEMA_MISS"
    return None


def run_loop(
    agent: Callable[[list[dict[str, Any]]], dict[str, Any]],
    tool: Callable[[str, dict[str, Any]], dict[str, Any]],
    goal: str,
) -> SpikeLog:
    log = SpikeLog()
    seen: set[str] = set()
    trace: list[dict[str, Any]] = [{"role": "user", "goal": goal}]

    for round_id in range(1, MAX_ROUNDS + 1):
        action = agent(trace)
        kind = action.get("kind")

        if kind == "stop":
            diff = action.get("diff", "")
            if not isinstance(diff, str) or not diff.strip():
                log.record("EMPTY_DIFF", f"round={round_id}")
                return log
            log.record("SHIP", f"round={round_id}")
            return log

        if kind != "tool":
            log.record("SCHEMA_MISS", f"round={round_id} bad_action")
            return log

        name = action.get("name")
        args = action.get("args")
        if not isinstance(name, str) or not isinstance(args, dict):
            log.record("SCHEMA_MISS", f"round={round_id} bad_tool")
            return log

        fp = fingerprint(name, args)
        if fp in seen:
            log.record("DUP_CALL", f"round={round_id} name={name}")
            return log
        seen.add(fp)

        payload = tool(name, args)
        miss = validate_tool_payload(payload)
        if miss:
            log.record(miss, f"round={round_id} name={name}")
            return log

        trace.append(
            {"round": round_id, "tool": name, "args": args, "payload": payload}
        )

    log.record("LOOP_CAP", f"rounds={MAX_ROUNDS}")
    return log
Enter fullscreen mode Exit fullscreen mode

Reject extra keys on purpose. Agents invent fields under retry pressure. A loose parser turns invention into fuel.

Why exact fingerprints first

Start with canonical JSON and a hash. Do not fuzzy-match arguments in 90 minutes. Fuzzy matching is a second spike.

Exact match catches the common storm: same test command, same path, same flags. Reformatted args can bypass this check. Record that limit in the spike note.

Round caps beat wall-clock caps here. Clock caps punish slow tools. Round caps punish uncontrolled planning.

Fixture suite, not vibes

Do not treat a chat transcript as proof. Run four fixtures with pytest.

# test_loop_cap_spike.py
# Unexecuted example.

def agent_retry_storm(trace):
    return {"kind": "tool", "name": "run_tests", "args": {"cmd": "pytest -q"}}


def agent_stops_with_diff(trace):
    if len(trace) == 1:
        return {"kind": "tool", "name": "run_tests", "args": {"cmd": "pytest -q"}}
    return {"kind": "stop", "diff": "--- a/x.py\n+++ b/x.py\n"}


def tool_ok(name, args):
    return {"ok": True, "stdout": "1 passed", "exit_code": 0}


def tool_extra_field(name, args):
    return {
        "ok": True,
        "stdout": "1 passed",
        "exit_code": 0,
        "hint": "retry",
    }


def test_duplicate_call_kills():
    log = run_loop(agent_retry_storm, tool_ok, "make tests pass")
    assert log.verdict() == "KILL"
    assert log.events[-1]["code"] == "DUP_CALL"


def test_schema_miss_kills():
    log = run_loop(agent_stops_with_diff, tool_extra_field, "make tests pass")
    assert log.verdict() == "KILL"
    assert log.events[-1]["code"] == "SCHEMA_MISS"


def test_loop_cap_kills():
    def agent_unique_retries(trace):
        n = sum(1 for item in trace if "tool" in item) + 1
        return {
            "kind": "tool",
            "name": "run_tests",
            "args": {"cmd": f"pytest -q {n}"},
        }

    log = run_loop(agent_unique_retries, tool_ok, "make tests pass")
    assert log.verdict() == "KILL"
    assert log.events[-1]["code"] == "LOOP_CAP"


def test_clean_stop_ships():
    log = run_loop(agent_stops_with_diff, tool_ok, "make tests pass")
    assert log.verdict() == "SHIP"
Enter fullscreen mode Exit fullscreen mode

Commands for the last fifteen minutes:

python -m pip install pytest
python -m pytest -q test_loop_cap_spike.py
python -c "from loop_cap_spike import SpikeLog; print('harness import ok')"
Enter fullscreen mode Exit fullscreen mode

Honest expected shape:

  • three KILL fixtures pass
  • one SHIP fixture passes
  • no extra test file appears in git status

If pytest cannot run inside the time box, kill the spike. Missing tooling is a kill, not a delay.

Decision table

Use this table as the only merge gate. Do not add a sixth row during the spike.

Observation Code Action
Same tool name and args twice DUP_CALL Kill. Do not prompt again.
Tool JSON has extra or missing keys SCHEMA_MISS Kill. Do not coerce types.
Round count exceeds MAX_ROUNDS LOOP_CAP Kill. Do not raise N.
Stop action with empty diff EMPTY_DIFF Kill. No explanation patch.
Stop action with non-empty diff SHIP Keep the diff. Review by hand.

Scope creep is how ninety minutes becomes three hours. A new row is a new spike.

Where a free coding server fits

This harness is boring on purpose. It needs a Python runtime and a log file. It does not need a GPU narrative.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source coding workspace. Operator-supplied facts used here: free model access, plus a free server option. Those facts explain how the spike can leave a laptop. They are not quality claims, and they are not benchmarks.

Keep the loop cap in the repo. Do not let the model own the cap. The cap is code. The model is a guest process.

A concrete check: run the four fixtures on that free server. Download the JSON event log. Compare each last code to the decision table. If the log and the table disagree, the verdict is KILL.

No model names appear here. No quota numbers appear here. Those values move. The contracts should not.

Ship-or-kill rule

At minute 90, pick one verdict. Do not split the difference.

  • SHIP if all four fixtures pass and logs use only the five codes.
  • KILL if any fixture is skipped, patched, or explained away.

Partial harnesses teach the wrong lesson. Agents already almost-work in demos.

Write one note and stop:

HYPOTHESIS: unbounded retries hide missing contracts
EVIDENCE: pytest four fixtures
VERDICT: SHIP|KILL
MAX_ROUNDS: 4
DUP_MATCH: exact json hash
Enter fullscreen mode Exit fullscreen mode

Commit that note with the harness. Or delete the branch.

What this does not prove

This spike does not prove agents cannot write useful code. It does not prove a loop cap raises patch quality. It does not measure tokens, latency, or review time.

It proves one control property only. The session can stop for a named reason.

Limitations:

  • MAX_ROUNDS = 4 is a fixture choice, not a law.
  • The schema covers one tool shape only.
  • Duplicate detection is exact-match on canonical JSON.
  • The fake agent is not a live model. Live models drift.
  • Fail closed can block a later valid tool with new arguments.
  • The harness does not inspect the semantic quality of a SHIP diff.

Who should not use this approach:

  • Teams with no test runner in the agent environment.
  • Workflows that require long research loops by design.
  • Anyone replacing a state machine with a larger context window.
  • Production incident response. This is a spike, not a pager playbook.

Adjacent defects to ignore today

Invented files, invented tests, and invented tickets are different defects. This account already covered invented context in a prior spike. Do not reopen that thesis here.

Auth shortcuts and health-check theater are also different defects. Leave them on their own branches. This spike is the tool loop only.

Close

Retry storms are observable in logs. Name them with small reason codes. Cap the loop in versioned code. Validate tool JSON before the next planning step.

If a free server is already available, run the four fixtures there and keep the kill log beside the diff.

Top comments (0)