DEV Community

Charlie Hu
Charlie Hu

Posted on

Trip the Loop: A Weekend Circuit Breaker for Agent Side Projects

Weekend agent side projects rarely die from a missing API key. They die from a loop that keeps calling the same tool, eating turns, and leaving a terminal full of identical errors at 1 a.m. A small in-process circuit breaker — turn cap, identical-call tripwire, consecutive-error open state — is enough to stop that spiral without a platform rewrite.

The pattern below is a scoped weekend demo. It is not a distributed reliability product. It is a fence around a local agent loop so a side project can fail closed instead of fail forever.

The failure that actually shows up

A typical weekend agent looks innocent. One planner. A handful of tools. A while loop until the model says it is done. That loop has no natural end when the model repeats a broken tool call, when JSON parsing fails, or when a search tool returns empty and the planner retries the same query.

Unit tests on the tools do not catch this. The tools are fine. The composition is not. Green tool tests and a runaway planner can coexist for hours, because the suite never asserted that the loop had to stop.

Three trip conditions cover most of that weekend damage:

  • Turn budget. Hard stop after N planner steps.
  • Identical call. Open the circuit when the same tool name and payload digest appear K times in a row.
  • Error streak. Open the circuit after M consecutive tool exceptions.

A wall-clock deadline is a fourth optional fence. It is cheap. It is also easy to get wrong on a laptop that sleeps, so the demo keeps cooldown as a reject counter the tests can drive without time.sleep.

Scope cut for one weekend

The build log is the product. Scope was cut on purpose.

Shipped

  • In-process state machine: closed, open, half_open
  • Turn, identical-call, and error-streak trips
  • A half-open probe that allows one trial call after cooldown
  • A scripted agent driver and a unit test file that treats "opened" as success

Skipped

  • Redis or any shared store
  • Multi-process leases (liveness belongs to a heartbeat, not this fence)
  • Token spend accounting
  • Prompt or model lockfiles
  • A web dashboard
  • Live model HTTP calls inside the test suite

The skip list is the point. A weekend fence that needs a cluster is not a weekend fence.

Decision table

The breaker answers one question before every tool call: may this loop continue?

State Event Next state Action
closed turn below max, call is new, tool succeeds closed execute
closed identical payload repeated k times open reject
closed consecutive errors reach m open reject
closed turn reaches max_turns open reject
open cooldown not elapsed open reject
open cooldown elapsed half_open reject this call, arm one probe
half_open probe succeeds closed execute and reset counters
half_open probe fails open reject and restart cooldown

Cooldown in the demo is measured in rejected attempts, not wall time. That choice keeps tests deterministic. Wall time can replace the counter later. It should not be the first thing built.

Working demo

The module is a single file. Copy it into tripwire.py. Python 3.10+ is enough. No third-party packages.

# tripwire.py
# Weekend demo: in-process circuit breaker for a local agent loop.
# Proposed pattern, not a production reliability library.

from __future__ import annotations

from dataclasses import dataclass, field
from hashlib import sha256
from typing import Any, Callable, Literal

State = Literal["closed", "open", "half_open"]


def payload_digest(tool: str, payload: Any) -> str:
    raw = f"{tool}:{payload!r}".encode("utf-8")
    return sha256(raw).hexdigest()[:16]


@dataclass
class TripConfig:
    max_turns: int = 8
    identical_limit: int = 3
    error_limit: int = 3
    open_cooldown_rejects: int = 4


@dataclass
class Tripwire:
    config: TripConfig = field(default_factory=TripConfig)
    state: State = "closed"
    turns: int = 0
    identical_streak: int = 0
    error_streak: int = 0
    last_digest: str | None = None
    rejects_while_open: int = 0
    probe_armed: bool = False
    last_reason: str = "ok"

    def precheck(self, tool: str, payload: Any) -> None:
        digest = payload_digest(tool, payload)

        if self.state == "open":
            self.rejects_while_open += 1
            if self.rejects_while_open >= self.config.open_cooldown_rejects:
                self.state = "half_open"
                self.probe_armed = True
                self.rejects_while_open = 0
            self.last_reason = "circuit_open"
            raise RuntimeError("circuit_open")

        if self.state == "half_open":
            if not self.probe_armed:
                self.last_reason = "circuit_open"
                raise RuntimeError("circuit_open")
            self.probe_armed = False

        if self.turns >= self.config.max_turns:
            self._open("max_turns")

        if self.last_digest == digest:
            self.identical_streak += 1
        else:
            self.identical_streak = 1
            self.last_digest = digest

        if self.identical_streak >= self.config.identical_limit:
            self._open("identical_call")

    def record_success(self) -> None:
        self.turns += 1
        self.error_streak = 0
        if self.state == "half_open":
            self.state = "closed"
            self.identical_streak = 0
            self.probe_armed = False
        self.last_reason = "ok"

    def record_error(self, exc: BaseException) -> None:
        self.turns += 1
        self.error_streak += 1
        if self.state == "half_open" or self.error_streak >= self.config.error_limit:
            self._open("error_streak")
        self.last_reason = type(exc).__name__

    def _open(self, reason: str) -> None:
        self.state = "open"
        self.rejects_while_open = 0
        self.probe_armed = False
        self.last_reason = reason
        raise RuntimeError(reason)


def run_agent(
    steps: list[tuple[str, Any, bool]],
    breaker: Tripwire,
    tool_fn: Callable[[str, Any], str],
) -> list[str]:
    """Drive a scripted agent. Each step is (tool, payload, should_succeed)."""
    outputs: list[str] = []
    for tool, payload, should_succeed in steps:
        breaker.precheck(tool, payload)
        try:
            if not should_succeed:
                raise ValueError("tool_failed")
            result = tool_fn(tool, payload)
            breaker.record_success()
            outputs.append(result)
        except RuntimeError:
            raise
        except Exception as exc:
            breaker.record_error(exc)
            raise
    return outputs
Enter fullscreen mode Exit fullscreen mode

A tiny driver shows the fence without a model:

# demo_loop.py
from tripwire import TripConfig, Tripwire, run_agent


def echo(tool: str, payload: object) -> str:
    return f"{tool}:{payload}"


def main() -> None:
    breaker = Tripwire(TripConfig(max_turns=5, identical_limit=3, error_limit=3))
    script = [
        ("search", "circuit breaker agent loop", True),
        ("search", "circuit breaker agent loop", True),
        ("search", "circuit breaker agent loop", True),  # should trip
    ]
    try:
        print(run_agent(script, breaker, echo))
    except RuntimeError as err:
        print(f"tripped state={breaker.state} reason={err}")


if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

Run it:

python demo_loop.py
Enter fullscreen mode Exit fullscreen mode

Expected line:

tripped state=open reason=identical_call
Enter fullscreen mode Exit fullscreen mode

The third identical search never becomes a fourth attempt. That is the whole weekend win. The planner can still be wrong. It just cannot be wrong forever in the same way.

Tests that fail closed

Naive agent tests assert that a tool returned JSON. They do not assert that the loop stopped. The suite below treats "the breaker opened" as the successful outcome.

# test_tripwire.py
import unittest

from tripwire import TripConfig, Tripwire, run_agent


def echo(tool: str, payload: object) -> str:
    return f"{tool}:{payload}"


class TripwireTests(unittest.TestCase):
    def test_identical_call_opens(self) -> None:
        b = Tripwire(TripConfig(identical_limit=3, max_turns=20))
        steps = [("search", "same", True)] * 3
        with self.assertRaisesRegex(RuntimeError, "identical_call"):
            run_agent(steps, b, echo)
        self.assertEqual(b.state, "open")

    def test_error_streak_opens(self) -> None:
        b = Tripwire(TripConfig(error_limit=3, max_turns=20, identical_limit=9))
        steps = [("fetch", i, False) for i in range(3)]
        with self.assertRaisesRegex(RuntimeError, "error_streak"):
            run_agent(steps, b, echo)
        self.assertEqual(b.state, "open")

    def test_max_turns_opens(self) -> None:
        b = Tripwire(TripConfig(max_turns=2, identical_limit=9, error_limit=9))
        steps = [("think", "a", True), ("think", "b", True), ("think", "c", True)]
        with self.assertRaisesRegex(RuntimeError, "max_turns"):
            run_agent(steps, b, echo)

    def test_half_open_probe_closes(self) -> None:
        b = Tripwire(
            TripConfig(
                max_turns=50,
                identical_limit=9,
                error_limit=2,
                open_cooldown_rejects=2,
            )
        )
        with self.assertRaisesRegex(RuntimeError, "error_streak"):
            run_agent([("fetch", 1, False), ("fetch", 2, False)], b, echo)

        for _ in range(2):
            with self.assertRaisesRegex(RuntimeError, "circuit_open"):
                b.precheck("fetch", "probe-wait")

        b.precheck("fetch", "unique-ok")
        b.record_success()
        self.assertEqual(b.state, "closed")


if __name__ == "__main__":
    unittest.main()
Enter fullscreen mode Exit fullscreen mode
python -m unittest test_tripwire.py -v
Enter fullscreen mode Exit fullscreen mode

Four tests. No network. No fixtures that rot when a vendor changes a chat template. The assertions pin control-plane behavior, which is the part a weekend agent actually loses.

Identical-call hashing uses repr of the payload. That is good enough for strings, ints, and small tuples in a demo. It is a poor canonical form for unordered dicts, objects with changing ids, or floating timestamps injected by the planner. If those show up, normalize the payload before payload_digest, or the tripwire will never see a repeat.

Where a hosted planner fits

Local scripted steps are enough to prove the fence. A real planner still needs a model at some point. That is a different layer from the tripwire. Mixing them in the first weekend usually produces an integration test that fails for reasons that have nothing to do with the breaker.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

When the scripted loop is green and the next step is to point a planner at a live model, MonkeyCode's free model access and free server option are one way to host that planner without standing up a GPU box first. Keep the tripwire on the caller side, next to the tool calls, so an open circuit does not depend on a remote dashboard.

That split is the useful part. The fence is local, deterministic, and tested. The model is optional and replaceable.

What this will not do

The demo does not coordinate two processes. Two agent workers can still double-call a tool. It does not inspect tool side effects, so a breaker that opens after a successful destructive write is already too late. It does not prove the model is truthful. It only proves the loop stopped.

People who should not use this as-is:

  • Anyone running multi-tenant or billed agents. In-process state is not an isolation boundary.
  • Anyone who needs a hard real-time deadline. The cooldown is a reject counter, not a clock.
  • Anyone who already has a workflow engine with retries and sagas. Do not stack a second policy object on top without mapping states.
  • Anyone treating "circuit closed" as a safety sign-off for generated code or destructive tools.

A merge gate for generated diffs is a different weekend. A tool allowlist is a different weekend. This fence only bounds the loop.

Build log: what moved and what did not

The first cut tried to hash the entire conversation and trip on a repeated trace. That was too coarse. A planner can legally retry with a small payload change. Hashing the tool name plus payload, and only counting consecutive repeats, matched the failure that actually happened: the same search, three times, no new information.

The second cut wanted wall-clock timeouts. Tests then needed sleep and became flaky on a laptop that throttled. The reject-counter cooldown replaced it. Wall time can come back when the breaker is wired to a real scheduler.

The third cut wanted to log every trip to a JSONL file. Useful later. Not required to prove the state machine. The last_reason field is enough for a weekend demo and for the tests.

Half-open took longer than the closed/open pair. Allowing the cooldown-completing call to run as a probe mixed "I am still rejecting" with "I am testing recovery" in the same request. The demo now rejects the call that arms the probe, then allows exactly one later call. That is easier to test. It also matches how most human operators think about a cooling-off period: the last failure does not get a free retry.

What shipped is a tripwire a side project can paste, run, and keep. What did not ship is a platform. That is the correct ending for a Sunday night.

Top comments (0)