The outage did not announce itself. A small agent pipeline consumed a free model route with no SLA. At 03:20, responses slowed. Retries doubled. The agent re-queued the same prompt. The dashboard stayed green because the job still "ran."
Free compute is best-effort. That is not a defect; it is the contract. The endpoint is available when capacity exists. Teams usually remember the first half of the sentence and forget the second one.
This field guide adds a tripwire. It places a circuit gate between your agent and a best-effort model route. The gate turns hidden degradation into an explicit fallback.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. I use MonkeyCode's free model access and free server option in the examples because that free tier is what I can stress without a budget.
The retry storm is the real incident
A single timeout is cheap. The follower then multiplies it. Every retry occupies a thread. The retry budget drains. Third-party rate limits arrive late. The user sees hangs where they used to see tokens.
The primary failure is rarely interesting. The feedback loop is the interesting part:
- One request times out.
- The client retries three times.
- Three queued calls wait behind each retry.
- The agent spawns more tasks because nothing finished.
The circuit gate breaks that loop before it starts. It tracks recent failures and refuses to call a degrading route.
What a circuit gate does
The gate has three states:
- closed — calls go through normally.
- open — calls are rejected fast, and a fallback runs instead.
- half_open — after a cooloff, one probe call is let through.
If the probe succeeds, the gate closes again. If it fails, the gate stays open. The logic is small, stateful, and easy to reason about.
The artifact: circuit_gate.py
Here is the whole gate, runnable as a standalone module:
# circuit_gate.py
import time
from collections import deque
from dataclasses import dataclass
@dataclass
class GateConfig:
window_secs: float = 60.0 # failure sliding window
max_failures: int = 5 # trip after 5 failures
cooloff_secs: float = 30.0 # wait before probe
class CircuitGate:
def __init__(self, cfg: GateConfig = GateConfig()):
self.cfg = cfg
self.state = "closed"
self.failures: deque[float] = deque()
self.opened_at = 0.0
self.probe_sent = False
def allow(self) -> bool:
now = time.monotonic()
if self.state == "open":
if now - self.opened_at >= self.cfg.cooloff_secs:
self.state = "half_open"
self.probe_sent = False
else:
return False
if self.state == "half_open":
if not self.probe_sent:
self.probe_sent = True
return True
return False
while self.failures and now - self.failures[0] > self.cfg.window_secs:
self.failures.popleft()
return len(self.failures) < self.cfg.max_failures
def record(self, ok: bool) -> None:
if self.state == "closed":
if not ok:
self.failures.append(time.monotonic())
if len(self.failures) >= self.cfg.max_failures:
self._open()
return
self.probe_sent = False
if ok:
self._reset()
else:
self._open()
def _open(self) -> None:
self.state = "open"
self.opened_at = time.monotonic()
self.failures.clear()
def _reset(self) -> None:
self.state = "closed"
self.failures.clear()
self.opened_at = 0.0
The sliding window forgets old failures. The cooloff prevents a hot loop against a broken endpoint. The half-open state gives the route one honest chance to recover.
Where to place it
Call allow() before the remote request. Call record(ok) once the request settles. The remote call below is pseudocode; wire it to your actual provider client.
from circuit_gate import CircuitGate, GateConfig
gate = CircuitGate(GateConfig(max_failures=5, cooloff_secs=30.0))
def summarize(ticket_text):
if not gate.allow():
return local_summary(ticket_text) # cheap fallback
try:
result = free_model_route(ticket_text) # MonkeyCode free route
except TimeoutError:
gate.record(ok=False)
return local_summary(ticket_text)
gate.record(ok=True)
return result
The fallback can be a local model, a cached answer, or a queue. Keep it cheap and deterministic.
Try the gate locally
import random
import time
from circuit_gate import CircuitGate
gate = CircuitGate()
for _ in range(30):
ok = random.random() < 0.5
if gate.allow():
print(f"routed (ok={ok})")
gate.record(ok)
else:
print("open: fallback used")
time.sleep(0.1)
Run it with a 50% failure rate. Watch the gate trip, probe, and recover. Then change max_failures and observe the behavior change.
Decision table
| Signal | Action |
|---|---|
| Failures below threshold, latency stable | Keep using the free route |
| Gate trips several times a day | Treat it as a capacity signal; plan a paid or self-hosted route |
| Half-open probes keep failing | Disable the route until the provider recovers |
| Batch job must finish by a hard deadline | Do not put the deadline on best-effort compute |
When not to use this approach
A circuit gate is not a fix. It is a reaction layer. Skip it when the real requirement is upstream.
- Regulated data. A free server may process data outside your control. A gate does not change where bytes land.
- Constant high volume. The gate does not create capacity. It only fails fast.
- SLA-bound workflows. You need a contract, not a circuit breaker.
- Latency-critical UI. A fallback that takes seconds is still a bad experience.
Also avoid abusing the gate as a silent mask. Log every trip. Count probe failures. If the open state becomes normal, the route is dead.
Limitations
The gate is process-local. It knows nothing about other replicas. Twenty instances can each send a probe after a cooloff. Tune that behavior for your fan-out.
time.monotonic() is local to the process. Restarts reset the state. The gate is not a load test either. Run a separate probe to measure latency and throughput before you trust the numbers.
Bottom line
Free AI compute is a constraint. Code for that constraint explicitly. The exit must exist before the outage, not after it.
If you want a free model route plus a free server, MonkeyCode has both. Wire the gate before you point real traffic at them. The tripwire is cheap. The outage is not.
Top comments (0)