When Retries Make It Worse: Build a Circuit Breaker for Free Model Endpoints

Retries fix blips. They amplify outages. A circuit breaker stops that loop. This tutorial builds one from zero. You verify every stage with a mock endpoint.

Free model endpoints fail in patterns. Shared infrastructure means shared pain. Rate limits and provider outages arrive without warning. Your retry ladder handles the first blip. It cannot handle a ten-minute outage. Every retry burns quota and time. A breaker fast-fails instead.

This is the layer above the retry ladder. It tracks consecutive failures. It opens when the endpoint degrades. It probes before allowing traffic again. You can run the whole harness on a free server. MonkeyCode offers free model access and a free server. Use them as a concrete target if you want one. The code below only needs a URL. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why retries fail alone

A retry ladder assumes transient failure. It backs off, adds jitter, and tries again. That works for a 503 blip. It fails for a sustained outage. Every attempt costs quota. Every attempt adds latency. Users wait longer for the same error. The breaker inverts the logic. It stops trying when failure looks systemic. It reopens only after a cooldown proves the endpoint recovered.

What you will build

A small Node.js service with three parts.

1. A CircuitBreaker class with three states.
2. A callModel wrapper around any JSON endpoint.
3. A mock-based test that proves every transition.

No dependencies. Node 20 or newer. Total setup under five minutes.

How a circuit breaker thinks

Three states. Closed, open, half-open.

State	Meaning	Action
CLOSED	Endpoint looks healthy	Let calls through, count failures
OPEN	Too many failures	Reject instantly, start cooldown
HALF_OPEN	Cooldown passed	Allow exactly one probe

Closed counts failures. Reach the threshold, and the breaker opens. Open rejects every call without touching the network. After cooldown, one probe passes. Success closes the circuit. Failure reopens it.

Step 1: Pick your thresholds

Three numbers matter.

• failureThreshold: consecutive failures before opening
• cooldownMs: how long the circuit stays open
• timeoutMs: how long one call may run before it counts as a failure

No universal values exist. Start with 5 failures, 30 seconds, 5 seconds. Tune against your own traffic. Free endpoints need conservative values. Their latency spikes during load.

Step 2: Scaffold the project

Node 20 or newer. No dependencies needed.

mkdir breaker-demo
cd breaker-demo
npm init -y
npm pkg set type=module
node -e "console.log(process.version)"

The last command verifies your runtime. You should see v20 or higher. The type=module line enables import syntax.

Step 3: Write the breaker

Create breaker.js.

class CircuitBreaker {
  constructor({ failureThreshold = 5, cooldownMs = 30_000, timeoutMs = 5_000 }) {
    this.failureThreshold = failureThreshold;
    this.cooldownMs = cooldownMs;
    this.timeoutMs = timeoutMs;
    this.state = 'CLOSED';
    this.failures = 0;
    this.openedAt = 0;
  }

  async call(fn) {
    if (this.state === 'OPEN') {
      if (Date.now() - this.openedAt >= this.cooldownMs) {
        this.state = 'HALF_OPEN';
      } else {
        throw new Error('circuit open');
      }
    }

    try {
      const result = await Promise.race([
        fn(),
        new Promise((_, reject) =>
          setTimeout(() => reject(new Error('call timeout')), this.timeoutMs)
        )
      ]);
      this.onSuccess();
      return result;
    } catch (err) {
      this.onFailure();
      throw err;
    }
  }

  onSuccess() {
    this.failures = 0;
    if (this.state === 'HALF_OPEN') this.state = 'CLOSED';
  }

  onFailure() {
    this.failures += 1;
    if (this.state === 'HALF_OPEN' || this.failures >= this.failureThreshold) {
      this.state = 'OPEN';
      this.openedAt = Date.now();
    }
  }
}

export { CircuitBreaker };

Key details. The timeout uses Promise.race. A slow call counts as a failure. Half-open allows exactly one probe. One failure reopens the circuit.

Step 4: Wire it to a model endpoint

Create model.js.

import { CircuitBreaker } from './breaker.js';

export const breaker = new CircuitBreaker({
  failureThreshold: 5,
  cooldownMs: 30_000,
  timeoutMs: 5_000
});

export async function callModel(prompt) {
  return breaker.call(async () => {
    const res = await fetch(process.env.MODEL_URL, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ prompt })
    });
    if (!res.ok) throw new Error(`HTTP ${res.status}`);
    return res.json();
  });
}

Set MODEL_URL to any JSON model endpoint. The breaker wraps the network call. Non-2xx responses count as failures. So do timeouts.

Step 5: Verify every state

Create test.js. This is the reproducible artifact.

import { CircuitBreaker } from './breaker.js';

let failuresLeft = 0;

async function fakeModel() {
  if (failuresLeft > 0) {
    failuresLeft -= 1;
    throw new Error('upstream 503');
  }
  return { ok: true };
}

const breaker = new CircuitBreaker({ failureThreshold: 3, cooldownMs: 200 });

// Stage 1: starts closed
console.assert(breaker.state === 'CLOSED', 'starts closed');

// Stage 2: three failures open the circuit
failuresLeft = 3;
for (let i = 0; i < 3; i += 1) {
  try { await breaker.call(fakeModel); } catch (err) {}
}
console.assert(breaker.state === 'OPEN', 'opens after threshold');

// Stage 3: open circuit rejects without calling
failuresLeft = 0;
try { await breaker.call(fakeModel); } catch (err) {}
console.assert(breaker.state === 'OPEN', 'stays open during cooldown');

// Stage 4: cooldown passes, probe succeeds, circuit closes
await new Promise((resolve) => setTimeout(resolve, 250));
const result = await breaker.call(fakeModel);
console.assert(result.ok, 'probe succeeds');
console.assert(breaker.state === 'CLOSED', 'closes after probe');

console.log('all checks passed');

Run it.

node test.js

Expected output: all checks passed. The mock fails three times, then recovers. That exercises every branch. console.assert prints to stderr on failure. It does not throw. For CI, replace asserts with explicit throws.

Step 6: Deploy to a free server

The breaker belongs where your code runs. A free server works for low-traffic tools. Create health.js.

import { callModel, breaker } from './model.js';

setInterval(async () => {
  try {
    await callModel('ping');
    console.log(new Date().toISOString(), breaker.state, 'ok');
  } catch (err) {
    console.log(new Date().toISOString(), breaker.state, err.message);
  }
}, 60_000);

Start it with your endpoint.

MODEL_URL=https://your-endpoint.example/v1/complete node health.js

Keep it alive with a process manager. Or use nohup for a quick check.

nohup node health.js > health.log 2>&1 &

Verify with the log.

tail -f health.log

Watch one full cycle. Healthy calls show CLOSED ok. An outage shows OPEN circuit open. After cooldown, you see a probe. Recovery returns to CLOSED ok. That is your verification loop. A free server suits this workload. One request per minute. State lives in memory. No database required.

Tuning guide

Symptoms point to the wrong knob.

Symptom	Adjustment
Opens during normal traffic	Raise failureThreshold
Stays open too long	Lower cooldownMs
Probe calls hang	Lower timeoutMs
Slow to detect real outages	Lower failureThreshold

Change one value at a time. Re-run the mock test after every change.

Limitations

A breaker is a shield. It is not a fix.

• It does not reduce 429s. Rate-limit compliance still needs your retry ladder.
• It does not save quota. Half-open probes cost one real request per cooldown.
• Thresholds need tuning. Wrong values cause false opens or slow detection.
• It hides provider health. Log state transitions separately.
• One breaker per endpoint. Do not share one across different models.

Who should not use this

Skip this pattern in three cases.

• One-off scripts. A single run does not need state.
• Batch jobs with deferred work. Retry later, not faster.
• Hard SLAs. Free endpoints cannot guarantee uptime. Use a paid provider with a contract.

The sweet spot is small, always-on tools. Cron checks, bots, personal assistants. Those run long enough to hit outages. Those benefit from fast failure.

The harness is endpoint-agnostic. Point it at any JSON model endpoint. Start with the mock test. Let real traffic teach you the thresholds. Log every state transition. You will need that data next month.