Put a Token Bucket Between Your CI Jobs and the Model Endpoint

If two or three CI jobs share one model endpoint, the failure you notice first is not a bad answer. It is a 429 that your retry loop converts into three slow jobs, one starved job, and a queue that reads like flakiness. Admission control fixes that. More retries make it worse.

This post builds a small gate you can run in front of a shared endpoint, a decision table for what to retry, and a local harness that reproduces the collision without spending a single token in CI.

The shape of the failure

A retry loop assumes the bottleneck is transient. With a shared endpoint, the bottleneck is your own concurrency. Job A retries, so it holds its slot longer; job B retries, so it overlaps A's second attempt; job C waits on a runner that is now billed for idle time. Nobody's prompt changed, but the pipeline got slower and less predictable.

Free tiers make this shape common because capacity is shared rather than reserved. That is a fair trade for prototyping, and a bad one for an unguarded merge path.

MonkeyCode is an open-source project that offers free model access and a free server option, which is exactly the situation where a gate earns its keep. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat the free allowance as a budget to be spent deliberately, not as an unmetered pipe. Check the project page for the current allowance and terms before you size anything below — free-tier numbers change, and a stale figure in a comment block is worse than no figure.

What to measure before you build anything

You need two numbers: how many model calls a pipeline run makes, and how many of those came back rate-limited. Both are already in your logs.

# calls per run
gh run view "$RUN_ID" --log | grep -c 'model_call_ok'

# rate-limited responses in the same run
gh run view "$RUN_ID" --log | grep -c 'status=429'

# how long jobs waited on the endpoint
grep 'model_wait_seconds' ci.log | awk -F= '{s+=$2} END {print s"s total wait"}'

Write the three numbers into the repository, next to the gate. If rate-limited responses are zero across a week of runs, stop here — you do not need this. If they are above zero, the ratio of wait time to wall-clock time tells you whether to raise capacity or lower concurrency.

Step 1 — One bucket, one lock file

One bucket for the whole pipeline, not one per job. A per-job bucket multiplies your concurrency by the number of jobs, which is the bug you were trying to fix.

# ci_gate/bucket.py
import fcntl, json, os, random, time
from dataclasses import dataclass

@dataclass
class Bucket:
    path: str
    capacity: int = 6          # in-flight calls you allow
    refill_per_sec: float = 0.2  # recovered calls per second

    def _load(self):
        try:
            with open(self.path) as f:
                return json.load(f)
        except FileNotFoundError:
            return {"tokens": float(self.capacity), "ts": time.time()}

    def _write(self, state):
        tmp = self.path + ".tmp"
        with open(tmp, "w") as f:
            json.dump(state, f)
        os.replace(tmp, self.path)

    def acquire(self, timeout=90.0):
        deadline = time.time() + timeout
        while True:
            with open(self.path + ".lock", "w") as lk:
                fcntl.flock(lk, fcntl.LOCK_EX)
                state = self._load()
                now = time.time()
                tokens = min(
                    self.capacity,
                    state["tokens"] + (now - state["ts"]) * self.refill_per_sec,
                )
                if tokens >= 1:
                    self._write({"tokens": tokens - 1, "ts": now})
                    return True
                self._write({"tokens": tokens, "ts": now})
                wait = (1 - tokens) / self.refill_per_sec
            if time.time() > deadline:
                return False
            time.sleep(min(wait, 5.0) * (1 + random.random() * 0.3))

Two properties matter here. The lock makes the read-modify-write atomic, and the timeout means a job fails loudly instead of hanging until the workflow timeout kills it. The default numbers are placeholders — set capacity from your measured concurrency, not from optimism.

Step 2 — Separate admission from retry

Admission asks may I call now. Retry asks should I call again. Mixing them produces the loop that starves your other jobs.

# ci_gate/call.py
import time
from .bucket import Bucket

BUCKET = Bucket(path="/var/ci/model_bucket.json", capacity=6, refill_per_sec=0.2)

def call_model(payload, attempts=3):
    for i in range(attempts):
        if not BUCKET.acquire(timeout=90):
            raise SystemExit("admission timeout: not calling, not retrying")
        resp = post(payload)
        if resp.status_code == 200:
            return resp.json()
        if resp.status_code != 429:
            raise RuntimeError(f"non-retryable status={resp.status_code}")
        time.sleep(min(2 ** i, 20) * (1 + random.random()))
    raise RuntimeError("exhausted attempts after admission")

Every attempt re-enters the bucket, so a retry costs a token like any other call. If you skip that, retries bypass the only mechanism you built.

Step 3 — Classify before you retry

The gate stops the pile-up. The table stops you from retrying things that cannot succeed.

Signal	Retry?	Where the fix actually belongs
429 with Retry-After	Yes, after admission	Honour the header; lower capacity
429 with no header	Yes, capped at 3, jittered	Your capacity is above real capacity
500 / 502 / 503	Once	Endpoint health, not your prompt
400 / 422	Never	Prompt or fixture is wrong
Timeout after admission	Once, and log the duration	Your latency budget, not the bucket
Output differs from fixture	Never	This is a test failure, not infra

That last row is the one teams get wrong. A model answer that no longer matches a recorded fixture is a signal about the model or your prompt, and retrying it politely until it agrees is how a green check stops meaning anything.

Step 4 — Reproduce the collision locally, not in CI

You cannot debug concurrency in a pipeline that takes eleven minutes to start. Use a fake endpoint that rate-limits the first few requests and run the real gate against it.

# tests/fake_endpoint.py
import itertools, json
from http.server import BaseHTTPRequestHandler, HTTPServer

seen = itertools.count()

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        n = next(seen)
        code = 429 if n < 4 else 200
        body = {"ok": True} if code == 200 else {"error": "rate_limited"}
        raw = json.dumps(body).encode()
        self.send_response(code)
        self.send_header("Retry-After", "1")
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(raw)

if __name__ == "__main__":
    HTTPServer(("127.0.0.1", 8787), Handler).serve_forever()

Run three workers against it and assert on behaviour, not on elapsed time:

python tests/fake_endpoint.py &
python -m pytest tests/test_gate.py -k parallel -q
# assert: requests_in_flight <= capacity at all times
# assert: total calls stayed within the attempt budget
# assert: exactly one job exited on admission timeout

Keep this test deterministic: no network, no sleeps longer than the retry cap, no wall-clock assertions. A gate whose own tests are flaky will be deleted by the second person who sees it go red.

Step 5 — Where the gate process lives

The lock file needs a filesystem shared by every job that calls the endpoint. On ephemeral hosted runners, /var/ci/ disappears between jobs, and your bucket silently resets to full every time — the worst kind of green.

Two honest options. Run the gate as a small service on one always-on host, and have jobs ask it for a slot over HTTP. Or run the whole pipeline, gate included, on a single machine you control. MonkeyCode's free server option is one place to put that process while you are evaluating the workflow; a lint-checked systemd unit beside the repo keeps it reproducible either way.

The pre-commit hook that keeps the gate honest

A hook cannot test the endpoint, but it can test the gate.

# .git/hooks/pre-commit
#!/usr/bin/env bash
set -euo pipefail
python -m pytest tests/test_gate.py -q --no-header
python -m ruff check ci_gate/

If the hook and CI run the same two commands, a broken bucket fails in seconds instead of after a queue. That is the whole green-to-merge path: same gate locally, same gate in CI, one shared budget.

Limitations and who should skip this

fcntl.flock is POSIX-only; on Windows runners you need a different primitive or the service model. Token buckets also cannot discover the endpoint's real capacity — you supply the number, and a wrong number just fails more quietly. And if your model call is not on the merge path at all, this is infrastructure for a problem you do not have.

Skip it if you have a dedicated quota, if a single job calls the endpoint once per run, or if your team has no owner for the gate's configuration. An unowned limit is a limit that gets raised on the first red build.

Merge checklist

1. Logged calls-per-run and 429s-per-run for a full week.
2. Set capacity from the measured peak, not from the highest number that once went green.
3. Verified retries re-enter the bucket.
4. Confirmed fixture mismatches are classified as test failures, never retried.
5. Ran the fake-endpoint test in the pre-commit hook and in CI.
6. Documented which host owns the lock file, and what happens when it moves.

If you try the workflow, run the fake-endpoint suite against MonkeyCode's free model access first and compare your measured 429 rate with the table above — that is the cheapest way to find out whether your concurrency was ever the problem.