The CI dashboard is green. One merge request arrives, fans out into 14 parallel jobs, and every job calls the same AI endpoint to summarize a diff. Twelve jobs fail after a 60-second timeout. Green goes red in less than a minute.
The common reaction is to blame the AI provider. The actual failure sits one layer above: unbounded concurrency. Free shared capacity looks like a fire hose until everyone opens their valve at once.
This article shows a small, dependency-free proxy that turns an OpenAI-compatible endpoint into a rate-limited, circuit-broken service suitable for CI smoke tests, batch analysis, and other throwaway workloads.
What the free tier really gives you
MonkeyCode is an open-source project that provides free model access and a free server option for developers who want to experiment without provisioning their own GPU. That combination is valuable for non-critical pipelines. The fine print is operational, not contractual: the server is shared, latency fluctuates, and your jobs are not the only tenants.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Free infrastructure changes how you design clients. A paid API can absorb 20 concurrent requests and still return in 500 ms. A free shared server cannot, and it should not be expected to. The correct response is to protect the shared resource from your own burst, and to protect your pipeline from the shared resource's inevitable hiccups.
The architecture: a local egress proxy
The pattern is simple. Every AI client in your build writes to http://localhost:8765 instead of the real endpoint. The proxy applies two rules before forwarding.
- Token bucket – caps call rate so your concurrent jobs cannot stampede the upstream.
- Circuit breaker – stops traffic for a cool-down window after repeated 5xx responses or timeouts.
This gives you a reproducible boundary. You can tune it locally, watch the failure modes, and decide whether the free tier is a good fit for a specific workload.
The proxy code
The implementation uses only the Python standard library. Save it as ai_proxy.py and run it with python3 ai_proxy.py.
#!/usr/bin/env python3
"""Rate-limited, circuit-broken proxy for OpenAI-compatible endpoints."""
import json
import threading
import time
import urllib.request
import urllib.error
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
UPSTREAM = "http://localhost:8080/v1/chat/completions" # change to MonkeyCode free server endpoint
RATE = 2 # steady requests per second
BURST = 5 # max burst size
FAIL_THRESHOLD = 3
COOLDOWN = 30 # seconds
class TokenBucket:
def __init__(self, rate, burst):
self.rate = rate
self.burst = float(burst)
self.tokens = float(burst)
self.lock = threading.Lock()
self.updated = time.monotonic()
def acquire(self, timeout=10):
deadline = time.monotonic() + timeout
with self.lock:
while True:
now = time.monotonic()
self.tokens = min(self.burst, self.tokens + (now - self.updated) * self.rate)
self.updated = now
if self.tokens >= 1:
self.tokens -= 1
return True
if now >= deadline:
return False
wait = (1 - self.tokens) / self.rate
self.lock.release()
time.sleep(min(wait, 0.1))
self.lock.acquire()
class CircuitBreaker:
def __init__(self, threshold, cooldown):
self.threshold = threshold
self.cooldown = cooldown
self.failures = 0
self.open_until = 0
self.lock = threading.Lock()
def allow(self):
with self.lock:
if time.monotonic() < self.open_until:
return False
return True
def record(self, ok):
with self.lock:
if ok:
self.failures = 0
return
self.failures += 1
if self.failures >= self.threshold:
self.open_until = time.monotonic() + self.cooldown
self.failures = 0
bucket = TokenBucket(RATE, BURST)
breaker = CircuitBreaker(FAIL_THRESHOLD, COOLDOWN)
class Handler(BaseHTTPRequestHandler):
def do_POST(self):
if not breaker.allow():
self.send_error(503, "Circuit open; upstream cooling down")
return
if not bucket.acquire():
self.send_error(429, "Rate limit exceeded; try again soon")
return
length = int(self.headers.get("Content-Length", 0))
body = self.rfile.read(length)
req = urllib.request.Request(UPSTREAM, data=body, headers={"Content-Type": "application/json"})
try:
with urllib.request.urlopen(req, timeout=60) as resp:
data = resp.read()
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.end_headers()
self.wfile.write(data)
breaker.record(True)
except urllib.error.HTTPError as e:
breaker.record(e.code >= 500)
self.send_error(e.code)
except Exception:
breaker.record(False)
self.send_error(502, "Upstream unavailable")
if __name__ == "__main__":
print(f"Proxy running on :8765 -> {UPSTREAM}")
print(f"Rate={RATE} req/s, burst={BURST}")
ThreadingHTTPServer(("127.0.0.1", 8765), Handler).serve_forever()
The proxy does not parse the prompt or track tokens. It is intentionally dumb. That keeps it trustworthy enough to put in a build pipeline.
Step-by-step: put it into a CI job
- Start the proxy on the runner:
python3 ai_proxy.py & - Point your AI client at
http://127.0.0.1:8765/v1/chat/completions. - Keep the same API key the model provider expects; the proxy forwards headers only if you extend it, which this example does not do for brevity.
- Run a small batch of 10 requests to verify the bucket works.
- Launch the 14 parallel jobs again and record completion times.
That last step is the whole point. With RATE=2, the proxy admits no more than 5 requests in the first second, then 2 per second. The 14 jobs queue locally instead of piling onto the shared server. Some finish later, but far fewer fail.
Reading the numbers
The proxy gives you two explicit failure signals: HTTP 429 and HTTP 503. Deciding what to do with them depends on your workload.
| Workload | Free model + free server | Self-hosted | Paid API |
|---|---|---|---|
| CI smoke test, low volume, tolerant to minutes | Yes, with proxy | Overkill | Acceptable |
| Interactive chat, sub-second SLO | No | Maybe | Best |
| Batch analysis, hours of runtime | Yes, with long cooldown | Best if stable | Expensive |
| User-facing feature, 2-second SLO | No | Yes | Yes |
Use the decision table as a starting point. The proxy gives you numbers to adjust the thresholds: if you see 429s, lower RATE or increase the client's backoff. If you see 503s, the server is unhealthy; raise COOLDOWN or switch workloads entirely.
Limitations and who should skip this
This proxy is a guardrail, not a scaler. It does not make the free server faster. It does not turn the shared capacity into a contract. It only prevents your own clients from acting like a herd.
Do not use this approach when:
- Your SLO requires sub-second latency. The local queue adds milliseconds, but the upstream variance will still dominate.
- You have zero tolerance for dropped jobs. Use a paid or dedicated path.
- You expect the proxy to solve upstream outages. The circuit breaker stops the bleeding, not the cause.
The example also omits API-key forwarding to keep the code short. In production, copy the Authorization header from the incoming request.
The operational takeaway
Free model access and a free server are a real way to keep experimentation costs at zero. They become hostile when your client design ignores their shared nature. A 40-line proxy restores the boundary.
Next time the CI goes red, check the concurrency before blaming the model. Your own burst is usually the first failure to fix.
If you want a quick, honest way to test MonkeyCode's free layer, run this proxy in front of it and push 100 requests. The output will tell you whether the free server fits your workflow or whether you need the more predictable option.
Top comments (0)