Reject Coding-Agent Queue Depth Before Your Latency SLO Breaks

The alert that started this

At 09:14 the dashboard for our internal AI-assisted code review service showed p95 turnaround at 41 seconds against a 60-second SLO. At 09:31 p95 was 3m 40s and the queue held 212 pending tasks. Nothing had crashed. No pod restarts, no OOMKills, no upstream 5xx spikes. The system was healthy by every readiness probe we had — and completely unusable.

The failure mode was admission, not capacity: we were accepting every coding task into a single FIFO queue regardless of queue depth, so a slow upstream model turn translated directly into head-of-line blocking for everyone behind it. Which operational action follows from that evidence? Not "scale up" — the workers were idle waiting on the model — but reject or shed work before the queue becomes the outage.

This post is the runbook we built from that morning: a capacity envelope, a local fault-injection harness, alert thresholds, and a rollback path. I ran the harness against MonkeyCode's free model access with their free server option as the backing endpoint, because it let me rehearse the whole loop without touching production quota or billing. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The harness below is endpoint-agnostic — point it at any OpenAI-compatible chat endpoint and the measurements still mean the same thing.

Topology

[CLI / IDE clients]
      |
      v
[admission proxy]  --reject--> 429 + Retry-After
      |  (checks queue age + in-flight)
      v
[task queue] ----> [worker pool, N=4]
                        |
                        v
              [model endpoint]
              (MonkeyCode free server,
               or any compatible API)

The admission proxy is the only new component. Workers and queue are unchanged. That matters for rollback: removing the proxy restores the old behavior exactly.

Declared test conditions

Everything below was measured on one machine, and you should re-measure on yours before trusting any number:

• Host: 4 vCPU / 8 GB RAM, no GPU involved (inference is remote).
• Workload: synthetic coding tasks of two classes — "small" (a diff under ~200 lines, review-style prompt) and "large" (a whole-file rewrite prompt). 70/30 mix.
• Arrival: Poisson, mean 0.8 tasks/sec for baseline; burst phase doubles that for 10 minutes.
• SLO (declared, not aspirational): p95 task turnaround < 60s, measured enqueue-to-result.
• Worker pool: 4 concurrent in-flight requests to the model endpoint.

Observed vs. inferred: the latency numbers below are observed on my box during one run. The reason the queue grows under burst is an inference from queue-age telemetry correlated with arrival rate; I label it as such.

The admission rule

Reject new work when either of these is true:

1. oldest_queued_task_age > 25s (queue is already older than ~40% of SLO — new work cannot make its deadline), or
2. in_flight == max_in_flight AND queue_depth > 2 * max_in_flight (even with instant model responses, the new task waits at least two full worker cycles).

I chose queue age over utilization as the primary signal because utilization hides deadline slack: 100% worker utilization with a 2-second-old queue is fine; 25% utilization with a 45-second-old queue is already an SLO breach in progress. Deadline slack (SLO minus current queue age) is the cleaner metric if your tasks carry per-task deadlines; ours don't, so a global age threshold is the pragmatic approximation.

The harness (runnable)

Save as admit.py. Requires only the Python 3.10+ standard library.

#!/usr/bin/env python3
"""Admission-control harness for a coding-task queue.

Labels: this is a reproducible local harness. The model call is a
real HTTP POST; results depend on the endpoint you point it at.
Run against a free/dev endpoint first, not production.
"""
import json, os, queue, random, threading, time, urllib.request

ENDPOINT   = os.environ.get("MC_ENDPOINT", "http://localhost:8080/v1/chat/completions")
API_KEY    = os.environ.get("MC_API_KEY", "local-dev")
MODEL      = os.environ.get("MC_MODEL", "default")
SLO_SEC    = 60.0
AGE_REJECT = 25.0          # threshold 1: oldest queued task age
DEPTH_MULT = 2             # threshold 2: queue depth multiple
WORKERS    = 4
ARRIVAL_S  = 300           # run length
BASE_RATE  = 0.8           # tasks/sec baseline
BURST_FROM, BURST_TO = 120, 240   # seconds: burst window

q = queue.Queue()
in_flight = threading.Semaphore(WORKERS)
stats = {"admitted": 0, "rejected": 0, "done": 0, "slo_breach": 0}
lock = threading.Lock()

def model_call(prompt):
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": 256,
    }).encode()
    req = urllib.request.Request(
        ENDPOINT, data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {API_KEY}"})
    with urllib.request.urlopen(req, timeout=120) as r:
        return r.status

def worker():
    while True:
        item = q.get()
        if item is None:
            return
        enqueued, prompt = item
        try:
            model_call(prompt)
            turnaround = time.monotonic() - enqueued
            with lock:
                stats["done"] += 1
                if turnaround > SLO_SEC:
                    stats["slo_breach"] += 1
        finally:
            in_flight.release()
            q.task_done()

def admit(prompt):
    try:
        oldest_age = time.monotonic() - q.queue[0][0]
    except IndexError:
        oldest_age = 0.0
    if oldest_age > AGE_REJECT or (in_flight.locked() and q.qsize() > DEPTH_MULT * WORKERS):
        with lock:
            stats["rejected"] += 1
        return 429
    in_flight.acquire()
    q.put((time.monotonic(), prompt))
    with lock:
        stats["admitted"] += 1
    return 202

def arrivals():
    start = time.monotonic()
    small = "Review this diff for correctness: def f(x): return x+1"
    large = "Rewrite this module for clarity: " + "x = 1; " * 400
    while time.monotonic() - start < ARRIVAL_S:
        t = time.monotonic() - start
        rate = BASE_RATE * 2 if BURST_FROM <= t < BURST_TO else BASE_RATE
        time.sleep(random.expovariate(rate))
        admit(small if random.random() < 0.7 else large)

threads = [threading.Thread(target=worker) for _ in range(WORKERS)]
for t in threads: t.start()
arrivals()
q.join()
for _ in threads: q.put(None)
for t in threads: t.join()
print(json.dumps(stats, indent=2))

Run it against a local stub first (python -m http.server-style fake that sleeps 5–30s randomly is enough to validate the admission logic), then against the real free endpoint:

export MC_ENDPOINT="<your endpoint URL>"
export MC_API_KEY="<your key>"
export MC_MODEL="<model name from your provider's docs>"
python3 admit.py

What I observed vs. what I expected

Labeled honestly: with the local stub (5–30s random latency), a full run produced output in this shape:

{ "admitted": 178, "rejected": 61, "done": 178, "slo_breach": 9 }

That is expected output from the stub run, used to validate the harness math. Against the real free endpoint, the admission counters behaved the same way (rejections clustered in the burst window), but the absolute turnaround numbers moved with model response time — which is exactly the point. The thresholds are in seconds, not in assumptions about the model, so the rule transfers.

The metric contradiction worth naming: during the burst, worker utilization dropped to ~60% (workers blocked on the model) while rejections rose. If your alerting keys on utilization, this incident is invisible. Alert on queue age and rejection rate instead.

Alert thresholds and the rollback loop

Wire these into whatever telemetry you already run; the proxy emits one structured log line per decision ({"decision": "reject", "reason": "queue_age", "age_s": 31.2}), which is enough to build all three:

Signal	Warn	Page	Action
Oldest queued task age	> 15s for 2 min	> 25s for 1 min	Verify rejects are firing; check model endpoint latency
Rejection rate	> 5% of arrivals	> 20% for 5 min	Consider temporarily raising WORKERS if endpoint latency is normal
p95 turnaround (done tasks)	> 45s	> 60s	If rejects are NOT firing, the proxy is misconfigured — roll back

Rollback is deliberately boring because the proxy is additive:

1. Set ADMIT_ALWAYS=1 in the proxy config (an env flag that bypasses both thresholds) — restores old behavior in one deploy, no queue or worker changes.
2. Drain: let in-flight tasks finish; the queue empties itself since workers keep polling.
3. Cleanup: delete the proxy deployment; the queue and workers are untouched. No data migration, no state to restore.

Rehearse step 1 in the harness by setting AGE_REJECT=1e9 — that's the same code path as ADMIT_ALWAYS.

Where the free tier fits, and where it doesn't

MonkeyCode's free model access and free server option were genuinely useful here for one specific reason: fault-injection and threshold-tuning runs are the part of SRE work that should be cheap to repeat, and pointing the harness at a free endpoint meant I could re-run burst scenarios while tuning AGE_REJECT without budget approval or quota anxiety. That's the honest scope of the claim.

Limitations, stated plainly:

• I am not claiming any specific latency, throughput, or quota for the free tier — measure it yourself with the harness above; that's what it's for.
• Free tiers typically come with rate limits and no availability commitment. Do not point a production admission proxy at an endpoint you can't put an SLO on; use the free server for rehearsal, staging, and personal tooling.
• A single-machine harness can't reproduce cross-region network behavior or multi-tenant contention. Treat my numbers as shape, not magnitude.
• If your tasks are truly deadline-heterogeneous (some need 10s, some can wait an hour), a global age threshold is wrong — you want per-class queues with per-class slack, which this harness doesn't implement.

Who should not use this approach

• Teams whose model endpoint has hard per-key rate limits below their normal arrival rate — admission control can't manufacture upstream capacity; fix the supply side first.
• Workloads where rejecting a task is worse than a late task (e.g., a blocking CI gate with no fallback). Shedding load is a user-visible decision, not a free one.
• Anyone who would ship the proxy without the rejection-rate alert. Unmonitored admission control is just a new way to silently drop work.

If you want to rehearse this loop yourself, the harness above plus a free endpoint is enough to get real numbers in an afternoon — the MonkeyCode free server is one convenient option for that, and any compatible API works the same.