Reject Agent Jobs Before Queue Age Breaks Your SLO: A Local Admission-Control Drill

The alert that motivated this

At 14:07 our serving dashboard showed a contradiction I see often in AI-backed backends: GPU-side utilization was under 40%, but p95 time-to-first-token had tripled. Requests weren't failing at the model. They were aging in the queue in front of it. By the time the autoscaler metric (utilization) moved, the SLO was already broken.

So the operational question is: which signal do you reject on — utilization, queue depth, or queue age — and at what threshold? This post is a reproducible local drill to answer that for your own stack before production answers it for you.

Observed behavior vs. inference, stated up front: I have observed queue age leading utilization as a failure precursor in my own serving setups. The thresholds below are starting points from a local drill, not validated production values — you must re-derive them on your hardware and workload.

Topology under test

client -> edge (envoy/nginx) -> admission gate -> work queue (Redis stream)
       -> worker pool (N=2) -> model endpoint (OpenAI-compatible HTTP)

The admission gate is the only new component: a small shim that decides accept, shed, or degrade per request, based on the oldest job's age in the queue — not on utilization.

For the model endpoint in this drill I used a free option rather than burning paid quota on synthetic load: MonkeyCode offers free model access and a free server option, which makes it a low-cost target for exactly this kind of pre-production fault injection. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The drill below is endpoint-agnostic — any OpenAI-compatible server works, including a local vLLM or llama.cpp instance — and the conclusions do not depend on which endpoint you point it at.

Declared workload and test conditions

• Machine: 8 vCPU / 32 GB RAM, single node, Docker Compose
• Workers: 2, concurrency 1 each (deliberately undersized so the queue actually fills)
• Load: k6, ramp 0 -> 20 RPS over 90s, hold 120s, ramp down
• Request shape: fixed prompt (~200 tokens in, max 256 out) to keep per-job latency variance low
• Failure injection: at T+60s, add 800ms artificial latency to the model endpoint via toxiproxy
• One run per rejection policy: (A) no admission control, (B) reject when oldest job age > 2s, (C) reject when queue depth > 10, (D) reject on utilization > 80%

The admission gate (reference implementation)

# gate.py — FastAPI shim in front of the queue. Labeled: reference code, run locally.
import time, os
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
import redis

app = FastAPI()
r = redis.Redis(host="redis", decode_responses=True)
STREAM = "jobs"
MAX_QUEUE_AGE_MS = int(os.getenv("MAX_QUEUE_AGE_MS", "2000"))

@app.post("/v1/completions")
async def submit(req: Request):
    # Oldest pending job age = now - entry time of stream head (ms-precision IDs)
    pending = r.xrange(STREAM, count=1)
    if pending:
        head_ts = int(pending[0][0].split("-")[0])
        age_ms = int(time.time() * 1000) - head_ts
        if age_ms > MAX_QUEUE_AGE_MS:
            return JSONResponse(
                status_code=429,
                content={"error": "shed", "queue_age_ms": age_ms},
                headers={"Retry-After": "1"},
            )
    body = await req.json()
    r.xadd(STREAM, {"payload": __import__("json").dumps(body)})
    return {"queued": True}

Worker telemetry — emit these fields per job, minimum:

time_enqueue_ms, time_dequeue_ms, queue_age_ms, worker_id,
endpoint_latency_ms, ttft_ms, tokens_out, result (ok|shed|timeout|error)

Results (clearly labeled)

Expected output shape from my local runs — treat as illustrative, re-run to get your own numbers:

Policy	p95 TTFT (no fault)	p95 TTFT (fault window)	Shed rate	SLO breach?
A: none	1.1s	14.8s	0%	yes, hard
B: queue age > 2s	1.1s	2.6s	11%	no
C: depth > 10	1.1s	9.4s	8%	yes
D: utilization > 80%	1.1s	13.2s	4%	yes

The pattern to look for in your own data: policy B sheds more requests but keeps the accepted ones inside the SLO. Depth and utilization both lag — utilization especially, because it only rises after workers are already saturated, while queue age rises the moment arrival rate exceeds service rate. That lag is the whole argument for rejecting on age.

Choosing the threshold operationally

Don't copy my 2s. Derive it:

1. Take your SLO budget for the queued stage (e.g., total TTFT SLO 5s minus observed p95 endpoint latency 1.5s = 3.5s of queue slack).
2. Set MAX_QUEUE_AGE_MS at roughly 60–70% of that slack (~2.1–2.4s here), leaving room for dequeue-to-dispatch overhead.
3. Prefer queue age over deadline slack when clients don't declare deadlines; prefer deadline slack when they do.

Failure handling, rollback, cleanup

• Failure modes of the gate itself: Redis unreachable -> fail closed only if your SLO values latency over availability; otherwise fail open and alert. Wrong threshold -> you shed healthy traffic; watch shed rate as a first-class metric with its own alert (e.g., shed > 5% for 5m with no latency fault = threshold too tight).
• Rollback: the gate is a sidecar/proxy hop. Rolling back = repointing the edge at the queue directly; keep that route as a live, tested config, not a runbook note.
• Cleanup: docker compose down -v, delete the toxiproxy container, flush the Redis stream. Don't leave the 800ms fault in place — I've personally debugged a "regression" that was a leftover fault injection.

Limitations and who should not do this

• Single-node, fixed-size requests. Real traffic has heavy-tailed token counts; re-derive thresholds under a realistic prompt mix.
• A free endpoint tier is fine for shedding behavior and queue dynamics, but don't extrapolate absolute latencies or capacity from it to a paid/production deployment — different hardware, limits, and queuing. Validate thresholds again against the real backend.
• If your workload is latency-insensitive batch (embeddings backfill, offline eval), admission shedding buys you nothing — just let the queue drain.
• If you can't tolerate any rejected requests (e.g., transactional flows), this pattern is wrong; you need backpressure into the producer instead.

Takeaway

Utilization tells you workers are busy; queue age tells you the SLO is already dying. The drill above takes an afternoon and gives you a threshold, a shed-rate alert, and a tested rollback path. If you need a cheap endpoint to run it against, the free model access and free server option I mentioned are one way to keep the experiment off your paid quota — but the artifact that matters is the gate and the telemetry, not the endpoint.

What signal does your serving stack reject on today, and where did that threshold come from?