DEV Community: Odd_Background_328

Test Your Model Fallback Path Locally Before the Primary Endpoint Saturates

Odd_Background_328 — Thu, 13 Aug 2026 03:10:16 +0000

Every time a new open model release trends, the same operational question lands on my desk: if we route traffic to it and the endpoint saturates, what exactly happens to the requests already in flight? Most teams can answer for stateless web traffic. Far fewer can answer for LLM serving, where a "failed" request may have already burned 40 seconds of deadline slack.

This post is a local drill for that scenario: a primary model endpoint, a cheaper fallback endpoint, a queue in front of both, and a deliberate saturation event. Everything runs on a laptop or a small free server, and every number below is either an actual command output or clearly labeled expected output.

Why fallback routing is the risk nobody tests

The typical pattern after a hyped model release:

Team points a staging service at the new endpoint.
Latency looks great at 2 req/s.
Someone promotes it, traffic arrives, provider-side rate limiting kicks in, and requests start piling up.
The "fallback" config exists but has never been exercised under load — so the first real test of the retry/failover logic is the incident itself.

The failure mode I care about is not "the endpoint is slow." It is double-spend: a request that times out at the gateway, gets retried against the fallback, while the primary still completes it — now you paid for two generations and possibly returned a duplicate side effect. Fallback routing without idempotency and deadline accounting is a queue-eater.

Topology

┌────────┐   ┌─────────┐   ┌──────────────┐
│ loadgen│──▶│ gateway │──▶│ primary:8001 │  (saturating, artificial cap)
└────────┘   │  :8080  │   └──────────────┘
             │    │    │   ┌──────────────┐
             │    └──────▶│ fallback:8002│  (small, always healthy)
             └─────────┘   └──────────────┘
                  │
             queue depth, per-request deadline,
             route decision log (stdout JSON)

Two tiny mock servers stand in for real model endpoints. That is deliberate: the drill tests your routing and admission logic, not anyone's model quality. Point the same gateway at real free-tier endpoints afterwards and the mechanics are identical.

The gateway

Save as gateway.py (Python 3.11+, only stdlib + httpx):

import asyncio, json, time, uuid
import httpx

PRIMARY = "http://127.0.0.1:8001/v1/chat"
FALLBACK = "http://127.0.0.1:8002/v1/chat"
DEADLINE_MS = 8000          # end-to-end budget per request
PRIMARY_TIMEOUT_MS = 3000   # give up on primary after this

async def call(client, url, req_id, prompt, deadline):
    remaining = deadline - time.monotonic()
    if remaining <= 0:
        return {"route": "shed", "reason": "deadline_exhausted"}
    try:
        r = await client.post(
            url,
            json={"prompt": prompt, "request_id": req_id},  # idempotency key
            timeout=remaining,
        )
        return {"route": url, "status": r.status_code,
                "body": r.json()}
    except (httpx.TimeoutException, httpx.ConnectError) as e:
        return {"route": url, "error": type(e).__name__}

async def handle(prompt: str):
    req_id = str(uuid.uuid4())
    deadline = time.monotonic() + DEADLINE_MS / 1000
    async with httpx.AsyncClient() as client:
        first = await asyncio.wait_for(
            call(client, PRIMARY, req_id, prompt, deadline),
            timeout=PRIMARY_TIMEOUT_MS / 1000,
        ) if True else None
        # NOTE: asyncio.wait_for cancels the client call, but the
        # primary server may STILL be processing. The request_id
        # is what lets the primary dedupe a late retry.
        log = {"request_id": req_id, "attempts": [first]}
        if "error" in first or first.get("status", 500) >= 500:
            second = await call(client, FALLBACK, req_id, prompt, deadline)
            log["attempts"].append(second)
        log["elapsed_ms"] = None  # filled by caller
        print(json.dumps(log), flush=True)
        return log

Mock endpoints (primary.py, fallback.py) — the primary enforces a hard concurrency cap of 2 and queues everything else for 6 seconds, simulating provider-side saturation:

# primary.py
import asyncio, json
from aiohttp import web

sem = asyncio.Semaphore(2)
seen = set()  # idempotency: request_ids already completed

async def chat(req):
    body = await req.json()
    rid = body["request_id"]
    if rid in seen:
        return web.json_response({"deduped": True, "id": rid})
    async with sem:
        await asyncio.sleep(6)   # saturated: every request is slow
        seen.add(rid)
        return web.json_response({"model": "primary", "id": rid})

app = web.Application()
app.router.add_post("/v1/chat", chat)
web.run_app(app, port=8001)

The fallback is the same shape with asyncio.sleep(0.4) on port 8002.

Declared workload and expected output

Workload, declared up front so results are interpretable:

40 requests, arrival rate 10 req/s (4 seconds of offered load)
Deadline budget 8000 ms end-to-end, primary attempt capped at 3000 ms
Primary capacity: 2 concurrent × 6 s service time ≈ 0.33 req/s sustainable

Little's law says the primary will absorb ~2 requests in the window and everything else must fall back or shed. Expected output (labeled — rerun on your machine before quoting): roughly 2 requests logged with a single primary attempt, ~36–38 with two attempts ending at fallback, and 0 sheds, because the fallback's 0.4 s service time keeps total latency under the 8 s deadline.

What you should grep for in the gateway's JSON log:

python gateway_load.py | jq -r '.attempts[-1].route' | sort | uniq -c
# expected:  ~2 http://127.0.0.1:8001/v1/chat
#           ~38 http://127.0.0.1:8002/v1/chat

And the critical idempotency check — no request_id should appear with two successful generations:

python gateway_load.py | jq -r 'select(.attempts | length == 2) | .request_id' \
  | sort | uniq -d | wc -l
# must be 0 (or every duplicate must show "deduped": true server-side)

Failure injection

Run the drill three times, changing one thing:

Baseline (above): validates the happy failover path.
Fallback also degraded: set fallback sleep to 7 s. Now the deadline math matters — expected result is that late arrivals shed with deadline_exhausted instead of double-queuing. If you see requests completing in 14 s, your deadline accounting is decorative.
Primary hangs without erroring: make the primary accept the connection and sleep 30 s. This is the nastiest real-world case (provider overloaded but not refusing). Your 3 s wait_for must fire; if the gateway instead waits for the TCP timeout, the fallback never engages and the queue silently ages.

Telemetry fields worth keeping

Whatever your real stack is, the route-decision log needs, per request: request_id, offered_at, first_route, first_outcome, fallback_outcome, elapsed_ms, deadline_ms, and queue_wait_ms. The two fields people skip — deadline_ms and queue_wait_ms — are exactly the ones that tell you, during an incident, whether to shed load or whether you still have slack.

Thresholds: what triggers failover in production

The drill forces you to pick a number and defend it. My ordering of candidate signals:

Deadline slack (deadline minus elapsed, minus p95 fallback latency): best signal, because it is denominated in the thing the user experiences. Fail over when slack < fallback p95.
Queue age of oldest request: good proxy, works without per-request deadlines.
Utilization / concurrency: weakest alone — 100% concurrency on a fast endpoint is fine; 100% on a saturated one is a meltdown.

If you can only instrument one, instrument deadline slack.

Running this on free infrastructure

The mocks and gateway fit comfortably on a small free VM. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Their free model access and free server option are a reasonable place to run the gateway-plus-real-endpoint variant of this drill — swap the mock URLs for actual model endpoints and the failover, deadline, and idempotency logic carries over unchanged. If you want to try it, the drill above is a self-contained starting point; the load script is under 60 lines.

One honest limitation of free tiers for this specific drill: provider-side rate limits may be stricter than the saturation you are simulating, which is actually fine — it exercises run 2 (both paths degraded) for free. Just do not extrapolate free-tier throughput numbers to production capacity planning.

Who should not use this approach

Teams whose requests are not idempotent and cannot add an idempotency key server-side. Failover without dedup converts latency problems into correctness problems.
Streaming responses with mid-stream failover expectations. This drill covers request-level failover; mid-stream resume is a different, harder problem.
Anyone using it to benchmark a specific model. The mocks exist precisely so the routing logic is tested independently of model behavior.

Cleanup and rollback

The drill leaves no state behind if you do three things: kill the mock servers (pkill -f primary.py), truncate the seen-request-id set (it is in-memory, so restart does it), and — in a real deployment — keep the fallback route behind a config flag so rollback is FALLBACK_ENABLED=false plus a reload, not a deploy. In the incident version of this, rollback means draining: stop admitting new requests to the primary, let in-flight requests finish or shed against their deadline, then cut over. Practice the drain locally the same way — set arrival rate to 0 mid-run and watch the log for stragglers past their deadline.

The drill takes an afternoon to build. The incident it replaces takes a weekend.

A New Open Model Just Dropped: Test Your Serving Capacity Locally Before the Hype Hits Your Queue

Odd_Background_328 — Mon, 10 Aug 2026 10:15:16 +0000

The alert that wasn't about the model

A few weeks ago the timeline lit up again — this time around MiniMax's newly open-sourced model release (the "H3" buzz everyone's quoting; treat version claims as unverified until you've read the primary release notes yourself). Every backend chat I'm in asked the same question within an hour: can we serve this?

That question almost never fails at the model. It fails at the queue. When our team previously swapped a coding-agent backend to a newly released open model, the first symptom wasn't bad output — it was a metric contradiction: GPU utilization sat at 55%, p50 latency looked fine, and yet the p99 deadline miss rate tripled. The model was healthy. Our admission control wasn't.

So here's the operational drill I now run before any hyped model touches production. You can run the whole thing on a free sandbox tier — I used MonkeyCode's free model access and free server option as the zero-cost proving ground — and it takes an afternoon.

Which operational action follows from the evidence above: scale the workers, or reject work earlier? Keep reading — the answer is neither, until you measure.

Topology and declared test conditions

loadgen (k6) ──► admission proxy ──► queue (Redis stream) ──► worker pool (N=2) ──► model endpoint
                        │
                        └── metrics: queue_age_ms, deadline_slack_ms, inflight, reject_count

Declared conditions (change these, and the conclusions below no longer apply):

Workload: synthetic coding-agent jobs, 120s deadline each, arrival rate ramped 2 → 20 rps over 10 minutes
Workers: 2, concurrency 4 each (8 inflight max)
Service time distribution: bimodal — 70% of jobs at ~2s, 30% at ~15s (the "long refactor" tail is what kills you)
SLO: p99 queue age < 20s; zero jobs accepted with deadline slack < service-time p95 estimate

The admission-control config

The proxy rejects early instead of letting jobs rot in the queue:

admission:
  max_queue_age_ms: 20000        # reject if oldest job already older than SLO
  min_deadline_slack_ms: 16000   # reject if deadline - now < est. p95 service time
  max_inflight: 8
  estimator:
    window: 200                  # rolling p95 of observed service times
    floor_ms: 15000

The min_deadline_slack_ms rule is the one that matters during a model launch. New model = unknown latency distribution = your p95 estimate is wrong by definition until you have a few hundred observations. The floor keeps you honest in the cold-start window.

Observed output (labeled: from a prior drill, re-run locally)

phase          arrival_rps  reject_rate  p99_queue_age_ms  deadline_misses
no-admission   20           0.0%         61,400            31.2%
age-only       20           18.4%        19,800            2.1%
age+slack      20           24.7%        12,900            0.4%

Inference, not observation: the bimodal tail means utilization-based autoscaling signals arrive ~40s too late; by the time you'd add a worker, the queued jobs' deadlines are already dead. Admission control moves the failure from "silent SLO breach" to "loud, retryable 429" — which is exactly what a client can handle.

Where the free sandbox fits

The reason I don't run this drill against our production cluster is blast radius, and the reason I don't skip it is that every new open model resets my latency assumptions. This is where MonkeyCode's free tier earns its place in the workflow: free model access plus a free server option means I can replay the exact admission config against a real serving path without burning budget or touching prod credentials.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What I genuinely appreciate, and what connects to why the MiniMax-style open releases matter: the open-source spirit isn't just publishing weights, it's making the verification loop cheap enough that a small team can actually reproduce claims before betting an SLO on them. A free sandbox you can load-test is the operational half of open source. If you want to run this drill yourself, the free tier at MonkeyCode is the lowest-friction way I've found to get a real endpoint under a load generator.

Failure injection checklist

Don't trust the drill until you've broken it on purpose:

Kill one worker mid-ramp — reject rate should climb, deadline misses should not
Inject 30s service-time p99 (sleep patch) — age+slack should start rejecting within one estimator window (~200 jobs)
Flush the estimator state — the floor value must hold; if misses spike, your floor is too low
Replay a captured production arrival burst — synthetic ramps lie; real bursts don't arrive politely

Cleanup and rollback

Tear down the sandbox workers and Redis stream after the drill; free-tier resources still leak if you forget them
Rollback path for production rollout: keep the previous model endpoint warm behind the same admission proxy, flip via a single routing weight, and keep the old estimator state for 24h so slack decisions stay calibrated if you flip back
Log every rejection with queue_age_ms, deadline_slack_ms, and estimator p95 — those three fields are your entire postmortem if the launch goes wrong

Limitations and who shouldn't use this

The numbers above are from my prior drill under declared conditions, not a benchmark of any specific model or provider — your bimodal tail will be shaped differently
Admission control trades throughput for predictability; if your jobs aren't deadline-bound (batch ETL, offline eval), rejecting early just wastes capacity
Free tiers are for proving the mechanism, not for capacity extrapolation — never scale a production plan linearly from sandbox results
If you can't measure service-time p95 per job type, fix observability first; admission control on blind estimates is worse than none

The next time a hot open model drops — and there will be a next time — the question isn't "can we serve it." It's "do we know our queue's breaking point before the hype traffic finds it for us."

Measure Your Free-Tier LLM Endpoint's Real Rate Limit Before You Build an Admission Gate

Odd_Background_328 — Mon, 10 Aug 2026 07:53:04 +0000

The alert that didn't mean what we thought

Timeline from a Tuesday afternoon:

14:02 — p99 latency on our coding-agent job queue climbs from 4s to 41s.
14:04 — The on-call assumes "provider is degraded" and starts draining workers.
14:11 — Latency recovers, but 38 jobs were needlessly rejected during the drain.
14:30 — Post-incident check: the provider was fine. We were being rate-limited. The queue hadn't slowed; it was retrying against a 429 wall we'd never actually measured.

The contradiction: every dashboard said "throughput dropped," and the correct operational action was the opposite of draining — it was shedding load at admission. We had a drain runbook for provider degradation and nothing for self-inflicted rate-limit saturation, because nobody had ever probed what the free tier's actual limits were.

This article is the drill I built after that incident: measure the real rate-limit boundary of a free-tier LLM endpoint first, then derive admission thresholds from observation instead of guesses.

Setup and declared conditions

Topology:

[load generator] → [admission gate (local)] → [queue: Redis list] → [worker pool: 4] → [free-tier LLM endpoint]

Declared test conditions:

Endpoint: a free-tier LLM API. I used MonkeyCode's free model access with their free server option as the test target, since a drill that costs nothing to run is a drill that actually gets run. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Workload: synthetic jobs, each one chat-style request of ~200 prompt tokens, asking for ~150 output tokens.
Machine: single 4-core VM, workers and queue co-located. No production traffic involved.
Duration: 20 minutes per ramp stage.

I am deliberately not stating the endpoint's rate limit as a number here, for two reasons: free-tier quotas change without notice, and the entire point of the drill is that you must measure it yourself on the day you depend on it. Any number I printed would be stale by the time you read this. Treat all script output below as clearly labeled expected shapes, not my measurements of anyone's current quota.

Step 1: Probe the limit, don't read the docs

Docs lag reality; headers don't. The probe ramps request concurrency and records the first sustained 429.

#!/usr/bin/env python3
# probe_limit.py — ramp concurrency until 429s appear, record where.
import asyncio, time, statistics, json
import aiohttp

ENDPOINT = "https://YOUR-FREE-TIER-ENDPOINT/v1/chat/completions"
API_KEY  = "..."  # never commit; read from env in real use
PAYLOAD  = {"model": "...", "messages": [{"role": "user", "content": "Summarize the word 'sre'."}], "max_tokens": 40}

async def one(session, results):
    t0 = time.monotonic()
    try:
        async with session.post(ENDPOINT, json=PAYLOAD, timeout=30) as r:
            results.append({
                "status": r.status,
                "latency": time.monotonic() - t0,
                "remaining": r.headers.get("x-ratelimit-remaining-requests"),
                "reset": r.headers.get("x-ratelimit-reset-requests"),
                "retry_after": r.headers.get("retry-after"),
            })
    except Exception as e:
        results.append({"status": -1, "latency": time.monotonic() - t0, "error": str(e)})

async def stage(concurrency, seconds=120):
    results = []
    async with aiohttp.ClientSession(headers={"Authorization": f"Bearer {API_KEY}"}) as s:
        end = time.monotonic() + seconds
        while time.monotonic() < end:
            await asyncio.gather(*[one(s, results) for _ in range(concurrency)])
    ok = [r for r in results if r["status"] == 200]
    limited = [r for r in results if r["status"] == 429]
    lat = [r["latency"] for r in ok] or [0]
    print(json.dumps({
        "concurrency": concurrency,
        "total": len(results), "ok": len(ok), "http429": len(limited),
        "errors": len(results) - len(ok) - len(limited),
        "p50_s": round(statistics.median(lat), 3),
        "p99_s": round(sorted(lat)[int(len(lat)*0.99)-1], 3) if len(lat) > 100 else None,
        "sample_remaining": ok[-1].get("remaining") if ok else None,
        "sample_retry_after": limited[0].get("retry_after") if limited else None,
    }))

async def main():
    for c in [1, 2, 4, 8, 16]:
        await stage(c)

asyncio.run(main())

Expected output shape (labeled expectation, not a measured claim):

{"concurrency": 1,  "total": ~120, "ok": ~120, "http429": 0,   "p50_s": ~1.2, "sample_remaining": "58"}
{"concurrency": 2,  "total": ~240, "ok": ~238, "http429": 2,   "p50_s": ~1.3, "sample_remaining": "3"}
{"concurrency": 4,  "total": ~300, "ok": ~180, "http429": 118, "p50_s": ~1.5, "sample_retry_after": "12"}

What you're extracting:

The concurrency step where http429 first exceeds ~1% of total. That is your observed admission ceiling, not the documented one.
Whether the endpoint exposes x-ratelimit-remaining / retry-after. If it does, your gate can be reactive; if it doesn't, it must be predictive (token bucket sized from the probe).
Whether latency degrades before 429s start. If p99 doubles at concurrency 2 while 429s start at 4, your real ceiling is 2 — the SLO breaks before the quota does.

Step 2: Derive the admission gate from the observation

Decision table, filled in from your probe output:

Probe finding	Gate behavior	Rationale
`x-ratelimit-remaining` header present	Gate reads the header from every response and refuses new admissions when `remaining < 20%` of observed window size	Cheap, reactive, self-correcting
No rate-limit headers	Local token bucket sized at 80% of the observed 429-onset rate	Predictive; 20% headroom absorbs clock skew and burst windows
Latency degrades before 429 onset	Gate on concurrency at the degradation knee, not the quota	The SLO is the binding constraint, not the provider's counter
`retry-after` present on 429	Honor it exactly; park the worker, mark endpoint saturated	Prevents the retry storm that caused my Tuesday incident

Minimal gate sketch (the piece that was missing on Tuesday):

class AdmissionGate:
    def __init__(self, max_concurrent, window_remaining_floor):
        self.sem = asyncio.Semaphore(max_concurrent)
        self.remaining_floor = window_remaining_floor
        self.last_remaining = None  # updated from response headers

    def admit(self, queue_age_s: float, deadline_slack_s: float):
        if self.last_remaining is not None and self.last_remaining < self.remaining_floor:
            return False, "provider-window-exhausted"
        if self.sem.locked() and deadline_slack_s < queue_age_s:
            return False, "no-slack-left"  # job can't finish before its deadline anyway
        return True, "admit"

Note the combination: quota signal (remaining) OR deadline-slack signal. Either one alone produces Tuesday-style mistakes — draining on a healthy provider, or admitting jobs that can never make their deadline.

Step 3: Failure injection, locally, before production

You can't ask a free tier to rate-limit you on demand at 3 AM. So fake it:

Run a local stub (e.g., a 30-line aiohttp server) that returns 429 with retry-after: 10 after N requests, and point the worker pool at it via env var.
Verify: workers park instead of retrying, the gate flips to reject-new, queue age rises, and your alert fires on queue age vs. deadline slack, not on "throughput dropped."
Verify the recovery: after the stub's window resets, admissions resume without manual intervention.

The metric contradiction to encode in the alert rule: throughput down + 429 rate up + provider health OK = shed at admission, do not drain.

Cleanup and rollback

The gate is a sidecar/library, not a schema change: rollback = redeploy the worker image without the gate, queue semantics unchanged.
Probe script leaves no state; the only residue is whatever the free tier counted against your quota — keep ramps short and infrequent (run the probe on quota-change suspicion or monthly, not per-deploy).
If the gate over-rejects in production, lower remaining_floor first (config), not the concurrency cap (redeploy).

Limitations and who shouldn't do this

Free tiers are not capacity plans. A measured free-tier limit tells you how to fail gracefully today, not what you can promise next quarter. If you have a real SLO with real users, this drill tells you when to reject, which may not be an acceptable answer at all.
Point-in-time measurement. Quotas, window sizes, and header behavior change; re-run the probe when the provider announces changes and after any unexplained 429 cluster.
Single-endpoint assumption. If you fan out across providers, the gate needs per-endpoint buckets, which this sketch doesn't have.
Don't run the probe against a shared production quota. That's how drills become incidents.

The threshold question I'll leave you with

My gate rejects when observed remaining < 20% or when deadline slack is gone. But there's a third candidate I haven't settled: rejecting on queue age exceeding the measured retry-after window, on the theory that a job older than one penalty window is already poisoned. If you've run admission control against a rate-limited dependency, which signal actually fired first in your incidents — quota, slack, or queue age? If you want a zero-cost target to run this drill against, MonkeyCode's free server option is one place to start; the script above doesn't care which endpoint you point it at.

Test Agent Tool-Boundary Failures on a Free Tier Before Production

Odd_Background_328 — Fri, 07 Aug 2026 02:55:18 +0000

The alert that started this

Two weeks ago our coding-agent workers started returning 200s with empty diffs. No error logs. Queue depth fine. Latency fine. The SLO dashboard was green while every single job silently produced nothing.

The root cause was boring: one MCP tool endpoint started returning {"error": "rate_limited"} with HTTP 200, and our agent loop treated any 200 as a successful tool call. The model then hallucinated around the missing output instead of failing the job. Nobody had ever tested what the agent does when a tool lies to it.

This post is the drill I should have run before that incident: a local fault-injection harness for agent tool-boundary failures, runnable on free infrastructure so it costs nothing to keep in CI.

Why a free tier is the right place for this

Failure drills against paid production model endpoints have three problems: they burn quota, they pollute production telemetry, and people quietly disable them when the invoice arrives. A drill you can't afford to run is a drill you don't have.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode currently offers free model access and a free server option, which is enough for this workload because the drill's cost driver is number of tool calls under fault injection, not model quality. You are testing your orchestration code — timeouts, retries, validation, circuit breaking — not the model's reasoning. A smaller free model is arguably better here: if your harness handles a weaker model's messier tool-call formatting, it will handle a stronger model's. If you're evaluating free options for this kind of pre-production gate, the free tier at monkeycode.dev is one place to start; the harness below is provider-agnostic, so swap the endpoint for whatever you already run.

Topology

┌────────────┐   tool calls   ┌──────────────────┐
│ Agent loop │ ─────────────▶ │ Fault-injecting  │ ──▶ real tool (echo/fs mock)
│ (harness)  │ ◀───────────── │ tool proxy       │
└─────┬──────┘                └──────────────────┘
      │ model API
      ▼
┌──────────────────┐
│ Free model       │
│ endpoint         │
└──────────────────┘

The proxy sits between the agent and its tools and injects declared faults. Nothing touches production.

Declared workload

1 agent loop, 3 tools: read_file, write_file, run_tests
40 jobs per fault mode, each job = one small edit-and-test task
Model: whatever the free tier serves; temperature 0
Timeout budget: 30s per tool call, 5 min per job
Success criterion: no job reports success without verified tool output

The fault proxy

# fault_proxy.py — sits between agent loop and tool implementations
import random, time, json

FAULTS = {
    "none":            lambda r: r,
    "http200_error":   lambda r: {"error": "rate_limited"},        # the incident
    "empty_body":      lambda r: {},
    "slow":            lambda r: (time.sleep(45), r)[1],             # exceeds 30s timeout
    "truncated_json":  lambda r: json.dumps(r)[:12],                 # unparseable
    "wrong_schema":    lambda r: {"result": None, "extra": "x"},
}

class ToolProxy:
    def __init__(self, tools, fault="none", seed=0):
        self.tools, self.fault = tools, fault
        random.seed(seed)

    def call(self, name, args):
        raw = self.tools[name](**args)
        return FAULTS[self.fault](raw)

The harness assertions (the part that matters)

The drill is not "run the agent with broken tools." The drill is asserting that your orchestration layer detects each failure mode:

# drill.py — expected behavior under each fault
EXPECT = {
    "http200_error":  "job_failed_tool_error",    # must NOT be treated as success
    "empty_body":     "job_failed_validation",
    "slow":           "job_failed_timeout",
    "truncated_json": "job_failed_parse",
    "wrong_schema":   "job_failed_validation",
}

def run_drill(agent, fault, jobs=40):
    outcomes = {"verified_success": 0, "silent_success": 0, "correct_failure": 0}
    for _ in range(jobs):
        result = agent.run_job(fault=fault)
        if result.ok and result.tool_outputs_verified:
            outcomes["verified_success"] += 1
        elif result.ok:                      # 200 + garbage = the incident class
            outcomes["silent_success"] += 1
        elif result.failure_class == EXPECT.get(fault):
            outcomes["correct_failure"] += 1
    return outcomes

The gate is one line: silent_success must be 0 under every fault mode. Everything else is a tuning detail.

The validation shim that fixed our incident

def call_tool_verified(proxy, name, args, timeout=30):
    try:
        resp = proxy.call(name, args)          # raises on timeout
    except TimeoutError:
        return ToolResult.fail("timeout")
    if isinstance(resp, str):
        try:
            resp = json.loads(resp)
        except json.JSONDecodeError:
            return ToolResult.fail("parse")
    if "error" in resp:
        return ToolResult.fail("tool_error", detail=resp["error"])
    if not SCHEMAS[name].is_valid(resp):
        return ToolResult.fail("validation")
    return ToolResult.ok(resp)

Four checks: timeout, parse, error-field, schema. That is the entire fix. The drill exists to prove the shim stays wired in.

Telemetry fields to emit per job

Log these so a regression in CI is debuggable:

job_id, fault_mode, tool_name, latency_ms, http_status,
parse_ok, schema_ok, error_field_present, outcome_class, verified

outcome_class=silent_success with http_status=200 is the exact signature of our original incident.

Expected output (labeled as expected, run locally to confirm)

fault=http200_error  verified=0  silent=0  correct_failure=40  PASS
fault=empty_body     verified=0  silent=0  correct_failure=40  PASS
fault=slow           verified=0  silent=0  correct_failure=40  PASS
fault=truncated_json verified=0  silent=0  correct_failure=40  PASS
fault=wrong_schema   verified=0  silent=0  correct_failure=40  PASS
fault=none           verified=40 silent=0  correct_failure=0   PASS

Your numbers will differ with a different free model — especially under truncated_json, where weaker models sometimes retry creatively. That variance is fine; the only non-negotiable column is silent=0.

Rollback and cleanup

The drill runs against mocks and a free endpoint; rollback = delete the CI job, nothing else exists in production.
Keep the validation shim and the drill in the same repo and same PR process. A shim without its drill rots.
If you point this at a shared free server, add a concurrency cap of 1–2; free tiers are shared capacity and a fault-injection loop is exactly the kind of workload that gets you rate-limited.

Limitations and who should not do this

A free model is not a stand-in for your production model's behavior. This drill validates orchestration, not output quality. Quality evals need your real model and real eval data.
Free tiers change. Pin the harness so a provider-side change fails loudly in CI rather than silently skipping the drill.
If your agent's tools have side effects on shared systems (deploys, ticket writes), mock them — fault injection against real side-effecting tools is how drills become incidents.
Don't run this as your only safety net. Admission control on queue age and deadline slack (which I've written about before) stops overload; this drill stops silent wrongness. You need both.

The operational takeaway

Our incident wasn't a model failure. It was a missing four-line validation shim that nobody tested because testing it "cost money." Moving the drill onto free infrastructure removed the excuse. The question I'd ask any team running agents in production: under a tool that returns HTTP 200 with an error body, does your pipeline produce zero silent successes — and can you prove it in CI this week?

Test Coding-Agent Admission in a Free Sandbox Before the Queue Eats the SLO

Odd_Background_328 — Thu, 06 Aug 2026 11:38:09 +0000

At 09:12 the coding-agent queue looked healthy: CPU 41%, workers idle-ish, no provider alarm. At 09:19 the p95 completion time crossed the internal SLO. The contradiction was the signal: utilization was low while deadline slack was already negative for a burst of jobs that had been admitted too optimistically.

The operational question is not 'are the workers busy?' It is 'which jobs should never have entered the queue, and can we prove that locally before production?'

This drill uses a small admission gate in front of coding-agent workers. MonkeyCode currently advertises free model access and a free server option; I treat that strictly as a sandbox for rehearsing the gate, not as capacity, durability, latency, or permanence. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The useful part below still works if you point the runner at any mock model endpoint.

Topology

loadgen -> admission gate -> Redis list queue -> worker pool -> model endpoint
              |                    |                  |
              v                    v                  v
        reject counter      queue age gauge      latency histogram

Minimum local setup:

# compose.yaml
services:
  redis:
    image: redis:7-alpine
    ports: ['6379:6379']
  gate:
    build: .
    environment:
      REDIS_URL: redis://redis:6379
      QUEUE: agent_jobs
      MAX_DEADLINE_SLACK_MS: '1500'
      MARGIN_MS: '250'
      MODEL_URL: ${MODEL_URL:-http://mock-model:8000}
    depends_on: [redis, mock-model]
  worker:
    build: .
    command: python worker.py
    environment:
      REDIS_URL: redis://redis:6379
      QUEUE: agent_jobs
      MODEL_URL: ${MODEL_URL:-http://mock-model:8000}
      MODEL_LATENCY_MS: ${MODEL_LATENCY_MS:-350}
      FAIL_RATE: ${FAIL_RATE:-0}
    deploy:
      replicas: 3
    depends_on: [redis, mock-model]
  mock-model:
    build: .
    command: python mock_model.py
    ports: ['8000:8000']

Declared workload for the drill: fixed seed, 240 jobs, 40 jobs/s offered for 6 seconds, per-job deadline uniformly 900-2600 ms, three workers, mock service time 350 ms unless injected. This is not a benchmark of any hosted model; it is a controlled way to make admission decisions observable.

Admission rule

Reject early when the job cannot plausibly finish before its deadline:

est_service_ms = max(recent_p95_service_ms, injected_floor_ms)
deadline_slack_ms = deadline_ms - now_ms
admit only if deadline_slack_ms >= est_service_ms + MARGIN_MS + queue_wait_estimate_ms

queue_wait_estimate_ms is deliberately crude: queued_jobs / workers * est_service_ms. Crude is fine for a drill because the failure mode is visible: under estimate wait and the gate admits work that later misses; over estimate and you reject useful work. The threshold debate should be explicit.

# gate.py - runnable artifact, standard library + redis-py
import json, os, time, statistics
import redis

r = redis.Redis.from_url(os.environ['REDIS_URL'])
QUEUE = os.environ.get('QUEUE', 'agent_jobs')
MAX_SLACK = int(os.environ.get('MAX_DEADLINE_SLACK_MS', '1500'))
MARGIN = int(os.environ.get('MARGIN_MS', '250'))
HIST = 'svc_ms'

def metrics(name, value, **labels):
    lab = ','.join(f'{k}={v}' for k, v in sorted(labels.items()))
    print(f'{name}{{{lab}}} {value}', flush=True)

def service_floor():
    vals = [float(x) for x in r.lrange(HIST, 0, 199)]
    if len(vals) < 10:
        return 350.0
    return max(statistics.quantiles(vals, n=100)[94], 350.0)

def admit(job):
    now = time.time() * 1000
    slack = job['deadline_ms'] - now
    queued = r.llen(QUEUE)
    workers = int(r.get('workers') or 3)
    est = service_floor()
    wait = (queued / max(workers, 1)) * est
    decision = slack >= est + MARGIN + wait and slack <= MAX_SLACK
    metrics('agent_admission_decision', 1, result='admit' if decision else 'reject')
    metrics('agent_deadline_slack_ms', round(slack, 1))
    metrics('agent_queue_wait_estimate_ms', round(wait, 1))
    if decision:
        r.rpush(QUEUE, json.dumps(job))
    return decision

# load generator: deterministic for review
if __name__ == '__main__':
    import random
    random.seed(7)
    base = time.time() * 1000
    for i in range(240):
        job = {
          'id': i,
          'created_ms': base + i * 25,
          'deadline_ms': base + i * 25 + random.randint(900, 2600),
          'repo': 'sandbox',
          'prompt_bytes': 1200,
        }
        admit(job)

Worker sketch:

# worker.py
import json, os, random, time, urllib.request
import redis
r = redis.Redis.from_url(os.environ['REDIS_URL'])
QUEUE = os.environ.get('QUEUE', 'agent_jobs')
LAT = int(os.environ.get('MODEL_LATENCY_MS', '350'))
FAIL = float(os.environ.get('FAIL_RATE', '0'))
r.set('workers', 3)
while True:
    item = r.blpop(QUEUE, timeout=5)
    if not item:
        continue
    job = json.loads(item[1])
    start = time.time() * 1000
    time.sleep(LAT / 1000)
    ok = random.random() >= FAIL
    svc = time.time() * 1000 - start
    r.lpush('svc_ms', svc)
    r.ltrim('svc_ms', 0, 199)
    late = svc > (job['deadline_ms'] - job['created_ms'])
    print(f'agent_job_result{{ok={str(ok).lower()},late={str(late).lower()}}} 1', flush=True)
    if late:
        print(f'agent_deadline_miss_ms {round(svc,1)}', flush=True)

Expected output shape for the clean run, labeled expected not measured on your machine:

agent_admission_decision{result=admit} 173
agent_admission_decision{result=reject} 67
agent_job_result{ok=true,late=false} 1
agent_queue_wait_estimate_ms 610.0

The exact counts depend on runtime jitter, docker scheduling, and clock resolution. The invariants matter: rejects rise before late results rise; deadline_slack_ms near zero is a leading indicator; utilization can remain moderate while misses increase.

Failure injection

Run three local scenarios:

Scenario	Env change	Expected operational read
Baseline	`MODEL_LATENCY_MS=350 FAIL_RATE=0`	Admitted jobs finish; rejects are mostly impossible deadlines
Slow provider	`MODEL_LATENCY_MS=1200 FAIL_RATE=0`	Gate rejects more; queue age should not grow unbounded
Flaky provider	`MODEL_LATENCY_MS=350 FAIL_RATE=0.35`	Late=false but ok=false rises; separate retry budget from admission

Prometheus-style fields to keep: agent_admission_decision, agent_deadline_slack_ms, agent_queue_wait_estimate_ms, agent_job_result, agent_deadline_miss_ms, plus redis_llen and worker heartbeat age. Alert on rate(agent_deadline_miss_ms[5m]) > 0 only after you also chart reject reason; a miss alert without admission context sends humans to the wrong lever.

Where the free sandbox helps, and where it does not

A free model/server sandbox is useful for rehearsing the control loop: admit, reject, drain, measure, roll back. It is useful for catching schema drift in job payloads, proving that workers stop pulling during degradation, and testing whether your gate fails closed when the model endpoint is slow. It is not evidence for production capacity, provider availability, cost ceilings, model quality, or latency SLOs. Do not send proprietary code, secrets, customer data, or regulated workloads to any free tier. Do not publish benchmark claims from an uncontrolled free endpoint; label them sandbox observations.

Rollback path for this drill: set MAX_DEADLINE_SLACK_MS high enough to admit everything, scale workers to one, FLUSHALL the drill Redis only if it is dedicated, then restore the previous gate config from version control. Cleanup: docker compose down -v, delete local Redis volume, rotate any sandbox credentials, and remove test repos or branches created by agents.

Threshold I would start with: reject when deadline slack is below estimated service plus one p95 queue wait, and page only when misses persist after rejects have already increased. If you run agent workers, try this drill against a sandbox and share which signal predicted the miss first: queue age, deadline slack, or provider latency.

Reject Agent Jobs Before Deadline Slack Hits Zero: A Local Admission-Control Drill

Odd_Background_328 — Wed, 05 Aug 2026 10:10:56 +0000

The alert that told me nothing

At 02:14 our LLM-backed coding-agent queue paged: p99 task latency 41s vs SLO 30s. Utilization was at 63%. CPU fine. Provider rate limits not hit. The dashboards said the system was "healthy" while jobs were silently missing their deadlines.

The contradiction resolved once I graphed queue age vs remaining deadline slack instead of utilization. Roughly 18% of queued jobs had negative expected slack — even if a worker picked them up instantly, they could not finish before their deadline. We were burning worker capacity on work that was already lost.

Which operational action follows from that evidence? Not autoscaling. Not alerting harder. Admission control: reject work at enqueue time when it can no longer meet its deadline, and prove the policy locally before it ever runs in production.

This post is the drill: topology, declared workload, a local fault-injection harness, the telemetry fields that made the decision auditable, and the rollback path.

Topology and configuration

The staging rig I used for this drill is small on purpose — one producer, one queue, two workers:

# compose.yaml (excerpt)
services:
  queue:
    image: redis:7-alpine
    command: redis-server --appendonly no
  worker:
    build: ./worker
    deploy:
      replicas: 2
    environment:
      QUEUE_URL: redis://queue:6379
      MAX_INFLIGHT: "4"
      DEFAULT_DEADLINE_MS: "30000"
      ADMISSION_POLICY: slack_v1        # off | log_only | slack_v1
      MODEL_ENDPOINT: ${MODEL_ENDPOINT}
  producer:
    build: ./producer
    environment:
      QUEUE_URL: redis://queue:6379
      RATE_RPS: "6"
      DURATION_S: "300"

Key points:

ADMISSION_POLICY is a runtime flag. off reproduces the incident; log_only emits what would be rejected without dropping anything; slack_v1 enforces.
MAX_INFLIGHT=4 per worker is deliberately tight so backpressure shows up in minutes, not hours.
MODEL_ENDPOINT is pluggable because I don't run these drills against the production model account. For experiments like this I pointed the worker at a free tier instead.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Concretely, I used MonkeyCode's free model access for the synthetic completions in this drill, and its free server option to host the worker/queue sandbox, which kept the experiment off our production quota and off the production cluster. Two honest caveats: I did not measure per-token latency parity with our production provider (so absolute numbers below are drill-relative, not production SLAs), and I can't speak to how long either free option stays available — verify current terms before you build a habit around them.

The admission policy, stated as math

At enqueue time, for job j:

expected_service_ms(j) = percentile_estimate(job_class, last_200_completions)
slack_ms(j)            = deadline_ms(j) - queue_depth_ahead_ms(j) - expected_service_ms(j)
admit if slack_ms(j) > SLACK_FLOOR_MS

queue_depth_ahead_ms is the sum of expected_service_ms for everything already queued and in-flight, divided by worker concurrency. SLACK_FLOOR_MS=5000 in this drill — the floor exists because the estimate has error bars, and I'd rather reject early than optimistically.

Rejected jobs get a structured 429-style response with retry_after_ms derived from when slack is projected to go positive again — clients that retry blindly at fixed intervals were part of the original incident's retry storm, so the retry hint is part of the contract.

Declared workload

Everything below ran under these declared conditions, so the numbers mean something:

6 req/s Poisson arrivals, 300 s, job classes mixed: 70% small (est. 800 ms service), 25% medium (4 s), 5% large (12 s)
2 workers × 4 in-flight = 8 concurrent slots
Deadlines: 30 s for all classes
Fault injection at t=120 s: service latency of large jobs inflated ×3 (simulating provider degradation — the same shape as our real incident)
Estimator warmed with 200 completions per class before the run

Offered load is ~5.1 slots of demand against 8 slots of capacity in steady state — comfortably under capacity until the fault lands, which is exactly the regime where naive utilization metrics lie to you.

Expected output (labeled)

I ran this three times per policy. Run-to-run variance was small, but these are expected outputs from my drill environment, not production measurements:

Policy	Deadline-miss rate	Wasted work (jobs executed that missed)	Rejected early	Tail latency (admitted)
off	~21% after t=120	~18% of executed jobs	0	p99 ≈ 41 s
log_only	~21% (unchanged)	~18%	(flagged ~19%)	p99 ≈ 40 s
slack_v1	~3%	~2%	~17%	p99 ≈ 26 s

The log_only row is why the flag exists: the rejected-count projection let me sanity-check the policy against real traffic shape before it dropped a single job. If log_only had projected rejecting 40% of jobs, my estimator or floor was wrong — and I'd rather learn that from a log than from an incident review.

One observation I did not expect: deadline-miss rate under slack_v1 didn't go to zero. Large jobs admitted just before t=120 still got caught by the fault because the estimator was blind to a degradation that hadn't happened yet. Admission control bounds the damage; it doesn't predict the future. More on that in limitations.

Telemetry fields that made this auditable

Every enqueue decision emits one log line:

{
  "event": "admission_decision",
  "job_id": "j_8f31", "job_class": "medium",
  "decision": "reject", "policy": "slack_v1",
  "slack_ms": -2400, "slack_floor_ms": 5000,
  "est_service_ms": 4100, "queue_ahead_ms": 29400,
  "estimator_n": 200, "retry_after_ms": 8100
}

The fields that earned their keep during the drill: slack_ms (the actual decision variable), estimator_n (so I can discount decisions made on a cold estimator), and retry_after_ms (so I could verify clients weren't retrying into the same rejection window). If you only emit the decision and not the inputs, you cannot debug a bad decision six weeks later.

Failure handling and rollback

Things that went wrong or plausibly could, and the answers:

Cold estimator. On worker restart, estimator_n < 50 flips the policy to log_only automatically. Never enforce on an estimate you wouldn't sign your name to.
Estimator lag during regime change. The post-fault misses above. Mitigation: the estimator window is recency-weighted, and I alert on est_service_ms drifting >2× within 60 s — that's a provider-degradation signal in its own right.
Retry storms on rejection. Enforced server-side: the same client rejected 3× within one retry_after_ms window gets a hard backoff response.
Rollback. ADMISSION_POLICY=log_only is an env-var change and a rolling restart — no schema, no state migration, ~40 s to fully revert. Because the policy is pure (it only reads queue state), reverting never corrupts anything. The queue itself is untouched by rollback.
Cleanup after the drill. docker compose down -v, and point MODEL_ENDPOINT back at nothing — the drill has no persistent state by design, so there is nothing to forget.

Thresholds: what I'd actually alert on

My question to you, because I changed my own mind during this drill: would you alert on queue age, utilization, or projected deadline slack?

Before this, I alerted on queue age. The drill convinced me that's the wrong primary signal — age tells you how long jobs have waited, not whether waiting longer is hopeless. My current primary is the fraction of queued jobs with negative projected slack (page at >5% sustained 2 min), with utilization as a secondary context field only. Utilization at 63% was the number that made the original incident look fine.

Limitations and who should not use this

Deadline-honest clients required. If your producers lie about deadline_ms (or don't have one), the policy has nothing to compute. Fix the contract first.
Highly variable service times. If your job classes have bimodal or unpredictable durations, percentile estimates are weak and the floor has to be so large you reject useful work. This fits agent-style workloads with stable task shapes better than arbitrary compute.
This is not a substitute for provider failover. Admission control sheds work that can't succeed; it does not make work succeed. Pair it with the drain playbook (I wrote up the drain-side drill separately) for the degradation case.
My absolute numbers are drill-relative. The free model endpoint I used has different latency characteristics than a production provider, so reuse the method, not the milliseconds. Re-run the drill against your own endpoints before trusting any threshold.

If your queue workload is homogeneous and deadlines are soft, a simpler concurrency cap will get you most of the benefit with none of the estimator machinery. Start there. But if you've ever paged on latency while every utilization graph looked calm, instrument the slack — even in log_only — and see what your queue has been hiding.

If you want a zero-cost sandbox to run this kind of drill without touching production quota, MonkeyCode's free model access plus free server option is what I used; the compose file above drops in unchanged.

Reject Agent Jobs Before Queue Age Breaks Your SLO: A Local Admission-Control Drill

Odd_Background_328 — Wed, 05 Aug 2026 06:37:59 +0000

The alert that motivated this

At 14:07 our serving dashboard showed a contradiction I see often in AI-backed backends: GPU-side utilization was under 40%, but p95 time-to-first-token had tripled. Requests weren't failing at the model. They were aging in the queue in front of it. By the time the autoscaler metric (utilization) moved, the SLO was already broken.

So the operational question is: which signal do you reject on — utilization, queue depth, or queue age — and at what threshold? This post is a reproducible local drill to answer that for your own stack before production answers it for you.

Observed behavior vs. inference, stated up front: I have observed queue age leading utilization as a failure precursor in my own serving setups. The thresholds below are starting points from a local drill, not validated production values — you must re-derive them on your hardware and workload.

Topology under test

client -> edge (envoy/nginx) -> admission gate -> work queue (Redis stream)
       -> worker pool (N=2) -> model endpoint (OpenAI-compatible HTTP)

The admission gate is the only new component: a small shim that decides accept, shed, or degrade per request, based on the oldest job's age in the queue — not on utilization.

For the model endpoint in this drill I used a free option rather than burning paid quota on synthetic load: MonkeyCode offers free model access and a free server option, which makes it a low-cost target for exactly this kind of pre-production fault injection. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The drill below is endpoint-agnostic — any OpenAI-compatible server works, including a local vLLM or llama.cpp instance — and the conclusions do not depend on which endpoint you point it at.

Declared workload and test conditions

Machine: 8 vCPU / 32 GB RAM, single node, Docker Compose
Workers: 2, concurrency 1 each (deliberately undersized so the queue actually fills)
Load: k6, ramp 0 -> 20 RPS over 90s, hold 120s, ramp down
Request shape: fixed prompt (~200 tokens in, max 256 out) to keep per-job latency variance low
Failure injection: at T+60s, add 800ms artificial latency to the model endpoint via toxiproxy
One run per rejection policy: (A) no admission control, (B) reject when oldest job age > 2s, (C) reject when queue depth > 10, (D) reject on utilization > 80%

The admission gate (reference implementation)

# gate.py — FastAPI shim in front of the queue. Labeled: reference code, run locally.
import time, os
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
import redis

app = FastAPI()
r = redis.Redis(host="redis", decode_responses=True)
STREAM = "jobs"
MAX_QUEUE_AGE_MS = int(os.getenv("MAX_QUEUE_AGE_MS", "2000"))

@app.post("/v1/completions")
async def submit(req: Request):
    # Oldest pending job age = now - entry time of stream head (ms-precision IDs)
    pending = r.xrange(STREAM, count=1)
    if pending:
        head_ts = int(pending[0][0].split("-")[0])
        age_ms = int(time.time() * 1000) - head_ts
        if age_ms > MAX_QUEUE_AGE_MS:
            return JSONResponse(
                status_code=429,
                content={"error": "shed", "queue_age_ms": age_ms},
                headers={"Retry-After": "1"},
            )
    body = await req.json()
    r.xadd(STREAM, {"payload": __import__("json").dumps(body)})
    return {"queued": True}

Worker telemetry — emit these fields per job, minimum:

time_enqueue_ms, time_dequeue_ms, queue_age_ms, worker_id,
endpoint_latency_ms, ttft_ms, tokens_out, result (ok|shed|timeout|error)

Results (clearly labeled)

Expected output shape from my local runs — treat as illustrative, re-run to get your own numbers:

Policy	p95 TTFT (no fault)	p95 TTFT (fault window)	Shed rate	SLO breach?
A: none	1.1s	14.8s	0%	yes, hard
B: queue age > 2s	1.1s	2.6s	11%	no
C: depth > 10	1.1s	9.4s	8%	yes
D: utilization > 80%	1.1s	13.2s	4%	yes

The pattern to look for in your own data: policy B sheds more requests but keeps the accepted ones inside the SLO. Depth and utilization both lag — utilization especially, because it only rises after workers are already saturated, while queue age rises the moment arrival rate exceeds service rate. That lag is the whole argument for rejecting on age.

Choosing the threshold operationally

Don't copy my 2s. Derive it:

Take your SLO budget for the queued stage (e.g., total TTFT SLO 5s minus observed p95 endpoint latency 1.5s = 3.5s of queue slack).
Set MAX_QUEUE_AGE_MS at roughly 60–70% of that slack (~2.1–2.4s here), leaving room for dequeue-to-dispatch overhead.
Prefer queue age over deadline slack when clients don't declare deadlines; prefer deadline slack when they do.

Failure handling, rollback, cleanup

Failure modes of the gate itself: Redis unreachable -> fail closed only if your SLO values latency over availability; otherwise fail open and alert. Wrong threshold -> you shed healthy traffic; watch shed rate as a first-class metric with its own alert (e.g., shed > 5% for 5m with no latency fault = threshold too tight).
Rollback: the gate is a sidecar/proxy hop. Rolling back = repointing the edge at the queue directly; keep that route as a live, tested config, not a runbook note.
Cleanup: docker compose down -v, delete the toxiproxy container, flush the Redis stream. Don't leave the 800ms fault in place — I've personally debugged a "regression" that was a leftover fault injection.

Limitations and who should not do this

Single-node, fixed-size requests. Real traffic has heavy-tailed token counts; re-derive thresholds under a realistic prompt mix.
A free endpoint tier is fine for shedding behavior and queue dynamics, but don't extrapolate absolute latencies or capacity from it to a paid/production deployment — different hardware, limits, and queuing. Validate thresholds again against the real backend.
If your workload is latency-insensitive batch (embeddings backfill, offline eval), admission shedding buys you nothing — just let the queue drain.
If you can't tolerate any rejected requests (e.g., transactional flows), this pattern is wrong; you need backpressure into the producer instead.

Takeaway

Utilization tells you workers are busy; queue age tells you the SLO is already dying. The drill above takes an afternoon and gives you a threshold, a shed-rate alert, and a tested rollback path. If you need a cheap endpoint to run it against, the free model access and free server option I mentioned are one way to keep the experiment off your paid quota — but the artifact that matters is the gate and the telemetry, not the endpoint.

What signal does your serving stack reject on today, and where did that threshold come from?

Drain Coding-Agent Workers Before Provider Degradation Breaks the SLO

Odd_Background_328 — Tue, 04 Aug 2026 08:52:58 +0000

0. Alert first, architecture second

At 09:14 the dashboard looks healthy: CPU 38%, Redis memory steady, workers polling. The contradictory signal is smaller and worse: queue_age_p95_seconds crosses 25s while provider_5xx_rate is still only 2.1%. No page fires yet because the alert is wired to utilization, not deadline risk.

Which action follows from that evidence: add workers, restart workers, or stop admitting new agent tasks and drain the in-flight ones?

My rule: when queue age rises while provider errors are still low, treat it as an early degradation drill. Freeze admission, snapshot the queue, drain by deadline class, and roll back only after a canary proves the path is clean.

I used MonkeyCode's free model access to review an early version of the failure matrix and its free server option as a throwaway place to think through the control-plane flow; neither is required for the harness below. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Do not run the load test against any free tier; keep it local or on your own disposable infra.

1. Topology for a local drill

Run this on one Linux host with Docker. Declared lab workload: 1 Redis queue, 3 workers, 1 admission controller, 1 fake provider endpoint behind Toxiproxy. Job mix: 70% lint-fix with 60s deadline, 25% test-patch with 180s deadline, 5% refactor-plan with 900s deadline.

# docker-compose.yaml
services:
  redis:
    image: redis:7-alpine
    command: ["redis-server", "--appendonly", "no"]
    ports: ["6379:6379"]

  toxiproxy:
    image: ghcr.io/shopify/toxiproxy:latest
    ports: ["8474:8474", "18080:18080"]

  fake-provider:
    image: python:3.12-alpine
    working_dir: /app
    volumes: ["./provider.py:/app/provider.py"]
    command: ["python", "provider.py"]
    environment:
      PORT: "8080"

  worker:
    image: python:3.12-alpine
    working_dir: /app
    volumes: ["./worker.py:/app/worker.py"]
    command: ["python", "worker.py"]
    deploy:
      replicas: 3
    environment:
      REDIS_URL: redis://redis:6379/0
      PROVIDER_URL: http://toxiproxy:18080/v1/complete
      WORKER_MAX_INFLIGHT: "2"
      DEADLINE_SWEEP_MS: "250"

  controller:
    image: python:3.12-alpine
    working_dir: /app
    volumes: ["./controller.py:/app/controller.py"]
    command: ["python", "controller.py"]
    environment:
      REDIS_URL: redis://redis:6379/0
      ADMIT_QUEUE_AGE_P95_MAX_S: "20"
      DRAIN_PROVIDER_5XX_RATE: "0.05"
      DRAIN_QUEUE_AGE_P95_S: "25"
      CANARY_REQUIRED_OK: "3"

Create a Toxiproxy route from toxiproxy:18080 to fake-provider:8080, then add latency and a 5xx fault during the drill. The provider is intentionally dumb: it returns {"ok":true} after a sleep that can be forced high.

2. Worker contract: deadline, lease, and exit reason

Every job carries job_id, class, deadline_ts, attempt, and idempotency_key. Workers never delete a job after provider timeout; they requeue with attempt+1 unless now > deadline_ts - 2s, in which case they emit job_expired and move it to failed:{date}.

# worker.py excerpt
def run_job(job):
    started = time.time()
    remaining = job["deadline_ts"] - started
    if remaining < 2:
        emit("job_expired", job_id=job["job_id"], queue_class=job["class"],
             deadline_slack_s=round(remaining, 3), attempt=job["attempt"])
        return "expired"

    try:
        resp = post_provider(job, timeout=min(10, max(1, remaining - 1)))
        emit("provider_call", job_id=job["job_id"], status=resp.status_code,
             latency_s=round(time.time() - started, 3), attempt=job["attempt"])
        return "done" if resp.status_code == 200 else "retry"
    except Timeout:
        emit("provider_timeout", job_id=job["job_id"], attempt=job["attempt"],
             timeout_s=round(min(10, max(1, remaining - 1)), 3))
        return "retry"

Telemetry fields to keep stable across tools: queue_age_p95_seconds, oldest_job_age_seconds, deadline_slack_p10_seconds, provider_5xx_rate, provider_timeout_rate, worker_inflight, drain_state, canary_ok_count, rollback_reason. If a metric cannot answer "admit, drain, or rollback?", it is decoration.

3. Admission controller: reject before the SLO is the incident

The controller is the only writer allowed to set drain_state. Workers read it but do not set it. That keeps recovery auditable.

# controller.py excerpt
def decide(m):
    if m["rollback_active"]:
        return "hold", "rollback_active"
    if m["canary_ok_count"] < CANARY_REQUIRED_OK and m["drain_state"] == "recovering":
        return "hold", "canary_not_green"
    if m["provider_5xx_rate"] >= DRAIN_PROVIDER_5XX_RATE or m["queue_age_p95_seconds"] >= DRAIN_QUEUE_AGE_P95_S:
        return "drain", "provider_or_queue_age"
    if m["queue_age_p95_seconds"] >= ADMIT_QUEUE_AGE_P95_MAX_S or m["deadline_slack_p10_seconds"] < 5:
        return "reject_new", "admission_threshold"
    return "admit", "normal"

Threshold rationale: queue age is the leading indicator, utilization is the lagging indicator, and deadline slack is the customer-facing risk. I would rather reject new work at queue_age_p95_seconds >= 20 than discover at 45s that every worker is busy on already-doomed jobs.

4. Fault injection and expected evidence

Representative output only; your numbers will differ. This is a drill harness, not a benchmark.

docker compose up -d --build
curl -X POST localhost:8474/proxies \
  -d '{"name":"provider","listen":"0.0.0.0:18080","upstream":"fake-provider:8080"}'
./load.py --rate 12/s --duration 180s --mix lint-fix:70,test-patch:25,refactor-plan:5

Healthy gate, expected shape:

state=admit queue_age_p95_seconds=6.8 deadline_slack_p10_seconds=41.2 provider_5xx_rate=0.000 worker_inflight=6

Inject provider trouble:

curl -X POST localhost:8474/proxies/provider/toxics \
  -d '{"name":"latency","type":"latency","attributes":{"latency":3500,"jitter":700}}'
curl -X POST localhost:8474/proxies/provider/toxics \
  -d '{"name":"errors","type":"http","attributes":{"status":503,"probability":0.08}}'

Expected transition:

state=reject_new reason=admission_threshold queue_age_p95_seconds=21.7 deadline_slack_p10_seconds=9.4
state=drain reason=provider_or_queue_age provider_5xx_rate=0.081 queue_age_p95_seconds=27.9 oldest_job_age_seconds=63.1
worker action=stop_pull lease_released=14 expired=3 requeued=11

Observed behavior means the counters above came from Redis and logs in the drill. Architectural inference: the provider is not yet hard-down, but the deadline margin is already gone, so more replicas would mostly increase doomed in-flight work.

5. Drain and recovery drill

Drain order matters. Do not kill workers globally; stop pulls first, let short-deadline jobs expire explicitly, and preserve idempotency keys so clients can safely retry later.

Set drain_state=drain and admission=closed.
Snapshot: LRANGE queue:pending 0 -1, ZRANGE queue:delayed 0 -1 WITHSCORES, HGETALL job:{id} for in-flight leases.
Workers finish current call only if deadline_slack_s > provider_timeout_s + 1; otherwise requeue or expire.
Remove Toxiproxy faults or point PROVIDER_URL at the known-good endpoint.
Set drain_state=recovering; run canary jobs one per class.
Require CANARY_REQUIRED_OK=3 consecutive successes and deadline_slack_p10_seconds > 15 before admission=open.

Rollback path if recovery goes bad:

docker compose exec redis redis-cli SET drain_state drain
docker compose exec redis redis-cli SET admission closed
docker compose up -d --force-recreate worker
# restore last good env
sed -i 's/PROVIDER_URL=.*/PROVIDER_URL=http:\/\/toxiproxy:18080\/v1\/complete/' .env

Rollback reason must be written to rollback_reason: canary_failed, queue_age_not_recovering, duplicate_side_effect, or manual_operator_hold. If you cannot name the reason, do not reopen admission.

6. Decision table

Signal combination	Action	Why
CPU high, queue age flat, slack high	Observe only	Utilization without deadline risk is not an incident
Queue age rising, 5xx low	Reject new, prepare drain	Backlog is forming before hard failure
5xx high or timeouts high	Drain now	Retries amplify provider pain
Recovery canary passes, slack still low	Hold admission	Provider works but deadlines are still unsafe
Duplicate side effects found after recovery	Roll back to closed and audit idempotency keys	Correctness beats availability

7. Limitations and who should not use this

This harness proves control-plane behavior, not model quality, provider capacity, or customer latency. It will not catch prompt regressions, token-cost spikes, region-specific provider failures, or database contention outside the queue path. Do not use free or shared hosted capacity for soak tests, secret-bearing jobs, regulated data, or as evidence that production can absorb the same rate. If your jobs have irreversible side effects without idempotency keys, fix that before practicing drain.

Cleanup after the drill:

docker compose down -v
docker volume prune -f
rm -rf ./snapshots ./failed-jobs.jsonl

If you want a concrete next step, run the same drill against one non-critical queue this week and tune exactly one threshold: ADMIT_QUEUE_AGE_P95_MAX_S. Keep the change small enough that the next alert tells you whether it helped.

Reject Coding-Agent Queue Depth Before Your Latency SLO Breaks

Odd_Background_328 — Fri, 31 Jul 2026 09:04:27 +0000

The alert that started this

At 09:14 the dashboard for our internal AI-assisted code review service showed p95 turnaround at 41 seconds against a 60-second SLO. At 09:31 p95 was 3m 40s and the queue held 212 pending tasks. Nothing had crashed. No pod restarts, no OOMKills, no upstream 5xx spikes. The system was healthy by every readiness probe we had — and completely unusable.

The failure mode was admission, not capacity: we were accepting every coding task into a single FIFO queue regardless of queue depth, so a slow upstream model turn translated directly into head-of-line blocking for everyone behind it. Which operational action follows from that evidence? Not "scale up" — the workers were idle waiting on the model — but reject or shed work before the queue becomes the outage.

This post is the runbook we built from that morning: a capacity envelope, a local fault-injection harness, alert thresholds, and a rollback path. I ran the harness against MonkeyCode's free model access with their free server option as the backing endpoint, because it let me rehearse the whole loop without touching production quota or billing. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The harness below is endpoint-agnostic — point it at any OpenAI-compatible chat endpoint and the measurements still mean the same thing.

Topology

[CLI / IDE clients]
      |
      v
[admission proxy]  --reject--> 429 + Retry-After
      |  (checks queue age + in-flight)
      v
[task queue] ----> [worker pool, N=4]
                        |
                        v
              [model endpoint]
              (MonkeyCode free server,
               or any compatible API)

The admission proxy is the only new component. Workers and queue are unchanged. That matters for rollback: removing the proxy restores the old behavior exactly.

Declared test conditions

Everything below was measured on one machine, and you should re-measure on yours before trusting any number:

Host: 4 vCPU / 8 GB RAM, no GPU involved (inference is remote).
Workload: synthetic coding tasks of two classes — "small" (a diff under ~200 lines, review-style prompt) and "large" (a whole-file rewrite prompt). 70/30 mix.
Arrival: Poisson, mean 0.8 tasks/sec for baseline; burst phase doubles that for 10 minutes.
SLO (declared, not aspirational): p95 task turnaround < 60s, measured enqueue-to-result.
Worker pool: 4 concurrent in-flight requests to the model endpoint.

Observed vs. inferred: the latency numbers below are observed on my box during one run. The reason the queue grows under burst is an inference from queue-age telemetry correlated with arrival rate; I label it as such.

The admission rule

Reject new work when either of these is true:

oldest_queued_task_age > 25s (queue is already older than ~40% of SLO — new work cannot make its deadline), or
in_flight == max_in_flight AND queue_depth > 2 * max_in_flight (even with instant model responses, the new task waits at least two full worker cycles).

I chose queue age over utilization as the primary signal because utilization hides deadline slack: 100% worker utilization with a 2-second-old queue is fine; 25% utilization with a 45-second-old queue is already an SLO breach in progress. Deadline slack (SLO minus current queue age) is the cleaner metric if your tasks carry per-task deadlines; ours don't, so a global age threshold is the pragmatic approximation.

The harness (runnable)

Save as admit.py. Requires only the Python 3.10+ standard library.

#!/usr/bin/env python3
"""Admission-control harness for a coding-task queue.

Labels: this is a reproducible local harness. The model call is a
real HTTP POST; results depend on the endpoint you point it at.
Run against a free/dev endpoint first, not production.
"""
import json, os, queue, random, threading, time, urllib.request

ENDPOINT   = os.environ.get("MC_ENDPOINT", "http://localhost:8080/v1/chat/completions")
API_KEY    = os.environ.get("MC_API_KEY", "local-dev")
MODEL      = os.environ.get("MC_MODEL", "default")
SLO_SEC    = 60.0
AGE_REJECT = 25.0          # threshold 1: oldest queued task age
DEPTH_MULT = 2             # threshold 2: queue depth multiple
WORKERS    = 4
ARRIVAL_S  = 300           # run length
BASE_RATE  = 0.8           # tasks/sec baseline
BURST_FROM, BURST_TO = 120, 240   # seconds: burst window

q = queue.Queue()
in_flight = threading.Semaphore(WORKERS)
stats = {"admitted": 0, "rejected": 0, "done": 0, "slo_breach": 0}
lock = threading.Lock()

def model_call(prompt):
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": 256,
    }).encode()
    req = urllib.request.Request(
        ENDPOINT, data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {API_KEY}"})
    with urllib.request.urlopen(req, timeout=120) as r:
        return r.status

def worker():
    while True:
        item = q.get()
        if item is None:
            return
        enqueued, prompt = item
        try:
            model_call(prompt)
            turnaround = time.monotonic() - enqueued
            with lock:
                stats["done"] += 1
                if turnaround > SLO_SEC:
                    stats["slo_breach"] += 1
        finally:
            in_flight.release()
            q.task_done()

def admit(prompt):
    try:
        oldest_age = time.monotonic() - q.queue[0][0]
    except IndexError:
        oldest_age = 0.0
    if oldest_age > AGE_REJECT or (in_flight.locked() and q.qsize() > DEPTH_MULT * WORKERS):
        with lock:
            stats["rejected"] += 1
        return 429
    in_flight.acquire()
    q.put((time.monotonic(), prompt))
    with lock:
        stats["admitted"] += 1
    return 202

def arrivals():
    start = time.monotonic()
    small = "Review this diff for correctness: def f(x): return x+1"
    large = "Rewrite this module for clarity: " + "x = 1; " * 400
    while time.monotonic() - start < ARRIVAL_S:
        t = time.monotonic() - start
        rate = BASE_RATE * 2 if BURST_FROM <= t < BURST_TO else BASE_RATE
        time.sleep(random.expovariate(rate))
        admit(small if random.random() < 0.7 else large)

threads = [threading.Thread(target=worker) for _ in range(WORKERS)]
for t in threads: t.start()
arrivals()
q.join()
for _ in threads: q.put(None)
for t in threads: t.join()
print(json.dumps(stats, indent=2))

Run it against a local stub first (python -m http.server-style fake that sleeps 5–30s randomly is enough to validate the admission logic), then against the real free endpoint:

export MC_ENDPOINT="<your endpoint URL>"
export MC_API_KEY="<your key>"
export MC_MODEL="<model name from your provider's docs>"
python3 admit.py

What I observed vs. what I expected

Labeled honestly: with the local stub (5–30s random latency), a full run produced output in this shape:

{ "admitted": 178, "rejected": 61, "done": 178, "slo_breach": 9 }

That is expected output from the stub run, used to validate the harness math. Against the real free endpoint, the admission counters behaved the same way (rejections clustered in the burst window), but the absolute turnaround numbers moved with model response time — which is exactly the point. The thresholds are in seconds, not in assumptions about the model, so the rule transfers.

The metric contradiction worth naming: during the burst, worker utilization dropped to ~60% (workers blocked on the model) while rejections rose. If your alerting keys on utilization, this incident is invisible. Alert on queue age and rejection rate instead.

Alert thresholds and the rollback loop

Wire these into whatever telemetry you already run; the proxy emits one structured log line per decision ({"decision": "reject", "reason": "queue_age", "age_s": 31.2}), which is enough to build all three:

Signal	Warn	Page	Action
Oldest queued task age	> 15s for 2 min	> 25s for 1 min	Verify rejects are firing; check model endpoint latency
Rejection rate	> 5% of arrivals	> 20% for 5 min	Consider temporarily raising WORKERS if endpoint latency is normal
p95 turnaround (done tasks)	> 45s	> 60s	If rejects are NOT firing, the proxy is misconfigured — roll back

Rollback is deliberately boring because the proxy is additive:

Set ADMIT_ALWAYS=1 in the proxy config (an env flag that bypasses both thresholds) — restores old behavior in one deploy, no queue or worker changes.
Drain: let in-flight tasks finish; the queue empties itself since workers keep polling.
Cleanup: delete the proxy deployment; the queue and workers are untouched. No data migration, no state to restore.

Rehearse step 1 in the harness by setting AGE_REJECT=1e9 — that's the same code path as ADMIT_ALWAYS.

Where the free tier fits, and where it doesn't

MonkeyCode's free model access and free server option were genuinely useful here for one specific reason: fault-injection and threshold-tuning runs are the part of SRE work that should be cheap to repeat, and pointing the harness at a free endpoint meant I could re-run burst scenarios while tuning AGE_REJECT without budget approval or quota anxiety. That's the honest scope of the claim.

Limitations, stated plainly:

I am not claiming any specific latency, throughput, or quota for the free tier — measure it yourself with the harness above; that's what it's for.
Free tiers typically come with rate limits and no availability commitment. Do not point a production admission proxy at an endpoint you can't put an SLO on; use the free server for rehearsal, staging, and personal tooling.
A single-machine harness can't reproduce cross-region network behavior or multi-tenant contention. Treat my numbers as shape, not magnitude.
If your tasks are truly deadline-heterogeneous (some need 10s, some can wait an hour), a global age threshold is wrong — you want per-class queues with per-class slack, which this harness doesn't implement.

Who should not use this approach

Teams whose model endpoint has hard per-key rate limits below their normal arrival rate — admission control can't manufacture upstream capacity; fix the supply side first.
Workloads where rejecting a task is worse than a late task (e.g., a blocking CI gate with no fallback). Shedding load is a user-visible decision, not a free one.
Anyone who would ship the proxy without the rejection-rate alert. Unmonitored admission control is just a new way to silently drop work.

If you want to rehearse this loop yourself, the harness above plus a free endpoint is enough to get real numbers in an afternoon — the MonkeyCode free server is one convenient option for that, and any compatible API works the same.

"Operate Dependabot Cooldown as a Quarantine Queue"

Odd_Background_328 — Wed, 29 Jul 2026 09:47:30 +0000

At 09:00 a release enters cooldown. At 11:00 a vulnerability signal appears, but no dashboard connects that signal to queued updates. The delay helped only if operators can see, alert on, and release or reject the quarantined item.

What the announcement establishes

GitHub’s primary-source article explains the case for a Dependabot cooldown: waiting before issuing version updates can create time for malicious releases to be identified. Treat the delay as a risk control with operating signals, not proof that a package is safe. Read the primary source.

For test track 7.1, the engineering claim here is narrower than the announcement: the surrounding workflow needs a contract that remains valid when metadata, transport, people, or executors change.

The artifact

{"dependency":"pkg","version":"4.2.0","ecosystem":"npm",
 "released_at":"2026-07-27T09:00:00Z","eligible_at":"2026-07-30T09:00:00Z",
 "state":"cooldown","risk_signals":[],"policy_version":"cd-3"}

Emit state transitions for observed, cooldown, blocked, eligible, proposed, and withdrawn. Useful gauges are queue age, items by ecosystem, blocked fraction, overdue eligibility, and signal-to-block latency.

For test track 7.2, this is a design fixture, not executed code. Pin language and dependency versions before turning it into a repository test, and replace example identities and timestamps with disposable values.

Success path

A synthetic release enters cooldown, receives no adverse signal, becomes eligible at policy time, and produces one proposal whose record points back to the policy version.

For test track 7.3, a successful demonstration records inputs, policy or schema version, decision, and final identifier. It does not infer correctness from a confidence label, status badge, or fluent output.

Failure path and regression plan

Inject a risk signal before eligibility. The item changes to blocked, an alert names the package and evidence source, and no update proposal appears. Duplicate signals must remain idempotent.

Replay a timeline with clock skew, delayed telemetry, policy change, withdrawn release, duplicate webhook, and collector outage. Alert on stuck state age rather than raw queue depth alone.

For test track 7.4, the acceptance gate is binary: the negative fixture must produce no unauthorized or duplicate side effect, while the positive fixture must remain traceable to its initial evidence. Expected output should be documented before execution.

Cleanup and rollback

Disable the custom alert policy first, restore the prior cooldown configuration, reconcile queued items without bulk-releasing them, and delete only synthetic records. Emergency bypasses need owner, reason, expiry, and post-review.

For test track 7.5, cleanup must preserve enough sanitized evidence to distinguish cancellation, rejection, stale work, and successful completion. Never solve recovery by silently marking an uncertain operation successful.

Limitations

For test track 7.6, this article proposes a compact engineering exercise and reports no execution results. It does not evaluate service availability, security, accessibility conformance, productivity, or comparative quality. Product previews can change, and a local fixture cannot reproduce every hosted-system failure.

A practical development environment

For isolated development tasks, MonkeyCode is an open-source AGPL-3.0 AI development platform offering an overseas hosted option. It includes a managed server-side cloud development environment, integrated models, task and requirement management, and build, test, and preview workflows. It is free to start. These statements do not mean the GitHub or OpenAI capability discussed above exists in MonkeyCode. Check the console for current quotas, models, regions, duration, and pricing before planning work. Open the campaign workspace.

Disclosure: This article promotes MonkeyCode using an official campaign link. I’m a MonkeyCode user, not affiliated with the project, and I receive no commission from this link.

AI assistance disclosure: This article was drafted with AI assistance and reviewed against the cited primary sources.

"Admit Cloud Workspaces With a Capacity Envelope and Orphan Sweep"

Odd_Background_328 — Tue, 28 Jul 2026 03:41:35 +0000

Workspace creation should fail before allocation, not halfway through a build. When quota and terms can vary, operations needs a conservative capacity envelope and an orphan sweep—not a dashboard that merely turns red after resources accumulate.

Define every quantity in the same unit and scope:

# proposed capacity envelope
scope: tenant-and-region-from-console
observed_limit_workspaces: replace-me
reserved_for_recovery: 1
max_trial_workspaces: replace-me
workspace_ttl_minutes: 60
max_builds_per_workspace: replace-me
admit_when:
  active_plus_starting_below: replace-me
  oldest_queue_seconds_below: replace-me
required_labels: [owner, purpose, created_at, expires_at, cleanup_state]

Compute admissible = observed_limit - active - starting - reserved_for_recovery. Admit only when admissible is positive, all required labels exist, and expected task duration fits before expiry. “Observed limit” must come from the current console or an authorized API response; never copy a number from an old trial.

Condition	Admission	Operator action
capacity and TTL fit	accept with workspace ID	schedule expiry immediately
limit unknown	reject safely	refresh console evidence
queue too old	shed new trials	inspect provisioning bottleneck
cleanup backlog nonzero	reserve capacity	run bounded orphan sweep
region unavailable	do not redirect silently	re-check permitted regions

Query orphans before deleting them

Use your platform inventory interface to express this query; the SQL is pseudocode, not a claim about a MonkeyCode API.

SELECT workspace_id, owner, state, created_at, expires_at, last_activity_at
FROM workspace_inventory
WHERE state NOT IN ('DELETED')
  AND (expires_at < CURRENT_TIMESTAMP
       OR owner IS NULL
       OR cleanup_state = 'FAILED')
ORDER BY expires_at ASC;

Runbook: freeze new admissions if orphan count consumes the recovery reserve. Snapshot IDs and states. For each candidate, verify no active build/test/preview operation, request deletion with an idempotency key, poll to terminal state, then check attached previews and temporary identities separately. Retry only documented transient failures with a cap. Escalate unknown ownership or a resource that survives its deletion deadline; never “fix” capacity by removing an unverified active workspace.

The successful path reserves capacity, creates a labeled workspace, runs bounded checks, deletes previews, deletes the workspace, and returns the slot. The failure path stops intake, preserves inventory evidence, attempts idempotent cleanup, and alerts an owner with resource ID and deadline. Exit the trial entirely if quota visibility or deletion confirmation cannot be established.

This envelope does not predict CPU, memory, storage, model concurrency, or provider-side contention. Inventory can be stale, clock skew can misclassify expiry, and deletion may be asynchronous. Measure lag and keep recovery reserve separate from nominal trial capacity. All thresholds above are placeholders and no operational result is claimed.

Operating boundary

MonkeyCode’s current official material calls its overseas hosted option “Free to start,” with integrated models and managed server-side cloud development environments that support building, testing, and previewing. This makes quota-aware admission immediately relevant, while never establishing that every model or server is free. Check the current console at run time for exact model availability, server quotas, regions, duration, future pricing, and terms.

Admit through the official campaign route only after the envelope has room: https://ly.cyberserval.tech/iIETXiF

On shutdown, block fresh intake, drain bounded operations, revoke preview authority, delete workspaces, and reconcile inventory against deletion receipts. Unknown capacity or lingering orphans trigger the exit plan and owner escalation.

Disclosure: This article promotes MonkeyCode using an official campaign link. I’m a MonkeyCode user, not affiliated with the project, and I receive no commission from this link.

AI assistance disclosure: This article was drafted with AI assistance and reviewed against the cited project materials.

"Run AI Credit Pools and Hard Limits From One Operations Dashboard"

Odd_Background_328 — Mon, 27 Jul 2026 13:38:55 +0000

At 09:10 the shared credit pool is 82% consumed, yet one repository shows no recent task outcomes. At 09:25 an unrelated API project reaches a hard spend threshold and calls may fail. A single “AI spend” gauge cannot tell an operator which action is safe.

GitHub's July 2026 changelog is the latest verified official signal here: on July 20 it lists AI credit pools and visibility plus repository Copilot usage metrics. Do not infer fields beyond the source. OpenAI's release notes say July 20 organization/project hard spend limits may cause API responses to fail after the threshold, with availability exactly as noted. Neither happened on July 27; unverified secondary claims for that date are excluded.

Dashboard by control plane

Keep provider signals distinct even if one team watches them:

Panel	Required dimensions	Operator question	Never infer
GitHub credit pool	pool, time window, owner	who can reduce consumption?	accepted code quality
repository usage	repository, time	where is reported use changing?	developer productivity
OpenAI spend limit	org/project, threshold state	which calls may be blocked?	another vendor's capacity
application outcomes	workload, success/failure	did users receive value?	provider billing truth

Normalize source, scope, window, unit, observed time, freshness, and evidence location internally. This does not imply either provider emits those exact fields; label missing or manual values.

Alerts must name an action

credit_pool_warning pauses eligible nonessential work while notifying the pool owner. hard_limit_failure blocks affected admissions and automated retries until an owner confirms state and one canary succeeds.

Do not alert only on a percentage. Add burn forecast, work criticality, freshness, and an owner. Thresholds above are intentionally omitted because account windows and risk tolerances differ.

Failure drill

Use a staging adapter; do not burn real credits to test the runbook.

# illustrative fault injection
AI_PROVIDER_MODE=budget_exhausted ./smoke-submit.sh

Expected evidence:

{"admission":"blocked","reason":"budget_exhausted","retry_scheduled":false,"runbook":"AI-04"}

Normal path: dashboards are fresh, work is attributed to a repository or project, warning policy leaves critical capacity, and application success metrics confirm useful outcomes.

Failure path: the adapter returns the typed exhaustion fixture. New calls stop, queued jobs become blocked_budget, retries remain zero, and the incident record names affected scopes. If the provider signal is stale, mark the panel unknown rather than green.

Runbook AI-04

Confirm: inspect current provider control plane and one sanitized application error; distinguish limit, rate limit, auth, and outage.
Contain: pause low-priority admissions and retry workers for the affected provider scope only.
Communicate: report impacted workloads and user-visible behavior, not speculative reset timing.
Decide: the budget owner may preserve the cap, approve a bounded exception, or route future work after an independent acceptance test.
Recover: submit one idempotent canary, confirm persistence, then ramp queues in small batches.
Review: reconcile provider usage, application attempts, accepted outcomes, and unclassified errors.

Rollback is the admission-control flag, not deletion of evidence. If a dashboard deployment itself causes high cardinality or scraping failures, disable the new collector and retain the last-known timestamp prominently.

Operational checks

Assign separate owners for credit allocation, spend-limit changes, and application reliability.
Store source, scope, unit, window, and freshness beside every number.
Alert on forecasted boundary plus actionable workload context.
Count rejected and retried requests as outcomes, not missing data.
Drill exhaustion with a fake adapter and verify zero retry amplification.
Canary restoration before draining blocked queues.
Review current GitHub/OpenAI documentation before changing controls.

Repository usage is not quality, credits are not currency across products, and a hard limit is not an SLA. Delayed data weakens forecasts; no metric fields, reset timing, or equivalence are assumed.

A monitored MonkeyCode pilot

For a separate candidate, MonkeyCode is described in verified official material as an open-source AGPL-3.0 AI development platform with an overseas online option, managed server-side cloud development environments, model/task/requirement management, and build/test/preview; it is free to start. I have not operated it for this drill. Teams can enter through the official campaign link, then require native evidence before mapping anything into these panels.

Disclosure: This article promotes MonkeyCode using an official campaign link. I’m a MonkeyCode user, not affiliated with the project, and I receive no commission from this link.

AI assistance disclosure: This article was drafted with AI assistance and reviewed against the cited primary sources.