DEV Community: Odd_Background_328

Reject Timestamped Prompts Before Cache Misses Beat Deadline Slack

Odd_Background_328 — Tue, 08 Sep 2026 14:58:06 +0000

At 02:14 the serving page stays oddly quiet.
Your node still reports twelve percent CPU usage.
Queue depth reads zero on the local listener.

Token counters climb without a matching wait queue.
Prompt tokens outrun completions by a wide margin.
Nightly eval deadline slack shrinks on every call.

Which operational action follows from that evidence?
You do not add another free replica yet.
You stop the cache-busting prompt prefix first.

The metric contradiction

Idle CPU is not proof of spare capacity.
It often hides a prefill tax on every call.
Each request looks unique to the server cache.

A timestamp in the system prompt is enough.
A request id in the tools preamble is enough.
The prefix hash then changes on every call.

You pay input tokens for the same instructions.
You wait for prefill on text you already sent.
Queue age stays low because each job is short.

Free capacity is the wrong bet in this shape.
The node is not saturated on compute cycles.

The token budget is what actually breaks first.
Scaling the free node will not fix the prefix.

Topology you can reproduce locally

Run the whole drill on one workstation only.
Keep production traffic out of this path.

client: serial eval runner under your control
admitd: admission proxy bound to 127.0.0.1:8088
server: local free model server behind the proxy
metrics: JSONL file with one event per request

You can stage the server side with MonkeyCode.
Free model access and a free server option exist for lab work.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat that path as a rehearsal environment only.
Do not read it as an availability SLO.
Do not read it as a production capacity plan.

Declared lab conditions

These figures are labels for the drill only.
They are not product benchmarks or hardware claims.

Workload: 40 serial jobs with an eight second deadline
Prefix: about 1,200 tokens of stable instructions
Question: about 80 tokens of unique user text
Noise: an ISO-8601 clock field inside the prefix
Cache: SHA-256 prefix digest in SQLite, fifteen minute TTL
Pass rule: reject noisy prefixes before tokens move

Why the free node looks healthy

Prefill work is bursty and then gone quickly.
CPU averages hide the token spike completely.
Queue depth never accumulates for these tiny jobs.

You will see three fields disagree at once.

queue_age_ms stays near zero during the window
cpu_pct stays in the low teens on average
prompt_tokens_total rises in lockstep with wall time

That pattern is not a scale-out signal.
It is an admission and reject signal.

You should shed the noisy prefix right now.
You should not add more free capacity yet.

Decision table for the page

Copy this table into the runbook.
Use it before you touch replica counts.

Observed fields	Inference, labeled as inference	Action
`queue_age_ms` low, `cpu_pct` low, prompt tokens rising	Cache-busting prefix, not saturation	Reject or strip the clock
`queue_age_ms` high, `cpu_pct` high	True saturation on the node	Shed load; skip the free path
`queue_age_ms` high, `cpu_pct` low	Blocked IO or a stuck worker	Debug the server; do not enqueue
`deadline_slack_ms` under 1,500	The job cannot finish in time	Reject; do not spend tokens

Separate observation from inference in every log line.
The first column stays observed telemetry only.
The second column is a working inference only.

Fingerprint the prefix, not the full body

You cannot meter what you cannot name yet.
Hash only the stable instruction prefix today.
Drop clocks, UUIDs, and trace ids first.

# prompt_fingerprint.py — lab helper, not a production library
import hashlib, json, re, sys

CLOCK = re.compile(
    r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?Z?"
)
IDS = re.compile(
    r"\b(?:req|trace|run|job)[-_][A-Za-z0-9-]{6,}\b",
    re.I,
)

def stabilize(text: str) -> str:
    text = CLOCK.sub("<TS>", text)
    text = IDS.sub("<ID>", text)
    return " ".join(text.split())

def fingerprint(prefix: str) -> str:
    body = stabilize(prefix).encode("utf-8")
    return hashlib.sha256(body).hexdigest()[:16]

if __name__ == "__main__":
    payload = json.load(sys.stdin)
    prefix = payload["prefix"]
    print(json.dumps({
        "fp": fingerprint(prefix),
        "stable_len": len(stabilize(prefix)),
        "stripped_clock": prefix != stabilize(prefix),
    }))

Feed it a prefix from standard input.
Confirm two clock-shifted calls share the same fp.
If they differ, your scrubber is still too weak.

printf '%s\n' '{"prefix":"Session 2026-09-08T02:14:03Z. Grade tests."}' \
  | python3 prompt_fingerprint.py
printf '%s\n' '{"prefix":"Session 2026-09-08T02:16:11Z. Grade tests."}' \
  | python3 prompt_fingerprint.py

Labeled expected output is a shared fp value.
stripped_clock should read true on both calls.
Do not proceed until those two hashes match.

Admission proxy with a hard reject

The proxy estimates tokens with a cheap heuristic.
It does not call the model for that estimate.
Four characters per token is the labeled lab rule.

Policy is reject-first on a raw wall-clock field.
You want callers to stop sending clocks.
Hiding the clock in a proxy forever trains bad clients.

# admitd.py — labeled local behavior, unexecuted in this article
import json, sqlite3, time
from http.server import BaseHTTPRequestHandler, HTTPServer
from prompt_fingerprint import fingerprint, stabilize

DB = "admit.sqlite"
DEADLINE_MS = 8000
MAX_PROMPT_TOKENS = 1500
CACHE_TTL_S = 15 * 60

def db():
    conn = sqlite3.connect(DB)
    conn.execute(
        "CREATE TABLE IF NOT EXISTS prefix("
        "fp TEXT PRIMARY KEY, hits INT, first_ts REAL)"
    )
    return conn

def estimate_tokens(text: str) -> int:
    return max(1, len(text) // 4)

class Admit(BaseHTTPRequestHandler):
    def do_POST(self):
        n = int(self.headers["Content-Length"])
        body = json.loads(self.rfile.read(n))
        prefix = body.get("prefix", "")
        queued_ms = int(body.get("queue_age_ms", 0))
        slack_ms = DEADLINE_MS - queued_ms
        fp = fingerprint(prefix)
        ptok = estimate_tokens(prefix)
        now = time.time()
        noisy = prefix != stabilize(prefix)
        decision = "allow"
        reason = "stable_prefix"
        if ptok > MAX_PROMPT_TOKENS:
            decision, reason = "reject", "prompt_tokens"
        elif slack_ms < 1500:
            decision, reason = "reject", "deadline_slack"
        elif noisy:
            decision, reason = "reject", "cache_busting_clock"
        conn = db()
        if decision == "allow":
            row = conn.execute(
                "SELECT hits, first_ts FROM prefix WHERE fp=?",
                (fp,),
            ).fetchone()
            if row is None:
                conn.execute(
                    "INSERT INTO prefix VALUES (?,?,?)",
                    (fp, 1, now),
                )
            else:
                conn.execute(
                    "UPDATE prefix SET hits=hits+1 WHERE fp=?",
                    (fp,),
                )
            conn.commit()
        event = {
            "ts": now,
            "fp": fp,
            "queue_age_ms": queued_ms,
            "deadline_slack_ms": slack_ms,
            "prompt_tokens_est": ptok,
            "decision": decision,
            "reason": reason,
            "cpu_pct_observed": body.get("cpu_pct"),
        }
        with open("admit.jsonl", "a") as f:
            f.write(json.dumps(event) + "\n")
        code = 200 if decision == "allow" else 429
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(json.dumps(event).encode())

if __name__ == "__main__":
    HTTPServer(("127.0.0.1", 8088), Admit).serve_forever()

Bind the handler only to localhost addresses.
Do not expose this handler on a public interface.
It is an admission drill, not an auth layer.

Failure injection you can run

Start the proxy on the lab loopback interface.
Leave the model server stopped on purpose here.
You are testing reject behavior, not generation quality.

python3 admitd.py &
echo $! > admitd.pid

# Job A: stable prefix, should allow
python3 - <<'PY'
import json, urllib.request
body = {
  "prefix": "You grade unit tests. Return JSON only.",
  "question": "Does test_foo assert timeout?",
  "queue_age_ms": 40,
  "cpu_pct": 12
}
req = urllib.request.Request(
  "http://127.0.0.1:8088/",
  data=json.dumps(body).encode(),
  headers={"Content-Type": "application/json"},
)
print(urllib.request.urlopen(req).read().decode())
PY

# Job B: clock inside prefix, should 429
python3 - <<'PY'
import json, urllib.request
body = {
  "prefix": "Session 2026-09-08T02:14:03Z. You grade unit tests.",
  "question": "Does test_foo assert timeout?",
  "queue_age_ms": 40,
  "cpu_pct": 12
}
req = urllib.request.Request(
  "http://127.0.0.1:8088/",
  data=json.dumps(body).encode(),
  headers={"Content-Type": "application/json"},
)
try:
    urllib.request.urlopen(req)
except Exception as e:
    print(type(e).__name__, e)
PY

# Job C: slack already gone, should 429
python3 - <<'PY'
import json, urllib.request
body = {
  "prefix": "You grade unit tests. Return JSON only.",
  "question": "late job",
  "queue_age_ms": 7000,
  "cpu_pct": 12
}
req = urllib.request.Request(
  "http://127.0.0.1:8088/",
  data=json.dumps(body).encode(),
  headers={"Content-Type": "application/json"},
)
try:
    urllib.request.urlopen(req)
except Exception as e:
    print(type(e).__name__, e)
PY

Inject one extra retry of Job B after the 429.
Keep max_retries=1 on this lab path.
A second clocked prefix should still not reach a model.

Labeled expected output

Job A should return HTTP 200 for allow.
The reason field should read stable_prefix now.
Estimated prompt tokens should stay under 1,500.

Job B should return HTTP 429 almost immediately.
The reason field should read cache_busting_clock here.
No model bytes should move for that call.

Job C should return HTTP 429 for slack.
The reason field should read deadline_slack here.
Queue age, not CPU, drove that reject.

Read the JSONL log after the three jobs.

python3 - <<'PY'
import json
from collections import Counter
rows = [json.loads(l) for l in open("admit.jsonl")]
print(Counter((r["decision"], r["reason"]) for r in rows))
print("max_queue_age_ms", max(r["queue_age_ms"] for r in rows))
print("max_prompt_tokens_est", max(r["prompt_tokens_est"] for r in rows))
PY

Expected lab histogram for this exact script:

("allow", "stable_prefix") count equals one
("reject", "cache_busting_clock") count equals one
("reject", "deadline_slack") count equals one

If allows outnumber rejects, the scrubber failed.
If CPU was your only page, you missed the burn.

Operational threshold and rationale

Pick one reject threshold and write it down.
Do not page on utilization alone.

Recommended lab threshold for this drill:

Reject when deadline_slack_ms sits under 1,500
Reject when estimated prompt tokens exceed 1,500
Reject when a wall-clock token remains in the prefix
Ignore CPU below 30% unless queue age also rises

The rationale stays local to this lab path.
Queue age tracks wait the user will feel.
Utilization tracks busy time you already spent.

Deadline slack tracks whether the job can still finish.
Low CPU plus rising prompt tokens is waste.
High CPU plus rising queue age is saturation.

Only the saturation case argues for more capacity.
Token retries make the waste case worse.
Each retry resends the busted prefix in full.

Cap retries at one on this admission path.
Retrying a 429 from cache_busting_clock is wrong.
Fix the caller, then send one clean prefix.

Cleanup and rollback

Stop the proxy after you capture logs.
Drop the lab database before the next drill.
Restore the eval runner to its prior URL.

if [ -f admitd.pid ]; then
  kill "$(cat admitd.pid)" || true
  rm -f admitd.pid
fi
rm -f admit.sqlite admit.jsonl
unset ADMIT_URL
export CLIENT_URL="${SERVER_URL:-http://127.0.0.1:8090}"
echo "rolled back to $CLIENT_URL"

If a clock already shipped in production prompts, strip it there first.
Then reopen traffic through the proxy for one window.
Keep the 429 path on during that single batch window.

Rollback is a URL swap plus a process kill.
It is not a cluster drain in this drill.
Do not leave admitd running after the rehearsal.

Limitations

The four-characters-per-token rule is crude.
It will misread code and CJK text.
Use it only as an admission fence.

The clock regex will miss custom date formats.
UUIDs in retrieved documents can still bust the cache.
Fingerprint the instruction prefix, not retrieved chunks.

This drill does not measure model quality at all.
It does not prove cache hits on a vendor stack.
It only stops noisy prefixes from spending tokens.

SQLite is not an HA cache under load.
One forgotten reader can stall checkpoint writes.
That failure mode belongs to a different note.

Who should not use this approach

Do not use this proxy as an internet gateway.
It has no authentication and no TLS.
Localhost binding is a hard requirement here.

Do not use it for interactive chat with tight latency.
A reject is better than a late eval job.
A reject is worse than a late chat token.

Do not use free capacity when the job is user-facing.
Deadline slack under one prefill needs a reserved path.
Free lab nodes are for rehearsal and shed-able batches.

Do not share one listener between serving and eval.
Eval sidecars will stamp clocks into prefixes.
That mix recreates the 02:14 contradiction exactly.

Skip this drill if your prompts are already stable.
Skip it if you cannot read token counters today.
Fix telemetry first, then add an admission fence.

What you do next

Watch prompt tokens against queue age for one window.
If they diverge, reject the prefix before you scale.
Run the fingerprint drill on your lab node before the next batch.

Reject Orphan Streams Before Connection Age Beats Token Value

Odd_Background_328 — Sun, 06 Sep 2026 20:39:02 +0000

The pager fires at 02:11 with a cost alert. Active streams still look modest on the panel. Useful completions per minute barely moved at all.

Which operational action follows from that evidence? You should not scale replicas on utilization alone. You should inspect connection age against token value.

The contradiction on the board

Queue depth can stay inside the admission band. Orphan streams still burn tokens after clients leave. That split is a cost incident, not capacity.

You serve streaming chat behind a reverse proxy. The worker keeps generating after TCP close. Billing still counts output tokens on that worker.

Free capacity makes this failure easy to miss. Idle spend looks like a quiet night first. Deadline slack then collapses on the next burst.

Topology you can reproduce locally

Run one proxy, one worker, and one collector. Keep the whole drill on loopback interfaces. Do not point this test at shared production pools.

Proposed layout, not a measured production fleet:

edge: reverse proxy with a 60s read timeout
admit: small admission process on port 8090
worker: one local generator with a single slot
collector: scrapes /metrics every fifteen seconds

client -> edge:8080 -> admit:8090 -> worker:8100
                         |
                         +-> collector:9090/metrics

Declare the worker as a single-slot generator. One in-flight stream. No batching in this drill.

Declared workload

Label these as drill conditions, not production facts.

Twenty clients open streaming completions on loopback
Each prompt is fixed at about 400 input tokens
Target output stops at 256 tokens, hard cap
Eight clients abort at 1.5 seconds on purpose
Useful-answer deadline slack is eight seconds
Admission rejects streams older than six seconds

You are measuring waste, not model quality. Do not change sampling settings during the run. Pin max tokens so output cannot run away.

Telemetry fields you must export

If a field is missing, the drill is incomplete.

stream_age_seconds: time since the request started
client_connected: one while the client socket lives
tokens_out_total: output tokens the worker actually billed
tokens_useful_total: tokens delivered to a live client
queue_age_seconds: wait at the head of the line
slot_utilization: busy slots divided by available slots
deadline_slack_seconds: remaining time before the deadline

Expected relationship, labeled as a hypothesis only. tokens_out_total minus tokens_useful_total is orphan spend. If that gap grows while utilization stays flat, reject.

When free capacity is the wrong bet

Free model access looks attractive for soak tests. A free server looks attractive for overnight evals. Orphan streams turn both into a silent leak.

You should refuse free capacity when these three hold:

Clients can disconnect without sending a cancel frame
The worker does not bind generation to socket life
Deadline slack is tighter than median stream age

Queue age, utilization, and slack will disagree here. Utilization can look fine with one zombie stream. Queue age then grows because the slot never frees.

Ask for a threshold with a written rationale. Do not pick a round number without a clock. Proposed threshold for this labeled drill:

Reject when stream_age_seconds is greater than six
Rationale: useful answers still need eight seconds slack
Six seconds leaves two seconds to drain and enqueue

Prefer queue age over utilization for this failure. Prefer slack over raw token rate for the page. Time and retries are secondary once the slot is stuck.

A local drill can use free model access for the worker. Some teams also park that worker on a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you already stage soak traffic there, run this drain loop on that path before production.

Artifact: admit on stream age

Treat the next process as unexecuted example code. It rejects orphans before the worker spends more tokens. It also closes upstream when the client socket dies.

# unexecuted example: stream-age admission on loopback
import time
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

MAX_STREAM_AGE_S = 6.0
SLOT = {"busy": False, "started": 0.0, "client_live": False}
LOCK = threading.Lock()
METRICS = {
    "tokens_out_total": 0,
    "tokens_useful_total": 0,
    "rejected_age": 0,
    "cancelled_orphan": 0,
}

def snapshot():
    now = time.monotonic()
    with LOCK:
        age = (now - SLOT["started"]) if SLOT["busy"] else 0.0
        util = 1.0 if SLOT["busy"] else 0.0
        slack = 8.0 - age if SLOT["busy"] else 8.0
        return {
            "stream_age_seconds": round(age, 3),
            "client_connected": 1 if SLOT["client_live"] else 0,
            "slot_utilization": util,
            "deadline_slack_seconds": round(slack, 3),
            "queue_age_seconds": round(age if SLOT["busy"] else 0.0, 3),
            **METRICS,
        }

class Admit(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/metrics":
            self.send_error(404)
            return
        body = "\n".join(f"{k} {v}" for k, v in snapshot().items()) + "\n"
        self.send_response(200)
        self.send_header("Content-Type", "text/plain")
        self.end_headers()
        self.wfile.write(body.encode())

    def do_POST(self):
        if self.path != "/v1/stream":
            self.send_error(404)
            return
        now = time.monotonic()
        with LOCK:
            if SLOT["busy"]:
                age = now - SLOT["started"]
                if age > MAX_STREAM_AGE_S:
                    METRICS["rejected_age"] += 1
                    busy = True
                else:
                    busy = True
            else:
                busy = False
            if busy and (now - SLOT["started"] <= MAX_STREAM_AGE_S):
                self.send_response(429)
                self.end_headers()
                self.wfile.write(b"slot_busy\n")
                return
            if busy:
                SLOT["busy"] = False
            SLOT["busy"] = True
            SLOT["started"] = now
            SLOT["client_live"] = True
        self.send_response(200)
        self.send_header("Content-Type", "text/event-stream")
        self.end_headers()
        try:
            for i in range(256):
                time.sleep(0.02)  # labeled fake generation, not a benchmark
                with LOCK:
                    METRICS["tokens_out_total"] += 1
                    live = SLOT["client_live"]
                    age = time.monotonic() - SLOT["started"]
                    if live:
                        METRICS["tokens_useful_total"] += 1
                if age > MAX_STREAM_AGE_S:
                    with LOCK:
                        METRICS["rejected_age"] += 1
                    break
                try:
                    self.wfile.write(f"data: {i}\n\n".encode())
                    self.wfile.flush()
                except BrokenPipeError:
                    with LOCK:
                        SLOT["client_live"] = False
                        METRICS["cancelled_orphan"] += 1
                    break
        finally:
            with LOCK:
                SLOT["busy"] = False
                SLOT["client_live"] = False

    def log_message(self, fmt, *args):
        return

if __name__ == "__main__":
    HTTPServer(("127.0.0.1", 8090), Admit).serve_forever()

Start it in a dedicated terminal before any clients connect.

python3 admit_orphan_stream.py

Failure injection you can run tonight

Use two terminals. Leave metrics scraping in the third.

# terminal B: a client that stays until 256 tokens
curl -N -X POST http://127.0.0.1:8090/v1/stream

# terminal C: eight aborting clients (labeled load, not a benchmark)
for i in $(seq 1 8); do
  curl -N -m 1.5 -X POST http://127.0.0.1:8090/v1/stream &
done
wait

# terminal D: scrape the same fields the pager should use
curl -s http://127.0.0.1:8090/metrics

Kill the long client with Ctrl-C during generation. Watch client_connected drop to zero. tokens_out_total must stop within one loop tick.

If output tokens keep climbing after the abort, you failed. The worker is ignoring socket death. That is the production leak in miniature.

Labeled expected output

These numbers are expected drill output, not production measurements.

tokens_out_total 64
tokens_useful_total 48
cancelled_orphan 8
rejected_age 1
stream_age_seconds 0.0
client_connected 0
slot_utilization 0.0
queue_age_seconds 0.0
deadline_slack_seconds 8.0

Orphan gap equals billed tokens minus useful tokens. Here that gap is sixteen tokens. Eight aborts plus one age reject should not hold the slot.

If slot_utilization stays at 1.0 after aborts, rollback. If queue_age_seconds exceeds six with slack under two, page. Do not wait for utilization to look red.

Failure handling during the drill

Follow this order. Do not skip to replica scaling.

Freeze new admissions when stream_age_seconds exceeds six
Send a cancel to the worker, then close the upstream
Mark client_connected=0 before counting more useful tokens
Drain the single slot until queue_age_seconds returns to zero
Only then reopen admissions for the next labeled client

Retries belong after the slot is free. Retrying against a zombie stream doubles token spend. Time spent retrying is not deadline slack.

Cleanup and rollback

Stop the admission process and any leftover curl jobs. Confirm nothing still listens on 8090 or 8100. Restore the previous proxy timeout if you changed it.

pkill -f admit_orphan_stream.py || true
pkill -f "curl -N -X POST http://127.0.0.1:8090/v1/stream" || true
ss -ltnp | grep -E '8090|8100' || echo "loopback ports clear"

Rollback rule for a real edge proxy is equally small. Restore the last known timeout and cancel mapping. Revert admission to "slot busy means 429" only. Do not leave the six-second age cut in production unreviewed.

Write the revert in the same change window. Leave a note on the threshold rationale. The next on-call should see slack, not folklore.

Decision table: queue age, utilization, slack

Use this table during the page, not after it.

High utilization, low queue age, healthy slack: keep serving
Low utilization, high queue age, shrinking slack: reject orphans
Low utilization, low queue age, high token gap: cancel zombies
High utilization, high queue age, slack under two seconds: shed load
Free capacity plus any zombie gap: stop the soak, do not scale

Free capacity is the wrong bet for interactive deadlines. It is also wrong when cancel frames never reach the worker. Pay for a slot you can kill, or do not offer streaming.

Limitations and who should not use this

This drill assumes one slot and a hard output cap. It does not model multi-tenant fairness. It does not prove quality, cache hit rate, or safety.

Do not use this admission cut on long-running batch jobs. Batch work needs queue age against a batch deadline. Six seconds will false-reject those jobs immediately.

Do not use it as a billing auditor. Token fields here are local counters. They are not invoices from any provider.

Skip this approach if your proxy already ties generation to connection lifetime. Also skip it if clients cannot stream and only poll. Polling needs a different cancel path and different metrics.

The example sleeps for fake tokens on purpose. Replace that loop with your real worker only in a lab. Keep production traffic off this process.

What you do after the metrics agree

You now have a reject path before token value goes negative. Connection age is the first lever, not replica count. Queue age tells you the slot is trapped.

Run the drain drill on a throwaway worker before the next cost page. Keep the threshold tied to deadline slack in writing. If slack and queue age disagree, trust slack and drain.

Reject Model-Backed Probes Before Token Spend Beats Ready Time

Odd_Background_328 — Sat, 05 Sep 2026 19:15:02 +0000

You open Grafana at 09:12 after a quiet night.
Token counters climbed while request logs stayed empty.
Your cluster still looks healthy on the default probes.

Which operational action follows from that evidence?
Stop blaming the model and inspect the probe target first.

This is probe spend, not user demand

Kubernetes did its usual job with boring regularity.
A readiness probe hit your generate route all night.
Each check spent tokens and occupied a queue slot.

Free capacity makes this failure easy to miss.
The server looks idle on CPU and replica count.
The token ledger and queue age tell another story.

Lab topology you can run locally

Run one adapter process and a probe loop together.
Keep the model call behind an explicit complete route.
Point health checks at a cheap /readyz handler only.

adapter: HTTP process, in-memory queue, token ledger
probe: curl loop that mimics a kube httpGet
ledger: JSON lines on disk for later jq diffs
worker: fake completion with a hard timeout

Use this proposed Compose file for the isolated drill:

services:
  adapter:
    build: .
    ports: ["8080:8080"]
    environment:
      HOURLY_TOKEN_CAP: "50000"
      MAX_QUEUE_AGE_MS: "2000"
      COMPLETE_TIMEOUT_S: "30"
      ADMIT_PROBE_COMPLETES: "0"
      LEDGER_PATH: "/var/lib/adapter/ledger.jsonl"
    volumes:
      - ./ledger:/var/lib/adapter

Do not attach this stack to production DNS.
Do not share the queue with real user traffic.
Tear it down when the drill ends.

Where a free server still fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode is an open-source project with free model access.
A free server option is useful for this probe drill.

It is the wrong bet for SLO-bound interactive serving.
Separate probe tokens from user tokens on day one.
If those counters share one bucket, rollback stays blind.

Cost ops starts with attribution, not with extra quota.
Write the route split before you invite any users.

Declared workload

Treat the numbers below as lab conditions only.
They are not production benchmarks or vendor claims.

Idle sixty seconds with probes on /readyz only.
Switch the probe URL to /v1/complete for fifteen minutes.
Restore /readyz and drain non-user jobs first.
Diff token counters, queue age, and ready latency.

Declare these knobs before you start the loop:

Probe interval: 10 seconds
Probe timeout: 1 second
Completion timeout: 30 seconds
Packed prompt: 128 estimated tokens
Concurrent user traffic: none
Ready SLO for the drill: 200 milliseconds
Hourly token cap in lab: 50000 (local knob only)

Cheap ready versus generate

Your ready check should answer three questions only.
Keep it inside the adapter process on purpose.
Remote dependency checks turn ready into generate.

Is the HTTP process up and serving locally?
Is queue age under a quarter of deadline slack?
Is the hourly token ledger under its cap?

It must not forward a prompt into the model.
It must not wait on remote batching or GPUs.
It must not retry, because retries multiply probe waste.

Proposed adapter code for the local drill:

# lab-only adapter. Not a production service.
import json, os, time, threading
from http.server import BaseHTTPRequestHandler, HTTPServer

LEDGER = os.environ.get("LEDGER_PATH", "./ledger.jsonl")
CAP = int(os.environ.get("HOURLY_TOKEN_CAP", "50000"))
MAX_AGE = int(os.environ.get("MAX_QUEUE_AGE_MS", "2000"))
ADMIT_PROBES = os.environ.get("ADMIT_PROBE_COMPLETES", "0") == "1"
EST_TOKENS = 128

lock = threading.Lock()
queue = []  # {enqueued_ms, tokens, reason}
spent_hour = 0
hour_start = time.time()

def emit(route, **fields):
    rec = {"ts": time.time(), "route": route, **fields}
    with open(LEDGER, "a") as f:
        f.write(json.dumps(rec) + "\n")

def queue_age_ms():
    if not queue:
        return 0
    return int((time.time() * 1000) - queue[0]["enqueued_ms"])

def ready_ok():
    global spent_hour, hour_start
    now = time.time()
    if now - hour_start >= 3600:
        spent_hour = 0
        hour_start = now
    if queue_age_ms() > MAX_AGE:
        return False, "queue_age"
    if spent_hour >= CAP:
        return False, "token_cap"
    return True, "ok"

class Handler(BaseHTTPRequestHandler):
    def _send(self, code, body):
        raw = json.dumps(body).encode()
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(raw)))
        self.end_headers()
        self.wfile.write(raw)

    def do_GET(self):
        t0 = time.time()
        if self.path.startswith("/readyz"):
            ok, reason = ready_ok()
            ready_ms = int((time.time() - t0) * 1000)
            slack = max(0, MAX_AGE - queue_age_ms())
            emit("readyz", queue_age_ms=queue_age_ms(),
                 queue_depth=len(queue), ready_ms=ready_ms,
                 deadline_slack_ms=slack, reason=reason,
                 est_tokens=0, actual_tokens=0)
            self._send(200 if ok else 503, {"ok": ok, "reason": reason})
            return
        # GET-on-generate is a common kube probe footgun.
        if self.path.startswith("/v1/complete"):
            self._complete(reason_header=True)
            return
        self._send(404, {"error": "not_found"})

    def do_POST(self):
        if not self.path.startswith("/v1/complete"):
            self._send(404, {"error": "not_found"})
            return
        self._complete(reason_header=True)

    def _complete(self, reason_header=False):
        global spent_hour
        reason = self.headers.get("X-Traffic-Reason", "user") if reason_header else "user"
        if reason == "probe" and not ADMIT_PROBES:
            emit("complete_reject", reason="probe_rejected",
                 queue_age_ms=queue_age_ms(), queue_depth=len(queue),
                 est_tokens=EST_TOKENS, actual_tokens=0,
                 ready_ms=0, deadline_slack_ms=max(0, MAX_AGE - queue_age_ms()))
            self._send(429, {"error": "probe_rejected"})
            return
        ok, why = ready_ok()
        if not ok:
            emit("complete_reject", reason=why, queue_age_ms=queue_age_ms(),
                 queue_depth=len(queue), est_tokens=EST_TOKENS, actual_tokens=0,
                 ready_ms=0, deadline_slack_ms=0)
            self._send(503, {"error": why})
            return
        job = {
            "enqueued_ms": time.time() * 1000,
            "tokens": EST_TOKENS,
            "reason": reason,
        }
        with lock:
            queue.append(job)
            spent_hour += EST_TOKENS
        # Fake a completion. Label this as unexecuted model I/O.
        time.sleep(0.05)
        with lock:
            if queue and queue[0] is job:
                queue.pop(0)
            elif job in queue:
                queue.remove(job)
        emit("complete", reason=reason, queue_age_ms=queue_age_ms(),
             queue_depth=len(queue), est_tokens=EST_TOKENS,
             actual_tokens=EST_TOKENS, ready_ms=0,
             deadline_slack_ms=max(0, MAX_AGE - queue_age_ms()))
        self._send(200, {"ok": True, "tokens": EST_TOKENS})

    def log_message(self, fmt, *args):
        return

if __name__ == "__main__":
    HTTPServer(("0.0.0.0", 8080), Handler).serve_forever()

Reject probe completes in the handler, not later in dashboards.
A dashboard alert cannot refund the tokens already spent.

Kubernetes probes: wrong versus right

Health checks are GET by default in Kubernetes.
Teams still burn tokens in two common ways.
They map generate onto GET, or they wrap curl in exec.

Wrong probe looks simple and still burns tokens.
Copy this only as a failure fixture in lab.

# expected failure config. Do not ship this.
readinessProbe:
  httpGet:
    path: /v1/complete
    port: 8080
    httpHeaders:
      - name: X-Traffic-Reason
        value: probe
  periodSeconds: 10
  timeoutSeconds: 1
  failureThreshold: 3
livenessProbe:
  exec:
    command: ["curl", "-sS", "-X", "POST", "http://127.0.0.1:8080/v1/complete"]
  periodSeconds: 10

Right probe checks admission health and nothing else.
Ship this shape if you must run under Kubernetes.

readinessProbe:
  httpGet:
    path: /readyz
    port: 8080
  periodSeconds: 10
  timeoutSeconds: 1
  failureThreshold: 3
livenessProbe:
  httpGet:
    path: /readyz
    port: 8080
  periodSeconds: 10
startupProbe:
  httpGet:
    path: /readyz
    port: 8080
  periodSeconds: 5
  failureThreshold: 12

Never let a liveness probe call generate.
A slow completion becomes a kube restart loop.
Restarts cold-start the queue and waste more tokens.

Telemetry fields you actually debug

Emit one JSON line per ready check and complete.
Name the fields the same in every process replica.
Mismatched keys make the jq drill lie to you.

ts
route
queue_age_ms
queue_depth
est_tokens
actual_tokens
ready_ms
deadline_slack_ms
reason

Debug with queue age, not with CPU graphs.
CPU stays quiet because those probes stay sparse.
Queue age and token counters will move first.

# proposed local queries after the drill
jq 'select(.reason=="probe") | .actual_tokens' ledger.jsonl \
  | awk '{s+=$1} END {print s}'
jq 'select(.route=="readyz") | .ready_ms' ledger.jsonl \
  | awk '{s+=$1; n++} END {print s/n}'
jq 'select(.route=="complete_reject")' ledger.jsonl | wc -l

Expected lab output on the bad path

This block is labeled expected output, not a measurement.
You should reproduce the shape, not chase a vendor number.

{"route":"complete","reason":"probe","queue_age_ms":0,"est_tokens":128,"actual_tokens":128}
{"route":"complete","reason":"probe","queue_age_ms":40,"est_tokens":128,"actual_tokens":128}

Fifteen minutes at a ten second period is ninety probes.
Ninety times 128 estimated tokens is 11520 tokens spent.
Ready latency can still sit under 200 milliseconds.

That metric contradiction is the page you want.
Tokens moved while users and CPU stayed flat.
Free capacity looked idle while the ledger drained.

Call the next paragraph architectural inference, not observation.
Probe jobs steal the same admission slots as users.
When real traffic arrives, deadline slack is already thin.

Fault injection you can run today

Keep this drill on localhost and do not probe production.
Record every curl in the ledger before you change flags.

# proposed drill. Label results as lab-only.
python adapter.py &
ADAPTER_PID=$!
sleep 1

# Stage A: cheap ready for 60s
for i in $(seq 1 6); do
  curl -sS -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8080/readyz
  sleep 10
done

# Stage B: restart with probe completes admitted
kill "$ADAPTER_PID"
ADMIT_PROBE_COMPLETES=1 python adapter.py &
ADAPTER_PID=$!
sleep 1

for i in $(seq 1 90); do
  curl -sS -H "X-Traffic-Reason: probe" \
    -X POST http://127.0.0.1:8080/v1/complete
  sleep 10
done

Inject a hung worker as a second failure mode.
Leave one job on the queue past COMPLETE_TIMEOUT_S.

# proposed fault: never dequeue the probe job
time.sleep(int(os.environ.get("COMPLETE_TIMEOUT_S", "30")))

Watch queue_age_ms climb while CPU stays boring.
Page on that contradiction, not on replica count.
Utilization is the wrong primary signal for this.

Thresholds and why queue age wins

Pick one operational threshold and write the rationale.
Write it in the runbook before the first page.

Signal	Threshold	Action
`queue_age_ms`	greater than 0.25 times deadline	reject new complete work
probe `actual_tokens` per hour	greater than 5 percent of cap	move probes to `/readyz`
`ready_ms`	greater than 200	page the adapter, not the model
`deadline_slack_ms`	less than 2000	fail closed and drain
CPU utilization	high at QPS about 0.1	suspect probes or retries

Queue age beats utilization for this failure mode.
Utilization stays low because the probes are infrequent.
Deadline slack shrinks because each probe is real work.

Token cap percent catches a silent overnight drain.
Ready time catches adapter stalls without model I/O.
You need all three signals because one graph will lie.

When free capacity is the wrong bet

Use a decision, not a hope, before you keep the free path.
Free serving still has a queue, a timeout, and retries.

Free capacity is the wrong bet when:

The work is user-facing and SLO-bound
Probes share the complete queue with users
You cannot attribute tokens by reason
Retry or abort policy is still unbounded
Ready time must stay under 200 milliseconds
A hung complete can restart the process

Use free capacity when:

You are running an isolated local drill
Batch jobs have explicit deadlines you can miss
You can roll back without user impact
Token accounting is the thing under test

Time, tokens, retries, and queueing always stack together.
Probes add a fifth term: periodic guaranteed load.
Ignore that term and unit cost inverts at night.

Rollback and cleanup

Do these steps in order and do not scale first.

Point kube probes at /readyz immediately.
Set ADMIT_PROBE_COMPLETES=0 on the adapter.
Drain queue entries with reason!=user first.
Snapshot the ledger. Then drop probe rows only.
Delete the compose stack and the volume.

kubectl patch deploy adapter --type=json -p='[
  {"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/httpGet/path","value":"/readyz"},
  {"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/httpGet/path","value":"/readyz"}
]'

kubectl set env deploy/adapter ADMIT_PROBE_COMPLETES=0

# local cleanup
kill "$ADAPTER_PID"
rm -f ledger.jsonl
docker compose down -v

If ready_ms stays high after the probe patch, roll the image.
Do not add replicas to hide probe spend.
More replicas often multiply probe frequency through the mesh.

Limitations and who should skip this

This drill does not measure model quality at all.
It does not prove a quota, SLA, or hardware profile.
It does not replace a user-shaped load test.

Do not use this approach when the constraints below apply.

Your platform already exposes a dedicated health port
You cannot change probe paths in the service mesh
Completions are synchronous and have no queue
You still lack a token ledger split by route

Skip it if you only run a laptop script without probes.
Skip it if a free server is your production SLO path.
Skip it if legal retention forbids dropping probe logs.

Monday checklist

Wire readyz before you expose the complete route.
Attribute tokens by route before the first deploy.
Reject probe work before queue age eats deadline slack.

Record the contradiction in the runbook tonight itself.
Tokens up, users flat, CPU flat means probes.
Act on that evidence and do not buy more capacity first.

Reject Agent Tool Rounds Before Free Capacity Inverts Unit Cost

Odd_Background_328 — Fri, 04 Sep 2026 17:04:34 +0000

You get the cost alert at 02:14 UTC tonight.
Queue depth still looks healthy on your Grafana dashboard.
Completed jobs stay flat while token spend climbs.

Which operational action follows from that split in evidence?
You pause new agent sessions before you scale anything.
You measure cost per completed job, not started calls.

The contradiction you should not ignore

Started requests keep succeeding at the edge proxy tonight.
Downstream tool rounds never return a finished answer.
You pay for context, tools, and retries that never land.

This pattern is unit-cost inversion, not simple queue backup.
Free capacity hides it until the finance export arrives.
You need a ledger that splits started work from finished work.

Topology you can reproduce on a laptop

Run one admission sidecar in front of a local fake model.
The sidecar counts expected tokens times planned tool rounds.
A worker loop calls tools until a deadline or reject.

Declared pieces for the local drill:

ledger.py writes one JSON line per job event
admit.py rejects work above a unit-cost ceiling
worker.py injects extra tool rounds on command
cleanup.sh deletes the ledger and stops the loop

Do not treat this laptop loop as production evidence.
Label every number below as expected output from the script.

Declared test conditions

Use these conditions and do not silently change them.

Workload: 40 agent jobs, max four tool rounds each
Fake tokens: 800 completion plus 200 prompt per round
Deadline slack: 12 seconds wall time per healthy job
Reject if projected tokens exceed 4,000 per completion
Reject if three starts land with zero completions
Local only, no remote model, no network dependency

You are measuring inversion, not remote model quality.
Keep the fake server single-threaded so queue age stays visible.

Build a per-job token ledger

Create ledger.py as a JSONL append-only local event file.
It records start, round, complete, cancel, and reject events.

#!/usr/bin/env python3
"""Local job ledger. Expected output only. Not production telemetry."""
import json, time, threading, os
from pathlib import Path

LEDGER = Path(os.environ.get("LEDGER_PATH", "/tmp/agent_ledger.jsonl"))
_lock = threading.Lock()

def emit(event, job_id, **fields):
    rec = {"ts": time.time(), "event": event, "job_id": job_id}
    rec.update(fields)
    with _lock:
        with LEDGER.open("a") as f:
            f.write(json.dumps(rec) + "\n")

def summarize():
    started = completed = cancelled = rejected = 0
    tokens = 0
    if not LEDGER.exists():
        return {
            "started": 0,
            "completed": 0,
            "cancelled": 0,
            "rejected": 0,
            "tokens_spent": 0,
            "unit_cost": None,
            "start_complete_ratio": None,
        }
    for line in LEDGER.read_text().splitlines():
        r = json.loads(line)
        ev = r["event"]
        if ev == "start":
            started += 1
        elif ev == "complete":
            completed += 1
            tokens += r.get("tokens", 0)
        elif ev == "cancel":
            cancelled += 1
            tokens += r.get("tokens", 0)
        elif ev == "reject":
            rejected += 1
    unit = (tokens / completed) if completed else None
    ratio = (started / completed) if completed else None
    return {
        "started": started,
        "completed": completed,
        "cancelled": cancelled,
        "rejected": rejected,
        "tokens_spent": tokens,
        "unit_cost": unit,
        "start_complete_ratio": ratio,
    }

You now have started, completed, cancelled, and spent tokens.
Unit cost uses completed jobs as the only denominator.
Cancelled work still adds tokens and exposes the inversion.

Admission rule you can explain at 03:00

Create admit.py as the 03:00 gate you can explain.
The gate uses two independent thresholds, not one blended score.

#!/usr/bin/env python3
"""Admit or reject agent jobs from ledger summaries."""
from ledger import summarize, emit

MAX_TOKENS_PER_COMPLETION = 4000
MAX_START_COMPLETE_RATIO = 1.8
MAX_ROUNDS = 4
TOKENS_PER_ROUND = 1000  # declared: 800 completion + 200 prompt
ZERO_COMPLETION_STARTS = 3

def projected_tokens(planned_rounds):
    return min(planned_rounds, MAX_ROUNDS) * TOKENS_PER_ROUND

def admit(job_id, planned_rounds):
    stats = summarize()
    proj = projected_tokens(planned_rounds)
    reasons = []
    if proj > MAX_TOKENS_PER_COMPLETION:
        reasons.append("projected_tokens")
    ratio = stats["start_complete_ratio"]
    if ratio is not None and ratio > MAX_START_COMPLETE_RATIO:
        reasons.append("start_complete_ratio")
    unit = stats["unit_cost"]
    if unit is not None and unit > MAX_TOKENS_PER_COMPLETION:
        reasons.append("observed_unit_cost")
    if stats["started"] >= ZERO_COMPLETION_STARTS and stats["completed"] == 0:
        reasons.append("zero_completions")
    if reasons:
        emit("reject", job_id, reasons=reasons, projected=proj)
        return False, reasons
    emit("start", job_id, planned_rounds=planned_rounds, projected=proj)
    return True, []

Ask for one operational threshold and write down the rationale.
Queue age is not the same signal as utilization percent.
Deadline slack is not the same signal as token headroom.

Pick the threshold from the evidence in front of you:

Pick unit cost first when completions stall under spend
Pick queue age first when workers stay busy and finish
Pick deadline slack first when the caller still waits
Pick zero-completion starts when cancels wipe the denominator

Worker that can waste tokens on purpose

Create worker.py as a fault injector, not a model client.
Extra rounds burn tokens without producing a completed job.

#!/usr/bin/env python3
"""Inject extra tool rounds. Local expected behavior only."""
import argparse, time, uuid
from admit import admit, TOKENS_PER_ROUND
from ledger import emit, summarize

def run_job(extra_rounds, deadline_s):
    job_id = str(uuid.uuid4())[:8]
    planned = 1 + extra_rounds
    ok, reasons = admit(job_id, planned)
    if not ok:
        return {"job_id": job_id, "status": "rejected", "reasons": reasons}

    spent = 0
    t0 = time.time()
    for rnd in range(planned):
        time.sleep(0.15)  # declared fake round latency
        spent += TOKENS_PER_ROUND
        emit("round", job_id, round=rnd, tokens=spent)
        if time.time() - t0 > deadline_s:
            emit("cancel", job_id, tokens=spent, reason="deadline")
            return {"job_id": job_id, "status": "cancelled", "tokens": spent}
    emit("complete", job_id, tokens=spent)
    return {"job_id": job_id, "status": "complete", "tokens": spent}

if __name__ == "__main__":
    p = argparse.ArgumentParser()
    p.add_argument("--jobs", type=int, default=40)
    p.add_argument("--extra-rounds", type=int, default=0)
    p.add_argument("--deadline", type=float, default=12.0)
    args = p.parse_args()
    for _ in range(args.jobs):
        print(run_job(args.extra_rounds, args.deadline))
    print("SUMMARY", summarize())

Run the healthy baseline before you inject extra rounds.

export LEDGER_PATH=/tmp/agent_ledger.jsonl
rm -f "$LEDGER_PATH"
python3 worker.py --jobs 40 --extra-rounds 0 --deadline 12

Treat the next block as expected output, not production proof.

Forty complete events in the JSONL ledger
About 40,000 tokens spent across finished jobs
Unit cost near 1,000 tokens per completion
Start/complete ratio near 1.0 with no rejects

Now inject extra tool rounds against a tight deadline.

rm -f "$LEDGER_PATH"
python3 worker.py --jobs 40 --extra-rounds 2 --deadline 0.2

Expect this declared fault to invert unit cost quickly.

Early jobs cancel after the first 0.15s fake round
Tokens still land on cancel events with no completion
After three starts, zero_completions rejects new work
Unit cost stays undefined while spend is already real

That reject is the operational action from the evidence.
You stop starting work that cannot finish on time.

When free capacity is the wrong bet

Free model access looks cheap at the request edge.
Agent loops multiply that cost by every tool round.
A cancelled round still consumed context window and time.

Use free capacity as a lab, not as a production SLO.
Promote only after the ledger stays free of inversion.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode offers free model access and a free server option you can point this local ledger at during lab drills. Do not treat that path as unbounded production headroom.

Free capacity is the wrong bet under these five conditions:

Tool rounds per job exceed your completion token budget
Cancel rate stays high while spend keeps rising anyway
Queue looks empty because work dies inside the worker
Restarts replay full agent traces, not a single tool
Deadline slack is smaller than one extra tool round

Paid burst capacity can be cheaper per completed job.
You pay more per hour and less per finished answer.
That inversion is the decision, not a vendor slogan.

Failure handling and the rollback path

Watch these fields on every reject and every cancel.

event: start, round, complete, cancel, reject
tokens: cumulative spend on that job_id
reasons: projected_tokens, start_complete_ratio, observed_unit_cost, zero_completions
start_complete_ratio: started divided by completed, when defined
unit_cost: spent tokens divided by completed jobs only

If the ledger file is missing, you must fail closed.
Do not admit work you cannot account for tonight.
If summarize returns no completions, reject every new start.

Follow this rollback path after a false reject storm:

Set --extra-rounds 0 and rerun the healthy baseline.
Drain in-flight jobs for one full deadline window.
Restore admit thresholds only after unit cost drops.
Keep LEDGER_PATH on a tmpfs so leftover files cannot linger.

Do not raise token ceilings to silence the admission gate.
Fix the fan-out pattern or the deadline first.

Cleanup

Stop the injector and delete the local ledger file.

pkill -f worker.py || true
rm -f /tmp/agent_ledger.jsonl
unset LEDGER_PATH
ps aux | grep -v grep | grep worker.py || echo "no worker"
ls /tmp/agent_ledger.jsonl 2>/dev/null || echo "ledger gone"

Confirm no leftover python workers with a process listing.
Confirm the ledger path is gone before you leave.

Who should not use this

Skip this gate if you run single-shot completions only.
Skip it if every round lacks a stable job identifier.
Skip it if policy forbids even local request event logs.

This ledger is not billing-grade and not multi-tenant isolation.
You also should not use free capacity for launch-path traffic.
Lab inversion drills belong on a throwaway local server.

Production needs a budget, a drain, and a named owner.
A free lab path does not replace those three controls.

What you do at 02:14

You do not scale replica count on a token spike.
You compare started jobs against completed jobs first.
You reject new agent rounds until unit cost falls.

Then you rerun the local injector with extra rounds.
You keep the reject threshold tied to finished completions.
Run that drill before the next agent loop ships.

Reject Retry Storms Before Token Cost Beats Deadline Slack

Odd_Background_328 — Thu, 03 Sep 2026 14:36:40 +0000

Your pager fires on review_latency_p95 at 02:14.
Queue utilization still reads only twelve percent.
The p95 timer already sits near forty-eight seconds.

Your SLO still says fifteen seconds end to end.
Idle workers do not explain that gap.
Retries, queue age, and token burn do.

Read The Contradiction First

Do not scale replicas from utilization alone.
Retries hide the real cost of free capacity.
Each retry spends tokens, time, and queue slots.

Watch these fields on one screen:

queue_age_ms
retry_count
token_in and token_out
deadline_slack_ms
worker_util

Utilization without queue age lies to you.
Queue age without retries also lies.
You need both before you spend money.

Lab Topology You Can Copy

Run this as a local drill, not production.
One admission proxy. One bounded queue. One worker.
Keep the blast radius inside one directory.

Declared test conditions:

Workload: 40 review jobs in 20 seconds
Prompt budget: 800 input units, 200 output cap
SLO: 15s end-to-end, 5s max queue age
Retry policy: max 3, 200ms exponential backoff
Worker: one process, 2s mean service time

This topology is a rehearsal, not a live outage.
Expected output is labeled in a later section.
Do not quote it as a vendor benchmark.

[loadgen] -> [admission] -> [queue depth=8]
                 |                |
            fail_closed        [worker]
                 |
          metrics.jsonl

The Four Costs You Actually Pay

Free capacity is not free under retries.
You pay wall time even when invoices stay quiet.
You also pay jobs that miss the deadline.

Track unit cost with this notebook formula:

cost_per_job =
  tokens_attempt
  + queue_age_ms
  + retry_n * tokens_attempt
  + missed_slo_penalty

On a free server, cash price can look like zero.
queue_age_ms and retry_n still compound fast.
A missed SLO is the invoice you cannot ignore.

Time, tokens, retries, and queueing move together.
If one field spikes, stop enqueueing new work.
Scaling later does not refund spent retries.

When Free Capacity Is The Wrong Bet

Use this table before you enqueue another job.
Treat it as an admission checklist, not folklore.
Three matching rows means fail closed now.

Evidence	Operational action	Rationale
`queue_age_ms > 5000` and `retry_count >= 1`	reject or shed	deadline slack is already gone
`retry_count >= 2` and `token_in` still rising	fail closed	retries are the spend
`worker_util < 30%` and p95 is high	inspect retries, do not scale	idle workers mean amplification
`deadline_slack_ms < service_p95`	do not enqueue	you cannot recover in line
reject ratio stays above 10%	leave free capacity	the queue became the product

If those rows fire inside one minute, stop.
Free capacity is the wrong bet in that window.
Pay for isolation, or cut the feature path.

Artifact: Retry-Budget Harness

The harness below is local and disposable.
It does not call a production model endpoint.
It records tokens, retries, and queue age.

Save it as retry_budget.py and run it once.

#!/usr/bin/env python3
"""Local retry-budget drill. Synthetic worker only."""
import json, os, random, time, uuid
from collections import deque

SLO_MS = 15000
MAX_AGE_MS = 5000
MAX_RETRY = 3
DEPTH = 8
TOKEN_IN = 800
TOKEN_OUT_CAP = 200

class Job:
    def __init__(self):
        self.id = uuid.uuid4().hex[:8]
        self.enqueued = time.monotonic()
        self.retry = 0
        self.tokens = 0

def deadline_slack_ms(job):
    age = (time.monotonic() - job.enqueued) * 1000
    return SLO_MS - age

def admit(queue, job, stats):
    age = (time.monotonic() - job.enqueued) * 1000
    if age > MAX_AGE_MS or deadline_slack_ms(job) < 2000:
        stats["rejected"] += 1
        return False
    if len(queue) >= DEPTH:
        stats["rejected"] += 1
        return False
    queue.append(job)
    stats["accepted"] += 1
    return True

def worker_once(job, inject_hold_s, inject_429):
    if inject_hold_s:
        time.sleep(inject_hold_s)
    if inject_429 and job.retry % 2 == 0:
        job.retry += 1
        job.tokens += TOKEN_IN
        return "retry"
    time.sleep(0.05)  # local stand-in for 2s service
    job.tokens += TOKEN_IN + random.randint(40, TOKEN_OUT_CAP)
    return "ok"

def main():
    queue = deque()
    stats = dict(accepted=0, rejected=0, retries=0, tokens=0, done=0)
    inject = os.getenv("INJECT", "none")
    hold = 8 if inject == "hold" else 0
    nack = inject == "429"
    t0 = time.monotonic()
    ages = []
    for i in range(40):
        job = Job()
        if not admit(queue, job, stats):
            continue
        while queue:
            cur = queue.popleft()
            ages.append((time.monotonic() - cur.enqueued) * 1000)
            result = worker_once(cur, hold, nack)
            stats["tokens"] += cur.tokens
            if result == "retry" and cur.retry < MAX_RETRY:
                stats["retries"] += 1
                if not admit(queue, cur, stats):
                    break
            else:
                stats["done"] += 1
        time.sleep(0.02)
    p95 = sorted(ages)[int(0.95 * (len(ages) - 1))] if ages else 0
    fail_closed = p95 > MAX_AGE_MS or stats["retries"] >= 12
    out = dict(stats, p95_age_ms=round(p95, 1),
               elapsed_ms=round((time.monotonic() - t0) * 1000, 1),
               decision="fail_closed" if fail_closed else "keep_open")
    print(json.dumps(out, indent=2))
    open("/tmp/retry-budget.json", "w").write(json.dumps(out))

if __name__ == "__main__":
    main()

Run the baseline, then inject one fault.

python3 retry_budget.py
INJECT=hold python3 retry_budget.py
INJECT=429 python3 retry_budget.py
rm -f /tmp/retry-budget.json

Expected output (local, labeled, not production):

{
  "accepted": 28,
  "rejected": 12,
  "retries": 19,
  "tokens": 9400,
  "done": 21,
  "p95_age_ms": 6800.0,
  "decision": "fail_closed"
}

Your numbers will differ on every laptop.
The fields must still move in the same direction.
If they do not, fix the harness before any server test.

Failure Injection And Rollback

Inject one fault at a time. Never stack them.
Record the five telemetry fields after each inject.
Reject work before p95 crosses fifteen seconds.

Hold the worker for eight seconds.
Force a 429 on every second attempt.
Duplicate the same job id twice.
Zero the token counter on purpose.

Rollback path after the drill:

Export ADMISSION=fail_closed on the proxy
Stop the injector with kill %1 or Ctrl-C
Drain remaining jobs with MAX_RETRY=0
Delete /tmp/retry-budget.json and any logs

Do not leave the injector running overnight.
A forgotten 429 loop burns the next rehearsal.
Cleanup is part of the test, not a courtesy.

Pick One Threshold And Defend It

Pick one primary threshold, not five equal ones.
For this workload, trip on queue age first.
Utilization lags. Token counters lag until completion.

Rationale you can hand a reviewer:

Queue age is visible before the SLO dies
Deadline slack equals remaining SLO minus age
Retries only matter after the first miss starts
Token totals arrive too late for admission

Local tripwires for the declared 15s SLO:

warn when queue_age_ms > 3000
reject when queue_age_ms > 5000
fail closed when retries are at least two and warn is already hot

Tune only after you replay the injector twice.
Do not copy these numbers into production configs.
They match this lab SLO and this queue depth only.

Where A Free Server Fits

You can stage the same admission loop against a free model endpoint and a free server after the harness fail-closes cleanly.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode currently offers free model access and a free server option you can use as a rehearsal lane. Use that lane to practice reject-and-drain, not to hide retry storms. If the local drill already fail-closes, dedicated capacity is the cheaper operational bet.

Confirm live quotas and server terms on the product page before you schedule a drill. Those numbers change. This article does not freeze them.

If you want that rehearsal, run this harness against the free server after local fail-closed stays green.

Limitations

This harness does not model GPU contention.
It does not model multi-tenant fairness rules.
It does not prove any vendor SLO or invoice.

Token units here are synthetic counters only.
They are not billed statements or capacity pledges.
Do not quote them in a budget review.

Short operational sentences skip distributed design.
Leave evaluation architecture to another owner.
You are diagnosing a deployed admission path.

Who Should Not Use This

Skip this if jobs have no deadline at all.
Skip this if the work is overnight batch only.
Skip this if you cannot shed or reject load.

Also skip it if you need named models or quotas.
This article does not claim hardware, duration, or permanence.
Read those from primary product docs at drill time.

What You Do At 02:14

You do not scale from twelve percent utilization.
You read queue age, retries, tokens, and slack.
Then you reject, drain, or pay for isolation.

Free capacity is a rehearsal tool under quiet load.
It is the wrong bet once retries amplify token cost.
Measure the four costs before the next page fires.

Your Free AI Server Has a Queue Ceiling: Measure It Before You Need It

Odd_Background_328 — Wed, 02 Sep 2026 13:35:51 +0000

It was 03:17 when the alert fired. The AI review bot had been silent for eleven minutes. No deploys. No config changes. The dashboard showed requests queuing behind a latency spike that came from nowhere.

That's the real cost of a free tier: you don't see the ceiling until you hit it.

This article is an ops drill for teams running on free AI infrastructure. I'll use MonkeyCode's free model access and free server option as the concrete example, because it's a usable entry point for small automation workloads. We'll measure the ceiling before the queue measures us.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free server is a shared bus

A free server is not your server. It's a shared resource with a capacity curve you don't control. But you still control how you load it.

Your job is to find the point where latency starts bending upward, then build a rollback path before that point.

Declare the workload first

Before running any numbers, write down exactly what you'll send:

Request type: code review prompt with a diff
Payload size: ~2,500 tokens per request
Concurrency: 1, 5, 10, 20
Timeout: 30 seconds
Success criteria: HTTP 2xx within 15 seconds

Don't invent a synthetic “average” workload. Use the real prompt your bot sends.

Probe with a small load

Start with a single request. Measure:

Time to first token
Total completion time
Response body length
Any retry or rate-limit headers

curl -s -o /dev/null -w "connect=%{time_connect} ttfb=%{time_starttransfer} total=%{time_total} code=%{http_code}\n" \
  -H "Content-Type: application/json" \
  -d '{"prompt":"Review this diff for concurrency bugs."}' \
  https://your-free-server.example.com/complete

That gives you a baseline. Write it down. If the baseline already exceeds 10 seconds, stop and reconsider the design.

Ramp concurrency until the curve bends

Use a load generator that prints latency percentiles. hey or vegeta both work. Here's a five-step ramp with a short cooldown between steps:

for c in 1 5 10 20; do
  echo "--- concurrency $c ---"
  hey -z 60s -c $c -m POST \
    -H "Content-Type: application/json" \
    -d '{"prompt":"Review this diff for concurrency bugs."}' \
    https://your-free-server.example.com/complete \
    | grep "Requests/sec\|p50\|p95\|p99\|Error"
done

Look for three signals:

p95 latency doubling from the previous step
Error rate climbing above 5%
Requests/sec plateauing while latency keeps rising

That intersection is your ceiling. Not the dashboard's number, not the vendor's claim. Yours.

Watch queue age, not just request rate

A flat request rate can hide an unhealthy queue. If the server accepts requests but processes them slowly, your pipeline accumulates stale jobs.

Add a metric for queue age: the time between a task entering the queue and the first model token arriving. If that age grows while request rate is stable, your ceiling is behind you.

# example: emit queue age from your worker
age_seconds=$(($(date +%s) - $task_enqueued_at))
if [ $age_seconds -gt 30 ]; then
  echo "WARNING queue_age=$age_seconds severity=high"
fi

Set an alert at 50% of your measured p95. The point of this drill is to know what “healthy” looks like before it stops looking that way.

Build the rollback decision table

After measuring, you get a small table like this:

Load	p95 latency	Error rate	Verdict
1	2.1s	0%	Safe
5	3.8s	0%	Acceptable for batch
10	9.4s	2%	Risky
20	23s	18%	Stop and roll back

Your thresholds will differ. Decide before production:

If p95 > 50% of request timeout, switch to a paid endpoint
If error rate > 5%, drain the queue and disable the bot
If queue age > 60s, emit an on-call page

Document the actual commands for each rollback step. A rollback you can't run at 3 AM is not a rollback.

When free capacity is the wrong bet

Free tiers are excellent for:

Non-blocking code review hints
Summaries that don't gate a deploy
Prototypes and internal tools
Bursts under a concurrency of 5

They are wrong for:

Any request that blocks a production release
Customer-facing latency SLAs
Sustained concurrency above your measured ceiling
Workloads where token consumption varies wildly per request

If your queue can fail independently of the model response, free is fine. If a late AI review stops a pipeline, pay for a bounded resource.

Limitations of this approach

This test measures one server at one moment. Shared free capacity means noisy neighbors can move the curve. Your production prompt might be longer than the test prompt. Quotas change without notice.

Schedule this drill monthly. Re-measure after every client or server upgrade. And never run this on a live workload that human users depend on.

The cheap option always has a cleanup cost

A free server saves money and costs time. The time is the measurement effort, the monitoring, and the rollback runbook. If you skip those, the only thing free is the failure.

Run the ramp, write down the numbers, and put the alert in before the queue does the same for you.

Stabilize AI Smoke Tests With a Token-Bucket Proxy Before the Free Server Bites Back

Odd_Background_328 — Tue, 01 Sep 2026 12:42:47 +0000

The CI dashboard is green. One merge request arrives, fans out into 14 parallel jobs, and every job calls the same AI endpoint to summarize a diff. Twelve jobs fail after a 60-second timeout. Green goes red in less than a minute.

The common reaction is to blame the AI provider. The actual failure sits one layer above: unbounded concurrency. Free shared capacity looks like a fire hose until everyone opens their valve at once.

This article shows a small, dependency-free proxy that turns an OpenAI-compatible endpoint into a rate-limited, circuit-broken service suitable for CI smoke tests, batch analysis, and other throwaway workloads.

What the free tier really gives you

MonkeyCode is an open-source project that provides free model access and a free server option for developers who want to experiment without provisioning their own GPU. That combination is valuable for non-critical pipelines. The fine print is operational, not contractual: the server is shared, latency fluctuates, and your jobs are not the only tenants.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Free infrastructure changes how you design clients. A paid API can absorb 20 concurrent requests and still return in 500 ms. A free shared server cannot, and it should not be expected to. The correct response is to protect the shared resource from your own burst, and to protect your pipeline from the shared resource's inevitable hiccups.

The architecture: a local egress proxy

The pattern is simple. Every AI client in your build writes to http://localhost:8765 instead of the real endpoint. The proxy applies two rules before forwarding.

Token bucket – caps call rate so your concurrent jobs cannot stampede the upstream.
Circuit breaker – stops traffic for a cool-down window after repeated 5xx responses or timeouts.

This gives you a reproducible boundary. You can tune it locally, watch the failure modes, and decide whether the free tier is a good fit for a specific workload.

The proxy code

The implementation uses only the Python standard library. Save it as ai_proxy.py and run it with python3 ai_proxy.py.

#!/usr/bin/env python3
"""Rate-limited, circuit-broken proxy for OpenAI-compatible endpoints."""
import json
import threading
import time
import urllib.request
import urllib.error
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

UPSTREAM = "http://localhost:8080/v1/chat/completions"  # change to MonkeyCode free server endpoint
RATE = 2        # steady requests per second
BURST = 5       # max burst size
FAIL_THRESHOLD = 3
COOLDOWN = 30   # seconds

class TokenBucket:
    def __init__(self, rate, burst):
        self.rate = rate
        self.burst = float(burst)
        self.tokens = float(burst)
        self.lock = threading.Lock()
        self.updated = time.monotonic()

    def acquire(self, timeout=10):
        deadline = time.monotonic() + timeout
        with self.lock:
            while True:
                now = time.monotonic()
                self.tokens = min(self.burst, self.tokens + (now - self.updated) * self.rate)
                self.updated = now
                if self.tokens >= 1:
                    self.tokens -= 1
                    return True
                if now >= deadline:
                    return False
                wait = (1 - self.tokens) / self.rate
                self.lock.release()
                time.sleep(min(wait, 0.1))
                self.lock.acquire()

class CircuitBreaker:
    def __init__(self, threshold, cooldown):
        self.threshold = threshold
        self.cooldown = cooldown
        self.failures = 0
        self.open_until = 0
        self.lock = threading.Lock()

    def allow(self):
        with self.lock:
            if time.monotonic() < self.open_until:
                return False
            return True

    def record(self, ok):
        with self.lock:
            if ok:
                self.failures = 0
                return
            self.failures += 1
            if self.failures >= self.threshold:
                self.open_until = time.monotonic() + self.cooldown
                self.failures = 0

bucket = TokenBucket(RATE, BURST)
breaker = CircuitBreaker(FAIL_THRESHOLD, COOLDOWN)

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        if not breaker.allow():
            self.send_error(503, "Circuit open; upstream cooling down")
            return
        if not bucket.acquire():
            self.send_error(429, "Rate limit exceeded; try again soon")
            return

        length = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(length)
        req = urllib.request.Request(UPSTREAM, data=body, headers={"Content-Type": "application/json"})
        try:
            with urllib.request.urlopen(req, timeout=60) as resp:
                data = resp.read()
                self.send_response(200)
                self.send_header("Content-Type", "application/json")
                self.end_headers()
                self.wfile.write(data)
                breaker.record(True)
        except urllib.error.HTTPError as e:
            breaker.record(e.code >= 500)
            self.send_error(e.code)
        except Exception:
            breaker.record(False)
            self.send_error(502, "Upstream unavailable")

if __name__ == "__main__":
    print(f"Proxy running on :8765 -> {UPSTREAM}")
    print(f"Rate={RATE} req/s, burst={BURST}")
    ThreadingHTTPServer(("127.0.0.1", 8765), Handler).serve_forever()

The proxy does not parse the prompt or track tokens. It is intentionally dumb. That keeps it trustworthy enough to put in a build pipeline.

Step-by-step: put it into a CI job

Start the proxy on the runner: python3 ai_proxy.py &
Point your AI client at http://127.0.0.1:8765/v1/chat/completions.
Keep the same API key the model provider expects; the proxy forwards headers only if you extend it, which this example does not do for brevity.
Run a small batch of 10 requests to verify the bucket works.
Launch the 14 parallel jobs again and record completion times.

That last step is the whole point. With RATE=2, the proxy admits no more than 5 requests in the first second, then 2 per second. The 14 jobs queue locally instead of piling onto the shared server. Some finish later, but far fewer fail.

Reading the numbers

The proxy gives you two explicit failure signals: HTTP 429 and HTTP 503. Deciding what to do with them depends on your workload.

Workload	Free model + free server	Self-hosted	Paid API
CI smoke test, low volume, tolerant to minutes	Yes, with proxy	Overkill	Acceptable
Interactive chat, sub-second SLO	No	Maybe	Best
Batch analysis, hours of runtime	Yes, with long cooldown	Best if stable	Expensive
User-facing feature, 2-second SLO	No	Yes	Yes

Use the decision table as a starting point. The proxy gives you numbers to adjust the thresholds: if you see 429s, lower RATE or increase the client's backoff. If you see 503s, the server is unhealthy; raise COOLDOWN or switch workloads entirely.

Limitations and who should skip this

This proxy is a guardrail, not a scaler. It does not make the free server faster. It does not turn the shared capacity into a contract. It only prevents your own clients from acting like a herd.

Do not use this approach when:

Your SLO requires sub-second latency. The local queue adds milliseconds, but the upstream variance will still dominate.
You have zero tolerance for dropped jobs. Use a paid or dedicated path.
You expect the proxy to solve upstream outages. The circuit breaker stops the bleeding, not the cause.

The example also omits API-key forwarding to keep the code short. In production, copy the Authorization header from the incoming request.

The operational takeaway

Free model access and a free server are a real way to keep experimentation costs at zero. They become hostile when your client design ignores their shared nature. A 40-line proxy restores the boundary.

Next time the CI goes red, check the concurrency before blaming the model. Your own burst is usually the first failure to fix.

If you want a quick, honest way to test MonkeyCode's free layer, run this proxy in front of it and push 100 requests. The output will tell you whether the free server fits your workflow or whether you need the more predictable option.

Free AI Capacity Is a Latency Bet: Run the 30-Minute Ops Drill

Odd_Background_328 — Mon, 31 Aug 2026 11:43:48 +0000

Your demo used a free AI endpoint. It answered in 800 ms. Then four teammates tried it. The queue doubled. Timeouts started. The free server never stopped accepting work. It just made you wait.

That is the real cost of free capacity. It is not the token price. It is the queue time, retry amplification, and developer hours spent debugging a "free" service that behaves like a shared weekend cluster.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Free Capacity Has a Queue Tax

Free model access and a free server option sound like a bargain. They are a bound on concurrency, not a guarantee of latency.

When you treat free capacity as a production promise, you are betting that the queue stays short. That bet fails at the worst moment: during a demo, a spike, or a teammate's urgent report.

I call this the cost_ops view. You measure time, tokens, retries, and queue depth. Then you decide whether free is still cheaper than paid.

The 30-Minute Ops Drill

Do this before you build a habit around free capacity. You need a load harness, a queue metric, and a threshold.

Step 1: Define the boundary

Write down the expected worst case:

Max concurrent requests: 5
Deadline slack: 2 seconds
Error budget: 1%
Token budget per request: 2,000

These numbers are your observable SLI targets. No target, no decision.

Step 2: Run a cheap load test

Use a simple Python script that sends concurrent requests and logs latency, queue age, and token usage. Here is a generic harness you can adapt to any free endpoint, including MonkeyCode's free server if you have access.

import asyncio, time, aiohttp

async def probe(session, url, results):
    start = time.perf_counter()
    try:
        async with session.post(url, json={"prompt": "ping"}) as resp:
            body = await resp.json()
            elapsed = time.perf_counter() - start
            results.append({
                "latency": elapsed,
                "status": resp.status,
                "tokens": body.get("usage", {}).get("total_tokens", 0)
            })
    except Exception as e:
        results.append({"latency": time.perf_counter() - start, "status": 0, "error": str(e)})

async def main(url, concurrency, requests):
    async with aiohttp.ClientSession() as session:
        results = []
        sem = asyncio.Semaphore(concurrency)
        async def limited():
            async with sem:
                await probe(session, url, results)
        await asyncio.gather(*[limited() for _ in range(requests)])
        return results

# Example: python load.py http://your-endpoint 5 50
if __name__ == "__main__":
    import sys, statistics
    url = sys.argv[1]
    concurrency = int(sys.argv[2])
    requests = int(sys.argv[3])
    results = asyncio.run(main(url, concurrency, requests))
    latencies = [r["latency"] for r in results if r.get("status") == 200]
    errors = [r for r in results if r.get("status") != 200]
    print(f"requests: {len(results)}")
    print(f"p50: {statistics.median(latencies):.3f}s")
    print(f"p95: {sorted(latencies)[int(len(latencies)*0.95)-1]:.3f}s")
    print(f"errors: {len(errors)} ({len(errors)/len(results)*100:.1f}%)")
    print(f"tokens: {sum(r.get('tokens', 0) for r in results)}")

This is a synthetic probe. It is not a benchmark. It gives you one afternoon's worth of evidence.

Step 3: Record the signal

Look at the output. Ask three questions:

Did p95 stay below your deadline slack?
Did the error rate stay below 1%?
Did token usage match your estimate?

If p95 is near the deadline, the queue is already eating your margin. Even with zero errors, latency is a cost.

Step 4: Add retry amplification

Your client retries on timeout. One slow request becomes three. That multiplies queue pressure. Track the retry count in your client logs.

A free server can handle 10 happy users. It cannot handle 10 users plus 20 retries. Retries are the hidden cost multiplier.

Decision Table: Free vs Paid Capacity

Condition	Stay Free	Move to Paid
Concurrent users < 3	Yes	No
p95 < 50% of deadline slack	Yes	No
Error rate < 1% over 1000 requests	Yes	No
Retry amplification > 1.5x	No	Yes
Token usage > 10x your quota estimate	No	Yes
Production traffic with no human fallback	No	Yes

This table is a starting point. Change the thresholds to match your actual workload.

Who Should Not Use This Approach

Do not use this drill if you have no deadline slack. If every request must return in 300 ms, free capacity is already the wrong bet. Do not use it if you cannot instrument the endpoint. No metrics, no decision.

Also avoid this approach if you need guaranteed isolation. Free servers are shared by definition. Your noise floor is someone else's batch job.

The Ops Conclusion

Free AI capacity is a latency bet, not a cost saving. Measure queue age, retry rate, and token utilization before you rely on it.

You can start today with MonkeyCode's free model access and free server option. Run this drill against their endpoint, or any free endpoint you already use.

Then decide with evidence, not hope. That is the only cost_ops move that survives production.

Free Capacity Is a Boundary: Load-Test Your AI Reviewer Before Production

Odd_Background_328 — Sun, 30 Aug 2026 10:51:52 +0000

The hot take says AI promotes every developer to reviewer. Nobody tests the reviewer. This post fixes the second part.

You added an AI reviewer. Merges got slower. Builds started failing. The AI code quality is not the problem. The queue is.

You operate an AI-backed queue now. Treat it like one. Measure it first. Use the free tier as a probe.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The Three Queues You Inherited

AI review is a network call. That call brings three hidden queues.

1. The API Queue

Your review request waits for GPU availability. You cannot see this queue. You only see latency.

2. The Rate Limit Queue

Exceed requests per minute. Now you wait or retry. The provider controls this timer.

3. The Retry Queue

Retries pile up behind fresh requests. Every retry adds more load. The system slows down. This is your fault.

You need a load test before production. Not for code accuracy. For operational boundaries.

Free Capacity as a Test Probe

MonkeyCode gives you free model access and a free server. Use that as your boundary probe.

Send a burst of review requests. Watch the latency curve. Watch the token burn rate. Watch the error rate.

A free server has shared CPU. Expect CPU steal. That is not a bug. That is a feature of the test.

Run It Locally First

Run the probe from your laptop before touching CI.

python probe_ai_reviewer.py --requests 50 --concurrency 5

Watch the token counter closely. A local run shows your baseline network cost. CI adds checkout time, runner startup, and shared CPU contention.

The Probe Artifact

Create a workflow file. Trigger it manually. Never put it in a critical path.

name: ai-review-load-test
on: workflow_dispatch
jobs:
  probe:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run review probe
        run: python probe_ai_reviewer.py --requests 200 --concurrency 10

Write the probe script. It posts a diff to your review endpoint. Then collects telemetry.

import time
import concurrent.futures
from collections import deque

retry_events = deque()
latencies = []
token_usage = []

def call_review(diff):
    start = time.monotonic()
    try:
        resp = post_review(diff)  # your integration
        latencies.append(time.monotonic() - start)
        token_usage.append(resp["usage"]["total_tokens"])
    except TimeoutError:
        retry_events.append(time.monotonic())

with concurrent.futures.ThreadPoolExecutor(max_workers=10) as ex:
    futures = [ex.submit(call_review, diff) for _ in range(200)]
    for f in concurrent.futures.as_completed(futures):
        pass  # collect results in call_review

latencies.sort()
p50 = latencies[len(latencies) // 2]
p95 = latencies[int(len(latencies) * 0.95)]
avg_tokens = sum(token_usage) / len(token_usage)
retry_rate = len(retry_events) / 200

print(f"p50: {p50:.2f}s")
print(f"p95: {p95:.2f}s")
print(f"avg tokens/review: {avg_tokens:.0f}")
print(f"retry rate: {retry_rate:.2%}")

Run it. Log the output. That tells you where the boundary sits.

What Signal to Track

Track four numbers. Latency median. Latency tail. Token consumption. Retry ratio.

A high p95 means unstable capacity. A high retry ratio means you are approaching a limit.

Token consumption gives you the true cost per review. Multiply that by daily reviews. Now you have a budget.

Decision Table: Free vs Paid

Workload Shape	Free Tier Works?	Ops Reason
Async batch, low urgency	Yes	Queues absorb latency spikes
Interactive CI gate	No	p95 breaks the GitHub timeout
Large refactor PRs	No	Token cap truncates analysis
Personal experiment	Yes	Zero cost, high signal

The Time Cost

Free tokens are not the only cost. Engineering time is the expensive part.

A blocked merge costs forty minutes. A rerun costs ten. A context switch costs hours.

Measure that. Reviews per day times p95 latency times waiting engineers. That is the queue tax.

This queue tax hits free tiers hardest. Retries amplify the load. Token burn accelerates.

Set a retry budget. Default to one attempt. Fail closed after that.

if attempts > 1:
    alert("AI review retrying. Check provider status.")

Rollback Runbook

Always ship the rollback before the feature. Use a feature flag.

export AI_REVIEW_ENABLED=false

Measure p95 in staging. If it exceeds your threshold, set the flag to false in production. No code rollback needed.

Pin the prompt. Pin the model version. Change one variable at a time.

Who Should Not Use This

Skip this if every PR is urgent. You do not need a review queue. You need a smaller batch.

Skip this if you have no monitoring. Free tokens cannot replace telemetry.

Skip this if you cannot act on the number. A load test without a rollback plan is theater.

The Takeaway

Free capacity is a maximum, not a promise. It shows you the breaking point.

Test the reviewer before it reviews you. Probe the queue. Set thresholds. Ship the rollback plan.

Reject Unmeasured AI Reviews Before Latency Breaks Your Pipeline

Odd_Background_328 — Sat, 29 Aug 2026 09:40:42 +0000

Monday, 09:14. Your AI reviewer approved a PR in 92 seconds.
Tuesday, 16:40. The review queue holds 214 items. P95 age is 41 minutes.
Nothing in your repository changed. The free token bucket just ran dry.
Which operational action do you take? Add capacity, throttle input, or switch models.

The recent AI trend made every developer a reviewer of machine output. Your toolchain changed. Your SLOs did not. This post is a cost-operations drill for free AI review capacity.

Free capacity is a queue, not a contract

Free model access and a free server option make experiments cheap. They also hide utilization until the bucket empties. You must measure time, tokens, retries, and queue age.

MonkeyCode is an open-source coding assistant project. Its current public materials advertise a 10M token free grant and a free server option. As of 2026-08-29, treat those terms as promotional, not as an SLO.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use the free grant for a controlled drill. Do not put it in an unmeasured production lane.

Topology: one review lane, two states

Here is the declared topology. A webhook receives PR events. The MonkeyCode server runs locally. It calls the free model gateway. Results enter a queue. A consumer applies labels. A CI gate blocks merge.

reviewer:
  provider: monkeycode-free
  server: self-hosted
  queue:
    max_age_seconds: 1800
    retry_limit: 3
    dead_letter_topic: reviews-dead
  gate:
    reject_on_queue_age: 2700
    block_merge: true

This YAML is illustrative. It is not a MonkeyCode-specific schema. It treats queue age as the contract. If a review waits longer than 2700 seconds, the gate rejects the PR. That is your rollback signal.

Declared test conditions

Run this before you trust any free tier.

Workload: 50 synthetic PR events per hour
Generator: local script posting to the webhook
Deadline slack: 30 minutes
Token budget: the 10M grant as published
Failure injection: at T+60, cap throughput to 20% for 20 minutes
Window: 3 hours
Output: queue age, token usage, retry count, tokens per review

Observable output

Record metrics every minute. Here is expected output from a local run under those conditions.

time, queued, p95_age_s, tokens, retries, tokens_per_review
T+00, 0, 0, 0, 0, 0
T+30, 12, 210, 1420, 1, 56
T+60, 9, 340, 3100, 2, 57
T+80, 41, 960, 3300, 7, 54
T+120, 88, 1880, 3800, 31, 59
T+180, 104, 2620, 4100, 68, 62

This output is labeled expected, not measured. Replicate the drill on your own hardware.

Why tokens per review misleads

Token usage per review looks stable. It stays near 56 tokens. Queue age climbs past 1800 seconds. The hidden variable is throughput, not unit cost.

Free tokens do not create capacity. They create a budget. Capacity still depends on the model gateway, the server, and the network. When the gateway throttles, your budget stays high and your queue grows.

Track four metrics:

p95_queue_age_s: deadline slack consumption
tokens_per_review: unit cost
retry_rate: amplification pressure
dead_letter_count: permanent loss

Alert when p95_queue_age_s passes 70 percent of your rejection threshold. At 2700 seconds, that means alert at 1890 seconds.

Decision table: when free is the wrong bet

Use this table before you route production work.

Situation	Use free tier?	Reason
Prototype with two developers	Yes	Cost zero, blast radius small
Production gate at 50 PR/hour	No	Queue age exceeds slack
Deadline slack under five minutes	No	Free quotas do not guarantee latency
Failover to paid gateway exists	Yes	High queue age triggers switch
Data residency required	No	Model endpoint may be remote
Nightly batch with no deadline	Yes	Draft reviews can wait

Free capacity is the right bet only when a queue can wait.

Failure drill: drain before the deadline breaks

The drill has four steps.

Step 1: pause new submissions.

curl -s -X POST http://localhost:8080/api/reviews/pause

Step 2: read queue age.

curl -s http://localhost:8081/metrics | grep review_queue_age_seconds

Step 3: drain with lower concurrency.

consumer --max-inflight 5 --topic reviews-queue

Step 4: roll back to a paid gateway.

curl -s -X PATCH http://localhost:8080/config -d '{"provider":"paid-gateway"}'

Do not restart the consumer before you check the dead-letter topic. That is how retry storms start.

Cleanup

After the drill, remove the test queue. Reset the gate to a neutral state. Delete synthetic PR events. Verify with a single canary PR.

curl -s -X DELETE http://localhost:8080/queues/review-drill
curl -s http://localhost:8081/health

The drill is complete only when the queue is empty and the canary PR passes.

Limitations

I did not benchmark MonkeyCode here. The 10M token figure is the published grant on the project page, not a measured quota. It can change without notice.

A free server option is good for isolation drills. It is not a substitute for availability. Treat it as an experiment environment.

Who should not use this approach

Skip this workflow if you need a written SLA. Skip it if your compliance policy prohibits third-party model endpoints. Skip it if your pipeline cannot tolerate a queue outage.

Free capacity is a starting condition, not a contract. Measure queue age, token burn, and retry rate. Reject unmeasured AI reviews before latency breaks your pipeline.

Try MonkeyCode's free grant on a disposable queue first. Then decide whether free is the bet your deadlines can survive.

Free Tokens Are Not an SLO: An Ops Cost Drill for AI Batch Queues

Odd_Background_328 — Fri, 28 Aug 2026 03:17:44 +0000

Free Tokens Are Not an SLO: An Ops Cost Drill for AI Batch Queues

This week, two numbers trended: a harness at 100%, a model at 30%. For platform teams, a better pair is queue age and deadline slack. This article is a cost drill for the simplest AI batch path: free tokens, free server, non-negotiable deadline.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. That capacity is real. It is not an SLO. The tokens cost nothing. The queue is patient. Your deadline is not.

The missing variable

Token cost is easy to measure. Operations cost is easy to ignore. A free endpoint converts a per-token bill into a per-hour bill. The bill becomes your time, your retries, and your queue age.

This drill keeps the ledger honest. It answers one question: what does a completed request cost when the token price is zero?

Topology

# worker.py (minimal, single-threaded)
import queue
import time
import csv

work = queue.Queue()
for i in range(1000):
    work.put({"id": i, "prompt_tokens": 512, "max_tokens": 256})

def call_model(payload):
    # replace with your free model endpoint
    return {"ok": True, "in_tokens": 512, "out_tokens": 180}

completed = 0
retries = 0
started_at = time.time()

while not work.empty():
    item = work.get()
    attempt = 0
    while attempt < 4:
        try:
            call_model(item)
            completed += 1
            break
        except Exception:
            retries += 1
            attempt += 1
            time.sleep(2 ** attempt)

The worker is deliberately single-threaded. Free capacity often serializes. Serialization turns a token problem into a time problem.

Declared test conditions

1,000 requests.
One worker process.
One free model endpoint.
No client-side rate limiting.
Deadline: 30 minutes.
Ledger: one CSV row per request.

Ledger and report

# cost_ledger.py
import csv
import time

HOURLY_OPS_COST = 50.0  # loaded engineering rate, adjust

def record(item, elapsed, retries):
    with open("ledger.csv", "a", newline="") as f:
        csv.writer(f).writerow([item["id"], round(elapsed, 3), retries])

def report(completed, retries, elapsed_s, deadline_s):
    ops_cost = (elapsed_s / 3600.0) * HOURLY_OPS_COST
    retry_ratio = retries / max(1, completed)
    slack = deadline_s - elapsed_s
    print(f"completed={completed}")
    print(f"retries={retries}")
    print(f"wall_clock_s={elapsed_s:.1f}")
    print(f"ops_cost_usd={ops_cost:.2f}")
    print(f"retry_ratio={retry_ratio:.3f}")
    print(f"deadline_slack_s={slack:.1f}")
    return retry_ratio, slack

The token spend is zero. The ledger rows still carry a cost.

Reading the ledger

A row looks like this:

id,elapsed_s,retries
0,1.234,0
1,3.456,2

High retries on early rows mean throttling, not a crash. Growing elapsed times mean the queue is the bottleneck. Both are signals for one control decision.

Observed output (labeled expected)

Normal free-tier conditions:

Metric	Expected value
Completed	987 / 1000
Retries	214
retry_ratio	0.217
wall_clock_s	1742 (29:02)
ops_cost_usd	24.19
deadline_slack_s	58

That is the good case. 58 seconds of slack. 24 dollars of human time on a "free" job.

Failure injection

Cut the network to the endpoint for three minutes.

tc qdisc add dev eth0 root netem loss 100%

The worker retries with backoff. The queue grows. The ledger fills.

After the fault:

Metric	After injection
Completed	801 / 1000
Retries	1034
retry_ratio	1.291
wall_clock_s	1900
ops_cost_usd	26.39
deadline_slack_s	-20

Negative slack. Deadline gone. Tokens still free.

Decision rule

Read the ledger as a control loop. Two thresholds matter more than token spend:

retry_ratio > 0.10
deadline_slack_s < 0

When either fires, stop the worker and re-route.

kill $(pgrep -f worker.py)
export MODEL_ENDPOINT="https://paid.example/v1"
./worker.py --resume ledger.csv

The ledger turns a crash into a resume. You know which items completed.

Cleanup

tc qdisc del dev eth0 root
kill %1
rm -f worker.py
# keep ledger.csv if you want a cost trend

Limitations

This drill assumes you control the client. It does not measure shared CPU noise, hidden rate limits, or model quality. Those need separate experiments.

Do not use this pattern for regulated data, payments, or any job with a hard SLO. The paid path exists for a reason.

Who should skip this

Teams with strict latency SLOs.
Teams with no retry budget.
Teams with no one awake when the free endpoint degrades.

Free capacity is a bet. The ledger is the odds table. Run the drill, set the thresholds, and stop before the deadline sign turns negative.

Your Free Token Budget Has a Queue Tax: Measure Cost Per Completed Request

Odd_Background_328 — Wed, 26 Aug 2026 10:50:07 +0000

Your nightly summarization job missed its 06:00 deadline. The model endpoint reported p50 latency at 812 ms. Healthy, said the dashboard.

The queue depth told a different story. You moved the job to a free model endpoint to cut spend. Token cost dropped to zero.

Completion time tripled. You traded a line item for a schedule risk. That trade is the queue tax.

Why the queue tax exists

Free model capacity is shared capacity. Your requests queue behind everyone else's. The endpoint's p50 latency stays flat. The queue depth grows. Your job's wall time grows with it.

Token price does not capture this. Wall time does.

Cost per token is the wrong metric

Finance sees $0 per million tokens. SRE sees a job that finishes when it finishes. Both are right. Neither is useful.

The metric that matters is cost per completed request. It includes tokens, retries, queue wait, and wall time. A free endpoint that fails 20% of requests is not free. It is a retry generator.

The probe

Here is a small, reproducible probe. It sends a declared workload to any OpenAI-compatible endpoint. It records queue wait, latency, retries, and tokens per request. Then it computes throughput and cost per completed request.

#!/usr/bin/env python3
"""cost_per_completed.py - measure the real cost of an LLM endpoint.

Declared workload: N requests, C concurrency, one prompt, max_tokens.
Output: JSON report with throughput, retries, and cost per completed request.
"""
import argparse
import asyncio
import json
import statistics
import time

import httpx


async def attempt(client, cfg, sem, results, created_at):
    async with sem:
        queue_ms = (time.monotonic() - created_at) * 1000
        for retry in range(1, cfg.max_retries + 1):
            t0 = time.monotonic()
            try:
                resp = await client.post(
                    f"{cfg.base_url}/chat/completions",
                    json={
                        "model": cfg.model,
                        "messages": [{"role": "user", "content": cfg.prompt}],
                        "max_tokens": cfg.max_tokens,
                        "temperature": 0,
                    },
                )
                resp.raise_for_status()
                data = resp.json()
                latency_ms = (time.monotonic() - t0) * 1000
                tokens = data.get("usage", {}).get("total_tokens", 0)
                results.append({
                    "ok": True,
                    "retries": retry,
                    "queue_ms": queue_ms,
                    "latency_ms": latency_ms,
                    "tokens": tokens,
                })
                return
            except Exception as exc:
                latency_ms = (time.monotonic() - t0) * 1000
                if retry == cfg.max_retries:
                    results.append({
                        "ok": False,
                        "retries": retry,
                        "queue_ms": queue_ms,
                        "latency_ms": latency_ms,
                        "tokens": 0,
                        "error": str(exc),
                    })
                    return
                await asyncio.sleep(cfg.backoff_base * (2 ** (retry - 1)))


async def run(cfg):
    results = []
    async with httpx.AsyncClient(
        timeout=cfg.timeout,
        headers={"Authorization": f"Bearer {cfg.api_key}"},
    ) as client:
        sem = asyncio.Semaphore(cfg.concurrency)
        start = time.monotonic()
        tasks = [
            attempt(client, cfg, sem, results, time.monotonic())
            for _ in range(cfg.requests)
        ]
        await asyncio.gather(*tasks)
        wall = time.monotonic() - start
    return results, wall


def report(results, wall, price_per_million):
    ok = [r for r in results if r["ok"]]
    completed = len(ok)
    latencies = sorted(r["latency_ms"] for r in ok)
    tokens = sum(r["tokens"] for r in ok)
    retries = sum(r["retries"] - 1 for r in ok) + sum(
        r["retries"] for r in results if not r["ok"]
    )
    token_cost = tokens * price_per_million / 1_000_000
    p95_index = min(len(latencies) - 1, int(len(latencies) * 0.95)) if latencies else 0
    return {
        "requests": len(results),
        "completed": completed,
        "failed": len(results) - completed,
        "retry_attempts": retries,
        "throughput_per_sec": round(completed / wall, 3) if completed else 0,
        "p50_latency_ms": round(statistics.median(latencies), 1) if latencies else None,
        "p95_latency_ms": round(latencies[p95_index], 1) if latencies else None,
        "tokens_per_completed": round(tokens / completed, 1) if completed else 0,
        "token_cost_usd": round(token_cost, 4),
        "seconds_per_completed": round(wall / completed, 3) if completed else None,
        "wall_seconds": round(wall, 3),
    }


def main():
    p = argparse.ArgumentParser(description="Measure cost per completed LLM request.")
    p.add_argument("--base-url", required=True, help="OpenAI-compatible base URL")
    p.add_argument("--api-key", required=True, help="API key")
    p.add_argument("--model", required=True, help="Model name")
    p.add_argument(
        "--prompt",
        default="Summarize this incident in one paragraph: queue depth spiked at 02:00 and the batch job missed its deadline.",
    )
    p.add_argument("--requests", type=int, default=50)
    p.add_argument("--concurrency", type=int, default=4)
    p.add_argument("--max-tokens", type=int, default=200)
    p.add_argument("--max-retries", type=int, default=3)
    p.add_argument("--timeout", type=float, default=60.0)
    p.add_argument("--backoff-base", type=float, default=1.0)
    p.add_argument(
        "--price-per-million",
        type=float,
        default=0.0,
        help="USD per million tokens. Use 0 for free endpoints.",
    )
    p.add_argument(
        "--deadline-seconds",
        type=float,
        default=0.0,
        help="Your real deadline slack. The report flags if the workload would miss it.",
    )
    cfg = p.parse_args()

    results, wall = asyncio.run(run(cfg))
    rep = report(results, wall, cfg.price_per_million)
    if cfg.deadline_seconds and rep["wall_seconds"] > cfg.deadline_seconds:
        rep["deadline_verdict"] = "MISS"
    else:
        rep["deadline_verdict"] = "OK"
    print(json.dumps(rep, indent=2))


if __name__ == "__main__":
    main()

How to run it

Install one dependency. Then run the probe against your paid endpoint first. The probe expects an OpenAI-compatible /chat/completions endpoint. Check your provider's docs for the base URL and model name.

pip install httpx

python3 cost_per_completed.py \
  --base-url "$PAID_ENDPOINT_URL" \
  --api-key "$PAID_API_KEY" \
  --model "$PAID_MODEL" \
  --requests 50 \
  --concurrency 4 \
  --deadline-seconds 300

Save the report. Then point the same probe at the free endpoint. Use the same workload, the same concurrency, the same deadline. The only variable is the endpoint.

What the report tells you

seconds_per_completed is the queue tax. Compare it across endpoints with the same workload. retry_attempts is the hidden cost. Each retry burns wall time and deadline slack. token_cost_usd is what finance sees. The other fields are what production feels.

Read failed first. Nine failed requests mean nine prompts that never completed. Read retry_attempts second. Seventeen retries mean seventeen extra round trips. Then read seconds_per_completed. It is the only number that combines all of it.

Expected output for a healthy endpoint (illustrative, not a measured run):

{
  "requests": 50,
  "completed": 50,
  "failed": 0,
  "retry_attempts": 0,
  "throughput_per_sec": 3.6,
  "p50_latency_ms": 812.4,
  "p95_latency_ms": 1940.2,
  "tokens_per_completed": 210.0,
  "token_cost_usd": 0.0,
  "seconds_per_completed": 0.278,
  "wall_seconds": 13.9,
  "deadline_verdict": "OK"
}

Expected output for a congested shared endpoint (illustrative, not a measured run):

{
  "requests": 50,
  "completed": 41,
  "failed": 9,
  "retry_attempts": 17,
  "throughput_per_sec": 0.9,
  "p50_latency_ms": 2410.7,
  "p95_latency_ms": 8800.3,
  "tokens_per_completed": 210.0,
  "token_cost_usd": 0.0,
  "seconds_per_completed": 1.11,
  "wall_seconds": 45.5,
  "deadline_verdict": "MISS"
}

The token cost is identical: zero. The operational cost is not. The second run burns 31.6 more seconds, fails 9 requests, and misses the deadline. That is the queue tax in numbers.

The decision table

Workload shape	Free capacity	Paid capacity
Background batch, no deadline, idempotent	Right bet	Overkill
Hard deadline, slack under 2× runtime	Wrong bet	Right bet
Bursty arrivals, spiky demand	Wrong bet	Right bet
Steady low rate, retry-tolerant	Right bet	Overkill
User-facing, p95-sensitive	Wrong bet	Right bet

The decision is about slack, not price. Free capacity is a bet that your workload can absorb queue time.

The table assumes one thing: you measured first. A decision without a probe is a guess. A guess about shared capacity is a bet against the queue.

When free capacity is the wrong bet

Hard deadline, less than 2× slack. The queue tax eats slack faster than tokens save money.
Callers retry without backoff. Retries amplify load on shared capacity. Your retry storm becomes everyone's latency.
You watch p50 latency, not queue depth. p50 hides the tail. Queue depth shows the backlog.
Partial completion is unacceptable. Free capacity is shared. Shared means noisy. Noisy means you cannot promise completeness.

When the verdict is MISS

Do not tune the prompt. Do not raise the timeout. The endpoint cannot carry the workload. That is the measurement.

Three options. First, reduce concurrency. Lower contention, fewer retries. Second, split the batch. Smaller jobs fit in slack windows. Third, keep the paid endpoint. Use free capacity for the parts that can wait.

The MonkeyCode workflow

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project that offers free model access and a free server option. Both fit this workflow. The free model access gives you an endpoint to probe. The free server gives you a place to run the probe.

Run the probe from the free server against the free model endpoint. That tests the full path: network, compute, and model. Then run the same probe against your paid endpoint. Compare seconds_per_completed and retry_attempts. The numbers decide. Not the price per token.

MonkeyCode's free model access is 10M tokens as of this writing. Quotas change. Re-run the probe before you commit a workload.

The free server is not a production cluster. Treat it as a staging target. Run the probe there. Run small jobs there. Keep the critical path elsewhere.

Run the probe against MonkeyCode's free model access and free server. The numbers will tell you if the fit is real.

Limitations

This probe uses one prompt shape. Real workloads mix prompt sizes and token counts. Run it with your actual prompt distribution.

The probe measures throughput, not correctness. A completed request can still contain a bad summary. Add your own validation pass before you trust the output.

Free capacity changes with demand. Last month's numbers are not this month's numbers. Re-measure before each schedule change.

The free server is shared infrastructure. Your neighbors affect your results. That is the point of the probe.

Who should not use this: workloads with hard external deadlines and no retry tolerance. Also, workloads that need guaranteed p95 latency. Free capacity cannot promise that.

Cleanup and rollback

The probe writes no state. Kill it with Ctrl-C. Delete the JSON report if you saved it.

Rollback is one environment variable. Keep the paid endpoint in your deployment manifest. Flip back when the free endpoint misses the deadline. Free capacity is an experiment, not a migration.

Free tokens are a budget decision. Queue time is a production decision. Measure both before you commit a workload.