Reject Unbounded Agent Loops Before Free CPU Steal Beats Deadline Slack

You open the pager at 08:02 UTC this morning. The free worker still reports eight percent CPU. Oldest batch job has waited forty-seven minutes already.

That contradiction is the whole incident. Idle CPU did not mean spare capacity. One agent eval loop held the only worker thread.

The story you are paging on

A nightly JSON-repair eval starts at 02:14 UTC. It calls a model, parses output, then retries on schema errors. The loop has no iteration cap and no wall-clock budget.

You expected the job to finish before 03:00 UTC. Interactive morning traffic has a ten-minute deadline. The eval job is still inside iteration eighty-six at 08:00.

Queue depth looks tiny because nothing else is admitted. Utilization looks healthy because the loop is mostly waiting. Deadline slack is the metric that actually broke.

Ask one operational question from that evidence. Which job do you kill, and which threshold justified it?

Separate observation from inference

Observed: one in-flight job, CPU near idle, queue age climbing. Observed: loop_iteration keeps incrementing in worker logs. Observed: interactive jobs never leave Redis.

Inference you must not treat as fact: the model is slow. Inference you must not treat as fact: you need more replicas. Inference you must not treat as fact: free capacity will absorb the next spike.

You are looking at intra-request looping, not queue backlog. A single request is a hidden scheduler. It steals the worker without showing busy CPU.

Topology you can stand up locally

Run one API process, one worker, and one Redis list. Do not add autoscaling. Do not add a second replica. The point is shared free-server contention, not cluster design.

Proposed lab topology, unexecuted until you run it:

client -> admission.py -> redis:6379 list key jobs
                       -> worker.py (concurrency = 1)
                       -> stdout JSON logs on fd 1

Keep the worker concurrency at one on purpose. Free-tier boxes often give you that shape. Your production page will rhyme with this lab.

Minimal compose file

# docker-compose.loop-lab.yml
services:
  redis:
    image: redis:7-alpine
    ports: ["6379:6379"]
  worker:
    build: .
    environment:
      REDIS_URL: redis://redis:6379/0
      WORKER_CONCURRENCY: "1"
      LOOP_HARD_CAP: "8"
      TOKEN_SOFT_CAP: "4000"
      DEADLINE_MS: "600000"
    depends_on: [redis]

Declare those env vars before you start traffic. Caps that exist only in a wiki will not fire. You want the worker process to enforce them.

Declared workload, not a vibe

Label this block as a lab recipe, not a production benchmark. You are measuring admission behavior under contention. You are not ranking model quality.

Declared conditions:

• One worker thread, no preemption, no autoscaler.
• Interactive jobs: 20 requests, 30-second deadline slack each.
• Eval job: schema-repair loop, one tool call per iteration.
• Injected failure: parser always rejects until iteration 80.
• Success rule: interactive p95 stays under 120 seconds.
• Fail rule: any interactive job with negative deadline slack.

If you change concurrency, you changed the experiment. If you add a second worker, you hid the bug. Keep the topology ugly and honest.

Artifact: admission plus loop budget

The original artifact is a tiny gate. It rejects or preempts work using three signals. Queue age, loop count, and deadline slack beat CPU idle.

# admission.py -- lab gate, not a framework
import json, time, os, redis

HARD_LOOP = int(os.environ.get("LOOP_HARD_CAP", "8"))
TOKEN_SOFT = int(os.environ.get("TOKEN_SOFT_CAP", "4000"))
MAX_QUEUE_AGE_MS = int(os.environ.get("MAX_QUEUE_AGE_MS", "120000"))
INTERACTIVE_SLACK_MS = int(os.environ.get("DEADLINE_MS", "600000"))

r = redis.Redis.from_url(os.environ["REDIS_URL"])

def queue_age_ms():
    raw = r.lindex("jobs", 0)
    if not raw:
        return 0
    job = json.loads(raw)
    return int(time.time() * 1000) - job["enqueued_at_ms"]

def admit(job):
    age = queue_age_ms()
    slack = job["deadline_at_ms"] - int(time.time() * 1000)
    if job["kind"] == "eval_loop" and age > MAX_QUEUE_AGE_MS:
        return "reject_eval_queue_age"
    if job["kind"] == "eval_loop" and slack < INTERACTIVE_SLACK_MS:
        return "reject_eval_slack"
    if job.get("loop_iteration", 0) >= HARD_LOOP:
        return "reject_loop_cap"
    if job.get("tokens_used", 0) >= TOKEN_SOFT:
        return "reject_token_soft"
    return "admit"

Put the same checks inside the worker loop. Admission at enqueue is not enough. A job that was healthy at 02:14 can become the thief at 03:00.

# worker.py excerpt -- preemption inside the loop
while True:
    job = json.loads(r.brpop("jobs")[1])
    for i in range(HARD_LOOP + 1):
        job["loop_iteration"] = i
        decision = admit(job)
        log({"event": "loop_tick", "job_id": job["id"], **job, "decision": decision})
        if decision != "admit":
            r.lpush("jobs.dlq", json.dumps(job))
            break
        # labeled stub: replace with your real model call
        job["tokens_used"] = job.get("tokens_used", 0) + 250
        if parse_ok(job):
            r.lpush("jobs.done", json.dumps(job))
            break

Log every tick as JSON. You need loop_iteration, tokens_used, queue_age_ms, and deadline_slack_ms on the same line. Graphs without those fields will lie to you.

Fault injection you can run in five minutes

Do not wait for another overnight miss. Inject the parser failure locally. Watch the gate choose eval death over interactive death.

export REDIS_URL=redis://127.0.0.1:6379/0
export LOOP_HARD_CAP=8
export TOKEN_SOFT_CAP=4000
export MAX_QUEUE_AGE_MS=120000
export DEADLINE_MS=600000
python worker.py &

python - <<'PY'
import json, time, os, redis, uuid
r = redis.Redis.from_url(os.environ["REDIS_URL"])
now = int(time.time() * 1000)
eval_job = {
  "id": str(uuid.uuid4()),
  "kind": "eval_loop",
  "enqueued_at_ms": now,
  "deadline_at_ms": now + 6 * 60 * 60 * 1000,
  "loop_iteration": 0,
  "tokens_used": 0,
  "force_parse_fail_until": 80
}
r.lpush("jobs", json.dumps(eval_job))
for n in range(20):
    t = int(time.time() * 1000)
    r.lpush("jobs", json.dumps({
      "id": f"interactive-{n}",
      "kind": "interactive",
      "enqueued_at_ms": t,
      "deadline_at_ms": t + 30000,
      "loop_iteration": 0,
      "tokens_used": 0
    }))
print("seeded eval thief plus 20 interactive jobs")
PY

Expected output, clearly labeled expected, not measured production data:

{"event":"loop_tick","kind":"eval_loop","loop_iteration":8,"decision":"reject_loop_cap"}
{"event":"loop_tick","kind":"interactive","job_id":"interactive-0","decision":"admit"}

If you instead see interactive jobs sitting behind iteration 40, the gate never loaded. Check env vars, then check that admit() runs inside the loop. Enqueue-time checks will not save you here.

Thresholds and why they exist

Pick thresholds from slack, not from CPU charts. CPU idle is a trailing comfort metric. Deadline slack is the leading one.

Use this rationale in the runbook:

• Queue age over 120 seconds: eval work is already stealing wake-ups.
• Loop count at 8: schema repair is no longer cheap search.
• Token soft cap at 4000: the loop is paying rent without finishing.
• Interactive slack under 10 minutes: batch eval loses, always.

Ask which signal you would page on. Queue age catches the silent wait. Loop count catches the busy thief. Slack tells you who must die first.

Utilization is the wrong primary page. A looping worker waits on I/O. You will see low CPU and still miss the SLO.

When free capacity is the wrong bet

Free model access and a free server option can host this drill. They are spare rehearsal capacity, not a contract. Treat them as a fault-injection box you can burn.

You can point the stubbed worker at MonkeyCode free model access for the same loop. You can park that worker on the free server option while you rehearse drain. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep the product out of the page policy. If the eval loop needs a hard SLO, free capacity is the wrong bet. Move it to reserved workers with preemption and a kill switch.

Decision table you can paste into the runbook:

Work	Loop cap	Queue age	Deadline slack	Action
Interactive chat	2	any	under 10 min	Admit; never queue behind eval
Nightly JSON eval	8	over 120s	interactive still positive	Reject eval to DLQ
Agent tool loop	8	any	negative	Kill in-flight; do not retry
Backfill embeddings	1	over 30s	any interactive waiting	Hold; free box is wrong
Incident debug prompt	1	any	page slack under 15 min	Reject; debug elsewhere

Free capacity is the wrong bet when any row needs preemption. One thread cannot preempt itself. Retries without a loop cap are just a quieter outage.

Failure handling and rollback

When reject_loop_cap fires, do not restart the worker first. Restarting without a DLQ just relaunches the thief. Drain, then reopen interactive traffic.

# 1. freeze enqueue of kind=eval_loop in admission.py
# 2. signal the worker to stop after the current tick
kill -TERM $(pgrep -f worker.py)

# 3. snapshot in-flight state before it vanishes
redis-cli LRANGE jobs 0 -1 > /tmp/jobs-live.jsonl
redis-cli LRANGE jobs.dlq 0 -1 > /tmp/jobs-dlq.jsonl

# 4. requeue only interactive kinds
python - <<'PY'
import json, redis, os
r = redis.Redis.from_url(os.environ["REDIS_URL"])
for raw in r.lrange("jobs", 0, -1):
    job = json.loads(raw)
    if job.get("kind") != "interactive":
        r.lpush("jobs.dlq", raw)
        continue
    r.lpush("jobs.replay", raw)
r.delete("jobs")
for raw in r.lrange("jobs.replay", 0, -1):
    r.rpush("jobs", raw)
print("interactive replay armed")
PY

# 5. start worker with caps still set
LOOP_HARD_CAP=8 python worker.py

Rollback means caps stay on. Turning caps off to "catch up" repeats 02:14. If eval must finish, schedule it on a reserved box after interactive slack recovers.

Cleanup after the drill:

redis-cli DEL jobs jobs.dlq jobs.done jobs.replay
kill -TERM $(pgrep -f worker.py) || true
docker compose -f docker-compose.loop-lab.yml down -v

Leave no orphaned worker on the free server. An orphaned loop is how this page starts. Confirm pgrep is empty before you walk away.

Who should not use this approach

Do not use a single-worker loop cap as a multi-tenant scheduler. Platform teams with GPU preemption already have a better hammer. Do not use free rehearsal capacity as high availability.

Skip this pattern if your eval loop is the product path. Skip it if legal retention needs every repair attempt stored. Skip it if you cannot log loop_iteration beside deadline_slack_ms.

This lab also will not help design-level routing debates. You are operating one deployed worker under steal. Leave cluster evaluation architecture to another desk.

Limitations

Numbers above are declared lab thresholds, not fleet measurements. Parser-fail-until-80 is injected, not observed in production. Token increments of 250 are stubs for log shape.

MonkeyCode free model access and the free server option have no SLA in this write-up. Do not invent quotas, hardware, or duration from this article. If the operator later publishes measured limits, replace these stubs with those primary numbers.

The gate rejects work; it does not make a bad schema succeed. If eight repairs never parse, fix the prompt or the schema offline. Burning loops overnight only hides that bug behind idle CPU.

What you do at 08:02

You do not scale from an idle CPU graph. You read loop ticks against deadline slack. You reject the eval loop before interactive jobs go negative.

Then you restore caps, drain the DLQ, and delete the lab keys. If you already rehearse on that free server, run this loop-starvation fixture before overnight evals share the worker. Spare capacity that cannot preempt is not spare when a loop never yields.