DEV Community

Odd_Background_328
Odd_Background_328

Posted on

Drain Coding-Agent Workers Before Provider Degradation Breaks the SLO

#devops #sre #ai #observability

0. Alert first, architecture second

At 09:14 the dashboard looks healthy: CPU 38%, Redis memory steady, workers polling. The contradictory signal is smaller and worse: queue_age_p95_seconds crosses 25s while provider_5xx_rate is still only 2.1%. No page fires yet because the alert is wired to utilization, not deadline risk.

Which action follows from that evidence: add workers, restart workers, or stop admitting new agent tasks and drain the in-flight ones?

My rule: when queue age rises while provider errors are still low, treat it as an early degradation drill. Freeze admission, snapshot the queue, drain by deadline class, and roll back only after a canary proves the path is clean.

I used MonkeyCode's free model access to review an early version of the failure matrix and its free server option as a throwaway place to think through the control-plane flow; neither is required for the harness below. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Do not run the load test against any free tier; keep it local or on your own disposable infra.

1. Topology for a local drill

Run this on one Linux host with Docker. Declared lab workload: 1 Redis queue, 3 workers, 1 admission controller, 1 fake provider endpoint behind Toxiproxy. Job mix: 70% lint-fix with 60s deadline, 25% test-patch with 180s deadline, 5% refactor-plan with 900s deadline.

# docker-compose.yaml
services:
  redis:
    image: redis:7-alpine
    command: ["redis-server", "--appendonly", "no"]
    ports: ["6379:6379"]

  toxiproxy:
    image: ghcr.io/shopify/toxiproxy:latest
    ports: ["8474:8474", "18080:18080"]

  fake-provider:
    image: python:3.12-alpine
    working_dir: /app
    volumes: ["./provider.py:/app/provider.py"]
    command: ["python", "provider.py"]
    environment:
      PORT: "8080"

  worker:
    image: python:3.12-alpine
    working_dir: /app
    volumes: ["./worker.py:/app/worker.py"]
    command: ["python", "worker.py"]
    deploy:
      replicas: 3
    environment:
      REDIS_URL: redis://redis:6379/0
      PROVIDER_URL: http://toxiproxy:18080/v1/complete
      WORKER_MAX_INFLIGHT: "2"
      DEADLINE_SWEEP_MS: "250"

  controller:
    image: python:3.12-alpine
    working_dir: /app
    volumes: ["./controller.py:/app/controller.py"]
    command: ["python", "controller.py"]
    environment:
      REDIS_URL: redis://redis:6379/0
      ADMIT_QUEUE_AGE_P95_MAX_S: "20"
      DRAIN_PROVIDER_5XX_RATE: "0.05"
      DRAIN_QUEUE_AGE_P95_S: "25"
      CANARY_REQUIRED_OK: "3"
Enter fullscreen mode Exit fullscreen mode

Create a Toxiproxy route from toxiproxy:18080 to fake-provider:8080, then add latency and a 5xx fault during the drill. The provider is intentionally dumb: it returns {"ok":true} after a sleep that can be forced high.

2. Worker contract: deadline, lease, and exit reason

Every job carries job_id, class, deadline_ts, attempt, and idempotency_key. Workers never delete a job after provider timeout; they requeue with attempt+1 unless now > deadline_ts - 2s, in which case they emit job_expired and move it to failed:{date}.

# worker.py excerpt
def run_job(job):
    started = time.time()
    remaining = job["deadline_ts"] - started
    if remaining < 2:
        emit("job_expired", job_id=job["job_id"], queue_class=job["class"],
             deadline_slack_s=round(remaining, 3), attempt=job["attempt"])
        return "expired"

    try:
        resp = post_provider(job, timeout=min(10, max(1, remaining - 1)))
        emit("provider_call", job_id=job["job_id"], status=resp.status_code,
             latency_s=round(time.time() - started, 3), attempt=job["attempt"])
        return "done" if resp.status_code == 200 else "retry"
    except Timeout:
        emit("provider_timeout", job_id=job["job_id"], attempt=job["attempt"],
             timeout_s=round(min(10, max(1, remaining - 1)), 3))
        return "retry"
Enter fullscreen mode Exit fullscreen mode

Telemetry fields to keep stable across tools: queue_age_p95_seconds, oldest_job_age_seconds, deadline_slack_p10_seconds, provider_5xx_rate, provider_timeout_rate, worker_inflight, drain_state, canary_ok_count, rollback_reason. If a metric cannot answer "admit, drain, or rollback?", it is decoration.

3. Admission controller: reject before the SLO is the incident

The controller is the only writer allowed to set drain_state. Workers read it but do not set it. That keeps recovery auditable.

# controller.py excerpt
def decide(m):
    if m["rollback_active"]:
        return "hold", "rollback_active"
    if m["canary_ok_count"] < CANARY_REQUIRED_OK and m["drain_state"] == "recovering":
        return "hold", "canary_not_green"
    if m["provider_5xx_rate"] >= DRAIN_PROVIDER_5XX_RATE or m["queue_age_p95_seconds"] >= DRAIN_QUEUE_AGE_P95_S:
        return "drain", "provider_or_queue_age"
    if m["queue_age_p95_seconds"] >= ADMIT_QUEUE_AGE_P95_MAX_S or m["deadline_slack_p10_seconds"] < 5:
        return "reject_new", "admission_threshold"
    return "admit", "normal"
Enter fullscreen mode Exit fullscreen mode

Threshold rationale: queue age is the leading indicator, utilization is the lagging indicator, and deadline slack is the customer-facing risk. I would rather reject new work at queue_age_p95_seconds >= 20 than discover at 45s that every worker is busy on already-doomed jobs.

4. Fault injection and expected evidence

Representative output only; your numbers will differ. This is a drill harness, not a benchmark.

docker compose up -d --build
curl -X POST localhost:8474/proxies \
  -d '{"name":"provider","listen":"0.0.0.0:18080","upstream":"fake-provider:8080"}'
./load.py --rate 12/s --duration 180s --mix lint-fix:70,test-patch:25,refactor-plan:5
Enter fullscreen mode Exit fullscreen mode

Healthy gate, expected shape:

state=admit queue_age_p95_seconds=6.8 deadline_slack_p10_seconds=41.2 provider_5xx_rate=0.000 worker_inflight=6
Enter fullscreen mode Exit fullscreen mode

Inject provider trouble:

curl -X POST localhost:8474/proxies/provider/toxics \
  -d '{"name":"latency","type":"latency","attributes":{"latency":3500,"jitter":700}}'
curl -X POST localhost:8474/proxies/provider/toxics \
  -d '{"name":"errors","type":"http","attributes":{"status":503,"probability":0.08}}'
Enter fullscreen mode Exit fullscreen mode

Expected transition:

state=reject_new reason=admission_threshold queue_age_p95_seconds=21.7 deadline_slack_p10_seconds=9.4
state=drain reason=provider_or_queue_age provider_5xx_rate=0.081 queue_age_p95_seconds=27.9 oldest_job_age_seconds=63.1
worker action=stop_pull lease_released=14 expired=3 requeued=11
Enter fullscreen mode Exit fullscreen mode

Observed behavior means the counters above came from Redis and logs in the drill. Architectural inference: the provider is not yet hard-down, but the deadline margin is already gone, so more replicas would mostly increase doomed in-flight work.

5. Drain and recovery drill

Drain order matters. Do not kill workers globally; stop pulls first, let short-deadline jobs expire explicitly, and preserve idempotency keys so clients can safely retry later.

  1. Set drain_state=drain and admission=closed.
  2. Snapshot: LRANGE queue:pending 0 -1, ZRANGE queue:delayed 0 -1 WITHSCORES, HGETALL job:{id} for in-flight leases.
  3. Workers finish current call only if deadline_slack_s > provider_timeout_s + 1; otherwise requeue or expire.
  4. Remove Toxiproxy faults or point PROVIDER_URL at the known-good endpoint.
  5. Set drain_state=recovering; run canary jobs one per class.
  6. Require CANARY_REQUIRED_OK=3 consecutive successes and deadline_slack_p10_seconds > 15 before admission=open.

Rollback path if recovery goes bad:

docker compose exec redis redis-cli SET drain_state drain
docker compose exec redis redis-cli SET admission closed
docker compose up -d --force-recreate worker
# restore last good env
sed -i 's/PROVIDER_URL=.*/PROVIDER_URL=http:\/\/toxiproxy:18080\/v1\/complete/' .env
Enter fullscreen mode Exit fullscreen mode

Rollback reason must be written to rollback_reason: canary_failed, queue_age_not_recovering, duplicate_side_effect, or manual_operator_hold. If you cannot name the reason, do not reopen admission.

6. Decision table

Signal combination Action Why
CPU high, queue age flat, slack high Observe only Utilization without deadline risk is not an incident
Queue age rising, 5xx low Reject new, prepare drain Backlog is forming before hard failure
5xx high or timeouts high Drain now Retries amplify provider pain
Recovery canary passes, slack still low Hold admission Provider works but deadlines are still unsafe
Duplicate side effects found after recovery Roll back to closed and audit idempotency keys Correctness beats availability

7. Limitations and who should not use this

This harness proves control-plane behavior, not model quality, provider capacity, or customer latency. It will not catch prompt regressions, token-cost spikes, region-specific provider failures, or database contention outside the queue path. Do not use free or shared hosted capacity for soak tests, secret-bearing jobs, regulated data, or as evidence that production can absorb the same rate. If your jobs have irreversible side effects without idempotency keys, fix that before practicing drain.

Cleanup after the drill:

docker compose down -v
docker volume prune -f
rm -rf ./snapshots ./failed-jobs.jsonl
Enter fullscreen mode Exit fullscreen mode

If you want a concrete next step, run the same drill against one non-critical queue this week and tune exactly one threshold: ADMIT_QUEUE_AGE_P95_MAX_S. Keep the change small enough that the next alert tells you whether it helped.

Top comments (0)