Reject Agent Jobs Before Deadline Slack Hits Zero: A Local Admission-Control Drill

The alert that told me nothing

At 02:14 our LLM-backed coding-agent queue paged: p99 task latency 41s vs SLO 30s. Utilization was at 63%. CPU fine. Provider rate limits not hit. The dashboards said the system was "healthy" while jobs were silently missing their deadlines.

The contradiction resolved once I graphed queue age vs remaining deadline slack instead of utilization. Roughly 18% of queued jobs had negative expected slack — even if a worker picked them up instantly, they could not finish before their deadline. We were burning worker capacity on work that was already lost.

Which operational action follows from that evidence? Not autoscaling. Not alerting harder. Admission control: reject work at enqueue time when it can no longer meet its deadline, and prove the policy locally before it ever runs in production.

This post is the drill: topology, declared workload, a local fault-injection harness, the telemetry fields that made the decision auditable, and the rollback path.

Topology and configuration

The staging rig I used for this drill is small on purpose — one producer, one queue, two workers:

# compose.yaml (excerpt)
services:
  queue:
    image: redis:7-alpine
    command: redis-server --appendonly no
  worker:
    build: ./worker
    deploy:
      replicas: 2
    environment:
      QUEUE_URL: redis://queue:6379
      MAX_INFLIGHT: "4"
      DEFAULT_DEADLINE_MS: "30000"
      ADMISSION_POLICY: slack_v1        # off | log_only | slack_v1
      MODEL_ENDPOINT: ${MODEL_ENDPOINT}
  producer:
    build: ./producer
    environment:
      QUEUE_URL: redis://queue:6379
      RATE_RPS: "6"
      DURATION_S: "300"

Key points:

• ADMISSION_POLICY is a runtime flag. off reproduces the incident; log_only emits what would be rejected without dropping anything; slack_v1 enforces.
• MAX_INFLIGHT=4 per worker is deliberately tight so backpressure shows up in minutes, not hours.
• MODEL_ENDPOINT is pluggable because I don't run these drills against the production model account. For experiments like this I pointed the worker at a free tier instead.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Concretely, I used MonkeyCode's free model access for the synthetic completions in this drill, and its free server option to host the worker/queue sandbox, which kept the experiment off our production quota and off the production cluster. Two honest caveats: I did not measure per-token latency parity with our production provider (so absolute numbers below are drill-relative, not production SLAs), and I can't speak to how long either free option stays available — verify current terms before you build a habit around them.

The admission policy, stated as math

At enqueue time, for job j:

expected_service_ms(j) = percentile_estimate(job_class, last_200_completions)
slack_ms(j)            = deadline_ms(j) - queue_depth_ahead_ms(j) - expected_service_ms(j)
admit if slack_ms(j) > SLACK_FLOOR_MS

queue_depth_ahead_ms is the sum of expected_service_ms for everything already queued and in-flight, divided by worker concurrency. SLACK_FLOOR_MS=5000 in this drill — the floor exists because the estimate has error bars, and I'd rather reject early than optimistically.

Rejected jobs get a structured 429-style response with retry_after_ms derived from when slack is projected to go positive again — clients that retry blindly at fixed intervals were part of the original incident's retry storm, so the retry hint is part of the contract.

Declared workload

Everything below ran under these declared conditions, so the numbers mean something:

• 6 req/s Poisson arrivals, 300 s, job classes mixed: 70% small (est. 800 ms service), 25% medium (4 s), 5% large (12 s)
• 2 workers × 4 in-flight = 8 concurrent slots
• Deadlines: 30 s for all classes
• Fault injection at t=120 s: service latency of large jobs inflated ×3 (simulating provider degradation — the same shape as our real incident)
• Estimator warmed with 200 completions per class before the run

Offered load is ~5.1 slots of demand against 8 slots of capacity in steady state — comfortably under capacity until the fault lands, which is exactly the regime where naive utilization metrics lie to you.

Expected output (labeled)

I ran this three times per policy. Run-to-run variance was small, but these are expected outputs from my drill environment, not production measurements:

Policy	Deadline-miss rate	Wasted work (jobs executed that missed)	Rejected early	Tail latency (admitted)
off	~21% after t=120	~18% of executed jobs	0	p99 ≈ 41 s
log_only	~21% (unchanged)	~18%	(flagged ~19%)	p99 ≈ 40 s
slack_v1	~3%	~2%	~17%	p99 ≈ 26 s

The log_only row is why the flag exists: the rejected-count projection let me sanity-check the policy against real traffic shape before it dropped a single job. If log_only had projected rejecting 40% of jobs, my estimator or floor was wrong — and I'd rather learn that from a log than from an incident review.

One observation I did not expect: deadline-miss rate under slack_v1 didn't go to zero. Large jobs admitted just before t=120 still got caught by the fault because the estimator was blind to a degradation that hadn't happened yet. Admission control bounds the damage; it doesn't predict the future. More on that in limitations.

Telemetry fields that made this auditable

Every enqueue decision emits one log line:

{
  "event": "admission_decision",
  "job_id": "j_8f31", "job_class": "medium",
  "decision": "reject", "policy": "slack_v1",
  "slack_ms": -2400, "slack_floor_ms": 5000,
  "est_service_ms": 4100, "queue_ahead_ms": 29400,
  "estimator_n": 200, "retry_after_ms": 8100
}

The fields that earned their keep during the drill: slack_ms (the actual decision variable), estimator_n (so I can discount decisions made on a cold estimator), and retry_after_ms (so I could verify clients weren't retrying into the same rejection window). If you only emit the decision and not the inputs, you cannot debug a bad decision six weeks later.

Failure handling and rollback

Things that went wrong or plausibly could, and the answers:

1. Cold estimator. On worker restart, estimator_n < 50 flips the policy to log_only automatically. Never enforce on an estimate you wouldn't sign your name to.
2. Estimator lag during regime change. The post-fault misses above. Mitigation: the estimator window is recency-weighted, and I alert on est_service_ms drifting >2× within 60 s — that's a provider-degradation signal in its own right.
3. Retry storms on rejection. Enforced server-side: the same client rejected 3× within one retry_after_ms window gets a hard backoff response.
4. Rollback. ADMISSION_POLICY=log_only is an env-var change and a rolling restart — no schema, no state migration, ~40 s to fully revert. Because the policy is pure (it only reads queue state), reverting never corrupts anything. The queue itself is untouched by rollback.
5. Cleanup after the drill. docker compose down -v, and point MODEL_ENDPOINT back at nothing — the drill has no persistent state by design, so there is nothing to forget.

Thresholds: what I'd actually alert on

My question to you, because I changed my own mind during this drill: would you alert on queue age, utilization, or projected deadline slack?

Before this, I alerted on queue age. The drill convinced me that's the wrong primary signal — age tells you how long jobs have waited, not whether waiting longer is hopeless. My current primary is the fraction of queued jobs with negative projected slack (page at >5% sustained 2 min), with utilization as a secondary context field only. Utilization at 63% was the number that made the original incident look fine.

Limitations and who should not use this

• Deadline-honest clients required. If your producers lie about deadline_ms (or don't have one), the policy has nothing to compute. Fix the contract first.
• Highly variable service times. If your job classes have bimodal or unpredictable durations, percentile estimates are weak and the floor has to be so large you reject useful work. This fits agent-style workloads with stable task shapes better than arbitrary compute.
• This is not a substitute for provider failover. Admission control sheds work that can't succeed; it does not make work succeed. Pair it with the drain playbook (I wrote up the drain-side drill separately) for the degradation case.
• My absolute numbers are drill-relative. The free model endpoint I used has different latency characteristics than a production provider, so reuse the method, not the milliseconds. Re-run the drill against your own endpoints before trusting any threshold.

If your queue workload is homogeneous and deadlines are soft, a simpler concurrency cap will get you most of the benefit with none of the estimator machinery. Start there. But if you've ever paged on latency while every utilization graph looked calm, instrument the slack — even in log_only — and see what your queue has been hiding.

If you want a zero-cost sandbox to run this kind of drill without touching production quota, MonkeyCode's free model access plus free server option is what I used; the compose file above drops in unchanged.