DEV Community

Odd_Background_328
Odd_Background_328

Posted on

Reject Agent Tasks Before Queue Age Breaks the SLO

#devops #ai #observability #reliability

A queue can accept work long after it has lost the ability to finish that work on time. “Message accepted” is not availability when the task's deadline will expire before a worker starts it.

Apply backpressure at admission using predicted queue age.

A small estimate

For one worker pool:

predicted_wait_seconds =
  queued_work_seconds / effective_worker_concurrency
Enter fullscreen mode Exit fullscreen mode

Use observed service time by task class rather than one global average. A five-second lint task and a twenty-minute repository migration should not contribute the same queue weight.

Admit a request only when:

predicted_wait + p95_service_time + safety_margin < task_deadline
Enter fullscreen mode Exit fullscreen mode

Also enforce per-tenant queued work and active concurrency. Without those limits, one bursty tenant can consume the entire future capacity of the pool.

Controller sketch 

def admit(task, queue, limits):
    tenant_work = queue.estimated_work(task.tenant)
    if tenant_work + task.estimated_seconds > limits.tenant_queue_seconds:
        return "reject_tenant_limit"

    wait = queue.total_estimated_work() / limits.effective_concurrency
    finish = wait + task.p95_seconds + limits.safety_margin
    if finish >= task.deadline_seconds:
        return "reject_deadline"

    return "accept"
Enter fullscreen mode Exit fullscreen mode

The estimate will be wrong sometimes. Measure the error and update task-class estimates from completed work. Do not let tasks self-report trusted cost without a server-side cap.

Metrics that explain overload 

agent_admission_total{decision,task_class,tenant_tier}
agent_queue_estimated_work_seconds
agent_queue_oldest_age_seconds
agent_task_start_delay_seconds
agent_service_seconds{task_class}
agent_deadline_miss_total{stage}
Enter fullscreen mode Exit fullscreen mode

Avoid tenant IDs in metric labels; that can create unbounded cardinality. Put sampled tenant detail in structured logs with retention controls.

Burst test

  1. Start a disposable pool with two workers.
  2. Submit two long tasks from tenant A.
  3. Burst short tasks from tenants A and B.
  4. Verify A reaches its queue budget without blocking all of B.
  5. Submit a task whose deadline is shorter than predicted wait.
  6. Confirm rejection includes a stable reason and retry guidance.
  7. Drain the queue and verify admission recovers without a restart.

Record prediction error, p50/p95 start delay, accepted tasks that miss deadlines, and rejected tasks that would have succeeded. Backpressure needs both false-accept and false-reject evidence.

A queue-length threshold is easy but workload-blind. Admission based on estimated work and deadlines fails earlier—and more honestly—before accepted tasks become guaranteed disappointments.

Top comments (0)