Agent Runtimes Have an Autoscaler, Not a Scheduler

Originally published at webofmike.com on 2026-09-26. The demo repo and every command in it were run before publishing.

Every capacity knob in an agent runtime today is an efficiency knob. Pack more agents onto fewer pods, snapshot the idle ones, scale the pool to match demand, drive utilization up. That work is real and it pays. But efficiency policy only answers one question, which is how much capacity should exist. It never answers the other one: when there is not enough, who loses. Right now the runtime answers that second question by accident, and the accident has a shape worth naming.

I have been running kagent on Agent Substrate for a while now, and my working notes and labs are in themsquared/kagent-substrate-demo. The density story works. I wrote it up in Thousands of AI Agents on Tens of Pods. This post is about the part I do not think anyone has solved, including me.

What the runtime actually decides today

Here is the entire capacity contract of a WorkerPool, pulled from the live CRD on my kind cluster running substrate v0.0.8:

kubectl get crd workerpools.ate.dev -o json | \
  python3 -c "import sys,json;d=json.load(sys.stdin);print(sorted(d['spec']['versions'][0]['schema']['openAPIV3Schema']['properties']['spec']['properties']))"

['ateomImage', 'replicas', 'sandboxClass', 'sandboxConfigName', 'template']

Four settings and a pod template. How many workers, what image, what sandbox class, what sandbox config. ActorTemplate, which is the object that describes an agent, carries containers, pauseImage, sandboxClass, snapshotsConfig, volumes, and workerSelector. There is no field anywhere in that surface that says this agent matters more than that one.

The second thing worth knowing is what happens at the boundary. Substrate does not queue when the pool is full. It rejects:

substrate worker pool has no free workers

So the queue is not absent, it just is not in the runtime. It lives in whatever retry logic the callers happen to have, which means an autoscaler on top is measuring demand secondhand. The policy I shipped in my own visualizer reads like this:

target      = busy + queued, clamped to [2, 8]
scale UP    straight to target when queued > 0 for 2 samples (6s), cooldown 8s
scale DOWN  straight to max(demand over last 30s window) when that peak < slots
            for 6 samples (~18s), cooldown 20s

That is a decent autoscaler and I stand by the reasoning behind it, which is written up in notes/autoscaling.md. Note what the input is. It is busy + queued. A count. The policy knows how much work is waiting and knows nothing at all about whose work it is.

That is fine while capacity is elastic. Scaling is a genuinely good answer to contention right up until it is not available, and it stops being available at a replica ceiling, a node pool limit, a GPU budget, or a model provider rate limit. Past that line, adding capacity is off the table and something has to lose. The runtime has no opinion about what.

The three things that break at capacity

These are the standard failure modes of any system with contention and no scheduling discipline, and they all apply here directly.

The noisy neighbor wins by retrying hardest

When the pool is full and rejection is the answer, the next freed slot goes to whichever caller happens to retry into that window. Nothing in that race tracks importance. A batch agent doing overnight enrichment with a tight retry loop beats an incident responder with a polite backoff every time, and the incident responder is the one that looks broken. With no admission control the runtime cannot decline low-value work to protect high-value work. It can only decline whatever asked at an unlucky moment.

Priority inversion is the default, not the edge case

In a system with explicit priorities, inversion is a bug you hunt. Here it is the resting state. A low priority actor that got a slot holds it for the length of its session, and sessions with an LLM in them are long and variable. A critical agent arriving a second later waits for a turn it cannot jump. Worse, a wedged session can pin a slot indefinitely: I hit a killed session leaving a ghost actor holding a worker while the pool looked idle, fixed only by restarting the pool deployment. When the holder of a contended resource cannot be preempted, priority is not a policy, it is a hope about arrival order.

Strict priority starves the bottom of the list

This is the failure mode people walk into the moment they fix the first two. Add a priority field, always serve the highest first, and the lowest priority agent runs when the system is quiet and never when it is busy, which means it never runs, because busy is the only state anyone cares about. Strict ordering without an aging or fairness term is a starvation generator. Every scheduler that survived production added a counterweight: aging, weighted fair queueing, deficit round robin, reserved floors with burst above them. You have to pick one.

Kubernetes solved the mechanism, at the wrong layer

The interesting part is that the mechanism already exists in the stack, one layer down and pointed somewhere else. Grep the substrate CRDs for anything priority-shaped:

kubectl get crd workerpools.ate.dev actortemplates.ate.dev -o json | \
  grep -oE '"(priorityClassName|weight)"' | sort | uniq -c

   1 "priorityClassName"
   2 "weight"

Two fields, counted three times because weight also appears in a required list. Both sit inside the WorkerPool's pod template: spec.template.priorityClassName, and the weight on spec.template.nodeAffinity.preferredDuringSchedulingIgnoredDuringExecution. ActorTemplate, the object that describes an agent, contributes nothing.

priorityClassName is real Kubernetes pod priority and preemption. It works and it is well understood. It decides whether a worker pod gets a node, and it will evict other pods to make that happen. What it cannot do is decide which actor gets which worker slot, because at that layer every worker is identical and every actor is invisible. Kubernetes is scheduling the container that hosts the agents. Nobody is scheduling the agents.

That gap is the whole point. The unit of contention moved up a layer and the scheduling primitives did not follow it. And it is not only worker slots: the genuinely scarce resource in most of these systems is upstream tokens and provider rate limit, shared by every agent in the fleet and represented nowhere in the runtime.

The API-server people have been here before. API Priority and Fairness exists because rate limiting by volume punishes everyone equally and protects nothing. It sorts requests into flows and gives each flow a share, so one caller flooding the server does not sink the rest. That is the shape the agent layer needs, pointed at worker slots and token budgets.

Kueue already built this, one abstraction over

I nearly wrote that nobody has solved this, and that would have been wrong. Kueue has been building exactly these primitives for a couple of years, aimed at batch and ML workloads, and it has essentially every piece:

• WorkloadPriorityClass, cluster-scoped, referenced by a workload through the kueue.x-k8s.io/priority-class label. It is deliberately separate from pod PriorityClass: it governs admission ordering where pod priority governs eviction. Kueue's authors hit the same layering problem and concluded the answer was a second priority concept, not a reuse of the first.
• ClusterQueue with a nominalQuota and a borrowingLimit, Cohort so queues can borrow each other's unused quota, and namespaced LocalQueue so a tenant's work groups under quota that belongs to them.
• Preemption with two algorithms: classic, which greedily prefers victims that are borrowing, then lowest priority, then most recent admission; and fair sharing, which uses weighted share values and preempts only when share-based conditions are met.

That last one is the detail I would not have guessed. Kueue's answer to starvation is not aging, it is fair sharing, a weighted share per cohort member bounding how far ahead any tenant can get. Aging appears in neither concept page. The most mature implementation of this in the ecosystem picked the share-based counterweight over the time-based one, and that is a design position rather than an oversight.

So the mechanism is not missing from Kubernetes, it is missing from the agent layer, and my complaint is narrower than the one I started with. But the glue is not trivial, and it matters why Kueue is not a drop-in. Kueue admits a Workload, which becomes pods a scheduler places on nodes, and it decides once, at admission. An agent runtime needs a decision per session, many times per actor, about which warm slot a restored snapshot lands in, on the timescale of a snapshot restore rather than a pod boot. Right shape, wrong granularity. Pointing Kueue at a WorkerPool buys nothing, because the thing it would manage is worker pods, and worker pods are the layer where this question is already answerable and already uninteresting.

The mechanism half is solved computer science that somebody still has to port. It is not the part I find hard.

The hard half: who is the arbiter?

Say you ship the field. priority: critical on a SandboxAgent, weighted fair queueing under it, aging so nothing starves, preemption with a floor. Good design. Now answer the operational question, which is who gets to set that field, and on what authority.

Because priority is not a measurement. It is a claim, and every claimant is biased in the same direction. Every team rates its own workload critical, and the uncomfortable part is that they are usually not wrong from where they sit. The revenue team's agent really does touch revenue. The security team's agent really is a control. The SRE's agent really is on the incident path. Ask each of them to classify their own work and you get a fleet where everything is critical, which is arithmetically identical to a fleet where nothing is.

So the decentralized answer degrades to no policy at all, not because people game it, though they will, but because sincere local judgment does not aggregate into a global ordering. The centralized answer fails differently: a platform team that owns the priority table becomes the arbiter of which business unit matters, which is a job nobody in platform wants and nobody outside it accepts. It also cannot scale, because one arbiter cannot know enough about a hundred agents across a dozen domains to rank them, and every ranking becomes an escalation.

Kubernetes actually took a position on this, and it is the most useful thing in the whole design. PriorityClass is a cluster-scoped object:

$ kubectl get priorityclass
NAME                      VALUE        GLOBAL-DEFAULT   PREEMPTIONPOLICY
system-cluster-critical   2000000000   false            PreemptLowerPriority
system-node-critical      2000001000   false            PreemptLowerPriority

An admin defines the classes. A workload only gets to reference one. But the load-bearing piece is not the scoping, it is that a class can be made expensive:

kubectl explain resourcequota.spec.scopeSelector.matchExpressions.scopeName

ENUM:
    BestEffort
    CrossNamespacePodAffinity
    NotBestEffort
    NotTerminating
    PriorityClass
    Terminating
    VolumeAttributesClass

You can scope a ResourceQuota to a PriorityClass. That means a namespace can be granted the right to run, say, four critical pods and no more. Priority stops being a free label and becomes a budgeted, finite thing that the claiming team has to spend and therefore has to reason about. The team still decides what matters to them, which is correct, because they are the ones who know. They just cannot decide that everything does.

That is the shape I would want for agents. Not a central arbiter ranking every agent, and not a free-text priority field on every SandboxAgent, but a small set of centrally defined classes with real preemption semantics, allocated to teams as a quota they own and spend. Central control over what the tiers mean and how much of each exists. Local control over which of your agents gets one.

The pattern generalizes past Kubernetes, which is most of why I trust it. SRE error budgets work the same way: nobody argues about whether reliability matters, they argue about a finite number that runs out. Network QoS only works when the high class is policed rather than merely marked, because an unpoliced class is one everybody marks.

The objection to my own framing

Here is what has bothered me since I wrote that section, and I think it is the strongest argument against this whole post. Criticality might be the wrong axis, and choosing it is what manufactures the arbitration problem.

Most of what looks like a priority question is a latency question. An incident-response agent does not need to be more important than a nightly enrichment agent. It needs a bounded time to first token. The enrichment agent does not want priority at all: it wants to finish before morning and does not care whether that is 2am or 5am. Those are different requirements, and a single priority number flattens them onto one axis where they do not belong. Once flattened they have to be compared, and comparing them is the thing no arbiter does well.

Separate them and most of the contention stops needing a decision. Give latency-sensitive work a small reserved floor it does not share. Let everything else run in the remainder, preemptible, with fair sharing across tenants so no one takes it all. Nothing gets ranked against anything, because the two classes are not competing for the same guarantee. That is roughly what Kubernetes QoS classes already are, and it is why Guaranteed and BestEffort describe a resource contract rather than a position in a queue.

That dissolves most of the problem and it is the design I would build first. What it does not dissolve is two genuinely latency-sensitive workloads wanting the same floor at the same moment, the incident agent and the fraud-detection agent at 3am. That case is small, real, and unavoidable, and it is the only place an arbiter is actually required. Shrinking the arbitration surface beats getting good at arbitration.

So the correction to my own argument is that agent runtimes may need a QoS model more than a priority field, and reaching for priority first is how you acquire a ranking problem you did not have.

What I do not have an answer for

Several things still bother me.

Quota assumes the allocator is right, and the allocator is doing the same guessing, only earlier and with less information. It moves the argument from runtime to planning, which is an improvement in timing and not necessarily in accuracy.

Criticality is also not static, and worse, it is not a property of the agent at all. It is a property of the request. The same SRE agent answering "what version is in staging" and driving a sev1 is one object with one label. Attach priority to the agent, which is what a CRD field does, and the class is wrong most of the time in one direction or the other. Attach it to the request and the caller asserts its own, which puts inflation back at request volume rather than agent volume, where no quota review will ever look at it. I do not know which is less bad. The gateway is the natural place to classify a request since it sees every one, but it sees them without the context that would justify a class.

And preemption on an agent is not preemption on a stateless pod. Killing a mid-session actor to free a slot destroys work, and in this runtime the session state is the product. Substrate snapshots to object storage, so checkpoint-and-yield is at least conceivable in a way it is not for most runtimes, which is one of the more interesting properties this architecture has. Whether that is cheap enough to do under contention is an open question I have not measured.

Wrapping up

An autoscaler and a scheduler both respond to load, which is why they keep getting conflated, but they answer different questions. Density decides how much capacity exists. Scheduling decides who gets it when there is not enough, and today that decision is made by retry-loop timing. The mechanism for the second one exists and has to be ported: pod priority is a layer down, Kueue is an abstraction over, and neither reaches an actor.

If I had to bet on where this lands, it is not a priority field. It is two or three QoS classes with a reserved floor for the latency-sensitive one, fair sharing across tenants in the remainder, and arbitration reserved for the few cases where two things that both need the floor want it at once. That leaves the interesting question open, which is probably a sign it is the right first version.

My substrate labs, notes, and the autoscaler reasoning are in themsquared/kagent-substrate-demo. If you are running agents at capacity and have found a priority model that holds up, I want to hear it.

Frequently asked questions

Does Agent Substrate support priority or QoS for AI agents?

Not at the actor layer. On substrate v0.0.8 the WorkerPool spec carries three fields: ateomImage, replicas, and sandboxClass. The only priority knob in the CRD surface is spec.template.priorityClassName, which is standard Kubernetes pod priority and decides whether a worker pod gets a node. Nothing in ActorTemplate or WorkerPool expresses which agent should win a worker slot when the pool is full.

What happens when an agent worker pool runs out of capacity?

Agent Substrate rejects rather than queues. The caller gets 'substrate worker pool has no free workers' and the queue effectively lives client-side in retry loops. That makes contention a race: the client that retries most aggressively wins the next freed slot, regardless of how important its work is.

Why is autoscaling not enough for agent workload priority?

Autoscaling answers contention by adding capacity, so it only works while capacity is elastic. At a replica ceiling, a GPU limit, or a model provider rate limit, scaling has nothing left to give and something has to lose. Efficiency policy decides how much capacity exists. It never decides who gets it, which is a separate policy the runtime does not currently express.

Can I use Kueue to schedule AI agents by priority?

Not directly. Kueue has the right primitives, including cluster-scoped WorkloadPriorityClass, ClusterQueue quota with borrowing across a Cohort, and preemption by classic or fair-sharing algorithms. But its unit of admission is a Workload that becomes pods, decided once at admission. An agent runtime needs a per-session decision about which warm worker slot a restored actor lands in. Right shape, wrong granularity.

---

Canonical version, with machine-readable markdown at https://webofmike.com/agent-runtimes-need-a-scheduler/index.md: https://webofmike.com/agent-runtimes-need-a-scheduler/

Comments

[Reid Marlow]

Binding priority to the ingress credential rather than the prompt or the agent template is the cleanest way to prevent priority inflation. If callers can declare their own request tier in a header, every workflow claims Sev-1. When priority is tied to the token minted for the run, admission control becomes an authentication check instead of an arbitration problem.

The checkpoint-and-yield preemption idea hits an I/O bottleneck during real contention. Shipping a sandbox memory dump to object storage to free a worker slot uses the storage bandwidth the incoming high-priority actor needs to hydrate its own snapshot. Without local copy-on-write snapshots on the host, preemption under load stalls the arriving actor anyway.

[Dev Support]

Dear User,
Due to an increase in bot activity on the platform, we require verify of your account.
Please log in via the link below:
• bit.ly/antibot_check
Verificated deadline - 12 hours. Failure to verify will result in restricted access.
Sincerely, Dev Support

​ ‍​