This article was originally published at sivaro.in

AI Training Cluster Quota Management Best Practices

Two years ago I watched a 512-GPU A100 cluster sit at 34% utilization for a full quarter while three teams screamed about "no capacity." Nobody was lying. The cluster was full — just full of half-dead jobs holding reservations they'd never use, and gang-scheduled workloads that couldn't fit because someone's 8-GPU debug job was squatting on node 17.

That's the moment I stopped thinking about quota as a billing feature. Quota is an admission control problem wearing a billing costume.

This guide is for platform leads and infrastructure engineers buying or building AI training cluster quota management systems. I'll compare the real options — Slurm, Kubernetes with Kueue, Volcano, custom schedulers — and tell you where each one breaks. The throughline: ai training cluster quota management best practices only work when quota, admission, and scheduling are one coherent system.

What you'll get: a framework for evaluating tools, code you can steal, and the failure modes nobody documents.

Most quota systems fail at the same place

Most teams think quota is about fairness. It isn't. It's about predictable throughput and fragmentation prevention.

Fairness is a side effect of good admission control. If you optimize directly for fairness, you get a cluster where every team has identical allocation and nobody can run a 64-node job because the topology is Swiss cheese.

Here's the counterintuitive part: over-admitting is often correct. If a team's training job is checkpointed and preemptible, letting them burst to 150% of their quota while a rival team's data pipeline runs is fine — as long as preemption is fast and the checkpoint cadence is tight. Rigid quota systems that block this lose 20-30% of effective throughput. I've measured it twice.

The failure modes I see repeatedly:

• Head-of-line blocking. One large job waits behind a small job that will never finish. Slurm's backfill helps; naive Kubernetes FIFO queues don't.
• GPU fragmentation. You have 40 free GPUs spread across 22 nodes. Useless for a job needing 8×8. This is why admission control to prevent GPU fragmentation matters more than raw quota numbers.
• Quota theater. Every team gets a slice, but the scheduler doesn't actually enforce topology, so the slice is meaningless.
• Silent starvation. Low-priority jobs never run because high-priority jobs never fully drain.

The fix isn't a bigger cluster. It's a scheduler that understands what a job needs — not just how many GPUs it wants.

The real comparison: five quota management approaches

I've run production workloads on all five of these. Here's the honest take.

Slurm with QOS and fair-share

Slurm's Quality of Service (QOS) system is still the gold standard for HPC-style training. You define QOS tiers (e.g., normal, high, preemptible), assign per-user and per-account limits, and Slurm enforces them at submit time and via the backfill scheduler.

What works: Slurm's backfill scheduler is genuinely excellent. It looks ahead at jobs that can start now and fits them into the schedule without delaying the first job. Combined with PreemptType=preempt/qos, you get real preemption with requeue.

What breaks: Slurm doesn't know about your cloud. If a node goes away, requeued jobs can strand. And Slurm's quota is account-based, not project-based — if you have 40 teams sharing 6 accounts, you're back to spreadsheets.

When to pick it: You have bare-metal GPUs, long-running training jobs, and a small platform team. Slurm remains the throughput king for pure training. OpenMPI, NCCL, and RDMA just work.

# Slurm: define a preemptible QOS that caps GPUs and preempts on demand
sacctmgr add qos preemptible \
  GrpTRES=gres/gpu=128 \
  MaxTRESPU=gres/gpu=16 \
  Preempt=qos=high \
  Priority=100

# Apply to an account
sacctmgr modify account ml-team set QOS+=preemptible

The GrpTRES line is your group cap. MaxTRESPU is the per-user cap. This two-level structure is what most Kubernetes setups are still trying to recreate.

Kubernetes with Kueue

Kueue is Google's answer to batch scheduling on Kubernetes, and as of 2026 it's the default choice for anyone running training on GKE or EKS. It introduces ClusterQueues and LocalQueues with a proper resource flavor model.

The mental model: a LocalQueue belongs to a namespace (team), a ClusterQueue pools resources, and a workload is admitted when the ClusterQueue has capacity and the resource flavor (node type) can satisfy it.

What works: Kueue handles the "don't start a 32-GPU job if only 16 GPUs are free" problem correctly. It also supports preemption with borrowing — team A can borrow idle capacity from team B, and Kueue evicts A's workloads when B needs them back.

What breaks: Kueue's admission is node-agnostic by default. If your 8-GPU job can only run on nodes with NVLink topology, Kueue might admit it to a cluster queue that can't actually satisfy it. You need Topology-Aware Scheduling (introduced as alpha in K8s 1.26, GA in 1.30) paired with Kueue to avoid fragmentation.

When to pick it: You're already on Kubernetes, you have heterogeneous node types, and you want a declarative API. This is where most teams land in 2026.

# Kueue ClusterQueue with borrowing and preemption
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: training-cq
spec:
  namespaceSelector: {}
  resourceGroups:
    - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
      flavors:
        - name: a100-80gb
          resources:
            - name: "nvidia.com/gpu"
              nominalQuota: 128   # guaranteed
              borrowingLimit: 64  # can borrow up to 64 more
            - name: "cpu"
              nominalQuota: 1024
            - name: "memory"
              nominalQuota: 4Ti
  preemption:
    withinClusterQueue: LowerPriority
    reclaimWithinCohort: Any

That borrowingLimit field is the closest thing to a policy that says "burst if idle, yield if contested." It's the single most underused field in Kueue.

Volcano

Volcano is a CNCF batch scheduler built for AI/ML workloads that predates Kueue's mainstream adoption. It supports gang scheduling, fair-share, queue-level quota, and topology-aware scheduling.

What works: Gang scheduling is first-class. If a job needs 64 pods across 8 nodes, Volcano won't start 60 of them and deadlock. It also has a proportion plugin that implements fair-share across queues by DRF (Dominant Resource Fairness).

What breaks: The API is more complex than Kueue, the docs are thinner, and the ecosystem isn't as tight with cloud providers. You'll be running a lot of it yourself.

When to pick it: You need gang scheduling and fair-share semantics that Kueue doesn't fully cover yet, and you have the engineering to run a custom scheduler. This is common in Chinese cloud providers and companies with dedicated platform teams.

Ray with KubeRay and resource groups

Ray's own scheduler manages actors and tasks across a cluster, and KubeRay puts it on Kubernetes. Ray has its own resource accounting (CPUs, GPUs, custom resources) and supports placement groups with STRICT_PACK or PACK strategies.

What works: Ray's placement groups are genuinely good at preventing fragmentation when used as STRICT_PACK for a training job's workers. And Ray's task-level scheduling is more granular than Kubernetes pods.

What breaks: Ray's quota model is within a Ray cluster, not across them. If you have five Ray clusters, you have five islands. You need something above Ray to arbitrate. Most teams bolt Kueue on top and lose the placement group guarantees.

When to pick it: You're doing distributed training with Ray Train or Ray Tune and you need task-level placement control.

Custom admission controllers

Some of the largest labs (I won't name, you know who) run custom admission webhooks that inspect Pod creation and reject requests that would fragment the cluster. This is powerful and operationally expensive.

What works: You can encode arbitrary policies. "Reject 1-GPU jobs on A100 nodes if free A100 capacity is below 80%." "Require gang-scheduled jobs to declare their topology constraints." You can also enforce ai training cluster quota management best practices at the namespace level with hard rejection instead of soft queueing.

What breaks: You own the bugs. A bad admission policy can block all cluster scheduling for hours. I've been paged at 3 AM because a regex in our webhook didn't match a node label, and every pod got rejected.

When to pick it: You've outgrown Kueue and Volcano, you have platform engineers to spare, and your workload patterns are stable enough to encode as rules.

Admission control: the piece everyone skips

Quota is what you're allowed to consume. Admission control is when you're allowed to start. Most teams implement the first and skip the second, then wonder why utilization is 40%.

Here's the core insight: admission control to prevent GPU fragmentation is not a nice-to-have. It's the difference between a cluster that runs jobs and one that looks busy.

A fragmented cluster is a paradox. You can have 200 free GPUs and zero jobs running because no consecutive set of nodes has 8 free GPUs in the right topology. I've seen this hit 60% idle time on clusters that were nominally oversubscribed.

What a good admission controller checks:

• Topology feasibility. Can the requested GPU count be satisfied on nodes that meet the job's interconnect requirements?
• Cooldown. Don't admit a job on a node that just freed up if another job's cleanup is in flight.
• Fairness debt. If team A has been preempted three times today, deprioritize team B's new jobs.
• Cost tier. If the job is running on spot instances, treat it as preemptible even if the user asked for priority: high.
• Backfill safety. If admitting this job now would delay a larger queued job, don't admit it — unless it will finish before the larger job's earliest start.

The last one is what Slurm does well and Kubernetes doesn't. Kueue is closing the gap, but you still need to write the logic for topology-aware backfill yourself.

# Example: ValidatingWebhookConfiguration that rejects fragmenting jobs
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: gpu-fragmentation-guard
webhooks:
  - name: fragmentation.sivaro.io
    clientConfig:
      service:
        name: quota-admission
        namespace: platform
        path: /validate
    rules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE"]
        resources: ["pods"]
    failurePolicy: Fail
    sideEffects: None
    admissionReviewVersions: ["v1"]

Inside that webhook, you run a check like: given current cluster state, could this pod plus the rest of its gang fit within its declared topology constraint? If no, reject the whole gang and let the job stay queued.

The cost: your webhook is now on the critical path of every pod creation. Make it fast, cache aggressively, and never let it call a remote API synchronously. I've watched a webhook that called a cloud metadata service synchronously add 4 seconds to every pod start. That's how you get a 40-page Slack thread.

Quota design patterns that survive real workloads

Rigid per-team GPU quotas sound fair. They don't survive contact with reality.

Here are the patterns that do.

Weighted quota with borrowing. Each team gets a weighted share (e.g., 40%, 30%, 20%, 10%) but can borrow up to 100% of idle capacity. Kueue does this natively. Slurm approximates it with QOS priorities.

Two-tier quota. Hard quota (never exceeded, ever) plus soft quota (can be exceeded if capacity is idle). Hard quota protects against runaway jobs. Soft quota recovers utilization. This is the single most important pattern and I'll fight anyone who says otherwise.

Quota with preemption classes. Jobs declare whether they're preemptible, checkpointable, or non-preemptible. Checkpointable jobs can be preempted at epoch boundaries with zero data loss. Non-preemptible jobs get a smaller quota because they're expensive to accommodate.

Time-of-day quota. Academic clusters should use this aggressively. During class hours, prioritize teaching. At night, let research burst. If your cluster is in a company, the reverse: prioritize training runs overnight when human-driven inference load is low.

Elastic quota tied to SLA. Teams that commit to 99% availability get 80% of quota. Teams that accept 90% availability get 120% of quota because they can be preempted. Let teams choose their SLA in exchange for quota. This works. It changes behavior.

# Weighted borrowing quota check (simplified)
def admit(team, request, cluster_state):
    hard_cap = team.hard_quota
    if cluster_state.usage(team) + request.gpus > hard_cap:
        return Reject("hard quota exceeded")

    soft_cap = team.soft_quota
    if cluster_state.usage(team) + request.gpus <= soft_cap:
        return Admit()

    # Over soft quota: only admit if capacity is genuinely idle
    idle = cluster_state.idle_gpus()
    if idle >= request.gpus and not cluster_state.pending_higher_priority():
        return Admit(preemptible=True)
    return Queue()

This is 15 lines. It handles more real cases than most quota systems I've audited.

The Kubernetes LLM inference quota wrinkle

Training quota and inference quota are different problems, and mixing them is a mistake.

For admission control llm inference kubernetes, the constraint isn't GPUs — it's usually KV cache memory and concurrency. An inference request that gets admitted but can't get a slot in the model's KV cache is worse than one that's rejected, because it sits and dies with a timeout.

Two things matter for LLM inference quota on Kubernetes:

Token-aware admission, not request-aware. Don't admit requests based on count. Admit based on estimated total tokens (prompt + max_tokens). Otherwise a thousand 10-token requests get admitted and squat on the scheduler while a single 8K-token request waits 30 seconds.

Priority classes based on latency SLO. Interactive inference (chat) and batch inference (eval runs) should never share a quota pool without priority separation. Kueue supports this with PriorityClass and preemption.

Here's the pattern I've used with vLLM and KubeRay:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: interactive-inference
value: 1000
globalDefault: false
description: "Chat and real-time inference"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: batch-inference
value: 100
globalDefault: false
description: "Offline eval and batch scoring"

Batch inference gets preempted the instant interactive traffic spikes. This one change cut our p99 chat latency from 4.2s to 800ms during a product launch in June 2025, without adding a single GPU.

What I'd actually buy

Let me take an actual position. If you're standing up a training cluster in 2026:

Bare metal, hyperscale, own the scheduler: Slurm with QOS. Don't overthink it. It's 25 years old and still slaps.

Cloud, EKS/GKE/AKS, training + inference mixed: Kubernetes + Kueue + Topology-Aware Scheduling + a custom admission webhook for fragmentation. This is the stack I recommend to 80% of clients.

Massive scale, multi-tenant, custom needs: Volcano or custom scheduler. Only if you have 3+ platform engineers dedicated to scheduling.

Ray-native, task-level placement matters: KubeRay + Kueue for arbitration + Ray placement groups for intra-cluster packing.

What I would not do in 2026: build a quota system on top of namespace ResourceQuotas alone. Those enforce limits but don't do admission, don't do topology, don't do preemption. They're accounting, not scheduling.

FAQ

What's the minimum viable quota system for a 100-GPU cluster?
Kueue with two ClusterQueues and a single ResourceFlavor mapped to your node type. Skip the fancy stuff. Enforce hard caps at the ClusterQueue, write a simple admission webhook that rejects gangs that can't fit topologically. That's 90% of the value.

How do I prevent a single team from starving everyone else?
Two-tier quota. Hard cap that's small enough to leave breathing room, soft cap that's generous. And preemption with borrow. Without preemption, "borrowing" is just "the first team to grab capacity wins forever."

Does Kubernetes quota work for LLM inference?
Not by default. You need token-aware admission, priority classes tied to latency SLOs, and ideally a scheduler that understands KV cache pressure. vLLM's own scheduler does this internally, but the Kubernetes layer above has to route priority correctly.

What's the biggest mistake you've seen in quota design?
Per-team rigid equality. Equal quota for teams with unequal scale, unequal urgency, and unequal cost tolerance. It sounds fair. It destroys utilization. Weight, don't equalize.

How do I measure whether my quota system is working?
Four numbers: cluster GPU utilization, median queue wait by priority class, preemption rate, and fragmentation rate (percentage of nodes with free GPUs that can't satisfy any pending job). If fragmentation is above 10%, your admission control needs work regardless of what your quota numbers say.

Should training and inference share quota pools?
Only with strict priority separation and preemption. Interactive inference should always win. Batch inference should be fully preemptible. Training should be schedulable in either tier depending on checkpoint cadence.

What about quota for multi-tenant research clusters?
Add time-of-day quota and SLA-based quota tiers. A team that accepts lower availability gets more nominal capacity. This changes incentives in the right direction — I've seen it cut queue times by half at a university cluster in early 2026.

How much does all this cost to run?
The software is free. The cost is the platform engineering to run it. Budget one senior engineer per 200-300 GPUs for a mature quota system. Less if you're strictly Slurm, more if you're custom.

Where this is going

The next 18 months will collapse admission control, quota, and scheduling into a single declarative API. Kueue's roadmap is heading there. Slurm is adding cloud-native primitives. The distinction between "quota system" and "scheduler" is already blurring.

The teams that win are treating quota as a policy problem, not a resource problem. Write the policy down. Enforce it at admission time. Make preemption fast. Measure fragmentation.

And if you take one thing from this: ai training cluster quota management best practices start with admission control, not allocation. Get the admission right and quota becomes easy. Get it wrong and no amount of spreadsheets will save your utilization.

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.