DEV Community

Cover image for Inside Kueue: How Kubernetes Decides What Runs Next
Ekamveer Walia
Ekamveer Walia

Posted on

Inside Kueue: How Kubernetes Decides What Runs Next

#kubernetes #ai #opensource #programming

Your Kubernetes cluster works great for microservices—until someone deploys a batch job
that eats every GPU and CPU on the machine. Suddenly critical workloads starve, distributed
jobs deadlock halfway through, and the scheduler has no idea what should run first.
Kueue fixes this. It's a smart traffic controller that queues batch workloads fairly,
prevents deadlock, and guarantees resources before a job ever touches your cluster.
In this blog, we'll see why Kubernetes' default scheduler breaks on batch jobs, and then
build Kueue from scratch with working demos you can run today.

How Scheduling actually works on Kubernetes

Kubernetes default scheduler works like this: 

Pod lands → Scheduler checks if it fits on any node.
Resources available? → Yes → Pod gets scheduled
Resources available? → No → Pod sits in ‘Pending’ state until resources free up.
Enter fullscreen mode Exit fullscreen mode

Meanwhile, other important jobs also queue up and fight for the same resources.

The key issues which one can face using default K8s scheduler:

  1. Job fairness across teams
  2. Deadlock issue (Some jobs require all pods to sync for run)
  3. Queue management (who should go first?)
  4. Resource quotas (how much can each team use?)
  5. Preemption (can I pause a low-priority job to run a critical one?)

The Relatable Problem

Let us suppose that your team is running Kubernetes, and everything is working well. Microservices deploy smoothly. Then one day, someone deploys a Batch Job—maybe it’s a machine learning model training job or a big data processing pipeline.

The job starts and immediately grabs every available GPU, CPU, and memory on the cluster. Meanwhile, other important workloads are left waiting for resources that won’t become available anytime soon. In some cases, this can even create a deadlock: workloads are waiting on resources held by other workloads, while the cluster has no effective way to decide what should run first.

Sound familiar? This is the issue that Kueue is built to solve.

Now let’s first understand what Batch Jobs are and the concept of Gang Scheduling, also how they create a DEADLOCK Issue through an example.

Let's start with the basics, because not everyone has run batch jobs before.

Traditional Microservices vs Batch Workloads

Microservices (what your cluster probably handles now):
  1. Run 24/7 (or close to it)
  2. Need modest, predictable resources
  3. React to incoming requests Example: A web API serving user requests
Batch Workloads (what breaks your cluster):
  1. Stateful (distributed state across pods)
  2. All or nothing (5 of 8 pods running = job hangs)
  3. Long running (hours, days, weeks)
  4. Coordinated (all pods must sync regularly)
  5. Resource intensive (GPUs, TPUs, high CPU)
  6. Run for a fixed time, then stop
  7. Don't react to requests; just 'process all this data' Example: Training an ML model on 1TB of data, processing tonight's logs, running backups
Real examples of Batch Workloads
  1. Machine Learning Training - Needs: 8 GPUs, 256GB RAM for 4 hours
  2. Then: Stops completely
  3. Data Pipeline - Needs: 64 CPUs, 512GB RAM to process logs
  4. Then: Stops, waits for tomorrow
  5. Big Data Job (Spark, Hadoop) - Needs: 100 CPUs, 500GB RAM in one shot
    • Then: Finishes

Why Jobs Need to Run Simultaneously (The Gang Scheduling Story)

Imagine you're running a distributed machine learning job. Think of it like a team project where 4 people need to work together:

Job = 4 workers (4 separate pods)
Team Member 1: "I'm ready!"
Team Member 2: "I'm ready!"
Team Member 3: "I'm ready!"
Team Member 4: "Still waiting for a computer..."
Enter fullscreen mode Exit fullscreen mode

What happens?

Members 1-3 sit around wasting time.
The job doesn't progress.
Resources are used but no work gets done.
This is the gang scheduling problem.

Why ALL Pods Must Start Together

Distributed jobs have dependencies between their pods:

Pod 1 needs to talk to Pod 2
Pod 2 needs to receive from Pod 3
Pod 3 needs data from Pod 4

If Pod 4 is stuck in "Pending..."
→ Pod 3 can't send data
→ Pod 2 can't receive from Pod 3
→ Pod 1 is blocked
→ All 4 pods run but do NOTHING
Without gang scheduling:

Scheduler tries to place 4 pods
Puts Pod 1 ✅
Puts Pod 2 ✅
Puts Pod 3 ✅
Can't fit Pod 4 ❌
Enter fullscreen mode Exit fullscreen mode

Result: 3 pods running, 1 waiting
Status: 3 pods doing nothing (waiting for Pod 4)
Wasted resources: 75% of the job's allocation is wasted
With gang scheduling (Kueue):
Job says: "I need 4 pods or nothing"
Kueue checks: Can I fit all 4?
→ Yes? Admit all 4, they start together ✅✅✅✅
→ No? Queue all 4, none start yet ⏳⏳⏳⏳

Result: Either 100% of the job runs, or 0%
Wasted resources: None (no idle pods)
This is Gang Scheduling, and it's why distributed jobs absolutely need it.

Why Batch Jobs Are Hard on Kubernetes, understanding Deadlock Scenario

Kubernetes scheduler doesn't understand gang scheduling:
It doesn't know: "These 8 pods are a team that needs resources together"
It treats each pod independently
So it partially schedules the job
Partially scheduled distributed job = DEADLOCK

This is where Kueue comes in.

Meet Kueue: Your Cluster's Traffic Controller

Kueue is a job queuing and quota management system for Kubernetes Batch Workloads.
It’s a smart traffic controller that:

  1. Collects all jobs in organized queues
  2. Checks available resources before admitting anything
  3. Allocates fairly based on priority and quotas
  4. Admits jobs atomically (all or nothing for distributed jobs)

With Kueue : 

User Job → KUEUE (Smart Gatekeeper) → Kubernetes Scheduler → Pods created → No deadlock
Enter fullscreen mode Exit fullscreen mode

Kueue’s Core principle is to only admit a job to the cluster when we're 100% sure we have enough resources for ALL its pods.

Why You Actually Need Kueue

  1. Fairness: Teams don't starve each other
  2. Gang Scheduling: Distributed jobs run all together or queue together
  3. Priorities: Critical jobs can be prioritized over experimental ones
  4. Visibility: You see exactly why a job is queued and when it'll run
  5. Resource Quotas: Each team gets a guaranteed slice of the cluster

Understanding Objects in Kueue

  1. Workload What it is: A wrapper around your Kubernetes Job that Kueue understands.

Plain English: When you submit a Job to Kueue, Kueue wraps it in a 'Workload' object that tracks its status in the queue.

  1. LocalQueue What it is: A queue for jobs in a specific namespace.

Plain English: Think of it as a 'job submission desk' in your namespace. When your team submits a job, it goes into this queue first.

  1. ClusterQueue What it is: A higher level queue that holds the actual resource budget.

Plain English: This is where the real resource management happens. It's the 'headquarters' that decides "OK, we have 100 CPUs available. Which job gets them?Jobs from all namespaces compete here based on priority and fairness."

  1. ResourceFlavor What it is: A label for a type of resource in your cluster.

Plain English: It's like saying "we have two types of computers: expensive GPUs and cheap CPUs. Let me label them differently."

  1. ResourceQuota What it is: How much of a resource a ClusterQueue can use.

Plain English: "This queue can use up to 100 CPUs, 500GB RAM, and 16 GPUs. Not more."

  1. Admission What it is: When Kueue says "yes, your job can now run."

Plain English: The job has been waiting in the queue. Kueue checked the available resources and decided "OK, go ahead and run."

How these objects Work Together 

Job → Workload → LocalQueue → ClusterQueue → Resources Available? → ADMITTED → Scheduler → Pods → Running → Complete → Resources Released → Next Job
Enter fullscreen mode Exit fullscreen mode

Installation of Kueue

Kueue is just a Kubernetes controller.

Step 1: Install Kueue from Official Manifests

kubectl apply -f kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.19.2/manifests.yaml
Enter fullscreen mode Exit fullscreen mode

That's it. Kueue controller is now running.

Step 2: Verify Installation

kubectl get pods -n kueue-system
Enter fullscreen mode Exit fullscreen mode

You should see:

NAME                                        READY   STATUS    RESTARTS   AGE
kueue-controller-manager-69866f4b8d-4vf5x   1/1     Running   0          65s
Enter fullscreen mode Exit fullscreen mode

Step 3: Verify Custom Resources are Installed

kubectl get crds | grep kueue
Enter fullscreen mode Exit fullscreen mode

You should see:

admissionchecks.kueue.x-k8s.io               2026-08-25T12:08:48Z
clusterqueues.kueue.x-k8s.io                 2026-08-25T12:08:48Z
cohorts.kueue.x-k8s.io                       2026-08-25T12:08:48Z
localqueues.kueue.x-k8s.io                   2026-08-25T12:08:48Z
multikueueclusters.kueue.x-k8s.io            2026-08-25T12:08:48Z
multikueueconfigs.kueue.x-k8s.io             2026-08-25T12:08:48Z
provisioningrequestconfigs.kueue.x-k8s.io    2026-08-25T12:08:48Z
resourceflavors.kueue.x-k8s.io               2026-08-25T12:08:49Z
topologies.kueue.x-k8s.io                    2026-08-25T12:08:49Z
workloadpriorityclasses.kueue.x-k8s.io       2026-08-25T12:08:49Z
workloads.kueue.x-k8s.io                     2026-08-25T12:08:49Z
Enter fullscreen mode Exit fullscreen mode

Done! Kueue is ready.

Demo: How Scheduling actually works in Kueue

Let's see Kueue in action with a simple scenario.

Setup: Create the Namespace 
kubectl create namespace kueue-demo
Enter fullscreen mode Exit fullscreen mode

Step 1: Create a ResourceFlavor

This tells Kueue about the resources available in your cluster:

apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: default
spec: {}
Enter fullscreen mode Exit fullscreen mode

Save as resource-flavor.yaml and apply:

kubectl apply -f resource-flavor.yaml
Enter fullscreen mode Exit fullscreen mode

Step 2: Create a ClusterQueue

This is where we set resource limits:

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: demo-queue
spec:
  namespaceSelector: {}
  resourceGroups:
    - coveredResources:
        - cpu
        - memory
      flavors:
        - name: default
          resources:
            - name: cpu
              nominalQuota: "10"        # Only 10 CPUs available
            - name: memory
              nominalQuota: "20Gi"      # Only 20GB RAM available
Enter fullscreen mode Exit fullscreen mode

Save as cluster-queue.yaml and apply:

kubectl apply -f cluster-queue.yaml
Enter fullscreen mode Exit fullscreen mode

Step 3: Create a LocalQueue

This connects the namespace to the ClusterQueue:

apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  name: default
  namespace: kueue-demo
spec:
  clusterQueue: demo-queue
Enter fullscreen mode Exit fullscreen mode

Save as local-queue.yaml and apply:

kubectl apply -f local-queue.yaml
Enter fullscreen mode Exit fullscreen mode

Step 4: Create Job A (The Resource Hog)

This job will use 8 out of 10 CPUs:

apiVersion: batch/v1
kind: Job
metadata:
  name: job-a-big
  namespace: kueue-demo
spec:
  completions: 1
  parallelism: 1
  template:
    metadata:
      labels:
        kueue.x-k8s.io/queue-name: default
    spec:
      restartPolicy: Never
      containers:
      - name: container
        image: ubuntu:22.04
        command: ["sleep", "300"]
        resources:
          requests:
            cpu: "8"
            memory: "12Gi"
          limits:
            cpu: "8"
            memory: "12Gi"
Enter fullscreen mode Exit fullscreen mode

Save as job-a.yaml and apply:

kubectl apply -f job-a.yaml
Enter fullscreen mode Exit fullscreen mode

Step 5: Watch What Happens

kubectl get workloads -n kueue-demo -w
Enter fullscreen mode Exit fullscreen mode

You should see:

NAME                  QUEUE     RESERVED IN   ADMITTED   FINISHED   AGE
job-job-a-big-cb1a1   default   demo-queue    True                  18s
Enter fullscreen mode Exit fullscreen mode

also check local queue, if the job is admitted or not

kubectl get localqueue -n kueue-demo
Enter fullscreen mode Exit fullscreen mode

should show

NAME      CLUSTERQUEUE   PENDING WORKLOADS   ADMITTED WORKLOADS
default   demo-queue     0                   1
Enter fullscreen mode Exit fullscreen mode

Job A is ADMITTED because 8 CPUs fit within the 10 CPUs available.

Step 6: Create Job B (The Starved Job)

Now create another job that also needs resources:

apiVersion: batch/v1
kind: Job
metadata:
  name: job-b-small
  namespace: kueue-demo
spec:
  completions: 1
  parallelism: 1
  template:
    metadata:
      labels:
        kueue.x-k8s.io/queue-name: default
    spec:
      restartPolicy: Never
      containers:
      - name: container
        image: ubuntu:22.04
        command: ["sleep", "300"]
        resources:
          requests:
            cpu: "5"
            memory: "8Gi"
          limits:
            cpu: "5"
            memory: "8Gi"
Enter fullscreen mode Exit fullscreen mode

Save as job-b.yaml and apply:

kubectl apply -f job-b.yaml -n kueue-demo
Enter fullscreen mode Exit fullscreen mode

Step 7: Watch the Queue

kubectl get workloads -n kueue-demo
Enter fullscreen mode Exit fullscreen mode

Now you see:

NAME                    QUEUE     RESERVED IN   ADMITTED   FINISHED   AGE
job-job-a-big-cb1a1     default   demo-queue    True                  4m40s
job-job-b-small-c8c54   default                                       47s
Enter fullscreen mode Exit fullscreen mode

Job A: ADMITTED (using 8 of 10 CPUs) Job B: NOT ADMITTED (only 2 CPUs available, but needs 5)
Job B is stuck in the queue! It's waiting for resources.

Step 8: See Why Job B Is Waiting

kubectl describe workload job-job-b-small-c8c54 -n kueue-demo
Enter fullscreen mode Exit fullscreen mode

Output:

Status:
  Conditions:
    Last Transition Time:  2026-08-25T12:30:48Z
    Message:               couldn't assign flavors to pod set main: insufficient unused quota for cpu in flavor default, 3 more needed
    Observed Generation:   1
    Reason:                Pending
    Status:                False
    Type:                  QuotaReserved
    Last Transition Time:  2026-08-25T12:30:48Z
    Message:               Not all pods are ready or succeeded
    Observed Generation:   1
    Reason:                WaitForStart
    Status:                False
    Type:                  PodsReady
  Resource Requests:
    Name:  main
    Resources:
      Cpu:     5
      Memory:  8Gi
Events:
  Type     Reason   Age    From             Message
  ----     ------   ----   ----             -------
  Warning  Pending  2m10s  kueue-admission  couldn't assign flavors to pod set main: insufficient unused quota for cpu in flavor default, 3 more needed
Enter fullscreen mode Exit fullscreen mode

Step 9: Free Up Resources (Delete Job A)

kubectl delete job job-a-big -n kueue-demo
Enter fullscreen mode Exit fullscreen mode

Now immediately check the workloads:

kubectl get workloads -n kueue-demo
Enter fullscreen mode Exit fullscreen mode

output

NAME                    QUEUE     RESERVED IN   ADMITTED   FINISHED   AGE
job-job-b-small-c8c54   default   demo-queue    True                  5m20s
Enter fullscreen mode Exit fullscreen mode

Magic! Job B is now ADMITTED. Kueue automatically moved it up the queue and gave it the freed resources.
What Just Happened
Job A grabbed the big resources
Job B arrived but couldn't fit
Job A finished, releasing resources
Kueue saw the freed resources
Kueue admitted Job B
Job B ran
This is fair resource management. This is what Kueue does.

The Real Deadlock Demo (Gang Scheduling)

The Real Problem (Without Kueue)

ClusterQueue has: 12 CPUs total

Job A arrives:
- Requests: 8 CPUs 
- Gets admitted, uses 8 CPUs
- Remaining: 4 CPUs free

Job B arrives (GANG JOB - needs ALL 6 CPUs at once):
- Requests: 6 CPUs SIMULTANEOUSLY 
- Only 4 CPUs available (less than 6)
- Kubernetes admits it anyway  (WRONG!)

Job B Pod 1 starts with 4 CPUs (partial):
- Job B NEEDS all 6 CPUs to coordinate with Pod 2
- But only 4 CPUs available
- Pod 2 has nowhere to go (0 CPUs left)
- Pod 1 is waiting for Pod 2
- Pod 2 is waiting for CPUs

Result: 
- Job B is half-running with only 4 CPUs 
- Job B Pod 2 is Pending, waiting for 3 CPUs 
- Job A is holding 8 CPUs 
- All 12 CPUs are consumed, NOTHING can progress = DEADLOCK
Enter fullscreen mode Exit fullscreen mode

Why is this Deadlock:

Job B CANNOT WORK with only 4 CPUs. It needs 6.
- If it's a distributed ML job with 2 workers
- Worker 1 needs to sync with Worker 2
- Worker 1 starts with 4 CPUs (wasting them)
- Worker 2 can't start (no CPUs)
- Worker 1 sits idle waiting for Worker 2 = DEADLOCK

Meanwhile:
- Job A holds 8 CPUs for 600 seconds
- Job B wastes 4 CPUs for 600 seconds
- 0 CPUs available for anything else
- System is stuck
Enter fullscreen mode Exit fullscreen mode

Below is the Yaml file which can cause the Deadlock Issue.

apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: problem-queue
spec:
  namespaceSelector: {}
  resourceGroups:
    - coveredResources:
        - cpu
        - memory
      flavors:
        - name: default
          resources:
            - name: cpu
              nominalQuota: "12"      # ← Only 12 CPUs total
            - name: memory
              nominalQuota: "24Gi"
---

#Job A (takes 6 CPUs)
apiVersion: batch/v1
kind: Job
metadata:
  name: job-a-takes-half
  namespace: kueue-demo
spec:
  completions: 1
  parallelism: 1
  template:
    metadata:
      labels:
        kueue.x-k8s.io/queue-name: default
    spec:
      restartPolicy: Never
      containers:
      - name: container
        image: ubuntu:22.04
        command: ["sleep", "600"]    # Runs for 10 minutes
        resources:
          requests:
            cpu: "6"
            memory: "8Gi"
          limits:
            cpu: "6"
            memory: "8Gi"
---

#Job B (Also needs 6 CPUs, but scheduler might give it partial)
apiVersion: batch/v1
kind: Job
metadata:
  name: job-b-needs-6-gets-3
  namespace: kueue-demo
spec:
  completions: 1
  parallelism: 2              # ← Needs 2 pods, 3 CPUs each = 6 total
  template:
    metadata:
      labels:
        kueue.x-k8s.io/queue-name: default
    spec:
      restartPolicy: Never
      containers:
      - name: container
        image: ubuntu:22.04
        command: ["sh", "-c", "echo 'I need my partner pod!'; sleep 600"]
        resources:
          requests:
            cpu: "3"          # Each pod needs 3 CPUs
            memory: "4Gi"
          limits:
            cpu: "3"
            memory: "4Gi"
Enter fullscreen mode Exit fullscreen mode

Solution: To create a Better ClusterQueue with gang scheduling awareness

apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: smart-queue
spec:
  namespaceSelector: {}
  resourceGroups:
    - coveredResources:
        - cpu
        - memory
      flavors:
        - name: default
          resources:
            - name: cpu
              nominalQuota: "12"
            - name: memory
              nominalQuota: "24Gi"
Enter fullscreen mode Exit fullscreen mode

Before Kueue:

apiVersion: batch/v1
kind: Job
metadata:
  name: job-b-needs-6-gets-4
  namespace: kueue-demo
  labels:
    kueue.x-k8s.io/queue-name: default
spec:
  completions: 2
  parallelism: 2
  template:
    metadata:
      labels:
        kueue.x-k8s.io/queue-name: default
    spec:
      restartPolicy: Never
      containers:
      - name: worker
        image: ubuntu:22.04
        resources:
          requests:
            cpu: "3"
            memory: "4Gi"
          limits:
            cpu: "3"
            memory: "4Gi"
Enter fullscreen mode Exit fullscreen mode

After Kueue:

apiVersion: batch/v1
kind: Job
metadata:
  name: job-b-fixed
  namespace: kueue-demo
  labels:
    kueue.x-k8s.io/queue-name: default
spec:
  completions: 2
  parallelism: 2
  suspend: true  #  THE FIX - Kueue will wait for ALL resources
  template:
    metadata:
      labels:
        kueue.x-k8s.io/queue-name: default
    spec:
      restartPolicy: Never
      containers:
      - name: worker
        image: ubuntu:22.04
        resources:
          requests:
            cpu: "3"
            memory: "4Gi"
          limits:
            cpu: "3"
            memory: "4Gi"
Enter fullscreen mode Exit fullscreen mode

Wrapping up

Kubernetes is great for microservices, but batch jobs need more. Kueue fills that gap it adds fair queuing, prevents deadlocks with gang scheduling, and gives you visibility into what's waiting and why.

The best part? One line fixes it: suspend: true. That's it. No complex configs, no custom schedulers just intelligent resource management that actually works.

Resources

Official Kueue Docs: https://kueue.sigs.k8s.io/
GitHub: https://github.com/kubernetes-sigs/kueue
Install: kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.19.2/manifests.yaml

Top comments (0)