DEV Community

Cover image for Part-98: ⚡Implementing Job Parallelism in Google Kubernetes Engine (GKE)
Latchu@DevOps
Latchu@DevOps

Posted on

Part-98: ⚡Implementing Job Parallelism in Google Kubernetes Engine (GKE)

#kubernetes #googlecloud #devops #cloud

Kubernetes Jobs are perfect for workloads that run to completion, like batch processing or data analysis.

Earlier, we saw how to use completions to specify how many Pods must finish successfully.

But what if you want multiple Pods to run at the same time instead of sequentially? 🤔

That’s where parallelism comes in.

🔹 Step 01: Introduction

  • By default → Job Pods do not run in parallel.
  • The parallelism field specifies the maximum number of Pods that can run concurrently.

👉 Example:

completions: 4 → Run 4 Pods in total.

parallelism: 2 → Run 2 Pods at a time.

🔹 Step 02: job4.yaml

Here’s the Job manifest with both completions and parallelism defined:

apiVersion: batch/v1
kind: Job
metadata:
  # Unique key of the Job instance
  name: job4
spec:
  template:
    metadata:
      name: job4
    spec:
      containers:
      - name: job4
        image: alpine
        command: ['sh', '-c', 'echo Kubernetes Jobs Demo - Job Completions and Parallelism Test ; sleep 20']
      # Do not restart containers after they exit
      restartPolicy: Never
  # Number of retries before marking as failed
  backoffLimit: 4
  # Total Pods that must complete
  completions: 4
  # Max Pods running at the same time
  parallelism: 2
Enter fullscreen mode Exit fullscreen mode

🔹 Step 03: Deploy and Verify 

# Deploy the Job
kubectl apply -f job4.yaml
# OR
kubectl create -f job4.yaml
Enter fullscreen mode Exit fullscreen mode

Check Jobs and Pods

kubectl get jobs
kubectl describe job job4
kubectl get pods
Enter fullscreen mode Exit fullscreen mode

🔎 Observations

  1. At first → 2 Pods run concurrently.
  2. Once they complete → next 2 Pods are created.
  3. Finally, all 4 Pods complete successfully.
  4. The Job is marked 4/4 completed.

✅ Example output:

$ kubectl get pods
NAME         READY   STATUS      RESTARTS   AGE
job4-6wxcv   0/1     Completed   0          16s
job4-c6j4c   0/1     Completed   0          34s
job4-fnvpt   0/1     Completed   0          34s
job4-jlfwr   0/1     Completed   0          16s

$ kubectl get jobs
NAME   COMPLETIONS   DURATION   AGE
job4   4/4           36s        45s
Enter fullscreen mode Exit fullscreen mode

j1

🔹 Step 04: Delete the Job 

# Delete the Job
kubectl delete job job4
Enter fullscreen mode Exit fullscreen mode

🎯 Conclusion

  • completions = total Pods that must complete.
  • parallelism = max Pods running at the same time.
  • Together, they give you fine-grained control over batch workloads in Kubernetes.

👉 Example:

  • completions: 10, parallelism: 5 → Run 10 Pods total, 5 at a time.

This makes Jobs highly flexible for parallel processing, ETL tasks, and

data crunching in GKE.

🌟 Thanks for reading! If this post added value, a like ❤️, follow, or share would encourage me to keep creating more content.

— Latchu | Senior DevOps & Cloud Engineer

☁️ AWS | GCP | ☸️ Kubernetes | 🔐 Security | ⚡ Automation
📌 Sharing hands-on guides, best practices & real-world cloud solutions

Top comments (0)