Create Countdown Job in Kubernetes

The Nautilus DevOps team is crafting jobs in the Kubernetes cluster. While they're developing actual scripts/commands, they're currently setting up templates and testing jobs with dummy commands. Please create a job template as per details given below:

1. Create a job named countdown-xfusion.
2. The spec template should be named countdown-xfusion (under metadata), and the container should be named container-countdown-xfusion
3. Utilize image debian with latest tag (ensure to specify as debian:latest), and set the restart policy to Never.
4. Execute the command sleep 5

---

Kubernetes Jobs: A Beginner's Guide to Batch Processing

Introduction

Imagine you need to run a one-time database migration, process a large dataset, or perform a system cleanup. You don't want this task running forever like a web server—you just want it to run, complete, and stop. This is exactly what Kubernetes Jobs are designed for!

In this beginner-friendly guide, we'll walk through creating and managing Jobs in Kubernetes. By the end, you'll be able to run any batch processing task in your cluster with confidence.

---

What You'll Learn

• What Kubernetes Jobs are and when to use them
• How to create and configure Jobs
• How Jobs differ from Pods and Deployments
• How to monitor and troubleshoot Jobs
• Best practices for production use

---

Table of Contents

1. What are Kubernetes Jobs?
2. Understanding the Problem Jobs Solve
3. Step-by-Step Job Creation
4. Job Lifecycle and States
5. Monitoring and Verification
6. Advanced Job Configuration
7. Real-World Examples
8. Common Issues and Troubleshooting
9. Best Practices
10. Conclusion

---

What are Kubernetes Jobs?

The Definition

A Job is a Kubernetes resource that creates one or more pods and ensures that a specified number of them successfully terminate. Think of it as a task runner that:

• Starts a pod
• Runs a command or script
• Waits for it to complete
• Reports success or failure

The Problem Jobs Solve

In any infrastructure, you need to run tasks that:

• Run once and stop - Not continuous like web servers
• Process data - Batch processing, ETL jobs
• Perform migrations - Database schema updates
• Run maintenance - Cleanup, backups, system checks

Without Jobs, you'd have to:

• Run scripts manually (error-prone)
• Use cron (limited scalability)
• Keep pods running indefinitely (waste resources)

How Jobs Work

┌─────────────────────────────────────────────┐
│              Job                            │
│         (countdown-xfusion)                 │
│         COMPLETIONS: 1/1                    │
└──────────────────┬──────────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────────┐
│              Pod                            │
│         (countdown-xfusion-xxxxx)           │
│         STATUS: Completed                   │
│                                             │
│  ┌─────────────────────────────────────┐   │
│  │   Container                         │   │
│  │   Image: debian:latest              │   │
│  │   Command: sleep 5                  │   │
│  │   STATUS: Exited (0)                │   │
│  └─────────────────────────────────────┘   │
└─────────────────────────────────────────────┘

---

Understanding the Problem Jobs Solve

Job vs Pod vs Deployment

Resource	Purpose	Runs Until	Restart Policy	Use Case
Pod	Run a container	Pod terminates	Always	Simple tasks
Job	Run to completion	Success/failure	Never/OnFailure	Batch processing
CronJob	Run on schedule	Each run completes	Never/OnFailure	Scheduled tasks
Deployment	Continuous service	Never	Always	Web servers, APIs
StatefulSet	Stateful service	Never	Always	Databases

When to Use a Job

✅ Use a Job when:

• You need to run a task once
• The task should complete (not run forever)
• You want to ensure the task succeeds
• You need to process a batch of data
• You're doing database migrations
• You're running system maintenance

❌ Don't use a Job when:

• Your application needs to run continuously
• You need auto-scaling
• You need rolling updates
• You need to expose a service

---

Step-by-Step Job Creation

The Scenario

The Nautilus DevOps team needs to create a Job that:

• Runs once and completes
• Uses the debian:latest image
• Executes the command: sleep 5
• Has restart policy: Never

Step 1: Generate the YAML

First, let's create the base YAML manifest:

kubectl create job countdown-xfusion \
  --image=debian:latest \
  --dry-run=client -o yaml > countdown-job.yaml

This generates the YAML template for us.

Step 2: Edit the YAML

Open the file and customize it:

nano countdown-job.yaml

Before (generated):

apiVersion: batch/v1
kind: Job
metadata:
  creationTimestamp: null
  name: countdown-xfusion
spec:
  template:
    metadata:
      creationTimestamp: null
    spec:
      containers:
      - image: debian:latest
        name: countdown-xfusion
        resources: {}
      restartPolicy: Never
status: {}

After (modified):

apiVersion: batch/v1
kind: Job
metadata:
  name: countdown-xfusion
spec:
  template:
    metadata:
      name: countdown-xfusion
    spec:
      containers:
      - name: container-countdown-xfusion
        image: debian:latest
        command: ["/bin/bash"]
        args: ["-c", "sleep 5"]
      restartPolicy: Never

Step 3: Apply the Job

Create the Job in your cluster:

kubectl apply -f countdown-job.yaml

Expected output:

job.batch/countdown-xfusion created

Step 4: Verify the Job

Check that the Job was created:

kubectl get jobs

Expected output:

NAME                COMPLETIONS   DURATION   AGE
countdown-xfusion   0/1           2s         5s

Step 5: Check Pods

View the pods created by the Job:

kubectl get pods

Expected output:

NAME                      READY   STATUS      RESTARTS   AGE
countdown-xfusion-xxxxx   0/1     Completed   0          10s

Step 6: View Logs

Check the output (sleep 5 produces no output):

kubectl logs countdown-xfusion-xxxxx
# No output (expected)

Step 7: Check Completion

Wait a few seconds and check again:

kubectl get jobs

Expected output:

NAME                COMPLETIONS   DURATION   AGE
countdown-xfusion   1/1           5s         15s

---

Job Lifecycle and States

Job Lifecycle Diagram

┌─────────────────────────────────────────────────────────────┐
│                     Job Lifecycle                          │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌──────────┐     ┌──────────┐     ┌──────────┐           │
│  │ Created  │────▶│ Pending  │────▶│ Running  │           │
│  └──────────┘     └──────────┘     └────┬─────┘           │
│                                         │                  │
│                         ┌───────────────┼───────────────┐  │
│                         │               │               │  │
│                         ▼               ▼               ▼  │
│                    ┌──────────┐   ┌──────────┐   ┌──────────┐
│                    │Complete  │   │  Failed  │   │  Retry   │
│                    │(Success) │   │  (Error) │   │          │
│                    └──────────┘   └──────────┘   └──────────┘
│                                                             │
└─────────────────────────────────────────────────────────────┘

Job States

State	Description	Indicates
Pending	Waiting for resources	Scheduling in progress
Running	Pods are executing	Task is in progress
Complete	All pods succeeded	Task finished successfully ✅
Failed	Some pods failed	Task encountered errors ❌

Pod States

State	Description
Pending	Waiting to be scheduled
Running	Container is executing
Completed	Container exited with 0 (success)
Error	Container exited with non-zero
CrashLoopBackOff	Container keeps crashing

---

Monitoring and Verification

1. Check Job Status

# Basic status
kubectl get job countdown-xfusion

# Detailed status
kubectl describe job countdown-xfusion

# Check completion status
kubectl get job countdown-xfusion -o jsonpath='{.status.conditions[?(@.type=="Complete")].status}'

# Check if failed
kubectl get job countdown-xfusion -o jsonpath='{.status.conditions[?(@.type=="Failed")].status}'

2. View Pods

# List all pods
kubectl get pods

# Filter by job name
kubectl get pods -l job-name=countdown-xfusion

# Get pod details
kubectl describe pod -l job-name=countdown-xfusion

# Pod status
kubectl get pod -l job-name=countdown-xfusion -o jsonpath='{.items[0].status.phase}'

3. Check Logs

# Get pod name
POD_NAME=$(kubectl get pods -l job-name=countdown-xfusion -o name | cut -d'/' -f2)

# View logs
kubectl logs $POD_NAME

# If command produced output
kubectl logs $POD_NAME

# View previous logs (if pod restarted)
kubectl logs $POD_NAME --previous

4. Check Events

# Events related to job
kubectl get events --field-selector involvedObject.name=countdown-xfusion

# All events
kubectl get events --sort-by='.lastTimestamp' | tail -10

# Events related to pods
kubectl get events --field-selector involvedObject.kind=Pod | grep countdown-xfusion

5. Complete Verification Script

#!/bin/bash
echo "=== Job Verification ==="

echo -e "\n1. Job Status:"
kubectl get job countdown-xfusion

echo -e "\n2. Pods:"
kubectl get pods -l job-name=countdown-xfusion

echo -e "\n3. Job Details:"
kubectl describe job countdown-xfusion | grep -E "Name:|Namespace:|Completions:|Duration:|Status:"

echo -e "\n4. Container Details:"
CONTAINER_NAME=$(kubectl get job countdown-xfusion -o jsonpath='{.spec.template.spec.containers[0].name}')
IMAGE=$(kubectl get job countdown-xfusion -o jsonpath='{.spec.template.spec.containers[0].image}')
COMMAND=$(kubectl get job countdown-xfusion -o jsonpath='{.spec.template.spec.containers[0].command}')
ARGS=$(kubectl get job countdown-xfusion -o jsonpath='{.spec.template.spec.containers[0].args}')
RESTART=$(kubectl get job countdown-xfusion -o jsonpath='{.spec.template.spec.restartPolicy}')

echo "Container Name: $CONTAINER_NAME"
echo "Image: $IMAGE"
echo "Command: $COMMAND"
echo "Args: $ARGS"
echo "Restart Policy: $RESTART"

echo -e "\n5. Pod Logs:"
POD_NAME=$(kubectl get pods -l job-name=countdown-xfusion -o name | cut -d'/' -f2)
if [ -n "$POD_NAME" ]; then
    echo "Logs from $POD_NAME:"
    kubectl logs $POD_NAME 2>/dev/null || echo "No logs (command produced no output)"
else
    echo "No pods found"
fi

---

Advanced Job Configuration

1. Parallel Jobs

Run multiple pods in parallel:

apiVersion: batch/v1
kind: Job
metadata:
  name: parallel-job
spec:
  completions: 5        # Total pods to complete
  parallelism: 2        # Run 2 pods at a time
  template:
    spec:
      containers:
      - name: worker
        image: debian:latest
        command: ["/bin/bash"]
        args: ["-c", "echo Working; sleep 10"]
      restartPolicy: Never

2. Job with Failure Handling

apiVersion: batch/v1
kind: Job
metadata:
  name: fault-tolerant-job
spec:
  backoffLimit: 5              # Retry up to 5 times
  activeDeadlineSeconds: 60    # Max 60 seconds
  template:
    spec:
      containers:
      - name: worker
        image: debian:latest
        command: ["/bin/bash"]
        args: ["-c", "sleep 5; exit 1"]  # Intentional failure
      restartPolicy: Never

3. Job with Resource Limits

apiVersion: batch/v1
kind: Job
metadata:
  name: resource-job
spec:
  template:
    spec:
      containers:
      - name: worker
        image: debian:latest
        command: ["/bin/bash"]
        args: ["-c", "echo Processing; sleep 5"]
        resources:
          requests:
            memory: "32Mi"
            cpu: "50m"
          limits:
            memory: "64Mi"
            cpu: "100m"
      restartPolicy: Never

4. Auto-Cleanup After Completion

apiVersion: batch/v1
kind: Job
metadata:
  name: auto-cleanup-job
spec:
  ttlSecondsAfterFinished: 60   # Delete after 60 seconds
  template:
    spec:
      containers:
      - name: worker
        image: debian:latest
        command: ["/bin/bash"]
        args: ["-c", "echo 'Job done!'; sleep 5"]
      restartPolicy: Never

5. Complete Production-Ready Job

apiVersion: batch/v1
kind: Job
metadata:
  name: countdown-xfusion
  namespace: production
  labels:
    app: countdown
    type: job
    environment: production
    team: nautilus
spec:
  completions: 1
  parallelism: 1
  backoffLimit: 3
  activeDeadlineSeconds: 30
  ttlSecondsAfterFinished: 3600
  template:
    metadata:
      name: countdown-xfusion
      labels:
        app: countdown
        type: job
    spec:
      containers:
      - name: container-countdown-xfusion
        image: debian:bullseye-slim
        imagePullPolicy: IfNotPresent
        command: ["/bin/bash"]
        args: ["-c", "sleep 5"]
        resources:
          requests:
            memory: "16Mi"
            cpu: "50m"
          limits:
            memory: "32Mi"
            cpu: "100m"
      restartPolicy: Never

---

Real-World Examples

Example 1: Database Migration

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration
spec:
  backoffLimit: 1
  template:
    spec:
      containers:
      - name: migration
        image: postgres:13
        command: ["/bin/sh"]
        args: ["-c", "psql -h db-service -U postgres -f /migrations/schema.sql"]
        env:
        - name: PGPASSWORD
          valueFrom:
            secretKeyRef:
              name: db-secret
              key: password
        volumeMounts:
        - name: migrations
          mountPath: /migrations
      restartPolicy: Never
      volumes:
      - name: migrations
        configMap:
          name: db-migrations

Example 2: Data Processing

apiVersion: batch/v1
kind: Job
metadata:
  name: data-processor
spec:
  completions: 10
  parallelism: 3
  template:
    spec:
      containers:
      - name: processor
        image: python:3.9
        command: ["/bin/sh"]
        args: ["-c", "python /scripts/process.py --batch=$BATCH_INDEX"]
        env:
        - name: BATCH_INDEX
          valueFrom:
            fieldRef:
              fieldPath: metadata.annotations['batch-index']
        volumeMounts:
        - name: scripts
          mountPath: /scripts
        - name: data
          mountPath: /data
      restartPolicy: Never
      volumes:
      - name: scripts
        configMap:
          name: data-processor-scripts
      - name: data
        persistentVolumeClaim:
          claimName: data-pvc

Example 3: System Cleanup

apiVersion: batch/v1
kind: Job
metadata:
  name: system-cleanup
spec:
  backoffLimit: 2
  template:
    spec:
      containers:
      - name: cleanup
        image: busybox:1.35
        command: ["/bin/sh"]
        args: ["-c", "find /tmp -name '*.tmp' -mtime +7 -delete"]
        volumeMounts:
        - name: tmp
          mountPath: /tmp
      restartPolicy: Never
      volumes:
      - name: tmp
        hostPath:
          path: /tmp

---

Common Issues and Troubleshooting

Issue 1: Pod Not Starting

Symptoms: Job created but no pods running

Solutions:

# Check job status
kubectl describe job countdown-xfusion

# Check pod events
kubectl describe pod -l job-name=countdown-xfusion

# Check if there are resource constraints
kubectl get nodes

# Check if image exists
kubectl run test --image=debian:latest --rm -it -- /bin/bash

Issue 2: Command Not Found

Error: /bin/bash: command not found

Solutions:

# Check if bash exists in the image
kubectl run test --image=debian:latest --rm -it -- which bash

# Use sh instead
kubectl patch job countdown-xfusion -p '{"spec":{"template":{"spec":{"containers":[{"name":"container-countdown-xfusion","command":["/bin/sh","-c","sleep 5"]}]}}}}'

Issue 3: Container Exits with Error

Symptoms: Pod shows Error status

Solutions:

# Check logs
kubectl logs countdown-xfusion-xxxxx

# Check exit code
kubectl get pod countdown-xfusion-xxxxx -o jsonpath='{.status.containerStatuses[0].state.terminated.exitCode}'

# Check if command is correct
kubectl get job countdown-xfusion -o yaml | grep -A 5 command

# Test the command manually
kubectl run test --image=debian:latest --rm -it -- /bin/bash -c "sleep 5"

Issue 4: Job Never Completes

Symptoms: Job stuck in Running state

Solutions:

# Check if command hangs
kubectl logs countdown-xfusion-xxxxx

# Set a deadline
kubectl patch job countdown-xfusion -p '{"spec":{"activeDeadlineSeconds":30}}'

# Check pod status
kubectl describe pod -l job-name=countdown-xfusion

# Delete the job
kubectl delete job countdown-xfusion

Issue 5: Too Many Failed Attempts

Symptoms: Many failed pods

Solutions:

# Check backoff limit
kubectl get job countdown-xfusion -o jsonpath='{.spec.backoffLimit}'

# Increase backoff limit if needed
kubectl patch job countdown-xfusion -p '{"spec":{"backoffLimit":5}}'

# Clean up failed jobs
kubectl delete pods -l job-name=countdown-xfusion

---

Best Practices

✅ DO's

1. Use Specific Image Tags

# BAD - Unpredictable
image: debian:latest

# GOOD - Specific version
image: debian:bullseye-slim

2. Set Resource Limits

resources:
  requests:
    memory: "16Mi"
    cpu: "50m"
  limits:
    memory: "32Mi"
    cpu: "100m"

3. Set Appropriate Backoff Limit

backoffLimit: 3  # Reasonable retry limit

4. Use RestartPolicy: Never

restartPolicy: Never  # For jobs that should run once

5. Test Commands Locally

# Test before deploying
docker run --rm debian:latest /bin/bash -c "sleep 5"

6. Add Labels

metadata:
  labels:
    app: countdown
    type: job
    team: nautilus

7. Set TTL for Auto-Cleanup

ttlSecondsAfterFinished: 3600  # Clean up after 1 hour

8. Monitor Job Completion

# Set up alerts for failures
# Check job status regularly

❌ DON'Ts

1. Don't Use :latest in Production

• Can cause unexpected behavior
• Harder to debug

2. Don't Use RestartPolicy: Always

• Jobs should use Never or OnFailure
• Always will cause infinite retries

3. Don't Ignore Failed Jobs

• Investigate failures
• Clean up failed jobs

4. Don't Create Jobs Without Resource Limits

• Can consume too many resources
• May affect other workloads

5. Don't Forget to Clean Up

• Completed jobs can accumulate
• Use TTL or manual cleanup

---

Job Management Commands

Create Job

# Create from YAML
kubectl apply -f job.yaml

# Create with command
kubectl create job NAME --image=IMAGE

View Jobs

# List jobs
kubectl get jobs

# List jobs in all namespaces
kubectl get jobs --all-namespaces

# Get job details
kubectl describe job NAME

# View job YAML
kubectl get job NAME -o yaml

Update Job

# Edit job
kubectl edit job NAME

# Patch job
kubectl patch job NAME -p '{"spec":{"backoffLimit":5}}'

Delete Job

# Delete job
kubectl delete job NAME

# Delete using YAML
kubectl delete -f job.yaml

# Delete with force
kubectl delete job NAME --force --grace-period=0

View Pods

# View pods for job
kubectl get pods -l job-name=NAME

# View pod logs
kubectl logs POD_NAME

# View pod details
kubectl describe pod POD_NAME

---

Quick Reference Card

Job Commands

Command	Description
kubectl create job NAME --image=IMAGE	Create a job
kubectl get jobs	List jobs
kubectl describe job NAME	Get job details
kubectl get pods -l job-name=NAME	List pods for job
kubectl logs POD_NAME	View pod logs
kubectl delete job NAME	Delete job
kubectl edit job NAME	Edit job

Job Configuration Keys

Key	Description	Default
completions	Total pods to complete	1
parallelism	Pods to run in parallel	1
backoffLimit	Retry attempts	6
activeDeadlineSeconds	Max runtime	None
ttlSecondsAfterFinished	Auto-cleanup delay	None
restartPolicy	Restart on failure	Always

---

Conclusion

You've now learned how to run batch processing tasks in Kubernetes using Jobs! This is a powerful feature that allows you to run any task that needs to complete, from database migrations to data processing.

Key Takeaways

1. Jobs run to completion - They finish and don't restart
2. Jobs are for batch tasks - Not for continuous services
3. Use restartPolicy: Never - Prevents infinite loops
4. Set resource limits - Prevents resource exhaustion
5. Test commands locally - Saves debugging time
6. Monitor job status - Know when tasks complete

What You Learned

✅ What Jobs are and when to use them
✅ How to create and configure Jobs
✅ The Job lifecycle and states
✅ How to monitor and verify Jobs
✅ Real-world Job examples
✅ Best practices for production

Next Steps

Now that you've mastered Jobs, consider exploring:

1. CronJobs - Schedule tasks to run periodically
2. Parallel Processing - Run multiple pods for large datasets
3. Workflows - Complex job chains with Argo Workflows
4. Event-driven Jobs - Trigger jobs based on events
5. Monitoring - Set up alerts for job failures