DEV Community

Cover image for Kubernetes on AKS: From Zero to Production Workloads -DaemonSets, Jobs and CronJobs (Part 7/9)
Rahimah Sulayman
Rahimah Sulayman

Posted on

Kubernetes on AKS: From Zero to Production Workloads -DaemonSets, Jobs and CronJobs (Part 7/9)

#daemonsets #cronjobs #aks #containers

Introduction

Kubernetes supports more than continuously running application workloads. It also provides controllers for node-level agents, one-off batch processing, and scheduled tasks.

In this mission, I implemented a DaemonSet to run a lightweight agent on each eligible AKS node, scaled the node pool to verify that the agent was scheduled on a newly added node, and then returned the pool to one node. I also configured a batch Job with three completions and a CronJob that runs every minute, verifying their execution through Job status and container logs.

These workload types address different operational requirements: DaemonSets distribute workloads across nodes, Jobs manage finite tasks, and CronJobs create Jobs on a recurring schedule.

What This Part Covers

  • Creating and validating a DaemonSet.
  • Injecting the node name into a container through the Downward API.
  • Scaling the AKS node pool and verifying DaemonSet placement.
  • Defining a batch Job with three successful completions.
  • Inspecting Job status and logs.
  • Scheduling recurring work with a CronJob.
  • Verifying scheduled executions and reviewing the resulting cluster resources.

1. Deploy a DaemonSet

A DaemonSet ensures that a copy of a specified Pod runs on each eligible node. This makes it suitable for node-level agents that need to operate wherever workloads are scheduled.

I defined the DaemonSet in manifests/07-daemonset.yaml:

cat << 'EOF' > manifests/07-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-agent
spec:
  selector:
    matchLabels:
      app: node-agent
  template:
    metadata:
      labels:
        app: node-agent
    spec:
      containers:
        - name: agent
          image: busybox:1.36
          command: ["sh", "-c", "while true; do echo \"$(date) agent running on node $NODE_NAME\"; sleep 30; done"]
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          resources:
            requests:
              cpu: 10m
              memory: 16Mi
            limits:
              cpu: 50m
              memory: 32Mi
EOF
Enter fullscreen mode Exit fullscreen mode

daemonsets

Configuration details

The manifest defines a DaemonSet named node-agent with a single container using busybox:1.36.

The container runs a loop that writes a timestamp and the node name to its output every 30 seconds. This provides a lightweight way to confirm which node is hosting each agent.

The node name is injected through the Kubernetes Downward API:

env:
  - name: NODE_NAME
    valueFrom:
      fieldRef:
        fieldPath: spec.nodeName
Enter fullscreen mode Exit fullscreen mode

This avoids hardcoding a node name into the container command. Each Pod receives the name of the node on which it is running.

The resource configuration also defines explicit requests and limits:

Resource Request Limit
CPU 10m 50m
Memory 16Mi 32Mi

These values keep the agent's resource requirements small while making its resource boundaries explicit.

I applied the manifest:

kubectl apply -f manifests/07-daemonset.yaml
Enter fullscreen mode Exit fullscreen mode

Kubernetes returned:

daemonset.apps/node-agent created
Enter fullscreen mode Exit fullscreen mode

I then monitored the rollout:

kubectl rollout status daemonset/node-agent
Enter fullscreen mode Exit fullscreen mode

The output confirmed:

daemon set "node-agent" successfully rolled out
Enter fullscreen mode Exit fullscreen mode

daemon set

2. Verify DaemonSet Placement and Logs

To inspect the DaemonSet's status, I ran:

kubectl get daemonset node-agent
Enter fullscreen mode Exit fullscreen mode

The output showed:

NAME         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE
node-agent   1         1         1       1            1
Enter fullscreen mode Exit fullscreen mode

At this stage, the cluster had one node, and the DaemonSet had one desired and ready Pod.

I checked the Pod's placement with:

kubectl get pods -l app=node-agent -o wide
Enter fullscreen mode Exit fullscreen mode

The output identified the Pod as node-agent-rdmcq, running on:

aks-nodepool1-33590657-vmss000000
Enter fullscreen mode Exit fullscreen mode

The Pod was in the Running state and had the internal IP 10.244.0.172.

To confirm that the agent was reporting the correct node, I inspected its recent logs:

kubectl logs -l app=node-agent --prefix --tail=1
Enter fullscreen mode Exit fullscreen mode

The output included:

[pod/node-agent-rdmcq/agent] Mon Oct 26 10:02:52 UTC 2026 agent running on node aks-nodepool1-33590657-vmss000000
Enter fullscreen mode Exit fullscreen mode

node-agent

This verified that the agent was running and that the node name supplied through the Downward API was available inside the container.

3. Scale the AKS Node Pool to Verify DaemonSet Behavior

To verify how the DaemonSet responds when the cluster gains another node, I scaled the AKS node pool from one node to two:

az aks scale \
  --resource-group $RESOURCE_GROUP \
  --name $CLUSTER_NAME \
  --nodepool-name nodepool1 \
  --node-count 2
Enter fullscreen mode Exit fullscreen mode

aks create

aks create2

The Azure CLI returned the node pool configuration. Its output also displayed a notice that the command's behavior had been altered by the aks-preview extension.

Once the second node was available, I checked the cluster:

kubectl get nodes
Enter fullscreen mode Exit fullscreen mode

The output showed two nodes in the Ready state:

NAME                                STATUS   ROLES    VERSION
aks-nodepool1-33590657-vmss000000   Ready    <none>   v1.36.4
aks-nodepool1-33590657-vmss000001   Ready    <none>   v1.36.4
Enter fullscreen mode Exit fullscreen mode

I then checked the DaemonSet Pods again:

kubectl get pods -l app=node-agent -o wide
Enter fullscreen mode Exit fullscreen mode

2 nodes

The output showed one agent Pod on each node:

Pod Status Node
node-agent-rdmcq Running aks-nodepool1-33590657-vmss000000
node-agent-mdscg Running aks-nodepool1-33590657-vmss000001

The second agent was running on the newly added node, demonstrating the DaemonSet's node-level placement behavior.

Unlike a Deployment, where the replica count is explicitly configured, a DaemonSet aims to maintain a Pod on every eligible node. As the eligible node count changes, the number of agent Pods changes accordingly.

Return the node pool to one node

After verifying the two-node behavior, I scaled the node pool back to one:

az aks scale \
  --resource-group $RESOURCE_GROUP \
  --name $CLUSTER_NAME \
  --nodepool-name nodepool1 \
  --node-count 1
Enter fullscreen mode Exit fullscreen mode

aks scale

This returned the lab environment to its previous node count. Scaling a node pool affects the cluster's compute capacity and is also relevant when managing cloud costs for a lab environment.

4. Create a Batch Job

A Kubernetes Job manages Pods that perform finite work and are expected to complete successfully. Unlike a continuously running Deployment, a Job tracks successful task completion.

I defined the Job in manifests/08-job.yaml:

cat << 'EOF' > manifests/08-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: report
spec:
  completions: 3
  parallelism: 3
  backoffLimit: 2
  ttlSecondsAfterFinished: 3600
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: report
          image: busybox:1.36
          command: ["sh", "-c", "echo \"Report part generated by $(hostname)\"; sleep 5"]
EOF
Enter fullscreen mode Exit fullscreen mode

batch job

Job configuration

The Job is named report and uses busybox:1.36 to print a message identifying the Pod that generated each report part.

The important settings are:

  • completions: 3 specifies that three successful completions are required.
  • parallelism: 3 allows up to three Job Pods to run concurrently.
  • backoffLimit: 2 sets the permitted retry count for failed Pods before the Job is considered failed.
  • ttlSecondsAfterFinished: 3600 allows the completed Job to be eligible for automatic cleanup one hour after it finishes.
  • restartPolicy: Never prevents a failed container from being restarted within the same Pod.

This configuration models a small batch operation with three successful executions. Each Pod prints its own hostname, making it possible to distinguish the individual task executions in the logs.

I applied the Job:

kubectl apply -f manifests/08-job.yaml
Enter fullscreen mode Exit fullscreen mode

The initial output confirmed:

job.batch/report created
Enter fullscreen mode Exit fullscreen mode

My first verification attempt returned a NotFound error when waiting for the Job to complete, and the immediate Job listing showed no resources. I reapplied the manifest and checked the resulting Job again.

kubectl apply -f manifests/08-job.yaml
Enter fullscreen mode Exit fullscreen mode

job yaml

The subsequent execution completed successfully.

5. Verify Job Completion and Inspect the Logs

I checked the Job status:

kubectl get job report
Enter fullscreen mode Exit fullscreen mode

The output showed:

NAME     STATUS     COMPLETIONS   DURATION
report   Complete   3/3           10s
Enter fullscreen mode Exit fullscreen mode

The 3/3 completion count confirmed that the Job had achieved its three required successful completions.

To inspect the output from all the Job's Pods, I ran:

kubectl logs -l job-name=report --prefix
Enter fullscreen mode Exit fullscreen mode

kubectl logs

The logs included messages such as:

[pod/report-2v8b5/report] Report part generated by report-2v8b5
[pod/report-bxzps/report] Report part generated by report-bxzps
[pod/report-plnqc/report] Report part generated by report-plnqc
Enter fullscreen mode Exit fullscreen mode

Each execution produced a message containing its own Pod hostname. This confirmed that the Job's three task executions ran and completed.

The logs also demonstrate why Job status and application output are complementary checks. The completion count confirms the controller's recorded outcome, while the logs show what the containers actually executed.

6. Schedule Recurring Work with a CronJob

A CronJob creates Jobs according to a schedule. It is useful for recurring operations such as periodic reporting, cleanup tasks, and scheduled maintenance.

I defined the CronJob in manifests/09-cronjob.yaml:

cat << 'EOF' > manifests/09-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: heartbeat
spec:
  schedule: "*/1 * * * *"
  successfulJobsHistoryLimit: 2
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: heartbeat
              image: busybox:1.36
              command: ["sh", "-c", "echo \"Heartbeat at $(date)\""]
EOF
Enter fullscreen mode Exit fullscreen mode

The CronJob is named heartbeat and uses the schedule:

*/1 * * * *
Enter fullscreen mode Exit fullscreen mode

This cron expression requests execution every minute.

The Job template uses a BusyBox container that prints a timestamp. Unlike the report Job, which required three successful completions, this CronJob creates a new Job each time its schedule is triggered.

The history settings control how many finished Jobs are retained by the CronJob controller:

  • successfulJobsHistoryLimit: 2 retains up to two successful Jobs.
  • failedJobsHistoryLimit: 1 retains up to one failed Job.

The Pod template uses restartPolicy: OnFailure, allowing a failed container to be restarted within its Pod according to Kubernetes' restart behavior.

I applied the manifest:

kubectl apply -f manifests/09-cronjob.yaml
Enter fullscreen mode Exit fullscreen mode

Kubernetes confirmed:

cronjob.batch/heartbeat created
Enter fullscreen mode Exit fullscreen mode

cronjob

7. Verify Scheduled Executions

I inspected the CronJob:

kubectl get cronjob heartbeat
Enter fullscreen mode Exit fullscreen mode

cronjob heartbeat

The output showed the every-minute schedule and the CronJob's status fields, including its suspension state, active Job count, and last scheduled execution.

The initial check occurred before the first scheduled run had been recorded, so the LAST SCHEDULE field was still empty.

After subsequent checks, I listed Jobs in creation order:

kubectl get jobs --sort-by=.metadata.creationTimestamp
Enter fullscreen mode Exit fullscreen mode

The output included the completed report Job and a completed Job created by the heartbeat CronJob.

To inspect the latest scheduled Job's output, I ran:

kubectl logs job/$(kubectl get jobs --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1].metadata.name}')
Enter fullscreen mode Exit fullscreen mode

The output was:

Heartbeat at Mon Oct 26 12:34:01 UTC 2026
Enter fullscreen mode Exit fullscreen mode

kubectl logs job

This confirmed that the scheduled workload had executed and produced its expected message.

The naming pattern also makes the controller relationship visible: the CronJob is named heartbeat, while the Jobs it creates include a generated suffix.

8. Inspect the Combined Cluster State

After deploying the DaemonSet, Job, and CronJob, I inspected the broader set of resources:

kubectl get all,configmap,secret,pvc
Enter fullscreen mode Exit fullscreen mode

kubectl get cm,pvc

kubectl get cm, secret,pvc

ls manifests
Enter fullscreen mode Exit fullscreen mode

The manifests folder contains nine yaml files.

manifests

The output showed the different workload types operating in the k8s-lab namespace.

The relevant resources included:

Resource Observed state
node-agent DaemonSet One desired and ready agent Pod after returning the node pool to one node
report Job Complete, with 3/3 successful completions
heartbeat CronJob Configured to run every minute
heartbeat Jobs Completed executions recorded
store StatefulSet 2/2 Pods ready
web Deployment 3/3 replicas available
web-svc Service Internal ClusterIP
web-public Service LoadBalancer with external IP 20.61.154.190
store Service Headless Service with no cluster IP
StatefulSet PVCs Two claims bound, each with 1Gi capacity

9. DaemonSets, Jobs and CronJobs: Key Differences

Although all three resources can create and manage Pods, they serve distinct purposes.

Resource Primary behavior Typical use
DaemonSet Maintains a Pod on each eligible node Node-level agents, monitoring, logging, and other node-scoped services
Job Runs Pods until a finite task reaches its required successful completion count Batch processing, report generation, and one-off operations
CronJob Creates Jobs according to a recurring schedule Scheduled reports, periodic maintenance, and recurring automation

The choice depends on the execution model.

A DaemonSet follows the eligible node count. A Job tracks successful completion of finite work. A CronJob adds a time-based schedule for creating those Jobs.

These resources are complementary rather than interchangeable.

10. Operational Takeaways

This mission demonstrated several useful Kubernetes operations:

  • Node-level coverage: A DaemonSet automatically placed an agent on each eligible node, including the second node added during the test.
  • Runtime visibility: The Downward API supplied the node name to the agent without hardcoding infrastructure details into the container command.
  • Finite task execution: The report Job completed three successful tasks and exposed their outputs through logs.
  • Recurring automation: The heartbeat CronJob created scheduled Jobs that printed timestamps.
  • Execution verification: Job completion counts and container logs provided separate ways to verify task outcomes.
  • Compute management: Scaling the AKS node pool demonstrated the relationship between node count and DaemonSet placement.
  • Resource awareness: The final resource listing showed how node-level, batch, scheduled, stateless, and stateful workloads coexist in the same cluster.

Conclusion

The node-agent DaemonSet was deployed and verified on the AKS node pool. Scaling the pool to two nodes resulted in one agent Pod on each node, and the pool was subsequently returned to one node.

The report Job completed all three required executions, with logs confirming that each Pod generated its report message. The heartbeat CronJob also ran successfully on its configured every-minute schedule, producing timestamped output.

Together, these implementations demonstrate how Kubernetes handles node-scoped agents, finite batch work, and recurring tasks through dedicated controllers.

In Part 8, the series will continue with the next mission, building on the workload and operational capabilities established here.

This article is Part 7 of the nine-part series, “Kubernetes on AKS: From Zero to Production Workloads.”

Top comments (1)

Collapse
 
rahimah_dev profile image
Rahimah Sulayman •

Not every Kubernetes workload needs to run continuously. DaemonSets provide node-level coverage, Jobs handle finite tasks, and CronJobs automate recurring operations. This mission brought these workload patterns together on AKS, with practical verification of node-level agent placement, batch-job completion, and scheduled execution. Understanding when to use each controller is essential for building efficient, maintainable cloud-native infrastructure.