Nalluri Gowtham

Posted on Jul 6

Not Every Workload Needs Replicas: Meet DaemonSets

#kubernetes #devops #cloudnative #containers

Introduction

Kubernetes offers different workload resources to handle different application requirements. Deployments manage long-running applications, Jobs execute one-time tasks, and CronJobs automate scheduled workloads.

But what if you need a Pod to run on every node in your Kubernetes cluster?

For example, suppose you want to collect logs from every node, monitor node-level metrics, or run a networking component that every worker node depends on. Creating individual Pods manually for each node would be inefficient and difficult to maintain.

This is where Kubernetes DaemonSets come in.

A DaemonSet ensures that a copy of a specific Pod runs on every eligible node in the cluster. Whenever a new node joins the cluster, Kubernetes automatically schedules the Pod on that node. Likewise, if a node is removed, the corresponding Pod is automatically removed.

In this article, we'll explore what DaemonSets are, how they work, how to create them, common use cases, and best practices for managing node-level workloads in Kubernetes.

What is a Kubernetes DaemonSet?

A DaemonSet is a Kubernetes workload resource that ensures one Pod runs on every eligible node in a cluster.

Unlike Deployments, which schedule Pods based on the desired number of replicas, DaemonSets automatically create exactly one Pod per node.

As your cluster grows, Kubernetes automatically adds another Pod to newly added nodes. If a node is removed, Kubernetes also removes the associated Pod.

This behavior makes DaemonSets ideal for applications that must be available on every worker node.

Some key characteristics of DaemonSets include:

Runs one Pod on every node
Automatically scales with cluster size
Automatically schedules Pods on new nodes
Removes Pods when nodes leave the cluster
Perfect for node-level services
Managed entirely by Kubernetes

Why Use DaemonSets?

Certain applications need direct access to every Kubernetes node rather than serving user traffic.

Instead of deploying multiple replicas manually, DaemonSets automate this process.

Common reasons for using DaemonSets include:

Collect logs from every node
Monitor node performance
Run networking plugins
Deploy storage drivers
Install security agents
Perform node maintenance

Because Kubernetes manages the Pods automatically, administrators don't need to worry about scheduling them manually whenever the cluster changes.

How Kubernetes DaemonSets Work

A DaemonSet continuously monitors the nodes in your cluster.

When a DaemonSet is created:

Kubernetes identifies all eligible worker nodes.
A Pod is scheduled on each node.
Every node runs exactly one copy of the Pod.
If a new node joins the cluster, Kubernetes automatically creates another Pod.
If a node is removed, the corresponding Pod is deleted.
If a DaemonSet Pod fails, Kubernetes recreates it automatically.

This ensures consistent node-level functionality across the cluster.

Figure 1: Kubernetes DaemonSet Workflow

DaemonSet Lifecycle

Let's understand how a DaemonSet behaves during its lifecycle.

1. DaemonSet Created

A DaemonSet resource is submitted to the Kubernetes API Server.

2. Node Discovery

The DaemonSet controller identifies all eligible worker nodes.

3. Pod Scheduling

Kubernetes creates one Pod on each node.

4. Continuous Monitoring

The DaemonSet controller continuously watches the cluster.

Whenever nodes are added or removed, Kubernetes automatically adjusts the Pods.

5. Pod Recovery

If a DaemonSet Pod crashes or is accidentally deleted, Kubernetes immediately recreates it.

6. Cluster Scaling

As the cluster scales up or down, the DaemonSet automatically scales with it.

No manual intervention is required.

Creating Your First DaemonSet

Let's create a simple DaemonSet using a YAML manifest.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentbit-daemonset
spec:
  selector:
    matchLabels:
      app: fluentbit
  template:
    metadata:
      labels:
        app: fluentbit
    spec:
      containers:
      - name: fluentbit
        image: fluent/fluent-bit:latest

Apply the DaemonSet:

kubectl apply -f daemonset.yaml

Kubernetes automatically schedules one Fluent Bit Pod on every eligible node.

Understanding the YAML

Let's break down each section of the DaemonSet manifest.

apiVersion

Specifies the Kubernetes API version.

apiVersion: apps/v1

DaemonSets belong to the apps/v1 API.

kind

Defines the Kubernetes resource type.

kind: DaemonSet

metadata

Contains information about the DaemonSet.

metadata:
  name: fluentbit-daemonset

selector

The selector identifies which Pods belong to the DaemonSet.

selector:
  matchLabels:
    app: fluentbit

The labels inside the selector must match the labels defined in the Pod template.

template

Defines the Pod that Kubernetes creates on every node.

template:

Everything inside the template describes the Pod specification.

labels

Assigns labels to each Pod.

labels:
  app: fluentbit

These labels allow Kubernetes to associate Pods with the DaemonSet.

containers

Defines the container running inside every Pod.

containers:
- name: fluentbit
  image: fluent/fluent-bit:latest

In production environments, this could be:

Fluent Bit
Prometheus Node Exporter
CNI plugins
Security agents
CSI storage drivers

Useful kubectl Commands

Create a DaemonSet:

kubectl apply -f daemonset.yaml

View all DaemonSets:

kubectl get daemonsets

View Pods created by the DaemonSet:

kubectl get pods

Describe the DaemonSet:

kubectl describe daemonset fluentbit-daemonset

View logs from a DaemonSet Pod:

kubectl logs <pod-name>

Delete a DaemonSet:

kubectl delete daemonset fluentbit-daemonset

Common Use Cases of Kubernetes DaemonSets

DaemonSets are commonly used for applications that need to run on every node in a Kubernetes cluster. Instead of manually deploying these services, Kubernetes ensures that each node automatically runs exactly one Pod.

Some of the most common real-world use cases include:

Log Collection

Applications running across different nodes generate logs that need to be collected and centralized.

Tools like Fluent Bit are commonly deployed as DaemonSets because they collect logs from every node and forward them to centralized logging platforms such as Elasticsearch or Loki.

Node Monitoring

Monitoring every node is essential for maintaining cluster health.

Tools such as Prometheus Node Exporter run as DaemonSets to collect CPU, memory, disk, and network metrics from every worker node.

Networking Components

Every Kubernetes node requires networking plugins for Pod communication.

Most Container Network Interface (CNI) plugins such as Calico, Cilium, and Flannel are deployed as DaemonSets.

Storage Drivers

Storage solutions often require software to run on every node.

Many Container Storage Interface (CSI) drivers are deployed using DaemonSets to provide persistent storage capabilities.

Security Agents

Security monitoring tools need visibility into every Kubernetes node.

Runtime security solutions like Falco are commonly deployed as DaemonSets to monitor system events and detect suspicious activities.

Figure 2: Common Use Cases of Kubernetes DaemonSets

Benefits of Kubernetes DaemonSets

Using DaemonSets offers several advantages for cluster-wide services.

Automatic Deployment

Pods are automatically deployed to every eligible node.

Automatic Scaling

Whenever a new worker node joins the cluster, Kubernetes immediately schedules another DaemonSet Pod.

Simplified Management

Administrators don't need to manually deploy Pods for every node.

Improved Reliability

If a DaemonSet Pod crashes, Kubernetes automatically recreates it.

Consistent Node Configuration

Every node runs the same service, ensuring consistent monitoring, logging, networking, or security.

Easy Cluster Expansion

As clusters grow, DaemonSets automatically grow with them.

DaemonSet vs Deployment

Although both DaemonSets and Deployments create Pods, they are designed for different purposes.

Feature	Deployment	DaemonSet
Number of Pods	User-defined replicas	One Pod per node
Scheduling	Based on replicas	Every eligible node
Scaling	Manual or Autoscaling	Automatic with nodes
Common Use Cases	Web applications, APIs	Logging, Monitoring, Networking
Cluster Growth	Replica count remains unchanged	New Pods created automatically

A Deployment focuses on application availability, while a DaemonSet focuses on node-wide services.

Figure 3: Deployment vs DaemonSet

Best Practices for Using DaemonSets

Following best practices helps improve reliability and resource efficiency.

Allocate Appropriate Resources

Define CPU and memory requests and limits to prevent DaemonSet Pods from consuming excessive resources.

Use Node Selectors

Deploy DaemonSets only on nodes where they are required.

This prevents unnecessary Pods from running on specialized nodes.

Configure Taints and Tolerations

Use tolerations if DaemonSets need to run on tainted nodes, such as control plane or GPU nodes.

Keep DaemonSets Lightweight

DaemonSets run on every node, so lightweight containers reduce overall resource consumption.

Monitor DaemonSet Health

Regularly verify that every node has a running DaemonSet Pod.

Useful commands include:

kubectl get daemonsets
kubectl get pods -o wide

Update Carefully

Perform rolling updates to avoid disrupting cluster-wide services.

Common Mistakes to Avoid

Even though DaemonSets are simple to use, there are some common mistakes.

Using DaemonSets for Regular Applications

Web applications and APIs should usually use Deployments instead of DaemonSets.

Ignoring Resource Limits

Running resource-intensive Pods on every node can significantly impact cluster performance.

Deploying Unnecessary Services

Only deploy services that truly need to run on every node.

Forgetting Node Selection

Some workloads should run only on worker nodes or only on specific hardware.

Node selectors and affinities help control placement.

Not Monitoring DaemonSet Status

Always verify that Pods are running on every expected node.

When Should You Use a DaemonSet?

A DaemonSet is the right choice when your application or service needs to run on every Kubernetes node.

Examples include:

Log collectors
Monitoring agents
Network plugins
Storage drivers
Security monitoring tools
Node maintenance utilities

If your workload serves user traffic or requires a fixed number of replicas, a Deployment is usually the better option.

Conclusion

Not every Kubernetes workload should be deployed using Deployments. Some services need to be present on every node to provide logging, monitoring, networking, storage, or security capabilities.

DaemonSets simplify this process by automatically ensuring that one Pod runs on every eligible node. As your cluster grows or shrinks, Kubernetes keeps the DaemonSet synchronized without requiring manual intervention.

Understanding DaemonSets is an important step toward managing production-ready Kubernetes clusters and building reliable cloud-native infrastructure.

Key Takeaways

DaemonSets ensure one Pod runs on every eligible node.
They automatically scale as nodes are added or removed.
DaemonSets are ideal for logging, monitoring, networking, storage, and security.
Kubernetes automatically recreates failed DaemonSet Pods.
They simplify cluster-wide service deployment.
Best practices improve performance and reliability.

FAQs

1. What is a Kubernetes DaemonSet?

A DaemonSet is a Kubernetes workload resource that ensures one Pod runs on every eligible node in a cluster.

2. What is the difference between a Deployment and a DaemonSet?

A Deployment manages a specified number of replicas, while a DaemonSet automatically runs one Pod on every node.

3. When should I use a DaemonSet?

Use a DaemonSet for node-level services such as log collection, monitoring, networking plugins, storage drivers, and security agents.

4. Does a DaemonSet automatically run on new nodes?

Yes. Whenever a new eligible node joins the cluster, Kubernetes automatically schedules a DaemonSet Pod on that node.

5. Can I run a DaemonSet only on selected nodes?

Yes. You can use node selectors, node affinity, or taints and tolerations to control where DaemonSet Pods are deployed.

6. Can a DaemonSet have multiple Pods on a node?

By default, a DaemonSet runs one Pod per eligible node.

7. How do I check if a DaemonSet is running?

Use:

kubectl get daemonsets
kubectl get pods -o wide

8. What are the most common DaemonSet applications?

Some common examples include Fluent Bit, Prometheus Node Exporter, Calico, Cilium, CSI drivers, and Falco.

Running essential services on every Kubernetes node is critical for maintaining observability, security, and cluster health. Pairing these services with intelligent resource optimization helps maximize performance while reducing cloud costs. EcScale automatically analyzes Kubernetes workloads, optimizes resource allocation, and eliminates idle resource waste—helping teams build efficient and cost-effective Kubernetes environments.