Inside kube-controller-manager: A Guide to Its 49 Controllers

A Deployment is not progressing, a Namespace is stuck terminating, or Pods remain on an unreachable Node. All three symptoms can involve kube-controller-manager, but they lead to different controllers, queues, and configuration settings.

This guide maps the 49 controllers registered in Kubernetes v1.37.1 to their responsibilities. It also explains why a registered controller might not start, how to inspect runtime configuration, and what to check before increasing concurrency or splitting the process.

The controller inventory and binary defaults target v1.37.1. The runtime output shown later was recorded with kindest/node:v1.37.0. kubeadm and managed Kubernetes services can override the binary defaults.

Follow the objects to find the responsible controller

kube-controller-manager runs many control loops in one process. A typical controller watches API objects through an informer, queues object keys when something changes, reads the cached state, and makes API requests to reconcile it with the desired state.

Source: Kubernetes documentation — Cluster Architecture. The diagram places kube-controller-manager in the control plane; it coordinates changes through the API server.

For a Deployment rollout, the responsibilities form a chain:

Component	Responsibility
Deployment controller	Creates and scales ReplicaSets during a rollout
ReplicaSet controller	Creates or deletes Pods to maintain the requested replica count
kube-scheduler	Assigns unscheduled Pods to Nodes
kubelet	Runs the Pods assigned to its Node through the container runtime

The Deployment controller does not start containers directly. When investigating a rollout, identify the object where progress stopped: Deployment, ReplicaSet, Pod scheduling, or Pod execution.

Multiple controllers can share an informer and its cache within a process. Their workqueues and reconciliation logic remain separate. Worker concurrency, API request throttling, and cache memory are therefore different bottlenecks.

The controller model describes the general pattern. The v1.37.1 Deployment implementation provides a concrete example.

Registered does not mean running

Controller names are stable identifiers defined in controller_names.go. Each registered descriptor associates a name with a constructor, aliases, required feature gates, and whether it is disabled by default.

Three checks determine whether a controller starts:

1. Selection: --controllers must select it. * means controllers enabled by default, rather than every registered controller. -<name> excludes one.
2. Feature gates: every gate required by the descriptor must be enabled.
3. Construction: controller-specific prerequisites must be satisfied. A constructor can decline to start a controller, for example when Node CIDR allocation is disabled.

With --controllers=* and the default feature gates, 43 of the 49 pass the first two checks. That is not a guarantee that 43 controllers are running: keys, certificates, API availability, and other settings still matter.

The startup checks and registration list are in controller_descriptor.go.

Three controllers are excluded from the default selection

Controller	Purpose
bootstrap-signer-controller	Signs the bootstrap discovery cluster-info ConfigMap
token-cleaner-controller	Deletes expired bootstrap token Secrets
selinux-warning-controller	Reports potential conflicts from Pods sharing a volume with different SELinux labels

kubeadm explicitly adds the first two using their aliases:

--controllers=*,bootstrapsigner,tokencleaner

That is a kubeadm configuration choice, not the kube-controller-manager binary's default. The SELinux warning controller can be added with --controllers=*,selinux-warning-controller.

Three more are blocked by default feature gates

Controller	Required gates	Default barrier in v1.37.1
podgroup-protection-controller	GenericWorkload	Beta, disabled by default
storageversion-garbage-collector-controller	APIServerIdentity, StorageVersionAPI	StorageVersionAPI is alpha and disabled
resourcepoolstatusrequest-controller	DRAResourcePoolStatus	Alpha, disabled by default

Adding a controller name does not override a disabled feature gate. Beta also does not necessarily mean enabled by default. The relevant gate definitions are in Kubernetes features and apiserver features.

Other prerequisites include:

Controller	Additional prerequisite
Node IPAM	--allocate-node-cidrs=true and CIDR configuration
ServiceAccount token	--service-account-private-key-file
CSR signing	Shared or signer-specific CA certificate and private key
API server serving ClusterTrustBundle publisher	A CA to publish and an available ClusterTrustBundle API
Garbage collector	--enable-garbage-collector=true
Storage version migrator	Garbage collection enabled

The complete controller inventory

The groups below are a way to navigate responsibilities, not categories defined by Kubernetes source code. Unless noted, a controller is included in the default selection and is not blocked by a default-disabled descriptor gate. Controller-specific prerequisites still apply.

Workloads: 11 controllers

Controller	Responsibility
deployment-controller	Manages ReplicaSets for Deployments and rollout progress
replicaset-controller	Maintains the requested Pod count for ReplicaSets
replicationcontroller-controller	Maintains Pod counts for the legacy ReplicationController API
statefulset-controller	Manages StatefulSet Pod identities, ordering, and PVC creation
daemonset-controller	Maintains Pods on eligible Nodes
job-controller	Manages Job Pods, completions, failures, and retry behavior
cronjob-controller	Creates Jobs according to a schedule
ttl-after-finished-controller	Deletes finished Jobs after ttlSecondsAfterFinished
horizontal-pod-autoscaler-controller	Adjusts replica counts through the scale subresource using metrics
disruption-controller	Calculates PodDisruptionBudget status
podgroup-protection-controller	Protects PodGroups in use; requires the default-disabled GenericWorkload gate

The disruption controller updates the budget; the API server's Eviction API uses that status when deciding whether to allow an eviction. Similarly, HPA reconciliation is built in, but its metric providers are separate components.

See the workload controller constructors and batch controller constructors.

Nodes: 4 controllers

Controller	Responsibility
node-lifecycle-controller	Monitors Node health, updates conditions, and applies health-related taints
taint-eviction-controller	Deletes Pods that do not tolerate a Node's NoExecute taints, including expired tolerations
node-ipam-controller	Allocates Pod CIDRs to Nodes when configured
ttl-controller	Annotates Nodes with a ConfigMap/Secret cache TTL hint

Node failure detection and Pod deletion are separate steps. Node lifecycle examines heartbeats from Node Leases and status updates; taint eviction acts on NoExecute taints and Pod tolerations. The default --node-monitor-grace-period is 50 seconds, but this alone does not define the time until a replacement Pod runs.

The ttl-controller is unrelated to finished Job cleanup. Its node.alpha.kubernetes.io/ttl annotation is used by the kubelet's TTL-based ConfigMap/Secret cache strategy. It does not set the default Watch strategy's update interval.

See the core constructors and TTL implementation.

Storage: 7 controllers

Controller	Responsibility
persistentvolume-binder-controller	Coordinates PV/PVC binding and provisioning-related lifecycle work
persistentvolume-attach-detach-controller	Coordinates volume attachment to and detachment from Nodes
persistentvolume-expander-controller	Coordinates expansion after a PVC's storage request increases
persistentvolume-protection-controller	Protects PVs in use from deletion
persistentvolumeclaim-protection-controller	Protects PVCs in use from deletion
volumeattributesclass-protection-controller	Protects VolumeAttributesClasses in use from deletion
ephemeral-volume-controller	Creates PVCs for generic ephemeral volumes, owned by the Pod

Storage operations cross component boundaries. For CSI volumes, expansion typically also involves an external-resizer and the CSI driver; filesystem expansion can involve the kubelet and CSI driver. A stuck operation is not necessarily a bottleneck inside kube-controller-manager.

The protection controllers use finalizers to defer deletion while the object remains in use. See persistent volume protection and volume expansion.

Services and networking: 4 controllers

Controller	Responsibility
endpoints-controller	Builds Endpoints from Pods matching Service selectors; the Endpoints API is deprecated
endpointslice-controller	Builds EndpointSlices from Pods matching Service selectors
endpointslice-mirroring-controller	Mirrors manually managed Endpoints into EndpointSlices; also deprecated
service-cidr-controller	Maintains ServiceCIDR status and deletion protection

Service ClusterIP allocation itself belongs to the API server's allocator. The ServiceCIDR controller protects and maintains the range objects.

Endpoints remains selected by default in this version despite API deprecation. For selectorless Services, prefer creating EndpointSlices directly rather than relying on Endpoints mirroring. See EndpointSlices and the discovery controller constructors.

Namespace lifecycle and garbage collection: 5 controllers

Controller	Responsibility
namespace-controller	Deletes namespaced content and completes Namespace termination
resourcequota-controller	Recalculates ResourceQuota usage
garbage-collector-controller	Deletes dependent objects according to ownership and deletion propagation
pod-garbage-collector-controller	Removes terminated Pods above the threshold and handles certain orphaned Pods
storageversion-garbage-collector-controller	Removes obsolete StorageVersion objects; requires StorageVersionAPI and APIServerIdentity

A terminating Namespace can be waiting on discovery, unavailable APIs, remaining objects, or their finalizers. More namespace workers cannot fix an unavailable API or a finalizer whose controller is absent.

ResourceQuota accounting also has two sides: this controller updates usage, while quota admission enforcement happens in the API server. The Pod GC threshold defaults to 12,500 terminated Pods; Pod GC is distinct from owner-reference garbage collection.

See garbage collection and the core constructors.

Security and certificates: 13 controllers

Controller	Responsibility
serviceaccount-controller	Creates the default ServiceAccount in each Namespace
serviceaccount-token-controller	Maintains Secret-based ServiceAccount tokens when a signing key is configured
legacy-serviceaccount-token-cleaner-controller	Cleans up eligible unused legacy Secret-based tokens
clusterrole-aggregation-controller	Aggregates rules for ClusterRoles with aggregationRule
root-ca-certificate-publisher-controller	Publishes the kube-root-ca.crt ConfigMap in Namespaces
certificatesigningrequest-approving-controller	Automatically approves specific classes of CSR
certificatesigningrequest-signing-controller	Signs approved CSRs for configured built-in signers
certificatesigningrequest-cleaner-controller	Removes old CSRs
podcertificaterequest-cleaner-controller	Removes old PodCertificateRequests
kube-apiserver-serving-clustertrustbundle-publisher-controller	Publishes the API server's serving CA as a ClusterTrustBundle
bootstrap-signer-controller	Signs bootstrap discovery information; excluded from *
token-cleaner-controller	Removes expired bootstrap tokens; excluded from *
validatingadmissionpolicy-status-controller	Reports expression type-checking results in ValidatingAdmissionPolicy status

Do not confuse the ServiceAccount token controller with projected, expiring Pod tokens. The latter are issued by the API server through TokenRequest and retrieved and refreshed by the kubelet.

PodCertificateRequest cleanup is built in; certificate issuance requires a separate signer controller. CSR approval also does not mean arbitrary CSR requests are automatically approved.

See the certificate controller constructors, ServiceAccount constructors, and certificate API documentation.

API maintenance and resource allocation: 5 controllers

Controller	Responsibility
storage-version-migrator-controller	Rewrites stored objects in response to StorageVersionMigration requests
resourceclaim-controller	Creates and deletes ResourceClaims from Pod ResourceClaimTemplates
resourcepoolstatusrequest-controller	Processes ResourcePoolStatusRequests; requires default-disabled DRAResourcePoolStatus
device-taint-eviction-controller	Evicts Pods in response to DRA device taints
selinux-warning-controller	Emits events and metrics about potential shared-volume SELinux conflicts; excluded from *

The storage version migrator processes migration requests. A trigger that detects storage-version changes and creates those requests is a separate concern. The in-tree migration design leaves automatic triggering to users; see KEP-4192.

What kube-controller-manager does not provide

An API object can exist without an implementation that acts on it. These components are outside kube-controller-manager and must be provided by your distribution, platform, or installation:

Need	Separate component
Vertical Pod resource adjustment	Vertical Pod Autoscaler
Node autoscaling or provisioning	Cluster Autoscaler or Karpenter
Ingress or Gateway traffic handling	An Ingress controller or Gateway API implementation
Cluster DNS	CoreDNS or another cluster DNS implementation
External DNS records	ExternalDNS or another DNS integration
Application certificate issuance and renewal	cert-manager or another certificate controller
PodCertificateRequest issuance	A signer controller
CPU/memory Metrics API	Metrics Server
Metrics derived from Kubernetes object state	kube-state-metrics
Configurable custom-resource state metrics	resource-state-metrics
Automatic storage version migration requests	A migration trigger, such as the separate kube-storage-version-migrator trigger

HPA is built in, while Metrics Server is not. Likewise, the presence of certificate cleanup controllers does not imply a complete application certificate issuance system.

Source: Kubernetes documentation — Kubernetes Components. kube-controller-manager and cloud-controller-manager are separate control-plane components.

Cloud-specific Node initialization, load balancer management, and route management belong to cloud-controller-manager, not this 49-controller inventory. See the cloud controller architecture and Kubernetes component overview.

Inspect the process before changing its configuration

The secure HTTPS port defaults to 10257. With appropriate network access and authentication/authorization, inspect:

Endpoint	Use
/flagz	Inspect runtime flag values
/statusz	Inspect binary version, start time, uptime, and related status
/metrics	Inspect controllers, queues, API clients, and process metrics

ComponentFlagz and ComponentStatusz are beta and enabled by default in v1.37.1. The endpoint registration is conditional on those gates. z-pages are intended for human debugging; their text output is not a stable parsing interface.

A disposable kind inspection example

The following lab configuration exposes the listener to the API proxy and bypasses endpoint authorization for the named debug paths. It is intended for this disposable kind cluster; it is not a production access configuration.

# kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    apiVersion: kubeadm.k8s.io/v1beta4
    kind: ClusterConfiguration
    controllerManager:
      extraArgs:
      - name: bind-address
        value: "0.0.0.0"
      - name: authorization-always-allow-paths
        value: "/healthz,/livez,/readyz,/flagz,/statusz"

kind create cluster --name kcm-inspect \
  --config kind-config.yaml --image kindest/node:v1.37.0

POD=$(kubectl -n kube-system get pods \
  -l component=kube-controller-manager -o name | cut -d/ -f2)

kubectl get --raw \
  "/api/v1/namespaces/kube-system/pods/https:${POD}:10257/proxy/flagz"

An excerpt from the recorded v1.37.0 output:

concurrent-deployment-syncs:5
concurrent-job-syncs:5
controllers:[*,bootstrapsigner,tokencleaner]
kube-api-burst:30
kube-api-qps:20
leader-elect:true
leader-elect-resource-name:kube-controller-manager
node-monitor-grace-period:50s
terminated-pod-gc-threshold:12500
use-service-account-credentials:true

Notice use-service-account-credentials:true: kubeadm supplies it, while the binary default is false. This is why inspecting the running process matters.

The same proxy path with /statusz provides runtime status. Structured JSON can be requested through a client that sets the full content-negotiation header:

# Accept header for /flagz
'Accept: application/json;v=v1beta1;g=config.k8s.io;as=Flagz'

# Accept header for /statusz
'Accept: application/json;v=v1beta1;g=config.k8s.io;as=Statusz'

The structured request needs g, v, and as; an incomplete structured header receives 406 Not Acceptable. See Kubernetes z-pages.

Clean up the inspection cluster afterward:

kind delete cluster --name kcm-inspect

Diagnose queues, API calls, and process resources separately

Start with the affected controller and the objects it is trying to reconcile. The common metrics provide different kinds of evidence:

Metric	What to investigate
running_managed_controllers	Which controllers are running, using name and manager labels
leader_election_master_status	Whether an instance is the leader
workqueue_depth	Queued work awaiting processing
workqueue_queue_duration_seconds	Time spent waiting before processing
workqueue_work_duration_seconds	Time spent processing an item
workqueue_unfinished_work_seconds	Accumulated elapsed time of unfinished work
workqueue_longest_running_processor_seconds	Longest-running current item
workqueue_retries_total	Rate-limited requeues, which can include causes other than errors
rest_client_request_duration_seconds	API request latency
rest_client_rate_limiter_duration_seconds	Time waiting for client-side request throttling
process_resident_memory_bytes	Process resident memory

Queue name values are not guaranteed to match canonical controller IDs one-to-one. Inspect actual labels. Metric availability also depends on controller startup, registration, and whether the corresponding activity occurred. Definitions are in the workqueue metrics and REST client metrics.

For classic histograms, queue waiting-time p99 can be calculated with this PromQL. Select the intended scrape targets and retain labels such as cluster or instance when comparing independent queues:

histogram_quantile(
  0.99,
  sum by (name, le) (
    rate(workqueue_queue_duration_seconds_bucket[5m])
  )
)

If your scrape configuration stores this metric as a native histogram, use the base metric name and remove le from the grouping. Check the actual stored representation before choosing the query.

Controller-specific metrics help connect queue symptoms to outcomes:

Controller	Useful metric
Job	job_controller_job_sync_duration_seconds, job_controller_job_syncs_total
CronJob	cronjob_controller_job_creation_skew_duration_seconds
HPA	horizontal_pod_autoscaler_controller_reconciliation_duration_seconds
Node lifecycle	node_collector_zone_health, node_collector_evictions_total
Taint eviction	taint_eviction_controller_pod_deletions_total
Node IPAM	node_ipam_controller_cidrset_usage_cidrs

node_collector_evictions_total counts application of eviction-related Node taints, not deleted Pods. node_collector_zone_health is a percentage from 0 to 100, while CIDR set usage is a ratio from 0 to 1. Similar names do not imply the same unit or lifecycle stage.

Tune the bottleneck you actually found

Worker concurrency

Increasing workers can help when independent objects are waiting and CPU and API capacity remain available. Representative binary defaults are:

Controller	Flag	Default
Deployment	--concurrent-deployment-syncs	5
ReplicaSet	--concurrent-replicaset-syncs	5
StatefulSet	--concurrent-statefulset-syncs	5
Job	--concurrent-job-syncs	5
CronJob	--concurrent-cron-job-syncs	5
DaemonSet	--concurrent-daemonset-syncs	2
Disruption	--concurrent-disruption-syncs	1
Namespace	--concurrent-namespace-syncs	10
Garbage collector	--concurrent-gc-syncs	20

More workers can also increase concurrent API requests and CPU use. If the API client is already throttled, worker count alone may not improve throughput.

API client rate limits

--kube-api-qps=20 and --kube-api-burst=30 are defaults applied when constructing API clients. They are not a process-wide total limit of 20 requests per second.

The normal client builder copies configuration to construct clientsets. Unless an explicit limiter is supplied, client-go creates a rate limiter during clientset construction. Independently constructed clients can therefore have separate limiters. Shared informer and discovery clients further complicate the total.

Inspect client-side waiting and API server latency together. Raising QPS can remove a local bottleneck while increasing pressure on the API server and etcd. Separate ServiceAccount credentials improve identity and permission separation; they do not themselves remove throttling.

See the tagged client builder and clientset rate-limiter initialization.

Batching, periodic work, and cache memory

Area	Tradeoff to consider
Endpoints and EndpointSlices	Batching updates can reduce writes while delaying propagation
EndpointSlice size	A larger --max-endpoints-per-slice reduces object count but increases each object's size; default is 100
HPA	A longer sync period reduces evaluation frequency but slows response; default is 15 seconds
ResourceQuota	A longer periodic sync reduces full recalculation frequency; default is five minutes
Pod GC	A lower terminated-Pod threshold removes historical objects earlier
Node lifecycle	Evaluate grace periods together with Lease/status reporting and tolerations

The default informer --min-resync-period is 12 hours, randomized into a 12–24 hour range. Resync gives cached objects another processing opportunity; normal change detection uses Watch. Increasing this period does not directly reduce the number of objects retained in memory.

Use the command reference to navigate flags, then confirm defaults against the relevant release and effective values through /flagz.

More replicas provide availability, not more active workers

With the same leader-election lock, one kube-controller-manager instance runs the controllers as leader. Adding identical replicas improves failover availability; it does not divide the reconciliation workload among them.

There are two distinct approaches when tuning one process is insufficient.

Split controller responsibilities across processes

You can select different controller groups and give each group its own leader-election lock. These commands illustrate selection and lock names only; they omit the other controllers and required authentication and configuration:

# Workload group
kube-controller-manager \
  --controllers=deployment-controller,replicaset-controller,job-controller,statefulset-controller \
  --leader-elect-resource-name=kcm-workload

# Node group
kube-controller-manager \
  --controllers=node-lifecycle-controller,node-ipam-controller,taint-eviction-controller \
  --leader-elect-resource-name=kcm-node

A complete design needs several decisions:

• Assign every required controller to a group. Explicit lists leave unlisted controllers out. A remainder group using * and exclusions can reduce omissions, while controllers excluded from * still need explicit inclusion.
• Avoid overlapping controller assignments. Independent locks allow both groups' leaders to act at the same time.
• Provide replicas for each group if each needs high availability.
• Expect duplicate informers, caches, and List/Watch traffic across processes.
• Decide where serviceaccount-token-controller runs. Startup gives this controller special treatment before the others.

Group controllers with shared informer needs where practical. Deployment and ReplicaSet share ReplicaSet/Pod data; Job, CronJob, and finished-Job TTL share Job data; endpoint controllers share Service and endpoint-related data. Pods are used widely, so splitting still tends to duplicate Pod caches.

Garbage collection, Namespace deletion, and quota accounting also use metadata-oriented resource discovery and informers. Their memory behavior is broader than one workload API. Measure total process RSS and API traffic across all groups, rather than comparing only the largest individual process.

The shared factory and startup logic are in controllermanager.go. Controller Manager Leader Migration, described in KEP-2436, addresses coordinated migration between controller managers; it is a different mechanism from manually dividing responsibilities into groups.

Sharding one controller requires more than a second lock

Splitting workload and Node controllers does not increase the capacity of a single Job controller beyond that group's resources and workers. To scale one controller horizontally, instances need disjoint ownership of reconciliation targets.

Kubernetes v1.37.1 has no general kube-controller-manager flag that shards built-in controllers such as Job across arbitrary object subsets. That requires implementation support. If every shard still watches all objects, caches and incoming Watch traffic also remain duplicated.

KEP-5866: Server-side Sharded List and Watch addresses source-side filtering for partitioned clients. Its ShardedListAndWatch gate is alpha and disabled by default in this release. Enabling it does not automatically turn existing built-in controllers into sharded controllers.

Use the inventory as a debugging map

Find the API object whose lifecycle stopped progressing, identify the responsible controller, and confirm that it is selected, allowed by feature gates, and actually running. Then distinguish queue waiting, slow reconciliation, API throttling, and API server latency before tuning.

If throughput remains insufficient, increase appropriate workers and client limits with API server headroom in view. Split controller groups when process isolation is useful; consider object sharding only when one controller itself needs implementation-level horizontal scaling.

The 49 names tell you what kube-controller-manager can run. Runtime settings and metrics tell you what your cluster is running and where it is spending time.