DEV Community: yosshi_

Inside kube-controller-manager: A Guide to Its 49 Controllers

yosshi_ — Wed, 21 Oct 2026 01:05:38 +0000

A Deployment is not progressing, a Namespace is stuck terminating, or Pods remain on an unreachable Node. All three symptoms can involve kube-controller-manager, but they lead to different controllers, queues, and configuration settings.

This guide maps the 49 controllers registered in Kubernetes v1.37.1 to their responsibilities. It also explains why a registered controller might not start, how to inspect runtime configuration, and what to check before increasing concurrency or splitting the process.

The controller inventory and binary defaults target v1.37.1. The runtime output shown later was recorded with kindest/node:v1.37.0. kubeadm and managed Kubernetes services can override the binary defaults.

Follow the objects to find the responsible controller

kube-controller-manager runs many control loops in one process. A typical controller watches API objects through an informer, queues object keys when something changes, reads the cached state, and makes API requests to reconcile it with the desired state.

Source: Kubernetes documentation — Cluster Architecture. The diagram places kube-controller-manager in the control plane; it coordinates changes through the API server.

For a Deployment rollout, the responsibilities form a chain:

Component	Responsibility
Deployment controller	Creates and scales ReplicaSets during a rollout
ReplicaSet controller	Creates or deletes Pods to maintain the requested replica count
kube-scheduler	Assigns unscheduled Pods to Nodes
kubelet	Runs the Pods assigned to its Node through the container runtime

The Deployment controller does not start containers directly. When investigating a rollout, identify the object where progress stopped: Deployment, ReplicaSet, Pod scheduling, or Pod execution.

Multiple controllers can share an informer and its cache within a process. Their workqueues and reconciliation logic remain separate. Worker concurrency, API request throttling, and cache memory are therefore different bottlenecks.

The controller model describes the general pattern. The v1.37.1 Deployment implementation provides a concrete example.

Registered does not mean running

Controller names are stable identifiers defined in controller_names.go. Each registered descriptor associates a name with a constructor, aliases, required feature gates, and whether it is disabled by default.

Three checks determine whether a controller starts:

Selection: --controllers must select it. * means controllers enabled by default, rather than every registered controller. -<name> excludes one.
Feature gates: every gate required by the descriptor must be enabled.
Construction: controller-specific prerequisites must be satisfied. A constructor can decline to start a controller, for example when Node CIDR allocation is disabled.

With --controllers=* and the default feature gates, 43 of the 49 pass the first two checks. That is not a guarantee that 43 controllers are running: keys, certificates, API availability, and other settings still matter.

The startup checks and registration list are in controller_descriptor.go.

Three controllers are excluded from the default selection

Controller	Purpose
`bootstrap-signer-controller`	Signs the bootstrap discovery `cluster-info` ConfigMap
`token-cleaner-controller`	Deletes expired bootstrap token Secrets
`selinux-warning-controller`	Reports potential conflicts from Pods sharing a volume with different SELinux labels

kubeadm explicitly adds the first two using their aliases:

--controllers=*,bootstrapsigner,tokencleaner

That is a kubeadm configuration choice, not the kube-controller-manager binary's default. The SELinux warning controller can be added with --controllers=*,selinux-warning-controller.

Three more are blocked by default feature gates

Controller	Required gates	Default barrier in v1.37.1
`podgroup-protection-controller`	`GenericWorkload`	Beta, disabled by default
`storageversion-garbage-collector-controller`	`APIServerIdentity`, `StorageVersionAPI`	`StorageVersionAPI` is alpha and disabled
`resourcepoolstatusrequest-controller`	`DRAResourcePoolStatus`	Alpha, disabled by default

Adding a controller name does not override a disabled feature gate. Beta also does not necessarily mean enabled by default. The relevant gate definitions are in Kubernetes features and apiserver features.

Other prerequisites include:

Controller	Additional prerequisite
Node IPAM	`--allocate-node-cidrs=true` and CIDR configuration
ServiceAccount token	`--service-account-private-key-file`
CSR signing	Shared or signer-specific CA certificate and private key
API server serving ClusterTrustBundle publisher	A CA to publish and an available ClusterTrustBundle API
Garbage collector	`--enable-garbage-collector=true`
Storage version migrator	Garbage collection enabled

The complete controller inventory

The groups below are a way to navigate responsibilities, not categories defined by Kubernetes source code. Unless noted, a controller is included in the default selection and is not blocked by a default-disabled descriptor gate. Controller-specific prerequisites still apply.

Workloads: 11 controllers

Controller	Responsibility
`deployment-controller`	Manages ReplicaSets for Deployments and rollout progress
`replicaset-controller`	Maintains the requested Pod count for ReplicaSets
`replicationcontroller-controller`	Maintains Pod counts for the legacy ReplicationController API
`statefulset-controller`	Manages StatefulSet Pod identities, ordering, and PVC creation
`daemonset-controller`	Maintains Pods on eligible Nodes
`job-controller`	Manages Job Pods, completions, failures, and retry behavior
`cronjob-controller`	Creates Jobs according to a schedule
`ttl-after-finished-controller`	Deletes finished Jobs after `ttlSecondsAfterFinished`
`horizontal-pod-autoscaler-controller`	Adjusts replica counts through the scale subresource using metrics
`disruption-controller`	Calculates PodDisruptionBudget status
`podgroup-protection-controller`	Protects PodGroups in use; requires the default-disabled `GenericWorkload` gate

The disruption controller updates the budget; the API server's Eviction API uses that status when deciding whether to allow an eviction. Similarly, HPA reconciliation is built in, but its metric providers are separate components.

See the workload controller constructors and batch controller constructors.

Nodes: 4 controllers

Controller	Responsibility
`node-lifecycle-controller`	Monitors Node health, updates conditions, and applies health-related taints
`taint-eviction-controller`	Deletes Pods that do not tolerate a Node's `NoExecute` taints, including expired tolerations
`node-ipam-controller`	Allocates Pod CIDRs to Nodes when configured
`ttl-controller`	Annotates Nodes with a ConfigMap/Secret cache TTL hint

Node failure detection and Pod deletion are separate steps. Node lifecycle examines heartbeats from Node Leases and status updates; taint eviction acts on NoExecute taints and Pod tolerations. The default --node-monitor-grace-period is 50 seconds, but this alone does not define the time until a replacement Pod runs.

The ttl-controller is unrelated to finished Job cleanup. Its node.alpha.kubernetes.io/ttl annotation is used by the kubelet's TTL-based ConfigMap/Secret cache strategy. It does not set the default Watch strategy's update interval.

See the core constructors and TTL implementation.

Storage: 7 controllers

Controller	Responsibility
`persistentvolume-binder-controller`	Coordinates PV/PVC binding and provisioning-related lifecycle work
`persistentvolume-attach-detach-controller`	Coordinates volume attachment to and detachment from Nodes
`persistentvolume-expander-controller`	Coordinates expansion after a PVC's storage request increases
`persistentvolume-protection-controller`	Protects PVs in use from deletion
`persistentvolumeclaim-protection-controller`	Protects PVCs in use from deletion
`volumeattributesclass-protection-controller`	Protects VolumeAttributesClasses in use from deletion
`ephemeral-volume-controller`	Creates PVCs for generic ephemeral volumes, owned by the Pod

Storage operations cross component boundaries. For CSI volumes, expansion typically also involves an external-resizer and the CSI driver; filesystem expansion can involve the kubelet and CSI driver. A stuck operation is not necessarily a bottleneck inside kube-controller-manager.

The protection controllers use finalizers to defer deletion while the object remains in use. See persistent volume protection and volume expansion.

Services and networking: 4 controllers

Controller	Responsibility
`endpoints-controller`	Builds Endpoints from Pods matching Service selectors; the Endpoints API is deprecated
`endpointslice-controller`	Builds EndpointSlices from Pods matching Service selectors
`endpointslice-mirroring-controller`	Mirrors manually managed Endpoints into EndpointSlices; also deprecated
`service-cidr-controller`	Maintains ServiceCIDR status and deletion protection

Service ClusterIP allocation itself belongs to the API server's allocator. The ServiceCIDR controller protects and maintains the range objects.

Endpoints remains selected by default in this version despite API deprecation. For selectorless Services, prefer creating EndpointSlices directly rather than relying on Endpoints mirroring. See EndpointSlices and the discovery controller constructors.

Namespace lifecycle and garbage collection: 5 controllers

Controller	Responsibility
`namespace-controller`	Deletes namespaced content and completes Namespace termination
`resourcequota-controller`	Recalculates ResourceQuota usage
`garbage-collector-controller`	Deletes dependent objects according to ownership and deletion propagation
`pod-garbage-collector-controller`	Removes terminated Pods above the threshold and handles certain orphaned Pods
`storageversion-garbage-collector-controller`	Removes obsolete StorageVersion objects; requires `StorageVersionAPI` and `APIServerIdentity`

A terminating Namespace can be waiting on discovery, unavailable APIs, remaining objects, or their finalizers. More namespace workers cannot fix an unavailable API or a finalizer whose controller is absent.

ResourceQuota accounting also has two sides: this controller updates usage, while quota admission enforcement happens in the API server. The Pod GC threshold defaults to 12,500 terminated Pods; Pod GC is distinct from owner-reference garbage collection.

See garbage collection and the core constructors.

Security and certificates: 13 controllers

Controller	Responsibility
`serviceaccount-controller`	Creates the default ServiceAccount in each Namespace
`serviceaccount-token-controller`	Maintains Secret-based ServiceAccount tokens when a signing key is configured
`legacy-serviceaccount-token-cleaner-controller`	Cleans up eligible unused legacy Secret-based tokens
`clusterrole-aggregation-controller`	Aggregates rules for ClusterRoles with `aggregationRule`
`root-ca-certificate-publisher-controller`	Publishes the `kube-root-ca.crt` ConfigMap in Namespaces
`certificatesigningrequest-approving-controller`	Automatically approves specific classes of CSR
`certificatesigningrequest-signing-controller`	Signs approved CSRs for configured built-in signers
`certificatesigningrequest-cleaner-controller`	Removes old CSRs
`podcertificaterequest-cleaner-controller`	Removes old PodCertificateRequests
`kube-apiserver-serving-clustertrustbundle-publisher-controller`	Publishes the API server's serving CA as a ClusterTrustBundle
`bootstrap-signer-controller`	Signs bootstrap discovery information; excluded from `*`
`token-cleaner-controller`	Removes expired bootstrap tokens; excluded from `*`
`validatingadmissionpolicy-status-controller`	Reports expression type-checking results in ValidatingAdmissionPolicy status

Do not confuse the ServiceAccount token controller with projected, expiring Pod tokens. The latter are issued by the API server through TokenRequest and retrieved and refreshed by the kubelet.

PodCertificateRequest cleanup is built in; certificate issuance requires a separate signer controller. CSR approval also does not mean arbitrary CSR requests are automatically approved.

See the certificate controller constructors, ServiceAccount constructors, and certificate API documentation.

API maintenance and resource allocation: 5 controllers

Controller	Responsibility
`storage-version-migrator-controller`	Rewrites stored objects in response to StorageVersionMigration requests
`resourceclaim-controller`	Creates and deletes ResourceClaims from Pod ResourceClaimTemplates
`resourcepoolstatusrequest-controller`	Processes ResourcePoolStatusRequests; requires default-disabled `DRAResourcePoolStatus`
`device-taint-eviction-controller`	Evicts Pods in response to DRA device taints
`selinux-warning-controller`	Emits events and metrics about potential shared-volume SELinux conflicts; excluded from `*`

The storage version migrator processes migration requests. A trigger that detects storage-version changes and creates those requests is a separate concern. The in-tree migration design leaves automatic triggering to users; see KEP-4192.

What kube-controller-manager does not provide

An API object can exist without an implementation that acts on it. These components are outside kube-controller-manager and must be provided by your distribution, platform, or installation:

Need	Separate component
Vertical Pod resource adjustment	Vertical Pod Autoscaler
Node autoscaling or provisioning	Cluster Autoscaler or Karpenter
Ingress or Gateway traffic handling	An Ingress controller or Gateway API implementation
Cluster DNS	CoreDNS or another cluster DNS implementation
External DNS records	ExternalDNS or another DNS integration
Application certificate issuance and renewal	cert-manager or another certificate controller
PodCertificateRequest issuance	A signer controller
CPU/memory Metrics API	Metrics Server
Metrics derived from Kubernetes object state	kube-state-metrics
Configurable custom-resource state metrics	resource-state-metrics
Automatic storage version migration requests	A migration trigger, such as the separate kube-storage-version-migrator trigger

HPA is built in, while Metrics Server is not. Likewise, the presence of certificate cleanup controllers does not imply a complete application certificate issuance system.

Source: Kubernetes documentation — Kubernetes Components. kube-controller-manager and cloud-controller-manager are separate control-plane components.

Cloud-specific Node initialization, load balancer management, and route management belong to cloud-controller-manager, not this 49-controller inventory. See the cloud controller architecture and Kubernetes component overview.

Inspect the process before changing its configuration

The secure HTTPS port defaults to 10257. With appropriate network access and authentication/authorization, inspect:

Endpoint	Use
`/flagz`	Inspect runtime flag values
`/statusz`	Inspect binary version, start time, uptime, and related status
`/metrics`	Inspect controllers, queues, API clients, and process metrics

ComponentFlagz and ComponentStatusz are beta and enabled by default in v1.37.1. The endpoint registration is conditional on those gates. z-pages are intended for human debugging; their text output is not a stable parsing interface.

A disposable kind inspection example

The following lab configuration exposes the listener to the API proxy and bypasses endpoint authorization for the named debug paths. It is intended for this disposable kind cluster; it is not a production access configuration.

# kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    apiVersion: kubeadm.k8s.io/v1beta4
    kind: ClusterConfiguration
    controllerManager:
      extraArgs:
      - name: bind-address
        value: "0.0.0.0"
      - name: authorization-always-allow-paths
        value: "/healthz,/livez,/readyz,/flagz,/statusz"

kind create cluster --name kcm-inspect \
  --config kind-config.yaml --image kindest/node:v1.37.0

POD=$(kubectl -n kube-system get pods \
  -l component=kube-controller-manager -o name | cut -d/ -f2)

kubectl get --raw \
  "/api/v1/namespaces/kube-system/pods/https:${POD}:10257/proxy/flagz"

An excerpt from the recorded v1.37.0 output:

concurrent-deployment-syncs:5
concurrent-job-syncs:5
controllers:[*,bootstrapsigner,tokencleaner]
kube-api-burst:30
kube-api-qps:20
leader-elect:true
leader-elect-resource-name:kube-controller-manager
node-monitor-grace-period:50s
terminated-pod-gc-threshold:12500
use-service-account-credentials:true

Notice use-service-account-credentials:true: kubeadm supplies it, while the binary default is false. This is why inspecting the running process matters.

The same proxy path with /statusz provides runtime status. Structured JSON can be requested through a client that sets the full content-negotiation header:

# Accept header for /flagz
'Accept: application/json;v=v1beta1;g=config.k8s.io;as=Flagz'

# Accept header for /statusz
'Accept: application/json;v=v1beta1;g=config.k8s.io;as=Statusz'

The structured request needs g, v, and as; an incomplete structured header receives 406 Not Acceptable. See Kubernetes z-pages.

Clean up the inspection cluster afterward:

kind delete cluster --name kcm-inspect

Diagnose queues, API calls, and process resources separately

Start with the affected controller and the objects it is trying to reconcile. The common metrics provide different kinds of evidence:

Metric	What to investigate
`running_managed_controllers`	Which controllers are running, using `name` and `manager` labels
`leader_election_master_status`	Whether an instance is the leader
`workqueue_depth`	Queued work awaiting processing
`workqueue_queue_duration_seconds`	Time spent waiting before processing
`workqueue_work_duration_seconds`	Time spent processing an item
`workqueue_unfinished_work_seconds`	Accumulated elapsed time of unfinished work
`workqueue_longest_running_processor_seconds`	Longest-running current item
`workqueue_retries_total`	Rate-limited requeues, which can include causes other than errors
`rest_client_request_duration_seconds`	API request latency
`rest_client_rate_limiter_duration_seconds`	Time waiting for client-side request throttling
`process_resident_memory_bytes`	Process resident memory

Queue name values are not guaranteed to match canonical controller IDs one-to-one. Inspect actual labels. Metric availability also depends on controller startup, registration, and whether the corresponding activity occurred. Definitions are in the workqueue metrics and REST client metrics.

For classic histograms, queue waiting-time p99 can be calculated with this PromQL. Select the intended scrape targets and retain labels such as cluster or instance when comparing independent queues:

histogram_quantile(
  0.99,
  sum by (name, le) (
    rate(workqueue_queue_duration_seconds_bucket[5m])
  )
)

If your scrape configuration stores this metric as a native histogram, use the base metric name and remove le from the grouping. Check the actual stored representation before choosing the query.

Controller-specific metrics help connect queue symptoms to outcomes:

Controller	Useful metric
Job	`job_controller_job_sync_duration_seconds`, `job_controller_job_syncs_total`
CronJob	`cronjob_controller_job_creation_skew_duration_seconds`
HPA	`horizontal_pod_autoscaler_controller_reconciliation_duration_seconds`
Node lifecycle	`node_collector_zone_health`, `node_collector_evictions_total`
Taint eviction	`taint_eviction_controller_pod_deletions_total`
Node IPAM	`node_ipam_controller_cidrset_usage_cidrs`

node_collector_evictions_total counts application of eviction-related Node taints, not deleted Pods. node_collector_zone_health is a percentage from 0 to 100, while CIDR set usage is a ratio from 0 to 1. Similar names do not imply the same unit or lifecycle stage.

Tune the bottleneck you actually found

Worker concurrency

Increasing workers can help when independent objects are waiting and CPU and API capacity remain available. Representative binary defaults are:

Controller	Flag	Default
Deployment	`--concurrent-deployment-syncs`	5
ReplicaSet	`--concurrent-replicaset-syncs`	5
StatefulSet	`--concurrent-statefulset-syncs`	5
Job	`--concurrent-job-syncs`	5
CronJob	`--concurrent-cron-job-syncs`	5
DaemonSet	`--concurrent-daemonset-syncs`	2
Disruption	`--concurrent-disruption-syncs`	1
Namespace	`--concurrent-namespace-syncs`	10
Garbage collector	`--concurrent-gc-syncs`	20

More workers can also increase concurrent API requests and CPU use. If the API client is already throttled, worker count alone may not improve throughput.

API client rate limits

--kube-api-qps=20 and --kube-api-burst=30 are defaults applied when constructing API clients. They are not a process-wide total limit of 20 requests per second.

The normal client builder copies configuration to construct clientsets. Unless an explicit limiter is supplied, client-go creates a rate limiter during clientset construction. Independently constructed clients can therefore have separate limiters. Shared informer and discovery clients further complicate the total.

Inspect client-side waiting and API server latency together. Raising QPS can remove a local bottleneck while increasing pressure on the API server and etcd. Separate ServiceAccount credentials improve identity and permission separation; they do not themselves remove throttling.

See the tagged client builder and clientset rate-limiter initialization.

Batching, periodic work, and cache memory

Area	Tradeoff to consider
Endpoints and EndpointSlices	Batching updates can reduce writes while delaying propagation
EndpointSlice size	A larger `--max-endpoints-per-slice` reduces object count but increases each object's size; default is 100
HPA	A longer sync period reduces evaluation frequency but slows response; default is 15 seconds
ResourceQuota	A longer periodic sync reduces full recalculation frequency; default is five minutes
Pod GC	A lower terminated-Pod threshold removes historical objects earlier
Node lifecycle	Evaluate grace periods together with Lease/status reporting and tolerations

The default informer --min-resync-period is 12 hours, randomized into a 12–24 hour range. Resync gives cached objects another processing opportunity; normal change detection uses Watch. Increasing this period does not directly reduce the number of objects retained in memory.

Use the command reference to navigate flags, then confirm defaults against the relevant release and effective values through /flagz.

More replicas provide availability, not more active workers

With the same leader-election lock, one kube-controller-manager instance runs the controllers as leader. Adding identical replicas improves failover availability; it does not divide the reconciliation workload among them.

There are two distinct approaches when tuning one process is insufficient.

Split controller responsibilities across processes

You can select different controller groups and give each group its own leader-election lock. These commands illustrate selection and lock names only; they omit the other controllers and required authentication and configuration:

# Workload group
kube-controller-manager \
  --controllers=deployment-controller,replicaset-controller,job-controller,statefulset-controller \
  --leader-elect-resource-name=kcm-workload

# Node group
kube-controller-manager \
  --controllers=node-lifecycle-controller,node-ipam-controller,taint-eviction-controller \
  --leader-elect-resource-name=kcm-node

A complete design needs several decisions:

Assign every required controller to a group. Explicit lists leave unlisted controllers out. A remainder group using * and exclusions can reduce omissions, while controllers excluded from * still need explicit inclusion.
Avoid overlapping controller assignments. Independent locks allow both groups' leaders to act at the same time.
Provide replicas for each group if each needs high availability.
Expect duplicate informers, caches, and List/Watch traffic across processes.
Decide where serviceaccount-token-controller runs. Startup gives this controller special treatment before the others.

Group controllers with shared informer needs where practical. Deployment and ReplicaSet share ReplicaSet/Pod data; Job, CronJob, and finished-Job TTL share Job data; endpoint controllers share Service and endpoint-related data. Pods are used widely, so splitting still tends to duplicate Pod caches.

Garbage collection, Namespace deletion, and quota accounting also use metadata-oriented resource discovery and informers. Their memory behavior is broader than one workload API. Measure total process RSS and API traffic across all groups, rather than comparing only the largest individual process.

The shared factory and startup logic are in controllermanager.go. Controller Manager Leader Migration, described in KEP-2436, addresses coordinated migration between controller managers; it is a different mechanism from manually dividing responsibilities into groups.

Sharding one controller requires more than a second lock

Splitting workload and Node controllers does not increase the capacity of a single Job controller beyond that group's resources and workers. To scale one controller horizontally, instances need disjoint ownership of reconciliation targets.

Kubernetes v1.37.1 has no general kube-controller-manager flag that shards built-in controllers such as Job across arbitrary object subsets. That requires implementation support. If every shard still watches all objects, caches and incoming Watch traffic also remain duplicated.

KEP-5866: Server-side Sharded List and Watch addresses source-side filtering for partitioned clients. Its ShardedListAndWatch gate is alpha and disabled by default in this release. Enabling it does not automatically turn existing built-in controllers into sharded controllers.

Use the inventory as a debugging map

Find the API object whose lifecycle stopped progressing, identify the responsible controller, and confirm that it is selected, allowed by feature gates, and actually running. Then distinguish queue waiting, slow reconciliation, API throttling, and API server latency before tuning.

If throughput remains insufficient, increase appropriate workers and client limits with API server headroom in view. Split controller groups when process isolation is useful; consider object sharding only when one controller itself needs implementation-level horizontal scaling.

The 49 names tell you what kube-controller-manager can run. Runtime settings and metrics tell you what your cluster is running and where it is spending time.

Native Histograms in Kubernetes 1.37: Migration Results and SLI Design

yosshi_ — Wed, 21 Oct 2026 00:39:46 +0000

Native histograms can remove thousands of _bucket series from a Kubernetes monitoring stack. The harder part is making sure the dashboards, alerts, and remote storage still see the same requests afterward.

I compared four Prometheus configurations against the same single-node kind cluster. The native-preferred configuration reduced head series by 76.4% and RSS by 34.7%. I also found queries that returned only one component's data, aggregations that returned an empty result, and histogram samples that never reached remote storage.

This article assumes familiarity with Prometheus scraping and basic PromQL. It covers Kubernetes v1.37.0 and Prometheus v3.13.2, using the measurements recorded in September 2026. The remote storage results apply to the specific versions listed below.

What changed, and what the experiment measured

Kubernetes 1.37 makes the NativeHistograms feature gate beta and enabled by default. Supported histograms expose both classic and native representations; Prometheus decides which to ingest. The gate does not make every Kubernetes histogram native. See the Kubernetes feature gate definition and KEP-5808.

Native histograms became stable in Prometheus 3.8.0. In the version used here, collection is controlled through scrape configuration, without a native-histogram feature flag.

All four Prometheus instances scraped the same kube-apiserver and kubelet jobs every 15 seconds:

Mode	What it stores
P1: Classic only	Classic histograms
P2: Both	Classic and native representations where available
P3: Native preferred	Native where available; classic otherwise
P4: Convert remaining classic	Native where available; remaining classic converted to native histograms with custom buckets (NHCB)

After approximately one hour:

Measurement	P1	P2	P3	P4
Head series	30,010	31,113	7,076	4,432
Snapshot block total	4,052 KiB	3,976 KiB	1,140 KiB	752 KiB
Block `index`	2,580 KiB	2,672 KiB	620 KiB	408 KiB
Block `chunks`	1,460 KiB	1,292 KiB	508 KiB	332 KiB
WAL	4,000 KiB	4,772 KiB	1,580 KiB	1,392 KiB
RSS	190 MiB	189 MiB	124 MiB	111 MiB
Histogram metric families still stored as classic	74	74	27	0

The block measurements came from a snapshot with skip_head=false, which writes the Head to a block. They were not measured after normal two-hour Head compaction or subsequent block compaction. WAL is the usage at that instant, not a long-term storage estimate.

These are single-point measurements from a small cluster. Scrape timing, changes in the source metrics, GC, and process state introduce variation. In particular, P2's slightly lower RSS and chunk usage do not establish that storing both formats saves resources.

For P3, the reductions relative to P1 were:

Head series: 76.4%
Block index: 76.0%
Block chunks: 65.2%
RSS: 34.7%

Series reduction is useful, but it is not a memory or storage sizing formula.

Why the series count falls

A classic histogram stores one float series per bucket boundary, plus _sum and _count. For a histogram with 20 bucket boundaries and 100,000 label combinations, that is:

label_combinations = 100_000
bucket_boundaries = 20
classic_series = label_combinations * (bucket_boundaries + 2)  # 2,200,000

A native histogram stores the distribution as one histogram sample per label combination:

native_series = label_combinations  # 100,000

The bucket counts move inside the sample. Increasing resolution therefore increases sample size, rather than creating more time series.

This explains the difference between index and chunk savings. The index contains series labels and postings; fewer series remove much of that overhead. Chunks store the samples themselves, and a histogram sample is larger than a float. WAL, network traffic, and in-memory chunks also depend on sample size.

Ordinary labels still matter. A route label with 5,000 distinct values continues to create 5,000 label combinations. Native histograms do not fix unbounded URLs, user IDs, or request IDs in labels.

What the distributions look like

Grafana Labs illustrates the difference using response-time heatmaps and histograms from the same application.

Native

Classic

Source: Grafana Labs — Prometheus native histograms in Grafana Cloud. These images come from Grafana's example, not the kind experiment. Axis ranges, scales, and displayed time windows differ, so compare the visible structure rather than bar heights or colors numerically. Classic histograms can also use finer boundaries, at the cost of more series.

Exponential buckets provide consistent relative resolution

For standard exponential native histograms, adjacent boundaries have the ratio:

g = 2 ** (2 ** (-schema))

Kubernetes configures a bucket factor of 1.1, which client_golang maps to schema 3. Its actual growth factor is approximately 1.0905: eight buckets per doubling, with a relative width of about 9.05% measured from the lower boundary.

Schema	Buckets per doubling	Relative bucket width
0	1	100%
2	4	18.9%
3	8	9.05%
4	16	4.43%
7	128	0.543%

The boundaries nest. Combining adjacent pairs of schema 3 buckets produces schema 2 buckets by adding their counts, without interpolation. You can reduce resolution exactly at the bucket-count level, but you cannot recover the finer distribution afterward.

Illustrative counts, with rounded boundaries. The diagram follows the Prometheus schema specification and OpenTelemetry exponential scale definition.

Sparse encoding avoids materializing the entire theoretical range. In the Mimir measurements, 1,259 active native histogram series contained 17,064 active buckets: an average of 13.6 buckets per series.

The Prometheus native histogram specification explains the data model, schemas, and encoding. NHCB uses a different schema: it retains explicitly supplied bucket boundaries rather than exponential ones.

Enable collection, then rewrite the queries

For an existing job, P3 requires this change:

scrape_configs:
  - job_name: kubernetes-apiservers
    scrape_native_histograms: true
    always_scrape_classic_histograms: false
    # Keep the existing discovery, authentication, and relabeling settings.

To keep classic series during migration, set always_scrape_classic_histograms: true for P2. To convert classic-only histograms to NHCB, also set convert_classic_histograms_to_nhcb: true for P4.

The first three switches can be set globally or per job. Starting with a job limits the scope of the change. Prometheus also supports target-level overrides through relabeling. The options are documented in the v3.13.2 configuration reference.

For the exporter libraries and exposition formats used in this experiment, native histogram collection requires protobuf negotiation. Enabling scrape_native_histograms makes protobuf preferred by default. If you explicitly configure scrape_protocols, check that PrometheusProto is preferred. A text response can contain only the classic representation even when the exporter supports native histograms.

The base name replaces the suffix series

When classic storage is disabled for a native-capable metric, the queryable series change:

Classic	Native
`X_bucket{le="..."}`	`X`, containing a histogram sample
`X_sum`	Extract with `histogram_sum()`
`X_count`	Extract with `histogram_count()`

For example, preserve the job grouping while removing le:

# Classic p99 per job
histogram_quantile(
  0.99,
  sum by (job, le) (rate(apiserver_request_duration_seconds_bucket[5m]))
)

# Native p99 per job
histogram_quantile(
  0.99,
  sum by (job) (rate(apiserver_request_duration_seconds[5m]))
)

Other common replacements are:

# Request rate
histogram_count(rate(http_request_duration_seconds[5m]))

# Mean duration per input series
histogram_avg(rate(http_request_duration_seconds[5m]))

# Fraction at or below 500 ms, aggregated across the selected series
histogram_fraction(
  -Inf, 0.5,
  sum(rate(http_request_duration_seconds[5m]))
)

Apply rate() before aggregation so resets are handled per series. Preserve the labels that define the service or operation you intend to measure.

A histogram is not a float. This comparison produces no usable alert result:

rate(http_request_duration_seconds[5m]) > 1

Extract a float first:

histogram_quantile(0.99, rate(http_request_duration_seconds[5m])) > 1

Audit dashboards, recording rules, alert rules, and queries against remote storage. A syntactically valid query can still return nothing after its _bucket, _sum, or _count series disappear.

The subtle failure: a query returns only one component

Of the 74 classic histogram metric families observed in the two jobs, 47 became native and 27 remained classic in P3.

Some remaining histograms use a different implementation, such as TimingHistogram; others come from the Go collector rather than Kubernetes component-base. A third group is affected by initialization order.

The same metric name can therefore have different representations in different components:

Metric	kubelet	kube-apiserver
`workqueue_work_duration_seconds`	Native	Classic
`workqueue_queue_duration_seconds`	Native	Classic
`rest_client_request_duration_seconds`	Native	Classic

This query returns kubelet data only:

sum(rate(workqueue_work_duration_seconds[10m]))

The kube-apiserver observations live in workqueue_work_duration_seconds_bucket and its companion series. Prometheus has no reason to warn that the native query omitted a component you expected to include.

Check coverage before replacing a query:

# Jobs exposing the base-name series
count by (job) (workqueue_work_duration_seconds)

# Jobs exposing classic buckets
count by (job) (workqueue_work_duration_seconds_bucket)

In kube-apiserver v1.37.0, informers and REST clients can be constructed before ApplyFeatureGates() propagates the gate state to the metrics subsystem. Histograms initialized at that point remain classic, while later ones become native. I reported this in kubernetes/kubernetes#142379.

NHCB changes the failure mode

NHCB packages classic bucket counts into native histogram samples while retaining their original boundaries. It reduces series count, but does not improve resolution.

P4 converts the remaining kube-apiserver workqueue histogram to custom buckets, while kubelet still exposes exponential buckets. Aggregating the two produces a warning and an empty result:

sum(rate(workqueue_work_duration_seconds[10m]))

{
  "status": "success",
  "data": { "resultType": "vector", "result": [] },
  "warnings": [
    "PromQL warning: vector contains a mix of histograms with exponential and custom buckets schemas for metric name \"\" (1:5)"
  ]
}

The original kind experiment found eight metric names with this problem. All eight returned an empty result when aggregated across the incompatible schemas.

Keep the aggregation groups separate, for example by job, and inspect API warnings. NHCB is useful for classic-only exporters you cannot change, provided the aggregation and storage paths support it. It is not an automatic final step for every migration.

Remote storage can undo the migration

There are two separate questions: did Prometheus send the samples, and how did the receiver store them?

Local collection and long-term availability require separate checks.

Remote write v1 does not send native histograms by default

For the default v1 protocol, explicitly enable sending:

remote_write:
  - url: http://thanos-receive:19291/api/v1/receive
    send_native_histograms: true

Without this option, Prometheus can store histograms locally while sending none of them to remote storage. The v3.13.2 queue implementation returns early without a log or a dropped-sample increment for this disabled path.

If classic storage has also been disabled, the long-term store loses that histogram entirely.

NHCB requires remote write v2. For a receiver that supports it:

remote_write:
  - url: http://mimir:8080/api/v1/push
    protobuf_message: io.prometheus.write.v2.Request

With v2, native histogram sending is always enabled; send_native_histograms is a no-op. With v1 and native sending enabled, NHCB samples are dropped with a warning and the reason nhcb_in_rw1_not_supported.

Receiver behavior differs

These results are scoped to the tested versions, rather than a claim about every release:

Receiver	Exponential native histograms	NHCB over remote write	Remote write v2
Thanos Receive v0.41.0	Stored as histogram samples	Unavailable through this v1-only Receive path	Unsupported in this version
Grafana Mimir 3.2.1	Stored as histogram samples	Supported	Supported, experimental
VictoriaMetrics v1.152.0	Converted into `_count`, `_sum`, and `_bucket` series with `vmrange` labels	Unsupported	Unsupported

The P4-to-Thanos path shows the additional v1 experiment: exponential samples were sent, while NHCB samples were dropped by Prometheus.

Thanos Receive ingested 12,580 histogram samples without an additional native histogram flag. Its v0.41.0 Receive implementation uses the v1 request type.

Mimir reported 1,259 native histogram series among 7,435 active series. When using query sharding in this version, its help text requires -query-frontend.query-result-response-format=protobuf for native histogram queries. This is an internal response-format requirement, separate from the public JSON query API.

VictoriaMetrics expanded the distributions back into bucket series. Comparing separate receivers fed classic and native data gave:

VictoriaMetrics measurement	Classic input	Native input
`totalSeries`	31,812	25,283
`totalLabelValuePairs`	255,703	191,272

Conversion during ingestion reintroduces multiple float series per distribution.

The receiver's series reduction was 20.5%, compared with 76.4% in the Prometheus Head. Some metrics gained bucket series because more converted buckets were populated than the original classic definition contained. The conversion is visible in the v1.152.0 ingestion implementation.

Head series and totalSeries cover different populations and time spans. These counts also do not measure stored bytes. Measure the receiving system's storage and query behavior before projecting long-term savings.

These choices apply to the tested versions. For P4, also verify aggregation compatibility between exponential and custom bucket schemas.

Better resolution does not make p99.9 stable

The migration changes what distributions you can retain economically. It does not create more requests.

There are two different sources of uncertainty:

Source	What affects it
Inferring typical or future behavior from a finite observation window	Request count, window length, distribution changes, and dependence between requests
Approximating recorded values with buckets	Bucket boundaries, resolution, and interpolation

Increasing schema addresses the second. It cannot resolve the first.

Count the observations in the tail

For N requests and quantile p, the expected number above a fixed population quantile is:

k = N * (1 - p)

Quantile	Tail fraction	Tail count with 1,000 requests	With 100,000 requests
p90	10%	100	10,000
p95	5%	50	5,000
p99	1%	10	1,000
p99.9	0.1%	1	100

With 1,000 requests, p99.9 is effectively tracking the very slowest observations. A finer bucket layout cannot make those observations repeat consistently.

Under independent observations from a fixed distribution, the count above a fixed population quantile is binomial. Its relative standard deviation is sqrt(p/k), approximately 1/sqrt(k) for high quantiles. At k = 1, that is about 100%; at k = 100, about 10%. See the NIST description of the binomial distribution.

Those percentages describe count variation, not percentage error in the latency value. They cannot be directly compared with a 9.05% bucket width. The value uncertainty also depends on the density of observations near the quantile.

Using 100 expected tail observations as a planning heuristic gives:

Quantile	Required requests	Average rate over one hour
p95	2,000	0.56 requests/s
p99	10,000	2.8 requests/s
p99.9	100,000	27.8 requests/s

This is a way to reason about window length, not an accuracy guarantee. Correlated traffic or a changing workload can still make the result unstable.

Scraping is not random request sampling: counters and histograms normally accumulate events between scrapes. Keep finite-window uncertainty separate from bucket approximation, missed scrapes, resets, and rate() extrapolation.

Threshold SLIs still need an error model

A threshold SLI expresses the fraction of requests that meet a latency target:

histogram_fraction(
  -Inf, 0.5,
  sum(rate(http_request_duration_seconds[1h]))
)

This fits the good events / total events model used for error budgets. It does not eliminate low-traffic variation.

It also introduces interpolation when the threshold is inside a bucket. A factor of 1.1 does not guarantee that the estimated good-event fraction is within 10% of the true fraction. If observations cluster near the threshold, even a narrow bucket can contain enough requests to materially change the result.

For exact threshold counts, retain a classic boundary at the SLO threshold or instrument dedicated good/total counters. Prometheus v3.13.2 also supports histogram_fraction() for classic histograms, so estimating arbitrary thresholds is not exclusive to native histograms. Native exponential buckets make relatively fine boundaries available across a broad range.

The Prometheus histogram guidance explains value-domain and quantile-domain errors. The Google SRE Workbook discusses multiple latency thresholds to capture both typical experience and the long tail.

A larger p99 does not automatically mean worse performance

In the P2 experiment, apiserver_request_duration_seconds produced:

Query result	Classic	Native
Mean	4.13 s	4.17 s
p99	60 s	318.6 s

The classic definition's largest finite boundary was 60 seconds. When the requested quantile falls in +Inf, histogram_quantile() returns the previous bucket's upper bound. Native buckets could represent observations beyond that boundary.

This is evidence of a representation limit, not a comparison against raw-event ground truth. The aggregate also included long-running requests such as watches. It is not an example of an appropriate SLI for ordinary API calls. Define the operations and labels first, then compare the same scope and observation window.

Instrumenting a Go exporter

For client_golang, native support starts in v1.14.0. The example below uses the options in v1.24.0:

var requestDuration = prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Name:    "myapp_request_duration_seconds",
        Help:    "Request duration in seconds.",
        Buckets: existingBuckets,

        NativeHistogramBucketFactor:     1.1,
        NativeHistogramMaxBucketNumber:  160,
        NativeHistogramMinResetDuration: time.Hour,
    },
    []string{"method", "route", "status"},
)

Keep the existing classic buckets during migration. Observe each event once; the histogram maintains both representations. promhttp.Handler() can negotiate the protobuf response.

If native histograms are enabled and Buckets is omitted, client_golang does not fill in its default finite classic buckets. Text exposition may still expose _sum, _count, and +Inf; do not assume every classic-related series disappears from the endpoint.

Choose the bucket limit from the observed range

For positive values, a rough estimate of the buckets needed to densely cover a range is:

from math import log, log2

# Approximate bucket count; both expressions are equivalent.
N = log(Vmax / Vmin) / log(g)
N = log2(Vmax / Vmin) * (2 ** schema)

At schema 3:

Range	Approximate bucket count
1 ms to 30 s	119
0.1 ms to 60 s	154
1 byte to 100 MB	213
1 byte to 1 GB	239

Sparse observations may occupy far fewer buckets. The estimate helps identify distributions where a limit of 160 could force coarser resolution, especially size metrics spanning many orders of magnitude.

Recovery from reduced resolution has a cost

In client_golang v1.24.0, exceeding NativeHistogramMaxBucketNumber can trigger a reset, widening of the zero bucket, or reduction of the schema. With NativeHistogramMinResetDuration left at zero, the reset-based recovery path is disabled. A reduced schema does not recover through that mechanism while the same histogram instance remains alive.

Setting time.Hour allows a reset after the minimum interval when needed; it is not an unconditional hourly reset. The reset affects the whole histogram, including classic cumulative values. rate() can correct an observed reset, but cannot recover observations accumulated after the preceding scrape and lost before the reset was scraped.

Kubernetes v1.37.0 component-base configures factor 1.1 and limit 160, but leaves the minimum reset duration unset. I raised the tradeoff in kubernetes/kubernetes#142377. For your own exporter, choose the limit and reset policy together, using the client_golang v1.24.0 option documentation and implementation.

A rollout that checks data coverage

The first migration task is an inventory of metric names and jobs, followed by queries and remote storage.

Find classic families with count by (__name__) ({__name__=~".+_bucket"}) and locate their consumers in dashboards, recording rules, and alerts.
Collect both formats for a bounded comparison period if the impact is broad. Compare counts, sums, and means over matching labels and windows. Expect quantiles to differ when boundaries or interpolation differ.
Check which jobs expose each representation. Verify that rewritten aggregations still include every intended component.
Check native sending and query the receiver directly. For NHCB, verify v2 support and keep incompatible schemas out of the same aggregation group.
Disable classic storage for native-capable metrics once those checks pass. Keep access to the classic queries for historical data until it ages out.

A direct P1-to-P3 switch can also work when the affected queries and storage paths are already known and checked. Configuration reloads change future collection; they do not convert existing samples.

For SLI design, decide the user-facing latency thresholds and required success fractions first. Then choose a window with enough observations and a bucket layout that meets the required approximation accuracy. Native histograms make broad distributions cheaper to retain, but neither tail traffic nor threshold accuracy comes for free.

Experiment versions and measurement commands

Component	Version
Kubernetes / kind node image	v1.37.0 / `kindest/node:v1.37.0`
kind	v0.33.0
Prometheus	v3.13.2
client_golang implementation reference	v1.24.0
Thanos Receive	v0.41.0
Grafana Mimir, single binary	3.2.1
VictoriaMetrics	v1.152.0

The kind cluster used one control-plane node. The local container environment was allocated eight CPUs and 16 GiB of memory for the four Prometheus instances and supporting services.

Head series were read from the TSDB status API:

kubectl -n nh-verify exec deploy/prom-native -- \
  wget -qO- 'http://localhost:9090/api/v1/status/tsdb'

The admin API snapshot included Head data:

kubectl -n nh-verify exec deploy/prom-native -- \
  wget -qO- --post-data='' \
  'http://localhost:9090/api/v1/admin/tsdb/snapshot?skip_head=false'

Block, index, chunk, and WAL sizes were measured with du -sk and reported in KiB. The other three instances used the same jobs and interval with the histogram switches changed according to P1–P4.

The experiment did not measure transfer bytes, long-term compacted storage, or quantile accuracy against raw request observations. Its results establish a comparison for this workload, rather than a production sizing ratio.

An Overview of Agent Substrate

yosshi_ — Tue, 20 Oct 2026 10:44:28 +0000

A Kubernetes Pod is a useful unit of deployment. An agent session has a different lifetime: it may run briefly, wait for minutes, then need its process and filesystem state back for the next turn. Keeping a dedicated environment resident through every wait can consume capacity that other sessions could use.

Agent Substrate separates those lifetimes. Kubernetes runs a pool of worker Pods; Substrate places sandboxed agent environments onto them, persists their state, and restores them when work arrives. This article follows one request through that system, then examines the components and a Counter demo you can try locally.

TL;DR

Kubernetes manages Workers; Substrate manages Actors. Actor state and assignments live behind a separate API backed by PostgreSQL.
Workers can serve many Actors over time. Capacity-based concurrency is combined with suspend/resume to reuse compute.
atelet and ateom save and restore sandbox state. They use gVisor or microVMs; the snapshot fidelity determines what is preserved.
Credentials can stay outside the Actor. An egress gateway and credential provider can inject API credentials into outgoing requests.
Check the implemented behavior, not just the design. Authorization enforcement is experimental and disabled by default for ordinary Control RPCs. Idle detection and performance claims also require context.

Implementation details target Agent Substrate v0.5.0. See Components and versions for the full scope.

This walkthrough assumes familiarity with Pods, Deployments and basic Kubernetes networking. Start with the request lifecycle, jump to the kind demo, or use the evaluation questions to assess the fit for your platform.

The design decision: keep the workers, move the sessions

An agent may execute code briefly, wait for inference or an external tool, then wait indefinitely for another user turn. Running untrusted code also creates a need for isolated environments, often with one tenant per sandbox.

The combination matters: a large fleet can contain many more waiting environments than actively executing ones. Reserving CPU and memory for every waiting agent wastes capacity. Snapshot-based multiplexing is intended to let those environments retain state without remaining assigned to running compute.

Source: docs/architecture.md#L17-L23.

What Kubernetes still does—and what Substrate moves elsewhere

The architecture document identifies three pressures behind its design:

Pressure	Why it matters for agents
API resource count and write traffic	Millions of frequently changing agent records are a different workload from ordinary Kubernetes reconciliation
Activation latency	Pod scheduling, reconciliation, network hops and image preparation add overhead to very short executions
State movement	Rapidly saving and restoring many environments is a different problem from attaching conventional persistent volumes

These are the project's design judgments for its intended scale, rather than universal Kubernetes limits or a claim that every Pod takes seconds to start.

Kubernetes still handles infrastructure provisioning and worker Pod lifecycles. Substrate introduces a separate control plane for Actor state, assignment and resumption, so an activation does not need to wait for a fresh Pod lifecycle.

Sources: docs/architecture.md#L153, docs/architecture.md#L160-L172.

The important boundary is the activation path: a new turn can restore an Actor onto an existing Worker without provisioning another Pod. Database write throughput, reconciliation and environment preparation all contribute to that choice.

Follow a request into a suspended Actor

The execution layer combines routing, state persistence and sandbox isolation:

Source: Agent Substrate on GKE, Google Cloud Blog (September 16, 2026).

Two lifecycles, two APIs

Resource	Meaning	Managed through
`ActorTemplate`	Immutable workload definition, including images and snapshot settings	Substrate API and database
`WorkerPool`	Defines a pool that `atecontroller` reconciles into a Deployment	Kubernetes CRD
`Atespace`	Logical grouping and identity scope for Actors; not a Kubernetes namespace	Substrate API and database
`Actor`	An instance of an ActorTemplate and the unit of suspend/resume	Substrate API and database
`Worker`	A record corresponding to a worker Pod; can host multiple Actors concurrently	Substrate API and database

Sources: docs/glossary.md#L10-L23, docs/glossary.md#L38-L52.

An Actor is the stateful environment that belongs to a session. A Worker is the reusable execution capacity. Suspending an Actor releases its assignment; it does not require destroying and recreating the worker Pod. Worker capacity can also permit several Actors to run concurrently.

Restore first, forward second

For an HTTP request targeting a suspended Actor, the activation path has four steps:

The router identifies the Actor from ate-target-actor.
The control plane assigns a Worker and coordinates restoration of the Actor's snapshot.
Once the assignment is ready, the router forwards the request through the Worker's tunnel to the Actor.
The Actor handles the request and returns its response.

For an Actor that is already running, ResumeActor returns the existing assignment. The call is idempotent.

Source: workflow_resume.go.

Returning a response does not automatically suspend the Actor. The application or higher-level system must choose a safe suspension boundary, taking account of outstanding work and network calls.

Suspend and resume

Source: Agent Substrate on GKE, Google Cloud Blog (September 16, 2026).

Here, “idle” means state has been persisted and the environment removed from running compute, rather than a process simply sleeping. The illustration is conceptual: it does not imply a fixed one-to-one relationship between Workers and Actors.

When a request arrives, atenet-router asks the control plane to resume the target Actor and resolve its Worker assignment. A suspended Actor is restored from its latest snapshot before traffic is forwarded. It may resume on a different Worker.

The router itself does not decide whether the Actor is running; that decision belongs to the control plane. Suspending the Actor saves state and releases its assignment for reuse.

Source: docs/architecture.md#L193-L204.

What runs where

Component	Responsibility	Relationship to Kubernetes
`ate-api-server` (`ateapi`)	Actor lifecycle, scheduling and snapshot coordination	Separate API, scheduler and workflow layer backed by PostgreSQL
`atecontroller`	Reconciles WorkerPool and related CRDs	Standard controller-runtime controller
`atelet` / `ateom`	Coordinates and executes sandbox checkpoint/restore	Additional execution layer inside worker Pod boundaries
`atenet` router and egress	Resumption-aware ingress and controlled outbound traffic	Routing plus Actor lifecycle coordination
`podcertcontroller`	Issues short-lived certificates for component mTLS	Signer controller for `PodCertificateRequest`

The threat-model diagram shows how these components connect:

Source: docs/threat-model.md#L30-L40.

Repository map: controllers, sidecars and tools

The repository also includes supporting programs:

Directory	Role
`cmd/ateapi`	gRPC control plane
`cmd/atecontroller`	Kubernetes controllers
`cmd/atelet`	Node supervisor deployed as a DaemonSet
`cmd/ateom-gvisor`	Drives gVisor inside worker Pods
`cmd/ateom-microvm`	Runs Actors with Cloud Hypervisor microVMs
`cmd/objectstore-plugin`	Snapshot transfer sidecar for node and control-plane operations
`cmd/atenet`	Networking daemon and proxy integration
`cmd/dataplane/envoy`	Envoy image with Rust Dynamic Modules for egress enforcement
`cmd/credential-provider/kubernetes-secrets`	Reference credential provider backed by Kubernetes Secrets
`cmd/podcertcontroller`	Pod certificate signers
`cmd/kubectl-ate`	Actor and other Substrate resource operations
`cmd/ate-setup`	Installation and removal
`cmd/benchmarking`	Synthetic resource-consuming test workloads
`tools/setup-gcp`	GKE, storage and IAM provisioning

The README's tour is not an exhaustive inventory; the sidecar, provider, dataplane and setup entries are supplemented from their own source and documentation.

Sources: README.md#L260-L272, cmd/objectstore-plugin/main.go#L15-L19, cmd/ate-setup/README.md#L3, cmd/dataplane/envoy/README.md.

Resource ownership and assignment

Source: Bringing you Agent Sandbox on GKE and Agent Substrate, Google Cloud Blog (May 20, 2026).

The green sandbox-isolation block is the foundation for the checkpoint and restore operations. Agent Substrate incorporates Agent Sandbox's secure runtime and snapshot capabilities into a separate execution layer.

classDiagram
    namespace kube-apiserver {
        class WorkerPool {
            <<CRD>>
        }
        class Deployment
        class WorkerPod {
            ateom
            runsc
        }
    }

    namespace ate-api-server {
        class ActorTemplate {
            <<record>>
        }
        class Actor {
            <<record>>
            status
            snapshotRefs
        }
        class Worker {
            <<record>>
            actorAssignments
            podIP
        }
    }

    ActorTemplate ..> WorkerPool : worker_selector (labels)
    WorkerPool ..> Deployment : reconciled by atecontroller
    Deployment "1" *-- "*" WorkerPod : manages
    Actor "*" --> "1" ActorTemplate : actor_template
    Actor "0..*" --> "0..1" Worker : runs on
    Worker "1" --> "1" WorkerPod : maps to

Adapted from docs/architecture.md#L268-L303; the multiplicity reflects the implementation's support for multiple Actors per Worker.

The split between API servers is central. Infrastructure resources stay in Kubernetes, while frequent Actor/Worker assignment changes go through ate-api-server and its state store.

Two resource layers

The architecture gives three reasons for this split:

Keep high-frequency Actor updates away from the Kubernetes control plane.
Use low-latency lookups and atomic assignments for resumption. The document's 100ms figure is a design target, not a result measured here.
Retain Kubernetes RBAC, auditing and policies for infrastructure resources such as WorkerPools and SandboxConfigs.

One detail in that document is stale: it says authorization has not been implemented. The same v0.5.0 tag contains OpenFGA-based authorization. --experimental-enable-authz defaults to false and controls enforcement for ordinary Control RPCs. AccessPolicy RPCs are checked regardless of that flag.

Sources: docs/architecture.md#L247-L261, cmd/ateapi/main.go#L84-L88, cmd/ateapi/internal/authz/interceptor.go#L28-L60.

ate-api-server: the Actor control plane

ateapi owns Actor lifecycles, schedules them onto Workers and coordinates snapshots. Its gRPC server connects to PostgreSQL, creates Kubernetes clients and configures its mTLS credentials.

Sources: docs/glossary.md#L56-L58, cmd/ateapi/main.go#L509-L521, cmd/ateapi/main.go#L560-L586.

templateReconciler := controlapi.NewActorTemplateReconciler(persistence, controlSrv, *templateResyncInterval)
templateReconciler.Start(shutdownCtx)

workerAssignmentReconciler := controlapi.NewWorkerAssignmentReconciler(persistence, workerCache)
workerAssignmentReconciler.Start(shutdownCtx)

Source: cmd/ateapi/main.go#L326-L330.

Despite their names, these reconcilers are internal Substrate loops rather than controller-runtime reconcilers watching custom resources. ActorTemplateReconciler prepares golden snapshots; WorkerAssignmentReconciler handles Actors lost when ateom restarts.

This component does not map one-to-one to kube-apiserver: it combines Actor-specific API handling, scheduling and workflows.

atecontroller: conventional Kubernetes reconciliation

atecontroller uses ctrl.NewManager and registers four reconcilers, including WorkerPoolReconciler. Its core structure is familiar:

func (r *WorkerPoolReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    log := log.FromContext(ctx)

    wp := &atev1alpha1.WorkerPool{}
    if err := r.Get(ctx, req.NamespacedName, wp); err != nil {
        if k8errors.IsNotFound(err) {
            return ctrl.Result{}, nil
        }
        return ctrl.Result{}, fmt.Errorf("failed to get worker pool %q: %w", req.NamespacedName, err)
    }

    if !wp.GetDeletionTimestamp().IsZero() {
        log.Info("WorkerPool is being deleted")
        return ctrl.Result{}, nil
    }

    if err := r.reconcileWorkerPool(ctx, wp); err != nil {
        log.Error(err, "Failed to reconcile worker pool")
        return ctrl.Result{}, err
    }

    return ctrl.Result{}, nil
}

Sources: cmd/atecontroller/main.go#L233-L276, cmd/atecontroller/internal/controllers/workerpool_controller.go#L105-L129.

Reconciliation does more than produce a Deployment. The WorkerPool reconciler protects referenced SandboxConfigs with finalizers; a separate SandboxConfigProtectionReconciler releases protection. Updating the recorded SandboxConfig uses two status writes so that status never references a configuration whose protection has already been released.

Owns(&Deployment{}) and Watches(&SandboxConfig{}, ...) connect these resources through normal controller-runtime mechanisms.

Sources: cmd/atecontroller/internal/controllers/workerpool_controller.go#L134-L136, cmd/atecontroller/internal/controllers/workerpool_controller.go#L416-L425.

atelet and ateom: checkpoint and restore

atelet runs on each node as a DaemonSet. ateom runs inside worker Pods and exposes operations such as RunWorkload, CheckpointWorkload and RestoreWorkload.

atelet coordinates worker supervision, image caching and CSI volumes. It also reuses kubelet's image credential-provider mechanism through --image-credential-provider-config.

Sources: docs/architecture.md#L324-L326, cmd/atelet/main.go#L103-L104.

Snapshot upload and download are delegated to objectstore-plugin sidecars over a Unix-socket gRPC API:

Plugin role	Operations
`NodeProvider`, alongside `atelet`	Upload and fetch snapshot files
`ControlProvider`, alongside `ate-api-server`	Copy and delete whole snapshots

The main processes do not directly transfer snapshots to object storage. This does not mean they hold no credentials at all: mTLS and database authentication remain separate concerns.

Source: docs/architecture.md#L330.

For gVisor, cmd/ateom-gvisor/runsc.go invokes runsc checkpoint and runsc restore. The microVM implementation uses Kata integration and Cloud Hypervisor, with userfaultfd demand paging when restoring memory.

Sources: docs/architecture.md#L328-L338.

Kubelet and the container runtime still run the worker Pod itself. Substrate manages sandboxed Actors inside it. Kubernetes already has a Kubelet Checkpoint API; Substrate's distinction is coordinating persistence, assignment, restore and reconnection as an Actor lifecycle.

atenet: routing that can resume its destination

Ingress requests identify an Actor with the ate-target-actor header, formatted as <atespace>/<actor>. In the Envoy configuration described here, an ext_proc processor calls the control plane's ResumeActor RPC before resolving the destination.

Once the assignment is available, the router opens an mTLS CONNECT tunnel to the Worker's atunnel listener on port 8443. atunnel, hosted by ateom, forwards traffic over the Actor's private veth interface. Worker Pod port 80 is not a direct Actor ingress endpoint.

Sources: docs/architecture.md#L344-L354.

The Kubernetes analogy is Service/Ingress/Gateway routing with an extra orchestration step: restore the destination if necessary before forwarding to it.

podcertcontroller: Pod certificate signers

podcertcontroller provides three signers:

Signer	Purpose
`servicedns.podcert.ate.dev/identity`	Service DNS certificates backed by a local CA
`podidentity.podcert.ate.dev/identity`	Pod identity certificates equivalent to Kubernetes service account identity
`postgres.podcert.ate.dev/identity`	PostgreSQL login certificates for `ate-api-server`

Source: cmd/podcertcontroller/main.go#L15-L22.

It watches certificates.k8s.io/v1 PodCertificateRequest objects. The API became GA in Kubernetes v1.37:

PodCertificateRequest: {
    {Version: version.MustParse("1.34"), Default: false, PreRelease: featuregate.Alpha},
    {Version: version.MustParse("1.35"), Default: false, PreRelease: featuregate.Beta},
    {Version: version.MustParse("1.37"), Default: true, PreRelease: featuregate.GA}, // LockToDefault in 1.38
},

Source: kube_features.go.

The API lets kubelet request certificates and project them into Pods. A signer must still process those requests. The well-known kubernetes.io/kube-apiserver-client-pod signer described in the API comments is unimplemented in this version.

Substrate fills its own needs with the three signers above. Its README calls the controller a “polyfill” for the missing signers.

Sources: certificate API types, README.md#L268-L269.

Keep session state and API credentials separate

Isolation, identity and credentials are separate controls

Source: Agent Substrate on GKE, Google Cloud Blog (September 16, 2026).

Kernel isolation does not by itself protect external API credentials. That also requires the gateway and credential-provider configuration described below.

The security document distinguishes Met, In Progress, Future and Out of Scope. Selected principles in v0.5.0 are:

Principle	Meaning	Documented status
Always Sandboxed	Run Actors in gVisor or Cloud Hypervisor microVMs rather than relying on ordinary Linux containers alone	Met
Zero Ambient Credentials	Do not implicitly give Actors infrastructure permissions or external API keys	Met; distinct from demos that explicitly pass keys in environment variables
No Access by Default	Restrict Actor access to the control plane, separate from required infrastructure traffic	Loopback/private netns: Met; DNS filtering: In Progress
Fail-Closed Policy-Enforced Egress Gateway	Control outbound traffic through an egress proxy	Gateway: Met; fail-closed behavior: In Progress, with UDP passthrough noted

Sources: docs/security.md#L30-L33, docs/security.md#L41, docs/security.md#L59, docs/security.md#L75-L77.

Warning: The same document lists Atespace authorization as In Progress, and node blast-radius containment and storage encryption as Future. The security architecture is not evidence that every control is fully implemented. Ordinary Control RPC authorization still requires the experimental flag.

Inject credentials outside the Actor

With egress credential injection, an Actor sends an HTTPS request containing a placeholder header. The gateway obtains the real credential and substitutes it outside the sandbox:

The Actor sends the request with a placeholder value.
The egress gateway terminates TLS and matches an EgressPolicy header-replacement rule.
The credential provider authorizes access and reads the secret.
The gateway replaces the header and makes a separate TLS connection to the destination.

The Actor must trust the gateway's CA. If the request does not contain the header to replace, credential lookup and replacement do not occur.

This keeps the API key outside Actor memory and snapshots. It does not prevent misuse of an allowed API; destination restrictions and application authorization still matter.

Sources: docs/egress-credential-injection.md#L1-L15, docs/egress-credential-injection.md#L41-L54.

The reference kubernetes-secrets provider starts with an empty deny-all policy. Readable namespaces must be explicitly allowed for each Atespace. Actors do not read Kubernetes Secrets themselves.

Source: README.md#L132-L136.

EgressPolicy defines destinations and header replacement

An EgressPolicy specifies allowed traffic and header replacement rules. The credential provider separately decides whether the Actor can use the referenced secret. The API defines the policy as an unordered set of allows: traffic matching no rule is denied.

Source: pkg/proto/ateapipb/ateapi.proto#L609-L610.

For example, allow HTTPS to api.example.com and replace its Authorization header:

rules:
- https:
    hostnames: ["api.example.com"]
    effects:
      replaceHeaders:
      - header: Authorization
        prefix: "Bearer "
        credentialUri: ate-secret://k8s.io/default/ns1/example-api/token

Source: docs/egress-credential-injection.md#L69-L76.

The policy is associated with an Actor rather than embedded in its ActorTemplate:

# Create an egress policy.
kubectl ate create egress-policy <actor-name> -a <atespace> -f policy.yaml

Source: cmd/kubectl-ate/README.md#L247-L253.

Note: The Claude Code demo discussed later directly places ANTHROPIC_API_KEY in the ActorTemplate environment. That differs from gateway injection. In either case, the Actor needs an egress policy permitting its HTTPS destination, such as api.anthropic.com.

The complete request path

With routing and credential injection in place, an HTTP-triggered agent can resume and call an external LLM as follows. This combines the documented flows; Counter itself does not call an LLM.

sequenceDiagram
    actor User as User / higher-level system
    participant Router as atenet-router
    participant API as ate-api-server
    participant Atelet as atelet
    participant Tunnel as ateom / atunnel
    participant Agent as Actor
    participant Egress as atenet-egress
    participant Cred as credential provider
    participant LLM as External LLM API

    User->>Router: HTTP request (ate-target-actor)
    Router->>API: ResumeActor
    alt Actor is suspended
        API->>Atelet: Restore request
        Atelet->>Tunnel: RestoreWorkload
        Tunnel-->>Atelet: Restore complete
        Atelet-->>API: Complete
    else Actor is running
        Note over API: Return existing assignment
    end
    API-->>Router: Worker assignment
    Router->>Tunnel: Forward through mTLS CONNECT
    Tunnel->>Agent: Forward request
    Agent->>Egress: HTTPS (placeholder header)
    Egress->>Cred: Request credential
    Cred-->>Egress: Return API key after authorization
    Egress->>LLM: Replace header and send over a separate TLS connection
    LLM-->>Egress: Response
    Egress-->>Agent: Response
    Agent-->>Tunnel: Result
    Tunnel-->>Router: Response
    Router-->>User: Response
    Note over User,API: A higher-level system can request SuspendActor to save state and release the assignment

Sources: Resume sequence, credential injection. Cold boot without a snapshot and other branches are omitted.

The Actor-to-gateway and gateway-to-provider connections use separate TLS sessions. The gateway must terminate the first to replace headers in the second.

Try the Counter demo with kind

The Counter workload makes state preservation visible: each request increments counters in process memory, a file and the root filesystem. The example below uses gVisor and includes recorded output from a local kind run.

Set up the cluster and demo

Building v0.5.0 requires Go 1.27.0 and a working Docker-based development environment. The microVM option additionally requires /dev/kvm and suitable virtualization support.

Sources: go.mod#L1-L5, hack/create-kind-cluster.sh#L74-L87.

Use a dedicated checkout and cluster name. The creation script deletes and recreates an existing cluster with the same name, so choose one that does not overlap with an environment you want to keep.

git clone --branch v0.5.0 --depth 1 \
  https://github.com/agent-substrate/substrate.git substrate-v0.5.0
cd substrate-v0.5.0
export KIND_CLUSTER_NAME=ate-overview

# create cluster and local registry (IPv4; IP_FAMILY=dual|ipv6 overrides)
hack/create-kind-cluster.sh

# install ate, PostgreSQL, rustfs
hack/install-ate-kind.sh --deploy-ate-system \
  --credential-provider='{"name":"k8s.io"}'

# install counter demo
hack/install-ate-kind.sh --deploy-demo-counter

# install kubectl-ate
go install ./cmd/kubectl-ate

# create a counter actor in the demo's atespace (--template names the
# actor template, resolved in the actor's atespace)
kubectl ate create actor my-counter-1 -a ate-demo-counter --template counter

# port-forward the network router to bind to local port `8000`
kubectl port-forward -n ate-system svc/atenet-router 8000:80

Source: README.md#L111-L129.

The creation script sets up a local registry and a kind cluster with kindest/node:v1.37.0. It explicitly enables certificate-related feature gates, including ClusterTrustBundleProjection; some comments still describe v1.36 requirements.

Source: hack/create-kind-cluster.sh#L104-L113.

The system installation uses ko to build and push its Go components and deploys PostgreSQL, RustFS, CRDs and SandboxConfig resources. The Counter installation adds its WorkerPool and ActorTemplate and waits for a golden snapshot.

WorkerPool and ActorTemplate

The demo's WorkerPool includes the following settings. The installer resolves template placeholders and ko:// references before applying the configuration.

apiVersion: ate.dev/v1alpha1
kind: WorkerPool
metadata:
  name: counter
  namespace: ate-demo-counter
  labels:
    workload: counter
spec:
  replicas: 3
  workerImage: ko://github.com/agent-substrate/substrate/cmd/ateom-gvisor
  sandboxClasses:
  - name: gvisor
  template:
    resources:
      limits:
        cpu: "1"
        memory: 1Gi
      requests:
        cpu: 250m
        memory: 1Gi

Source: demos/counter/counter.yaml.tmpl#L26-L63.

An ActorTemplate is not a Kubernetes object. This YAML maps to Substrate's protobuf fields and is passed through kubectl ate create actor-template:

metadata:
  atespace: ate-demo-counter
  name: counter
workerSelector:
  matchLabels:
    workload: counter
containers:
- name: counter
  image: ko://github.com/agent-substrate/substrate/demos/counter
  wakeupProbe:
    httpGet:
      path: /readyz
      port: 80
  volumeMounts:
  - name: data
    mountPath: /home/counter
resources:
  limits:
  - name: cpu
    quantity: "1"
  - name: memory
    quantity: 512Mi
snapshotConfig:
  preferredFidelity: SNAPSHOT_FIDELITY_MEMORY
  storageLocation: gs://${BUCKET_NAME}/ate-demo-counter/
sandboxConfig:
  sandboxClass: gvisor
  configName: gvisor-default
volumes:
- name: data
  durableDir: {}

Source: demos/counter/counter-template.yaml.tmpl#L22-L69.

SNAPSHOT_FIDELITY_MEMORY requests preservation of process memory. gvisor-default names a cluster-scoped SandboxConfig whose assets specify where to download the runtime binaries:

apiVersion: ate.dev/v1alpha1
kind: SandboxConfig
metadata:
  name: gvisor-default
  annotations:
    sandboxconfig.ate.dev/is-class-default: "true"
spec:
  sandboxClass: gvisor
  defaultVersion: v3
  versions:
  - name: v3
    state: Enabled
    pauseImage: "registry.k8s.io/pause:3.10.2@sha256:f548e0e8e3dc1896ca956272154dde3314e8cc4fde0a57577ee9fa1c63f5baf4"
    assets:
      arm64:
        gvisor:
          url: "gs://gvisor/releases/nightly/2026-10-10/aarch64/gvisor.tar.zstd"
          sha256: "aaf414f06eea7960238e43165c691f0a36cefc0593fb141e011e96211c7b6f57"

Source: manifests/ate-install/sandboxconfig-gvisor.yaml#L22-L48. Only the arm64 assets are shown; amd64 assets are omitted.

Observe Actor state and preserved counters

A newly created Counter Actor starts suspended: its snapshot exists, but it has no Worker assignment yet.

kubectl ate create actor my-counter-1 -a ate-demo-counter --template counter

Recorded output:

ATESPACE           NAME           TEMPLATE                   STATE                   WORKER POD   WORKER IPS   VERSION   AGE
ate-demo-counter   my-counter-1   ate-demo-counter/counter   ACTOR_STATE_SUSPENDED   <none>                    1         0s

Send a request through the router. It resumes the Actor before forwarding, and the recorded response is HTTP 200:

curl -X POST \
  -H "ate-target-actor: ate-demo-counter/my-counter-1" \
  -i http://localhost:8000/

Recorded output:

HTTP/1.1 200 OK
Content-Length: 111
Content-Type: text/plain; charset=utf-8

hello from: 169.254.17.2 | preserved memory count: 1 | preserved file counter: 1 | preserved rootfs counter: 1

A second POST increments the counter to 2. The Actor is now running and assigned to a worker Pod:

curl -X POST \
  -H "ate-target-actor: ate-demo-counter/my-counter-1" \
  http://localhost:8000/

kubectl ate get actor my-counter-1 -a ate-demo-counter

Recorded output:

ATESPACE           NAME           TEMPLATE                   STATE                 WORKER POD                                  WORKER IPS    VERSION   AGE
ate-demo-counter   my-counter-1   ate-demo-counter/counter   ACTOR_STATE_RUNNING   ate-demo-counter/counter-796d677995-v55tv   10.244.0.23   3         20s

Explicitly suspend it, then send another request:

kubectl ate suspend actor my-counter-1 -a ate-demo-counter
curl -X POST \
  -H "ate-target-actor: ate-demo-counter/my-counter-1" \
  http://localhost:8000/

Recorded output:

hello from: 169.254.17.2 | preserved memory count: 3 | preserved file counter: 3 | preserved rootfs counter: 3

The memory, file and root-filesystem counters continue at 3. This illustrates the documented snapshot-and-resume lifecycle, including preserved process memory.

Stop the port-forward when finished, then remove the dedicated cluster. The kind-registry container is separate and remains after cluster deletion.

hack/kind.sh delete cluster --name ate-overview

Plan the deployment around more than Pods

Worker Pods and CRDs are only part of the deployment. The following table distinguishes infrastructure that must be supplied from components the installer can deploy:

Dependency	Purpose	Installation considerations
PostgreSQL	Actor/Worker state store	The kind setup includes a development database; production operators supply database identities, schema and connection settings
GCS or S3-compatible object storage	Durable snapshot storage	Configure `storage.bucketName` / `BUCKET_NAME`; kind uses `ate-snapshots` and an in-cluster RustFS service
Egress credential provider	Resolve credentials outside Actors	Provider selection is required, including explicit `{"enabled":false}` to disable injection; `{"name":"k8s.io"}` deploys the reference provider
Sandbox runtime binaries	Run Actors with gVisor or microVMs	Selected through a cluster-scoped SandboxConfig; default gVisor assets come from a public GCS bucket
Proxy dataplane image	Router and egress proxies	`--atenet-dataplane` selects Envoy, requiring its custom build, or agentgateway, using a published image
Object-storage permissions	Let plugin sidecars transfer snapshots	For GCS, configure the required Workload Identity/IAM bindings; the kind substitute uses local credentials
CSI driver, when needed	Attach CSI-backed persistent storage to Actors	Not needed for the demo's `durableDir`; Substrate imposes additional controller/node deployment requirements

External databases are not provisioned by ateapi or ate-setup. Snapshot-bucket selection and credential-provider selection also need explicit configuration, even when other defaults are available.

Sources: docs/postgres.md#L8, docs/operator-install.md#L314, docs/egress-credential-injection.md#L128-L131.

Missing GCS IAM bindings can produce a 403 on the first suspend/resume, and also prevent control-plane snapshot operations and image pulls.

Source: tools/setup-gcp/README.md#L154-L159.

Note: Substrate directly orchestrates CSI operations from ateapi and atelet. A CSI controller must be reachable as a cluster-network gRPC service, unlike the usual controller-side Unix socket arrangement. See docs/csi-deployment.md#L5.

For the Kubernetes v1.37 setup covered here, Pod certificate and ClusterTrustBundle features are GA and enabled by default. Substrate also includes its certificate controller. Provider-specific installation requirements should still be checked separately.

Source: ClusterTrustBundle feature definition.

GKE availability

Google announced Agent Substrate availability on GKE on September 16, 2026.

The GKE offering installs open-source Substrate onto GKE Standard; Autopilot is not supported. Evaluation and non-production use are available, while production support is allowlisted under a private GA program. Substrate has no additional product charge, but the GKE and other resources created still incur their normal charges.

Source: Install Agent Substrate on GKE.

The announcement also describes ComputeClasses and optional Filestore agent volumes. Shared workspace storage serves a different purpose from snapshots of execution state.

Google reports sub-500ms resumption, more than 500 suspend/resume activations per second and 10x density. These are Google's published figures, separate from upstream design targets and the Counter example.

Source: Agent Substrate on GKE, Google Cloud Blog (September 16, 2026).

What to evaluate in your own workload

Substrate is worth examining when a fleet has many isolated, stateful environments but only a fraction need to execute at once. The useful question is how much resident capacity suspend/resume can release, and whether restoration meets the application's latency needs.

Start with three checks:

Suspension boundaries: identify when an environment can safely stop, including outstanding network calls and application work.
State requirements: decide what must survive, then select snapshot fidelity and any persistent workspace storage accordingly.
Operational ownership: account for the database, snapshot backend, identity and egress services as well as the Workers.

The Counter demo exposes the core mechanism. An agent platform must then connect it to real session events and measure its own activation latency and capacity use.

Further implementation details

Claude Code demo: workload execution versus UI state

What the Claude Code demo actually implements

claude-code-multiplex defines three Claude Code Actors—luna, mars and orion—with a two-Pod WorkerPool. It uses the control API rather than sending workload HTTP through atenet-router, so it does not use the routing header.

Source: demos/claude-code-multiplex/README.md#L14-L21.

An ActorTemplate supplies the task and interval as environment variables:

  env:
  - name: ACTOR_NAME
    value: "luna"
  - name: TASK
    value: "Tell me one short, surprising fact about the Moon. One sentence."
  - name: INTERVAL_SECONDS
    value: "45"
  - name: ANTHROPIC_API_KEY
    value: "${ANTHROPIC_API_KEY}"

Source: demos/claude-code-multiplex/agent-luna-template.yaml.tmpl#L34-L42.

The workload repeatedly runs claude --print and then sleeps:

while true; do
  ...
  claude --print "${TASK}" 2>&1 || echo "[demo-actor:${ACTOR_NAME}] claude exited non-zero"
  ...
  sleep "${INTERVAL_SECONDS}"
done

Source: demos/claude-code-multiplex/workload/run.sh#L17-L49.

That idle interval is an opportunity for multiplexing. However, the v0.5.0 UI does not implement everything its README's task-flow description suggests:

UI behavior	Implementation
“Give a task”	Records the selected Actor and task in UI-server assignment state
`queued` / `running` / `completed`	Derived from timestamps and randomized waiting/execution durations
Actor state and logs	Retrieved from the Substrate and Kubernetes APIs

giveTask does not send a task to the Actor or call ResumeActor / SuspendActor. A comment in computeState identifies the task state as a time-based UI display rather than Substrate state.

Source: demos/claude-code-multiplex/ui/server.go#L252-L315.

A changing UI status therefore does not establish that a real LLM task completed or automatic multiplexing occurred. The useful pieces are the ActorTemplates, the periodic Claude Code workload and the state-viewing UI. Verify suspend/resume through real API operations and Actor state, as in the Counter demo.

RustFS: the local snapshot backend

RustFS

RustFS is a separate, Rust-based S3-compatible object store. The kind installation uses it as its local snapshot backend. The v0.5.0 manifest selects rustfs/rustfs:1.0.0-beta.3.

This development setup does not replace production storage availability, persistence or operational design. objectstore-plugin uses RustFS through its S3 backend; GCS uses the same plugin interface with a different backend.

Sources: manifests/ate-install/kind/rustfs.yaml#L55-L75, rustfs/rustfs.

ReBAC: deriving permissions from relationships

ReBAC and Kubernetes RBAC

Substrate's API authorization uses OpenFGA's relationship-based access control model. In this version, ordinary Control RPC enforcement remains experimental and disabled by default.

Source: docs/security.md#L46.

Kubernetes RBAC checks applicable RoleBindings and ClusterRoleBindings against Role/ClusterRole rules: API groups, resources, verbs, namespaces and sometimes resource names. It does not automatically derive access from ownership or arbitrary parent-child relationships.

Source: Using RBAC Authorization.

ReBAC derives access from relations between subjects and objects, and potentially relations between objects. Inheritance is explicit in the authorization model, rather than automatic for every parent-child association.

Source: OpenFGA: Modeling—Getting Started.

Substrate's .fga model makes a Worker's read permission depend on its parent global object's viewer or controller relation:

type worker
  relations
    define parent_global: [global]
    define host_node: [node]

    define can_get: viewer from parent_global or controller from parent_global

Source: cmd/ateapi/internal/authz/model.fga#L29-L34.

An Atespace can inherit its owner from its parent global object:

type atespace
  relations
    define parent_global: [global]
    define owner: [user] or owner from parent_global

An ActorTemplate can derive read permission from the viewer relation on its parent Atespace:

type actor_template
  relations
    define parent_atespace: [atespace]

    define can_get: viewer from parent_atespace

Sources: cmd/ateapi/internal/authz/model.fga#L71-L77, cmd/ateapi/internal/authz/model.fga#L101-L106.

The relation data connects global → atespace → actor_template and global → worker. Permission checks follow those modelled relations. The practical difference from ordinary Kubernetes RoleBindings is the ability to derive child access from parent relations according to an explicit model.

Components and versions

Component	Version	Repository	Scope
Agent Substrate	v0.5.0	agent-substrate/substrate	Implementation, configuration and source examples
Kubernetes	v1.37.1	kubernetes/kubernetes	Pod certificate APIs and feature gates
kind	v0.33.0	kubernetes-sigs/kind	Local example using `kindest/node:v1.37.0`

GKE availability information reflects the official documentation checked on October 20, 2026. The GKE installer does not necessarily distribute the same version as the upstream tag used here.

Note: Agent Substrate is pre-1.0 and does not guarantee backward compatibility. Unless another repository is named, source links in this article refer to agent-substrate/substrate v0.5.0.

References

Kubernetes Memory on cgroup v2: Debugging OOM Kills and Evictions

yosshi_ — Fri, 09 Oct 2026 07:20:35 +0000

A container is OOMKilled, but kubectl top never showed it reaching its limit. Or a Pod is evicted while your node dashboard still shows free memory. In both cases, the first question is which memory value the decision actually used.

This guide separates cgroup-limit OOM, node-side OOM and kubelet eviction, then shows how to inspect the evidence and review memory requests and limits. It assumes familiarity with Pod resources and basic PromQL.

TL;DR

The working set is a heuristic, not a measurement of unreclaimable memory. It subtracts inactive_file from usage but still includes reclaimable active page cache and slab.
Distinguish memcg OOM from global OOM using kernel logs. A container can be below its own limit when an ancestor cgroup reaches its limit or a node-side allocation fails. OOMKilled and SystemOOM alone cannot establish the cause.
Eviction uses the kubelet's memory.available, not Linux MemAvailable. Check the kubelet's statistics and configured thresholds.
Review requests and limits separately. Requests influence placement and victim selection. To address a cgroup limit, reduce usage or adjust the limit after checking node capacity.
Check Memory QoS configuration as well as its feature gate. In v1.37, the gate defaults to enabled, but request-based protection and memory.high remain disabled by default.

This article targets cgroup v2, Kubernetes v1.37.1 and containerd. See Components and versions for exact versions. The kind examples use a v1.37.0 node image.

Use the investigation workflow for an incident, the metric map to understand a dashboard, or requests and limits for prevention. The kind examples explore reclaim and Memory QoS in more detail.

Identify the failure before choosing a metric

Observation	Mechanism	Evidence to check
Container termination reason is `OOMKilled`	The kernel killed a process; this alone does not distinguish the OOM source	Kernel logs, container and ancestor cgroup limits, node allocation constraints
Pod is `Evicted` with `The node was low on resource: memory.`	The kubelet responded to node memory pressure	Kubelet `memory.available`, configured thresholds and eviction message
Exit code is 137 without an OOM reason	The process received SIGKILL; the exit code alone does not establish OOM	Termination details, events and kernel logs

memcg OOM means an allocation cannot fit within a cgroup limit after reclaim. Global OOM means a node-side allocation cannot be satisfied after reclaim, potentially within NUMA or cpuset constraints. A container below its own limit can be affected by either an ancestor limit or a node-side failure.

Investigating OOM and eviction

Investigation workflow

Start with the observed termination or eviction, then compare metrics from the same time. Scrapes can miss the peak; kernel logs are stronger evidence of the OOM source than a sampled usage graph.

1. When a Pod was OOMKilled

Check whether container_memory_usage_bytes is close to the limit. memcg OOM is evaluated against this value
Compare with container_memory_rss to tell whether anonymous pages are the main cause. An OOM can occur even when RSS is low
If the gap between the working set or usage and RSS is large, page cache, shmem or kernel memory may be pushing usage up
If OOMKilled occurs despite headroom under the limit, suspect global OOM. Correlate SystemOOM with kernel logs and inspect ancestor limits. A missing event does not establish memcg OOM
If you see no spike in the metrics, suspect a surge shorter than the scrape interval. See My working set is below the limit, but the container was OOMKilled for what to check

2. When a Pod was evicted

If the message is The node was low on resource: memory., node-wide memory pressure is the cause
Use the node-utilization query to check the kubelet-view node memory utilization. The node-exporter-based utilization alone is not enough to judge
Check whether a Pod with a small (or unset) memory request relative to actual usage is on the same node. Compare each Pod's container_memory_working_set_bytes (cAdvisor) against kube_pod_resource_request{resource="memory"} (kube-scheduler). Note that Pods with large requests make the scheduler keep placement density low, so they are unlikely to be the cause of pressure

3. To prevent recurrence

Set a memory request informed by workload history to avoid excessive placement density. PromQL for checking is collected in Align memory request with actual usage
For containers where memcg OOM occurs, also review the limit
Control placement with Pod anti-affinity or topology spread constraints so that Pods with large memory usage do not gather on the same node

Telling which path occurred

Using Kubernetes information

Start by narrowing down with what you can check using kubectl.

Where to look	What it tells you
`Last State` in `kubectl describe pod`, or `lastState.terminated` in `kubectl get pod -o yaml`	That an OOM kill happened (`reason: OOMKilled`). It cannot distinguish the path
Events in `kubectl describe node <node name>`	Use `SystemOOM` as a clue and verify the source in kernel logs. If there is no record, that does not make it a memcg OOM; it just means you cannot tell

There is no OOM-specific event on the Pod side. You only see things like BackOff accompanying the restart.

Check it as follows. SystemOOM is an event attached to the Node, and since Node is not a namespaced object, the event itself is recorded in the default namespace.

# 1. Find which node the OOMKilled Pod is on
kubectl get pod -n <namespace> <Pod name> -o wide

Output:

NAME       READY   STATUS    RESTARTS      AGE   IP            NODE
myapp-...  1/1     Running   5 (2m ago)    1h    10.0.0.1      node-1

# 2. Check whether that node's events contain SystemOOM
kubectl describe node node-1

Output:

...
Events:
  Type     Reason     Age   From     Message
  ----     ------     ----  ----     -------
  Warning  SystemOOM  3m    kubelet  System OOM encountered, victim process: java, pid: 12345

# To list only events
kubectl get events -n default \
  --field-selector reason=SystemOOM,involvedObject.name=node-1

If SystemOOM exists, inspect kernel messages from the same time: constraint, oom_memcg, task_memcg and the killed PID. A parsing failure can also produce this event. Match the event and kernel log to the container's termination time (lastState.terminated.finishedAt).

Not finding SystemOOM is not evidence of memcg OOM. The same state results from event TTL expiry, a kubelet restart, or a missed event.

The event retention period is set by kube-apiserver's --event-ttl, and the default is 1 hour (options.go). So an old SystemOOM may no longer be available through kubectl. If you forward events to an external logging platform, check there.

On the node, inspect the kernel log:

sudo journalctl -k -b -o short-iso \
  | rg 'oom-kill:|Out of memory:|Memory cgroup out of memory:|Killed process'

CONSTRAINT_MEMCG and oom_memcg identify the limiting cgroup; task_memcg identifies the victim's cgroup. If these differ, inspect the Pod and ancestor limits. CONSTRAINT_CPUSET and CONSTRAINT_MEMORY_POLICY call for checking NUMA and cpuset restrictions. Align timestamps with the container's finishedAt.

If logs are unavailable, use the following as supporting evidence rather than proof:

Whether container_memory_usage_bytes was close to the limit at that time (if so, memcg OOM is likely)
Whether the kubelet-view node memory utilization was high at that time (if so, global OOM is likely)

Why OOMKilled cannot distinguish the paths

When a task exits with code 137, the container runtime (containerd) checks oom_kill in the cgroup's memory.events, and if it is 1 or more, it updates the termination reason to OOMKilled (events.go). The check is oomMetricsEventOccurred() in the same file.

    switch v := taskMetricsAny.(type) {
    case *cg1.Metrics:
        return v.GetMemoryOomControl().GetOomKill() > 0, nil
    case *cg2.Metrics:
        return v.GetMemoryEvents().GetOomKill() > 0, nil

The definition of oom_kill in memory.events is "The number of processes belonging to this cgroup killed by any kind of OOM killer". So a process chosen and killed by the kernel in a global OOM is counted too, and both paths produce OOMKilled.

How the kubelet parser produces SystemOOM

When SystemOOM is recorded

SystemOOM is an event the kubelet records on the Node object. The kubelet parses OOM messages in the kernel log and records it only when the cgroup where the OOM happened is judged to be the root (/) (oom_watcher_linux.go).

        for event := range outStream {
            // Count every OOM kill per container to back the
            // container_oom_events_total metric.
            recordOOMKill(event.ContainerName)

            if event.VictimContainerName == recordEventContainerName {
                ...
                ow.recorder.WithLogger(logger).Eventf(ref, v1.EventTypeWarning, systemOOMEvent, "%s", eventMsg)
            }
        }

The usual association with global OOM comes from the combination of the kernel log format and the parser. For an OOM caused by a memcg limit, the kernel prints oom_memcg=<cgroup path>, whereas for a global OOM it prints ,global_oom instead (mem_cgroup_print_oom_context() in mm/memcontrol.c).

    if (memcg) {
        pr_cont(",oom_memcg=");
        pr_cont_cgroup_path(memcg->css.cgroup);
    } else
        pr_cont(",global_oom");

The modern-format regular expression in the kubelet's forked parser matches the format containing oom_memcg= (oomparser_linux.go).

    containerRegexp = regexp.MustCompile(`oom-kill:constraint=(.*),nodemask=(.*),cpuset=(.*),mems_allowed=(.*),oom_memcg=(.*),task_memcg=(.*),task=(.*),pid=(.*),uid=(.*)`)

So no cgroup can be extracted from a global OOM message, and the initial value / of OomInstance is left as is (StreamOoms in oomparser_linux.go).

            oom := &OomInstance{
                ContainerName:       "/",
                VictimContainerName: "/",
                TimeOfDeath:         msg.Timestamp,
            }

In other words, the presence of SystemOOM is not "the result of directly determining the path" but "the result of whether the kernel log could be parsed". Still, in practice it is enough to understand that a recorded event is a reason to inspect node-side OOM, while neither presence nor absence establishes the source.

How it appears in metrics

Metrics do not distinguish the path with a label. The kernel's __oom_kill_process() records the same event regardless of the path.

Metric	Emitted by	memcg OOM	global OOM
`node_vmstat_oom_kill`	node-exporter (`oom_kill` in `/proc/vmstat`)	Increases	Increases
`container_oom_events_total`	kubelet (result of kernel log parsing)	The series of the container the killed process belonged to increases	The `id="/"` series increases, not a container series
`oom_kill` in cgroup v2 `memory.events`	(a cgroup file, not a metric; containerd reads it)	Increases	Increases

Only container_oom_events_total behaves differently because, as described earlier, the parser cannot extract a cgroup in a global OOM and treats it as the root (/). This is only a result of whether the log could be parsed, so do not use it to determine the path; confirm the source from constraint and oom_memcg in the kernel log.

Note: Although container_oom_events_total has container_ in its name, in v1.37 it is counted by the kubelet, not cAdvisor (recordOOMKill() in oom_counter.go). The meaning of the value does not change, but you have to look elsewhere when reading the code.

What to look at after narrowing down

If you judge it to be a global OOM, inspect node-side memory pressure and any NUMA or cpuset constraints. Compare the memory requests and actual usage of the Pods placed on the same node, and check whether any Pod has too small a request
If you judge it to be a memcg OOM, review that container's memory limit and its actual usage. If the container seems to have headroom under its limit, it may have reached the limit of the Pod as a whole or of /kubepods.slice
If you cannot decide, keep both possibilities open and check both the container's limit and usage and the node's memory utilization

Kernel OOM behavior and victim selection

The two OOM kill paths

There are two paths by which a container's process gets OOM killed. Both are SIGKILLs from the kernel's OOM killer, but the conditions under which they occur and the scope they affect differ. As defined in the terminology notes, memcg OOM enforces a configured limit and global OOM protects the node, so their purposes differ at the root.

Path	Condition	What gets killed	Node event
memcg OOM	`memory.max` exceeded for a container, Pod or `/kubepods.slice`	Processes inside the cgroup that hit its limit	Not recorded
global OOM	A node-side allocation cannot be satisfied after reclaim, possibly within NUMA or cpuset constraints	Chosen from eligible processes under the allocation constraints	`SystemOOM` is recorded when the kubelet parses the log and treats the triggering cgroup as `/`

Even if your container is under its limit, it can be OOM killed because of another Pod on the same node. That is global OOM.

SystemOOM is a kubelet-generated clue to investigate, not a definitive classification of the OOM source. Events expire according to the API server's --event-ttl (one hour by default), so a missing event does not rule out global OOM.

OOM kill from exceeding a cgroup limit (memcg OOM)

resources.limits.memory is set as the container cgroup's memory.max. When a cgroup's memory.current reaches memory.max, this is what happens. As described above, limits are set not only on containers but also on Pods and /kubepods.slice, and the same flow applies whichever level reaches its limit.

The kernel tries to reclaim memory inside that cgroup. Page cache is basically reclaimed here.
If usage still does not fit under memory.max after reclaim, the cgroup's OOM killer SIGKILLs processes in that cgroup.
With cgroup v2, the kubelet defaults to singleProcessOOMKill: false and sets memory.oom.group=1 for the container. A memcg OOM therefore normally kills the container's processes together. With singleProcessOOMKill: true, a child process can be killed while the container keeps running. Container restarts follow restartPolicy. See the implementation.

The implementation is try_charge_memcg() in mm/memcontrol.c. It is a goto loop returning to the retry: label, and proceeds in this order.

Step	Location	What happens
1. Try to charge	L2177-L2178	If `page_counter_try_charge()` succeeds, the allocation completes and it returns
2. Reclaim	L2211-L2216	On failure it reclaims with `try_to_free_mem_cgroup_pages()`, and if `mem_cgroup_margin()` satisfies the request, goes back to 1
3. Count the retry	L2244-L2245	If reclaim is not enough, it decrements `nr_retries` and goes back to 1. The initial value is `MAX_RECLAIM_RETRIES` (= `16`; mm/internal.h)
4. Invoke the OOM killer	L2259-L2264	Only after the retries are used up does it call `mem_cgroup_oom()`

MAX_RECLAIM_RETRIES is a retry budget, not a guarantee of exactly 16 reclaim attempts. Allocation flags and reclaim progress change the control flow. The check in step 2 looks like this in code.

    nr_reclaimed = try_to_free_mem_cgroup_pages(mem_over_limit, nr_pages,
                            gfp_mask, reclaim_options, NULL);
    psi_memstall_leave(&pflags);

    if (mem_cgroup_margin(mem_over_limit) >= nr_pages)
        goto retry;

The kernel documentation for memory.max says the same.

Memory usage hard limit. ... If a cgroup's memory usage reaches this limit and can't be reduced, the OOM killer is invoked in the cgroup.

Warning: memory.current reaching memory.max does not by itself cause an OOM. A cgroup sitting at its limit with page cache is a normal state. For the same reason, the working set reaching memory.max is not an OOM condition either. Only inactive_file is subtracted from the working set, and reclaimable memory remains in the rest:

active_file: demoted to inactive under pressure and then reclaimed

slab_reclaimable (dentry / inode caches): in memcg reclaim too, shrink_slab() is called after shrink_lruvec() (shrink_node_memcgs() in mm/vmscan.c)

So there is no threshold of "OOM once either value reaches the limit". The actual condition is when the charge fails and reclaim cannot make room. What cannot be reclaimed is mainly anonymous pages (without swap) and shmem, so that is where to look when narrowing down.

That said, what the kernel compares against memory.max in a memcg OOM is the whole of memory.current, and it is not decided by any single metric. memory.current includes, besides anonymous pages, kernel memory, shmem (tmpfs and shared memory), and page cache that cannot be reclaimed, or cannot be reclaimed right away.

container_memory_rss is an indicator for determining whether anonymous pages are the main cause. If cgroup RSS is close to the limit, anonymous pages are a likely major contributor, but RSS being below the limit is no evidence that an OOM will not occur. When the gap between RSS and the working set is large, check container_memory_usage_bytes (the value of memory.current) together with the breakdown in memory.stat on the node: anon, file, shmem, slab and so on.

OOM kill from node memory exhaustion (global OOM)

When a node-side allocation cannot be satisfied after reclaim, the kernel evaluates eligible processes using oom_badness(). cpusets, NUMA policies and oom_score_adj=-1000 can restrict the candidates. A container need not exceed its own limit to be selected.

There is no guaranteed ordering between kubelet eviction and the kernel's OOM killer. The kubelet evicts Pods to protect the node, and its selection puts "Pods whose usage exceeds their memory request" first (rankMemoryPressure() in helpers.go).

// rankMemoryPressure orders the input pods for eviction in response to memory pressure.
// It ranks by whether or not the pod's usage exceeds its requests, then by priority, and
// finally by memory usage above requests.
func rankMemoryPressure(pods []*v1.Pod, stats statsFunc) {
    orderedBy(exceedMemoryRequests(stats), priority, memory(stats)).Sort(pods)
}

A Pod with a zero memory request and positive usage belongs to the "exceeds request" group. Within that group, priority and then usage above request determine the order, so it is not necessarily evicted first. If eviction acts in time, the result is Evicted, not OOMKilled.

Global OOM happens first when eviction cannot reclaim memory as fast as memory grows. There are several reasons it cannot keep up.

Factor	Value	Effect
Evaluation interval	10 seconds (`evictionMonitoringPeriod`)	Detection can be up to 10 seconds late
Pods evicted per evaluation	Only one ("we kill at most a single pod during each eviction interval" in eviction_manager.go)	At most one Pod is evicted per synchronization; notifications and cleanup waits mean this is not a fixed rate
Waiting for cleanup	Up to 30 seconds (`podCleanupTimeout`)	The next evaluation may be delayed further
Default threshold	`memory.available<100Mi`	Extremely thin on large nodes
How `memory.available` is computed	From the root cgroup's `anon + file`	Contains no kernel memory, so it overestimates free memory

The official Kubernetes documentation also says (Node out of memory behavior):

If the node experiences an out of memory (OOM) event prior to the kubelet being able to reclaim memory, the node depends on the oom_killer to respond.

It is more likely when applications that allocate a lot of memory right at startup, or in response to a sudden surge in requests, share a node.

Warning: Eviction protects the node; it does not protect individual Pods from OOM. Containers without limits have no memory.max backstop at all, so eviction is merely the last line of defense.

Moreover, rankMemoryPressure() looks at Pod priority before usage size. If the offending Pod has high priority, low-priority Pods with small usage get evicted first, and global OOM can occur without the pressure being resolved. The proper countermeasure is to set limits and requests rather than relying on eviction.

Which process gets killed

The kernel evaluates eligible processes using resident pages, swap, page tables and oom_score_adj; the adjustment alone does not establish a fixed kill order. The kubelet sets oom_score_adj (a bonus added to oom_score) according to the container's QoS class, so the QoS class and the memory request setting directly become how likely a container is to be killed. You can check the QoS class with kubectl get pod <Pod name> -o jsonpath='{.status.qosClass}', and how it is determined is described in Pod Quality of Service Classes.

Condition	`oom_score_adj`	Likelihood of being killed
`system-node-critical` (evaluated before the QoS class)	`-997`	Least likely
Guaranteed	`-997`	Least likely
Burstable	`1000 - 1000 × memory request / node memory capacity` (clamped to 3–999)	The smaller the request, the more likely
BestEffort	`1000`	Most likely

The implementation is GetContainerOOMScoreAdjust() in Kubernetes' policy.go. system-node-critical is evaluated before the QoS class decision.

    if types.IsNodeCriticalPod(pod) {
        // Only node critical pod should be the last to get killed.
        return guaranteedOOMScoreAdj
    }

The formula above is the container-level baseline. Pod-level requests and restartable init containers can introduce adjustments. An omitted memory request can also default from the limit, so inspect the applied Pod spec.

The Burstable clamp is implemented with a lower bound of 1000 + guaranteedOOMScoreAdj (= 3) and an upper bound of besteffortOOMScoreAdj - 1 (= 999). A Pod whose request is set smaller than its actual usage is more likely to be targeted in both eviction and global OOM.

Warning: PriorityClass works differently for eviction and for global OOM. It is a misreading to think that raising priority protects you from both.

PriorityClass Eviction global OOM (oom_score_adj)

User defined (up to 1000000000) The higher the value, the later it is evicted Determined by QoS class and request

system-cluster-critical Exempt from node-pressure eviction Determined by QoS class and request (not protected)

system-node-critical Exempt from node-pressure eviction Fixed at -997

static / mirror Pod Exempt from node-pressure eviction Determined by QoS class and request

The three exempt rows identify critical Pods for which IsCriticalPod() is true; the kubelet checks static and mirror Pods as well as system-critical priority classes.

What oom_score_adj protects is only system-node-critical, for which IsNodeCriticalPod() is true. If a system-cluster-critical Pod is BestEffort, its oom_score_adj is 1000, and it is more likely to be selected in a global OOM. "I raised the priority but it was still OOMKilled" comes from this path.

Warning: Neither an OOM kill nor node-pressure eviction directly deletes the Pod API object. What happens next depends on restartPolicy.

Always or OnFailure: the container is recreated on the same node. The Pod stays on the same node, so if the cause is not resolved, OOM kills will happen again

Never: the container is not recreated and the Pod remains Failed. If a controller such as a Job creates a successor Pod, it may be placed on a different node

If OOMKilled repeats even though there is headroom under the limit, suspect this path.

Note: Because the OOM kill frees memory, the pressure may already be gone when the kubelet next evaluates free memory. In that case the node may get no MemoryPressure condition and no node.kubernetes.io/memory-pressure:NoSchedule taint, and scheduling of new Pods does not stop. Conversely, if the pressure continues, a later evaluation sets the condition and taint, and eviction also occurs. Note that it is decided by memory.available at evaluation time, not by whether an OOM kill happened.

PriorityClass	Eviction	global OOM (`oom_score_adj`)
User defined (up to `1000000000`)	The higher the value, the later it is evicted	Determined by QoS class and request
`system-cluster-critical`	Exempt from node-pressure eviction	Determined by QoS class and request (not protected)
`system-node-critical`	Exempt from node-pressure eviction	Fixed at `-997`
static / mirror Pod	Exempt from node-pressure eviction	Determined by QoS class and request

Checking with PSI (Pressure Stall Information)

This is an auxiliary indicator for the case where things are slow although usage has not reached the limit. It shows how long processes were actually stalled waiting for memory reclaim.

How to read PSI and PromQL

You can obtain the time a container was stalled waiting for memory allocation as PSI. Even when usage has not reached the limit, you can detect being made to wait by memory reclaim.

Metric	Emitted by	Description
`container_pressure_memory_waiting_seconds_total`	cAdvisor (embedded in kubelet)	Time some processes were waiting for memory (`total` of `some` in `memory.pressure`)
`container_pressure_memory_stalled_seconds_total`	cAdvisor (embedded in kubelet)	Time all processes were stalled waiting for memory (`total` of `full` in `memory.pressure`)

Note: These metrics are controlled by the KubeletPSI feature gate, which went GA in v1.36 and is enabled by default (it cannot be disabled). The kernel must support PSI, though, and the kubelet decides this by whether cpu.pressure exists at the cgroup root (cadvisor_linux.go). If the metrics do not appear, check whether the kernel was built with CONFIG_PSI=y and whether psi=0 is set as a kernel parameter.

memory.pressure also has avg10 / avg60 / avg300, "the average over the last N seconds (%)", but only the cumulative value (total) is exposed as a Prometheus metric. So use rate() to see a ratio.

The unit is seconds, so the result of rate() is "how many seconds of waiting per second", a ratio from 0 to 1. Multiply by 100 to get a percentage directly.

# Per Pod, the share of time (%) all processes were stalled waiting for memory
100 * max by (namespace, pod) (
  rate(container_pressure_memory_stalled_seconds_total{container="", image="", pod!=""}[5m])
)

# Per Pod, the share of time (%) some processes were waiting for memory
100 * max by (namespace, pod) (
  rate(container_pressure_memory_waiting_seconds_total{container="", image="", pod!=""}[5m])
)

When there is plenty of memory, neither increases, so rate() stays at 0. Under pressure, waiting (some) rises first, and if it gets worse stalled (full) follows. waiting counts even when only some processes waited, so to judge severity, use stalled. It is the time all processes in the container were stopped, so if it shows a value, throughput is directly affected.

To see memory pressure on the whole node, use the root cgroup series.

# Share of time (%) the whole node was completely stalled waiting for memory
100 * max by (node) (
  rate(container_pressure_memory_stalled_seconds_total{id="/"}[5m])
  * on(instance) group_left(node) kubelet_node_name
)

Containers that are struggling although they are below their limit can be found by combining PSI with utilization. container_spec_memory_limit_bytes is the container limit exposed by cAdvisor.

# Containers whose usage is under 80 % of the limit but that are stalled waiting for memory
(
  100 * rate(container_pressure_memory_stalled_seconds_total{container!="", image!=""}[5m]) > 1
)
and
(
  container_memory_usage_bytes{container!="", image!=""}
  / container_spec_memory_limit_bytes{container!="", image!=""} < 0.8
)

and keeps only series whose label sets match exactly on both sides. A container with no limit has container_spec_memory_limit_bytes of 0, so the division gives +Inf, which does not satisfy < 0.8, and it is excluded automatically.

Here is an example alert. It catches stalled that keeps occurring.

- alert: ContainerMemoryStalled
  # Fires when the 5-minute average of fully stalled time is 5 % or more for 10 minutes
  expr: |
    100 * max by (namespace, pod) (
      rate(container_pressure_memory_stalled_seconds_total{container="", image="", pod!=""}[5m])
    ) > 5
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "A Pod is stalled waiting for memory allocation (add the namespace and pod labels with your own template)"
    description: "It may be made to wait by reclaim even though it has not reached its memory limit. Check the gap between usage and RSS and the amount of page cache."

The right threshold of 5 % depends on the environment. First look at the distribution of values without an alert to understand the normal level, then decide.

For details on PSI see Understand Pressure Stall Information (PSI) Metrics.

Quick reference: contradictions explained and a metric map

The working set, cgroup usage, process RSS and node availability answer different questions. Use this map to identify the value behind the dashboard before comparing it with a limit or threshold. For a non-root cgroup, working set is max(memory.current - inactive_file, 0). It still includes some reclaimable memory.

Scope	Metric	Emitted by	Backing value in cgroup v2 / `/proc`	Page cache handling	Main use
Container	`container_memory_working_set_bytes`	cAdvisor (embedded in kubelet)	`max(memory.current − inactive_file, 0)`	Active pages count as in use	`kubectl top pod`, usage visualization
Container	`container_memory_rss`	cAdvisor (embedded in kubelet)	`anon` in `memory.stat`	Not included (anonymous pages only)	Understanding what the app allocated
Container	`container_memory_usage_bytes`	cAdvisor (embedded in kubelet)	`memory.current`	Everything included	What memcg OOM is evaluated against
Node	`container_memory_working_set_bytes{id="/"}`	cAdvisor (embedded in kubelet)	`anon + file − inactive_file` in the root cgroup's `memory.stat` (see below)	Active pages count as in use	Eviction decisions, `kubectl top node`
Node	`node_memory_MemAvailable_bytes`	node-exporter	`MemAvailable` in `/proc/meminfo`	Reclaimable pages count as free	Visualizing node memory utilization
Node	`kube_node_status_capacity{resource="memory"}`	kube-state-metrics	(`status.capacity` of the Node object)	(capacity, not usage)	Denominator when converting to utilization

Looking at the same node, utilization computed from node_memory_MemAvailable_bytes does not match the utilization the kubelet uses for eviction decisions. With workloads that do a lot of file I/O or logging, the difference can reach several GiB.

Different emitters also mean different labels on the metrics. cAdvisor metrics carry no label identifying the node, so when you compute per-node utilization you need to align labels with kubelet_node_name, as shown later.

Kubernetes memory usage and cgroup v2

Why the numbers differ

A process RSS measurement includes its resident mappings, including shared and file-backed pages. Cgroup accounting charges memory to a group, while working set subtracts one category of file cache from usage. An application's heap metric covers a different subset again.

Reclaimability is a separate question: clean file cache can often be reclaimed, dirty pages may need writeback, and anonymous memory normally needs swap or explicit release by the process. Calling free does not necessarily return allocator-held memory to the OS.

These differences explain why heap size, process RSS, cgroup RSS and working set need not match. The following sections connect those values to cgroup v2 accounting.

Memory breakdown in cgroup v2

Container memory usage is managed in a per-container cgroup. In cgroup v2, memory.current is the current usage and memory.stat is its breakdown.

# Example values in a container's cgroup (checked on the node)
cat /sys/fs/cgroup/<container cgroup>/memory.current

Output:

5368709120

cat /sys/fs/cgroup/<container cgroup>/memory.stat

Output:

anon 2147483648           # anonymous pages (heap, stack: memory not backed by a file)
file 3087007744           # page cache (including tmpfs and shared memory)
shmem 268435456           # the part of file that is tmpfs or shared memory
...
inactive_anon 134217728    # anon LRU: not accessed for a while
active_anon 2013265920     # anon LRU: recently accessed
inactive_file 1073741824   # file LRU: not accessed for a while
active_file 1744830464     # file LRU: recently accessed
unevictable 268435456      # pages that cannot be reclaimed (here, the tmpfs part)
...

The following relationships hold in this example. Note that inactive_file + active_file does not equal file; it equals file - shmem. The next section explains why.

anon + file + kernel memory = 5368709120 (memory.current)
inactive_file + active_file = 2818572288 = file - shmem
inactive_anon + active_anon = 2147483648 = anon        (shmem is not in here)
unevictable                 =  268435456 = shmem       (tmpfs without swap)

The main fields of memory.stat are:

`anon` (anonymous pages)

Memory not backed by a file. Regions the application allocated with malloc and similar belong here.
Unlike page cache, the kernel cannot reclaim it independently of the process. With swap it can write pages out and free up space, but on nodes that do not use swap it is memory the kernel cannot reclaim under pressure. When swap is unavailable, the kernel sets scan_balance = SCAN_FILE and excludes the anon LRU from scanning (get_scan_count() in mm/vmscan.c).
On the other hand, this value drops if the process itself returns memory to the OS with munmap or madvise(MADV_DONTNEED). Note that calling free does not reduce it unless the allocator or language runtime returns the memory to the OS.

`file` (page cache)

Contents of files held in memory by the kernel during reads and writes. The application did not allocate it explicitly.
When memory runs short the kernel discards (reclaims) it automatically, so it is basically reclaimable memory.
Cache of files on disk is split into active_file (recently accessed) and inactive_file (not accessed for a while). Reclaim takes inactive_file first.
active_file + inactive_file does not equal file. These are totals of the LRU lists, and as described below shmem is on the anon LRU rather than the file LRU. Besides shmem, pages that fall off the LRU, such as mlocked pages, also show up as a difference.
It includes not only the cache of files on disk but also shmem (tmpfs and shared memory), because Linux treats these as page cache too. shmem is written out to swap rather than to a file, so on nodes without swap it cannot be reclaimed. The breakdown is visible as shmem in memory.stat.

Warning: shmem is counted in file, but it is not on the file LRU. The kernel classifies swap-backed pages as anon, and tmpfs pages are among them. folio_is_file_lru() in mm_inline.h is defined as "0 if @folio is a normal anonymous folio, a tmpfs folio or otherwise ram or swap backed folio".

The counters they are charged to are different, though. A tmpfs page is added to both NR_FILE_PAGES (= file) and NR_SHMEM (= shmem) (mm/shmem.c):
  __lruvec_stat_mod_folio(folio, NR_FILE_PAGES, nr);
  __lruvec_stat_mod_folio(folio, NR_SHMEM, nr);
Furthermore, for a Kubernetes emptyDir with medium: Memory, the kubelet mounts tmpfs with the noswap option when the node supports it (generateTmpfsMountOptions() in empty_dir.go). The kernel makes the inode of a noswap tmpfs unevictable (shmem_get_inode() in mm/shmem.c), so these pages sit on the unevictable LRU, not the anon LRU.
  if (sbinfo->noswap)
      mapping_set_unevictable(inode->i_mapping);
So shmem is accounted as follows:

mount LRU Where it shows up in memory.stat

emptyDir with medium: Memory (noswap) unevictable unevictable

Any other tmpfs / shared memory anon LRU inactive_anon / active_anon

In neither case does it enter inactive_file / active_file. That matters in two ways later:

inactive_file + active_file generally corresponds to file - shmem; unevictable pages and asynchronous statistics can cause differences

Only inactive_file is subtracted from the working set, so shmem stays in the working set in full

With workloads that write a lot of logs or read and write large files, the page cache can grow to several GiB even though the application did not intend that.

Note: memory.current includes memory the kernel itself uses (slab, socket buffers and so on) in addition to anon and file. So adding anon and file alone does not give memory.current.

How cgroup v2 memory controls map to Pod settings

cgroup v2 memory control interface

cgroup v2 offers four levels of control over memory usage. memory.min and memory.low are protections ("do not reclaim up to here"), and memory.high and memory.max are limits ("do not allow use beyond here").

File	Type	Behavior
`memory.min`	Protection (hard)	Protects memory within the effective boundary determined by ancestors and usage. It does not preallocate physical memory
`memory.low`	Protection (soft)	Best-effort avoidance of reclaim up to this value. It is reclaimed if nothing else can be
`memory.high`	Limit (soft)	Exceeding it applies reclaim pressure and throttling. Crossing `memory.high` does not itself invoke the OOM killer; `memory.max` and global OOM still apply
`memory.max`	Limit (hard)	Exceeding it triggers reclaim, and if usage still does not fit, the cgroup's OOM killer runs

Other files that report state:

memory.current: current usage (what container_memory_usage_bytes is backed by)
memory.stat: usage breakdown
memory.events: counts of low, high, max, oom, oom_kill and oom_group_kill
memory.swap.max: swap limit
memory.pressure: PSI for waiting on memory allocation

For exact definitions of each file see the kernel's Memory Interface Files, and for how Kubernetes treats cgroup v2 see About cgroup v2.

How Pod settings are reflected

Pod settings are reflected into cgroup v2 as follows.

Pod setting	Target in cgroup v2
`resources.limits.memory`	`memory.max`
`resources.requests.memory`	Not reflected (with the default kubelet configuration in v1.37; you can reflect it by changing settings → KEP-2570)
`resources.limits.cpu`	`cpu.max`
`resources.requests.cpu`	`cpu.weight`

cgroup v2 is a directory tree. Its top is the root cgroup (/sys/fs/cgroup), where any process that is in no child cgroup belongs, so it contains not only Pods but every process on the node, such as the kubelet and containerd. Kubernetes creates the hierarchy below it: "all Pods on the node → Pod → container".

# With the systemd cgroup driver (the kubeadm default; names differ with the cgroupfs driver, see below)
/sys/fs/cgroup                                  ← root cgroup (id="/")
│                                                 contains every process on the node
│                                                 the value used for eviction decisions
│
├── system.slice/                               kubelet, containerd, sshd, etc.
│   ├── kubelet.service                           (processes that are not Pods)
│   └── containerd.service
│
└── kubepods.slice/                             ← all Pods on the node
    │                                             (id="/kubepods.slice")
    ├── kubepods-pod<UID>.slice/                ← Pod (Guaranteed)
    │   ├── cri-containerd-<ID>.scope           ← container
    │   └── cri-containerd-<pause ID>.scope     ← pause container
    │
    ├── kubepods-burstable.slice/                 (intermediate slice per QoS class)
    │   └── kubepods-burstable-pod<UID>.slice/  ← Pod (Burstable)
    │       └── cri-containerd-<ID>.scope       ← container
    │
    └── kubepods-besteffort.slice/
        └── kubepods-besteffort-pod<UID>.slice/ ← Pod (BestEffort)
            └── cri-containerd-<ID>.scope       ← container

When filtering metrics, the root cgroup is id="/", all Pods on the node are id="/kubepods.slice", a Pod is container="", image="", pod!="", and a container is container!="", image!="". What the kubelet uses for eviction decisions is the working set of the topmost root cgroup (see below). That is a different level from memcg OOM caused by a container or Pod exceeding its limit.

Memory limits are set not only on containers but also on higher levels.

Level	cgroup	Memory limit
Whole node (root cgroup)	`/`	Not set (the root has no `memory.max` → see below)
All Pods on the node	`/kubepods.slice`	`Capacity - kube-reserved - system-reserved` when Pod allocatable enforcement is enabled
Pod	Differs by QoS class (see below)	The Pod's effective limit, including Pod-level `spec.resources.limits.memory` when configured
Container	`cri-containerd-<ID>.scope` under the Pod's cgroup	That container's `limits.memory`

The examples here use container-level resources and omit Pod-level resources. Without a Pod-level limit, the required container limits must be present for a finite Pod limit to be set. A Pod's effective limit is not simply the sum of its regular containers' limits.memory. If there are init containers, their maximum is also considered, and spec.overhead is added if set. For the calculation see Resource sharing within containers and Pod Overhead.

So even if a container has not reached its own limit, an OOM kill can occur when the Pod as a whole or /kubepods.slice hits its limit.

Note: The paths above assume the kubelet uses the systemd cgroup driver. With the cgroupfs driver, paths look like /kubepods/burstable/pod<UID>/<container ID>, with no .slice or .scope and no cri-containerd- prefix on the container.

The kubelet's own default is cgroupfs, but kubeadm sets systemd when the value is empty (defaults.go, kubelet.go). In v1.37, a supported runtime's CRI RuntimeConfig response takes precedence through driver auto-detection. Check the effective driver with:
$ kubectl get --raw "/api/v1/nodes/<node name>/proxy/configz" \
  | jq '.kubeletconfig.cgroupDriver'
"systemd"
The id label of cAdvisor metrics also changes with the cgroup driver, so when filtering in PromQL, check the value in your own cluster with this query:
group by (id) (container_memory_working_set_bytes{id=~"/kubepods|/kubepods.slice"})
A Pod's cgroup path depends on its QoS class. As the tree above shows, Burstable and BestEffort go through an intermediate slice per QoS class.

In the <UID> part, systemd escaping replaces the hyphens of the Pod UID with underscores (example: 6983351e-4ede-... → pod6983351e_4ede_...). The UID of a static Pod (mirror Pod) is a hash with no hyphens, so it looks like pod1ff70b81885169da4e9e32c9530b77c9.slice.

The QoS class and UID of a Pod can be read from the Pod resource, and once you have those two you can assemble the cgroup path.
$ kubectl get pod -n <namespace> <Pod name> -o jsonpath='{.status.qosClass}{"\n"}{.metadata.uid}{"\n"}'
Burstable
6983351e-4ede-4611-8bab-b986614af18d

# To list them
$ kubectl get pods -n <namespace> -o custom-columns='NAME:.metadata.name,QOS:.status.qosClass'

Container metrics

The cAdvisor built into the kubelet exposes the cgroup values above as Prometheus metrics. They are served from the kubelet's /metrics/cadvisor endpoint, and you do not need to run a separate cAdvisor Pod or DaemonSet.

Metric	Emitted by	Backing value in cgroup v2	Description
`container_memory_usage_bytes`	cAdvisor (embedded in kubelet)	`memory.current`	Usage including anonymous pages, page cache and kernel memory. The value memcg OOM is evaluated against
`container_memory_working_set_bytes`	cAdvisor (embedded in kubelet)	`memory.current - inactive_file`	Usage minus `inactive_file`, clamped to zero. A heuristic that still includes some reclaimable memory
`container_memory_rss`	cAdvisor (embedded in kubelet)	`anon` in `memory.stat`	Amount of anonymous pages. Unlike true RSS, it does not include `mmap`ed file pages (see below)
`container_memory_cache`	cAdvisor (embedded in kubelet)	`file` in `memory.stat`	Page cache size. Includes `shmem` (tmpfs and shared memory)
`container_memory_total_active_file_bytes`	cAdvisor (embedded in kubelet)	`active_file` in `memory.stat`	Active page cache
`container_memory_total_inactive_file_bytes`	cAdvisor (embedded in kubelet)	`inactive_file` in `memory.stat`	Inactive page cache
`container_memory_mapped_file`	cAdvisor (embedded in kubelet)	`file_mapped` in `memory.stat`	Amount of `mmap`ed files
`container_pressure_memory_waiting_seconds_total`	cAdvisor (embedded in kubelet)	`total` of `some` in `memory.pressure`	Time some processes were waiting for memory (see below)
`container_pressure_memory_stalled_seconds_total`	cAdvisor (embedded in kubelet)	`total` of `full` in `memory.pressure`	Time all processes were stalled waiting for memory (see below)
`container_oom_events_total`	kubelet (v1.37; the name starts with `container_` but it is not from cAdvisor)	(not a cgroup value)	Counted by parsing `oom-kill:` lines in the kernel log. The live series disappears when the container cgroup is removed (whether or not it is recreated)

Note: The kubelet has endpoints such as /metrics, /metrics/cadvisor and /metrics/resource (server.go). The last two relate to container and node usage. /metrics/cadvisor exposes the container_* metrics above, while /metrics/resource exposes only CPU, memory and swap usage (resource_metrics.go). The latter has container_memory_working_set_bytes and also per-Pod and per-node series (pod_memory_working_set_bytes, node_memory_working_set_bytes) (see podMemoryUsageDesc and nodeMemoryUsageDesc in resource_metrics.go). When to use which for per-Pod usage is covered in Align memory request with actual usage.

The important thing here is the definition of the working set.

container_memory_working_set_bytes = max(
    container_memory_usage_bytes - container_memory_total_inactive_file_bytes,
    0,
)

In other words, only inactive_file is subtracted from usage, with the result clamped to zero; active page cache remains in the working set. For the implementation see cAdvisor's handler.go.

    workingSet := ret.Memory.Usage
    if v, ok := s.MemoryStats.Stats[inactiveFileKeyName]; ok {
        ret.Memory.TotalInactiveFile = v
        if workingSet < v {
            workingSet = 0
        } else {
            workingSet -= v
        }
    }
    ret.Memory.WorkingSet = workingSet

As a breakdown, it is roughly the sum of the following. RSS (container_memory_rss) is exactly anon in memory.stat under cgroup v2, that is, the amount of anonymous pages, so it is included in the working set.

container_memory_working_set_bytes
  ≒ container_memory_rss                        (anonymous pages; shmem not included)
  + shmem                                       (tmpfs and shared memory; stays in full)
  + container_memory_total_active_file_bytes    (active page cache)
  + unreclaimable pages, kernel memory, etc.    (unevictable, slab, socket buffers, ...)

shmem is listed as a separate term because neither container_memory_rss (= anon = NR_ANON_MAPPED) nor active_file includes shmem. As described above, shmem is not on the file LRU, so it never appears in active_file / inactive_file, and since only inactive_file is subtracted from the working set, it remains in full. No container_* metric is exposed for shmem, so this term can only be checked from memory.stat on the node.

The approximation reflects overlapping accounting categories and LRU classifications in memory.stat. cAdvisor also exposes container_memory_kernel_usage, but this alone cannot reconstruct the full breakdown. Read memory.stat directly for more detail.

Caution: RSS in container_memory_rss stands for Resident Set Size, but its scope differs from the real meaning. True RSS means "the pages a process holds in physical memory" and includes mmaped file pages.

But memory.stat in cgroup v2 has no rss field, and cAdvisor exposes anon as container_memory_rss. The kernel documentation defines anon as "Amount of memory used in anonymous mappings such as brk(), sbrk(), and mmap(MAP_ANONYMOUS)", which does not include file mappings. So it does not match the RSS column of ps or VmRSS in /proc/[pid]/status.

Warning: A large container_memory_working_set_bytes does not necessarily mean the application needs that much memory. It may just be accumulated page cache. To see how much memory the application has actually allocated, compare against container_memory_rss.

Node metrics

There are two families of metrics representing whole-node memory usage, with different emitters.

From cAdvisor (the kubelet's view)

cAdvisor exposes memory usage not only for containers but also for the root cgroup (id="/", the topmost cgroup that contains every process on the node, as described above) and for the cgroup of all Pods. The id of the all-Pods cgroup depends on the cgroup driver, so the queries below match both values with =~.

container_memory_working_set_bytes{id="/"}: working set of the whole node (Pods + other processes). But the root cgroup has no memory.current, so the backing value differs from containers (see below)
container_memory_working_set_bytes{id=~"/kubepods|/kubepods.slice"}: working set of all Pods on the node
container_memory_total_active_file_bytes{id="/"}: active page cache of the whole node

Both the MEMORY(bytes) column of kubectl top node and the kubelet's eviction decision described below use this working set.

From node-exporter (`/proc/meminfo` as is)

node-exporter turns the contents of /proc/meminfo directly into metrics.

node_memory_MemTotal_bytes: total physical memory on the node
node_memory_MemAvailable_bytes: the kernel's estimate of "how much can be allocated when new memory is requested"
node_memory_Cached_bytes: page cache size. As the kernel documentation says, "In-memory cache for files read from the disk (the pagecache) as well as tmpfs & shmem", it includes tmpfs and shared memory. The breakdown is in node_memory_Shmem_bytes

node_memory_MemAvailable_bytes counts reclaimable page cache as free. Computing node memory utilization as 1 - MemAvailable / MemTotal is also in node-exporter's official mixin as the NodeMemoryHighUtilization alert (alerts.libsonnet). Its expr uses variables for the selector and threshold; stripped of those it is 100 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100) > <threshold>. However, as shown below, this value does not match the utilization the kubelet uses for eviction decisions.

Values the kubelet uses for eviction decisions

The kubelet normally evaluates memory availability every 10 seconds. Crossing a threshold contributes to MemoryPressure; eviction also depends on soft-threshold grace periods and reclaim handling. This memory.available is not /proc/meminfo MemAvailable: it is computed from the root cgroup's working set.

memory.available = node_memory_MemTotal_bytes - container_memory_working_set_bytes{id="/"}

For the implementation see Kubernetes' helper.go and helpers_others.go. The 10-second interval is defined as evictionMonitoringPeriod.

Warning: The root cgroup working set used for eviction decisions (and by kubectl top node) is not the same thing as a container's working set. In cgroup v2, memory.current has the CFTYPE_NOT_ON_ROOT flag, so the root cgroup has no memory.current (memory_files[] in mm/memcontrol.c).
  {
      .name = "current",
      .flags = CFTYPE_NOT_ON_ROOT,
      .read_u64 = memory_current_read,
  },
memory.stat, on the other hand, has no such flag, so it can be read even on the root. The cgroup library cAdvisor uses therefore substitutes anon + file from memory.stat as the usage when reading memory.current fails with ENOENT, for the root only (rootStatsFromMeminfo() in opencontainers/cgroups).
  // sum `anon` + `file` to report the same value as `usage_in_bytes` in v1.
  stats.MemoryStats.Usage.Usage = stats.MemoryStats.Stats["anon"] + stats.MemoryStats.Stats["file"]
Kubernetes v1.37.1 selects opencontainers/cgroups v0.0.7. For a real host root cgroup, the id="/" series is therefore:
container_memory_usage_bytes{id="/"}       = anon + file
container_memory_working_set_bytes{id="/"} = anon + file - inactive_file
Unlike memory.current, this includes no kernel memory at all, such as slab, pagetables, kernel_stack, percpu and sock. This is mainly why the node working set tends to come out smaller than usage derived from /proc/meminfo. For container and Pod cgroups memory.current can be read directly, so this substitution does not occur.

Because the working set includes active page cache, the kubelet treats active page cache as "memory in use".

For memory on Linux, the default is the hard-eviction threshold memory.available<100Mi, as in DefaultEvictionHard in defaults_linux.go; no soft eviction is configured by default.

Choose thresholds appropriate to the node size and workload. In practice, though, I think it is better to override this value explicitly. In the v1.37 cluster I used for verification I also pass the following flags to the kubelet to set both hard and soft thresholds.

--eviction-hard=memory.available<5%,nodefs.available<5%,pid.available<5%
--eviction-soft=memory.available<10%,nodefs.available<10%,pid.available<10%
--eviction-soft-grace-period=memory.available=2m,nodefs.available=5s,pid.available=1m

The default 100Mi is a fixed value, so it becomes relatively thinner on nodes with more memory. If you specify percentages, the threshold follows the node size. Placing soft eviction ahead of hard (10 %) and giving it a grace period with --eviction-soft-grace-period avoids evicting Pods on momentary spikes while still evicting them when the pressure persists.

You can check your cluster's settings with the following query.

Setting only some evictionHard signals does not inherit the unspecified defaults unless mergeDefaultEvictionSettings is enabled. Include all required disk and memory signals or check that setting.

kubectl get --raw "/api/v1/nodes/<node name>/proxy/configz" \
  | jq '.kubeletconfig | {evictionHard, evictionSoft, evictionSoftGracePeriod}'

Output:

{
  "evictionHard": {
    "memory.available": "5%",
    "nodefs.available": "5%",
    "pid.available": "5%"
  },
  "evictionSoft": {
    "memory.available": "10%",
    "nodefs.available": "10%",
    "pid.available": "10%"
  },
  "evictionSoftGracePeriod": {
    "memory.available": "2m",
    "nodefs.available": "5s",
    "pid.available": "1m"
  }
}

Note: The denominator is Capacity (node.status.capacity.memory, kube_node_status_capacity{resource="memory"}), not Allocatable. These normally match node_memory_MemTotal_bytes; verify that the metrics describe the same host and cgroup namespace. The kubelet uses cAdvisor's value as the machine's memory capacity, and cAdvisor reads it from MemTotal in /proc/meminfo (raw/handler.go). A threshold like memory.available<5% is also evaluated as a percentage of this Capacity.

Allocatable is Capacity minus the system reservation and the hard eviction threshold, and is the upper bound the scheduler uses when assigning Pod requests. It is not used to decide whether eviction happens. See Reserve Compute Resources for System Daemons for details.

Why node-exporter and kubelet usage disagree

Putting the above together, there are two views of the same node.

View	Calculation	Emitted by	Active page cache
Node memory utilization from node-exporter	`1 - MemAvailable / MemTotal`	node-exporter (`/proc/meminfo`)	The part estimated reclaimable counts as free
Kubelet's eviction decision	`node working set / MemTotal`	cAdvisor (embedded in kubelet; root cgroup `anon + file - inactive_file`)	Counted entirely as in use

MemAvailable is the kernel's estimate of "how much can be newly allocated", and it adds part of the page cache and the reclaimable slab to free memory. It does not count the whole page cache as free. The implementation is si_mem_available() in mm/show_mem.c.

    /*
     * Estimate the amount of memory available for userspace allocations,
     * without causing swapping or OOM.
     */
    available = global_zone_page_state(NR_FREE_PAGES) - totalreserve_pages;

    /*
     * Not all the page cache can be freed, otherwise the system will
     * start swapping or thrashing. Assume at least half of the page
     * cache, or the low watermark worth of cache, needs to stay.
     */
    pagecache = global_node_page_state(NR_ACTIVE_FILE) +
        global_node_page_state(NR_INACTIVE_FILE);
    pagecache -= min(pagecache / 2, wmark_low);
    available += pagecache;

    /*
     * Part of the reclaimable slab and other kernel memory consists of
     * items that are in use, and cannot be freed. Cap this estimate at the
     * low watermark.
     */
    reclaimable = global_node_page_state_pages(NR_SLAB_RECLAIMABLE_B) +
        global_node_page_state(NR_KERNEL_MISC_RECLAIMABLE);
    reclaimable -= min(reclaimable / 2, wmark_low);
    available += reclaimable;

Three things can be read from this.

What is subtracted is min(half, watermark_low), so at least half of the page cache and the reclaimable slab is counted as free
From the free pages, totalreserve_pages is subtracted, not watermark_low
The page cache term is the file LRU (NR_ACTIVE_FILE + NR_INACTIVE_FILE), so shmem, which sits on the anon LRU, is never counted as free. On nodes that use tmpfs heavily, MemAvailable also comes out small

The important point is that the definitions differ, and the size and direction of the difference depend on the environment. There are four factors behind the divergence, and they push in opposite directions.

Factor	Which utilization comes out higher
The working set counts `active_file` entirely as "in use"	The kubelet's utilization is higher
The working set removes `inactive_file` entirely as "free"	node-exporter's utilization is higher
`MemAvailable` subtracts `min(half, watermark_low)` from file-LRU cache before counting it as available (`shmem` is excluded)	node-exporter's utilization is higher
The root's usage is computed as `anon + file`, so it includes no kernel memory such as slab or pagetables	node-exporter's utilization is higher

These rows are not independent additive terms; some describe the same cache treatment from different angles. Compare statistics from the same time rather than counting how many factors favor either side.

The kubelet side is higher on nodes running workloads where active_file builds up easily, for example:

Batch jobs and ETL that read and write large files
Services that emit a lot of logs (stdout is written to files on the node by the container runtime, so it becomes page cache)
Middleware designed around the page cache (Kafka, Elasticsearch, etc.)
Log collection agents that keep reading log files on the node

What matters is the amount of file access, not the language or runtime. For example, a JVM heap is anonymous pages, so it raises RSS, not active_file.

On such a node it can happen that utilization computed from MemAvailable looks like 33 % while the kubelet's side has reached 53 %. If you look at the MemAvailable-based utilization and conclude there is plenty of room, the kubelet may in fact be approaching its threshold and evictions will occur.

On the other hand, on a node where page cache has not built up much:

MemTotal          3.820 GiB
MemAvailable      2.692 GiB   →  1 - 2.692/3.820 = 29.6 %  (node-exporter side)
usage(/)          2.557 GiB   (= anon + file of the root)
inactive_file(/)  1.622 GiB
active_file(/)    0.227 GiB
Working Set(/)    0.935 GiB   →  0.935/3.820     = 24.5 %  (kubelet side)

active_file is only 0.23 GiB while inactive_file is 1.6 GiB, so the working set removes it all as "free", and the kubelet side comes out about 5 points lower. The kubelet side is not always higher.

Either way, the conclusion is that you cannot judge whether there is headroom from only one of the values. active_file is an auxiliary indicator for confirming the cause of the divergence, and its value is not itself the difference. When investigating node memory pressure, also check the kubelet-view utilization with the following PromQL.

# Node memory utilization from the kubelet's view (the value used in eviction decisions)
max by (node) (
  container_memory_working_set_bytes{id="/"}
  * on(instance) group_left(node) kubelet_node_name
)
/ max by (node) (kube_node_status_capacity{resource="memory"})

# Amount of active page cache, the main cause of the divergence
max by (node) (
  container_memory_total_active_file_bytes{id="/"}
  * on(instance) group_left(node) kubelet_node_name
)

Note: The label alignment is there because no cAdvisor container_* metric carries any label identifying the node. Looking directly at the kubelet's /metrics/cadvisor, the root cgroup series looks like this:
container_memory_working_set_bytes{container="",id="/",image="",name="",namespace="",pod=""} 8.92522496e+08
The only thing that tells you the node is the instance label Prometheus adds at scrape time, and labels such as kubernetes_io_hostname are added by your Prometheus relabel configuration. Because the name depends on the relabel configuration (or may not exist at all), referring to it with label_replace is environment specific.

So the queries above use kubelet_node_name, which the kubelet exposes on /metrics.
kubelet_node_name{node="memqos-control-plane"} 1
This is a gauge whose value is always 1, and the node name is in the node label. The value itself is meaningless; it is the info metric pattern, which carries the information in labels. The kubelet's help text says the same.
  // NodeName is a Gauge that tracks the node's name. The count is always 1.
  NodeName = metrics.NewGaugeVec(
      &metrics.GaugeOpts{
          Subsystem: KubeletSubsystem,
          Name:      NodeNameKey,
          Help:      "The node's name. The count is always 1.",
In OpenMetrics, Info is defined as an independent metric type, with a mandatory _info suffix and a sample value that is always 1 (OpenMetrics spec - Suffixes).

Info metrics are used to expose textual information which SHOULD NOT change during process lifetime.

kubelet_node_name has no _info suffix and is exposed as a gauge, so strictly speaking it is not an OpenMetrics Info type. But "value always 1, information in labels" is the same usage, so this article treats it as an info-metric pattern. Similar examples are kube-state-metrics' kube_node_info and node-exporter's node_uname_info. When both endpoints are scraped with matching target labels, instance matches, and * on(instance) group_left(node) brings in the node label. And the resulting node label has the same name as kube-state-metrics' node label, so they join directly.

If your environment adds its own node label by relabeling, you can use label_replace as follows (replace kubernetes_io_hostname with the value in your environment).
max by (node) (
  label_replace(
    container_memory_working_set_bytes{id="/"},
    "node", "$1", "kubernetes_io_hostname", "(.*)"
  )
)
Note: Eviction candidates are not limited to Pods in particular namespaces. When node memory gets tight, Pods in every namespace on that node are candidates. Which Pod is chosen is decided in this order: (1) whether usage exceeds the memory request, (2) Pod priority, (3) how far above the request it is. See Pod selection for kubelet eviction for details.

However, critical Pods are exempt from node-pressure eviction by the kubelet (evictPod() in eviction_manager.go). "Critical pod" is a term in the official Kubernetes documentation and refers to a Pod whose PriorityClass is system-cluster-critical or system-node-critical (Guaranteed Scheduling For Critical Add-On Pods says "To mark a Pod as critical, set priorityClassName for that Pod to system-cluster-critical or system-node-critical.").
  if kubelettypes.IsCriticalPod(pod) {
      logger.Error(nil, "Eviction manager: cannot evict a critical pod", "pod", klog.KObj(pod))
      return false
  }
The kubelet's check is broader than this definition: IsCriticalPod() is also true for static Pods and mirror Pods. The remaining condition is that pod.Spec.Priority is at least SystemCriticalPriority (= 2 × 1000000000), and since the upper limit of user-definable priority is HighestUserDefinablePriority (= 1000000000), the only PriorityClasses that match are the two above. These priority classes are exempt from node-pressure eviction, but that exemption does not itself determine their protection during global OOM. See Which process gets killed for details.

What to do about it

Once the cause is clear, review requests and placement for node pressure, and usage and limits for memcg OOM.

Start with the limiting cgroup

For memcg OOM, identify the cgroup named by oom_memcg before changing resources. It may be the container, its Pod or an ancestor such as /kubepods.slice.

Compare memory.current with that cgroup's memory.max, then inspect memory.stat to see whether anonymous memory, shmem, file cache or kernel memory dominates. Reduce the required memory or adjust the relevant limit after checking node capacity. Raising a request alone does not raise that limit.

Source: cgroup v2 memory interfaces.

Align memory request with actual usage

Aligning memory requests with actual usage helps prevent excessive placement density on a node.

With the default v1.37 kubelet configuration, resources.requests.memory is not applied as cgroup memory protection, so it does not directly limit how much memory a container can use. It nevertheless matters because the request determines how likely OOM kills and evictions are, in these three ways:

Placement (how likely both global OOM and eviction are): the standard resource-fit check uses requests and Allocatable rather than actual usage. Affinity, taints and other rules also affect placement. A Pod with no request, or one smaller than actual usage, is placed while overestimating the node's free memory. As a result, Pods with large memory usage concentrate on the same node and the whole node becomes tight, inviting both global OOM and eviction
Eviction candidate selection: the kubelet preferentially evicts Pods whose actual usage exceeds their memory request. If you set the request smaller than actual usage, that Pod is evicted more easily
oom_score_adj (the order of kills in a global OOM): as described earlier, the Burstable oom_score_adj is 1000 - 1000 × memory request / node memory capacity. The smaller the request, the larger the oom_score_adj, and the more likely the Pod is chosen as a kill target in a global OOM. BestEffort Pods receive 1000. An unset memory request alone does not imply BestEffort: CPU resources can make a Pod Burstable, and a memory limit can default its request

"Whether the event happens" and "which Pod is chosen" are separate matters, so organized by effect:

How the request acts	memcg OOM occurs	global OOM occurs	global OOM target selection	Eviction occurs	Eviction target selection
Placement (scheduler)	—	○	—	○	—
`oom_score_adj`	—	—	○	—	—
Eviction candidate selection	—	—	—	—	○

Only placement affects whether an event occurs, and it affects both global OOM and eviction, because both start from node pressure. The effect is indirect, though: the request does not appear in the computation of memory.available (the kubelet's memory-availability calculation), so raising a request does not relax the threshold. The node just becomes less likely to get tight as a result of lower placement density.

The other two only affect whether your Pod is chosen after the pressure has built up. With the default settings, increasing a request does not raise memory.max or resolve a memcg limit violation.

Raising requests can reduce placement density and alter victim selection. It can also leave Pods Pending or trigger node expansion, so account for capacity and cost.

Warning: Increasing a request alone does not resolve memcg OOM. Reduce usage or review the limiting cgroup's limit after checking node capacity. If Memory QoS protection or throttling is configured, requests also affect reclaim behavior. This section focuses on reducing global OOM and eviction.

PromQL for comparing requests and actual usage

You can check whether a request matches actual usage with PromQL. I included queries for the per-Pod ratio, for finding Pods with no request, for per-node comparison, and for deriving a recommended value from past usage. I also explain the pitfalls that come from where the metrics originate (such as double counting).

PromQL for checking and notes on filters

What to know before comparing

These queries assume one cluster and that the required metrics are collected. For multiple clusters, filter by cluster or include a cluster label in every aggregation and join key. When joining on instance, restrict jobs as needed so that the right-hand kubelet_node_name has one series per instance.

A Pod's request is available as kube_pod_resource_request, which kube-scheduler exposes on its /metrics/resources endpoint. You compare it with actual usage (container_memory_working_set_bytes), but the two come from different sources, so dividing them as is gives the wrong result. Keep these five points in mind.

Point	Details
Scrape configuration is needed	`kube_pod_resource_request` is on kube-scheduler's secure port (default 10259) at `/metrics/resources`. It is a different endpoint from `/metrics`, so Prometheus needs explicit configuration
No series for Pods with no request	Because the collector does `if val.IsZero() { return }`, Pods whose request is 0 or unset do not appear in the metric. Pods with zero requests drop out of the ratio query, so find them separately with `unless` below
Aggregated per Pod	`kube_pod_resource_request` is the total for the whole Pod and has no `container` label. Actual usage must also be aligned to Pod level before comparing (how is shown below)
Duplicate scrapes	kube-scheduler runs multiple replicas in an HA setup, so the same Pod's series may be scraped several times with different `instance` values. Summing them double counts, so normalize with `max by (...)` before aggregating
Terminated Pods are excluded	Pods that are Succeeded / Failed are not included

Which metric gives per-Pod memory usage

kube_pod_resource_request is a per-Pod value, so actual usage must also be aligned to Pod level. You may wonder "doesn't kube-state-metrics have a Pod memory usage metric?", but it does not.

kube-state-metrics converts Kubernetes API objects (spec and status) directly into metrics and does not handle actual resource usage at all. It provides configured values written in manifests, such as kube_pod_container_resource_requests / kube_pod_container_resource_limits, but usage is out of scope. Usage requires reading cgroups, which is the job of cAdvisor and the kubelet.

There are three ways to get actual per-Pod usage.

Method	Metric	Emitted by	Characteristics
Sum the containers	`sum by (namespace, pod) (container_memory_working_set_bytes{...})`	cAdvisor (`/metrics/cadvisor`)	Works when the required container metrics are collected. Double counts if the filter is wrong (see below)
Look at the Pod's cgroup directly	`container_memory_working_set_bytes{container="", image="", pod!=""}`	cAdvisor (`/metrics/cadvisor`)	No summing needed. Slightly larger than the container total because it includes the pause container and Pod-level cgroup charges. `image=""` is required (see below)
The kubelet's per-Pod metric	`pod_memory_working_set_bytes{namespace, pod}`	kubelet `/metrics/resource`	The most straightforward, but it is on a different path from `/metrics/cadvisor`, so you must add a scrape configuration to Prometheus

With the default cAdvisor statistics provider, the second and third use the same Pod cgroup. Sampling times and CRI statistics settings can produce differences. The kubelet gets pod_memory_working_set_bytes from the Pod-level cgroup, not as a sum of containers.

        podUID := types.UID(podStats.PodRef.UID)
        // Lookup the pod-level cgroup's CPU and memory stats
        podInfo := getCadvisorPodInfoFromPodUID(podUID, allInfos)
        if podInfo != nil {
            cpu, memory := cadvisorInfoToCPUandMemoryStats(podInfo)
            podStats.CPU = cpu
            podStats.Memory = memory

pod_memory_working_set_bytes is a STABLE metric, so if Prometheus scrapes /metrics/resource in your environment, using it is the simplest.

# Actual per-Pod usage (when scraping the kubelet's /metrics/resource)
pod_memory_working_set_bytes

If you do not scrape /metrics/resource, specify the Pod cgroup series directly. /metrics/resource is a different path from /metrics/cadvisor, so it is not collected unless you add it to the Prometheus configuration.

# Actual per-Pod usage (looking at the Pod's cgroup series directly)
max by (namespace, pod) (
  container_memory_working_set_bytes{container="", image="", pod!=""}
)

container="" gets the Pod's cgroup because when the kubelet labels metrics it also attaches pod and namespace to the Pod's cgroup series. The code explains it as "Associate pod cgroup with pod so we have an accurate accounting of sandbox" (server.go).

Warning: Do not omit image="". If you filter with only container="", the series of the pause container (sandbox) also matches along with the Pod's cgroup. In containerd environments the pause container's container label is an empty string and its image label holds the pause image.

For example, in a cluster running 100 Pods, the series count splits into these three kinds.

Selector	Series	What it is
`{container="", pod!=""}`	200	Pod cgroups + pause containers
`{container="", image="", pod!=""}`	100	Pod cgroups only
`{container="", image!="", pod!=""}`	100	Pause containers only

With max by (namespace, pod) the Pod's cgroup is larger, so the correct value comes back, but with sum by the pause part is added and it is double counted. For one Pod it looks like this:

Pod cgroup                666,116,096   ← correct value
Sum of real containers     665,882,624
pause container                225,280
sum{container="",pod!=""}  666,341,376   ← Pod cgroup + pause, double counted

If you do sum by (namespace, pod) without a container filter, each container's share and the Pod cgroup's share are both added, so the value is almost doubled. When summing containers, narrow to real containers only, as follows.

# Actual per-Pod usage (when summing containers)
sum by (namespace, pod) (
  container_memory_working_set_bytes{container!="", image!=""}
)

In containerd environments the pause container's container is empty, so container!="" alone excludes the Pod cgroup, the root cgroup and pause. image!="" is added because series other than real containers have no image, and it is insurance for environments where container!="" alone leaves some series. With CRI-O the pause container appears as container="POD", so also add container!="POD" there.

The following queries use the Pod cgroup series; verify that it is collected with these labels in your environment.

Compare request and actual usage for each Pod

This compares the observed working set with the request. A value above 1 means usage currently exceeds the request. Check history and the breakdown before changing it: a temporary peak or reclaimable cache does not by itself establish that the request is too small.

# Ratio of observed working set to memory request (above 1 means usage exceeds the request)
max by (namespace, pod) (
  container_memory_working_set_bytes{container="", image="", pod!=""}
)
/
max by (namespace, pod) (
  kube_pod_resource_request{resource="memory", unit="bytes"}
)

To see the difference in bytes, do the following. A positive value is "the amount by which the request is exceeded".

# Amount over the request (bytes)
max by (namespace, pod) (
  container_memory_working_set_bytes{container="", image="", pod!=""}
)
-
max by (namespace, pod) (
  kube_pod_resource_request{resource="memory", unit="bytes"}
)

Find Pods with no request

As noted above, Pods with no request do not appear in kube_pod_resource_request, so they do not show up in the ratio query. Use unless to find "Pods that have a usage series but no request series". These may have a zero memory request, or their request series may be missing from collection. Check the scrape configuration and the applied Pod resources; CPU settings can make them Burstable.

# Pods with no memory request, and their actual usage
max by (namespace, pod) (
  container_memory_working_set_bytes{container="", image="", pod!=""}
)
unless
max by (namespace, pod) (
  kube_pod_resource_request{resource="memory", unit="bytes"}
)

Compare the sum of requests and the sum of actual usage per node

For each node, compare "usage the scheduler knows about (sum of requests)" with "actual usage". A node where actual usage is larger is one where the scheduler overestimates free space, and the risk of global OOM and eviction is high.

kube_pod_resource_request and kube_node_status_allocatable both have a node label, so these two can be joined without label_replace.

# Per node: sum of memory requests / Allocatable
sum by (node) (
  max by (node, namespace, pod) (
    kube_pod_resource_request{resource="memory", unit="bytes"}
  )
)
/
max by (node) (
  kube_node_status_allocatable{resource="memory"}
)

The actual-usage side comes from cAdvisor and has no node label, so align labels as in the PromQL above. Using the working set of /kubepods.slice gives the usage of all Pods on the node without summing per Pod.

# Actual usage of all Pods on the node / Allocatable
max by (node) (
  container_memory_working_set_bytes{id=~"/kubepods|/kubepods.slice"}
  * on(instance) group_left(node) kubelet_node_name
)
/
max by (node) (
  kube_node_status_allocatable{resource="memory"}
)

Put these two side by side and look for nodes where the lower query (actual usage) exceeds the upper one (requests).

Derive a recommended request from past usage

Decide the request not from an instantaneous value but from usage over a period. Anonymous memory required by the application cannot be time-shared like CPU. Use representative peaks as well as averages, while separating reclaimable cache from the workload's persistent needs.

# Maximum memory usage per Pod over the past 7 days (a guide for the request)
max_over_time(
  max by (namespace, pod) (
    container_memory_working_set_bytes{container="", image="", pod!=""}
  )[7d:5m]
)

If you do not want to be pulled around by transient spikes, use a quantile. Be aware, though, that if you use this value as the request, the Pod will exceed its request at peak times and become an eviction candidate.

# 95th percentile over the past 7 days
quantile_over_time(0.95,
  max by (namespace, pod) (
    container_memory_working_set_bytes{container="", image="", pod!=""}
  )[7d:5m]
)

Warning: The maximum obtained by this query can miss spikes shorter than the scrape interval or the subquery evaluation step (5 minutes here). For applications that temporarily allocate a lot of memory right after startup, the real peak may be higher than this value. It is the same reason as in My working set is below the limit, but the container was OOMKilled. Leave some margin, or decide after understanding the application's startup behavior.

Also, the working set includes active page cache. For containers with a lot of logging or file I/O, using this value as is for the request makes it too large. Compare with container_memory_rss and decide which to base it on.

Do not base request design on kubectl top

kubectl top pod is convenient, but it is not suited to designing requests. The reason lies in the design of Metrics Server itself.

What Metrics Server gets from the kubelet is the value of the /metrics/resource endpoint, and for memory it is the working set ("Memory is reported as the working set at the instant the metric was collected"). It is the same value as container_memory_working_set_bytes seen through Prometheus, so it includes page cache in the same way
Metrics Server keeps only the latest value. The binary default for --metric-resolution is 60 seconds, while the v0.8.1 deployment manifest sets it to 15 seconds. Metrics Server keeps no long-term history, so it cannot show past peaks
Metrics Server's own README states in its use cases "Don't use Metrics Server when you need: ... An accurate source of resource usage metrics" (README.md)

Use kubectl top to get a feel for "what is using a lot right now", and when you decide a request, specify a period and check with PromQL for comparing requests and actual usage.

Adjust requests automatically with VPA

Instead of running the PromQL in the previous section by hand, you can have the Vertical Pod Autoscaler (VPA) compute recommendations. The idea is the same (derive from quantiles of past usage), and VPA runs it continuously, combined with OOM detection.

VPA configuration examples and defaults

VPA consists of three components.

Component	Role
Recommender	Computes recommendations from usage history and writes them to `status.recommendation` of the VPA object
Updater	Applies in-place updates or evicts Pods for recreation, depending on `updateMode`
Admission Controller	Rewrites `resources` to the recommendation with a mutating webhook at Pod creation

Start by looking at recommendations with updateMode: Off

You would not want Pods to be rebuilt abruptly, so start with updateMode: Off to inspect recommendations. In this mode VPA does not touch Pods at all and only writes recommendations to the VPA object. The API definition also says "This can be used for a \"dry run\"".

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: sample-vpa
  namespace: default
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: sample
  updatePolicy:
    updateMode: "Off"          # only compute recommendations. Pods are not changed
  resourcePolicy:
    containerPolicies:
    - containerName: app
      controlledResources: ["memory"]   # default is ["cpu", "memory"]
      controlledValues: RequestsOnly    # default is RequestsAndLimits
      minAllowed:
        memory: 128Mi
      maxAllowed:
        memory: 4Gi

Check the recommendation as follows.

kubectl describe vpa sample-vpa

Output:

...
Status:
  Recommendation:
    Container Recommendations:
      Container Name:  app
      Lower Bound:
        Memory:  262144k
      Target:
        Memory:  367001600
      Uncapped Target:
        Memory:  367001600
      Upper Bound:
        Memory:  524288k

Target is the recommended request. Lower Bound and Upper Bound account for the history available and recommendation confidence; they do not guarantee the minimum runnable or maximum useful memory. The Updater uses these bounds when deciding whether a Pod needs an update.

Switch to automatic application

Once you confirm the recommendation is reasonable, change updateMode to switch to automatic application. The modes are:

`updateMode`	Behavior
`Off`	Only computes recommendations. Pods are not changed
`Initial`	Applies recommendations only at Pod creation. Running Pods are not changed
`Recreate`	Applies at creation, and also updates running Pods by evicting and recreating them
`InPlaceOrRecreate`	Tries In-Place Resize first, and falls back to recreation if that is not possible. Requires the cluster's `InPlacePodVerticalScaling` feature gate
`InPlace`	Tries only In-Place Resize and does not evict. On failure it leaves it to the kubelet's retry. Requires VPA's own `InPlace` feature gate in addition to the above
`Auto`	Deprecated. Currently equivalent to `Recreate`

Auto has been deprecated as of VPA 1.7, and the API comment also says "Use explicit update modes like \"Recreate\", \"Initial\", or \"InPlaceOrRecreate\" instead". Do not use Auto in newly written manifests.

This is a configuration example that automatically adjusts only the memory request.

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: sample-vpa
  namespace: default
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: sample
  updatePolicy:
    updateMode: "Recreate"
    minReplicas: 2             # do not evict when replicas is below this (the global default is also 2)
  resourcePolicy:
    containerPolicies:
    - containerName: app
      controlledResources: ["memory"]
      controlledValues: RequestsOnly
      minAllowed:
        memory: 128Mi
      maxAllowed:
        memory: 4Gi

With controlledValues: RequestsOnly, you can manage the limit yourself and leave only the request to VPA. In the context of this article (reducing global OOM and eviction) this is the easy setting to work with. With the default RequestsAndLimits, both are rewritten while keeping the ratio of request to limit, so note that the limit moves too.

Behavior when an OOM occurs

For memory, VPA has a mechanism that raises the recommendation when it detects an OOM kill. The defaults are:

Setting	Default	Meaning
`oomBumpUpRatio`	`1.2`	When an OOM kill is detected, record usage at that time multiplied by 1.2 as a sample
`oomMinBumpUp`	`100Mi` (104857600)	If the bump is smaller than this, bump up by at least this much
`memoryAggregationIntervalSeconds`	`86400` (24 hours)	Record one peak sample per this interval
`memoryAggregationIntervalCount`	`8`	How many such intervals to keep. By default the memory recommendation is computed from 24 hours × 8 = 8 days of history
`evictAfterOOMSeconds`	(unset)	Make Pods that OOM within this many seconds of starting eviction targets

To use these per-container overrides in containerPolicies, enable the PerVPAConfig feature gate in the VPA components that handle them. In VPA 1.7.1 it is Alpha and disabled by default. See the aggregation implementation.

    containerPolicies:
    - containerName: app
      controlledResources: ["memory"]
      oomBumpUpRatio: "1.5"                    # bump up more strongly on OOM
      oomMinBumpUp: 256Mi
      memoryAggregationIntervalSeconds: 3600   # record a peak every hour
      memoryAggregationIntervalCount: 24       # compute from 24 intervals = 1 day of history

The memory recommendation itself is computed from quantiles of a usage histogram. The Recommender defaults are:

Flag	Default	Use
`--target-memory-percentile`	`0.9`	Quantile used to compute `Target` (the recommended request)
`--recommendation-lower-bound-memory-percentile`	`0.5`	Quantile used to compute `Lower Bound`
`--recommendation-upper-bound-memory-percentile`	`0.95`	Quantile used to compute `Upper Bound`
`--recommendation-margin-fraction`	`0.15`	Safety margin added to the computed value (15 %)

So by default, a 90th percentile from a weighted histogram of interval peaks, with a 15% margin and other bounds applied becomes the recommended request. The idea is the same as the quantile_over_time(0.95, ...) PromQL introduced in the previous section, and it is easiest to think of VPA as running that continuously, combined with OOM detection.

Warning: The usage VPA looks at is also the working set. The Recommender gets usage from the Metrics Server API (metrics.k8s.io/v1beta1), so as described earlier it derives recommendations from values that include page cache.

So for containers with a lot of logging or file I/O, reclaimable cache can increase VPA's recommendation. For containers where the gap between container_memory_rss and container_memory_working_set_bytes is large, it may waste less in the end to cap with maxAllowed or to decide by hand instead of leaving it to VPA.

metrics.k8s.io became GA (v1) in v1.37 through KEP-5207. Metrics Server v0.8.1, the version covered here, uses v1beta1. The working-set definition does not depend on the API version.

Note: Operational points when using VPA:

Combining with HPA: do not use HPA and VPA on the same resource at the same time. The official docs also state "should not be used with the HPA on the same resource metric (CPU or memory)" (known-limitations.md). If you split the targets they can be combined, and VPA for memory and HPA for CPU is the officially recommended example. With controlledResources: ["memory"] as in this article's examples, CPU can be left to HPA
Recreate rebuilds Pods: the Updater evicts Pods using the Eviction API, so set a PodDisruptionBudget. It does not evict when replicas is below minReplicas (the global --min-replicas, default 2, if not specified in VPA)
In-Place modes: they can change the request without rebuilding the Pod, but require the cluster's InPlacePodVerticalScaling feature gate
Usage collection is required: this example uses Metrics Server; VPA can also be configured to use Prometheus for historical data

FAQ

My working set is below the limit, but the container was OOMKilled

What memcg OOM compares against the limit is memory.current (container_memory_usage_bytes), not the working set. Furthermore, memory.current reaching the limit does not cause an OOM by itself; the kill happens when reclaim fails to bring usage back under the limit. Also, even if your container is under its limit, it can be killed in a node-wide memory exhaustion (global OOM). First use The two OOM kill paths to tell the path.

If none of the values seems to have reached the limit, suspect a spike shorter than the scrape interval. A spike in memory usage shorter than Prometheus' scrape interval (the global scrape_interval defaults to 1 minute; see Configuration) does not show up in metrics. Batch processing right after startup, or suddenly accepting a very large request, are examples.

In that case, check values like these:

Metric	Emitted by	Description
`kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}`	kube-state-metrics	The last termination reason. The value does not count up when OOMs repeat, so from the second time on, detect it by combining with the increase of `kube_pod_container_status_restarts_total` (also kube-state-metrics)
`container_oom_events_total`	kubelet	Number of OOM kills. But the series disappears once the container cgroup is removed. Detecting an increase depends on scraping it before removal; already collected historical samples remain in Prometheus
`node_vmstat_oom_kill`	node-exporter	Number of OOM kills on the node. It cannot identify the Pod or container, but is not affected by container recreation

Besides metrics, also check Last State in kubectl describe pod (exit code 137 means termination by SIGKILL).

Node usage computed from MemAvailable is low, but Pods are evicted

MemAvailable counts part of the page cache as "free", but the kubelet is working-set based and treats all of active_file as "in use". With workloads that do a lot of file I/O or logging, active_file can grow to several GiB, so even if the MemAvailable-based utilization looks low, the kubelet side may be approaching its threshold.

Conversely, on nodes where page cache has not built up much, node-exporter's side comes out higher. Which is higher depends on the environment, so you cannot judge from just one of them. For the breakdown of factors and the PromQL to get the kubelet-side utilization, see Why node-exporter and kubelet usage disagree.

kubectl top pod and the Prometheus working set differ

kubectl top pod shows the sum of the containers' working sets obtained from the kubelet (via Metrics Server). Prometheus' container_memory_working_set_bytes comes from the same source, but it is as stale as the scrape interval, so instantaneous values differ.

Memory working set keeps growing

First check whether container_memory_rss or container_memory_cache is the one growing.

container_memory_rss is growing: suspect a memory leak in the application
container_memory_cache is growing: page cache is accumulating. But it includes not only the cache of files on disk but also tmpfs and shared memory. To tell which is growing, check shmem in memory.stat on the node. tmpfs (such as emptyDir with medium: Memory) is not reclaimed on nodes without swap, so accumulation leads to OOM kills. shmem sits on the anon or unevictable LRU, so it does not appear in inactive_file and is not subtracted from the working set. That means the increase pushes the working set up as is

For containers with no memory limit, page cache accumulates with the whole node's memory as the ceiling. Set a memory limit on containers that handle large files.

The RSS my application reports differs from the RSS metric

They measure different things. The value called "RSS" in Go, Java, Node.js and so on is per-process RSS, whereas container_memory_rss is anonymous pages charged to the container's cgroup.

Target	Source	Backing value	What it includes
`process.memoryUsage().rss` in Node.js, the `RSS` column of `ps` and `top`	Language runtime or tools such as `ps` (per process)	`VmRSS` in `/proc/[pid]/status`	`RssAnon` + `RssFile` + `RssShmem`
`container_memory_rss`	cAdvisor (embedded in kubelet; per cgroup)	`anon` in cgroup v2 `memory.stat`	Anonymous pages only

There are two main differences:

Whether file mappings are included: a process's RSS includes the pages of the executable, shared libraries and mmaped files that are in physical memory (RssFile). container_memory_rss does not. So process RSS also includes RssFile and RssShmem. Shared-page charging and sampling mean the difference is not necessarily an exact subtraction
Unit of aggregation: a process's RSS is for one process. container_memory_rss is per container cgroup, so it is the total of all processes in the container. Also, a process's RSS counts shared pages in each process, whereas in a cgroup a page is charged to only one cgroup

Note that JVM Runtime.totalMemory() and JMX heap usage are not RSS to begin with; they are the heap size. Metaspace, code cache, thread stacks and GC bookkeeping areas are not included, so you cannot compare these values directly with the memory limit.

Neither value is used in the OOM kill decision. What memcg OOM compares against the limit is memory.current (container_memory_usage_bytes). Global OOM follows a node-side allocation failure after reclaim, potentially within allocation constraints; it has no per-container usage threshold.

Which metric should I use for node memory utilization

Use them according to purpose.

Purpose	Metric to look at	Emitted by
Get a rough sense of the node's headroom	`node_memory_MemAvailable_bytes` / `node_memory_MemTotal_bytes`	node-exporter
Judge whether eviction will occur	`container_memory_working_set_bytes{id="/"}` / `kube_node_status_capacity{resource="memory"}`	cAdvisor (embedded in kubelet) / kube-state-metrics
Judge whether a Pod can be placed	The sum of `kube_pod_resource_request{resource="memory"}` (the sum of requests, not usage)	kube-scheduler `/metrics/resources`

Note: What emits kube_pod_resource_request is kube-scheduler, not kube-state-metrics (resources.go). kube-state-metrics also has a similarly named kube_pod_container_resource_requests, but kube-state-metrics itself says in its help text that it recommends using the more accurate kube-scheduler metrics. Unless Prometheus is configured to scrape kube-scheduler's /metrics/resources, kube_pod_resource_request is not collected.

Version and configuration differences

Statistics collection and Memory QoS behavior depend on the Kubernetes version and kubelet configuration.

cgroup v2 prerequisites

Kubernetes support for cgroup v2 became stable in v1.25. The documented requirements include Linux 5.8 or later, a compatible runtime and matching systemd cgroup drivers. Confirm that the following command prints cgroup2fs:

stat -fc %T /sys/fs/cgroup/

See About cgroup v2. The official blog, The Shift to cgroup v2 in Kubernetes, also explains active page cache and group OOM handling. Its Memory QoS discussion uses v1.36 settings; the v1.37 defaults below differ.

KEP-2371: cAdvisor-less, CRI-full Container and Pod Stats

KEP-2371 moves the source of container and node metrics from the kubelet's embedded cAdvisor to the container runtime (CRI). Its goal is to remove the situation where the kubelet collects duplicate statistics from both cAdvisor and CRI.

The /metrics/cadvisor endpoint and the container_* metric names are kept, and the plan is to change only where the values come from to CRI
The feature gate PodAndContainerStatsFromCRI was Alpha in v1.23 and becomes Beta in v1.37, but even at Beta it is disabled by default

So the definition of the working set and the eviction decision method described in this article will not change for the time being. If this migration becomes the default in the future, the source of values becomes the runtime side, so small differences may arise from differences in how cgroups are read.

KEP-2570: Support Memory QoS with cgroups v2

KEP-2570 reflects requests.memory into cgroup v2 memory protection settings. When enabled, the memory for the request is protected from reclaim, and throttling kicks in before the limit is reached.

The feature gate MemoryQoS has been Alpha (disabled by default) since v1.22 and became Beta and enabled by default in v1.37. However, just enabling the feature gate does not make requests.memory reflected in the cgroup. To reflect it, the kubelet needs the following settings, both disabled by default as of v1.37.

memoryReservationPolicy: TieredReservation: sets requests.memory into memory.min for Guaranteed and memory.low for the rest. A setting added to KubeletConfiguration in v1.36, with a default of None. With None, memory.min is not set
memoryThrottlingFactor: sets memory.high to requests.memory + factor × (limits.memory - requests.memory). Up to v1.36 the default was 0.9, but the feature gate itself was disabled by default then, so it had no actual effect. In v1.37 the default is nil, and unless you set it explicitly memory.high is not set

Both can be confirmed in the KubeletConfiguration type definition.

    // MemoryThrottlingFactor specifies the factor multiplied by the memory limit or node allocatable memory
    // ...
    // Default: nil
    MemoryThrottlingFactor *float64 `json:"memoryThrottlingFactor,omitempty"`
    // MemoryReservationPolicy controls how the kubelet applies cgroup v2 memory protection.
    // "None" (default): The kubelet does not set memory.min for containers and pods,
    // ...
    MemoryReservationPolicy MemoryReservationPolicy `json:"memoryReservationPolicy,omitempty"`

So even on a v1.37 cluster, requests.memory is not reflected in the cgroup, as in the table above. "Not reflected" is only about the default configuration, and it is reflected if you set the two above.

What is actually written with the default configuration is as in the implementation. When memoryReservationPolicy is None, an explicit 0 is written instead of requests.memory (equivalent to no protection).

        if memoryRequest != 0 && m.memoryReservationPolicy == kubeletconfiginternal.TieredReservationMemoryReservationPolicy {
            ...
        } else {
            unified[cm.Cgroup2MemoryMin] = "0"
            unified[cm.Cgroup2MemoryLow] = "0"
        }

So with the v1.37 default configuration, a container's cgroup has memory.min = 0, memory.low = 0 and memory.high unset (max), and only memory.max (= limits.memory) is in effect. If you use a managed service, check the provider's release notes to see whether these settings get enabled.

What the cgroup v2 values become if you enable it

This section works out what gets written to each cgroup v2 file when you explicitly set memoryThrottlingFactor: 0.9, the default up to v1.36, and also enable memoryReservationPolicy: TieredReservation.

kubelet configuration example and the values written for each QoS class

The kubelet configuration looks like this. MemoryQoS is enabled by default in v1.37, so you can omit featureGates, but I include it to make the intent explicit.

# /var/lib/kubelet/config.yaml (KubeletConfiguration)
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
featureGates:
  MemoryQoS: true            # true by default in v1.37
memoryReservationPolicy: TieredReservation   # default is None
memoryThrottlingFactor: 0.9                  # default in v1.37 is nil (= memory.high is not set)

Suppose you create the following Pod in this state. Assume the node's Allocatable memory is 16Gi.

apiVersion: v1
kind: Pod
metadata:
  name: sample
spec:
  containers:
  - name: app
    image: nginx
    resources:
      requests:
        memory: 256Mi
      limits:
        memory: 1Gi

requests.memory and limits.memory differ, so this Pod's QoS class is Burstable. The following values are written to the container's cgroup.

File	Value	Calculation
`memory.max`	`1073741824` (1Gi)	`limits.memory` as is (set as before, independent of MemoryQoS)
`memory.low`	`268435456` (256Mi)	Burstable, so `requests.memory` goes to `memory.low`
`memory.min`	`0`	`requests.memory` goes in only for Guaranteed. Otherwise an explicit `0` is written
`memory.high`	`993210368` (about 947 MiB)	`floor((256Mi + (1Gi - 256Mi) × 0.9) / 4096) × 4096`

The memory.high calculation is rounded down to a multiple of the page size (4 KiB), as in the implementation.

            memoryHigh = int64(math.Floor(
                float64(memoryRequest)+
                    (float64(memoryLimitVal)-float64(memoryRequest))*float64(*m.memoryThrottlingFactor))/float64(defaultPageSize)) * defaultPageSize

So this container is subject to strong reclaim pressure and throttling at about 947 MiB, before the 1Gi hard limit. Up to 256Mi it is also less likely to be reclaimed thanks to memory.low.

Organized by QoS class, it looks like this. What matters is that even with the same requests / limits, the files that get set change with the QoS class.

QoS class	`memory.min`	`memory.low`	`memory.high`
Guaranteed (`requests` = `limits`)	`requests.memory`	`0`	Not set (stays `max`; excluded when `requests` and `limits` are equal)
Burstable (`requests` < `limits`)	`0`	`requests.memory`	`floor((req + (lim − req) × factor) / page size) × page size`
Burstable (no `limits`)	`0`	`requests.memory`	Same formula using the node's Allocatable instead of `limits`
BestEffort (no `requests`/`limits`)	`0`	`0`	`floor(Allocatable × factor / page size) × page size`

For a BestEffort Pod on the same node as the example above (Allocatable 16Gi), memory.high is floor(16Gi × 0.9 / 4096) × 4096 = 15461879808 (about 14.4 GiB). memory.high is also set on containers with no limit, so enabling memoryThrottlingFactor changes how page cache accumulates as well.

memory.min and memory.low are set not only on the container cgroup but also on higher levels (qos_container_manager_linux.go). The kernel evaluates protection by walking up through ancestor cgroups, so the same protection is needed above as well.

Level	`memory.min`	`memory.low`
`/kubepods.slice`	Sum of Guaranteed requests + sum of Burstable requests	Sum of Burstable requests
`/kubepods.slice/kubepods-burstable.slice`	`0`	Sum of Burstable requests
Pod cgroup	The Pod's request if Guaranteed	The Pod's request if Burstable
Container cgroup	As in the table above	As in the table above

You can check how much is protected across the whole node with the following metrics the kubelet exposes (both ALPHA as of v1.37).

Metric	Emitted by	Description
`kubelet_memory_qos_node_memory_min_bytes`	kubelet	Configured `memory.min` protection budget for Guaranteed Pods, not actual resident memory or the effective protected amount
`kubelet_memory_qos_node_memory_low_bytes`	kubelet	Configured `memory.low` protection budget for Burstable Pods

Warning: With memoryReservationPolicy: TieredReservation, Guaranteed requests become memory.min protection budgets. This does not preallocate physical memory. The kernel protects the used portion within the effective boundary determined by ancestors and competing protection. Excessive protection can leave too little reclaimable memory elsewhere and destabilize the node. Right-size requests before enabling it.

If you are considering enabling it, the precondition is to first right-size requests with Align memory request with actual usage.

Hands-on verification with kind

The following examples compare protection settings, page cache and anonymous memory in a local kind cluster.

Verifying it with kind

Steps to bring up a local v1.37 cluster with kind and verify

These configuration examples and recorded values use kind with Kubernetes v1.37.0. The implementation references elsewhere target v1.37.1. The following configuration enables Memory QoS protection and throttling:

# kind-memqos.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: memqos
nodes:
- role: control-plane
  # the kind default image changes with the version, so pin it explicitly
  image: kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5
  kubeadmConfigPatches:
  - |
    kind: KubeletConfiguration
    featureGates:
      MemoryQoS: true
    memoryReservationPolicy: TieredReservation
    memoryThrottlingFactor: 0.9

If you omit image, the default image embedded in kind is used, but that changes with each kind version. The digest above is the same as the default of kind v0.33.0; to try a different Kubernetes version, replace it with the digest listed in the kind release notes.

kind create cluster --config kind-memqos.yaml

# Check that the settings reached the kubelet
kubectl get --raw "/api/v1/nodes/memqos-control-plane/proxy/configz" \
  | jq '.kubeletconfig | {memoryThrottlingFactor, memoryReservationPolicy, featureGates}'

Output:

{
  "memoryThrottlingFactor": 0.9,
  "memoryReservationPolicy": "TieredReservation",
  "featureGates": { "MemoryQoS": true }
}

You read cgroup files by going into the node container. kind puts the cgroup root at /kubelet.slice/kubelet-kubepods.slice, so note that the path differs from /kubepods.slice in a real cluster.

Note: Two behaviors differ from a host running Kubernetes directly.

One is the root cgroup fallback described earlier. A kind node is a container and has its own cgroup namespace, so the /sys/fs/cgroup visible from inside the node is actually a delegated non-root cgroup. As a result memory.current exists, and the anon + file substitution does not happen. The difference from a real machine is as follows.

# Inside a kind node (root of the cgroup namespace = actually a non-root cgroup)
docker exec memqos-control-plane ls /sys/fs/cgroup/memory.current

Output:

/sys/fs/cgroup/memory.current

# On the real host (the root cgroup of cgroup v2)
colima ssh -- ls /sys/fs/cgroup/memory.current

Output:

ls: /sys/fs/cgroup/memory.current: No such file or directory

colima ssh -- ls /sys/fs/cgroup/memory.stat

Output:

/sys/fs/cgroup/memory.stat

The other is eviction. kind overrides evictionHard with disk-related thresholds and omits memory.available. In this configuration there is no memory hard-eviction threshold; without other reservations, memory Capacity and Allocatable match. Soft thresholds and custom settings can change this. Set evictionHard explicitly to test memory eviction.

Create sample-burstable.yaml and wait for the application container before inspecting its cgroup:

apiVersion: v1
kind: Pod
metadata:
  name: sample-burstable
spec:
  containers:
    - name: app
      image: nginx:1.27
      resources:
        requests:
          memory: 256Mi
        limits:
          memory: 1Gi

kubectl apply -f sample-burstable.yaml
kubectl wait --for=condition=Ready pod/sample-burstable --timeout=120s

# Identify the cgroup by the real container ID (a glob can pick up the pause side, as explained below)
CID=$(kubectl get pod sample-burstable \
    -o jsonpath='{.status.containerStatuses[0].containerID}' | sed 's|containerd://||')
docker exec memqos-control-plane sh -c "
    d=\$(find /sys/fs/cgroup/kubelet.slice -type d -name 'cri-containerd-${CID}.scope')
    grep . \$d/memory.max \$d/memory.min \$d/memory.low \$d/memory.high"

Output:

/sys/fs/cgroup/.../cri-containerd-50fc89e1....scope/memory.max:1073741824
/sys/fs/cgroup/.../cri-containerd-50fc89e1....scope/memory.min:0
/sys/fs/cgroup/.../cri-containerd-50fc89e1....scope/memory.low:268435456
/sys/fs/cgroup/.../cri-containerd-50fc89e1....scope/memory.high:993210368

These are measured values on a node with Allocatable of 2005512Ki (= 2053644288 bytes) with memoryThrottlingFactor: 0.9 set.

Pod	QoS	`memory.min`	`memory.low`	`memory.high`
`requests: 256Mi` / `limits: 1Gi`	Burstable	0	268435456	993210368
`requests: 128Mi` / no limits	Burstable	0	134217728	1861701632
memory only `requests` = `limits` = 512Mi	Burstable	0	536870912	max (not set)
cpu and memory `requests` = `limits`	Guaranteed	536870912	0	max (not set)
nothing specified	BestEffort	0	0	1848279040

Each memory.high follows the formula floor((req + (lim − req) × factor) / page size) × page size described earlier. For Pods with no limits, the node's Allocatable is used for lim, and for BestEffort req is 0.

Warning: Setting only memory to requests = limits does not make a Pod Guaranteed. Guaranteed requires both cpu and memory to have requests = limits in every container. A Pod with only memory matched is Burstable, so requests.memory goes to memory.low, not memory.min.

memory.high stays unset (max) not because of the QoS class but on the condition requests.memory == limits.memory (the memoryRequest != memoryLimitSpec check in the implementation), so even a Burstable Pod with only memory matched stays at max.

                                    memory.min   memory.low   memory.high
Burstable with only memory matched           0    536870912           max
Guaranteed with cpu and memory matched  536870912          0           max

Also, besides the real container, a Pod has a pause (sandbox) container's cri-containerd-*.scope alongside it, so a glob for cri-containerd-*.scope can match both pause and application containers even in a single-container Pod. Which comes first depends on the container ID order, so using the first glob result may read the pause side, and memory.low and memory.high can look like 0 / max. Identify it with .status.containerStatuses[].containerID as in the command above.

Protection is also set on higher levels, not only on the container cgroup. On kubelet-kubepods.slice (equivalent to /kubepods.slice in a real cluster), memory.min is the sum of Guaranteed and Burstable requests and memory.low is the sum of Burstable requests.

kubelet-kubepods.slice            memory.min 1243611136   memory.low 706740224
kubelet-kubepods-burstable.slice  memory.min 0            memory.low 706740224
kubelet-kubepods-besteffort.slice memory.min 0            memory.low 0

1243611136 - 706740224 = 536870912 = 512Mi  ← matches the Guaranteed Pod's request

The kubelet metrics described earlier can also be obtained from this endpoint. memory_min_bytes covers only the Guaranteed part, so note that its value differs from the parent Pod cgroup's memory.min (Guaranteed + Burstable).

kubectl get --raw "/api/v1/nodes/memqos-control-plane/proxy/metrics" \
  | grep kubelet_memory_qos

Output:

kubelet_memory_qos_node_memory_low_bytes 7.06740224e+08
kubelet_memory_qos_node_memory_min_bytes 5.36870912e+08

Comparing page cache with anonymous memory

Comparing reclaimable page cache with an anonymous-memory OOM

To compare reclaimable cache with anonymous memory, write six times the container's 256Mi limit to a disk-backed emptyDir. This example uses the Memory QoS configuration above, including memory.high.

Create memtest.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: memtest
spec:
  containers:
    - name: app
      image: busybox:1.37
      command: ["sh", "-c", "sleep 3600"]
      resources:
        requests:
          memory: 64Mi
        limits:
          memory: 256Mi
      volumeMounts:
        - name: disk
          mountPath: /disk
  volumes:
    - name: disk
      emptyDir: {}

kubectl apply -f memtest.yaml
kubectl wait --for=condition=Ready pod/memtest --timeout=120s

kubectl exec memtest -- sh -c 'dd if=/dev/zero of=/disk/big bs=1M count=1536'

Output:

1610612736 bytes (1.5GB) copied, 1.501464 seconds, 1023.0MB/s

Looking at the container's cgroup, usage is pinned just short of the limit, but no OOM occurred even once.

memory.max      268435456   (256Mi)
memory.high     248299520   ← floor((64Mi + (256Mi - 64Mi) * 0.9) / 4096) * 4096
memory.current  246484992
memory.peak     249049088

--- memory.events ---
low 0
high 2666        ← number of times it was throttled for exceeding memory.high
max 0            ← memory.max was not reached
oom 0
oom_kill 0       ← no OOM kill occurred
oom_group_kill 0

--- memory.stat (excerpt) ---
file           235982848
inactive_file  235937792   ← nearly all of it is reclaimable inactive page cache
active_file        45056

At this point the working set is 246484992 - 235937792 = 10547200 (about 10 MiB). Here reclaimable page cache dominates usage while the working set stays small. Because high increased and max remained zero, this record demonstrates reclaim and throttling at memory.high; it does not directly test reclaim at memory.max.

On the other hand, doing the same with anonymous pages leads to an OOM kill. tail /dev/zero keeps allocating memory until it finishes reading its input, and on a node without swap the retained anonymous pages cannot normally be reclaimed.

kubectl run oomtest --image=busybox:1.37 --restart=Never \
    --overrides='{"spec":{"containers":[{"name":"app","image":"busybox:1.37",
      "command":["sh","-c","tail /dev/zero"],
      "resources":{"limits":{"memory":"64Mi"}}}]}}'

kubectl get pod oomtest -o jsonpath='{.status.containerStatuses[0].state.terminated}'

Output:

{"exitCode":137,"reason":"OOMKilled",...}

The kernel log has a line containing oom_memcg=. As described earlier, the kubelet's parser matches this format, so the cgroup can be extracted, and as a result no SystemOOM event is recorded.

oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-9be8306a....scope,
mems_allowed=0,oom_memcg=/docker/.../kubelet-kubepods-burstable-pod....slice,
task_memcg=/docker/.../cri-containerd-9be8306a....scope,task=tail,pid=4867,uid=0

Memory cgroup out of memory: Killed process 4867 (tail) total-vm:68880kB,
anon-rss:64768kB, file-rss:1280kB, shmem-rss:0kB, UID:0 pgtables:176kB oom_score_adj:984

kubectl get events -A --field-selector reason=SystemOOM

Output:

No resources found

Note that after an OOM kill the container's cgroup disappears, so the container_oom_events_total series itself goes away. It is the same even if you do not let it be recreated with restartPolicy: Never. To trace the number of OOMs afterward, use node_vmstat_oom_kill (per node) or kube_pod_container_status_last_terminated_reason.

When the dedicated test cluster is no longer needed, remove it:

kind delete cluster --name memqos

Terminology and kernel references

Why this article distinguishes memcg OOM and global OOM

How this article names the two OOM kill paths

There are two paths by which a Pod's process gets killed for memory reasons, and they differ in both cause and blast radius. If you call both "OOM" you cannot tell them apart, so this article uses the following two names throughout.

Name in this article	Purpose	Triggering cause	What gets killed
memcg OOM	Enforcing a configured limit	An allocation charge fails against a container, Pod or `/kubepods.slice` `memory.max`, and reclaim cannot create enough headroom	Only processes inside the cgroup that hit its limit
global OOM	Keeping the whole system alive (protecting the node)	A node-side allocation cannot be satisfied, potentially within NUMA or cpuset constraints	Eligible processes within the allocation constraints

memcg OOM starts from a cgroup limit; global OOM starts from a node-side allocation failure. When a container is below its own limit, check both ancestor cgroup limits and node-side constraints.

The kernel documentation describes global OOM as a way to keep the rest of the system alive:

In order to save the rest of the system, it invokes the OOM killer

Source: Concepts overview — OOM killer.

The description of memory.max states:

This is the main mechanism to limit memory usage of a cgroup

Source: Memory Interface Files.

cgroup v2 separates limits (memory.max / memory.high) from protection (memory.min / memory.low). memcg OOM enforces a limit.

memcg OOM can happen even when the node has plenty of free memory, because the allocation cannot fit within the cgroup constraint after reclaim. Global OOM does not originate from a memcg limit; NUMA policies or cpuset restrictions can cause it even when memory remains elsewhere on the node.

Kubernetes also uses a cgroup limit for the /kubepods.slice hierarchy when Pod allocatable enforcement is enabled. Its memory.max is set to Capacity - kube-reserved - system-reserved (see below), which effectively reserves memory for non-Pod processes such as the kubelet and containerd. That is not the purpose of the kernel's memcg OOM; it is Kubernetes using the mechanism that way.

Note: These two names are not official Kubernetes terms. Kubernetes only exposes the termination reason OOMKilled, and it does not distinguish the paths (the reason is explained in Why OOMKilled cannot distinguish the paths).

Both are terms the kernel itself uses, and this article adopts them as they are. The two are used side by side in a kernel comment (mm/oom_kill.c, pagefault_out_of_memory()):
/*
 * The pagefault handler calls here because some allocation has failed. We have
 * to take care of the memcg OOM here because this is the only safe context without
 * any locks held but let the oom killer triggered from the allocation context care
 * about the global OOM.
 */
The actual distinction is whether the memcg field of struct oom_control is NULL. If it holds a value, the victim is chosen from within that cgroup; if NULL, from eligible processes under the allocation constraints (oom.h). is_memcg_oom() just checks this field.
  /* Memory cgroup in which oom is invoked, or NULL for global oom */
  struct mem_cgroup *memcg;
memcg stands for memory cgroup (memory control group), a short form of struct mem_cgroup in the code above. Variable and function names in the source (is_memcg_oom(), try_charge_memcg()) use the same spelling, and the kernel documentation also writes "Memory cgroup (memcg)" (Documentation/mm/hmm.rst).

This is how they appear in the kernel log:

Name in this article How it appears in the kernel log

memcg OOM constraint=CONSTRAINT_MEMCG, oom_memcg=<cgroup path>, Memory cgroup out of memory: Killed process ...

global OOM constraint=CONSTRAINT_NONE (or cpuset / memory-policy constraints), ,global_oom, Out of memory: Killed process ...

The prefix of the Killed process line directly indicates the path (oom_kill.c):
      oom_kill_process(oc, !is_memcg_oom(oc) ? "Out of memory" :
               "Memory cgroup out of memory");

Also note that eviction is a separate mechanism from an OOM kill. It is the kubelet, not the kernel, that removes the Pod, and it is observed as Evicted. It is covered below as well.

Closing thoughts

Separate the values used to display usage from those used to enforce limits and trigger eviction. That distinction narrows an OOM investigation to the relevant container, ancestor cgroup or node-side constraint. Then review requests against workload history and placement density, and review limits against the memory the workload actually needs.

References

Components and versions

This article assumes cgroup v2. With cgroup v1, the mapping between metrics and memory.stat fields is different.

The statements in this article target the following versions.

Component	Version	Repository	Role in this article
Linux kernel	v6.12	https://github.com/torvalds/linux	cgroup v2 memory control and the OOM killer itself; the OOM message format in kernel logs
Kubernetes (kubelet)	v1.37.1	https://github.com/kubernetes/kubernetes	Eviction decisions, `oom_score_adj`, the `SystemOOM` event, counting for `container_oom_events_total`
Kubernetes (kube-scheduler)	v1.37.1	https://github.com/kubernetes/kubernetes	Exposing `kube_pod_resource_request` / `kube_pod_resource_limit` (`/metrics/resources`)
cAdvisor	v0.60.5	https://github.com/google/cadvisor	Embedded in the kubelet; exposes `container_*` metrics; computes the working set
containerd	v2.3.5	https://github.com/containerd/containerd	Decides that a container's termination reason is `OOMKilled`
node-exporter	v1.12.1	https://github.com/prometheus/node_exporter	Exposes `node_memory_*` (`/proc/meminfo`) and `node_vmstat_oom_kill` (`/proc/vmstat`)
kube-state-metrics	v2.20.0	https://github.com/kubernetes/kube-state-metrics	Exposes `kube_node_status_capacity` and `kube_pod_container_status_*`
Metrics Server	v0.8.1	https://github.com/kubernetes-sigs/metrics-server	Source of the values `kubectl top` reads
Vertical Pod Autoscaler	1.7.1	https://github.com/kubernetes/autoscaler	Computing and automatically applying request/limit recommendations
kind	v0.33.0	https://github.com/kubernetes-sigs/kind	Recorded examples use kindest/node:v1.37.0

Note: I assume containerd as the container runtime. With other runtimes such as CRI-O the meaning of cgroup v2 values is the same, but the OOMKilled detection logic and cgroup path naming differ.

I also assume the cluster has Prometheus, node-exporter and kube-state-metrics installed and that you can run PromQL. These are not part of Kubernetes itself, so if you do not have them, substitute kubectl top or read memory.stat directly on the node.

Metrics covered in this article

Grouped by emitting component. The takeaway of this article is that even metrics representing the same "memory usage" have different definitions if they come from different emitters, so when investigating, first check which component emits the value.

cAdvisor (kubelet `/metrics/cadvisor`)

cAdvisor embedded in the kubelet reads cgroups and exposes them. You do not need to run a separate cAdvisor.

Metric	Backing value in cgroup v2	Summary
`container_memory_usage_bytes`	`memory.current`	Usage including everything. What memcg OOM is evaluated against
`container_memory_working_set_bytes`	`memory.current - inactive_file`	The value Kubernetes treats as usage. The basis for `kubectl top` and eviction decisions
`container_memory_rss`	`anon` in `memory.stat`	Amount of anonymous pages. Its scope differs from true RSS
`container_memory_cache`	`file` in `memory.stat`	Page cache size (includes `shmem`)
`container_memory_total_active_file_bytes`	`active_file` in `memory.stat`	Active page cache. Included in the working set
`container_memory_total_inactive_file_bytes`	`inactive_file` in `memory.stat`	Inactive page cache. Subtracted from the working set
`container_memory_mapped_file`	`file_mapped` in `memory.stat`	Amount of `mmap`ed files
`container_spec_memory_limit_bytes`	`memory.max`	The container's memory limit. Exposed as 0 when `memory.max` is `max` (no limit)
`container_pressure_memory_waiting_seconds_total`	`total` of `some` in `memory.pressure`	Cumulative time (seconds) some processes were waiting for memory
`container_pressure_memory_stalled_seconds_total`	`total` of `full` in `memory.pressure`	Cumulative time (seconds) all processes were stalled waiting for memory

The whole node can be selected with id="/", all Pods on the node with id=~"/kubepods|/kubepods.slice", and a single Pod with container="", image="", pod!="". Only id="/" has no memory.current, so its usage is substituted with anon + file from memory.stat (kernel memory is not included). For details see Values the kubelet uses for eviction decisions.

Metric	Emitted by	Summary
`container_oom_events_total`	kubelet (the name says `container_` but it is not cAdvisor)	Number of OOM kills parsed from `oom-kill:` lines in the kernel log. Exposed on the same `/metrics/cadvisor`

kubelet (`/metrics/resource`)

An endpoint that exposes only CPU, memory and swap usage per container, per Pod and per node. It is a different path from /metrics/cadvisor, so to scrape it with Prometheus you need to add configuration.

Metrics Server gets values from here, and the official documentation also says "The metrics-server fetches resource metrics from the kubelets" (Resource metrics pipeline). It is not exclusive to Metrics Server, though, and Metrics Server's own README directs you to scrape this endpoint directly for monitoring purposes ("In such cases please collect metrics from Kubelet /metrics/resource endpoint directly.").

Metric	Summary
`container_memory_working_set_bytes`	Per-container working set (the same value as the metric of the same name on `/metrics/cadvisor`)
`pod_memory_working_set_bytes`	Per-Pod working set. Obtained from the Pod's cgroup
`node_memory_working_set_bytes`	Per-node working set

kubelet (`/metrics`)

Metric	Summary
`kubelet_node_name`	A gauge whose value is always `1` (the info-metric pattern). The node name is in the `node` label, so you can bring `node` into the `container_*` metrics, which have no node label, with `on(instance) group_left(node)`
`kubelet_memory_qos_node_memory_min_bytes`	Configured `memory.min` protection budget for Guaranteed Pods (MemoryQoS; ALPHA)
`kubelet_memory_qos_node_memory_low_bytes`	Configured `memory.low` protection budget for Burstable Pods (when MemoryQoS is enabled. ALPHA)

node-exporter

Turns the contents of /proc directly into metrics. It knows nothing of Kubernetes concepts.

Metric	Source	Summary
`node_memory_MemTotal_bytes`	`/proc/meminfo`	Total physical memory on the node
`node_memory_MemAvailable_bytes`	`/proc/meminfo`	The kernel's estimate of allocatable memory. Counts reclaimable page cache as free
`node_memory_Cached_bytes`	`/proc/meminfo`	Page cache size (includes tmpfs / shmem)
`node_memory_Shmem_bytes`	`/proc/meminfo`	The tmpfs / shared memory part of the above
`node_vmstat_oom_kill`	`/proc/vmstat`	Number of OOM kills on the node. Includes both memcg OOM and global OOM

kube-state-metrics

Converts Kubernetes API objects (spec / status) into metrics. It does not handle actual resource usage.

Metric	Summary
`kube_node_status_capacity{resource="memory"}`	The node's Capacity. The denominator for evaluating eviction thresholds
`kube_node_status_allocatable{resource="memory"}`	The amount assignable to Pods. The upper bound the scheduler uses
`kube_pod_container_status_last_terminated_reason`	The last termination reason (`reason="OOMKilled"` detects an OOM kill). No series appears for a container that has never terminated
`kube_pod_container_status_restarts_total`	Container restart count
`kube_pod_container_resource_requests`	The request written in the manifest (per container)
`kube_pod_container_resource_limits`	The limit written in the manifest (per container)

kube-scheduler (`/metrics/resources`)

Exposed on the secure port (default 10259) at /metrics/resources. It is a different endpoint from /metrics.

Metric	Summary
`kube_pod_resource_request{resource="memory", unit="bytes"}`	The sum of requests per Pod. No series is emitted for a Pod whose value is 0
`kube_pod_resource_limit{resource="memory", unit="bytes"}`	The sum of limits per Pod

mount	LRU	Where it shows up in `memory.stat`
`emptyDir` with `medium: Memory` (`noswap`)	unevictable	`unevictable`
Any other tmpfs / shared memory	anon LRU	`inactive_anon` / `active_anon`

Name in this article	How it appears in the kernel log
memcg OOM	`constraint=CONSTRAINT_MEMCG`, `oom_memcg=<cgroup path>`, `Memory cgroup out of memory: Killed process ...`
global OOM	`constraint=CONSTRAINT_NONE` (or cpuset / memory-policy constraints), `,global_oom`, `Out of memory: Killed process ...`