Kubernetes OCI image volumes hit stable in v1.36: ship model weights without rebuilding images

Summary. The image volume source went stable in Kubernetes v1.36, enabled by default, ending a graduation that began as alpha in v1.31 in August 2024 and passed through beta in v1.33 (April 2025) before becoming on-by-default in v1.35 (December 2025). It lets a pod mount the contents of an OCI object, read-only, at a mount path, with no rebuild of the container image that consumes it. For teams serving models, that separates a 40 GB weights artifact from a 500 MB model-server image on independent release cycles. KEP-4639 names this case directly: mounting "large language model weights or machine learning model weights in a pod alongside a model-server". You need containerd v2.1 or later, or CRI-O, and you need to understand three constraints the announcements underplay. Storage cost is not the win. At Amazon ECR's $0.10 per GB-month, registry deduplication means baking weights into images was never costing you what you assumed. The win is decoupling, pull behaviour, and rollback granularity.

What actually shipped, and when

The timeline matters, because a lot of published material about this feature is describing an older version of it.

Kubernetes version	Stage	Default	What changed
v1.31 (Aug 2024)	Alpha	Off	image volume source introduced under the ImageVolume feature gate
v1.33 (Apr 2025)	Beta	Off	subPath and subPathExpr support added
v1.35 (Dec 2025)	Beta	On	Enabled by default; ImageVolumeWithDigest lands as alpha
v1.36	Stable	On	GA; feature gate no longer needed

The Kubernetes documentation for image volumes now carries the header "FEATURE STATE: Kubernetes v1.36 stable". The v1.35 release announcement puts the intermediate step plainly: "The image volume type has been in beta since v1.33 and is enabled by default in v1.35."

One wrinkle worth knowing if you go reading the source material. KEP-4639's own metadata still lists beta at v1.34, which contradicts both the feature-gate table and the v1.33 beta announcement. The KEP's implementation history explains the drift: the beta target moved from v1.33 to v1.34, then to "beta (enabled by default) in v1.35", and finally GA was retargeted to v1.36 in January 2026. The documentation and the release blogs are correct. The KEP metadata is stale.

Sascha Grunert of Red Hat, who co-authored KEP-4639 and wrote the Kubernetes release announcements for the feature, marked the beta graduation this way: "SIG Node is proud and happy to deliver this feature graduation as part of Kubernetes v1.33."

The YAML

This is the canonical example from the Kubernetes documentation, unchanged since beta:

apiVersion: v1
kind: Pod
metadata:
  name: image-volume
spec:
  containers:
  - name: shell
    command: ["sleep", "infinity"]
    image: debian
    volumeMounts:
    - name: volume
      mountPath: /volume
  volumes:
  - name: volume
    image:
      reference: quay.io/crio/artifact:v2
      pullPolicy: IfNotPresent

Applied to model serving, the shape becomes:

apiVersion: v1
kind: Pod
metadata:
  name: model-server
spec:
  containers:
  - name: server
    image: registry.example.com/vllm-server:1.9.2   # never rebuilt for a new model
    args: ["--model", "/models/weights"]
    volumeMounts:
    - name: weights
      mountPath: /models
  volumes:
  - name: weights
    image:
      reference: registry.example.com/models/llama-70b:2026-06-01
      pullPolicy: IfNotPresent

The server image and the weights artifact now have separate tags, separate release cadences, and separate rollbacks. Changing the model is a pod spec edit, not a build.

The API surface is two fields

ImageVolumeSource has exactly two fields, reference and pullPolicy. That is the whole API.

reference is the OCI object to mount. The current documentation describes it as optional, "to allow higher level config management to default or override container images in workload controllers like Deployments and StatefulSets", which is a useful detail if you template these.

pullPolicy accepts Always, Never, or IfNotPresent, and defaults to Always if the :latest tag is specified, or IfNotPresent otherwise. The failure semantics of each are specific, and they are the part that will page you:

pullPolicy	Behaviour	Failure mode
Always	The kubelet always attempts to pull the reference	If the pull fails, the kubelet sets the Pod to Failed
IfNotPresent	The kubelet pulls if the reference isn't already present on disk	The Pod becomes Failed if the reference isn't present and the pull fails
Never	The kubelet never pulls and only uses a local image or artifact	The Pod becomes Failed if any layers aren't already present locally, or if the manifest isn't cached
Default with :latest	Always	As Always
Default without :latest	IfNotPresent	As IfNotPresent

Never pin a weights volume to a mutable :latest tag. That combination gives you Always, which means every pod start is a registry round trip for tens of gigabytes, and a registry hiccup is a Failed pod rather than a slow one.

Three constraints the announcements underplay

It is read-only, permanently

The documentation is unambiguous: the OCI object "gets mounted in a single directory (spec.containers[*].volumeMounts[*].mountPath) and will be mounted read-only". The KEP explains the reasoning and the roadmap: "For security reasons, ro (read-only) option is set by default. Having rw (read-write) support will require a follow-up enhancement."

For weights this is correct behaviour and not a limitation. For anything that expects to write into its data directory, including some model servers that build an index or cache on first load, this will fail. Check before you migrate. Point the writable cache at an emptyDir and the weights at the image volume.

One correction if you read the v1.31 alpha announcement: it stated the volume is mounted non-executable (noexec). That requirement was dropped in June 2025 during the beta retarget. Do not design around it.

The volume re-resolves on pod recreation

This is the behaviour that surprises people, and the docs state it directly: "The volume gets re-resolved if the Pod gets deleted and recreated, which means that new remote content will become available on Pod recreation. A failure to resolve or pull the image during Pod startup will block containers from starting and may add significant latency. Failures will be retried using normal volume backoff and will be reported on the Pod reason and message."

Read that twice if you are running a mutable tag. A pod rescheduled at 03:00 onto a fresh node can come back with different weights than its neighbours. That is a silent correctness bug in a serving fleet, not a deployment event. Pin to an immutable tag or a digest.

This is also why ImageVolumeWithDigest, alpha in v1.35, matters. It records the resolved image digest in the pod status, so you can tell which bytes a running pod actually has.

fsGroupChangePolicy does nothing here

The docs note that spec.securityContext.fsGroupChangePolicy "has no effect on this volume type". If your model server runs as a non-root user and expects group ownership fixups on the mount, verify permissions on a real cluster before you roll it out. The AlwaysPullImages admission controller, for what it is worth, does work on this volume source the same way it works for container images.

The cost argument is not the one you have been told

Here is where most write-ups on this feature get it wrong, and it is worth being precise because the wrong version is expensive to act on.

The usual claim is that baking model weights into a container image wastes registry storage, because every rebuild stores another copy. That is not how OCI registries work. Registries deduplicate by layer digest. If your weights sit in a layer that does not change between builds, thirty tagged builds of your server image reference the same weights layer, and the registry stores those bytes once.

Run the arithmetic at Amazon ECR's published price of $0.10 per GB-month for private repository storage. A 40 GB weights layer shared across thirty tags costs $4.00 a month, not $120. Splitting it into a separate artifact costs the same $4.00. Storage is a wash. In-region transfer is free anyway: data moving between ECR and EC2, Lambda, App Runner, or Fargate in the same Region carries no charge.

So if the storage saving is roughly zero, why do this at all? Four reasons that are actually real.

Rebuild coupling disappears. Patching your server image for a CVE means a rebuild. If the weights are in that image, every rebuild has to move 40 GB through your CI, and every model change has to rebuild the server. The two artifacts have completely different change rates: your server image might ship weekly, your weights quarterly. Coupling them means the fast one drags the slow one.

Layer invalidation stops being a trap. The dedup that saves you storage only holds while the weights layer is byte-identical. Reorder a COPY in the Dockerfile, change a base image, or rebuild in a way that alters layer digests, and you have just written another 40 GB to the registry. The saving is real but fragile, and it depends on Dockerfile discipline that nobody audits.

Rollback gets granular. With weights in the image you roll back both or neither. Split, you can roll the model back to last quarter's tag while keeping this week's security-patched server.

Init containers and startup scripts go away. The v1.35 release announcement makes this the headline benefit: the feature "lets you package and deliver data-only artifacts such as configs, binaries, or machine learning models using standard OCI registry tools", and "you can fully separate your data from your container image and remove the need for extra init containers or startup scripts".

That last one is the honest sell. Most teams serving models today have an init container that curls weights from S3 into an emptyDir. That init container is code you wrote, with credentials you rotate, retry logic you maintain, and no layer caching. Deleting it is the win.

Concern	Weights in the image	Weights as an image volume
Registry storage	Deduplicated by layer; $4.00/month for 40 GB at ECR list	The same $4.00/month
Server CVE patch	Rebuild moves 40 GB through CI	Rebuild moves 500 MB
Model swap	Full image rebuild and redeploy	Pod spec tag change
Rollback	Server and model roll back together	Independent tags
Node disk	One combined image	Server image plus mounted artifact
Credentials	Registry pull secret	Registry pull secret; no S3 keys in an init container

What you need on the nodes

The feature is only as available as your runtime. The v1.35 announcement is explicit: "Please note that using this feature requires a compatible container runtime, such as containerd v2.1 or later." CRI-O has supported the initial feature since v1.31 and added beta support in v1.33.

The failure mode on an old runtime is described in the KEP and is not obvious from the pod events: "Pods using the new VolumeSource combined with a not supported container runtime version will fail to run on the node, because the Mount.host_path field is not set for those mounts."

If you run a managed cluster, check the runtime version before the Kubernetes version. A control plane on v1.36 with nodes on containerd v1.7 gives you an API that accepts the pod spec and nodes that cannot honour it.

Mounting one subdirectory with subPath

The beta release added the ability to mount a subdirectory rather than the whole artifact, which matters when one artifact carries several model variants. From the v1.33 announcement: "This allows end-users to mount a certain subdirectory of an image volume, which is still mounted as readonly (ro). This means that non-existing subdirectories cannot be mounted by default. As for other subPath and subPathExpr values, Kubernetes will ensure that there are no absolute path or relative path components part of the specified sub path. Container runtimes are also required to double check those requirements for safety reasons."

  volumeMounts:
  - name: weights
    mountPath: /models
    subPath: llama-70b-int4

A non-existent subdirectory fails at container creation, surfaced through kubelet events. That is a deployment-time failure rather than a silent empty mount, which is the behaviour you want.

Watching it in production

The v1.33 beta announcement documents three kubelet metrics for image volumes: kubelet_image_volume_requested_total ("Outlines the number of requested image volumes"), kubelet_image_volume_mounted_succeed_total ("Counts the number of successful image volume mounts"), and kubelet_image_volume_mounted_errors_total ("Accounts the number of failed image volume mounts").

Check the exact names against your own cluster before you build alerts on them. KEP-4639's metadata lists a different naming scheme for the same three counters, without the kubelet_ prefix, and the two have not been reconciled in the source material. Scrape the endpoint and read what is actually there.

The alert that matters is on the error counter. A rising errors_total with healthy pods means nodes are failing to pull weights and backing off, which shows up as capacity that never arrives rather than as a crash.

India-specific considerations

For teams running model-serving clusters from Indian regions, two things follow from the mechanics above.

Registry locality does most of the work on cost. Transfer between ECR and compute in the same Region is free, so a weights artifact in an India region pulled by nodes in that Region carries the $0.10 per GB-month storage charge and no egress. The expensive pattern is a registry in one Region serving nodes in another, which turns every node scale-up into cross-Region transfer of tens of gigabytes. If you serve from Mumbai, publish to Mumbai.

The second is a data governance point rather than a cost one. A weights artifact is not personal data, but fine-tuned weights derived from customer records sit closer to it than most teams admit, and adapters trained on support transcripts are a live question under the Digital Personal Data Protection Act 2023 (DPDP). Image volumes make this easier to reason about, not harder: the artifact has a digest, a registry, and an access policy, rather than living inside a container image that also contains your application. Treat the weights registry as a system holding derived data and apply the same access controls you apply in production.

Teams building on this will find the related mechanics in our guides to gang scheduling for GPU jobs with the Workload API, in-place pod resize for rightsizing, and pod certificates and constrained impersonation in v1.35.

Migrating off an init container

The realistic path from a curl-from-S3 init container is four steps, and only the first is interesting.

Package the weights as an OCI artifact and push them to your registry. Pin the tag to something immutable, dated or digest-based, never :latest. Point the volume at it with pullPolicy: IfNotPresent. Delete the init container, the bucket credentials it used, and the retry logic you wrote.

Test one thing before you roll out: kill a pod and let it reschedule onto a node that has never pulled the artifact. That path exercises the cold pull, the backoff, and the startup latency the docs warn "may add significant latency". Measure it, then set your readiness probe and pod disruption budget against the number you measured rather than the number you hoped for.

The feature is stable, on by default, and the API is two fields. The work is in your tags.

FAQ

Is the OCI image volume source production ready in 2026?

Yes. The Kubernetes documentation lists image volumes as stable and enabled by default since v1.36, after alpha in v1.31 and beta from v1.33. The feature gate is no longer required. The constraint is the node runtime, which must be containerd v2.1 or later, or a supporting CRI-O version.

Can I write to an image volume?

No. The Kubernetes documentation states the OCI object is mounted read-only, and KEP-4639 confirms the read-only option is set by default for security reasons, with read-write support requiring a follow-up enhancement. Mount an emptyDir alongside it for any cache or index your model server needs to write during startup.

Does using an image volume reduce my registry storage bill?

Usually not. OCI registries deduplicate by layer digest, so an unchanged weights layer shared across many image tags is stored once. At Amazon ECR's $0.10 per GB-month, a 40 GB weights layer costs about $4.00 either way. The real gains are decoupled rebuilds, granular rollback, and deleting init containers.

What happens if the registry is unavailable when a pod starts?

The pod fails rather than starting degraded. With pullPolicy Always the kubelet sets the Pod to Failed if the pull fails. With IfNotPresent the pod fails only when the reference is absent locally and the pull fails. Kubernetes retries using normal volume backoff and reports the reason on the pod.

Should I use the latest tag for a weights volume?

No. pullPolicy defaults to Always when the latest tag is specified, and IfNotPresent otherwise. Always means every pod start pulls tens of gigabytes from the registry. Worse, the docs confirm volumes re-resolve on pod recreation, so a rescheduled pod could load different weights than its neighbours.

Can I mount just one model from a multi-model artifact?

Yes, using subPath or subPathExpr, added in the v1.33 beta. Kubernetes rejects absolute paths and relative path components in the sub path, and container runtimes double check this. If the subdirectory does not exist, runtimes fail at container creation and report it through kubelet events rather than mounting an empty directory.

What minimum container runtime version do I need?

The Kubernetes v1.35 release announcement states the feature requires a compatible runtime such as containerd v2.1 or later. CRI-O supported the initial feature from v1.31 and added beta support in v1.33. On an unsupported runtime, pods fail on the node because the Mount.host_path field is not set for those mounts.

How do I know which weights a running pod actually loaded?

Pin to an immutable tag or digest, and watch ImageVolumeWithDigest, which landed as alpha in Kubernetes v1.35. It records the resolved image digest in the pod status, so you can confirm the exact bytes a running pod mounted rather than inferring it from a tag that may have moved underneath you.

How eCorpIT can help

eCorpIT builds and operates Kubernetes platforms for teams serving models in production. Our senior engineering teams handle the parts of this migration that are not in the documentation: packaging weights as OCI artifacts, setting registry locality so node scale-ups do not cross Regions, replacing init-container download scripts and their long-lived credentials, and validating that your node runtime actually supports the API your control plane accepts. We design these systems aligned with DPDP 2023 requirements where fine-tuned weights carry derived customer data. If you are serving models on Kubernetes and your model release still means an image rebuild, talk to our team.

References

1. Kubernetes documentation, Use an Image Volume With a Pod
2. Kubernetes v1.33: Image Volumes graduate to beta
3. Kubernetes 1.31: Read Only Volumes Based On OCI Artifacts (alpha)
4. KEP-4639: OCI VolumeSource, kubernetes/enhancements
5. VolumeSource: OCI Artifact and/or Image, enhancements issue #4639
6. Kubernetes v1.35: Timbernetes (The World Tree Release)
7. Kubernetes v1.35 Sneak Peek
8. Kubernetes v1.31: Elli release announcement
9. Amazon ECR pricing
10. Amazon ECR FAQs
11. Understanding data transfer costs for AWS container services
12. KEP-4639 writeable, container isolated volume content, containerd issue #11853
13. Support Kubernetes image volume mounting by using standard OCI layer media types, kitops issue #1144

Last updated: 16 July 2026.