writable Kubernetes volumes are security contracts now

There is a familiar Kubernetes hardening checklist that makes everyone feel briefly responsible.

Run as non-root. Drop Linux capabilities. Disable privilege escalation. Set readOnlyRootFilesystem: true. Add resource limits. Avoid host paths. Use admission policy so nobody has to remember all of this by hand.

Good checklist. Keep it.

Then the application fails because it cannot write to /tmp, a framework wants a cache directory, a sidecar needs a handoff path, an upload flow stages files before scanning them, or a model server wants somewhere to unpack artifacts. So we add an emptyDir, or a persistent volume, or a shared mount between containers.

And just like that, the supposedly immutable container has a writable filesystem again.

Not the root filesystem, sure. But still a writable filesystem. One the process can use to write scripts, unpack binaries, race another container, leave junk for a later step, or turn "temporary storage" into the most interesting part of the compromise.

This is the part of container hardening we have been hand-waving for years. We treated writable volumes as plumbing. Kubernetes v1.37 is a useful reminder that they are policy.

the old checklist was incomplete

readOnlyRootFilesystem: true is still one of the better low-drama controls in Kubernetes. It breaks lazy application assumptions. It makes persistence harder for an attacker. It forces teams to separate the image from runtime state.

But it has always had an escape hatch: workloads still need writable paths.

That escape hatch is not a footnote. It is where many real applications do their work:

• /tmp for framework scratch files
• caches for package managers, templates, compiled views, or ML runtime state
• upload staging directories
• sidecar exchange volumes
• init-container-to-app handoff paths
• plugin directories
• generated config
• logs when the app is not properly writing to stdout

Some of these paths should be boring scratch space. Some are intentionally shared. Some should never execute code. Some are explicitly designed to receive executable code, which is another way of saying they deserve a much louder review.

If all of those end up as "just mount an emptyDir", the security contract is missing the important sentence.

what Kubernetes v1.37 changes

Kubernetes v1.37 introduced two alpha storage-hardening features that look small and are not.

The first is bindMountOptions on volumeMounts. It lets a pod specify Linux bind mount flags such as noexec, nodev, and nosuid per volume mount. That means a container can mount a writable directory while still saying: files written here must not be executable, device files should not matter, and setuid/setgid bits should be ignored.

The second is emptyDir.mode. It lets the pod spec declare the Unix permission mode for an emptyDir, including the sticky-bit pattern people expect from a real /tmp, such as 01777.

Both are alpha in v1.37. They require feature gates. bindMountOptions also depends on runtime support for the CRI mount options field. This is not a "go turn it on everywhere by Friday" post.

The point is more important than the rollout status: Kubernetes is finally giving the pod spec vocabulary for writable mount intent.

Here is the shape of it:

apiVersion: v1
kind: Pod
metadata:
  name: hardened-scratch-example
spec:
  os:
    name: linux
  containers:
    - name: app
      image: example/app:latest
      securityContext:
        readOnlyRootFilesystem: true
        allowPrivilegeEscalation: false
      volumeMounts:
        - name: scratch
          mountPath: /tmp
          bindMountOptions:
            - noexec
            - nodev
            - nosuid
  volumes:
    - name: scratch
      emptyDir:
        mode: 01777

That is still YAML, yes. Kubernetes remains undefeated at producing YAML.

But this YAML says something that was previously buried in convention: this writable directory exists for scratch data, not for executable code, device files, or privilege tricks.

PV mount options were not enough

One subtle but important detail: this is not the same as PersistentVolume mountOptions.

PersistentVolume mount options operate at the storage layer. They are about how the volume is mounted by the storage system. The new bindMountOptions field applies to the bind mount inside the container.

That distinction matters because a pod can have multiple containers mounting the same volume for different reasons. The storage backend may be the same, but the intent is not.

An init container may need to write generated files. The main application may need to read them but never execute from that path. A sidecar may need a shared exchange directory. Another container may not need the mount at all.

Per-container mount policy is the interesting part.

If the only policy lives at the storage object, the pod loses nuance. If the policy lives at the volumeMount, the platform can say: this container gets this path with these constraints.

That is a much better security contract.

writable does not mean executable

The most useful default is also the most obvious one:

Most writable volumes should be non-executable.

If the application needs /tmp, a cache directory, or an upload staging path, why should a binary copied there be executable? If the answer is "because Linux defaults", that is not a good answer. It is just the absence of a decision.

The Kubernetes feature blog uses exactly this example: without noexec, a compromised process can use any writable volume to download or create a binary, mark it executable, and run it even when the root filesystem is read-only.

This does not make noexec magic. Interpreters exist. Shells exist. A process can still feed data to something already executable elsewhere. Security controls rarely delete entire attack classes by themselves.

But noexec does remove a very common and very convenient step. That matters. Good hardening is often about making the cheap path expensive.

The same logic applies to nodev and nosuid. Most application scratch paths have no business honoring device special files or setuid bits. If a workload needs those semantics, it should say so loudly.

emptyDir permissions were also a contract

emptyDir has always been a little too easy.

Need a writable directory? Add this:

volumes:
  - name: tmp
    emptyDir: {}

Done. Ship it.

The default behavior is convenient, but convenience is not the same as intent. If the directory behaves like shared /tmp, it probably needs the sticky bit so one user cannot delete another user's files. If it is an app-private cache, maybe it should not be world-writable at all. If it is a sidecar exchange path, the permissions should match the actual producers and consumers instead of whatever happened after an init container ran chmod.

That last workaround is worth calling out. Before emptyDir.mode, teams often used init containers to create directories and fix permissions. It worked, but it made the pod's storage policy procedural. You had to read the init container command to understand the contract.

An API field is better. It is visible to admission policy. It is visible to review. It is visible to humans who are tired and trying to understand why a pod is allowed to have a writable mount.

classify mounts by intent

The practical move is not to wait for every managed Kubernetes provider to make these alpha fields boring. The practical move is to start classifying writable mounts now.

I would use buckets like these.

scratch temp

This is the ordinary /tmp case: framework scratch files, temporary decompression, request-local intermediate data, things that should not survive the pod.

Default posture:

bindMountOptions:
  - noexec
  - nodev
  - nosuid

Use emptyDir.mode: 01777 only when it is really shared temp with multiple users or containers that need normal /tmp behavior. If it is private to one container and one user, consider narrower permissions.

shared tmp

Shared temp is where people get casual and attackers get comfortable.

If multiple containers or multiple users share the directory, say that explicitly. Use the sticky bit when the model is "everyone can create files, but cannot delete each other's files." Do not let world-writable become the default just because it made the first test pass.

Also ask why it is shared. Many "shared temp" volumes are actually lazy IPC, lazy logging, or lazy configuration handoff.

app cache

Most caches should not be executable.

Template caches, dependency caches, image transformation caches, model runtime caches, and web framework caches are usually data. If a runtime insists on generating executable code into a cache, that should be documented as an exception, not discovered during an incident.

This is where admission policy can help later: executable writable cache directories should need a reason.

uploads and quarantine

Upload staging directories should be treated as hostile by default.

That means non-executable, no device files, no suid semantics, and ideally separated from the final storage path. If the application scans or transforms uploads, make the handoff stages explicit. "Uploaded file exists somewhere under /tmp" is not a design.

This matters even more when sidecars are involved. The scanner and the app may need different views of the same volume.

plugin and executable drop zones

Some systems really do need writable executable directories.

Plugin systems. Build workers. Notebook environments. CI runners. Developer sandboxes. Some ML serving setups that materialize runtime code or native extensions.

Fine. But call them what they are: privileged exceptions.

An executable writable mount is not automatically wrong. It is just not ordinary storage. It should have an owner, a threat model, a retention story, a review path, and ideally a much smaller blast radius than the main application container.

If a team cannot explain why a writable mount must be executable, it probably should not be executable.

sidecar handoff volumes

Sidecars are where per-container mount policy becomes more than a nice detail.

Imagine an init container writes generated config to a volume. The app reads it. A reload sidecar watches it. Those containers do not need identical mount semantics.

Or imagine an upload scanner. The app writes incoming files. The scanner reads and marks results. The app consumes only approved outputs. Again, different containers have different intent.

The old pattern treated the volume as the policy unit. The better pattern treats each mount as the policy unit.

admission policy should ask for reasons

The future I want here is not every developer memorizing mount flags.

The future is admission policy that understands writable storage intent.

For example:

• Writable mounts must declare a purpose label or annotation.
• Writable mounts default to noexec, nodev, and nosuid where supported.
• Executable writable mounts require an explicit exception.
• Shared emptyDir volumes require a declared permission mode.
• Upload staging mounts cannot be executable.
• Plugin directories must be isolated from the main app container unless approved.

This is the same pattern we already use for capabilities, host networking, privileged pods, and service account tokens. Make the common safe thing easy. Make the dangerous thing possible but explicit.

The alpha fields are not required to start this work. You can inventory pod specs today. You can classify mounts today. You can add annotations today. You can update review templates today. You can teach platform teams to stop saying "it only writes to /tmp" as if that ends the conversation.

When the fields become stable enough for your environment, the policy vocabulary is already there.

migration advice

I would not start by enabling feature gates in production.

I would start with an inventory:

1. List every workload with a writable volume mount.
2. Record the mount path, volume type, containers that mount it, and whether it is shared.
3. Assign a purpose: scratch, shared temp, cache, upload staging, generated config, plugin/executable, sidecar handoff, or "unknown".
4. Treat "unknown" as a bug in the platform contract.
5. Add admission warnings for new writable mounts without a purpose.
6. Test bindMountOptions and emptyDir.mode on non-critical clusters once your Kubernetes version, feature gates, and runtime support line up.
7. Require a written reason for executable writable mounts.

The important artifact is not the YAML field. It is the classification.

Once a team has that inventory, the v1.37 features become a way to encode decisions instead of a last-minute compliance retrofit. Without that inventory, the fields become another checkbox nobody trusts.

the boring field is the lesson

I like features like this because they expose where the old mental model was too simple.

Read-only root filesystem was never the same thing as read-only workload state. It was a good control with an obvious exception. We just spent years pretending the exception was operational trivia.

Writable pod storage is not trivia. It is where applications stage data, share state, receive hostile input, and sometimes execute code. That makes it part of the workload security contract.

Kubernetes v1.37 did not invent that contract. It just made the missing words harder to ignore.

references

• Kubernetes v1.37: Hardening Container Storage with Bind Mount Options and EmptyDir Permissions
• Set Bind Mount Options on Volume Mounts
• Kubernetes volumes documentation
• KEP-5855: Allow bind mount options on volumeMounts
• KEP-5502: EmptyDir Volume Permission Mode
• Kubernetes v1.37 release announcement
• NSA/CISA Kubernetes Hardening Guidance v1.2

To test my projects, I use Railway. If you want $20 USD to get started, use this link.