DEV Community: Ganesh Viswanathan

Compute Containers

Ganesh Viswanathan — Fri, 02 Jan 2026 19:20:11 +0000

Kata Containers is a secure container runtime that runs each container inside a lightweight virtual machine, combining the isolation of VMs with the speed and developer experience of containers. It enables both speed and isolation for running containerized workloads.

Goal: strengthen multi-tenant isolation beyond what namespaces/cgroups alone can safely provide, especially against kernel-level escapes.

Approach: replace the usual low-level runtime (e.g., runc) with a runtime that boots a micro-VM per sandbox and runs the container inside it.

Result: similar workflow (docker, containerd, Kubernetes), but each workload gets its own guest kernel and userspace boundary.

How it started?

Started Dec 15 2017, via merger of two secure container efforts - Intel’s Clear Containers and Hyper.sh’s runV both of which used hardware virtualization to isolate containers inside lightweight virtual machines.

Part of OpenStack (OpenInfra) Foundation, positioned as a community-driven way to blend VM-grade isolation with container workflows.

Default runtimes such as runc already offer strong performance and a smooth developer experience. However, in multi-tenant or untrusted workload scenarios, a simple container boundary may not provide adequate isolation since all containers share the same host kernel. Kata Containers introduces an additional layer of defense by leveraging hardware virtualization to isolate each workload. This design makes it significantly more difficult for a compromised container to escape or impact other containers and the host system.

Description

Prior - software isolation between containers

With Kata - hardware (hypervisor driven) isolation between containers

At a high level, a Kata deployment introduces:

Hypervisor layer: QEMU, Cloud Hypervisor, or Firecracker, using KVM to create a minimal VM per pod or container.

Guest assets: a trimmed guest kernel and rootfs optimized for fast boot and small memory footprint.

Kata agent: runs inside the guest; receives commands from the host runtime to create namespaces, mounts, and processes for the container.

Runtime integration: an OCI-compatible runtime that containerd/Docker/Kubelet can call instead of runc.

Make creation of container faster: Prepare rootfs/images for container, and other side is to create sandbox for pod. Use hotplug to combine them together and start pod.

Make small as a container: Don’t need to allocate real memory, use nvdimm to allocate virtual memory to device. Additionally use KSM to efficiently use memory; very high density than traditional VMs.

Networking: MacVTAP to bridge veth pair to TAP device. Additionally, apply tc rules to perform traffic transforms.

Storage: support volumes either from block-device or from 9pfs. With virtio-blk, when you have block-device storage on the host, performance is closer to BM / native performance than not using block-device on host. See https://dev.to/gansvv/storage-for-kata-containers-9pfs-vs-virtio-blk-f6n

CPU/Memory multi-tenancy:

Networking support for K8s multi-tenancy:

From the workload’s perspective, it is still a container; from the host’s perspective, it is a VM with a single-tenant container payload.

Containers

Firecracker doesn’t support emulation environments, so GPUs don’t work with Firecracker.

Outer runtime meant for lifecycling of VM. Inner runtime is OCP compliant, adheres to CNI/CSI/CRI, enables k8s networking and storage functionality.

Fortify applications running on cluster:

Kata has already been validated with GPU on BM:

Running a confidential environment: components should be trustworthy - kernel, guest image, memory are in a specific known state. Workload owner can then review the attestation report and decide actions, e.g., releasing secrets into that environment.

RATS reference arch and attestation:

Runtime class on pod spec - kata-qemu-gpu-snp for AMD system, tdx for tdx system, cca for arm system:

Confidential containers: encrypted/signed container images (host no longer sees the images), confidential containers for every pod.

Terminology

GDR, GPS: Meaning: GPUDirect RDMA (Remote Direct Memory Access) and GPUDirect Storage (GDS) are NVIDIA technologies for high-performance data transfer, enabling direct paths between GPUs and network/storage devices, bypassing the CPU to slash latency and boost bandwidth, crucial for AI/HPC; while GPUDirect RDMA connects GPUs to network interfaces (NICs) or storage adapters, GDS specifically links GPUs directly to local/remote storage (like NVMe-oF), both leveraging PCIe for speed.
Cold plug vs Hot plug VFIO: Context: VFIO devices are hot-plugged on a bridge by default. In a confidential compute environment, hot-plugging can compromise security. Kata supports cold-plugging of VFIO devices to a bridge-port, root-port, or switch-port. Meaning: In VFIO (Virtual Function I/O), "cold plug" means attaching a device (like a GPU) to a VM before the VM starts, while "hot plug" means attaching it while the VM is running, with "cold" being the standard, reliable way (passthrough) and "hot" being a more advanced, dynamic (but potentially complex) method for live device addition/removal

Good References

Kata launch:https://youtu.be/VupUc88FV9Q?si=KuL4fI7FEMkpOHbx
Nvidia - Kata Containers:https://youtu.be/a3HzBmPuw5g?si=xM96k3Ji4PoBaFfX
PodSandbox and container lifecycle mgmt: https://kubernetes.io/blog/2016/12/container-runtime-interface-cri-in-kubernetes/#:~:text=Before%20starting%20a%20pod%2C%20kubelet%20calls%20RuntimeService.RunPodSandbox%20to%20create%20the%20environment
https://youtu.be/odBdXHU8W9U?si=S7OK98RWs-griBGh

Funny: AI mixing computing terminology with mechnical engineering in one sentence

Ganesh Viswanathan — Mon, 29 Dec 2025 20:24:21 +0000

On 29 Dec 2025, Google search term:
cold plug vs hot plug VFIO

Yielded:
In VFIO (Virtual Function I/O), "cold plug" means attaching a device (like a GPU) to a VM before the VM starts, while "hot plug" means attaching it while the VM is running, with "cold" being the standard, reliable way (passthrough) and "hot" being a more advanced, dynamic (but potentially complex) method for live device addition/removal, *requiring specific hardware/software support to avoid system instability or data loss, unlike simple spark plug heat ranges which are about engine temperature management. *

Storage for Kata Containers - 9pfs vs virtio-blk

Ganesh Viswanathan — Mon, 29 Dec 2025 18:34:34 +0000

9pfs (Plan 9 File System via virtio-9p) and virtio-blk are two different methods for providing storage to virtual machines, differing primarily in their level of abstraction (file-level vs. block-level).

9pfs (virtio-9p)

9pfs exposes a specific directory on the host machine directly to the guest. It is often used for "shared folders" where the host and guest need simultaneous access to the same files.

Pros:

Elasticity: Storage is only consumed by actual files on the host; no need to pre-allocate a large image file.
Ease of Use: Useful for development, where you want to edit code on the host and run it immediately in the guest.

Cons:

Performance: Generally slower than block devices because every file operation must go through the 9P protocol and host system calls.
Compatibility: While Linux support is excellent, Windows support is limited and often requires third-party drivers.
Modern Alternative: virtio-fs is the successor to 9pfs in many modern QEMU setups, offering significantly better performance and POSIX compliance.

virtio-blk

This presents a virtual raw block device (a "hard drive") to the guest. The guest OS treats it like a physical disk and manages its own filesystem (e.g., ext4, XFS) on top of it.

Pros:

High Performance: Optimized for low latency and high throughput. Features like IOThread Virtqueue Mapping (introduced in QEMU 9.0) allow it to scale across multiple vCPUs efficiently.
Full Feature Support: Supports TRIM/Discard (to reclaim space), bootable partitions, and standard disk management tools.

Cons:

Isolation: The host cannot easily "see" or edit files inside the block device while the guest is running without risking corruption.
Fixed Size: Typically requires pre-allocation or using "sparse" image formats (like QCOW2) which can have their own performance trade-offs.

When to use which

Use 9pfs when:

You need to share a host directory into a guest for development, simple file exchange, or testing.
You value ease of sharing over raw performance and strict isolation.

Use virtio-blk when:

You are provisioning a root disk or data disk for a VM.
You care about performance, isolation, and standard disk semantics (snapshots, independent filesystems per VM).