DEV Community: James Joyner

Keep Docker Engine as Your Kubernetes Runtime on Ubuntu with cri-dockerd

James Joyner — Tue, 04 Aug 2026 23:35:35 +0000

Sometimes you genuinely want Docker Engine as the Kubernetes node runtime — a team standardized on the Docker CLI/API for tooling, an image-build box that doubles as a node, or a legacy playbook you can't rewrite yet. Since dockershim was removed in Kubernetes 1.24, that path now runs through cri-dockerd, an open-source CRI adapter maintained by Mirantis. Here's how to set it up on Ubuntu, and an honest note on whether you should.

Should you? A quick reality check

For most people the answer is no — use containerd (see the previous post). Reach for cri-dockerd only when you have a concrete reason:

Existing automation that talks to the Docker socket on the node.
You want docker build and the kubelet on the same host sharing one image store.
A vendor or product that still assumes Docker Engine as the runtime.

If none of those apply, containerd is less to install and less to break.

1. Install Docker Engine on Ubuntu

Use Docker's official repo, not the docker.io package, so you get current Engine:

sudo apt-get update
sudo apt-get install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" \
  | sudo tee /etc/apt/sources.list.d/docker.list >/dev/null

sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io

Set Docker's cgroup driver to systemd — on Ubuntu's cgroup v2 this must match the kubelet or the node will not stay Ready:

sudo mkdir -p /etc/docker
cat <<'EOF' | sudo tee /etc/docker/daemon.json
{ "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": {"max-size": "100m"} }
EOF
sudo systemctl restart docker
sudo systemctl enable docker

2. Install cri-dockerd

Grab the latest release for your architecture:

VER=0.3.15   # check github.com/Mirantis/cri-dockerd/releases for current
ARCH=$(dpkg --print-architecture)
curl -fsSLo cri-dockerd.deb \
  "https://github.com/Mirantis/cri-dockerd/releases/download/v${VER}/cri-dockerd_${VER}.3-0.ubuntu-noble_${ARCH}.deb"
sudo dpkg -i cri-dockerd.deb

sudo systemctl enable --now cri-docker.socket
systemctl status cri-docker.socket

The CRI socket it exposes is the value you'll hand to kubeadm:

unix:///var/run/cri-dockerd.sock

3. Point kubeadm at the cri-dockerd socket

Everything else in the node setup (kernel modules, sysctl, swap off, the pkgs.k8s.io repo) is identical to the containerd post — only the CRI endpoint changes:

sudo kubeadm init \
  --cri-socket=unix:///var/run/cri-dockerd.sock \
  --pod-network-cidr=10.244.0.0/16

Worker joins must carry the same flag:

sudo kubeadm join <cp>:6443 \
  --token <token> --discovery-token-ca-cert-hash sha256:<hash> \
  --cri-socket=unix:///var/run/cri-dockerd.sock

4. Verify

kubectl get nodes -o wide
# CONTAINER-RUNTIME now reads docker://<engine-version>

The gotcha nobody warns you about

With cri-dockerd, images pulled by the kubelet and images you docker pull live in the same store, which is convenient — but docker ps will show a lot of Kubernetes-managed containers you didn't start. Use crictl (CRI-aware) for cluster containers and reserve docker for images you manage by hand, so you don't accidentally docker rm something the kubelet owns.

sudo crictl --runtime-endpoint unix:///var/run/cri-dockerd.sock ps

If the node comes up NotReady or pods can't pull, it's nearly always the cgroup-driver mismatch or the socket path — both covered in the Kubernetes pod-startup error hub and the Docker runtime error guides.

Next in the series: dropping the whole node dance and running Kubernetes inside Docker with kind for local development.

Set Up a Kubernetes Node on Ubuntu 24.04 with containerd (the Docker-Compatible Runtime)

James Joyner — Fri, 31 Jul 2026 16:28:05 +0000

This is the mainstream way to run Kubernetes "with Docker" on Ubuntu in 2026: containerd as the runtime (the same engine Docker uses under the hood), your Docker-built images running unchanged. Here's a clean kubeadm node bring-up on Ubuntu 24.04 (Noble) with the sharp edges called out.

Run everything below as root or with sudo. This sets up a single control-plane node; join workers with the kubeadm join command printed at the end.

1. Kernel prerequisites

Kubernetes needs bridged traffic to hit iptables and IP forwarding on:

cat <<'EOF' | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter

cat <<'EOF' | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
sudo sysctl --system

Disable swap — the kubelet refuses to start with swap on by default:

sudo swapoff -a
sudo sed -i.bak '/\bswap\b/s/^/#/' /etc/fstab

2. Install containerd

sudo apt-get update
sudo apt-get install -y containerd
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml >/dev/null

The one edit that matters on Ubuntu. Ubuntu 22.04+ uses cgroup v2, so containerd and the kubelet must both use the systemd cgroup driver or the node flaps between Ready/NotReady:

sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl restart containerd
sudo systemctl enable containerd

(This mismatch is common enough that I gave it its own post later in the series.)

3. Install kubeadm, kubelet, kubectl

The old apt.kubernetes.io repo was retired — use pkgs.k8s.io. Pin the minor version you want (1.30 shown):

sudo apt-get install -y apt-transport-https ca-certificates curl gpg
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key \
  | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' \
  | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl   # don't let an unattended upgrade skip a minor

4. Initialize the control plane

Pick a pod CIDR that matches your CNI. This uses Flannel's default:

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

# set up kubectl for your user
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

5. Install a CNI, then verify

kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

kubectl get nodes -o wide
# STATUS should go Ready within a minute; CONTAINER-RUNTIME shows containerd://1.7.x

For a single-node lab, let workloads schedule on the control plane:

kubectl taint nodes --all node-role.kubernetes.io/control-plane-

6. Prove Docker images run

kubectl create deployment web --image=nginx:1.27
kubectl expose deployment web --port=80
kubectl get pods -o wide     # Running, on your containerd node

That nginx image was built by Docker. It runs on containerd without translation because it's an OCI image — which is the whole point of this series.

When it doesn't come up

The usual first-node failures are the kubelet not starting (almost always swap or the cgroup driver), or pods stuck pending (no CNI yet). I keep the specific fixes here: Kubernetes pod-startup errors and the Kubernetes troubleshooting stack.

Next: if you specifically need Docker Engine as the runtime rather than bare containerd, that's cri-dockerd.

Run Kubernetes in Docker on Ubuntu for Local Development

James Joyner — Sun, 26 Jul 2026 18:26:50 +0000

There's a delightfully literal answer to "Kubernetes with Docker": kind — Kubernetes IN Docker. Each node is a Docker container running a full Kubernetes node image. On an Ubuntu workstation it gives you a real, throwaway, multi-node cluster in about 30 seconds. It's my default for local dev and for CI.

Prerequisites on Ubuntu

You need Docker Engine and kubectl. If you don't have Docker yet:

sudo apt-get update && sudo apt-get install -y docker.io
sudo usermod -aG docker $USER && newgrp docker   # run docker without sudo

Install kind (single static binary):

curl -fsSLo ./kind https://kind.sigs.k8s.io/dl/latest/kind-linux-amd64
chmod +x ./kind && sudo mv ./kind /usr/local/bin/kind
kind version

A one-command cluster

kind create cluster --name dev
kubectl cluster-info --context kind-dev
docker ps    # you'll see a dev-control-plane container — that's your node

kind wrote a kubeconfig context for you. Tear the whole thing down just as fast:

kind delete cluster --name dev

A realistic multi-node cluster

Most bugs only show up with more than one node (scheduling, affinity, PodDisruptionBudgets). Define it in a config file:

# kind-cluster.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    kubeadmConfigPatches:
      - |
        kind: InitConfiguration
        nodeRegistration:
          kubeletExtraArgs:
            node-labels: "ingress-ready=true"
    extraPortMappings:
      - containerPort: 80
        hostPort: 8080
        protocol: TCP
  - role: worker
  - role: worker

kind create cluster --name dev --config kind-cluster.yaml
kubectl get nodes

The extraPortMappings bit is the trick people miss: it forwards a port from your Ubuntu host into the control-plane container, so an ingress controller inside the cluster is reachable at http://localhost:8080.

Loading a locally-built image (no registry needed)

This is kind's best feature for the Docker workflow. Build with Docker, push straight into the cluster's nodes — no registry round-trip:

docker build -t myapp:dev .
kind load docker-image myapp:dev --name dev

kubectl create deployment myapp --image=myapp:dev
kubectl set image deployment/myapp myapp=myapp:dev   # after a rebuild + reload

Set imagePullPolicy: IfNotPresent (or Never) in your manifest for locally-loaded images, or the kubelet will try to pull myapp:dev from a registry and fail with ImagePullBackOff.

Ubuntu-specific gotchas

cgroup / inotify limits. Big clusters on kind can exhaust inotify watches. If pods crashloop with "too many open files," raise them:

  sudo sysctl fs.inotify.max_user_watches=524288
  sudo sysctl fs.inotify.max_user_instances=512

Rootless Docker works but needs cgroup v2 delegation configured; if kind create hangs, test with rootful Docker first.
It's ephemeral by design. A kind node is a container — restart Docker and the cluster state is gone unless you use extraMounts for persistence. That's a feature for testing, a footgun if you treated it like a server.

When a kind pod won't start, it's the same Kubernetes failures as anywhere — ImagePullBackOff from the pull-policy trap above, or CrashLoopBackOff from the app itself. Fixes here: ImagePullBackOff and CrashLoopBackOff.

Next: minikube with the Docker driver — a heavier but more full-featured local option.

minikube with the Docker Driver on Ubuntu: A Practical Local Cluster

James Joyner — Sat, 25 Jul 2026 12:34:10 +0000

minikube is the other "Kubernetes in Docker" option on Ubuntu, and with --driver=docker it runs the cluster inside a Docker container just like kind — but ships with addons (ingress, metrics-server, dashboard, a built-in registry) that make it feel more like a real cluster. Here's a practical setup and how it differs from kind.

Install on Ubuntu

You need Docker first (sudo apt-get install -y docker.io, then add yourself to the docker group). Then:

curl -fsSLo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube /usr/local/bin/minikube
minikube version

Start with the Docker driver

minikube start --driver=docker
# make it the default so you don't repeat the flag:
minikube config set driver docker

kubectl get nodes
docker ps    # a 'minikube' container is your node

Size it for real work:

minikube start --driver=docker --cpus=4 --memory=8g --disk-size=40g

The addons are the reason to pick minikube

minikube addons list
minikube addons enable ingress
minikube addons enable metrics-server
minikube dashboard        # opens the web UI

ingress gives you a working NGINX ingress controller with no manifest wrangling — genuinely useful when you want to test ingress routing locally.

The Docker image workflow

minikube runs its own Docker daemon inside the node container. The neat trick is pointing your shell's Docker CLI at that daemon, so images you build are immediately visible to the cluster with no push:

eval $(minikube docker-env)     # your `docker` now talks to minikube's daemon
docker build -t myapp:dev .
kubectl create deployment myapp --image=myapp:dev
# remember: imagePullPolicy: IfNotPresent so it doesn't try a registry pull

Undo it when you're done so docker points back at your host daemon:

eval $(minikube docker-env -u)

There's also a built-in registry if you prefer the push model:

minikube addons enable registry

Accessing services from Ubuntu

Two common patterns:

# quick tunnel to a single service (prints a URL)
minikube service myapp --url

# LoadBalancer support on your host (needs sudo; keep it running in a terminal)
minikube tunnel

minikube vs kind — how I choose

kind: faster start, lighter, multi-node config is trivial, ideal for CI and quick throwaways.
minikube: heavier, single-node by default, but addons (ingress, metrics, registry, dashboard) and the docker-env trick make it nicer for interactive local development.

Both run "Kubernetes with Docker" in the most literal sense — the cluster is a Docker container. Neither reflects how production nodes run (that's the containerd/cri-dockerd posts earlier in this series), so don't debug production runtime issues on them.

Cleanup

minikube stop        # keep the cluster, free the resources
minikube delete      # nuke it entirely

When a workload misbehaves on minikube it's ordinary Kubernetes troubleshooting — start with kubectl describe pod and kubectl logs, and the pod-startup error hub for the specific messages.

Next: the failure mode that quietly breaks more Ubuntu nodes than anything else — the systemd cgroup driver mismatch.

From Docker Build to Kubernetes Deploy on Ubuntu: The Image Workflow That Never Changed

James Joyner — Wed, 22 Jul 2026 21:31:12 +0000

Amid all the noise about dockershim, one thing got lost: the everyday workflow of building an image with Docker and running it on Kubernetes never changed. Docker is still an excellent build tool, Kubernetes still runs OCI images, and on Ubuntu the loop is clean. Here it is end to end.

1. A build-friendly Dockerfile

Multi-stage keeps the runtime image small and the attack surface low — this matters more on Kubernetes, where you pull the image onto every node that schedules the pod:

# build stage
FROM golang:1.22 AS build
WORKDIR /src
COPY go.* ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /out/api ./cmd/api

# runtime stage — distroless, no shell, tiny
FROM gcr.io/distroless/static:nonroot
COPY --from=build /out/api /api
USER nonroot:nonroot
EXPOSE 8080
ENTRYPOINT ["/api"]

2. Build and push with Docker on Ubuntu

Use buildx (bundled with modern Docker) so you can build multi-arch — worth it if any nodes are arm64:

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t registry.example.com/api:1.4.2 \
  --push .

Tag with an immutable version, never rely on :latest. Kubernetes caches images per node; :latest makes "which build is actually running?" unanswerable and breaks rollbacks.

3. A deployment that behaves in production

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  selector: { matchLabels: { app: api } }
  template:
    metadata: { labels: { app: api } }
    spec:
      containers:
        - name: api
          image: registry.example.com/api:1.4.2   # the exact tag you pushed
          imagePullPolicy: IfNotPresent
          ports: [{ containerPort: 8080 }]
          resources:
            requests: { cpu: "100m", memory: "128Mi" }
            limits:   { memory: "256Mi" }
          readinessProbe:
            httpGet: { path: /healthz, port: 8080 }
            initialDelaySeconds: 3
          livenessProbe:
            httpGet: { path: /healthz, port: 8080 }
            initialDelaySeconds: 10

The readinessProbe is the piece people skip and regret: without it, Kubernetes sends traffic to a pod before your app is listening, and you get intermittent 502s during every rollout.

kubectl apply -f deployment.yaml
kubectl rollout status deployment/api

4. Pulling from a private registry

If your registry needs auth, the cluster needs a pull secret — this is the same regardless of whether nodes run containerd or cri-dockerd:

kubectl create secret docker-registry regcred \
  --docker-server=registry.example.com \
  --docker-username=ci \
  --docker-password="$REGISTRY_TOKEN"

spec:
  imagePullSecrets:
    - name: regcred

Miss this and pods sit in ImagePullBackOff with pull access denied.

5. Ship a new build

docker buildx build -t registry.example.com/api:1.4.3 --push .
kubectl set image deployment/api api=registry.example.com/api:1.4.3
kubectl rollout status deployment/api
# rollback is one command because you used immutable tags:
kubectl rollout undo deployment/api

The mental model

Docker = how you build and push images. Unchanged, still great.
containerd / cri-dockerd = how the node runs them. This is what dockershim's removal was about.
Your manifests = don't care which runtime is underneath.

If you keep those three separate in your head, none of the 2022 runtime drama affects your daily loop.

When a rollout goes wrong it's usually the image pull or the probe: ImagePullBackOff, and the wider Docker runtime error guides for build-side failures.

Last in the series: when the node or the runtime itself is the problem — troubleshooting kubelet and containerd on Ubuntu.

State Encryption in OpenTofu: How It Works and How to Roll It Out

James Joyner — Mon, 20 Jul 2026 15:54:27 +0000

If you've ever cat-ed a Terraform or OpenTofu state file, you already know the uncomfortable truth: it's a plaintext JSON dump of everything your infrastructure knows, including secrets. Database passwords, generated private keys, API tokens injected through providers — they all land in state, in the clear. OpenTofu is the one place where you can fix this at the source, because native state and plan encryption is a first-class OpenTofu feature that upstream Terraform does not have. Here's how it actually works and how I roll it out on existing projects without breaking them.

Why plaintext state is a real risk

State is not a cache you can regenerate. It's the authoritative map between your HCL and the real resources, and OpenTofu has to store the values of attributes to compute diffs. That includes sensitive ones. Marking an output sensitive = true only hides it from the CLI output — it's still written verbatim to state.

On real infra I've seen state end up in three places it shouldn't: an S3 bucket without SSE and with overly broad read IAM, a CI artifact that got uploaded to a build cache, and a developer laptop with terraform.tfstate committed to a feature branch by accident. Backend encryption (like S3 SSE) helps for one of those. It does nothing for the other two, because the moment state leaves the backend it's plaintext again.

OpenTofu's encryption operates at the data layer, before the bytes ever hit the backend or a local file. The state is encrypted at rest everywhere: in the backend, in local copies, in CI artifacts. That's the property I want.

Anatomy of the `encryption` block

Encryption lives in a terraform { encryption { ... } } block. It has three moving parts: a key provider (where the encryption key comes from), a method (the actual cipher), and targets (state and/or plan) that bind a method to what you want encrypted.

terraform {
  encryption {
    key_provider "pbkdf2" "passphrase" {
      passphrase = var.tofu_encryption_passphrase
    }

    method "aes_gcm" "default" {
      keys = key_provider.pbkdf2.passphrase
    }

    state {
      method = method.aes_gcm.default
    }

    plan {
      method = method.aes_gcm.default
    }
  }
}

That's the whole minimal setup. pbkdf2 derives a key from a passphrase, aes_gcm is AES-GCM authenticated encryption, and both the state and plan targets use it. Note the passphrase comes from a variable — never hardcode it.

One caveat worth knowing (as of 2026 — check current docs): you generally can't pull the passphrase from a normal input variable defined elsewhere, because the encryption block is evaluated very early, before most of the graph. In practice I feed it from the environment instead, which I'll cover below. Treat the var. reference above as illustrative and prefer the env-var approach for the passphrase itself.

Key providers: passphrase vs KMS

The pbkdf2 passphrase provider is the easiest to start with and the easiest to get wrong operationally. If you lose the passphrase, the state is gone — there is no recovery. It's great for a solo project or a quick proof of concept, but the passphrase becomes a secret you now have to manage carefully.

For anything shared or production, I use a cloud KMS provider so the key material lives in a managed HSM-backed service and access is controlled by IAM:

terraform {
  encryption {
    key_provider "aws_kms" "main" {
      kms_key_id = "arn:aws:kms:us-east-1:111122223333:key/abcd-1234"
      region     = "us-east-1"
      key_spec   = "AES_256"
    }

    method "aes_gcm" "kms" {
      keys = key_provider.aws_kms.main
    }

    state {
      method   = method.aes_gcm.kms
      enforced = true
    }

    plan {
      method = method.aes_gcm.kms
    }
  }
}

There are equivalent gcp_kms and openbao/Vault-style providers too. The pattern is identical: the key_provider block changes, the method and targets stay the same. With KMS, access control and audit logging come for free — I can see in CloudTrail exactly who decrypted state and when, and I can revoke a role without rotating the underlying data key.

Encrypt the plan file too

People forget plan files. A tofu plan -out=tfplan binary contains the same resource values as state, plus the proposed changes. If your CI pipeline runs plan in one job and apply in another, that plan artifact is passed between jobs — and it's just as sensitive as state. The plan {} target above encrypts it with the same method. Do not skip it.

Rolling it out on an existing project with `fallback`

The scary part is turning encryption on when you already have unencrypted state. If you just add the encryption block, the next tofu plan will fail trying to decrypt state that was never encrypted. The fallback block is the migration escape hatch: it tells OpenTofu "if you can't decrypt with the primary method, treat the data as unencrypted."

terraform {
  encryption {
    key_provider "aws_kms" "main" {
      kms_key_id = "arn:aws:kms:us-east-1:111122223333:key/abcd-1234"
      region     = "us-east-1"
    }

    method "aes_gcm" "kms" {
      keys = key_provider.aws_kms.main
    }

    state {
      method = method.aes_gcm.kms

      fallback {
        # no method = read plaintext state, write encrypted
      }
    }
  }
}

The migration is a one-time apply:

tofu plan -out=tfplan
tofu apply tfplan
# state is now written back ENCRYPTED with the primary method

Because fallback with no method means "read as plaintext," OpenTofu can read your old state, and because the primary method is set, it writes the new state encrypted. Run one apply, confirm the state in the backend is now ciphertext, then remove the fallback block so plaintext state can no longer be silently accepted.

Lock it down with `enforced`

Once you've migrated, add enforced = true on the target. This is the setting that turns encryption from optional to mandatory:

state {
  method   = method.aes_gcm.kms
  enforced = true
}

With enforced, OpenTofu refuses to read or write unencrypted state at all. No accidental fallback, no misconfiguration silently dropping to plaintext. On a team, this is the line that guarantees nobody's local run produces a plaintext terraform.tfstate.

Key rotation

Rotation doesn't require re-encrypting everything in one shot. A method can hold a list of keys; the first is used to encrypt, and all of them are tried for decryption. To rotate, add a new key provider and put it ahead of the old one in the method's keys:

method "aes_gcm" "kms" {
  keys = [
    key_provider.aws_kms.new,   # new key: used for encryption
    key_provider.aws_kms.old,   # old key: still valid for decryption
  ]
}

The next apply writes state encrypted with the new key while still being able to read anything encrypted with the old one. After you've applied and confirmed all state is on the new key, drop the old provider. Same mechanism works for migrating from passphrase to KMS.

CI and secrets handling

For the passphrase provider, feed the secret through the environment, not a .tfvars file. OpenTofu reads encryption config from env vars prefixed appropriately, and in CI I inject it as a masked secret:

export TF_ENCRYPTION='key_provider "pbkdf2" "passphrase" {
  passphrase = "'"$TOFU_PASSPHRASE"'"
}'
tofu apply -auto-approve

For KMS, there's no passphrase to leak — the CI runner just needs an IAM role that can call kms:Decrypt/kms:GenerateDataKey on that key, which is exactly the least-privilege boundary you want. That's the strongest argument for KMS over passphrase in a pipeline: the secret never exists as a string anywhere.

When I get stuck on the migration edge cases — mixed fallback states, rotation ordering, backend quirks — I keep notes and error write-ups in my OpenTofu troubleshooting guides, because the failure messages during a half-migrated encryption rollout are not always obvious.

Takeaway

Turn on encryption with a fallback block, run one tofu apply to migrate existing state, remove fallback, then set enforced = true and encrypt the plan target too. Use pbkdf2 only for throwaway projects and reach for aws_kms/gcp_kms for anything real, so the key lives in a managed service with IAM and audit logging. Rotate by prepending a new key provider ahead of the old one and applying once. Plaintext state is a solved problem in OpenTofu — you just have to opt in.

Migrating from Terraform to OpenTofu: A Low-Risk Playbook

James Joyner — Sun, 19 Jul 2026 19:30:09 +0000

I've migrated a few real environments from Terraform to OpenTofu now, and the good news is that a careful migration is almost boring. The state format is compatible, the CLI is a near drop-in, and the whole thing can be done with a rollback path at every step. The bad news is that "almost boring" still has a couple of sharp edges, and the teams that get hurt are the ones who skip the parity check and go straight to apply. Here's the calm, low-risk playbook I actually follow.

Step 0: Know what "low-risk" means here

The core insight that makes this safe: OpenTofu reads the same HCL and the same state file that Terraform does. A migration is not a rewrite — it's swapping which binary talks to your existing state. That means at almost every step, your rollback is just "keep using the terraform binary." As long as you don't trigger a one-way-door feature (more on those later), you can walk back.

So the whole strategy is: prove parity before you change anything real, change one thing at a time, and keep the old binary installed until you're confident.

Step 1: Check and pin your Terraform version first

Before you touch OpenTofu, get your current setup deterministic. OpenTofu forked from the last MPL-licensed Terraform, so very old or very new Terraform configs can have edges. Find out exactly what you're running:

terraform version

Pin it. If you're not already using a version manager or a pinned CI image, do that now — you want a fixed, known-good Terraform baseline to compare against and to fall back to. Also pin your provider versions in a lockfile:

terraform providers lock

A migration where both the tool version and the provider versions are floating is a migration where you can't tell what caused a diff. Lock everything down first.

Step 2: Install `tofu` alongside, not instead

Install OpenTofu without removing Terraform. On a workstation or a scratch CI runner:

# Verify the binary is there and note the version
tofu version

Keep both terraform and tofu on PATH during the migration. You'll be running them back to back to compare, and having both is what makes rollback trivial. As of 2026 the install methods and current versions are in the OpenTofu docs — check them rather than trusting a version number from a blog post (including this one).

Step 3: The parity check — init and plan, no apply

This is the heart of the whole exercise. Work on a copy or a non-production workspace first. Point OpenTofu at your existing configuration and existing state, initialize, and produce a plan — but do not apply.

# Fresh working dir state, same backend/state as before
tofu init

# The critical test: does OpenTofu see zero changes?
tofu plan -out tofu.plan

What you want to see is a clean, no-changes plan. If OpenTofu reads your Terraform-written state and reports that nothing needs to change, you have parity. That's the green light.

If the plan shows drift, stop and read it carefully before doing anything. Common causes I've hit:

Provider version differences. OpenTofu resolved a slightly different provider than your pinned Terraform lockfile. Reconcile the versions.
Registry source differences. OpenTofu uses its own registry; a provider or module might resolve from a different source. Verify the provider actually publishes where OpenTofu looks.
A genuinely different interpretation of some config. Rare, but read the diff — do not apply your way past it.

Do this parity check per module/workspace, not once globally. State lives per-workspace, and a clean plan in one doesn't guarantee a clean plan in another.

Step 4: Apply once, deliberately, in a safe place

Once you've got a clean plan in a non-prod workspace, run the apply there so OpenTofu writes state at least once:

tofu apply tofu.plan

Even a no-op apply may rewrite state metadata. That's fine and expected — but it's the moment worth noting, because after OpenTofu writes state, that workspace's state has been touched by tofu. Terraform can generally still read it, but this is the point where you start treating that workspace as "OpenTofu-managed." Do it somewhere you can afford to be wrong before you do it in production.

Step 5: Swap CI, one pipeline at a time

Now change the automation. In your CI config, this is usually as small as swapping the binary and the command name:

# Before
terraform init && terraform plan -out plan.tfout

# After
tofu init && tofu plan -out plan.tfout

Roll it out per-pipeline, lowest-stakes environment first. Keep the plan-review gate in your pipeline — a human or a required approval looking at the plan output — for the first few runs on each environment. The whole point of a slow rollout is that if OpenTofu ever produces a plan you didn't expect, you catch it at plan time, not after apply.

I also recommend keeping a terraform-based fallback job available (even if disabled) during the transition, so reverting CI is a one-line change rather than an archaeology project.

Step 6: Watch for the one-way doors

Everything above is reversible as long as your config stays compatible with both tools. The way you lose your rollback is by adopting an OpenTofu-only feature. The big ones to be aware of:

Native state/plan encryption. Once OpenTofu encrypts your state, stock Terraform can't read it. This is a feature you may want — but adopt it as a deliberate, post-migration decision, not mid-migration.
Early variable evaluation in backend blocks or module sources. Configs that rely on it won't parse under Terraform.
.tofu / .tofu.json override files and provider-defined functions via the provider:: namespace. Both are OpenTofu-specific surface area.

My rule during the migration window: change the tool, not the config. Keep your HCL dual-compatible until every environment is on OpenTofu and stable. Only then start adopting the divergent features — and when you do, understand you're closing the door behind you. If you want more detail on the specific compatibility gotchas and error messages these features throw, I keep a running set of OpenTofu troubleshooting notes from real migrations.

Step 7: How to actually roll back

If something goes wrong before you've crossed a one-way door, rollback is genuinely simple:

Switch the binary back. In CI and locally, tofu becomes terraform again.
Re-init with Terraform so its lockfile and provider selections are in place: terraform init.
Run a plan and confirm a clean, no-change result: terraform plan.
Restore state from backup only if you actually corrupted or encrypted it. This is why you keep versioned state — an S3 bucket with versioning, or whatever your backend offers, so you can retrieve the pre-migration state object.

Back up your state before you start. A cheap tofu state pull > backup.tfstate (or the Terraform equivalent) before the first apply gives you a plain escape hatch. I've never had to use it on a careful migration, but the whole reason the migration feels calm is that the backup exists.

Takeaway

Migrating from Terraform to OpenTofu is mostly a swap, not a rewrite, and the parity check is what makes it safe: prove OpenTofu reads your existing state with a clean plan before you change anything real. Pin your versions, keep both binaries installed, roll CI out one environment at a time, and don't adopt one-way-door features until you're fully migrated and stable. Do it in that order and the scariest part of the whole thing will be how uneventful it is.

OpenTofu vs Terraform in 2026: What Actually Changed

James Joyner — Sat, 18 Jul 2026 14:30:45 +0000

I've been running both Terraform and OpenTofu across production infra for a while now, and the number one question I still get is some version of "wait, aren't they the same thing?" The honest answer in 2026 is: they share a common ancestor and a lot of DNA, but they are no longer the same tool. Here's what actually changed, from someone who has to keep both green in CI.

A very short history of the fork

If you missed the drama: HashiCorp relicensed Terraform from the MPL open-source license to the Business Source License (BSL) in 2023. The BSL is source-available but not OSI-approved open source, and it carries a use restriction aimed at competitors. A chunk of the community, backed by a group of vendors and users, forked the last MPL-licensed Terraform codebase. That fork landed under the Linux Foundation as OpenTofu.

So the core distinction is governance, not features: OpenTofu is a Linux Foundation project with open governance and an MPL-2.0 license, and Terraform is a HashiCorp product under the BSL. That licensing split is the reason a lot of teams looked at OpenTofu at all — if your legal team is nervous about the BSL's use restriction, an actual open-source license is the whole ballgame.

The CLI and state are (mostly) drop-in compatible

The thing that makes OpenTofu practical to adopt is that it started as a literal fork. The binary is tofu instead of terraform, and for a lot of everyday work it behaves identically:

tofu init
tofu plan -out plan.tfout
tofu apply plan.tfout
tofu state list

It reads your existing .tf files, understands the same HCL, and uses the same state file format. On real infra I've pointed tofu at a state file that was last touched by Terraform and had it produce a clean, no-change plan. That parity is not an accident — keeping the state format compatible is what makes migration a low-drama exercise rather than a rewrite.

But "mostly compatible" is doing real work in that sentence. The two projects have been diverging since the fork, and the gap widens with every release. Treating them as interchangeable is where teams get burned, so let's talk about the divergences that actually matter.

The real divergences in 2026

State and plan encryption, natively

This is the feature I care about most. OpenTofu ships native state and plan encryption. You configure it directly in your OpenTofu configuration, pick a key provider (PBKDF2 with a passphrase, a cloud KMS, and so on) and a method, and OpenTofu encrypts state at rest — including the plan file, which can leak secrets just as badly as state.

terraform {
  encryption {
    key_provider "pbkdf2" "mykey" {
      passphrase = var.encryption_passphrase
    }

    method "aes_gcm" "default" {
      keys = key_provider.pbkdf2.mykey
    }

    state {
      method = method.aes_gcm.default
    }

    plan {
      method = method.aes_gcm.default
    }
  }
}

In the Terraform world you traditionally solved this at the backend level — a KMS-encrypted S3 bucket, restrictive IAM, and hoping nobody terraform shows a plan file into a CI log. OpenTofu moves encryption into the tool itself. If you handle regulated data, this alone can justify the switch.

Early variable evaluation

For years the answer to "why can't I use a variable in my backend block?" was a shrug. OpenTofu added early variable evaluation, which lets you use variables (and locals) in places that used to demand static literals — most notably backend configuration and module sources. That means you can drive your backend bucket or key by variable instead of maintaining a wall of -backend-config flags or partial-backend hacks.

It's genuinely useful, and it's also a one-way door: a config that relies on early eval in a backend block won't parse cleanly under stock Terraform. Keep that in mind before you sprinkle it everywhere.

Provider-defined functions

Both ecosystems moved toward letting providers ship their own functions rather than waiting for the core team to add every string-munging helper. In OpenTofu you call them through the provider:: namespace:

locals {
  parsed = provider::aws::arn_parse(var.role_arn)
}

The exact set of available functions depends on the provider version you're pinning, so I won't quote a catalog — check the current provider docs. The point is that the language surface is no longer frozen to whatever core ships.

`.tofu` and `.tofu.json` override files

This is a small feature with big ergonomic payoff. OpenTofu recognizes .tofu and .tofu.json files, and it prefers them over the equivalent .tf/.tf.json when both exist. That gives you a clean way to keep a shared codebase that runs under both tools: keep the common config in .tf, and drop OpenTofu-specific overrides in .tofu files that Terraform simply ignores.

main.tf          # shared, runs under both
backend.tofu     # OpenTofu-only overrides, invisible to terraform

If you're maintaining a module that needs to support both tools during a transition, this is the mechanism that keeps you sane.

The registry

OpenTofu runs its own provider and module registry rather than depending on HashiCorp's. In practice most of the popular providers are mirrored and resolve fine, but the source of truth is different, and provider/module availability is something to actually verify rather than assume. If you have a niche or internal provider, confirm it publishes where OpenTofu looks for it.

A note on version numbers

I'm deliberately not quoting exact version numbers or "OpenTofu is X% faster" benchmarks, because those age badly and half the ones you'll see online are made up. As of 2026 both projects are shipping regularly and the feature sets keep moving — treat any specific version claim (including mine) as something to verify against the current docs before you build a decision on it.

So which do you pick, and should you switch?

Here's my honest take.

If you're starting green today, I'd default to OpenTofu. You get a real open-source license, native state/plan encryption, and the divergent features are mostly additive quality-of-life wins. The compatibility story means you lose almost nothing by choosing it.

If you're an existing Terraform shop, the calculus is about your actual pain. Switch if the BSL license is a genuine legal or procurement problem, or if native state encryption solves a compliance requirement you're currently duct-taping. Don't switch just to be on the trendy side of a fork — a migration is still real work and real risk, even when it's low-risk.

If you depend on Terraform Cloud / HCP-specific features or a paid workflow tightly coupled to HashiCorp's platform, weigh that integration honestly. OpenTofu is the engine, not the whole platform, and you'll be assembling backend, state, and workflow pieces yourself or via third-party platforms.

Whatever you choose, the one thing I'd avoid is drifting into using divergent features by accident and then being surprised you can't go back. If you want a cross-referenced dive into the specific errors and edge cases I hit while running OpenTofu on real clusters, I keep my OpenTofu troubleshooting guides updated as I trip over new ones.

Takeaway

OpenTofu and Terraform are close cousins, not twins. The compatibility is real enough that adoption is cheap, but the divergences — state/plan encryption, early evaluation, provider-defined functions, .tofu overrides, and a separate registry — are real enough that you should choose deliberately and know which one-way doors you're walking through. Pick based on your license posture and your compliance needs, pin your versions, and read the current docs before betting on any specific feature.

The OpenTofu Errors You'll Actually Hit (and How to Fix Them Fast)

James Joyner — Fri, 17 Jul 2026 23:14:11 +0000

Every OpenTofu user hits the same wall of errors eventually, usually at the worst possible moment — mid-deploy, in CI, with a teammate waiting. After enough tofu apply runs on real infra I've learned that most of these have a fast, deterministic fix once you recognize the message. This is the field guide I wish I'd had: the errors you'll actually see and the shortest path out of each.

The CLI is tofu (OpenTofu is the Linux Foundation fork of Terraform), but almost all of these apply identically to terraform if you're still on it.

1. State lock: "Error acquiring the state lock"

You'll see a ConditionalCheckFailedException or a lock ID dump. It means a previous run died without releasing the lock, or someone is genuinely running apply right now.

First, make sure nobody is actually applying. Then force-unlock with the ID from the error:

tofu force-unlock 1a2b3c4d-5e6f-7890-abcd-ef1234567890

Do not reach for -force flags or delete the DynamoDB lock item by hand unless force-unlock refuses. Ninety percent of the time a stale lock from a crashed CI job is the cause.

2. Provider checksum mismatch / dependency lock

Error: registered checksum for provider ... does not match

Your .terraform.lock.hcl was generated on one platform (say, macOS arm64) and CI runs on linux amd64, so the recorded hashes don't cover the platform being used. The fix is to record hashes for all the platforms your team and CI use:

tofu providers lock \
  -platform=linux_amd64 \
  -platform=darwin_arm64 \
  -platform=linux_arm64

Commit the updated .terraform.lock.hcl. If you legitimately upgraded a provider and want to accept the new hash, tofu init -upgrade regenerates it. Never delete the lock file to "fix" this — that just moves the problem to the next person.

3. "Backend initialization required, please run tofu init"

The backend config changed, a new module was added, or you're in a fresh checkout. This is not a real error, just an uninitialized working directory:

tofu init

If you changed backends (e.g. local to S3), you'll need to migrate state:

tofu init -migrate-state

And if init complains the backend config differs from what's cached and you want to blow away the cached backend, tofu init -reconfigure. Use -migrate-state when you want to keep state, -reconfigure when you want to point at a fresh one.

4. "Invalid for_each argument" — unknown values

Error: Invalid for_each argument
The "for_each" value depends on resource attributes that cannot be
determined until apply.

This is the single most common structural error I see. for_each needs to know its keys at plan time, but you fed it a value that only exists after another resource is created — an ARN, a generated ID, a computed name.

The fix is to key the map on something static. Use the input you control, not the computed output:

# BAD: keys depend on a created resource's attribute
resource "aws_route53_record" "r" {
  for_each = { for s in aws_subnet.this : s.id => s }
}

# GOOD: key on the static input you already know
resource "aws_route53_record" "r" {
  for_each = var.subnet_defs   # a map you defined
}

If you truly can't avoid it, a -targeted apply to create the upstream resource first, then a normal apply, is the escape hatch — but restructuring the keys is the real fix.

5. "Unsupported argument" / "Unsupported attribute"

Error: Unsupported argument
An argument named "foo" is not expected here.

Two usual causes. Either the provider version changed and renamed/removed the argument, or you're referencing an attribute that doesn't exist on that resource. Check the exact version you have and its docs:

tofu version
tofu providers

If a provider upgrade renamed things, pin the version you were on while you migrate:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.40"
    }
  }
}

For "Unsupported attribute," run tofu state show <address> on the resource to see exactly which attributes it actually exposes — I've wasted real time guessing at attribute names that the provider simply renamed between versions.

6. Dependency cycle

Error: Cycle: module.a.aws_x.foo, module.b.aws_y.bar

Two resources (or modules) reference each other, directly or through a chain, so OpenTofu can't order them. Visualize it instead of squinting at HCL:

tofu graph | dot -Tsvg > graph.svg

The fix is to break the loop. Usually one of the two references can be replaced with a static value, moved into a separate resource (like an aws_security_group_rule broken out of the group), or resolved by passing a value in as a variable rather than reading it back. Self-referential security groups are the classic offender — split the ingress rule into its own resource.

7. "Provider produced inconsistent final plan"

Error: Provider produced an inconsistent final plan
... produced an invalid new value for .some_attr ...

This is a provider bug, not your HCL — the value the provider promised at plan time didn't match apply time. It's rarely something you can fix in config directly. Fastest mitigations, in order:

First, upgrade the provider — these are frequently patched:

tofu init -upgrade

If that doesn't help, tell OpenTofu to stop tracking the flapping computed attribute with ignore_changes, or drop lifecycle { ignore_changes = [some_attr] } on the resource. As a last resort, tofu apply -replace=<address> forces a clean recreate so the provider computes the attribute fresh. If it persists, it's worth an upstream issue with the provider.

8. Registry service discovery failure

Error: Failed to query available provider packages
could not connect to registry.opentofu.org

Network, proxy, or registry outage. First confirm it's reachable:

curl -sSf https://registry.opentofu.org/.well-known/terraform.json

If you're behind a corporate proxy, set HTTPS_PROXY and NO_PROXY before init. If the public registry is flaky or you need reproducible CI, configure a provider mirror and point OpenTofu at local packages:

tofu providers mirror ./tofu-mirror

Then reference that directory with a provider_installation { filesystem_mirror { ... } } block in your CLI config. A mirror also inoculates you against the next registry hiccup, which is why every serious CI pipeline I run has one.

When the message isn't in this list

Some errors are stack- or provider-specific and don't have a one-line fix. When I hit one that isn't obvious, I check my OpenTofu error library for the exact message before I start guessing, because reading the actual failure text carefully almost always beats trial-and-error re-applies.

Takeaway

Most tofu errors fall into a handful of buckets: locking (force-unlock), lock-file/checksum drift (providers lock with all platforms), uninitialized dirs (init / -migrate-state), unknown-value for_each (key on static inputs), version-renamed arguments (pin and read state show), cycles (tofu graph, then break the loop), provider plan bugs (init -upgrade / -replace), and registry outages (mirror). Learn to recognize the message and the fix is usually one command away. Keep a mirror and a pinned lock file and you'll pre-empt half of these before they ever fire.

The 12 DevOps Errors That Page Teams Most (And the First Thing to Check)

James Joyner — Wed, 15 Jul 2026 14:58:06 +0000

Over the last while I've been cataloguing production DevOps errors — the exact strings that show up in logs at 2 a.m. — and writing a fix for each one. A pattern jumps out fast: a small number of errors account for a huge share of the pages. Here are the twelve that come up most, with the one-thing-to-check-first for each.

None of these are exotic. That's the point. The stuff that actually pages you is rarely exotic — it's the same dozen failure modes wearing different hats.

1. `CrashLoopBackOff`

The pod started, died, and Kubernetes is now backing off between restarts. CrashLoopBackOff is a symptom, never a cause. Go straight to kubectl logs <pod> --previous — the logs from the crashed container are where the real error lives. Nine times out of ten it's a bad config value, a missing env var, or a failed migration on startup.

2. `ImagePullBackOff` / `ErrImagePull`

Kubernetes can't pull the image. Don't guess — kubectl describe pod spells it out in Events. It's almost always a typo in the tag, a missing imagePullSecret, or a registry rate limit.

3. `OOMKilled` (exit code 137)

This is not "the node ran out of memory." It's "this container hit its own cgroup memory limit and the kernel killed it." Different problem, different fix. Compare the pod's resources.limits.memory against what it actually uses (kubectl top pod) before you touch anything at the node level.

4. `No space left on device`

The classic — and the trap is when df -h shows free space anyway. Then it's one of two things: you're out of inodes (df -i), or a process is holding a deleted-but-still-open file (lsof +L1). rm won't reclaim that space until you restart the process holding the file descriptor.

5. DNS timeouts inside pods

An external lookup that works from the node but intermittently times out inside a pod is almost always the ndots:5 search-domain cascade colliding with a conntrack UDP race — you get a flat 5-second stall that blows your client timeout. Overriding ndots on the pod spec and running NodeLocal DNSCache is the fix.

6. `FATAL: sorry, too many clients already` (Postgres)

Bumping max_connections is the trap, not the fix — each connection costs real memory. You need a pooler (PgBouncer), not 500 backend processes.

7. `Connection refused`

Something reached the host and nothing was listening on that port. It's rarely DNS or the network — it's the service being down, bound to 127.0.0.1 instead of 0.0.0.0, or a firewall. ss -tlnp on the target tells you in one line.

8. `TLS handshake timeout`

Usually not a cert problem at all — it's a network path problem (MTU, a proxy, or a firewall silently dropping the handshake) masquerading as TLS. Test raw connectivity first with openssl s_client -connect host:443.

9. `Read-only file system`

A filesystem that was mounted read-write and is suddenly read-only almost always means the kernel remounted it ro after detecting I/O errors. Check dmesg — you may be looking at a failing disk, not a permissions issue.

10. `Multi-Attach error for volume`

A ReadWriteOnce volume can attach to exactly one node at a time — not one pod, one node. If a node goes NotReady with the volume still attached, a pod rescheduled elsewhere gets this error. Kubernetes waits ~6 minutes before force-detaching on purpose — to protect your data from being written by two hosts at once.

11. `502 Bad Gateway`

502 means your proxy reached the upstream and the upstream said no (or died). It's rarely the proxy. connect() failed (111: Connection refused) in the NGINX error log → your app isn't listening where the proxy thinks it is.

12. `exec format error`

You built an image for one CPU architecture and ran it on another (hello, Apple Silicon → x86 clusters). Build multi-arch, or match your --platform.

The pattern

Every one of these has the same shape: the error message describes the symptom the system noticed, not the cause you need to fix. CrashLoopBackOff isn't why your pod is dying. OOMKilled isn't the node. The skill isn't memorizing fixes — it's knowing which single command turns the symptom back into a cause.

I keep a full, searchable library of these — every error above has a complete guide with the diagnostic workflow, an example root-cause analysis, and the prevention checklist. If you want the deeper version of any of them:

The Kubernetes ones live in the Kubernetes troubleshooting toolkit
Everything else is in the full error-guide library (Linux, Postgres, Docker, NGINX, and more)

What's the error that pages your team most? Curious whether it's on this list or something I should go write up next.

I Built Free Browser-Based Validators for YAML, Kubernetes and Terraform (No Upload, No Signup)

James Joyner — Tue, 14 Jul 2026 15:32:55 +0000

Every DevOps engineer has done this dance: you've got a chunk of YAML or a Terraform file that looks right, something's rejecting it, and you want a fast sanity check. So you paste it into some random online validator — and a small voice asks, wait, where did that config just go?

That config often has structure, comments, sometimes internal hostnames or resource names in it. Pasting infrastructure definitions into an unknown server is a habit worth breaking. So I built a set of validators that never send your config anywhere — they run entirely in your browser.

What they are

Free, browser-based validators for the formats DevOps folks paste-and-pray most:

YAML — catches the indentation and structure errors that make Kubernetes and CI configs fail with cryptic messages
Kubernetes manifests — schema-aware checks beyond "is it valid YAML," so you catch the wrong apiVersion or a misplaced field before kubectl apply does
Terraform / HCL — structural validation for the syntax slips that terraform validate flags only after you've context-switched away

The one design decision that matters

100% client-side. No upload, no signup, no server round-trip. Your config is parsed by JavaScript running in your own tab — it never leaves your machine. You can literally open dev-tools, watch the network panel, and see nothing go out. Turn off your wifi and they still work.

This isn't a privacy gimmick — it's the correct architecture for a tool that handles infrastructure definitions. A validator has no business seeing your config on a server it doesn't need to.

Why I bother

Two reasons, honestly.

One: I kept wanting this exact thing and kept not trusting the options. The nth time I hesitated before pasting a manifest into a stranger's website, I decided to just build the version I'd trust.

Two: fast feedback loops are the whole game in this job. The gap between "save the file" and "find out it's malformed" is pure friction — and the tighter that loop, the less of your working memory it burns. A validator that's one tab away and gives an answer in milliseconds is a small thing that compounds.

Try them

The validator workbench — YAML, Kubernetes, and Terraform, all client-side

If you're the kind of person who'd rather script it, a lot of the underlying tooling is open source — CLIs and a small read-only API for the prompt and error-guide data — over on the developer page and the GitHub org.

Client-side tools have real limits — they can't know your cluster's live state, and schema validation isn't the same as a policy check. But for the "did I just fat-finger the indentation" question, having the answer without a network request is exactly the trade I want.

What config format do you most wish had a trustworthy, offline, no-signup validator? That's genuinely how I decide what to build next.

Fix Docker Exit Code 137 (OOMKilled): Why It Happens and How to Stop It

James Joyner — Tue, 14 Jul 2026 03:21:09 +0000

Your container died and docker ps -a shows something like Exited (137) 4 minutes ago. Nine times out of ten that's the kernel's OOM killer, not your app crashing on its own. Here's what exit code 137 actually means and how I go about stopping it from happening again.

What exit code 137 actually means

Exit code 137 is 128 + 9. The 128 + part is the shell convention for "terminated by a signal," and 9 is SIGKILL. So 137 means your process was hard-killed — no chance to clean up, no graceful shutdown.

Two things commonly send that SIGKILL:

The kernel OOM killer. Your container hit its memory cgroup limit, or the whole host ran out of RAM, and the kernel picked a process to kill to stay alive.
A docker stop that timed out. Docker sends SIGTERM, waits (10s by default), and if the process is still alive it escalates to SIGKILL. That also produces 137.

Both look identical in docker ps -a. The rest of this is about the first case — OOMKilled — because that's the one that quietly recurs.

Confirm it was OOM, not something else

Before changing anything, confirm the cause. Docker records whether the OOM killer was involved:

docker inspect my-service --format '{{.State.OOMKilled}} {{.State.ExitCode}}'

If that prints true 137, you're done guessing — it was OOM. If it prints false 137, the SIGKILL came from somewhere else (most often a docker stop timeout).

For the fuller picture:

docker inspect my-service --format '{{json .State}}' | python3 -m json.tool

You can also confirm from the kernel side. The OOM killer logs every kill:

sudo dmesg -T | grep -i -E 'out of memory|oom-kill|killed process'
# or, on a systemd host:
journalctl -k | grep -i -E 'out of memory|oom-kill'

You're looking for a line like Out of memory: Killed process 12345 (java).

One distinction that changes your fix: did you hit the container's own --memory limit, or did the whole host run out of RAM? If OOMKilled is true but the host has plenty of free memory, the container hit its own cgroup limit. If the host itself was starved, the kernel may kill the biggest process regardless of which container it's in — sometimes an innocent bystander. dmesg shows the cgroup and total-vm in the kill line, which tells you which case you're in.

Why it happens

A few root causes cover most of what I see:

No --memory limit at all. The container can grow until the host is exhausted. This is the one that takes down neighbours.
A limit that's just too low for the real working set. The app was always going to need ~400 MB and you capped it at 256 MB.
A real leak. Memory climbs steadily under load and never comes back down. A limit only changes when it dies, not whether.
A runtime that ignores the cgroup limit. This is the classic. An old JVM sees the host's total RAM, sizes its heap for that, and blows past the container limit. Node has a similar story — its old-space heap defaults to roughly 1.5–2 GB regardless of the container limit unless you tell it otherwise with --max-old-space-size.
A batch job that spikes. Steady-state memory is fine, but one large request or a big file load briefly doubles it and trips the limit.

The runtime-unaware case is worth dwelling on because it surprises people: the container limit and the runtime's idea of "how much memory exists" are two different numbers, and if the runtime's number is bigger, it will happily allocate its way into an OOM kill.

How I'd diagnose it (in order)

Cheapest, least invasive first. I don't reach for a profiler until the simple checks rule things out.

Watch live usage against the limit. docker stats shows current memory and the limit side by side:

   docker stats --no-stream my-service

If MEM USAGE / LIMIT reads 254MiB / 256MiB right before it dies, you're pegged at the limit — that's your answer.

Reproduce and watch it climb. Leave docker stats streaming (drop --no-stream) while you drive load. Steady climb that never recedes points at a leak; a sharp spike on one operation points at a batch/request problem.
Check the configured limit. Confirm what the container was actually given:

   docker inspect my-service --format 'mem={{.HostConfig.Memory}} memswap={{.HostConfig.MemorySwap}}'

0 means no limit. Otherwise it's bytes.

Check the app-level heap settings. Look at how the runtime was told to size itself — JVM flags, NODE_OPTIONS, whatever applies. Mismatch between this and the container limit is a common culprit.
Only now, a profiler. If usage is legitimately high and you need to know what's holding memory, attach the language's heap profiler. This is the expensive step, so I earn my way to it.

A worked example

Say I've got a JVM service running with a tight limit:

docker run -d --name my-service --memory=256m my-registry/my-service:1.4.2

It OOMs under load. docker inspect confirms it:

docker inspect my-service --format '{{.State.OOMKilled}} {{.State.ExitCode}}'
# true 137

docker stats shows it pinned at the limit before each death. The problem is twofold: the limit is a bit low for the real working set, and the JVM isn't sizing its heap to the container.

First, give it a limit that reflects reality. I measured the steady-state working set at around 350 MB, so I'll allow headroom for the JVM's non-heap overhead (metaspace, thread stacks, off-heap buffers) on top of the heap:

docker run -d --name my-service \
  --memory=512m \
  -e JAVA_OPTS="-XX:MaxRAMPercentage=70.0" \
  my-registry/my-service:1.4.2

-XX:MaxRAMPercentage=70.0 tells the JVM to cap its heap at 70% of the container limit — leaving the other 30% for non-heap memory so the process total stays under 512 MB. On JDK 10+ container support (-XX:+UseContainerSupport) is on by default, so the JVM reads the cgroup limit rather than the host's RAM. On older JVMs you'd set an explicit -Xmx instead, but percentage-based is more robust across environments.

For a Node service the equivalent is bounding the old-space heap under the limit:

docker run -d --name my-worker \
  --memory=512m \
  -e NODE_OPTIONS="--max-old-space-size=384" \
  my-registry/my-worker:2.1.0

384 (MB) sits comfortably under the 512 MB container limit, leaving room for Node's other allocations.

The pattern in both cases: pick the container limit from measured usage plus headroom, then tell the runtime to keep its heap under that limit.

What to watch out for

Setting the limit too low just to "cap" it. If the app has a leak, a tight limit doesn't fix the leak — it converts a slow degradation into a fast crash loop. You've made it more visible, not healthier. Fine as a deliberate blast-radius guard; not fine as a substitute for fixing the leak.
No limit at all. One unbounded container can consume the host and get other containers OOM-killed. The victim in dmesg may be a service that did nothing wrong.
Swap accounting. --memory and --memory-swap are different knobs. If you set --memory=512m and leave swap unset, Docker may allow up to twice the memory in swap, which masks the real usage — the container limps along swapping instead of failing cleanly. Set --memory-swap equal to --memory to disable swap for that container when you want hard, predictable behaviour.
Measuring the wrong number. MEM USAGE in docker stats includes page cache, which can make usage look scarier than the actual anonymous (unreclaimable) memory that drives OOM decisions. Watch the trend and the kill line in dmesg rather than a single snapshot.

Making it repeatable

To stop this being a recurring surprise:

Right-size from data. Use docker stats or your metrics stack to find the real working set under load, then set --memory to that plus honest headroom.
Pin the limit and the runtime heap together. Set --memory and a matching runtime flag (-XX:MaxRAMPercentage, --max-old-space-size, etc.) so the two numbers can't drift apart.

  # docker-compose.yml
  services:
    my-service:
      image: my-registry/my-service:1.4.2
      mem_limit: 512m
      restart: on-failure
      environment:
        JAVA_OPTS: "-XX:MaxRAMPercentage=70.0"

Add a restart policy and an alert. restart: on-failure keeps you online through a transient spike, and an alert on OOMKilled events or restart count means you hear about it before your users do. Be honest about the restart policy though: it buys time, it does not fix a leak. A container that restarts every ten minutes is telling you something you shouldn't silence.

If you'd rather not reassemble all of this under pressure the next time a container flaps, I keep the reusable Docker patterns — limits, healthchecks, restart policies — as a reference set of Docker runbook patterns so it's a lookup, not an investigation.

Wrapping up

Exit code 137 is almost always a memory conversation, not a crash. The durable fix isn't a bigger number — it's making the container limit and the runtime's own idea of "available memory" agree, sized from what the app actually uses. Get those two to match and 137 stops being a mystery.

DEV Community: James Joyner

Keep Docker Engine as Your Kubernetes Runtime on Ubuntu with cri-dockerd

Should you? A quick reality check

1. Install Docker Engine on Ubuntu

2. Install cri-dockerd

3. Point kubeadm at the cri-dockerd socket

4. Verify

The gotcha nobody warns you about

Set Up a Kubernetes Node on Ubuntu 24.04 with containerd (the Docker-Compatible Runtime)

1. Kernel prerequisites

2. Install containerd

3. Install kubeadm, kubelet, kubectl

4. Initialize the control plane

5. Install a CNI, then verify

6. Prove Docker images run

When it doesn't come up

Run Kubernetes in Docker on Ubuntu for Local Development

Prerequisites on Ubuntu

A one-command cluster

A realistic multi-node cluster

Loading a locally-built image (no registry needed)

Ubuntu-specific gotchas

minikube with the Docker Driver on Ubuntu: A Practical Local Cluster

Install on Ubuntu

Start with the Docker driver

The addons are the reason to pick minikube

The Docker image workflow

Accessing services from Ubuntu

minikube vs kind — how I choose

Cleanup

From Docker Build to Kubernetes Deploy on Ubuntu: The Image Workflow That Never Changed

1. A build-friendly Dockerfile

2. Build and push with Docker on Ubuntu

3. A deployment that behaves in production

4. Pulling from a private registry

5. Ship a new build

The mental model

State Encryption in OpenTofu: How It Works and How to Roll It Out

Why plaintext state is a real risk

Anatomy of the encryption block

Key providers: passphrase vs KMS

Encrypt the plan file too

Rolling it out on an existing project with fallback

Lock it down with enforced

Key rotation

CI and secrets handling

Takeaway

Migrating from Terraform to OpenTofu: A Low-Risk Playbook

Step 0: Know what "low-risk" means here

Step 1: Check and pin your Terraform version first

Step 2: Install tofu alongside, not instead

Step 3: The parity check — init and plan, no apply

Step 4: Apply once, deliberately, in a safe place

Step 5: Swap CI, one pipeline at a time

Step 6: Watch for the one-way doors

Step 7: How to actually roll back

Takeaway

OpenTofu vs Terraform in 2026: What Actually Changed

A very short history of the fork

The CLI and state are (mostly) drop-in compatible

The real divergences in 2026

State and plan encryption, natively

Early variable evaluation

Provider-defined functions

.tofu and .tofu.json override files

The registry

A note on version numbers

So which do you pick, and should you switch?

Takeaway

The OpenTofu Errors You'll Actually Hit (and How to Fix Them Fast)

1. State lock: "Error acquiring the state lock"

2. Provider checksum mismatch / dependency lock

3. "Backend initialization required, please run tofu init"

4. "Invalid for_each argument" — unknown values

5. "Unsupported argument" / "Unsupported attribute"

6. Dependency cycle

7. "Provider produced inconsistent final plan"

8. Registry service discovery failure

When the message isn't in this list

Takeaway

Anatomy of the `encryption` block

Rolling it out on an existing project with `fallback`

Lock it down with `enforced`

Step 2: Install `tofu` alongside, not instead

`.tofu` and `.tofu.json` override files

1. `CrashLoopBackOff`

2. `ImagePullBackOff` / `ErrImagePull`

3. `OOMKilled` (exit code 137)

4. `No space left on device`

6. `FATAL: sorry, too many clients already` (Postgres)

7. `Connection refused`

8. `TLS handshake timeout`

9. `Read-only file system`

10. `Multi-Attach error for volume`

11. `502 Bad Gateway`

12. `exec format error`