DEV Community: James Joyner

DNS Troubleshooting with dig: The Commands DevOps Engineers Actually Need

James Joyner — Wed, 19 Aug 2026 12:41:16 +0000

A surprising share of "the app is down" pages resolve to a name-resolution problem, not a broken service. The service is fine; the client can't turn a name into an address. dig is the precision tool for proving that in seconds instead of guessing.

Think about it as a resolution chain, not "is DNS broken"

When a name fails, work the chain: which resolver did the client ask, what did that resolver return, and does it match what authoritative DNS actually says? Most incidents live in the gap between those three. The method is boring and reliable: observe the symptom, form a hypothesis about where in the chain it breaks, test with one query, read the evidence, fix, then validate.

The single most important habit: query the name from the same host and the same resolver the app uses. Running dig from your laptop proves nothing about what the pod or VM sees.

The record types worth knowing

You don't need all of them, but you need to recognize them:

A / AAAA — name to IPv4 / IPv6 address. The usual suspect.
CNAME — an alias pointing at another name. A stale or wrong CNAME sends traffic somewhere unexpected.
MX — mail routing. TXT — SPF, DKIM, domain verification, and other metadata.
NS — which servers are authoritative for a zone. SOA — the zone's serial and TTL defaults; the serial tells you whether a change has propagated.
PTR — reverse lookup, IP back to name.

The commands that actually earn their place

Start with the quick answer, then get precise.

dig +short api.internal.example.com

+short strips everything except the answer. If it prints an IP, resolution works from this host. If it prints nothing, you have a real failure to chase. Empty output is a signal, not an error.

dig api.internal.example.com A

The full form. Read the status in the header: NOERROR with an ANSWER section is good; NXDOMAIN means the name genuinely doesn't exist; SERVFAIL points at a broken upstream or DNSSEC issue. Also note which SERVER answered at the bottom — that's the resolver you're actually testing.

dig -x 10.20.30.40

Reverse lookup (PTR). Handy when logs show an IP and you need the name, or when validating that forward and reverse records agree.

dig +trace api.example.com

+trace walks delegation from the root servers down, showing each handoff. Use it when a name works from one resolver but not another — it reveals whether the authoritative servers themselves disagree with your cache.

resolvectl status

On systemd hosts, this shows the actual resolver and search domains in effect per interface. This is the ground truth that /etc/resolv.conf often only hints at, especially when systemd-resolved owns the stub at 127.0.0.53.

Real scenarios, and how to read them

Wrong A record. dig +short returns an IP, but it's the old one. Confirm authoritative truth with dig @<authoritative-ns> name A and compare to the zone's SOA serial. If authoritative is correct but your resolver isn't, you're looking at caching.

Stale cache / TTL. A record changed an hour ago but clients still hit the old address. Check the TTL counting down in the ANSWER section — a long TTL means old answers linger. Query the authoritative server directly to confirm the new value, then wait out or flush the cache rather than "restarting things."

Wrong resolver in /etc/resolv.conf. The app queries a resolver that can't see internal zones. dig shows the wrong SERVER at the bottom of its output. Cross-check with resolvectl status. This is common on cloud VMs where DHCP overwrites resolver config.

Search-domain surprises. dig +short api returns nothing but dig +short api.internal.example.com works. A bare name gets search domains appended by the resolver, and dig does not apply them the way your app's resolver library does. Always test both the short name and the FQDN.

Container DNS. Inside Docker, /etc/resolv.conf points at the embedded resolver 127.0.0.11, which forwards to the host and resolves other containers by name. In Kubernetes, pods use cluster DNS (CoreDNS) with search domains like svc.cluster.local. The rule holds: dig from inside the container or pod, not from the host. A name that resolves on the node but not in the pod is a cluster-DNS or search-domain issue, not a broken app.

I wrote this up as a full, reproducible walkthrough — resolver chain, capture, and fixes — in the DNS troubleshooting lesson, part of a 15-lesson networking series aimed at DevOps and DevSecOps troubleshooting. There's also a container-focused DNS lesson that goes deeper on the 127.0.0.11 resolver and cluster DNS.

One note on ethics: only inspect, query, or capture DNS traffic on systems you own or are explicitly authorized to assess. This is defensive infrastructure validation, not reconnaissance.

The takeaway

DNS failures feel mysterious because people guess. dig, run from the right host against the right resolver, turns guessing into evidence: you can see the status code, the resolver, the TTL, and the delegation path in one command each.

If you want the hands-on version, the DNS troubleshooting walkthrough steps through each scenario end to end — and the site has more free, hands-on Kali learning paths built specifically for DevOps engineers.

tcpdump for DevOps: Read a TCP Handshake and Stop Guessing

James Joyner — Tue, 18 Aug 2026 17:07:07 +0000

When a service "can't connect," the app logs and the network rarely tell the same story. tcpdump settles the argument: it shows what actually went over the wire, not what the client library claims happened. Learning to read three packet signatures is usually enough to split a network problem from an application problem in under a minute.

One ground rule first: only capture on systems you own or are explicitly authorized to assess. A pcap can hold credentials, tokens, and personal data, so treat capture files as sensitive and delete them when you're done. This is defensive infrastructure validation, not snooping.

Capture just enough, and nothing else

The mistake most people make is capturing everything and drowning in it. Filter at the source instead.

sudo tcpdump -i eth0 -n host 10.0.4.20 and port 443 -c 20

Breaking that down:

-i eth0 — the interface to listen on (any works if you're unsure).
-n — don't resolve IPs or ports to names, so DNS lag never distorts the output.
host 10.0.4.20 and port 443 — a BPF filter: only packets to/from that host on that port.
-c 20 — stop after 20 packets so the terminal stays readable.

Add tcp to drop everything else, or write the raw capture to disk for later analysis:

sudo tcpdump -i eth0 -n host 10.0.4.20 and tcp -w handshake.pcap

-w handshake.pcap writes packets untouched — open it later in the same tool or in Wireshark. The point is to reproduce the failing request while this runs, then read back exactly what the kernel saw.

Signature 1: a healthy connection

A working TCP connection always opens with the three-way handshake — SYN, SYN-ACK, ACK:

10.0.1.5.51420 > 10.0.4.20.443: Flags [S], seq 12345
10.0.4.20.443 > 10.0.1.5.51420: Flags [S.], seq 98765, ack 12346
10.0.1.5.51420 > 10.0.4.20.443: Flags [.], ack 98766

[S] is SYN, [S.] is SYN-ACK, [.] is a bare ACK. Three lines, both directions, done. If you see this, the network path and the listening service are both fine — your problem is above the transport layer: TLS, auth, a slow query, a 500. Stop blaming the firewall and go read the app.

Signature 2: connection refused

10.0.1.5.51422 > 10.0.4.20.443: Flags [S], seq 22222
10.0.4.20.443 > 10.0.1.5.51422: Flags [R.], seq 0, ack 22223

The SYN gets an immediate RST ([R.]). That's a fast, deliberate "no." The packet reached the host, but nothing is listening on that port — the process is down, bound to the wrong interface (127.0.0.1 instead of 0.0.0.0), or a proxy is rejecting it. Notice the speed: refusals come back in milliseconds. Check that the service is up and bound correctly.

Signature 3: timeout / silence

10.0.1.5.51424 > 10.0.4.20.443: Flags [S], seq 33333
10.0.1.5.51424 > 10.0.4.20.443: Flags [S], seq 33333
10.0.1.5.51424 > 10.0.4.20.443: Flags [S], seq 33333

Your SYN goes out, gets retransmitted after ~1s, ~2s, ~4s — and nothing ever comes back. No RST, no SYN-ACK, just your own side talking to a wall. That silence is the signature of a dropped packet: a security group, iptables rule, missing route, or a host that's simply gone. A refusal answers; a firewall drop stays quiet. That distinction alone routes the ticket to the right team.

Don't forget DNS

Half of "network" outages are name resolution. DNS rides UDP/53, and a healthy lookup is one query, one response:

10.0.1.5.40311 > 10.0.0.2.53: A? api.internal.example. (37)
10.0.0.2.53 > 10.0.1.5.40311: A 10.0.4.20 (53)

Query out, answer back with an address. If the query leaves and no response returns, you have a resolver problem, not a connectivity problem — and every downstream connect() will fail for the wrong-looking reason. I walk through this end to end in the full tcpdump packet-analysis lesson.

Capturing inside a container

Containers don't get raw-socket access by default. The wrong fix is --privileged, which hands the container broad host-level capabilities — and never mount the Docker socket to work around it. Grant exactly the one capability packet capture needs:

services:
  debug:
    image: nicolaka/netshoot
    cap_add:
      - NET_RAW
    network_mode: "service:app"

NET_RAW is enough for tcpdump; sharing the target's network namespace lets you watch its traffic without touching the host. There's a container-specific walkthrough in the Docker packet-capture lesson.

The habit

Observe the packets, form a hypothesis (handshake completes → look higher; RST → nothing listening; silence → something dropping), then test and validate the same way. The capture is evidence, not a guess.

If you want the guided version, start with the tcpdump lesson in the networking series. The site has free, hands-on Kali learning paths built for DevOps engineers — Kali as a portable troubleshooting toolbox, not a hacking course.

Network Troubleshooting as a Stack: Find Which Layer Is Broken First

James Joyner — Sat, 15 Aug 2026 18:34:37 +0000

The difference between a good infrastructure troubleshooter and someone who restarts services and hopes is a mental model. When "HTTPS times out" lands in your inbox, you don't guess — you know exactly which layer to interrogate first, and in what order.

The network is a stack, so treat it like one

Every request rides through the same layers, top to bottom:

Application → TLS → Port → DNS → Gateway → Route → Interface

That's the dependency order — TLS can't work if the port is closed, the port is meaningless if DNS resolved to the wrong host, and none of it matters if your interface has no IP. So you verify in the inverse order, from the ground up:

Interface → IP → Route → Gateway → DNS → Port → TLS → Application

Start at the bottom because a broken lower layer produces confusing symptoms higher up. Confirm each layer is healthy before you climb. The moment a layer fails, you've found your problem — everything above it is a red herring.

Walk it: "HTTPS to api.example.com times out"

1. Interface — do we have a link and an address?

ip addr show

Look for your primary interface (say eth0) in state UP with an inet line like 192.168.1.20/24. No inet? DHCP failed or the link is down — stop here, nothing above will work. If the address is present and sane, climb.

2. Route — is there a path to the destination?

ip route get 93.184.216.34

This shows the exact route the kernel would pick, including the source IP and gateway (via 192.168.1.1 dev eth0 src 192.168.1.20). If you get "Network is unreachable" or no default route, you've found it. This is also the signature behind the classic curl error "No route to host."

3. Gateway — can we reach the first hop?

ping -c3 192.168.1.1
ip neigh show

ping tests reachability; ip neigh shows the ARP table. A gateway entry in state REACHABLE with a MAC address means L2 is fine. FAILED or INCOMPLETE means the gateway isn't answering ARP — a VLAN, cabling, or firewall problem. Note that many hosts drop ICMP, so treat a failed ping as a hint, not a verdict — trust the neighbor state.

4. DNS — does the name resolve, and to the right thing?

dig +short api.example.com

An empty answer or "could not resolve host" from curl means DNS is your layer. Compare the returned IP against what you expect — resolving successfully to a stale address is a subtle failure that looks like an application bug. Add dig api.example.com (no +short) to see the SERVER: line and confirm which resolver answered.

5. Port — is the service actually listening/reachable?

nc -vz api.example.com 443

Three outcomes, three different meanings — this is the most diagnostic step:

"succeeded" / "open" — TCP handshake completed, move up to TLS.
"Connection refused" — you reached the host but nothing is listening on 443 (service down, or wrong port). The host answered fast with a RST.
Timeout / hangs — a firewall or security group is silently dropping packets. The signature is the wait, not an immediate error.

Refused vs. timeout is the single most valuable distinction in network debugging: refused = reached the host, timeout = something ate the packet. On the server, ss -tlnp | grep :443 confirms the process is bound to the right address (a service on 127.0.0.1 won't accept external traffic).

6. TLS — does the handshake complete and the cert validate?

openssl s_client -connect api.example.com:443 -servername api.example.com </dev/null

Read the Verify return code at the bottom. 0 (ok) means the chain is valid. An expired cert, wrong hostname, or missing intermediate shows up here as a nonzero code (e.g. 21 unable to verify the first certificate) — long before your application logs blame something vague. The -servername flag sends SNI, which matters on shared hosts serving multiple certs.

7. Application — now, finally, the app

curl -v https://api.example.com/health

Only when everything below is green does an app-layer error (HTTP 500, 401, a slow backend) actually mean the app. The -v output replays the whole climb — DNS, connect, TLS handshake, request, response headers — so it's also a great one-shot sanity check to confirm your layer-by-layer conclusion.

Why this beats guessing

Each command targets exactly one layer and each failure has a distinct signature: no route to host (route), could not resolve (DNS), connection refused (port, service down), timeout (port, firewall drop), verify return code != 0 (TLS). Learn the signatures and you skip straight to the broken layer instead of restarting things at random.

I put the full layer-by-layer walkthrough, with these tools built into a portable toolbox, in the Kali Linux Networking for DevOps series — a 15-lesson path that frames Kali as a disposable, throwaway diagnostics box, not a "learn to hack" course. The Linux routing lesson and the DNS troubleshooting lesson go deep on the two layers people misdiagnose most.

One ground rule: only probe, scan, or capture on systems you own or are explicitly authorized to assess — this is defensive infrastructure validation, not an excuse to poke at other people's networks.

Try it next time

Next incident, resist the restart reflex — climb the stack from ip addr upward and let the failure signature name the layer. If you want the guided version, work through the full 15-lesson networking series; the site has free hands-on Kali learning paths built specifically for DevOps engineers.

Nmap for Authorized Infrastructure Validation (Not Hacking)

James Joyner — Thu, 13 Aug 2026 21:39:05 +0000

Every deploy makes a promise about the network: "this box only exposes SSH and HTTPS," "the database is never reachable from outside the app tier." Nmap is how you turn that promise into a test that either passes or fails. Nobody has to take the security group's word for it.

One rule before anything else: only scan systems you own or are explicitly authorized to assess. Point Nmap at a lab, a VM you control, or your own infrastructure. This is authorized infrastructure validation — a defensive check on exposure you're responsible for, not "hacking."

Start with what's actually listening

The most basic useful run is a host scan:

nmap 192.168.56.10

This does host discovery and a default TCP scan of the common ports. The output lists each port as open, closed, or filtered. open means something accepted the connection. filtered usually means a firewall or security group silently dropped the packet — which is exactly the signal you want when validating that a rule is doing its job. If you expected a wall of filtered and instead see open, that's your finding.

When you already know what should be exposed, scan for exactly that and nothing else:

nmap -p 22,80,443 host

Narrowing to the declared ports keeps the scan fast and the output readable. The question you're answering isn't "what's out there" — it's "does observed reality match what I declared?"

Confirm what's really on the port

An open port tells you a socket is listening. It does not tell you what. For that, add version detection:

nmap -sV -p 22,80,443 host

-sV probes each open port and reports the service and, when it can, the version banner. This matters because ports lie. A service you assumed was nginx on 443 might be something a teammate stood up last week. Read the SERVICE and VERSION columns and ask: is this the thing I expected, at the version I expected? A mismatch here is often the first sign of drift or a forgotten container.

A methodology, not just commands

Running Nmap ad hoc gives you trivia. Running it as a pipeline gives you a regression test for your network posture. The loop I use after every deploy:

Declared Configuration → Expected Exposure → Observed Exposure → Compare → Correct → Retest.

Declared Configuration — the source of truth: the security group, firewall rules, or Terraform that says what's allowed.
Expected Exposure — translate that into a concrete list of ports that should answer. For a web tier: 22 (from bastion only) and 443.
Observed Exposure — what Nmap actually sees.
Compare — diff the two. Every open port that isn't in the expected list is a finding.
Correct — fix the rule, the container publish, or the config.
Retest — scan again. The finding must be gone. No retest, no fix.

A concrete finding

Say the declared config for a web host is SSH and HTTPS only. Expected exposure: 22, 443. You run:

nmap -sV -p 22,443,3306 web-host.internal

And 3306 comes back:

PORT     STATE  SERVICE  VERSION
22/tcp   open   ssh      OpenSSH ...
443/tcp  open   https    ...
3306/tcp open   mysql    MySQL ...

3306 is MySQL, and it was never in the declared config. Observed doesn't match expected — that's the finding. The hypothesis writes itself: a container published 3306:3306 to the host instead of binding to an internal network, or a security group rule is broader than intended. Correct it (bind the DB to the app network, tighten the rule), then rerun the exact same scan. When 3306 comes back filtered or absent, the loop closes.

That Declared → Observed → Retest discipline is the core of the full Nmap network validation walkthrough, which builds this into a repeatable post-deploy check.

A note on scan types and privileges

Some scan types need elevated privileges. A SYN scan (-sS), for example, crafts raw packets and requires CAP_NET_RAW — run unprivileged, Nmap quietly falls back to a slower connect scan. In a container, grant only that capability:

docker run --rm --cap-add NET_RAW instrumentisto/nmap -sS -p 22,443 host

Never reach for --privileged. It hands the container the full capability set and effectively removes the kernel's isolation between it and the host — a wildly oversized trade for one raw-socket permission. --cap-add NET_RAW grants exactly what the scan needs and nothing more. If you're wiring Nmap into container-based checks, the Docker-based service discovery lesson covers the disposable-toolbox pattern in depth.

Wrap up

Nmap earns its place in a DevOps toolkit not as an offensive tool but as an assertion engine: it makes "the firewall should only allow X" something you can prove after every change. Work through the Nmap network validation lesson to turn this loop into a habit — and the site has free, hands-on Kali networking learning paths aimed squarely at DevOps engineers who'd rather test their infrastructure than trust it.

Keep Docker Engine as Your Kubernetes Runtime on Ubuntu with cri-dockerd

James Joyner — Tue, 04 Aug 2026 23:35:35 +0000

Sometimes you genuinely want Docker Engine as the Kubernetes node runtime — a team standardized on the Docker CLI/API for tooling, an image-build box that doubles as a node, or a legacy playbook you can't rewrite yet. Since dockershim was removed in Kubernetes 1.24, that path now runs through cri-dockerd, an open-source CRI adapter maintained by Mirantis. Here's how to set it up on Ubuntu, and an honest note on whether you should.

Should you? A quick reality check

For most people the answer is no — use containerd (see the previous post). Reach for cri-dockerd only when you have a concrete reason:

Existing automation that talks to the Docker socket on the node.
You want docker build and the kubelet on the same host sharing one image store.
A vendor or product that still assumes Docker Engine as the runtime.

If none of those apply, containerd is less to install and less to break.

1. Install Docker Engine on Ubuntu

Use Docker's official repo, not the docker.io package, so you get current Engine:

sudo apt-get update
sudo apt-get install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" \
  | sudo tee /etc/apt/sources.list.d/docker.list >/dev/null

sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io

Set Docker's cgroup driver to systemd — on Ubuntu's cgroup v2 this must match the kubelet or the node will not stay Ready:

sudo mkdir -p /etc/docker
cat <<'EOF' | sudo tee /etc/docker/daemon.json
{ "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": {"max-size": "100m"} }
EOF
sudo systemctl restart docker
sudo systemctl enable docker

2. Install cri-dockerd

Grab the latest release for your architecture:

VER=0.3.15   # check github.com/Mirantis/cri-dockerd/releases for current
ARCH=$(dpkg --print-architecture)
curl -fsSLo cri-dockerd.deb \
  "https://github.com/Mirantis/cri-dockerd/releases/download/v${VER}/cri-dockerd_${VER}.3-0.ubuntu-noble_${ARCH}.deb"
sudo dpkg -i cri-dockerd.deb

sudo systemctl enable --now cri-docker.socket
systemctl status cri-docker.socket

The CRI socket it exposes is the value you'll hand to kubeadm:

unix:///var/run/cri-dockerd.sock

3. Point kubeadm at the cri-dockerd socket

Everything else in the node setup (kernel modules, sysctl, swap off, the pkgs.k8s.io repo) is identical to the containerd post — only the CRI endpoint changes:

sudo kubeadm init \
  --cri-socket=unix:///var/run/cri-dockerd.sock \
  --pod-network-cidr=10.244.0.0/16

Worker joins must carry the same flag:

sudo kubeadm join <cp>:6443 \
  --token <token> --discovery-token-ca-cert-hash sha256:<hash> \
  --cri-socket=unix:///var/run/cri-dockerd.sock

4. Verify

kubectl get nodes -o wide
# CONTAINER-RUNTIME now reads docker://<engine-version>

The gotcha nobody warns you about

With cri-dockerd, images pulled by the kubelet and images you docker pull live in the same store, which is convenient — but docker ps will show a lot of Kubernetes-managed containers you didn't start. Use crictl (CRI-aware) for cluster containers and reserve docker for images you manage by hand, so you don't accidentally docker rm something the kubelet owns.

sudo crictl --runtime-endpoint unix:///var/run/cri-dockerd.sock ps

If the node comes up NotReady or pods can't pull, it's nearly always the cgroup-driver mismatch or the socket path — both covered in the Kubernetes pod-startup error hub and the Docker runtime error guides.

Next in the series: dropping the whole node dance and running Kubernetes inside Docker with kind for local development.

Set Up a Kubernetes Node on Ubuntu 24.04 with containerd (the Docker-Compatible Runtime)

James Joyner — Fri, 31 Jul 2026 16:28:05 +0000

This is the mainstream way to run Kubernetes "with Docker" on Ubuntu in 2026: containerd as the runtime (the same engine Docker uses under the hood), your Docker-built images running unchanged. Here's a clean kubeadm node bring-up on Ubuntu 24.04 (Noble) with the sharp edges called out.

Run everything below as root or with sudo. This sets up a single control-plane node; join workers with the kubeadm join command printed at the end.

1. Kernel prerequisites

Kubernetes needs bridged traffic to hit iptables and IP forwarding on:

cat <<'EOF' | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter

cat <<'EOF' | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
sudo sysctl --system

Disable swap — the kubelet refuses to start with swap on by default:

sudo swapoff -a
sudo sed -i.bak '/\bswap\b/s/^/#/' /etc/fstab

2. Install containerd

sudo apt-get update
sudo apt-get install -y containerd
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml >/dev/null

The one edit that matters on Ubuntu. Ubuntu 22.04+ uses cgroup v2, so containerd and the kubelet must both use the systemd cgroup driver or the node flaps between Ready/NotReady:

sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl restart containerd
sudo systemctl enable containerd

(This mismatch is common enough that I gave it its own post later in the series.)

3. Install kubeadm, kubelet, kubectl

The old apt.kubernetes.io repo was retired — use pkgs.k8s.io. Pin the minor version you want (1.30 shown):

sudo apt-get install -y apt-transport-https ca-certificates curl gpg
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key \
  | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' \
  | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl   # don't let an unattended upgrade skip a minor

4. Initialize the control plane

Pick a pod CIDR that matches your CNI. This uses Flannel's default:

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

# set up kubectl for your user
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

5. Install a CNI, then verify

kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

kubectl get nodes -o wide
# STATUS should go Ready within a minute; CONTAINER-RUNTIME shows containerd://1.7.x

For a single-node lab, let workloads schedule on the control plane:

kubectl taint nodes --all node-role.kubernetes.io/control-plane-

6. Prove Docker images run

kubectl create deployment web --image=nginx:1.27
kubectl expose deployment web --port=80
kubectl get pods -o wide     # Running, on your containerd node

That nginx image was built by Docker. It runs on containerd without translation because it's an OCI image — which is the whole point of this series.

When it doesn't come up

The usual first-node failures are the kubelet not starting (almost always swap or the cgroup driver), or pods stuck pending (no CNI yet). I keep the specific fixes here: Kubernetes pod-startup errors and the Kubernetes troubleshooting stack.

Next: if you specifically need Docker Engine as the runtime rather than bare containerd, that's cri-dockerd.

Run Kubernetes in Docker on Ubuntu for Local Development

James Joyner — Sun, 26 Jul 2026 18:26:50 +0000

There's a delightfully literal answer to "Kubernetes with Docker": kind — Kubernetes IN Docker. Each node is a Docker container running a full Kubernetes node image. On an Ubuntu workstation it gives you a real, throwaway, multi-node cluster in about 30 seconds. It's my default for local dev and for CI.

Prerequisites on Ubuntu

You need Docker Engine and kubectl. If you don't have Docker yet:

sudo apt-get update && sudo apt-get install -y docker.io
sudo usermod -aG docker $USER && newgrp docker   # run docker without sudo

Install kind (single static binary):

curl -fsSLo ./kind https://kind.sigs.k8s.io/dl/latest/kind-linux-amd64
chmod +x ./kind && sudo mv ./kind /usr/local/bin/kind
kind version

A one-command cluster

kind create cluster --name dev
kubectl cluster-info --context kind-dev
docker ps    # you'll see a dev-control-plane container — that's your node

kind wrote a kubeconfig context for you. Tear the whole thing down just as fast:

kind delete cluster --name dev

A realistic multi-node cluster

Most bugs only show up with more than one node (scheduling, affinity, PodDisruptionBudgets). Define it in a config file:

# kind-cluster.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    kubeadmConfigPatches:
      - |
        kind: InitConfiguration
        nodeRegistration:
          kubeletExtraArgs:
            node-labels: "ingress-ready=true"
    extraPortMappings:
      - containerPort: 80
        hostPort: 8080
        protocol: TCP
  - role: worker
  - role: worker

kind create cluster --name dev --config kind-cluster.yaml
kubectl get nodes

The extraPortMappings bit is the trick people miss: it forwards a port from your Ubuntu host into the control-plane container, so an ingress controller inside the cluster is reachable at http://localhost:8080.

Loading a locally-built image (no registry needed)

This is kind's best feature for the Docker workflow. Build with Docker, push straight into the cluster's nodes — no registry round-trip:

docker build -t myapp:dev .
kind load docker-image myapp:dev --name dev

kubectl create deployment myapp --image=myapp:dev
kubectl set image deployment/myapp myapp=myapp:dev   # after a rebuild + reload

Set imagePullPolicy: IfNotPresent (or Never) in your manifest for locally-loaded images, or the kubelet will try to pull myapp:dev from a registry and fail with ImagePullBackOff.

Ubuntu-specific gotchas

cgroup / inotify limits. Big clusters on kind can exhaust inotify watches. If pods crashloop with "too many open files," raise them:

  sudo sysctl fs.inotify.max_user_watches=524288
  sudo sysctl fs.inotify.max_user_instances=512

Rootless Docker works but needs cgroup v2 delegation configured; if kind create hangs, test with rootful Docker first.
It's ephemeral by design. A kind node is a container — restart Docker and the cluster state is gone unless you use extraMounts for persistence. That's a feature for testing, a footgun if you treated it like a server.

When a kind pod won't start, it's the same Kubernetes failures as anywhere — ImagePullBackOff from the pull-policy trap above, or CrashLoopBackOff from the app itself. Fixes here: ImagePullBackOff and CrashLoopBackOff.

Next: minikube with the Docker driver — a heavier but more full-featured local option.

minikube with the Docker Driver on Ubuntu: A Practical Local Cluster

James Joyner — Sat, 25 Jul 2026 12:34:10 +0000

minikube is the other "Kubernetes in Docker" option on Ubuntu, and with --driver=docker it runs the cluster inside a Docker container just like kind — but ships with addons (ingress, metrics-server, dashboard, a built-in registry) that make it feel more like a real cluster. Here's a practical setup and how it differs from kind.

Install on Ubuntu

You need Docker first (sudo apt-get install -y docker.io, then add yourself to the docker group). Then:

curl -fsSLo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube /usr/local/bin/minikube
minikube version

Start with the Docker driver

minikube start --driver=docker
# make it the default so you don't repeat the flag:
minikube config set driver docker

kubectl get nodes
docker ps    # a 'minikube' container is your node

Size it for real work:

minikube start --driver=docker --cpus=4 --memory=8g --disk-size=40g

The addons are the reason to pick minikube

minikube addons list
minikube addons enable ingress
minikube addons enable metrics-server
minikube dashboard        # opens the web UI

ingress gives you a working NGINX ingress controller with no manifest wrangling — genuinely useful when you want to test ingress routing locally.

The Docker image workflow

minikube runs its own Docker daemon inside the node container. The neat trick is pointing your shell's Docker CLI at that daemon, so images you build are immediately visible to the cluster with no push:

eval $(minikube docker-env)     # your `docker` now talks to minikube's daemon
docker build -t myapp:dev .
kubectl create deployment myapp --image=myapp:dev
# remember: imagePullPolicy: IfNotPresent so it doesn't try a registry pull

Undo it when you're done so docker points back at your host daemon:

eval $(minikube docker-env -u)

There's also a built-in registry if you prefer the push model:

minikube addons enable registry

Accessing services from Ubuntu

Two common patterns:

# quick tunnel to a single service (prints a URL)
minikube service myapp --url

# LoadBalancer support on your host (needs sudo; keep it running in a terminal)
minikube tunnel

minikube vs kind — how I choose

kind: faster start, lighter, multi-node config is trivial, ideal for CI and quick throwaways.
minikube: heavier, single-node by default, but addons (ingress, metrics, registry, dashboard) and the docker-env trick make it nicer for interactive local development.

Both run "Kubernetes with Docker" in the most literal sense — the cluster is a Docker container. Neither reflects how production nodes run (that's the containerd/cri-dockerd posts earlier in this series), so don't debug production runtime issues on them.

Cleanup

minikube stop        # keep the cluster, free the resources
minikube delete      # nuke it entirely

When a workload misbehaves on minikube it's ordinary Kubernetes troubleshooting — start with kubectl describe pod and kubectl logs, and the pod-startup error hub for the specific messages.

Next: the failure mode that quietly breaks more Ubuntu nodes than anything else — the systemd cgroup driver mismatch.

From Docker Build to Kubernetes Deploy on Ubuntu: The Image Workflow That Never Changed

James Joyner — Wed, 22 Jul 2026 21:31:12 +0000

Amid all the noise about dockershim, one thing got lost: the everyday workflow of building an image with Docker and running it on Kubernetes never changed. Docker is still an excellent build tool, Kubernetes still runs OCI images, and on Ubuntu the loop is clean. Here it is end to end.

1. A build-friendly Dockerfile

Multi-stage keeps the runtime image small and the attack surface low — this matters more on Kubernetes, where you pull the image onto every node that schedules the pod:

# build stage
FROM golang:1.22 AS build
WORKDIR /src
COPY go.* ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /out/api ./cmd/api

# runtime stage — distroless, no shell, tiny
FROM gcr.io/distroless/static:nonroot
COPY --from=build /out/api /api
USER nonroot:nonroot
EXPOSE 8080
ENTRYPOINT ["/api"]

2. Build and push with Docker on Ubuntu

Use buildx (bundled with modern Docker) so you can build multi-arch — worth it if any nodes are arm64:

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t registry.example.com/api:1.4.2 \
  --push .

Tag with an immutable version, never rely on :latest. Kubernetes caches images per node; :latest makes "which build is actually running?" unanswerable and breaks rollbacks.

3. A deployment that behaves in production

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  selector: { matchLabels: { app: api } }
  template:
    metadata: { labels: { app: api } }
    spec:
      containers:
        - name: api
          image: registry.example.com/api:1.4.2   # the exact tag you pushed
          imagePullPolicy: IfNotPresent
          ports: [{ containerPort: 8080 }]
          resources:
            requests: { cpu: "100m", memory: "128Mi" }
            limits:   { memory: "256Mi" }
          readinessProbe:
            httpGet: { path: /healthz, port: 8080 }
            initialDelaySeconds: 3
          livenessProbe:
            httpGet: { path: /healthz, port: 8080 }
            initialDelaySeconds: 10

The readinessProbe is the piece people skip and regret: without it, Kubernetes sends traffic to a pod before your app is listening, and you get intermittent 502s during every rollout.

kubectl apply -f deployment.yaml
kubectl rollout status deployment/api

4. Pulling from a private registry

If your registry needs auth, the cluster needs a pull secret — this is the same regardless of whether nodes run containerd or cri-dockerd:

kubectl create secret docker-registry regcred \
  --docker-server=registry.example.com \
  --docker-username=ci \
  --docker-password="$REGISTRY_TOKEN"

spec:
  imagePullSecrets:
    - name: regcred

Miss this and pods sit in ImagePullBackOff with pull access denied.

5. Ship a new build

docker buildx build -t registry.example.com/api:1.4.3 --push .
kubectl set image deployment/api api=registry.example.com/api:1.4.3
kubectl rollout status deployment/api
# rollback is one command because you used immutable tags:
kubectl rollout undo deployment/api

The mental model

Docker = how you build and push images. Unchanged, still great.
containerd / cri-dockerd = how the node runs them. This is what dockershim's removal was about.
Your manifests = don't care which runtime is underneath.

If you keep those three separate in your head, none of the 2022 runtime drama affects your daily loop.

When a rollout goes wrong it's usually the image pull or the probe: ImagePullBackOff, and the wider Docker runtime error guides for build-side failures.

Last in the series: when the node or the runtime itself is the problem — troubleshooting kubelet and containerd on Ubuntu.

State Encryption in OpenTofu: How It Works and How to Roll It Out

James Joyner — Mon, 20 Jul 2026 15:54:27 +0000

If you've ever cat-ed a Terraform or OpenTofu state file, you already know the uncomfortable truth: it's a plaintext JSON dump of everything your infrastructure knows, including secrets. Database passwords, generated private keys, API tokens injected through providers — they all land in state, in the clear. OpenTofu is the one place where you can fix this at the source, because native state and plan encryption is a first-class OpenTofu feature that upstream Terraform does not have. Here's how it actually works and how I roll it out on existing projects without breaking them.

Why plaintext state is a real risk

State is not a cache you can regenerate. It's the authoritative map between your HCL and the real resources, and OpenTofu has to store the values of attributes to compute diffs. That includes sensitive ones. Marking an output sensitive = true only hides it from the CLI output — it's still written verbatim to state.

On real infra I've seen state end up in three places it shouldn't: an S3 bucket without SSE and with overly broad read IAM, a CI artifact that got uploaded to a build cache, and a developer laptop with terraform.tfstate committed to a feature branch by accident. Backend encryption (like S3 SSE) helps for one of those. It does nothing for the other two, because the moment state leaves the backend it's plaintext again.

OpenTofu's encryption operates at the data layer, before the bytes ever hit the backend or a local file. The state is encrypted at rest everywhere: in the backend, in local copies, in CI artifacts. That's the property I want.

Anatomy of the `encryption` block

Encryption lives in a terraform { encryption { ... } } block. It has three moving parts: a key provider (where the encryption key comes from), a method (the actual cipher), and targets (state and/or plan) that bind a method to what you want encrypted.

terraform {
  encryption {
    key_provider "pbkdf2" "passphrase" {
      passphrase = var.tofu_encryption_passphrase
    }

    method "aes_gcm" "default" {
      keys = key_provider.pbkdf2.passphrase
    }

    state {
      method = method.aes_gcm.default
    }

    plan {
      method = method.aes_gcm.default
    }
  }
}

That's the whole minimal setup. pbkdf2 derives a key from a passphrase, aes_gcm is AES-GCM authenticated encryption, and both the state and plan targets use it. Note the passphrase comes from a variable — never hardcode it.

One caveat worth knowing (as of 2026 — check current docs): you generally can't pull the passphrase from a normal input variable defined elsewhere, because the encryption block is evaluated very early, before most of the graph. In practice I feed it from the environment instead, which I'll cover below. Treat the var. reference above as illustrative and prefer the env-var approach for the passphrase itself.

Key providers: passphrase vs KMS

The pbkdf2 passphrase provider is the easiest to start with and the easiest to get wrong operationally. If you lose the passphrase, the state is gone — there is no recovery. It's great for a solo project or a quick proof of concept, but the passphrase becomes a secret you now have to manage carefully.

For anything shared or production, I use a cloud KMS provider so the key material lives in a managed HSM-backed service and access is controlled by IAM:

terraform {
  encryption {
    key_provider "aws_kms" "main" {
      kms_key_id = "arn:aws:kms:us-east-1:111122223333:key/abcd-1234"
      region     = "us-east-1"
      key_spec   = "AES_256"
    }

    method "aes_gcm" "kms" {
      keys = key_provider.aws_kms.main
    }

    state {
      method   = method.aes_gcm.kms
      enforced = true
    }

    plan {
      method = method.aes_gcm.kms
    }
  }
}

There are equivalent gcp_kms and openbao/Vault-style providers too. The pattern is identical: the key_provider block changes, the method and targets stay the same. With KMS, access control and audit logging come for free — I can see in CloudTrail exactly who decrypted state and when, and I can revoke a role without rotating the underlying data key.

Encrypt the plan file too

People forget plan files. A tofu plan -out=tfplan binary contains the same resource values as state, plus the proposed changes. If your CI pipeline runs plan in one job and apply in another, that plan artifact is passed between jobs — and it's just as sensitive as state. The plan {} target above encrypts it with the same method. Do not skip it.

Rolling it out on an existing project with `fallback`

The scary part is turning encryption on when you already have unencrypted state. If you just add the encryption block, the next tofu plan will fail trying to decrypt state that was never encrypted. The fallback block is the migration escape hatch: it tells OpenTofu "if you can't decrypt with the primary method, treat the data as unencrypted."

terraform {
  encryption {
    key_provider "aws_kms" "main" {
      kms_key_id = "arn:aws:kms:us-east-1:111122223333:key/abcd-1234"
      region     = "us-east-1"
    }

    method "aes_gcm" "kms" {
      keys = key_provider.aws_kms.main
    }

    state {
      method = method.aes_gcm.kms

      fallback {
        # no method = read plaintext state, write encrypted
      }
    }
  }
}

The migration is a one-time apply:

tofu plan -out=tfplan
tofu apply tfplan
# state is now written back ENCRYPTED with the primary method

Because fallback with no method means "read as plaintext," OpenTofu can read your old state, and because the primary method is set, it writes the new state encrypted. Run one apply, confirm the state in the backend is now ciphertext, then remove the fallback block so plaintext state can no longer be silently accepted.

Lock it down with `enforced`

Once you've migrated, add enforced = true on the target. This is the setting that turns encryption from optional to mandatory:

state {
  method   = method.aes_gcm.kms
  enforced = true
}

With enforced, OpenTofu refuses to read or write unencrypted state at all. No accidental fallback, no misconfiguration silently dropping to plaintext. On a team, this is the line that guarantees nobody's local run produces a plaintext terraform.tfstate.

Key rotation

Rotation doesn't require re-encrypting everything in one shot. A method can hold a list of keys; the first is used to encrypt, and all of them are tried for decryption. To rotate, add a new key provider and put it ahead of the old one in the method's keys:

method "aes_gcm" "kms" {
  keys = [
    key_provider.aws_kms.new,   # new key: used for encryption
    key_provider.aws_kms.old,   # old key: still valid for decryption
  ]
}

The next apply writes state encrypted with the new key while still being able to read anything encrypted with the old one. After you've applied and confirmed all state is on the new key, drop the old provider. Same mechanism works for migrating from passphrase to KMS.

CI and secrets handling

For the passphrase provider, feed the secret through the environment, not a .tfvars file. OpenTofu reads encryption config from env vars prefixed appropriately, and in CI I inject it as a masked secret:

export TF_ENCRYPTION='key_provider "pbkdf2" "passphrase" {
  passphrase = "'"$TOFU_PASSPHRASE"'"
}'
tofu apply -auto-approve

For KMS, there's no passphrase to leak — the CI runner just needs an IAM role that can call kms:Decrypt/kms:GenerateDataKey on that key, which is exactly the least-privilege boundary you want. That's the strongest argument for KMS over passphrase in a pipeline: the secret never exists as a string anywhere.

When I get stuck on the migration edge cases — mixed fallback states, rotation ordering, backend quirks — I keep notes and error write-ups in my OpenTofu troubleshooting guides, because the failure messages during a half-migrated encryption rollout are not always obvious.

Takeaway

Turn on encryption with a fallback block, run one tofu apply to migrate existing state, remove fallback, then set enforced = true and encrypt the plan target too. Use pbkdf2 only for throwaway projects and reach for aws_kms/gcp_kms for anything real, so the key lives in a managed service with IAM and audit logging. Rotate by prepending a new key provider ahead of the old one and applying once. Plaintext state is a solved problem in OpenTofu — you just have to opt in.

Migrating from Terraform to OpenTofu: A Low-Risk Playbook

James Joyner — Sun, 19 Jul 2026 19:30:09 +0000

I've migrated a few real environments from Terraform to OpenTofu now, and the good news is that a careful migration is almost boring. The state format is compatible, the CLI is a near drop-in, and the whole thing can be done with a rollback path at every step. The bad news is that "almost boring" still has a couple of sharp edges, and the teams that get hurt are the ones who skip the parity check and go straight to apply. Here's the calm, low-risk playbook I actually follow.

Step 0: Know what "low-risk" means here

The core insight that makes this safe: OpenTofu reads the same HCL and the same state file that Terraform does. A migration is not a rewrite — it's swapping which binary talks to your existing state. That means at almost every step, your rollback is just "keep using the terraform binary." As long as you don't trigger a one-way-door feature (more on those later), you can walk back.

So the whole strategy is: prove parity before you change anything real, change one thing at a time, and keep the old binary installed until you're confident.

Step 1: Check and pin your Terraform version first

Before you touch OpenTofu, get your current setup deterministic. OpenTofu forked from the last MPL-licensed Terraform, so very old or very new Terraform configs can have edges. Find out exactly what you're running:

terraform version

Pin it. If you're not already using a version manager or a pinned CI image, do that now — you want a fixed, known-good Terraform baseline to compare against and to fall back to. Also pin your provider versions in a lockfile:

terraform providers lock

A migration where both the tool version and the provider versions are floating is a migration where you can't tell what caused a diff. Lock everything down first.

Step 2: Install `tofu` alongside, not instead

Install OpenTofu without removing Terraform. On a workstation or a scratch CI runner:

# Verify the binary is there and note the version
tofu version

Keep both terraform and tofu on PATH during the migration. You'll be running them back to back to compare, and having both is what makes rollback trivial. As of 2026 the install methods and current versions are in the OpenTofu docs — check them rather than trusting a version number from a blog post (including this one).

Step 3: The parity check — init and plan, no apply

This is the heart of the whole exercise. Work on a copy or a non-production workspace first. Point OpenTofu at your existing configuration and existing state, initialize, and produce a plan — but do not apply.

# Fresh working dir state, same backend/state as before
tofu init

# The critical test: does OpenTofu see zero changes?
tofu plan -out tofu.plan

What you want to see is a clean, no-changes plan. If OpenTofu reads your Terraform-written state and reports that nothing needs to change, you have parity. That's the green light.

If the plan shows drift, stop and read it carefully before doing anything. Common causes I've hit:

Provider version differences. OpenTofu resolved a slightly different provider than your pinned Terraform lockfile. Reconcile the versions.
Registry source differences. OpenTofu uses its own registry; a provider or module might resolve from a different source. Verify the provider actually publishes where OpenTofu looks.
A genuinely different interpretation of some config. Rare, but read the diff — do not apply your way past it.

Do this parity check per module/workspace, not once globally. State lives per-workspace, and a clean plan in one doesn't guarantee a clean plan in another.

Step 4: Apply once, deliberately, in a safe place

Once you've got a clean plan in a non-prod workspace, run the apply there so OpenTofu writes state at least once:

tofu apply tofu.plan

Even a no-op apply may rewrite state metadata. That's fine and expected — but it's the moment worth noting, because after OpenTofu writes state, that workspace's state has been touched by tofu. Terraform can generally still read it, but this is the point where you start treating that workspace as "OpenTofu-managed." Do it somewhere you can afford to be wrong before you do it in production.

Step 5: Swap CI, one pipeline at a time

Now change the automation. In your CI config, this is usually as small as swapping the binary and the command name:

# Before
terraform init && terraform plan -out plan.tfout

# After
tofu init && tofu plan -out plan.tfout

Roll it out per-pipeline, lowest-stakes environment first. Keep the plan-review gate in your pipeline — a human or a required approval looking at the plan output — for the first few runs on each environment. The whole point of a slow rollout is that if OpenTofu ever produces a plan you didn't expect, you catch it at plan time, not after apply.

I also recommend keeping a terraform-based fallback job available (even if disabled) during the transition, so reverting CI is a one-line change rather than an archaeology project.

Step 6: Watch for the one-way doors

Everything above is reversible as long as your config stays compatible with both tools. The way you lose your rollback is by adopting an OpenTofu-only feature. The big ones to be aware of:

Native state/plan encryption. Once OpenTofu encrypts your state, stock Terraform can't read it. This is a feature you may want — but adopt it as a deliberate, post-migration decision, not mid-migration.
Early variable evaluation in backend blocks or module sources. Configs that rely on it won't parse under Terraform.
.tofu / .tofu.json override files and provider-defined functions via the provider:: namespace. Both are OpenTofu-specific surface area.

My rule during the migration window: change the tool, not the config. Keep your HCL dual-compatible until every environment is on OpenTofu and stable. Only then start adopting the divergent features — and when you do, understand you're closing the door behind you. If you want more detail on the specific compatibility gotchas and error messages these features throw, I keep a running set of OpenTofu troubleshooting notes from real migrations.

Step 7: How to actually roll back

If something goes wrong before you've crossed a one-way door, rollback is genuinely simple:

Switch the binary back. In CI and locally, tofu becomes terraform again.
Re-init with Terraform so its lockfile and provider selections are in place: terraform init.
Run a plan and confirm a clean, no-change result: terraform plan.
Restore state from backup only if you actually corrupted or encrypted it. This is why you keep versioned state — an S3 bucket with versioning, or whatever your backend offers, so you can retrieve the pre-migration state object.

Back up your state before you start. A cheap tofu state pull > backup.tfstate (or the Terraform equivalent) before the first apply gives you a plain escape hatch. I've never had to use it on a careful migration, but the whole reason the migration feels calm is that the backup exists.

Takeaway

Migrating from Terraform to OpenTofu is mostly a swap, not a rewrite, and the parity check is what makes it safe: prove OpenTofu reads your existing state with a clean plan before you change anything real. Pin your versions, keep both binaries installed, roll CI out one environment at a time, and don't adopt one-way-door features until you're fully migrated and stable. Do it in that order and the scariest part of the whole thing will be how uneventful it is.

OpenTofu vs Terraform in 2026: What Actually Changed

James Joyner — Sat, 18 Jul 2026 14:30:45 +0000

I've been running both Terraform and OpenTofu across production infra for a while now, and the number one question I still get is some version of "wait, aren't they the same thing?" The honest answer in 2026 is: they share a common ancestor and a lot of DNA, but they are no longer the same tool. Here's what actually changed, from someone who has to keep both green in CI.

A very short history of the fork

If you missed the drama: HashiCorp relicensed Terraform from the MPL open-source license to the Business Source License (BSL) in 2023. The BSL is source-available but not OSI-approved open source, and it carries a use restriction aimed at competitors. A chunk of the community, backed by a group of vendors and users, forked the last MPL-licensed Terraform codebase. That fork landed under the Linux Foundation as OpenTofu.

So the core distinction is governance, not features: OpenTofu is a Linux Foundation project with open governance and an MPL-2.0 license, and Terraform is a HashiCorp product under the BSL. That licensing split is the reason a lot of teams looked at OpenTofu at all — if your legal team is nervous about the BSL's use restriction, an actual open-source license is the whole ballgame.

The CLI and state are (mostly) drop-in compatible

The thing that makes OpenTofu practical to adopt is that it started as a literal fork. The binary is tofu instead of terraform, and for a lot of everyday work it behaves identically:

tofu init
tofu plan -out plan.tfout
tofu apply plan.tfout
tofu state list

It reads your existing .tf files, understands the same HCL, and uses the same state file format. On real infra I've pointed tofu at a state file that was last touched by Terraform and had it produce a clean, no-change plan. That parity is not an accident — keeping the state format compatible is what makes migration a low-drama exercise rather than a rewrite.

But "mostly compatible" is doing real work in that sentence. The two projects have been diverging since the fork, and the gap widens with every release. Treating them as interchangeable is where teams get burned, so let's talk about the divergences that actually matter.

The real divergences in 2026

State and plan encryption, natively

This is the feature I care about most. OpenTofu ships native state and plan encryption. You configure it directly in your OpenTofu configuration, pick a key provider (PBKDF2 with a passphrase, a cloud KMS, and so on) and a method, and OpenTofu encrypts state at rest — including the plan file, which can leak secrets just as badly as state.

terraform {
  encryption {
    key_provider "pbkdf2" "mykey" {
      passphrase = var.encryption_passphrase
    }

    method "aes_gcm" "default" {
      keys = key_provider.pbkdf2.mykey
    }

    state {
      method = method.aes_gcm.default
    }

    plan {
      method = method.aes_gcm.default
    }
  }
}

In the Terraform world you traditionally solved this at the backend level — a KMS-encrypted S3 bucket, restrictive IAM, and hoping nobody terraform shows a plan file into a CI log. OpenTofu moves encryption into the tool itself. If you handle regulated data, this alone can justify the switch.

Early variable evaluation

For years the answer to "why can't I use a variable in my backend block?" was a shrug. OpenTofu added early variable evaluation, which lets you use variables (and locals) in places that used to demand static literals — most notably backend configuration and module sources. That means you can drive your backend bucket or key by variable instead of maintaining a wall of -backend-config flags or partial-backend hacks.

It's genuinely useful, and it's also a one-way door: a config that relies on early eval in a backend block won't parse cleanly under stock Terraform. Keep that in mind before you sprinkle it everywhere.

Provider-defined functions

Both ecosystems moved toward letting providers ship their own functions rather than waiting for the core team to add every string-munging helper. In OpenTofu you call them through the provider:: namespace:

locals {
  parsed = provider::aws::arn_parse(var.role_arn)
}

The exact set of available functions depends on the provider version you're pinning, so I won't quote a catalog — check the current provider docs. The point is that the language surface is no longer frozen to whatever core ships.

`.tofu` and `.tofu.json` override files

This is a small feature with big ergonomic payoff. OpenTofu recognizes .tofu and .tofu.json files, and it prefers them over the equivalent .tf/.tf.json when both exist. That gives you a clean way to keep a shared codebase that runs under both tools: keep the common config in .tf, and drop OpenTofu-specific overrides in .tofu files that Terraform simply ignores.

main.tf          # shared, runs under both
backend.tofu     # OpenTofu-only overrides, invisible to terraform

If you're maintaining a module that needs to support both tools during a transition, this is the mechanism that keeps you sane.

The registry

OpenTofu runs its own provider and module registry rather than depending on HashiCorp's. In practice most of the popular providers are mirrored and resolve fine, but the source of truth is different, and provider/module availability is something to actually verify rather than assume. If you have a niche or internal provider, confirm it publishes where OpenTofu looks for it.

A note on version numbers

I'm deliberately not quoting exact version numbers or "OpenTofu is X% faster" benchmarks, because those age badly and half the ones you'll see online are made up. As of 2026 both projects are shipping regularly and the feature sets keep moving — treat any specific version claim (including mine) as something to verify against the current docs before you build a decision on it.

So which do you pick, and should you switch?

Here's my honest take.

If you're starting green today, I'd default to OpenTofu. You get a real open-source license, native state/plan encryption, and the divergent features are mostly additive quality-of-life wins. The compatibility story means you lose almost nothing by choosing it.

If you're an existing Terraform shop, the calculus is about your actual pain. Switch if the BSL license is a genuine legal or procurement problem, or if native state encryption solves a compliance requirement you're currently duct-taping. Don't switch just to be on the trendy side of a fork — a migration is still real work and real risk, even when it's low-risk.

If you depend on Terraform Cloud / HCP-specific features or a paid workflow tightly coupled to HashiCorp's platform, weigh that integration honestly. OpenTofu is the engine, not the whole platform, and you'll be assembling backend, state, and workflow pieces yourself or via third-party platforms.

Whatever you choose, the one thing I'd avoid is drifting into using divergent features by accident and then being surprised you can't go back. If you want a cross-referenced dive into the specific errors and edge cases I hit while running OpenTofu on real clusters, I keep my OpenTofu troubleshooting guides updated as I trip over new ones.

Takeaway

OpenTofu and Terraform are close cousins, not twins. The compatibility is real enough that adoption is cheap, but the divergences — state/plan encryption, early evaluation, provider-defined functions, .tofu overrides, and a separate registry — are real enough that you should choose deliberately and know which one-way doors you're walking through. Pick based on your license posture and your compliance needs, pin your versions, and read the current docs before betting on any specific feature.

DEV Community: James Joyner

DNS Troubleshooting with dig: The Commands DevOps Engineers Actually Need

Think about it as a resolution chain, not "is DNS broken"

The record types worth knowing

The commands that actually earn their place

Real scenarios, and how to read them

The takeaway

tcpdump for DevOps: Read a TCP Handshake and Stop Guessing

Capture just enough, and nothing else

Signature 1: a healthy connection

Signature 2: connection refused

Signature 3: timeout / silence

Don't forget DNS

Capturing inside a container

The habit

Network Troubleshooting as a Stack: Find Which Layer Is Broken First

The network is a stack, so treat it like one

Walk it: "HTTPS to api.example.com times out"

1. Interface — do we have a link and an address?

2. Route — is there a path to the destination?

3. Gateway — can we reach the first hop?

4. DNS — does the name resolve, and to the right thing?

5. Port — is the service actually listening/reachable?

6. TLS — does the handshake complete and the cert validate?

7. Application — now, finally, the app

Why this beats guessing

Try it next time

Nmap for Authorized Infrastructure Validation (Not Hacking)

Start with what's actually listening

Confirm what's really on the port

A methodology, not just commands

A concrete finding

A note on scan types and privileges

Wrap up

Keep Docker Engine as Your Kubernetes Runtime on Ubuntu with cri-dockerd

Should you? A quick reality check

1. Install Docker Engine on Ubuntu

2. Install cri-dockerd

3. Point kubeadm at the cri-dockerd socket

4. Verify

The gotcha nobody warns you about

Set Up a Kubernetes Node on Ubuntu 24.04 with containerd (the Docker-Compatible Runtime)

1. Kernel prerequisites

2. Install containerd

3. Install kubeadm, kubelet, kubectl

4. Initialize the control plane

5. Install a CNI, then verify

6. Prove Docker images run

When it doesn't come up

Run Kubernetes in Docker on Ubuntu for Local Development

Prerequisites on Ubuntu

A one-command cluster

A realistic multi-node cluster

Loading a locally-built image (no registry needed)

Ubuntu-specific gotchas

minikube with the Docker Driver on Ubuntu: A Practical Local Cluster

Install on Ubuntu

Start with the Docker driver

The addons are the reason to pick minikube

The Docker image workflow

Accessing services from Ubuntu

minikube vs kind — how I choose

Cleanup

From Docker Build to Kubernetes Deploy on Ubuntu: The Image Workflow That Never Changed

1. A build-friendly Dockerfile

2. Build and push with Docker on Ubuntu

3. A deployment that behaves in production

4. Pulling from a private registry

5. Ship a new build

The mental model

State Encryption in OpenTofu: How It Works and How to Roll It Out

Why plaintext state is a real risk

Anatomy of the encryption block

Key providers: passphrase vs KMS

Encrypt the plan file too

Rolling it out on an existing project with fallback

Lock it down with enforced

Key rotation

CI and secrets handling

Takeaway

Anatomy of the `encryption` block

Rolling it out on an existing project with `fallback`

Lock it down with `enforced`

Step 2: Install `tofu` alongside, not instead

`.tofu` and `.tofu.json` override files