Making kubectl diff show what actually changed

kubectl diff is the closest thing Kubernetes has to a plan step: it shows what kubectl apply would change before you apply it. Its output is a unified diff of two YAML dumps, and that is where it stops being useful once a manifest is longer than a screen.

Here is a small test. A Deployment with two containers, api and sidecar, runs in a kind cluster. The new version of the manifest makes two real changes, replicas from 3 to 5 and the api image from nginx:1.27 to nginx:1.28. It also lists the two containers in the opposite order and swaps the two env vars. To the cluster the reordering means nothing. To kubectl diff it means this:

@@ -29,17 +29,6 @@
         app: api
     spec:
       containers:
-      - env:
-        - name: LOG_LEVEL
-          value: info
-        - name: REGION
-          value: eu
-        image: nginx:1.27
-        imagePullPolicy: IfNotPresent
-        name: api
-        resources: {}
-        terminationMessagePath: /dev/termination-log
-        terminationMessagePolicy: File
       - command:
         - sleep
         - infinity
@@ -49,6 +38,17 @@
         resources: {}
         terminationMessagePath: /dev/termination-log
         terminationMessagePolicy: File
+      - env:
+        - name: REGION
+          value: eu
+        - name: LOG_LEVEL
+          value: info
+        image: nginx:1.28
+        imagePullPolicy: IfNotPresent
+        name: api
+        resources: {}
+        terminationMessagePath: /dev/termination-log
+        terminationMessagePolicy: File

That is 26 changed lines for the Deployment, the replicas hunk not shown. The image bump is in there: one line in a removed block and one in an added block, which you have to compare by eye.

How kubectl diff works

kubectl fetches the live object, asks the API server for the object as it would look after the apply (a dry-run request), strips managedFields from both, and writes them as YAML into two temporary directories, LIVE-… and MERGED-…. Each object becomes one file named group.version.Kind.namespace.name, for example apps.v1.Deployment.default.api. Then it runs diff -u -N on the two directories, or the program named in KUBECTL_EXTERNAL_DIFF. The exit code follows diff's convention: 0 for no differences, 1 for differences, above 1 for an error.

So any tool that can compare two directories can take over. I maintain datadiff, a CLI that compares JSON, YAML and other structured files by their parsed data instead of their lines, and since 0.5.0 it compares directories too:

export KUBECTL_EXTERNAL_DIFF="datadiff --key name"
kubectl diff -f deploy.yaml

Same cluster, same manifest:

apps.v1.Deployment.default.api
~ metadata.generation: 1 → 2
~ spec.replicas: 3 → 5
~ spec.template.spec.containers[name=api].image: "nginx:1.27" → "nginx:1.28"
3 changes (0 added, 0 removed, 3 modified)
v1.ConfigMap.default.api-config
+ apiVersion: "v1"
+ data: {"feature_flags":"beta"}
+ kind: "ConfigMap"
+ metadata: {"creationTimestamp":"2026-10-07T18:27:38Z","name":"api-config","namespace":"default","uid":"11c4ba1a-b34f-4fc5-98b1-b20d769f536b"}
4 changes (4 added, 0 removed, 0 modified)

--key name tells datadiff to match list items by their name field rather than their position, so the swapped containers and env vars are not a change. Without it the Deployment comes out as ten changes of the containers[0].name: "api" → "sidecar" kind. The ConfigMap is new: for an object that does not exist yet, kubectl writes an empty file into the LIVE directory, and an empty file reads as "nothing here", so every entry shows up as added.

kubectl drops some of your arguments

The next thing I tried was a policy: make the exit code 1 only when the replica count changes.

export KUBECTL_EXTERNAL_DIFF="datadiff --fail-on spec.replicas"
kubectl diff -f deploy.yaml

error: a value is required for '--fail-on <FAIL_ON>' but none was supplied
error: failed to run "/home/runner/work/datadiff/datadiff/target/release/datadiff": exit status 2

The value never arrived. The reason is a few lines in kubectl's diff.go:

diffCommand := strings.Split(envDiff, " ")
diff = diffCommand[0]

if len(diffCommand) > 1 {
    // Regex accepts: Alphanumeric (case-insensitive), dash and equal
    isValidChar := regexp.MustCompile(`^[a-zA-Z0-9-=]+$`).MatchString
    for i := 1; i < len(diffCommand); i++ {
        if isValidChar(diffCommand[i]) {
            args = append(args, diffCommand[i])
        }
    }
}

Every word after the program name has to consist of letters, digits, - and =, or it is dropped without a warning. --key name gets through; spec.replicas has a dot and does not. Quoting does not help either, since the string is split on single spaces with no shell parsing, and a quote character fails the same check.

The environment, on the other hand, reaches the child process untouched. datadiff reads each of its options from a DATADIFF_* variable as well, so path options go there:

export KUBECTL_EXTERNAL_DIFF="datadiff --key name"
export DATADIFF_FAIL_ON='spec.replicas,*.image'
kubectl diff -f deploy.yaml   # exits 1 only if replicas or an image changed

On the same manifest, DATADIFF_FAIL_ON=spec.replicas exits 1 and DATADIFF_FAIL_ON=spec.template.spec.volumes exits 0: the changes still print, but none of them is one you asked to gate on. That makes kubectl diff usable as a drift check in CI against a live cluster.

The noise that is left

Two fields still change without anyone editing them. metadata.generation goes up with every spec change, so it is in every real diff. And kubectl diff --server-side on a Deployment that had been created with client-side apply reported two changes for a manifest that had not changed at all: generation 1 → 2 and a rewritten kubectl.kubernetes.io/last-applied-configuration annotation, the whole previous manifest as one JSON string. Exit code 1, nothing to look at.

0.5.0 adds --ignore for this. It takes the same patterns as --fail-on: a path, a dot-separated prefix of one, or * globs. Ignored changes are not printed, not counted and not checked:

export DATADIFF_IGNORE='metadata.generation,metadata.annotations.kubectl.kubernetes.io/last-applied-configuration'

With that, the unchanged manifest exits 0 in both client-side and server-side mode, and the real diff is down to the replica count and the image.

Try it

brew install dimanovikov/datadiff/datadiff   # or: cargo install datadiff
export KUBECTL_EXTERNAL_DIFF="datadiff --key name"
kubectl diff -f your-manifest.yaml

With Nix: nix run github:dimanovikov/datadiff -- --help. If you already use dyff, it documents the same KUBECTL_EXTERNAL_DIFF setup, and the argument filter applies to it just the same.

All the output above comes from a kind cluster in GitHub Actions, run in both client-side and server-side mode. The README has the setup.

I wrote this post with the help of an AI assistant. The commands and outputs are from that run.