Fix Flux SOPS MAC mismatch in kustomize-controller

A SOPS MAC mismatch in Flux almost always means one thing: the encrypted file was changed outside of sops. SOPS signs every file with a message authentication code computed over the plaintext at encrypt time. When kustomize-controller decrypts the file and recomputes that code, it no longer matches, so Flux refuses the Secret and stops reconciling. You cannot patch the ciphertext by hand to fix this. The reliable fix is to recover the real values and re-encrypt the file from scratch, which is what the rest of this page walks through.

What the error looks like

The failure shows up on the Kustomization, not the Secret. Check the object status and the controller logs:

$ flux get kustomizations --all-namespaces
$ kubectl -n flux-system logs deploy/kustomize-controller | grep -i "mac mismatch"

You will see a line similar to this:

Kustomization/flux-system/apps: Reconciliation failed after 1.2s:
failed to decrypt secret 'db-credentials': Error getting data key:
Error decrypting tree: MAC mismatch. Expected '9f8c...', got 'a71b...'

The two hashes are the point. "Expected" is the MAC that sops stored when the file was last encrypted correctly. "Got" is the MAC recomputed from the plaintext it just decrypted. They differ, so the content changed since the last clean encryption.

Why it happens

Four situations produce a MAC mismatch in practice.

Cause	Signal	Fix path
File hand-edited outside sops	Recent commit touched the .enc.yaml directly	Recover plaintext, re-encrypt
Git merge conflict resolved by hand	Merge commit on the encrypted file	Re-merge from plaintext
encrypted_regex / mac_only_encrypted changed	.sops.yaml edited since last encrypt	Align rules, re-encrypt
Copy-paste truncation or corruption	Value looks short or malformed	Restore from git, re-encrypt

The first row covers most incidents. Opening an encrypted YAML in a text editor and changing a value, a key name, or even reindenting it alters the plaintext that sops will recompute the MAC over. The same goes for resolving a Git merge conflict by editing the encrypted file directly: you end up with a document sops never signed.

The mac_only_encrypted case is subtler. When your .sops.yaml sets mac_only_encrypted: true, only the encrypted values feed the MAC. Flip that flag, or change which fields encrypted_regex selects, and the recomputed MAC covers a different set of values than the stored one, even though nothing looks wrong in the diff.

Fix it step by step

Step 1: Confirm the file is the problem, not the key

Decrypt the file locally with the same age key Flux uses. If you get a MAC mismatch here too, the file is corrupt and the cluster key secret is fine:

$ export SOPS_AGE_KEY_FILE=$HOME/.config/sops/age/keys.txt
$ sops --decrypt clusters/prod/db-credentials.enc.yaml

If instead you see no key could decrypt or a base64 error, that is a different failure. The related Flux SOPS illegal base64 error fix covers the key-encoding case.

Step 2: Recover the real plaintext

You need a trustworthy copy of the values. Pick whichever source you actually trust.

Restore the last known-good version from Git history:

$ git log --oneline -- clusters/prod/db-credentials.enc.yaml
$ git show <good-sha>:clusters/prod/db-credentials.enc.yaml > /tmp/recovered.enc.yaml
$ sops --decrypt /tmp/recovered.enc.yaml > /tmp/plain.yaml

If you trust the current values but only the MAC is stale (for example the file was reindented, not semantically changed), decrypt while skipping MAC verification:

$ sops --decrypt --ignore-mac clusters/prod/db-credentials.enc.yaml > /tmp/plain.yaml

One caveat: --ignore-mac does not work with --in-place, and it will not rescue a file whose ciphertext was actually modified. Use it only to pull known-good plaintext back out so you can re-encrypt it.

As a last resort, if the Secret already applied to the cluster at least once, read the live values back:

$ kubectl -n app get secret db-credentials -o jsonpath='{.data.password}' | base64 -d

Step 3: Re-encrypt from clean plaintext

Encrypt the recovered plaintext into a fresh file with a valid MAC:

$ sops --encrypt --age age1yourclusterpublickey /tmp/plain.yaml > clusters/prod/db-credentials.enc.yaml
$ rm /tmp/plain.yaml

If you keep a .sops.yaml with creation_rules (recommended, so everyone encrypts identically), sops picks up the key and regex automatically:

$ sops --encrypt /tmp/plain.yaml > clusters/prod/db-credentials.enc.yaml

Step 4: Verify before you push

Never push an encrypted file you have not decrypted at least once. This one command is the difference between a clean reconcile and another red Kustomization:

$ sops --decrypt clusters/prod/db-credentials.enc.yaml | head -5

If that returns plaintext with no MAC error, the file is good.

Step 5: Commit and reconcile

$ git add clusters/prod/db-credentials.enc.yaml
$ git commit -m "fix: re-encrypt db-credentials to repair sops mac"
$ git push
$ flux reconcile kustomization apps --with-source

Watch the Kustomization go Ready again with flux get kustomizations.

Can you tell Flux to ignore the MAC?

No. The kustomize-controller always verifies the MAC during decryption and exposes no ignore-mac option, by design: a Secret that fails integrity checks should not be applied to a cluster. Tested with Flux v2.4.0 and sops v3.9.4, there is no spec.decryption field that disables MAC verification. The only durable fix is a file that decrypts cleanly, so treat the mismatch as a signal to re-encrypt rather than something to bypass.

Prevention

• Never open a .enc.yaml in a plain text editor. Run sops clusters/prod/db-credentials.enc.yaml, which decrypts into your editor and recomputes the MAC on save.
• Add a CI check or pre-commit hook that runs sops --decrypt on every changed encrypted file, so a broken MAC fails the pull request instead of the cluster.
• Commit a .sops.yaml with creation_rules so encrypted_regex and key groups are identical for everyone. Inconsistent rules are a quiet source of MAC drift.
• Resolve merge conflicts on encrypted files by decrypting both sides, merging the plaintext, and re-encrypting. Do not hand-edit ciphertext to settle a conflict.

For the canonical setup, the official Flux SOPS guide documents the age and key-secret wiring end to end. If you are still deciding how to structure encrypted secrets across environments, the Argo CD vs Flux multi-cluster GitOps guide and the Argo CD GitOps setup tutorial walk through the surrounding reconciliation model that this Secret plugs into.