DEV Community

Mohammed Tayeh
Mohammed Tayeh

Posted on AI-assisted

Kubernetes Secrets That Sync Themselves — the SnapEnv Operator, Auto-Reload, and Helm

#kubernetes #devops #linux #security

Part 3 of 3. Part 1 covers the dashboard and CLI; part 2 covers a bare Linux server with systemd. This one's for Kubernetes.

If your deployment story today is "someone runs kubectl create secret by hand, or a value gets base64-encoded into a values file that shouldn't really be in git" — the SnapEnv Kubernetes operator replaces that with a SnapEnvSecret custom resource that keeps a native Secret in sync automatically, and can roll the Deployments that use it the moment something changes.

What we're building

  • The operator installed cluster-wide, watching for SnapEnvSecret resources in any namespace
  • One SnapEnvSecret synced into a native Secret, wired into a Deployment via envFrom
  • That Deployment auto-restarting when the underlying variables change — no manual rollout
  • The same pattern generalized into a drop-in block for your own Helm chart

1. Install the operator

The operator is a single, small (32Mi request) controller — one manifest installs the CRD, RBAC, and the Deployment itself:

kubectl apply -f https://raw.githubusercontent.com/snapenv-io/operator/main/config/deploy/install.yaml
Enter fullscreen mode Exit fullscreen mode

That creates a dedicated snapenv-operator namespace, a ClusterRole scoped to exactly what it needs (read SnapEnvSecret CRs, read/write Secrets, read/patch Deployments and StatefulSets for the restart feature, write Events), and a one-replica Deployment running ghcr.io/snapenv-io/operator:latest.

$ kubectl -n snapenv-operator get pods
NAME                                READY   STATUS    RESTARTS   AGE
snapenv-operator-669d6b9d54-gwcql   1/1     Running   0          16m
Enter fullscreen mode Exit fullscreen mode

2. The SnapEnvSecret custom resource 

apiVersion: snapenv.io/v1alpha1
kind: SnapEnvSecret
metadata:
  name: my-web-app-dev
  namespace: my-namespace
spec:
  tokenSecret: snapenv-token      # name of a Secret in this namespace holding the snp_live_ token
  project: ad12d532-0779-49fa-8045-eed069db8597
  env: dev
  target: my-web-app-secret       # the native Secret the operator will create/keep in sync
  syncInterval: 1m                # optional — default 30m, minimum 1m
Enter fullscreen mode Exit fullscreen mode
Field Required Notes
tokenSecret yes Name of a Secret in the same namespace, containing the token under a token key (override with tokenKey)
project yes The SnapEnv project UUID
env yes Environment name (prod, staging, dev, …)
target yes The native Secret name the operator manages
apiUrl no Override the API base URL — must be https://, enforced by the CRD schema itself
syncInterval no Poll interval absent a webhook push. Default 30m, clamps to a 1m minimum
allowEmpty no Off by default — a sync that returns zero variables won't overwrite a target that previously had data, since an empty response is far more likely to be a permissions/deploy mistake upstream than an intentional wipe

The token only ever needs read scope, pinned to the one project and environment it's syncing — exactly the same least-privilege instinct as the systemd guide's token.

Applying it against a real (disposable, demo) project:

$ kubectl create secret generic snapenv-token \
    --from-file=token=./token.txt -n my-namespace

$ kubectl apply -f snapenvsecret.yaml
snapenvsecret.snapenv.io/my-web-app-dev created

$ kubectl get snapenvsecret my-web-app-dev
NAME             PROJECT                                ENV   TARGET              READY   VARS   LAST SYNC
my-web-app-dev   ad12d532-0779-49fa-8045-eed069db8597   dev   my-web-app-secret   true    8       9s

$ kubectl describe secret my-web-app-secret
Name:         my-web-app-secret
Labels:       app.kubernetes.io/managed-by=snapenv-operator
              snapenv.io/env=dev
              snapenv.io/project=ad12d532-0779-49fa-8045-eed069db8597
Annotations:  snapenv.io/data-hash: W/"d2fb7d8f49100323"
              snapenv.io/last-sync-time: 2026-09-15T12:22:27Z
Type:  Opaque
Data
====
API_KEY:               28 bytes
DATABASE_URL:          39 bytes
LOG_LEVEL:             5 bytes
PORT:                  4 bytes
REDIS_URL:             24 bytes
SNAPENV_ENV:           3 bytes
SNAPENV_PROJECT:       36 bytes
SNAPENV_PROJECT_NAME:  10 bytes
Enter fullscreen mode Exit fullscreen mode

Same three synthetic SNAPENV_* context variables the CLI injects on every pull show up here too — useful for a pod to introspect which project/env it's actually running against without you wiring that up separately.

3. Wire it into a Deployment 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-web-app
  namespace: my-namespace
spec:
  replicas: 2
  selector:
    matchLabels: { app: my-web-app }
  template:
    metadata:
      labels: { app: my-web-app }
      annotations:
        snapenv.io/sync-secret: my-web-app-secret   # see step 4
    spec:
      containers:
        - name: my-web-app
          image: ghcr.io/you/my-web-app:latest
          envFrom:
            - secretRef:
                name: my-web-app-secret
Enter fullscreen mode Exit fullscreen mode

Nothing SnapEnv-specific in the container itself — it's a normal envFrom.secretRef, exactly like referencing any other Kubernetes Secret.

4. Auto-reload when secrets change

The snapenv.io/sync-secret: "<target-secret-name>" annotation above is what makes this self-driving. It can sit on the Deployment's own metadata or on its pod template (the operator checks both, since some Helm conventions put annotations in one place or the other) — when the operator resyncs and the secret's content actually changed, it patches kubectl.kubernetes.io/restartedAt onto the pod template of every Deployment/StatefulSet carrying that annotation. That's the same mechanism kubectl rollout restart uses — a normal, native rolling restart, not a hard delete.

Critically, it only restarts on an actual content change — the operator stores the API's ETag on the target Secret (snapenv.io/data-hash) and compares it before writing anything, so a poll that finds nothing new is a no-op (no restart, and if the API returns 304 Not Modified, not even a Secret write). That distinction matters: an earlier version of this exact operator had a bug where a broken ETag comparison made every poll look like a change, restart-looping every linked Deployment roughly once a minute. Diffing before you act — not "sync on every poll" — is the difference between a self-healing system and a self-DDoSing one.

5. Verify from the dashboard

Every sync the operator performs sends X-SnapEnv-Client: operator, which the dashboard's Integrations page uses to show a live "last synced" time per project — a quick way to confirm the operator is actually running against a given project without reaching for kubectl at all.

6. The Helm chart pattern

If you maintain your own chart, the pattern that works well is a snapenv block in values.yaml, mutually exclusive with however you currently inject secrets (a plain base64 Secret, Doppler, whatever):

# values.yaml
snapenv:
  enabled: false
  tokenSecret:
    create: true
    name: ""              # defaults to <app-name>-snapenv-token
    value: ""              # snp_live_... — pass via --set-string or a values overlay, never committed
  managedSecret:
    name: ""              # defaults to <app-name>-secret
  project: ""              # required when enabled
  env: prod
  apiUrl: ""
  syncInterval: 30m
Enter fullscreen mode Exit fullscreen mode 
# templates/snapenvsecret.yaml
{{- if .Values.snapenv.enabled }}
{{- if .Values.snapenv.tokenSecret.create }}
apiVersion: v1
kind: Secret
metadata:
  name: {{ .Values.snapenv.tokenSecret.name | default (printf "%s-snapenv-token" .Values.name) }}
type: Opaque
stringData:
  token: {{ .Values.snapenv.tokenSecret.value | quote }}
---
{{- end }}
apiVersion: snapenv.io/v1alpha1
kind: SnapEnvSecret
metadata:
  name: {{ .Values.name }}-snapenv
spec:
  tokenSecret: {{ .Values.snapenv.tokenSecret.name | default (printf "%s-snapenv-token" .Values.name) }}
  project: {{ .Values.snapenv.project | quote }}
  env: {{ .Values.snapenv.env | quote }}
  target: {{ .Values.snapenv.managedSecret.name | default (printf "%s-secret" .Values.name) }}
  {{- if .Values.snapenv.syncInterval }}
  syncInterval: {{ .Values.snapenv.syncInterval | quote }}
  {{- end }}
{{- end }}
Enter fullscreen mode Exit fullscreen mode

Two helpers make this play nicely alongside whatever secret mode the chart already supports:

{{/* Fail the render early if more than one secret mode is enabled at once */}}
{{- define "mychart.validateSecrets" -}}
{{- $enabled := list }}
{{- if .Values.secret.enabled }}{{- $enabled = append $enabled "secret" }}{{- end }}
{{- if .Values.snapenv.enabled }}{{- $enabled = append $enabled "snapenv" }}{{- end }}
{{- if gt (len $enabled) 1 }}
{{- fail (printf "Enable exactly one secret mode, not: %s" (join ", " $enabled)) }}
{{- end }}
{{- end }}

{{/* One name to envFrom.secretRef against, regardless of which mode is active */}}
{{- define "mychart.secretName" -}}
{{- if .Values.snapenv.enabled }}
{{- .Values.snapenv.managedSecret.name | default (printf "%s-secret" .Values.name) }}
{{- else }}
{{- printf "%s-secret" .Values.name }}
{{- end }}
{{- end }}
Enter fullscreen mode Exit fullscreen mode

Then every container's envFrom just references {{ include "mychart.secretName" . }} and doesn't care which mode produced it — flipping snapenv.enabled: true in an environment's values file is the entire migration.

Troubleshooting

kubectl logs on the operator is flooded with deployments.apps is forbidden. The ClusterRole is missing the apps apiGroup grant on deployments/statefulsets — required for the auto-restart feature (step 4), not for syncing itself, so secrets will keep updating fine while this silently breaks restarts. This is easy to end up with after a redeploy: if your pipeline only updates the operator's image tag and never re-applies its RBAC manifest, a ClusterRole edit that's sitting correctly in your repo can quietly drift out of sync with what's actually applied to the cluster. Worth an explicit check after any operator upgrade:

kubectl apply -f config/deploy/install.yaml   # idempotent — re-applies CRD + RBAC + Deployment
kubectl -n snapenv-operator logs deploy/snapenv-operator --tail=20 | grep -i forbidden
Enter fullscreen mode Exit fullscreen mode

A SnapEnvSecret shows READY: false. kubectl describe snapenvsecret <name> — the Last Sync Error field in status has the real reason, almost always a token/project/env mismatch or an expired/revoked token.

A new status field you added to the CRD never shows up. Kubernetes' apiserver prunes any field not declared in the CRD's OpenAPI schema — silently, no error. If you're extending this operator yourself, any new field goes in the Go type and config/crd/snapenv.io_snapenvsecrets.yaml's status.properties, or it vanishes on every write.

Restarts happen on every sync, even with nothing changed. That means the ETag comparison is broken somewhere in the chain — check whether anything between the operator and the API (an ingress, a proxy) is rewriting the ETag/If-None-Match headers; a reverse proxy's gzip layer downgrading a strong ETag to a weak W/"..." one is a common culprit, and the comparison needs to tolerate that (RFC 9110 weak comparison), not do a byte-for-byte string match.

Wrap-up

That's all three parts of the series: dashboard + CLI for how your team works day to day, systemd for a plain VPS, and the operator here for Kubernetes — three different delivery mechanisms, one source of truth underneath all of them. Full reference docs at docs.snapenv.io.

🎉 Code HELLOSNAP gets you the Pro plan free for 3 months, first 100 redemptions — redeem it from Workspace → Plan & Billing at snapenv.io.

Top comments (0)