DEV Community: Guatu

Prometheus Alerting Rules That Don't Cry Wolf

Guatu — Mon, 29 Jun 2026 16:15:49 +0000

The fastest way to get a team to ignore Prometheus is to point an alert at up == 0 and ship it. Week one it catches a real outage and everyone's impressed. By week three it has fired a couple dozen times for scrape blips that healed themselves before anyone looked, and people have quietly built a mental filter that drops every page from that channel. The alert still fires. Nobody reads it.

That's the failure mode I want to talk about: alerts that are technically correct and operationally useless. An alert you've trained yourself to ignore is worse than no alert, because it gives you the feeling of coverage without the substance.

If you're running kube-prometheus-stack or plain Prometheus and you've reached the point where your alert channel is mostly noise, this is for you. If you've ever gotten a page that said "GPU temperature high" followed by a 36-character UUID and zero indication of which physical card was cooking, you're exactly the audience. The fix isn't fewer alerts. It's alerts that know what they're talking about.

What I reached for first (and why it didn't hold up)

The instinct when you stand up monitoring is to alert on everything you can see. Scrape is failing? Page. CPU over 80%? Page. Memory climbing? Page. Disk filling? Page. It feels responsible. You're "covering your bases."

What you actually build is a slot machine that pays out anxiety. Two specific mistakes show up over and over, and I've made both.

The first is alerting without a for: clause. A bare expression like up == 0 evaluates on every rule interval, and the instant a single scrape fails the alert goes active. Scrapes fail for boring reasons all the time: a pod gets rescheduled, a node has a half-second of network weather, an exporter restarts during a Helm upgrade. None of those are incidents. All of them page you.

The second mistake is alerting on raw hardware identifiers with no human context. Most low-level exporters emit metrics keyed by opaque IDs. nvidia_smi_temperature_gpu carries a uuid label and nothing friendly. So your alert fires, you read it half-asleep, and now you're SSHing into nodes running nvidia-smi -L to figure out which UUID maps to which card in which chassis. By the time you've found it, the alert has either resolved or escalated. Either way you've spent ten minutes doing a join in your head that Prometheus should have done for you.

Both mistakes share a root cause. The alert describes a metric, not a situation. "up is zero" and "the GPU is at 89 degrees" are facts. "The monitoring for Node A's backup exporter has been unreachable for ten minutes, and that node is the only one tagged backup-critical" is a situation. Situations are actionable. Facts are homework.

The fixes, in order of how much noise they kill

1. Make every alert survive five minutes before it fires

This is the single highest-use change and it costs one line. The for: clause tells Prometheus to hold an alert in the pending state until the condition has been continuously true for that duration. Only then does it transition to firing and hit your notification path.

groups:
  - name: targets.rules
    rules:
      - alert: ExporterDown
        expr: up{job=~"pve-exporter|unpoller|nvidia-gpu-exporter"} == 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Exporter {{ $labels.job }} is unreachable"
          description: "{{ $labels.job }} on {{ $labels.instance }} has had failed scrapes for 10m+."

A scrape blip that recovers in 30 seconds never leaves pending. You never hear about it. A genuine outage that persists past ten minutes pages you, with the job and instance already in the message. The number isn't sacred: use for: 2m for things that genuinely matter fast, for: 15m for slow-burn conditions like disk fill. The point is that almost nothing on your infrastructure justifies paging on a single failed evaluation.

If you take only one thing from this post, take this. Most "Prometheus is too noisy" complaints are really "we forgot the for: clause" complaints.

2. Join hardware IDs to human names inside the alert expression

PromQL can do the UUID-to-name lookup for you at evaluation time, so the name lands in the alert annotation instead of in your debugging session. The trick is a vector match with group_left, multiplying your metric by an _info series whose value is always 1.

nvidia_smi_temperature_gpu
  * on(uuid) group_left(name) nvidia_smi_gpu_info
  > 85

nvidia_smi_gpu_info is a metric whose only job is to carry labels: it's 1 for every GPU, tagged with uuid and name. Multiplying your temperature reading by 1 doesn't change the value, but group_left(name) copies the name label from the info series onto the result. Now the alert knows it's a "Tesla P40" and not just a hex string.

Wired into a rule:

      - alert: GPUTemperatureHigh
        expr: |
          nvidia_smi_temperature_gpu
            * on(uuid) group_left(name) nvidia_smi_gpu_info
            > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "GPU {{ $labels.name }} is running hot"
          description: "{{ $labels.name }} ({{ $labels.uuid }}) has been above 85°C for 5m+."

The page now reads "GPU Tesla P40 is running hot" with the temperature and the UUID still available for the deep dive. That one annotation turns a cry-wolf alert into one you can act on without leaving the notification. This same pattern works for any exporter that splits readings from metadata: node info series, disk SMART data, network interface names. If you're already wrestling with the realities of monitoring passthrough hardware, the GPU passthrough field guide covers why those IDs are so unstable in the first place.

When you put these joined values on a dashboard, the same context discipline applies; I wrote about not drowning the signal in chrome in Grafana dashboards: information density vs readability.

3. Suppress alerts for failures you already expect

Not every failure is a problem. Velero is a good example: snapshot a Longhorn volume and it works, snapshot an NFS-backed volume and the CSI snapshot fails because the driver doesn't support it. That second failure is expected. It will happen on every backup run, forever, and alerting on it teaches people that backup alerts are noise, which is precisely the day your actual backup breaks and nobody notices.

The clean fix is to scope the alert to the targets you actually care about using a label match instead of trying to enumerate exceptions.

      - alert: CriticalVolumeSnapshotFailed
        expr: |
          increase(velero_volume_snapshot_failure_total[1h]) > 0
          and on(namespace, name)
            kube_persistentvolumeclaim_info{storageclass="longhorn"}
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "Snapshot failed for {{ $labels.namespace }}/{{ $labels.name }}"
          description: "A backup-critical Longhorn volume failed to snapshot for 30m+."

The and on(namespace, name) ...{storageclass="longhorn"} clause filters the failures down to volumes whose storage class is one you expect snapshots to work on. NFS volumes drop out of the result set entirely, so they can't fire. You've encoded "I know NFS snapshots fail and I don't care" directly into the rule, which is far better than a Slack message someone wrote in 2024 saying "just ignore the NFS ones."

You can push the same idea further with explicit labels. Relabel your backup targets to carry backup_critical="true" and only alert on those, so a dev-scratch volume failing to back up never wakes anyone. Storage alerts are their own rabbit hole, and the gap between "Longhorn says healthy" and "actually recoverable" is worth understanding before you decide what to alert on; that's Longhorn volume health in detail.

4. Separate "the service is down" from "the monitoring is broken"

This is the distinction that fixes the worst class of false positive. The up metric only tells you whether Prometheus could scrape the exporter. It says nothing about whether the exporter could reach the thing it's monitoring.

pve-exporter makes this concrete. It scrapes the Proxmox API on your behalf and returns a pve_up metric per target. If the exporter's API token expires or loses permissions, the exporter itself is healthy and scrapeable (up == 1), but it returns pve_up == 0 because it can't authenticate upstream. Alert on up alone and you'll either miss the auth failure entirely or, worse, misread it as the node being down and start a recovery you don't need.

Split it into two alerts that say two different things:

      - alert: PVEExporterUnreachable
        expr: up{job="pve-exporter"} == 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Prometheus cannot reach pve-exporter"
          description: "The monitoring path is broken, not necessarily the node."

      - alert: PVEAPIAuthFailing
        expr: pve_up == 0 and on(instance) up{job="pve-exporter"} == 1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "pve-exporter is up but cannot talk to the Proxmox API"
          description: "Token expired or lost permissions. Check the exporter credentials."

The first alert means "I'm blind." The second means "my eyes work but the door is locked." Those need different responses, and naming them differently is what stops you from running the wrong runbook at the wrong hour. Designing alerts around the credential boundary like this is the same instinct behind keeping monitoring identities scoped and rotatable, which is part of why I treat alerting trust as a consulting problem for production systems, not an afterthought.

One trap to avoid here: don't write a thermal alert against pve-exporter. As of v3.5.5 it doesn't expose temperature metrics at all, so a "node too hot" rule against that job produces a no data state forever, which Grafana will happily render as a permanently confusing yellow tile. Alert on what the exporter actually emits, not what you wish it did.

5. Make sure your alert rules are even being scraped

This is the silent killer, and it's specific to the Prometheus Operator world. You write a perfect ServiceMonitor, your exporter is running, and no data ever appears. No alert ever fires. Everything looks deployed and nothing works.

The cause is almost always a label mismatch. The Prometheus custom resource has a serviceMonitorSelector, and kube-prometheus-stack defaults it to match release: <your-helm-release-name>. A ServiceMonitor without that label is invisible to Prometheus. It exists, kubectl get servicemonitor shows it, and Prometheus ignores it completely.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: pve-exporter
  labels:
    release: monitoring   # MUST match the Prometheus serviceMonitorSelector
spec:
  selector:
    matchLabels:
      app: pve-exporter
  endpoints:
    - port: metrics
      interval: 60s

Check your Prometheus CR with kubectl get prometheus -o yaml and look at spec.serviceMonitorSelector. Whatever it matches, your ServiceMonitor labels have to satisfy it. The same applies to PrometheusRule objects via ruleSelector: get the label wrong and your beautifully written alert rules are never loaded. Verify they made it by opening the Prometheus UI under Status → Rules. If your group isn't listed, the selector dropped it.

While you're in there, one format gotcha. If you copied alert syntax from an old StackOverflow answer that uses ALERT FooBar IF expr FOR 5m, that's the Prometheus 1.x DSL and it will not load in 2.x or 3.x. The current format is the YAML groups: structure shown throughout this post. Mixing the two is a common reason rules silently fail validation.

Why these changes actually move the needle

Every one of these fixes pushes alerts in the same direction: from describing a metric to describing a situation a human can act on.

The for: clause works because it converts instantaneous facts into sustained conditions. Infrastructure is noisy at the millisecond scale and calm at the multi-minute scale. By requiring a condition to persist, you're matching the alert's time horizon to the time horizon a human actually responds on. Nobody pages on a 200-millisecond blip because nobody can act on one.

The PromQL joins work because the cost of context should be paid once, at rule-evaluation time, not every time a tired engineer reads the page. Prometheus already has the mapping from uuid to name sitting in an info metric. Doing the join in the rule means the expensive lookup happens automatically and the result is baked into the annotation. You move work from the human to the query engine, which is the whole point of having a query engine.

Label-based suppression works because it encodes intent. "Ignore NFS snapshot failures" is a decision, and decisions that live in someone's head rot the moment that person goes on vacation. Putting the decision in the rule expression makes it durable, reviewable, and visible in the same place as the alert it governs.

The target-versus-exporter split works because "I can't see" and "the thing is broken" demand different responses, and an alert that conflates them sends you down the wrong path under pressure. Naming the two states separately is cheap insurance against the expensive mistake of treating a permissions error like an outage.

A note on the Grafana alerting overlap

One source of duplicate pages worth flagging: you can define alerts as Prometheus/Mimir rules (the Ruler) and also define them in Grafana's unified alerting, and both can route to the same contact points. People wire up both without realizing it and then wonder why every incident pages twice.

Pick one home for alert definitions. I keep mine as PrometheusRule objects in Git alongside the rest of the GitOps config, because that's where the rest of the platform lives and code review catches bad expressions before they ship. Grafana then handles dashboards and visualization, not alert routing. If you prefer Grafana-managed alerts, that's a defensible choice, but run one or the other for any given signal. Two systems firing on the same metric is how you rebuild the noise problem you just spent a weekend fixing.

What I'd do differently and what still bites

If I were starting a monitoring stack from scratch, I'd write the first alert with a for: clause and a join already in place, before I ever shipped a bare up == 0. The bad habits are easy to form and annoying to unwind once a team has learned to distrust the channel.

A few things that still catch me:

Exporters that get OOMKilled look like target outages. nvidia-gpu-exporter (v1.4.1) needs more memory than you'd guess, around 256Mi rather than the 64Mi you might set by reflex, because the NVIDIA runtime injects libraries into the container at start. Set the limit too low and it restarts in a loop, your GPU alerts flap, and you blame the GPU. The behavior ties back to how the runtime mounts things; the NVIDIA Container Toolkit post explains why that injection happens.
no data is not OK. An alert that can't evaluate because its metric vanished is a failure state, not a passing one. Use absent() rules to catch metrics that should always exist and have gone missing, otherwise a deleted exporter reads as eternal calm.
Severity is a routing decision, not a vibe. Reserve paging severity for things that need a human awake right now. Everything else is a warning that lands in a channel you read during business hours. If every alert is critical, none of them are.

The goal was never zero alerts. It's that when one fires, you believe it. An alert channel people trust is worth more than one that's technically complete, because the complete-but-ignored channel catches nothing the moment it matters. Spend the extra line on for:, the extra clause on the join, and the extra label on suppression. That's the difference between a monitoring system and a smoke detector with a dead battery that beeps at random.

SealedSecrets: Storing Secrets in Git Without the Risk

Guatu — Sat, 27 Jun 2026 14:15:48 +0000

A base64-encoded string in a Git repo is not a secret. It's plaintext with one extra step, and echo "cGFzc3dvcmQ=" | base64 -d undoes that step in about a second. Plenty of teams discover this the hard way when a secret scanner flags their carefully "encoded" data: block as a live database password sitting in the commit history, readable by anyone who ever cloned the repo.

That's the core tension of GitOps. You want Git to be the single source of truth for everything in the cluster, but "everything" includes API keys, TLS private keys, and database passwords, and committing those in the clear is how breaches start. SealedSecrets resolves that tension with a small controller and a CLI, and it's the lightest-weight tool I know of that does the job properly. If you run ArgoCD or Flux and you're still keeping a secrets/ folder out of version control "for now," this is the post for you.

The false starts I'd warn you about

Before SealedSecrets clicks, most people cycle through the same set of wrong answers. I did too.

The first instinct is to just base64-encode the value and call it encrypted. Kubernetes stores secret data as base64, so it looks scrambled, and that visual scramble fools a surprising number of code reviews. Base64 is an encoding, not encryption. There is no key. Anyone with the repo has the secret. Cross this one off immediately.

The second instinct, at least mine, was to reach for Vault. HashiCorp Vault (or its OpenBazaar-era forks) is a genuinely good piece of software, and if you're running a platform team with dynamic secrets, lease revocation, and audit requirements, it earns its keep. For a homelab or a small cluster, it's a second stateful system to run, unseal, back up, and babysit. You're now protecting your secrets manager more carefully than the secrets it holds. That's a lot of operational surface area to avoid committing a few API keys to Git.

Then there's the git history problem, which trips people who try to fix this reactively. Say you already committed a plaintext secret last month. Deleting it from the current file does nothing. It lives forever in the history, retrievable with git log -p. You'd have to rewrite history with git filter-repo or BFG, force-push, and rotate the secret anyway because you have to assume it leaked. The lesson there is that secret management is a preventative discipline. The tool has to make the wrong thing hard to do in the first place.

SOPS with age or KMS keys is a legitimate alternative, and I'd put it second on my list. It encrypts files before they hit Git and decrypts them at apply time. The reason I land on SealedSecrets for Kubernetes-native GitOps is that it lives inside the cluster as a CRD, the decryption never leaves the controller, and the developer workflow is a single command. Different tradeoffs, both valid.

How SealedSecrets actually works

The model is asymmetric encryption, and once that clicks, everything else follows.

You install a controller (Bitnami's sealed-secrets-controller, typically into kube-system). On startup it generates an RSA key pair. The public key is freely distributable. The private key never leaves the cluster. You then use the kubeseal CLI to encrypt a normal Kubernetes Secret against that public key, producing a SealedSecret custom resource. That resource is safe to commit, safe to put in a public repo, safe to paste in a Slack channel. The only thing in the universe that can decrypt it is the controller holding the matching private key.

When the SealedSecret lands in the cluster, the controller watches for it, decrypts the payload, and produces a regular Secret with the same name and namespace. Your workloads consume that Secret exactly as they always have. Nothing downstream knows the difference.

Here's the workflow loop. Start with a normal secret manifest you generate locally and never commit:

# Generate a plain Secret manifest (stays on your machine, never committed)
kubectl create secret generic db-credentials \
  --namespace=app-prod \
  --from-literal=password='example-not-a-real-password' \
  --dry-run=client -o yaml > secret.yaml

Then seal it:

# kubeseal fetches the controller's public cert and encrypts against it
kubeseal --format yaml < secret.yaml > sealed-secret.yaml

# Delete the plaintext, commit the sealed version
rm secret.yaml
git add sealed-secret.yaml

The output is a SealedSecret whose encryptedData is an opaque blob:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: db-credentials
  namespace: app-prod
spec:
  encryptedData:
    password: AgBy3i4OJSWK+PiTySYZ...truncated...rO43cGDEq
  template:
    metadata:
      name: db-credentials
      namespace: app-prod

That blob is what goes in Git. ArgoCD syncs the SealedSecret, the controller unseals it into a real Secret, and your app starts. The person writing the manifest never has to see the production value in plaintext, which is the zero-trust angle I'll come back to.

One operational detail worth setting up early: seal offline using a saved public cert instead of hitting the cluster every time. Pull the cert once, commit it (it's public), and point kubeseal at it:

# Fetch the public cert once
kubeseal --fetch-cert > pub-cert.pem

# Seal against the saved cert, no cluster connection needed
kubeseal --cert pub-cert.pem --format yaml < secret.yaml > sealed-secret.yaml

This matters because it lets you seal secrets in CI without giving the pipeline cluster access. A GitHub Actions job can encrypt a new secret during a PR using only the public cert, which pairs nicely with the kind of automated pipeline I described in Infrastructure as Code, but Automated: OpenTofu and GitHub Actions.

The scopes gotcha

By default, a SealedSecret is bound to both its name and its namespace. This is "strict" scope, and it's the right default, but it bites people who don't expect it. If you seal a secret for app-prod and then try to reuse that same sealed blob in app-staging, the controller refuses to decrypt it. Same if you rename it. The name and namespace are mixed into the encryption, so changing either invalidates the seal.

You can loosen this when you genuinely need to:

# Namespace-wide: any name within the namespace can decrypt it
kubeseal --scope namespace-wide --format yaml < secret.yaml > sealed.yaml

# Cluster-wide: any name in any namespace can decrypt it
kubeseal --scope cluster-wide --format yaml < secret.yaml > sealed.yaml

My advice is to stay on strict scope unless you have a concrete reason not to. Loosening the scope trades blast radius for convenience. Cluster-wide in particular means anyone who can create a Secret object anywhere can pull your value out, which quietly undoes part of the boundary you set up with least-privilege service accounts.

The part nobody warns you about: this is a backup problem

Here's the reframe that took me a while to internalize. SealedSecrets makes your encryption essentially bulletproof. What it does instead is concentrate all your risk into one tiny object: the controller's private key. Lose that key, and every SealedSecret in every repo becomes permanently undecryptable. The blobs are fine. The cluster that can read them is gone.

Picture the disaster cleanly. Your cluster dies (a failed etcd restore, a wiped control plane, a homelab node that won't come back). You rebuild from scratch, install a fresh SealedSecrets controller, and point ArgoCD at your repo. ArgoCD dutifully applies every SealedSecret. The new controller generated a new key pair on install, so it cannot decrypt a single one of the blobs sealed against the old public key. Every secret-dependent workload fails to start. Your Git repo is intact, your manifests are perfect, and none of it works.

The fix is to treat the private key as the single most important artifact in your infrastructure and back it up off-cluster. The keys live as Secrets in the controller's namespace, labeled so you can grab the full set in one shot:

# Back up the active sealing keys (THIS FILE CONTAINS THE PRIVATE KEYS)
kubectl get secret -n kube-system \
  -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o yaml > sealed-secrets-master.key

That output is the actual private key material, so it never goes near Git. It goes into a password manager or an encrypted, off-cluster backup. Restoring is the inverse: apply the key backup before the controller fully reconciles, then restart it so it picks up the old keys:

# On the rebuilt cluster, restore keys then restart the controller
kubectl apply -f sealed-secrets-master.key
kubectl rollout restart deployment sealed-secrets-controller -n kube-system

Now the controller holds both the old and new private keys, decrypts all your historical blobs, and your secrets come back to life. I wrote a dedicated walkthrough on exactly this lifecycle in SealedSecrets Key Backup: Don't Lose Your Encryption Keys, because it's the one step that separates a working SealedSecrets setup from a time bomb.

One thing that helps here: the controller rotates its key every 30 days by default, generating a fresh pair while keeping all the old ones for decryption. That's good for forward security, but it also means your key backup is never "done." A backup from two months ago is missing two rotations of keys. Either automate the backup on a schedule or pin the key and disable rotation deliberately, knowing the tradeoff. Stale backups are how people think they're safe right up until they aren't.

The GitOps cleanup reality check

The other thing tutorials skip is what happens when you remove or rotate a secret in a GitOps world, because ArgoCD's relationship with deleted resources is more complicated than "delete from Git, it disappears from the cluster."

By default, the controller sets an owner reference on the Secret it generates, pointing back to the SealedSecret. That's the well-behaved case: delete the SealedSecret and Kubernetes garbage-collects the generated Secret along with it. Clean. The trouble starts when that chain is broken, or when you've turned off ArgoCD's pruning.

If your ArgoCD Application doesn't have automated pruning on, removing a SealedSecret from Git leaves the object sitting in the cluster as an orphan. ArgoCD shows the app as synced because everything in Git exists in the cluster; it just won't remove the things that used to be there. You end up with ghost SealedSecrets, and sometimes the Secrets they spawned, lingering long after you deleted them from the repo. To get the behavior most people actually want, turn on prune and self-heal:

spec:
  syncPolicy:
    automated:
      prune: true      # delete cluster resources removed from Git
      selfHeal: true   # revert manual drift back to Git state

Be deliberate about prune: true, though, because it's the same flag that creates the scarier failures. When ArgoCD prunes a whole application, it deletes resources in dependency order and respects finalizers, and that's where things get stuck. I've watched deletions hang on a PVC whose finalizer waits on a volume that's still attached, or an IngressRoute that won't go because something downstream holds a reference. The SealedSecret deletes instantly; it's the stateful neighbors (PVCs, certificates, anything with a finalizer) that turn a clean delete into a half-removed application you have to untangle by hand. None of that is SealedSecrets' fault directly, but rotating or removing a secret often means touching the surrounding app, and that's when the orphans surface. The App-of-Apps structure I described in GitOps for Homelabs makes this more predictable by keeping deletion boundaries clear, but you still want to watch a prune happen the first few times rather than trusting it blindly.

Why this is the right shape of tool

The reason SealedSecrets earns a place in a GitOps stack comes down to where the trust boundary sits. The developer, the CI pipeline, and the Git host all handle only the encrypted blob. None of them can decrypt it. The only component that ever sees plaintext is the controller, which already runs inside the trust boundary of the cluster and is governed by RBAC. You've moved the secret out of every system that shouldn't have it and into the one place that already has to.

That's a genuine zero-trust property, not marketing. A new engineer can submit a PR that adds a production secret without ever knowing its value, sealing it against the public cert from CI. Review it, merge it, and the controller is the only thing that ever decrypts it. Compare that to a model where someone has to know the real password to put it in the right place, and you can see why this is the better default for a team.

It also stays lightweight in a way that matters operationally. One controller, one CRD, one CLI. No second clustered datastore to keep alive, no unseal ritual, no separate HA story. For homelabs and small-to-medium clusters, the cost-to-value ratio is hard to beat, and it composes cleanly with the rest of a Kubernetes-native stack like cert-manager handling your TLS. If you're designing a GitOps platform for a team and want the secret lifecycle and disaster-recovery story built in from the start rather than bolted on later, that's the kind of thing I help with through infrastructure consulting.

Lessons learned

If I were setting this up fresh on a new cluster, here's what I'd do in order and what I'd watch for.

Back up the private key on day one, before you seal anything. The temptation is to install the controller, seal a few secrets, and move on, treating the backup as a later task. That's exactly backwards. The key is most valuable and most likely to be forgotten right at the start. Automate the backup to an off-cluster, encrypted destination and verify the restore at least once, because an untested backup is a guess.

Stay on strict scope and resist the urge to go cluster-wide for convenience. Every loosening of scope widens who can extract the value, and you rarely get a reminder of that decision until it matters.

Treat removal as carefully as creation. Adding a SealedSecret is the easy, well-lit path. Removing or rotating one drags in ArgoCD pruning, finalizers, and owner references, and that's where the orphaned resources hide. Turn on prune and selfHeal so Git stays authoritative, but watch the first few prunes and keep an eye on anything stateful nearby.

Validate sealed manifests in CI before they reach main. A malformed SealedSecret won't decrypt, and you'd rather catch that in a PR check than at sync time when a workload won't start. The same manifest-validation habit from Stop Merging Broken YAML applies cleanly here.

The thing that surprised me most was how completely SealedSecrets shifts the problem. You go in worried about encryption strength and whether someone can crack the blob in your repo. You come out realizing the encryption was never the weak point. The weak point is a single RSA private key that, if you lose it, turns your perfectly version-controlled secrets into perfectly version-controlled garbage. Get the backup right, and the rest of the tool more or less takes care of itself.

MetalLB on Bare Metal: LoadBalancer Without a Cloud Provider

Guatu — Sat, 27 Jun 2026 12:15:48 +0000

Deploy a Service of type LoadBalancer on a bare-metal cluster and kubectl get svc hands you a permanent <pending> in the EXTERNAL-IP column. No error, no event, no hint. The Service just sits there forever, waiting for an external IP that will never arrive, because nothing in your cluster is responsible for assigning one.

That gap is the entire reason MetalLB exists. On AWS or GCP, the cloud controller manager watches for LoadBalancer Services, calls the provider API, provisions an actual load balancer, and writes the IP back into the Service status. On bare metal there is no API to call. Kubernetes ships the abstraction but not the implementation, and you're left holding a Service type that does nothing until you supply the missing piece yourself.

Who should care

If you run Kubernetes on Proxmox VMs, on a stack of mini PCs, or on anything that isn't a managed cloud, you've hit this. The moment you try to expose ingress-nginx, Traefik, a database, or an LLM inference endpoint to the rest of your LAN, you need a stable IP that lives outside the cluster and routes in. MetalLB is the most common answer, and the config has changed enough across versions that a 2021 tutorial will actively mislead you. This walks through MetalLB v0.15.x, the L2-versus-BGP decision, and the connectivity traps that make an IP look assigned while traffic quietly goes nowhere.

What I reached for first (and why it was wrong)

Before MetalLB, the obvious-looking options all have a catch.

NodePort. It works, technically. You get a port in the 30000-32767 range on every node, and you can hit http://node-ip:31234. Then you try to use it for real and remember why nobody runs production this way: the ports are ugly, they're not 80/443, and you've now hardcoded a specific node's IP into whatever's calling the service. If that node reboots or drains, your "load balancer" is a single point of failure pointing at a machine that's gone.

hostNetwork on the ingress controller. Bind ingress-nginx directly to ports 80 and 443 on the host. This also works, and it's genuinely fine for a single-node setup. On a multi-node cluster it falls apart: now you need an external load balancer or DNS round-robin in front of the nodes to spread traffic, and you've moved the problem one layer up the stack instead of solving it. You also lose the clean LoadBalancer Service abstraction that everything else in the ecosystem expects.

Assuming spec.loadBalancerIP would just work. I set spec.loadBalancerIP: 10.0.0.210 on a Service, expecting Kubernetes to honor it. Nothing happened. That field assigns an IP only if a controller is watching for it and acts on the request. Without MetalLB or a cloud provider, it's an inert string. Worse, in newer MetalLB versions that field is deprecated in favor of an annotation, so even after installing MetalLB the old syntax silently does nothing.

The mistake underneath all three: treating type: LoadBalancer as a feature Kubernetes provides, when it's really a contract Kubernetes defines and expects something else to fulfill.

Installing MetalLB

MetalLB has two pieces. A controller Deployment handles IP allocation, watching Services and writing the assigned address into their status. A speaker DaemonSet runs on every node and is responsible for actually announcing the IP to the network, either via ARP (L2 mode) or BGP.

I install it with Helm so it fits a GitOps flow. If you run ArgoCD app-of-apps, this slots in as one more application.

helm repo add metallb https://metallb.github.io/metallb
helm install metallb metallb/metallb \
  --namespace metallb-system \
  --create-namespace \
  --version 0.15.3

Installing the chart does nothing visible on its own. MetalLB ships with no IP pool by default, which is deliberate: handing out addresses you didn't explicitly authorize would be a great way to start an IP conflict war with your DHCP server. You have to tell it which addresses it owns.

That configuration moved to CRDs back in v0.13. If a guide tells you to edit a ConfigMap named config in the metallb-system namespace, it predates that change and the syntax no longer applies. The two objects you need are IPAddressPool and L2Advertisement.

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: lan-pool
  namespace: metallb-system
spec:
  addresses:
    - 10.0.0.200-10.0.0.250  # carve this OUT of your DHCP range
  autoAssign: true

apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: lan-l2
  namespace: metallb-system
spec:
  ipAddressPools:
    - lan-pool

The pool defines which addresses MetalLB may hand out. The L2Advertisement says "announce these addresses using Layer 2." Without the advertisement object, MetalLB will assign an IP from the pool but never tell the network it exists, which produces one of the most confusing failure states: a Service with an EXTERNAL-IP that nothing on the LAN can reach. The pool without the advertisement is half a configuration.

The address range matters. Those IPs must live in the same subnet as your nodes, and they must be outside whatever your DHCP server hands out. Pick a slice your router will never lease. A .200-.250 block at the top of a /24 is a common choice precisely because most DHCP pools stop well before there.

Apply both, deploy a test Service, and the <pending> finally resolves:

$ kubectl get svc -n ingress-nginx
NAME                 TYPE           EXTERNAL-IP   PORT(S)
ingress-nginx        LoadBalancer   10.0.0.200    80:31...,443:32...

L2 versus BGP: pick the simple one until you can't

MetalLB runs in one of two modes, and most homelab and small-cluster setups want Layer 2.

L2 mode uses ARP for IPv4 (NDP for IPv6). One speaker pod becomes the leader for a given Service IP and answers ARP requests for it. When a host on the LAN asks "who has 10.0.0.200?", that node replies with its own MAC, traffic arrives at the node, and kube-proxy forwards it to a backend pod. The whole thing needs zero cooperation from your router or switch. It looks like a normal host claiming an IP, which is exactly what it is.

BGP mode has each speaker peer with your router and advertise the Service IPs as routes. The router then load-balances across nodes via ECMP. This gives you real multi-node traffic distribution and faster failover, but it requires a router that speaks BGP and a willingness to manage routing config. That's a reasonable trade in a rack with a proper top-of-rack switch. In a homelab where the "router" is a consumer box, it's usually more than you want to take on.

	L2 (ARP/NDP)	BGP
Router requirement	None	Must speak BGP
Traffic distribution	One node per IP at a time	ECMP across nodes
Failover speed	Seconds (ARP re-announce)	Sub-second
Setup complexity	Low	Moderate to high
Best for	Homelabs, small clusters	Datacenter, high throughput

The honest caveat about L2: it is not true load balancing. For any single Service IP, all traffic lands on one node, the elected leader, and that node forwards internally. You get failover (if the leader dies, another speaker takes over and re-announces via gratuitous ARP) but not bandwidth aggregation. For a homelab serving an ingress controller or an Ollama API, that's completely fine. A single 10G node will saturate most home internet connections long before it becomes the bottleneck. If you genuinely need to spread inbound traffic across nodes, that's the line where BGP earns its complexity.

Why L2 mode actually works

The mechanism is worth understanding, because it explains every weird failure you'll hit later.

When the leader speaker announces a Service IP, it doesn't add the IP to a network interface in the usual sense. It answers ARP requests at the protocol level. A client on the LAN broadcasts an ARP query, the leading node responds with its MAC address, and the client's ARP table now maps the Service IP to that node's hardware address. Traffic flows to the node, hits iptables/IPVS rules installed by kube-proxy, and gets DNAT'd to a pod endpoint.

When the leader node goes down, the other speakers notice through MetalLB's memberlist-based election, a new leader is chosen, and it sends a gratuitous ARP. Gratuitous ARP is an unsolicited "hey, this IP is at my MAC now" broadcast that updates every device's ARP cache without them asking. That's the failover. It takes a few seconds for caches to update, which is why L2 failover is measured in seconds rather than milliseconds.

The reason this is all Layer 2 and not Layer 3 is also the reason for the biggest constraint: ARP doesn't cross subnet boundaries. A router won't forward an ARP request from one VLAN into another. So the Service IP, the nodes, and the client all have to sit on the same broadcast domain. The instant you want a Service IP that lives on a different VLAN than your nodes, plain L2 mode stops being enough, and you're into either BGP or careful per-interface advertisement.

The port-ownership trap

Here's the failure that eats the most debugging time, and it's the one tutorials never warn you about. The IP gets assigned. kubectl get svc shows the EXTERNAL-IP. You telnet the port and it connects. Everything says "working." But the actual application returns connection resets or timeouts, and you can't figure out why a connection that succeeds is also broken.

A successful TCP handshake to a MetalLB IP proves exactly one thing: a node owns the IP and accepted the connection. It says nothing about whether a healthy backend pod exists behind it. The classic cause is externalTrafficPolicy.

apiVersion: v1
kind: Service
metadata:
  name: ollama
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local  # the trap
  selector:
    app: ollama
  ports:
    - port: 11434
      targetPort: 11434

With externalTrafficPolicy: Local, traffic is only delivered to pods on the same node that received it, and the source IP is preserved. The problem: MetalLB elects a leader node to announce the IP, but if your single Ollama pod is scheduled on a different node, the leader has no local endpoint. Traffic arrives, finds nothing local to forward to, and gets dropped. The TCP connect still succeeds because the node owns the IP; the request just dies after that.

MetalLB is supposed to handle this by only announcing from nodes that have a ready endpoint, but the interaction with a single-replica workload pinned to one node is genuinely confusing the first time you meet it. The fix, when you don't need to preserve the client source IP, is externalTrafficPolicy: Cluster. That lets any node accept the traffic and forward it internally to wherever the pod actually runs, at the cost of an extra hop and a SNAT'd source address.

The lesson generalizes past MetalLB: stop treating "the port is open" as "the service works." Verify the full path.

# 1. Did MetalLB assign an IP from the pool?
kubectl get svc -A | grep LoadBalancer

# 2. Which node is announcing it? (look for the assigned IP)
kubectl logs -n metallb-system -l component=speaker | grep 10.0.0.200

# 3. From ANOTHER host on the same VLAN, does the IP answer ARP?
arping -I eth0 10.0.0.200

# 4. Does the backend actually respond, not just accept the connection?
curl -v http://10.0.0.200:11434/api/tags

That step 3 is the one people skip. arping confirms the IP is genuinely claimed at Layer 2 on your network, independent of anything Kubernetes thinks. If arping gets no reply but kubectl swears the IP is assigned, your speaker isn't announcing on the interface you expect, and you've found your problem before wasting an hour in application logs.

The hybrid VLAN reality

Most tutorials assume a flat network where every node has one NIC on one subnet. Real labs are segmented. You might want ingress traffic on one VLAN and management on another, with nodes carrying multiple interfaces.

MetalLB v0.15.x handles this through interface and node selectors on the L2Advertisement. You can scope a pool so it only ever announces on a specific NIC, which keeps an ingress IP from accidentally being advertised on your management network.

apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: ingress-vlan
  namespace: metallb-system
spec:
  ipAddressPools:
    - ingress-pool
  interfaces:
    - eth1          # only announce on the ingress-facing NIC
  nodeSelectors:
    - matchLabels:
        node-role: ingress

This is where L2's same-subnet constraint bites hardest. Each pool's addresses still have to be reachable on the broadcast domain of the interface announcing them. You can segment which IPs go where, but you can't make ARP magically route across VLANs. If your design genuinely needs Service IPs that live on subnets your nodes don't touch, that's the signal to either bring those VLANs to the nodes as tagged interfaces or move to BGP and let the router do the routing it's built for.

MetalLB as the foundation, not the finish line

A MetalLB IP is rarely the whole story. It's the stable entry point everything else builds on. The typical flow looks like this:

External client → MetalLB IP → ingress controller Service → Ingress/IngressRoute → backend Service → pod.

MetalLB hands ingress-nginx or Traefik a single, stable LAN IP. Your ingress controller terminates TLS and routes by hostname to dozens of backends. You point a wildcard DNS record at that one MetalLB IP, and now *.lab.example.com resolves to your cluster. This is also the precondition for automated certificates: cert-manager with a DNS-01 solver needs your ingress reachable at a known address before it can prove ownership and issue certs. MetalLB provides the address; cert-manager provides the identity.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ollama
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  ingressClassName: nginx
  tls:
    - hosts: [ollama.lab.example.com]
      secretName: ollama-tls
  rules:
    - host: ollama.lab.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: ollama
                port: { number: 11434 }

That Ingress doesn't reference a MetalLB IP directly, and that's the point. It targets the ingress controller's Service, which is the MetalLB-assigned IP. The same pattern extends upward to the Gateway API: a Gateway's listener address comes from a LoadBalancer Service, so gatewayAddresses in tools layered on top ultimately resolve to an IP MetalLB allocated. Get the bottom layer right and everything above inherits a stable address.

If you want a Service to keep a fixed IP across restarts, annotate it (the deprecated spec.loadBalancerIP is the trap here):

metadata:
  annotations:
    metallb.io/loadBalancerIPs: 10.0.0.210

What I'd do differently

A few things I wish I'd internalized before the first install.

Carve the IP pool out of DHCP on day one. The single most common MetalLB incident is an address conflict because the pool overlapped the DHCP lease range, and your router handed the same IP to a laptop. Decide the block, exclude it in your DHCP server, and document it. This is boring and it prevents a class of intermittent failures that are miserable to diagnose.

Default to externalTrafficPolicy: Cluster unless you have a specific reason to preserve source IPs. Source IP preservation matters for some applications, geo-logic, and rate limiting, but if you don't need it, Local only buys you the single-node-endpoint trap. Reach for Cluster first and switch to Local deliberately when a workload demands the real client IP.

Treat the arping check as a permanent part of your toolkit, not a one-time debug step. Separating "is the pod running" from "is the network path open" is the difference between an hour of log-reading and a thirty-second confirmation. MetalLB sits exactly at the seam between Kubernetes and your physical network, so when it misbehaves, you have to test both sides independently.

And keep the whole config in Git. The pool, the advertisement, and the per-Service annotations are small, declarative, and exactly the kind of thing that drifts when you edit it by hand at 11pm. Folding MetalLB into a GitOps repo alongside the rest of the cluster, the same way I treat Longhorn for bare-metal storage, means the networking layer is as reproducible as everything else. Bare-metal Kubernetes is mostly the work of rebuilding the conveniences a cloud provider gave you for free, one controller at a time. If you're standing up this kind of infrastructure and would rather not rediscover every trap yourself, that's the kind of work I do.

The surprise, looking back, was how little of the difficulty was MetalLB itself. The install is two Helm commands and two small CRDs. The hard part was everything around it: which subnet, which VLAN, which traffic policy, and the stubborn assumption that a connection accepting means a service working. Get those right and the <pending> that started all of this never comes back.

OpenTofu State in MinIO: An S3-Compatible Backend for IaC

Guatu — Sun, 21 Jun 2026 10:15:26 +0000

A .tfstate file on your laptop is a single point of failure with no redundancy and no audit trail. The first time a GitHub Actions runner applies against the same infrastructure you just touched locally, you get drift, a lock error, or the failure mode nobody warns you about: a perfectly clean apply that quietly tears down a container. None of that is a state-storage problem in isolation, but remote state is where you fix the storage half and where you have to start thinking about the destruction half.

If you run infrastructure-as-code against a homelab or any self-hosted environment, you already have a perfectly good S3 endpoint sitting there: MinIO. You probably stood it up for backups. It can hold your OpenTofu state too, inside your own perimeter, with no Terraform Cloud account and no real AWS bill. The catch is that "S3-compatible" hides a few sharp edges, and pointing CI at your real compute changes the blast radius of a typo.

Who should care

This is for people running OpenTofu (or Terraform) against on-prem providers like Proxmox, where the resources under management are real boxes you'd rather not lose. If your state already lives in AWS S3 with DynamoDB locking and you're happy, you don't need this. The interesting territory is self-hosted: you want cloud-native state management without renting the cloud, and the provider you're driving can replace resources in ways a managed cloud rarely does.

What I tried first

The obvious starting point is local state. It works, right up until it doesn't. You write your .tf files, run tofu apply, and the state lands in terraform.tfstate next to your code. Then you add a CI pipeline, the runner starts from an empty checkout with no state, and OpenTofu happily plans to create everything you already created. Now you've got two sources of truth and a plan that wants to make duplicates.

The next instinct is to gitignore the state and commit it anyway "just for CI," or to scp it into the runner. Both are bad. State holds resource IDs and, depending on your providers, secrets in plaintext. It does not belong in Git. So remote state it is, and since MinIO is already running, the S3 backend looks like a five-minute job.

It is not a five-minute job the first time. My first MinIO backend block was copied from an AWS example with the endpoint swapped in:

terraform {
  backend "s3" {
    bucket   = "opentofu-state"
    key      = "proxmox/terraform.tfstate"
    region   = "us-east-1"
    endpoint = "https://minio.example.com"
  }
}

tofu init rejected endpoint as deprecated, and once I got past that, the AWS SDK underneath tried to reach opentofu-state.minio.example.com. That hostname does not exist and the wildcard cert doesn't cover it, so I got TLS and DNS errors that looked like a MinIO problem and were actually an addressing-style problem. Then OpenTofu tried to call the AWS instance-metadata endpoint to figure out a region and credentials, which obviously goes nowhere on a bare-metal runner. Every error pointed at a different layer. None of them said "you forgot path-style addressing," which is what every single one of them meant.

The actual solution

Here's the backend block that actually works against MinIO with current OpenTofu. The shape of this changed across versions, so the keys matter:

terraform {
  backend "s3" {
    bucket = "opentofu-state"
    key    = "proxmox/terraform.tfstate"
    region = "us-east-1" # arbitrary, but required by the SDK

    endpoints = {
      s3 = "https://minio.example.com"
    }

    use_path_style              = true # bucket in the path, not the hostname
    use_lockfile                = true # S3-native locking, no DynamoDB
    skip_credentials_validation = true
    skip_requesting_account_id  = true
    skip_metadata_api_check     = true
    skip_region_validation      = true
  }
}

A few of those lines are the whole game. endpoints = { s3 = ... } replaced the old top-level endpoint attribute. use_path_style = true is the rename of what older guides call force_path_style. The four skip_* flags stop the AWS SDK from trying to validate credentials, look up an account ID, hit the metadata service, or sanity-check the region against AWS's real region list. On MinIO every one of those checks either fails or wastes time.

Credentials do not go in the block. Keep them out of .tf files entirely and feed them through the standard AWS environment variables, which the S3 backend reads natively:

export AWS_ACCESS_KEY_ID="$(op read 'op://Infra/minio-opentofu/access-key')"
export AWS_SECRET_ACCESS_KEY="$(op read 'op://Infra/minio-opentofu/secret-key')"
tofu init

That access key should not be your MinIO root user. Create a dedicated user with a policy scoped to exactly the one bucket OpenTofu touches. Least privilege here is the difference between "a leaked CI secret can read one state file" and "a leaked CI secret owns your entire object store." This is the same mindset I wrote about for agent service accounts: give the automation the smallest grant that lets it do its job.

The MinIO policy looks like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
      "Resource": ["arn:aws:s3:::opentofu-state"]
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": ["arn:aws:s3:::opentofu-state/*"]
    }
  ]
}

ListBucket and GetBucketLocation are on the bucket itself. The object actions live on the /* path. DeleteObject is non-optional even though that surprises people: the lockfile is written before an operation and deleted after, so without delete you can acquire a lock and never release it. Apply it with the MinIO client:

mc admin policy create local opentofu-state ./opentofu-state-policy.json
mc admin user add local opentofu-ci "$(op read 'op://Infra/minio-opentofu/secret-key')"
mc admin policy attach local opentofu-state --user opentofu-ci

Wiring it into CI

In GitHub Actions, the MinIO credentials come from repository secrets and get injected as the AWS env vars the backend expects. The job stays small:

- name: OpenTofu Init
  env:
    AWS_ACCESS_KEY_ID: ${{ secrets.MINIO_ACCESS_KEY }}
    AWS_SECRET_ACCESS_KEY: ${{ secrets.MINIO_SECRET_KEY }}
  run: tofu init

- name: OpenTofu Plan
  env:
    AWS_ACCESS_KEY_ID: ${{ secrets.MINIO_ACCESS_KEY }}
    AWS_SECRET_ACCESS_KEY: ${{ secrets.MINIO_SECRET_KEY }}
  run: tofu plan -detailed-exitcode -out=tfplan

-detailed-exitcode is the piece most pipelines skip. It returns 0 for no changes, 2 for a non-empty plan, and 1 for an error. That lets you gate apply on human approval whenever the plan is non-empty, instead of letting every push to main run straight through to apply -auto-approve. If your runner can't reach MinIO over the public internet, this is also where a Tailscale step earns its keep; I covered that access pattern in Tailscale subnet routers.

The safety net that actually matters

Storing state remotely solves the "where does the truth live" problem. It does nothing for the "a successful apply destroyed something" problem, and on Proxmox that problem is real. The bpg and Telmate providers both have attributes marked ForceNew. Change one of those on an LXC, the network bridge or the rootfs storage, and the provider cannot update in place. It plans a destroy followed by a create. Replacing an LXC means deleting the container and its filesystem, then building a fresh one. The data that lived on it does not come back.

Read that plan by hand and you'd catch it instantly: -/+ destroy and then create replacement. Run it under -auto-approve in CI off a push to main, and the box is gone before anyone looks at the diff. Remote state didn't cause that, but remote state is what made unattended applies easy, so it's the moment to add guards. The first guard is a lifecycle block on anything stateful:

resource "proxmox_virtual_environment_container" "app" {
  # ... container config ...

  lifecycle {
    prevent_destroy = true
  }
}

With prevent_destroy = true, any plan that would delete or replace this resource fails at plan time with an error instead of proceeding. That's the behavior you want: the pipeline stops and shouts rather than quietly doing the destructive thing. When you genuinely need to replace the resource, you remove the block deliberately, in a reviewed commit, with your eyes open. The same defensive instinct shows up in provider-driven recreation elsewhere; I hit a version of it with hardware, where a card got replaced out from under me, in the GPU passthrough field guide.

Why it works

Two mechanics are worth understanding instead of cargo-culting, because both will bite you again in other contexts.

The path-style flag comes down to URL construction. AWS S3 prefers virtual-hosted-style addressing: the bucket becomes a subdomain, bucket.s3.amazonaws.com. MinIO serves buckets path-style by default: minio.example.com/bucket. When the SDK defaults to virtual-hosted style against MinIO, it builds opentofu-state.minio.example.com, which has no DNS record and no matching certificate. Setting use_path_style = true tells the SDK to keep the bucket in the path. Every confusing TLS-and-DNS error I got was this and only this, wearing four different costumes.

State locking is the other one. The classic Terraform/OpenTofu S3 backend needed a DynamoDB table to hold a lock item, because plain S3 historically had no way to do an atomic "create this only if it doesn't already exist." Self-hosters without DynamoDB faked it with sidecar services or just turned locking off and prayed. Recent OpenTofu (1.10+) added use_lockfile = true, which writes a .tflock object next to your state and relies on S3 conditional writes (PutObject with If-None-Match) to make lock acquisition atomic. MinIO implements conditional writes, so the lock is honest: a second apply that arrives while the first holds the lock gets refused at the object-store level, not by a polite convention. No extra database, no extra moving part. That's why the same MinIO instance you use for Velero Kubernetes backups can back your IaC state without bolting anything else on. One S3 layer, two jobs.

prevent_destroy works because it intercepts the plan graph before execution. OpenTofu walks the dependency graph, decides each resource needs no-op, update, create, or replace, and a lifecycle guard converts any planned delete on that resource into a hard error. It is a plan-time veto, not a runtime one, which is exactly why it's safe in CI: nothing has touched real infrastructure yet when it fires.

Lessons learned

The version churn on the S3 backend is the thing that wasted the most of my time, and most of it is invisible until init complains. endpoint became endpoints.s3. force_path_style became use_path_style. DynamoDB locking became use_lockfile. If you're following a guide written for an older OpenTofu or for Terraform pre-1.10, half the keys are renamed and you'll burn an afternoon on deprecation warnings that don't clearly explain the replacement. Check the backend docs for your exact version before copying anyone's block, including mine.

prevent_destroy is a seatbelt, not an airbag, and it has a real gap: it only protects a resource that's still in your configuration. Delete the resource block from your .tf files and the lifecycle rule goes with it, so the next plan cheerfully schedules the destroy with nothing left to veto it. Treat removing a stateful resource from config as a destructive operation in its own right, and review those diffs as carefully as you'd review an apply.

The destruction risk reframed how I think about remote state entirely. Moving state into MinIO felt like a storage decision, but the real change is that you've made unattended applies cheap, and cheap unattended applies against real compute need guardrails that local-state workflows never forced you to build. -detailed-exitcode to gate on human review, prevent_destroy on anything holding data, and a habit of actually reading replacement plans matter more than the backend block itself. This is the part of the workflow that complements the GitHub Actions automation I described in automating OpenTofu with GitHub Actions: the automation gets you speed, the guards keep the speed from being a liability.

Credentials were the last lesson, and an easy one to get lazy about. The MinIO key OpenTofu uses should be scoped to one bucket, stored in a secret manager, and pulled into env vars at runtime rather than parked in a .tf file or, worse, a shell history. If you've ever fought your shell over special characters in an automation token, you already know how these leak; I wrote about one such trap in Proxmox API tokens and the ! character. A scoped key plus a lifecycle guard turns the worst-case CI accident from "lost the cluster" into "lost one state file I can restore from a versioned bucket."

That last point is the quiet upside of running this on MinIO instead of a SaaS backend. Turn on bucket versioning and your state file gets the same history your backups do, inside the same perimeter, under the same retention you already control. If you want help designing that kind of self-hosted IaC and storage layer for a production environment, that's the sort of thing I do through GuatuLabs. The state of your infrastructure is some of the most sensitive data you have. Keeping it on hardware you own, behind credentials you scoped, with guards that refuse to delete the wrong box, is a reasonable thing to want, and it's maybe a weekend of work once you know which four flags actually matter.

CloudNativePG: Running PostgreSQL in Kubernetes Without the Pain

Guatu — Tue, 16 Jun 2026 00:15:32 +0000

A CloudNativePG cluster that sits in Setting up primary forever, with zero error events on the Cluster resource and a perfectly healthy operator, is one of the more frustrating ways to spend an afternoon. The operator says it's working. The pods never appear. And the actual cause has nothing to do with the database at all.

Running stateful databases on Kubernetes used to be the thing everyone told you not to do. CloudNativePG (CNPG) changed that calculus for a lot of people, including me. It's a proper operator: it handles failover, backups, connection routing, and rolling upgrades through native Kubernetes primitives instead of bolting Postgres onto a StatefulSet and praying. If you run a hardened cluster with admission controllers, network policies, and least-privilege RBAC, this post is about the friction you'll hit that the quickstart never mentions.

Who should care

If your cluster is vanilla, kubectl apply the operator and a Cluster manifest, and you're done in ten minutes. The CNPG docs are genuinely good for that path. This is for the rest of us: people running Kyverno or OPA Gatekeeper, self-signed cert chains, and the kind of policy-as-code setup where every workload has to justify its existence. That's where CNPG stops being a ten-minute install and starts being an integration project.

What I tried first

The first instinct, when a CNPG cluster hangs, is to assume you got the database config wrong. So you go read your Cluster manifest line by line. You check the storage class. You check that the PVC bound. You bump the operator log level and watch it cheerfully report that it's reconciling, over and over, with no complaints.

Here's the trap: the CNPG operator doesn't run initdb itself. It creates a Kubernetes Job to bootstrap the primary. That Job spawns a Pod. And in a hardened cluster, the Pod is where everything dies, because your admission controller is judging it against policies the operator's own Pods were exempted from but the bootstrap Job was not.

The mistake I see constantly is reading the wrong resource. People kubectl describe cluster and kubectl describe pod on the operator, find nothing, and conclude CNPG is broken. The events you need are on the Job and on the Pod the Job tries to create. A blocked Pod creation shows up as an event on the Job's owning controller, not on the Cluster:

# The Cluster looks stuck here, but says nothing useful
kubectl get cluster -n databases
# NAME       AGE   INSTANCES   READY   STATUS                    PRIMARY
# pg-main    8m    3           0       Setting up primary

# The real story is on the bootstrap Job's events
kubectl describe job -n databases pg-main-1-initdb

If a policy is the culprit, that describe output is where you'll finally see something like admission webhook "validate.kyverno.svc" denied the request: validation error: every container must define resource limits. The bootstrap Job's Pod template didn't set CPU/memory limits, your require-resource-limits policy rejected it, and the operator quietly retries forever because, from its perspective, it asked Kubernetes nicely and Kubernetes said no.

I spent longer than I'd like to admit assuming the storage layer was at fault before I went and looked at the Job. The lesson stuck: when an operator hangs, find the resource the operator creates, not the resource it manages.

The actual solution

1. Exempt CNPG lifecycle resources from blocking policies

CNPG generates Jobs and Pods on your behalf, and you can't directly edit their pod templates the way you would a Deployment you wrote. So the fix isn't to add resource limits to the Job. It's to teach your policy engine that CNPG-owned resources are allowed to skip the rule that's blocking them.

Every resource CNPG creates carries the cnpg.io/cluster label. That label is your exclusion key. For Kyverno, add an exclude block to the rule that's firing:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
spec:
  validationFailureAction: Enforce
  rules:
    - name: validate-resources
      match:
        any:
          - resources:
              kinds: ["Pod"]
      exclude:
        any:
          - resources:
              # CNPG-managed Pods (instances + bootstrap Jobs) carry this label
              selector:
                matchLabels:
                  cnpg.io/cluster: "*"
      validate:
        message: "Every container must define CPU and memory limits."
        pattern:
          spec:
            containers:
              - resources:
                  limits:
                    memory: "?*"
                    cpu: "?*"

This is a deliberately narrow exclusion. You're not disabling the policy. You're carving out resources that match a specific operator-owned label, which means a developer can't accidentally smuggle a limitless Pod past the gate by slapping a random label on it. If you want to be stricter, scope the exclusion to the databases namespace as well so the label only grants an exemption where CNPG is actually allowed to run.

The same idea applies to OPA Gatekeeper, just expressed differently: add the label to the constraint's match.excludedNamespaces or write a labelSelector exclusion in the constraint spec. The principle doesn't change. Match the operator's label, exempt the lifecycle resources, leave everything else under enforcement. I wrote about the general shape of this in Kyverno Admission Controllers: Policy-as-Code That Actually Works, and CNPG's initdb Job is the cleanest real-world example I've found of policy breaking infrastructure in a way that's invisible until you know where to look.

2. Give the operator the RBAC it actually needs

If you provision service accounts by hand instead of trusting the operator's defaults, remember that CNPG needs to manage Jobs, Pods, PVCs, Secrets, and Services on your behalf. A read-only or overly-scoped account will fail in the same silent way a policy block does: the reconcile loop runs, the create call gets a 403, and nothing visible happens.

The operator's ClusterRole covers this out of the box. If you're tightening it, the non-obvious permissions are the ability to create and delete Jobs (for initdb and restores) and to manage PVCs (for volume expansion and replica provisioning). Strip those and your cluster bootstraps fine until the first time it needs to scale or recover, then breaks. I go deeper on scoping accounts like this in Kubernetes RBAC: Building Least-Privilege Service Accounts.

3. Pin your PostgreSQL minor version away from 16.4

There's a known regression in PostgreSQL 16.4 where the server can hit a segmentation fault under certain memory conditions on nodes with large amounts of RAM available. If you're running CNPG on beefy worker nodes (16GB+ of available memory is the trigger zone), this is exactly the kind of thing that looks like a CNPG bug, a storage bug, or a kernel OOM, when it's actually upstream Postgres.

The fix is boring and effective: pin the image to a known-good minor and don't float the tag.

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: pg-main
  namespace: databases
spec:
  instances: 3
  # Pin explicitly. Do not use a floating major-version tag in production.
  imageName: ghcr.io/cloudnative-pg/postgresql:16.6
  storage:
    size: 20Gi
    storageClass: longhorn
  resources:
    requests:
      memory: "2Gi"
      cpu: "500m"
    limits:
      memory: "2Gi"
      cpu: "1"

Note the memory requests and limits are set to the same value. For a database, you almost never want Postgres getting throttled or evicted because a noisy neighbor ballooned and the scheduler decided your requests were a polite suggestion. Equal requests and limits put the Pod in the Guaranteed QoS class, which is what you want for a stateful workload you can't afford to lose to memory pressure.

4. Understand the three Services CNPG hands you

This is the part that pays off long after install. For a cluster named pg-main, CNPG creates a set of Services automatically, and each one routes to a different role:

Service	Routes to	Use it for
`pg-main-rw`	Current primary	Writes, migrations, anything that mutates
`pg-main-ro`	Replicas only	Read-only queries, reporting, analytics
`pg-main-r`	Any instance (primary or replica)	Reads where you don't care which node

The -rw Service is the important one: when CNPG fails over, it repoints -rw at the new primary. Your application doesn't need to know a failover happened. It keeps connecting to pg-main-rw.databases.svc.cluster.local and the operator handles the rest. That's the entire value proposition of running Postgres under an operator instead of as a hand-rolled StatefulSet.

For read/write splitting, point your app at two connection strings instead of one. Most ORMs and connection libraries support a primary/replica config:

# In your app's config or Secret
env:
  - name: DATABASE_URL_PRIMARY
    value: "postgresql://app:$(PGPASSWORD)@pg-main-rw.databases.svc.cluster.local:5432/appdb"
  - name: DATABASE_URL_REPLICA
    value: "postgresql://app:$(PGPASSWORD)@pg-main-ro.databases.svc.cluster.local:5432/appdb"

Send SELECTs that tolerate slight replication lag to -ro, and send everything else to -rw. The catch worth stating plainly: replicas are asynchronous by default, so a read immediately after a write can return stale data. If you need read-your-writes consistency for a given query, send it to -rw. Don't blanket-route all reads to replicas and then act surprised when a user doesn't see the row they just created.

5. Connection SSL: the untrusted-certificate wall

CNPG enables TLS by default and issues its own certificates through an internal CA. That's good for in-cluster security and annoying the first time a client refuses to connect because it doesn't trust the CA.

The error you'll see from a client is some flavor of SSL error: certificate verify failed or self-signed certificate in certificate chain. The wrong reaction is to globally disable TLS on the cluster. The right reaction depends on who's connecting:

# In-cluster clients: trust CNPG's CA. The operator publishes it as a Secret.
kubectl get secret pg-main-ca -n databases -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
# Then point the client at it:
# postgresql://...?sslmode=verify-full&sslrootcert=/etc/pg/ca.crt

For clients that genuinely can't do certificate verification (some managed platforms and serverless backends only support a binary "SSL on/off" toggle and can't be handed a custom CA), you have two honest options. Either set sslmode=require on the client, which encrypts the connection but skips CA verification, or terminate trust at a proxy you control. sslmode=require is the pragmatic middle ground: you keep encryption in transit and drop only the identity check. It's not as strong as verify-full, but it's a deliberate, documented tradeoff rather than turning TLS off entirely.

Here's the quick reference I keep around for the sslmode ladder:

`sslmode`	Encrypted?	Verifies CA?	Verifies hostname?
`disable`	No	No	No
`require`	Yes	No	No
`verify-ca`	Yes	Yes	No
`verify-full`	Yes	Yes	Yes

Aim for verify-full for anything in-cluster, where you control the CA distribution. Drop to require only for external clients that can't be handed the CA, and never to disable. If you're already running cluster-wide TLS automation, the CA-distribution problem is the same one cert-manager solves for ingress; I covered that workflow in cert-manager + Cloudflare DNS-01: Automated TLS for Everything.

6. Exposing pgAdmin without poking a hole in the cluster

You'll eventually want a GUI to poke at the database. The pattern I'd reach for is pgAdmin4 in its own namespace, reachable through your existing ingress controller, never exposed directly. Keep it in a separate namespace from the database so your network policies can treat it as an external-ish client that's explicitly allowed to reach the -rw/-ro Services, rather than something that lives inside the data tier.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: pgadmin
  namespace: pgadmin
  annotations:
    # Force HTTPS and lean on cert-manager for the cert
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    # pgAdmin needs a bigger body size for imports/exports
    nginx.ingress.kubernetes.io/proxy-body-size: "16m"
spec:
  ingressClassName: nginx
  tls:
    - hosts: ["pgadmin.example.com"]
      secretName: pgadmin-tls
  rules:
    - host: pgadmin.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: pgadmin
                port:
                  number: 80

Put authentication in front of it. pgAdmin's own login is fine, but I'd add an ingress-level auth layer (OAuth proxy or basic auth) so a leaked pgAdmin password isn't a direct line to your database. And lock down the NetworkPolicy so only the pgAdmin namespace can reach the database Services. A database admin GUI on the public internet with default credentials is how clusters become someone else's crypto miner.

Why it works

The thing that finally made CNPG click for me is that it's not pretending Postgres is stateless. It embraces the fact that a database has a primary and replicas, that failover is a real event, and that bootstrapping is a one-time Job rather than a steady-state process. Every piece of the design maps a Postgres concept onto a native Kubernetes object you can inspect with kubectl.

That's also why the failure modes are sneaky. The operator delegates the actual work to Jobs and Pods, so when an admission controller or RBAC rule blocks one of those, the operator has no good way to surface it beyond a stalled status. There's no exception thrown into your terminal. The reconcile loop is doing exactly what it's designed to do, which is keep trying, and "keep trying against a wall" looks identical to "working" until you go read the Job's events.

The Service abstraction works because CNPG owns the failover decision and the endpoint update atomically. When it promotes a replica, it updates the -rw Service's selector in the same control loop. There's no DNS TTL to wait out, no client-side failover logic to get wrong, no floating VIP to manage. Kubernetes Service routing was already solving "send traffic to whichever Pod currently has this role," and CNPG just plugs the primary/replica roles into that existing machinery. Running databases reliably on Kubernetes is the kind of platform-engineering work that separates a homelab toy from production infrastructure, and it's a chunk of what I do in consulting engagements.

Lessons learned

The biggest shift was learning to debug the resources the operator creates, not the ones it manages. kubectl describe cluster will lie to you by omission. The Job and its Pod tell the truth. If a CNPG cluster hangs in Setting up primary, my first move now is straight to the bootstrap Job's events, and nine times out of ten it's a policy or RBAC denial, not a database problem.

What surprised me was how much the hardened-cluster setup matters. Every CNPG tutorial assumes a permissive cluster, so the exact features that make a cluster production-grade (enforced resource limits, least-privilege RBAC, default-deny network policies) are the features that break the install. None of them are CNPG's fault. They're the cost of doing security right, and the fix is always a narrow, labeled exclusion rather than a blanket exception. If you run CNPG via GitOps, put those policy exclusions in the same ArgoCD app as the operator so they're never out of sync; the App-of-Apps pattern handles this cleanly.

If I were starting over, I'd pin the PostgreSQL minor version from day one and treat floating tags as a production smell, set Guaranteed QoS on the database Pods before the first incident rather than after, and write the read/write split into the application from the start instead of routing everything at the primary and refactoring later. None of those are hard. They're just the kind of decision that's cheap to make early and expensive to retrofit once you have data and uptime to protect.

CNPG genuinely delivers on running Postgres in Kubernetes without the pain, but only if you account for the cluster you actually have, not the empty one the docs assume. The operator is excellent. The integration with your security posture is the part you own.

Proxmox Backup Server: Incremental Backups for Your Whole Cluster

Guatu — Mon, 15 Jun 2026 18:15:32 +0000

A full Proxmox cluster rebuild from scratch takes somewhere between a weekend and a week, depending on how much of your config lives in Git versus your head. The VMs and LXCs themselves, the ones with actual state in them, those are the things you can't reconstruct from memory. Proxmox Backup Server (PBS) exists specifically for this problem: deduplicated, incremental backups of your entire virtualization layer, with verification built in.

If you're running a multi-node Proxmox cluster and your backup strategy is still "I'll just snapshot it manually before I do anything scary," this is the upgrade path. PBS slots into an existing cluster with surprisingly little friction, but the authentication model and a few operational quirks will trip you up if you don't know they're coming.

Why Not Just Use vzdump to NFS?

The built-in vzdump tool works. You can schedule backups to an NFS share and call it a day. I've seen plenty of homelabs run this way for years. The problem is what happens at scale.

With vzdump to a plain NFS target, every backup is a full copy. A 50 GB VM backed up daily for 30 days is 1.5 TB of storage, most of it identical data. PBS changes this fundamentally. It chunks the data, deduplicates across all backups (and across all VMs), and only transfers the changed chunks on subsequent runs. That 1.5 TB becomes something closer to 80-120 GB depending on churn rate.

The other thing vzdump alone doesn't give you is backup verification. PBS can mount and verify the integrity of every backup after it completes, checking that the data is actually restorable. That matters more than most people think. A backup you've never tested is just a hope.

I initially tried running vzdump backups to a Synology NFS share. It worked, but retention management was manual, dedup was nonexistent, and I had zero confidence that any given backup was actually restorable until I tried. PBS replaced all of that with a single integration point.

The PBS Placement Decision

Before installing anything, you need to answer one question: where does PBS run?

There are two reasonable options for a homelab:

Option 1: PBS as a VM on the cluster itself. Quick to set up, uses existing hardware, but your backup server lives on the infrastructure it's backing up. If you lose the node hosting PBS, you lose your backup target at the exact moment you need it most.

Option 2: PBS on a dedicated machine, physically separate from the cluster. This is the correct answer for anything you actually care about. A small mini-PC with a large spinning disk, or even an old desktop with a few terabytes of storage, is enough. The key property is that it's not on the same failure domain as your cluster.

I'd go with option 2 every time. A used mini-PC with a 4 TB drive costs less than the time you'll spend rebuilding a cluster from scratch. PBS itself is lightweight. It doesn't need much CPU or RAM. What it needs is disk space and network connectivity to your Proxmox nodes.

If you're running PBS on a NAS via NFS (mounting the NAS storage into a PBS VM), be aware that deduplication performance degrades over NFS compared to local storage. PBS's chunked dedup store does a lot of random I/O, and NFS adds latency to every operation. Local disk or direct-attached storage is preferable.

Installing PBS

PBS installs like any other Debian-based system. Download the ISO from the Proxmox site, boot it, run through the installer. The whole process takes about 10 minutes.

After installation, you'll access the web UI on port 8007:

https://10.0.0.50:8007

First thing to configure is a datastore, which is just a directory path where PBS will store backup chunks:

# On the PBS host, create the datastore directory
mkdir -p /mnt/backups/pbs-store

# Add it via the CLI (or through the web UI under Storage > Datastore)
proxmox-backup-manager datastore create main-store /mnt/backups/pbs-store

The datastore is where all the deduplicated chunks live. PBS handles the internal structure. You don't need to think about the file layout.

Adding PBS as Storage in Proxmox VE

On each Proxmox VE node (or once in a cluster, since storage config is shared), you add the PBS instance as a storage target. This is where the first gotcha lives.

In the PVE web UI, go to Datacenter > Storage > Add > Proxmox Backup Server. You'll need:

Server address (the IP of your PBS host)
Username and password (or API token)
Datastore name
Fingerprint (PBS uses a self-signed cert by default)

The fingerprint is available on the PBS dashboard or via:

# On the PBS host
proxmox-backup-manager cert info | grep Fingerprint

For a basic setup with username/password, this works out of the box. But if you're automating backup jobs or integrating with scripts, you'll want API tokens. And that's where things get interesting.

The API Token Authentication Trap

If you've worked with Proxmox API tokens before, you know PVE uses the format user@realm!tokenname with the secret passed as a separate header or parameter. PBS uses a similar but subtly different format, and the distinction will cost you hours if you don't catch it early.

The token format for PBS authentication:

# PVE token format (for reference)
user@realm!tokenname    (secret passed separately)

# PBS token format in storage config
user@realm!tokenname    (same structure, but permission model differs)

The real trap isn't the format. It's privilege separation.

When you create an API token in PBS, there's a checkbox labeled "Privilege Separation" that defaults to on. With privsep enabled, the token has its own independent permission set, completely separate from the user it belongs to. This means if your user backup@pbs has DatastoreBackup and DatastoreAudit roles on the datastore, but you created the token with privsep on and didn't assign those same roles to the token specifically, the token will authenticate successfully but return empty results or 403 errors on actual operations.

The fix:

# Create a user for backups
proxmox-backup-manager user create backup@pbs

# Create a token WITHOUT privilege separation
proxmox-backup-manager user generate-token backup@pbs pve-integration --privsep 0

# If you want privsep on (recommended for production), assign roles to the token directly
proxmox-backup-manager acl update / DatastoreBackup --auth-id backup@pbs!pve-integration

The --privsep 0 flag is the quick path for homelabs. The token inherits all permissions from its parent user. For a more locked-down setup, keep privsep on and explicitly grant the token the roles it needs. Either way, test the token before you walk away:

# Verify the token can actually list datastore contents
proxmox-backup-client list --repository 'backup@pbs!pve-integration@10.0.0.50:main-store'

If this returns an empty list (for a new datastore) or your existing backups, you're good. If it returns a 403 or permission error, check the privsep settings.

Scheduling Backup Jobs

With PBS added as a storage target in PVE, you schedule backups the same way you would any other vzdump job. Datacenter > Backup > Add. Select your PBS storage, pick the VMs and LXCs to include, set the schedule.

A reasonable starting configuration:

Schedule:     daily at 02:00
Selection:    all VMs and LXCs
Mode:         snapshot (for running machines)
Retention:    keep-last=7, keep-weekly=4, keep-monthly=3
Compression:  zstd

This gives you a week of daily recovery points, a month of weekly snapshots, and three months of monthly archives. Because PBS deduplicates, the storage cost of this retention policy is a fraction of what you'd expect.

The snapshot mode is important. It creates a consistent point-in-time snapshot without stopping the VM. For most workloads this is fine. If you're running a database directly in a VM (not in Kubernetes), consider using the stop mode or pre-freeze hooks to ensure filesystem consistency.

# You can also trigger a one-off backup via CLI
vzdump 100 --storage pbs-target --mode snapshot --compress zstd

The Stale Lock File Problem

Backup jobs will occasionally fail with an error like:

ERROR: backup of VM 101 failed - can't acquire lock '/var/lock/pve-manager/vzdump-101.lck'

This happens when a previous vzdump process was interrupted (killed, node rebooted during backup, OOM, etc.) and didn't clean up its lock file. The fix is straightforward:

# Check for stale lock files
ls -la /var/lock/pve-manager/vzdump-*.lck

# Remove the stale lock (only if no vzdump process is actually running)
ps aux | grep vzdump
# If no vzdump is running for that VMID:
rm -f /var/lock/pve-manager/vzdump-101.lck

There's a less obvious variant of this problem. On some nodes, the /var/lock/pve-manager/ directory itself can disappear after a reboot. This directory lives on a tmpfs and should be recreated by systemd-tmpfiles on boot. If it's missing:

# Recreate the lock directory
mkdir -p /var/lock/pve-manager

To make this persistent, verify that the tmpfiles configuration includes it:

# Check if the config exists
cat /usr/lib/tmpfiles.d/pve-manager.conf
# Should contain a line like:
# d /var/lock/pve-manager 0755 root root -

If that file is missing or doesn't include the lock directory, create a drop-in:

echo 'd /var/lock/pve-manager 0755 root root -' > /etc/tmpfiles.d/pve-manager.conf
systemd-tmpfiles --create

Backup Verification

PBS has a built-in verification system that reads back every chunk in a backup and checks its integrity. This is the feature that separates "I have backups" from "I have backups I can actually restore from."

Schedule verification jobs in the PBS web UI under Datastore > Verify Jobs. A good cadence is to verify the most recent backup daily and do a full verification of all backups weekly. Verification is I/O intensive but doesn't affect PVE operations since it runs on the PBS host.

# Manual verification via CLI
proxmox-backup-client verify --repository 'backup@pbs@10.0.0.50:main-store'

If verification fails for a specific snapshot, PBS will flag it in the UI. Don't ignore these warnings. A failed verification means that backup may not be restorable.

PBS in the Context of a Full 3-2-1 Strategy

PBS handles one layer of your backup stack: the hypervisor layer. VMs and LXCs, their disks, their configs. But if you're running Kubernetes on top of those VMs, there's application-level state that PBS backs up only indirectly.

Consider the layers:

Layer	What It Contains	Backup Tool	Recovery Speed
Hypervisor	VM disks, LXC rootfs, configs	PBS	Full VM restore in minutes
Kubernetes	PV data, etcd, secrets	Velero + MinIO	Namespace-level restore
GitOps	Manifests, Helm values, configs	Git (ArgoCD)	Re-sync from repo

PBS gives you the "bare metal to running VMs" recovery path. If a node dies, you restore the VMs to another node and they come up exactly as they were. But the Kubernetes workloads inside those VMs have their own state (persistent volumes, databases, application data) that benefits from Velero-level backups running in parallel.

The combination is what makes 3-2-1 actually work:

Three copies: live data + PBS backup + offsite copy (Synology, cloud bucket, second PBS instance)
Two media types: local SSD/NVMe (live) + HDD (PBS datastore)
One offsite: PBS supports built-in sync to a remote PBS instance, or you can replicate the datastore to a NAS for geographic separation

For the GitOps layer, ArgoCD already handles the "config as code" part. You don't need to back up Kubernetes manifests the traditional way because they're already in Git. What you need to back up is the state that isn't in Git: persistent volumes, database contents, secrets.

Garbage Collection and Datastore Maintenance

PBS deduplicates by storing data as content-addressed chunks. When you prune old backups, the chunks aren't immediately deleted. They become unreferenced. Garbage collection (GC) is the process that identifies and removes unreferenced chunks to reclaim disk space.

GC runs on a schedule within PBS. The default is usually fine, but keep an eye on the "Deduplication Factor" metric in the PBS dashboard. For a homelab with similar VMs (same base OS, similar packages), you'll typically see dedup factors between 3x and 8x. That means your backups are using 3-8x less space than the raw data size.

# Check datastore status including dedup factor
proxmox-backup-manager datastore list

# Manually trigger garbage collection
proxmox-backup-manager garbage-collection start main-store

If your dedup factor is close to 1x, something is off. Either your VMs have very little data in common (unlikely if they're running the same distro), or the chunk size configuration isn't optimal for your workload.

Monitoring Backup Health

PBS exposes metrics that you can pull into Grafana or any monitoring stack. The key things to watch:

Last backup timestamp per VM/LXC: if a backup hasn't run in 24+ hours, something is broken
Backup duration trends: a backup that used to take 10 minutes and now takes 60 suggests disk issues or unexpected data growth
Verification status: any failed verifications need immediate attention
Datastore usage: track the growth rate to predict when you'll need more storage

A simple monitoring approach is a cron job that checks for recent backups:

#!/bin/bash
# Check that every VM has a backup from the last 24 hours
CUTOFF=$(date -d '24 hours ago' +%s)

proxmox-backup-client list \
  --repository 'backup@pbs@10.0.0.50:main-store' \
  --output-format json | \
  jq -r '.[] | select(.backup_time < '$CUTOFF') | .backup_id' | \
  while read vm; do
    echo "WARNING: $vm has no backup in the last 24 hours"
  done

Lessons Learned

Test restores, not just backups. At least once a quarter, pick a VM and restore it to a temporary location. Verify it boots, verify the data is intact. A backup system you've never restored from is a hypothesis, not a strategy.

Privilege separation on API tokens is the silent killer. If your automated backups authenticate fine but return empty data or permission errors on operations, check privsep. This one issue probably accounts for half the "PBS isn't working" posts on the Proxmox forums.

Separate your failure domains. PBS running as a VM on the cluster it's backing up is better than no backups, but only barely. The whole point of backups is surviving hardware failure. A dedicated, physically separate PBS host (even a cheap one) fundamentally changes your recovery posture.

PBS handles the hypervisor layer, not the application layer. If you're running Kubernetes, you still need something like Velero for PV snapshots and namespace-level restores. PBS gives you "get back to running VMs." Velero gives you "get back to running applications." Both are necessary. Building a production homelab is only half the work if you don't have a plan for when things go wrong.

Deduplication makes aggressive retention policies cheap. Don't be stingy with retention. The marginal cost of keeping an extra month of weekly snapshots is tiny after dedup. The value of having that three-month-old snapshot when you discover slow data corruption is enormous.

Lock file issues are operational, not architectural. They're annoying, but they're just stale state from interrupted processes. Know where the lock files live, know how to check if a vzdump is actually running, and clean up when needed. Don't let a stuck lock file make you think PBS itself is broken.

When Agents Should Stop: Designing Safety Boundaries That Work

Guatu — Mon, 15 Jun 2026 12:38:16 +0000

An agent in my homelab posted "HEARTBEAT_OK" to the ops channel 47 times over one weekend. Every message was technically correct. The scheduled jobs were healthy, the agent verified them, and it reported in exactly like it was told to. By Monday morning I had muted the channel, which meant the one message that mattered (a failed backup verification) scrolled past unread sometime around 3 AM.

That incident wasn't an alignment problem or a runaway loop. It was a stopping problem. The agent had no concept of "nothing to say," so it said something every time it woke up. Most agent safety writing focuses on preventing harmful actions. In practice, the boundary I've had to engineer most carefully is more mundane: teaching agents when to do nothing and exit quietly.

If you run scheduled agents, autonomous loops, or anything where an LLM makes decisions on a timer, this is for you. The patterns below come from running multi-agent pipelines on my own infrastructure, and from the specific ways they've failed. I covered the theory in Three-Layer Safety for Autonomous Agents; this post is the operational follow-up, the part where theory meets a crash-looping gateway at 2 AM.

Stopping is a feature, not a failure state

The core mistake I made early on: treating an agent that stops as an agent that failed. My first orchestration scripts retried everything. Agent exits without completing the task? Retry. Agent says it's blocked? Rephrase the prompt and retry. The result was agents that burned tokens grinding against problems they'd already correctly identified as unsolvable from inside the loop.

What fixed it was giving agents a vocabulary for stopping. Mine boils down to three boundary types, all enforced outside the model:

Budget boundaries cap what an agent can spend: iterations, tokens, wall-clock time. These are the easy ones, and most frameworks give you something here. The mistake is setting them as emergency brakes (high enough that they never trigger) instead of as scoping decisions. If a task should take 3 iterations, cap it at 5, not 50. A cap that triggers at 50 means you've already wasted 45 iterations of spend before learning anything. I also set budgets per stage rather than per pipeline: a global 30-minute cap on a five-stage pipeline tells you nothing about which stage ran away.

Progress boundaries detect when the agent is still spending but no longer changing anything. This is the infinite-loop killer, and it's the one almost nobody implements. An agent can stay under every budget cap while making zero progress: rewriting the same file back and forth, re-running the same failing test with cosmetic tweaks. You detect this by hashing the observable state between iterations and stopping when the hash stops changing.

Reporting boundaries define when the agent is allowed to speak. This is the HEARTBEAT_OK lesson: an agent that reports success on every run trains humans to ignore it. Silence on success, noise on failure. The inversion matters more than it looks.

The configs

Progress detection is the highest-value boundary, so start there. The wrapper below runs an agent task in a loop and kills it when two consecutive iterations produce identical state:

#!/usr/bin/env bash
# agent-loop.sh: run an agent task with hard stop conditions
MAX_ITERATIONS=5
previous_state=""

for i in $(seq 1 "$MAX_ITERATIONS"); do
  run_agent_iteration "$TASK_FILE"   # your agent invocation here

  # Hash everything the agent can change: working tree + state dir
  current_state=$(
    { git diff; git status --porcelain; cat state/*.json 2>/dev/null; } \
    | sha256sum | cut -d' ' -f1
  )

  if [[ "$current_state" == "$previous_state" ]]; then
    echo "iteration $i produced no state change, stopping" >&2
    exit 2   # stopped at boundary, not failed
  fi
  previous_state="$current_state"

  task_complete && exit 0
done

echo "hit iteration cap ($MAX_ITERATIONS) without completing" >&2
exit 2

Exit code 2 is doing real work there. I use a three-value contract for every agent wrapper:

0 = done: task complete, verified
1 = failed: something broke, a human needs to look
2 = stopped: hit a boundary with partial progress, safe to resume

The distinction between 1 and 2 is the whole point. A failure pages someone. A boundary stop writes a state file and waits for the next scheduled run, which picks up where the last one left off. Collapsing those into one exit code gives you either alert fatigue or silent data loss, depending on which direction you collapse them.

Notice that all of this lives in the wrapper, not in the prompt. You can (and should) tell the agent about its budget in the prompt, because a model that knows it has two iterations left plans differently. But the prompt is advice. The wrapper is the boundary.

Reporting boundaries live in the scheduler config. Here's the shape I use for scheduled agent jobs after the heartbeat incident:

{
  "name": "nightly-health-check",
  "schedule": "0 6 * * *",
  "task": "Verify backup jobs completed and volumes are healthy.",
  "notify": {
    "on_success": "silent",
    "on_failure": "channel:#ops",
    "on_boundary_stop": "channel:#ops-low"
  },
  "deadman": "https://hc.example.com/ping/nightly-health"
}

Two things to notice. Success is silent: the channel only gets a message when something needs a human. And the deadman URL replaces the heartbeat message entirely: instead of the agent telling humans "I'm alive," it pings a dead-man's-switch endpoint (Healthchecks.io, or any self-hosted equivalent) that alerts only when the ping stops arriving. Machines are good at noticing absence. Humans are terrible at it. Route the liveness signal to the machine and the failure signal to the human.

Gotcha 1: silence can hide breakage

About a month after I made my agents quiet on success, a memory MCP server's tools started failing silently. Calls returned empty results instead of errors. The agents treated "no results" as "nothing to report" and exited cleanly, status 0, for eleven days. From the outside everything looked healthy: exit codes were green and the dead-man pings kept arriving, because the agent itself was running fine. Only the tools inside it were broken.

The lesson: "silence on success" requires verifying success, not just the absence of an exception. My health-check agents now end every run with an assertion phase that demands positive evidence:

# Don't trust "no errors". Demand proof of work.
results=$(query_memory_store "test-canary-record")
if [[ -z "$results" ]]; then
  echo "canary record missing: memory store is lying to us" >&2
  exit 1
fi

Plant a canary record you know exists, and fail loudly if the tooling can't find it. A tool that fails silently turns every downstream stop condition into a lie, because the agent is deciding "nothing to do" based on data it never received.

Gotcha 2: validate config before the gateway eats it

Stop conditions usually live in config files, which means they inherit every config-deployment failure mode. I learned this when I added a plausible-looking concurrency cap to an agent gateway's config. The key didn't exist in the schema. Older versions ignored unknown keys; the version I was running had switched to strict validation and rejected the whole file. The gateway crash-looped on restart, taking every scheduled agent down with it, including the ones whose job was to report that things were down.

Strict validation is the right behavior (a typo'd max_iteratons silently ignored is a budget cap that doesn't exist), but it means you treat agent config like any other production config: validate before reload, never after.

# Never restart a gateway on unvalidated config
agentctl validate --config /etc/agent/gateway.json || {
  echo "config invalid, refusing to restart" >&2
  exit 1
}
systemctl restart agent-gateway

If your agent platform ships a doctor or validate subcommand, wire it into the deploy path and make the restart conditional on it passing. If it doesn't ship one, a JSON Schema check in CI is twenty minutes of work and saves you a crash-looped orchestrator. Same idea as validating Kubernetes manifests before merge, just pointed at your agent stack.

Gotcha 3: a stopped agent must leave a note

Early versions of my boundary stops just exited. The next scheduled run started from scratch, re-derived the same context, hit the same boundary, and exited again. Functionally an infinite loop, just with a 24-hour period and a cron job in the middle.

Now every boundary stop writes a handoff file before exiting:

{
  "stopped_at": "2026-06-08T03:12:44Z",
  "reason": "no_progress",
  "iterations_used": 4,
  "progress_summary": "Identified failing PVC, replica rebuild blocked on node disk pressure",
  "blocking_on": "needs human: node disk cleanup or replica eviction",
  "resume_hint": "check node disk usage before retrying"
}

The next run reads the handoff first. If blocking_on names a human action and nothing in the environment has changed, it exits immediately at near-zero cost instead of re-deriving the same dead end. When the blocker clears, it resumes from the summary instead of from nothing. This one file turned boundary stops from an expensive pause into an actual checkpoint mechanism.

What I considered and rejected

Letting the model decide when to stop. Tempting, because the model often knows it's stuck. But a stop condition that lives inside the thing being bounded isn't a boundary, it's a suggestion. Models are also systematically optimistic that one more iteration will help. I let agents request an early stop (which short-circuits the loop), but enforcement stays in the wrapper, outside the model's reach.

Confidence thresholds. Some frameworks stop when the model's self-reported confidence drops below a cutoff. I tried it; self-reported confidence was noise, uncorrelated with whether the next iteration helped. The state-hash check costs one sha256sum and doesn't depend on the model grading its own homework.

Watchdog agents. A second agent that monitors the first and decides whether to kill it. This works, and for high-stakes pipelines I still use a reviewer stage (the pattern shows up in Multi-Agent AI Systems: Architecture Patterns That Actually Work). But as a stop mechanism it's expensive and introduces a new question: who stops the watchdog? Deterministic boundaries in the wrapper give you 90% of the value at roughly zero marginal cost.

Where this lands

Stopping is the cheapest safety mechanism you have, and it's the one most agent deployments skip because it doesn't feel like a feature. Nobody demos an agent exiting cleanly. But the boundaries above have prevented more incidents on my cluster than any prompt-engineering guardrail I've written: budget caps treated as scoping decisions instead of emergency brakes, state-hash progress detection, the 0/1/2 exit contract, silent success paired with loud failure and a machine-checked dead-man switch, and handoff files so a stop is a checkpoint instead of a discard.

Reach for this the moment any agent runs without a human watching: scheduled jobs, overnight batch pipelines, CI agents. Building agent systems that run unattended against real infrastructure is part of what I help teams with at GuatuLabs, and stop-condition design is reliably the piece nobody thought about before the first incident. If your ops channel has a recurring message in it right now that everyone has learned to scroll past, that's not a reporting feature. That's a stop condition nobody designed.

Network Policies with Calico: Default Deny and Namespace Isolation

Guatu — Mon, 15 Jun 2026 12:38:04 +0000

A default-deny NetworkPolicy is five lines of spec. Those five lines will also kill DNS resolution for every pod they select, because an egress deny blocks UDP packets to kube-dns just as happily as it blocks the traffic you were actually worried about. The distance between "I understand network policies" and "I rolled out default deny without an outage" is mostly three blind spots: DNS, your ingress controller, and admission webhooks.

Out of the box, Kubernetes runs a flat pod network. Every pod can open a connection to every other pod in the cluster, across namespaces, no questions asked. If you've already done the work of building least-privilege service accounts, a flat network is the same problem one layer down: identity is locked tight while the network is wide open. This post is about closing that gap with Calico on a bare-metal cluster (K8s 1.31, Calico 3.x), in an order that doesn't take the cluster down while you do it.

One prerequisite worth stating plainly: the NetworkPolicy API objects exist in every cluster, but they do nothing unless your CNI enforces them. Calico does. If you're on a CNI without policy support, you can apply these manifests all day and traffic flows anyway, which is its own special category of false confidence.

The rollout that looks right and isn't

The tempting approach goes like this: write one default-deny policy, template it across every namespace, apply, done. Security checkbox ticked before lunch.

Here's the policy everyone starts with:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: team-a
spec:
  podSelector: {}        # selects every pod in the namespace
  policyTypes:
    - Ingress
    - Egress

The empty podSelector selects all pods, and listing both policy types makes them isolated in both directions. Correct, minimal, and the moment it lands cluster-wide, three things break in a predictable order.

Failure one: DNS dies first, and it dies slowly

Every pod in a selected namespace loses the ability to resolve names, because queries to kube-dns in kube-system are egress traffic like any other. The nasty part is the failure mode. Connections to a denied endpoint fail fast with a timeout you'll notice. DNS failures look different: each lookup waits out a 5-second timeout per attempt, multiplied by the search domain list your ndots config generates. Apps get slow before they get broken, which sends you debugging application performance instead of network policy. I wrote about how the search domain expansion amplifies this in the ndots:5 post; default deny turns every one of those expanded lookups into a 5-second black hole.

Failure two: your ingress controller can't reach anything

Traffic from Traefik or ingress-nginx to your backend pods is just pod-to-pod traffic crossing a namespace boundary. Default deny on the application namespace blocks it, and every service behind the ingress starts returning 502s and 504s. The application pods are healthy, the Service endpoints are populated, readiness probes pass (kubelet probes come from the node, and Calico permits them). Everything looks green except the part where users reach it. This also bites cert-manager: an HTTP-01 challenge needs the ingress controller to reach the temporary solver pod, so default deny can silently stall certificate issuance long after the initial rollout.

Failure three: the webhook deadlock

This is the one that turns a degraded cluster into a stuck one. Admission webhooks (Kyverno, cert-manager's webhook, anything with a ValidatingWebhookConfiguration) receive calls from the API server. Deny ingress to the webhook pod and those calls time out. With failurePolicy: Fail, the API server now rejects the operations that webhook gates, and the trap closes: the NetworkPolicy you're trying to apply to fix the problem is itself an API operation that flows through admission. You're locked out of the fix by the thing you broke.

It gets worse if the policies are managed by automation. With a Kyverno generate rule or a GitOps controller syncing the policy, deleting the offending NetworkPolicy by hand buys you a few seconds before it's regenerated. You end up playing whack-a-mole against your own reconciliation loop while the cluster burns. The escape hatch is to pause the automation first (scale down Kyverno, disable ArgoCD auto-sync for that app), then remove the policy.

A detail that matters here: API server traffic to webhooks often originates from the control plane host network, not from a pod you can match with a podSelector. Allowing it means an ipBlock rule for your control plane CIDR, or excluding webhook namespaces from default deny entirely. I do the latter.

A rollout order that works

The fix for all three failures is the same discipline: never apply a deny you haven't already written the allows for, and never apply it wider than you can watch.

Step 1: one namespace, not the cluster

Pick a single application namespace with low blast radius. Resist the urge to start cluster-wide; the whole point of the first namespace is to discover the flows you forgot existed. kubectl get networkpolicy -A should stay boring while you learn.

Step 2: the baseline trio

Default deny ships as a set of three policies applied together, in one kubectl apply -f of one directory. The deny:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: team-a
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]

The DNS allow, which goes everywhere the deny goes, no exceptions:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: team-a
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Both protocols matter. DNS falls back to TCP for large responses, and an egress rule that only allows UDP produces intermittent failures that are miserable to track down.

The intra-namespace and ingress-controller allow:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-baseline-ingress
  namespace: team-a
spec:
  podSelector: {}
  policyTypes: [Ingress]
  ingress:
    # any pod in this same namespace
    - from:
        - podSelector: {}
    # everything in the ingress controller's namespace
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress

That kubernetes.io/metadata.name label is the load-bearing trick here. Since K8s 1.22, every namespace carries it automatically with its own name as the value, which gives you a stable way to select namespaces without inventing and maintaining your own labeling scheme.

With the trio applied, check behavior from inside the namespace before moving on:

# throwaway pod inside the locked-down namespace
kubectl -n team-a run probe --rm -it --image=busybox:1.36 --restart=Never -- sh
# inside the pod:
nslookup kubernetes.default                                  # should answer instantly
wget -qO- -T 2 http://api.team-b.svc.cluster.local           # should time out

Fast DNS plus a slow, eventually-failing cross-namespace connection is the signature of a healthy baseline. Instant DNS failure means the allow-dns policy didn't land; an instant cross-namespace success means the deny didn't.

Step 3: log before you deny

Calico's Log rule action is the visibility tool the vanilla NetworkPolicy API doesn't have. Before tightening further, I put a logging policy behind the allows so I can see what the deny is about to catch:

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: log-unmatched
spec:
  order: 4000                    # evaluated after everything else
  namespaceSelector: projectcalico.org/name == 'team-a'
  types: [Ingress, Egress]
  ingress:
    - action: Log
  egress:
    - action: Log

With the iptables dataplane, Log uses the kernel LOG target, so dropped-candidate packets show up in the kernel log with a calico-packet: prefix (configurable via logPrefix in FelixConfiguration):

journalctl -k --grep calico-packet

Two caveats. Kernel logging is noisy, so treat this as a diagnostic you enable for hours, not a permanent fixture. And the eBPF dataplane doesn't support the Log action, so if you've switched dataplanes this tool isn't available.

This step is where "set and forget" turns into something closer to auditing. Run a logging policy for a day against a namespace before enforcing, and you find the flows nobody documented: the metrics scraper, the backup job, the sidecar that phones a service in another namespace.

One class of flow deserves special mention: anything running with hostNetwork: true. Node-level monitoring agents and some bare-metal ingress deployments source their traffic from the node's IP, not a pod IP, so podSelector and namespaceSelector rules never match them. If scraping or health checks break only after enforcement, this is usually why, and the fix is an ipBlock rule covering your node CIDR rather than another selector you'll fight with.

Step 4: the cluster-wide backstop

Once the per-namespace pattern is proven, Calico's GlobalNetworkPolicy enforces namespace isolation as a guardrail across every tenant namespace at once, with infrastructure explicitly carved out:

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: tenant-isolation-backstop
spec:
  order: 3000
  namespaceSelector: >-
    projectcalico.org/name not in
    {"kube-system", "calico-system", "calico-apiserver",
     "ingress", "argocd", "cert-manager", "kyverno"}
  types: [Ingress, Egress]
  egress:
    # DNS keeps working even where namespace policies are missing
    - action: Allow
      protocol: UDP
      destination:
        selector: k8s-app == 'kube-dns'
        ports: [53]
    - action: Allow
      protocol: TCP
      destination:
        selector: k8s-app == 'kube-dns'
        ports: [53]

No explicit Deny rule, and that's deliberate. In Calico, when at least one policy selects an endpoint and no rule allows the packet, the packet is dropped at the end of evaluation. The backstop selects everything outside the exclusion list, allows DNS, and lets the implicit deny do the rest.

The order: 3000 is doing real work. Calico assigns Kubernetes NetworkPolicies an order of 1000, and lower order means earlier evaluation. An allow in a namespace's own policy terminates evaluation before the backstop is ever consulted. The backstop only catches traffic nothing else has claimed, which means namespaces with proper policies behave per their policies, and namespaces without any get isolation by default instead of the flat network.

That exclusion list is the "infrastructure exclusion" pattern, and I'd argue it's the single most important decision in the whole rollout. The namespaces that run your CNI, your ingress, your GitOps controller, and your admission webhooks are the namespaces where a policy mistake costs you the ability to fix policy mistakes. Leave them out of automated enforcement. Write their policies by hand, later, one at a time, with the logging step in between.

Step 5: automate generation, with the same exclusions

For new namespaces, a Kyverno generate rule stamps the baseline trio in automatically:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: generate-default-deny
spec:
  rules:
    - name: default-deny
      match:
        any:
          - resources:
              kinds: [Namespace]
      exclude:
        any:
          - resources:
              kinds: [Namespace]
              names: ["kube-system", "kube-public", "kube-node-lease",
                      "calico-system", "ingress", "argocd", "kyverno"]
      generate:
        apiVersion: networking.k8s.io/v1
        kind: NetworkPolicy
        name: default-deny
        namespace: "{{request.object.metadata.name}}"
        synchronize: true
        data:
          spec:
            podSelector: {}
            policyTypes: [Ingress, Egress]

Two operational notes. synchronize: true is what creates the regeneration loop from failure three: hand-deleting the generated policy gets it recreated within seconds, so during an incident you pause the ClusterPolicy before touching its output. And Kyverno treats generate rules as effectively immutable: if the generated resource definition is wrong, plan on deleting and recreating the ClusterPolicy rather than patching it in place.

Why this works

The mental model that makes all of this predictable: Kubernetes NetworkPolicies are additive allow-lists with an implicit deny that activates the moment any policy selects a pod. There is no deny rule in the vanilla API. A pod selected by zero policies accepts everything; a pod selected by any policy accepts only what the union of matching policies allows. That's why the baseline trio works as a set: the deny policy flips the pod into isolated mode, and the other two define the allowed surface.

Calico layers an ordered evaluation model on top. Policies are sorted by order, rules within a policy run top to bottom, and the first Allow or Deny terminates evaluation. Kubernetes-native policies slot in at order 1000 (you can see the converted versions with calicoctl get networkpolicy --all-namespaces, prefixed knp.default.). Pods matched by no policy at all fall through to Calico's per-namespace profiles, which default to allow. That layering is exactly what makes the backstop-at-3000 pattern safe: specific intent at 1000 wins, the guardrail catches the remainder, and the logging policy at 4000 sees only what's about to die.

Felix, Calico's per-node agent, also quietly saves you from the worst self-own. Its failsafe port list (SSH on 22, the API server on 6443, BGP on 179, etcd, Typha) is exempt from policy on host endpoints by default, so a bad policy can break your workloads without also locking you out of the nodes you need to fix it from. Don't shrink that list without a very specific reason.

Lessons learned

The failure modes are knowable in advance. DNS, ingress, and webhooks fail in that order every time, and writing the allows before the deny is cheaper in every way than discovering them from a monitoring graph. If a rollout plan doesn't mention kube-dns, port 53, or failurePolicy, it isn't done.

Namespace-by-namespace beats cluster-wide, even though it feels slower. The first namespace takes a day because you're discovering undocumented flows. The tenth takes ten minutes because there's nothing left to discover. Going cluster-wide first inverts that: you discover everything at once, in production, with automation re-applying the breakage faster than you can remove it.

Exclude infrastructure from automation permanently, not temporarily. Every system that can generate or sync policies (Kyverno, ArgoCD, your own scripts) should carry the same exclusion list for kube-system, the CNI namespace, ingress, GitOps, and webhook namespaces. The asymmetry is stark: a missing policy in those namespaces costs you some security posture, while a wrong policy there costs you the control plane's ability to accept the fix.

Logging is the difference between policy as guesswork and policy as engineering. The Log action is crude (kernel log lines, iptables dataplane only), but it converts "why is this connection failing" from a hypothesis into a grep. I'd take crude visibility over elegant blindness in any network debugging session. This pattern, restrict by default and watch the boundary, is the same shape as the guardrails I build around autonomous agent infrastructure: the deny is easy, and the engineering is in the observability that tells you what the deny will cost before you pay it.

The thing the docs undersell is that default deny is a migration, not a manifest. The YAML is trivial. The work is the inventory of flows your cluster actually depends on, and you only get that inventory by watching one namespace at a time with the logs on.

Velero + MinIO: Kubernetes Backup Strategy for Bare Metal

Guatu — Wed, 10 Jun 2026 12:15:14 +0000

I spent three hours staring at a PartiallyFailed status in Velero, wondering why my backups were failing despite the logs claiming the S3 connection was healthy. The culprit wasn't the network or the credentials. It was a handful of NFS-backed persistent volumes that Velero was trying to snapshot using a CSI driver that didn't support them.

If you're running Kubernetes on bare metal, you don't have the luxury of a "managed" backup service. You have to build the storage backend, the orchestration layer, and the recovery path yourself. Most of the documentation assumes you're pushing to AWS S3, but when you're running your own hardware, that's usually not the goal. You want your data on your own disks, under your own control.

The False Starts

My first attempt was naive. I thought I could just install Velero, point it at a MinIO instance running inside the same cluster, and call it a day. This was a mistake for two reasons.

First, backing up a cluster to a storage provider running inside that same cluster is a circular dependency. If the cluster goes down, your backups are gone. I quickly moved MinIO to a separate set of machines to ensure the backup target lived outside the blast radius of the Kubernetes API.

Second, I relied entirely on the "happy path" of CSI snapshots. I assumed that because I was using Longhorn for most of my stateful workloads, everything would just work. I forgot that I had a few legacy NFS mounts for shared configuration files. Velero tried to trigger a CSI snapshot on those NFS volumes, failed, and marked the entire backup as PartiallyFailed. I spent an hour chasing "S3 timeout" errors when the real issue was a storage class mismatch.

I also tried using the default Velero installation without specifying the S3 URL explicitly in the environment variables of the pod. I assumed the plugin would magically find MinIO if the credentials were correct. It didn't. I ended up with a loop of 403 Forbidden errors because Velero was trying to hit the actual AWS S3 endpoints instead of my local MinIO instance.

The Actual Solution

To get a reliable bare-metal backup strategy, you need three distinct layers: the S3-compatible target (MinIO), the orchestrator (Velero), and the control plane safety net (ETCD snapshots).

1. The Storage Backend (MinIO)

I run MinIO on a separate set of bare-metal nodes. For the sake of this setup, I've created a dedicated bucket called k8s-backups and a specific service account with read/write access to that bucket.

Running MinIO outside the cluster is non-negotiable. If you have a power failure on your K8s rack and your backups are on the same rack, you haven't built a backup system: you've just built a very expensive way to lose your data twice.

2. Installing Velero with MinIO

The trick here is the AWS plugin. Since MinIO uses the S3 API, we use the AWS provider but override the endpoint to point to the local MinIO server.

I used the following command to deploy Velero 1.14 on K8s 1.31:

velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.14.0 \
  --bucket k8s-backups \
  --secret-file ./credentials-velero \
  --use-volume-snapshots=true \
  --backup-destination-type=s3 \
  --s3-url http://minio.example.com:9000 \
  --namespace velero

The credentials-velero file is a standard AWS credentials format. To keep these secure and avoid committing them to Git, I use SealedSecrets to manage the secrets across my environments.

If you're deploying this via GitOps, I highly recommend using the official Helm chart but overriding the configuration.s3Url value. This ensures that when you scale your cluster or move nodes, the backup configuration remains consistent.

3. Handling the "PartiallyFailed" Nightmare

To stop Velero from trying to snapshot volumes that don't support it (like NFS), I had to be explicit. Labeling volumes to exclude them is a start, but the most effective way to handle a mixed-storage environment is to patch the backup schedule to ignore volume snapshots for specific workloads or to use Restic/Kopia for file-level backups.

If you have a schedule that keeps failing due to incompatible PVs, you can disable snapshot volumes for that specific schedule:

kubectl patch schedule daily-cluster-backup -n velero \
  --type=merge \
  -p '{"spec":{"template":{"snapshotVolumes":false}}}'

For the volumes that actually need backing up (like my Longhorn volumes), I rely on the Longhorn integration, which allows Velero to trigger native Longhorn snapshots.

4. The ETCD Safety Net

Velero is great for resources and PVs, but if your ETCD cluster completely collapses, you're in for a bad time. I don't trust a single tool for the control plane. I implemented a systemd timer on the control plane nodes to take raw ETCD snapshots every 24 hours.

I use this unit file to handle the snapshot and a basic retention policy:

[Unit]
Description=ETCD Snapshot Backup
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/bin/etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /var/backups/etcd/etcd-snapshot-$(date +%Y%m%d).db
ExecStartPost=/bin/sh -c '/usr/bin/find /var/backups/etcd -type f -name "etcd-snapshot-*.db" -mtime +7 -exec rm -f {} \;'

I then use a simple cron job to rsync these .db files to the MinIO server. This gives me a raw binary backup of the cluster state that is completely independent of the Velero operator.

Troubleshooting the Gap

When things go wrong with Velero and MinIO, the errors are rarely helpful. You'll see Backup failed in the high-level status, but the real gold is in the pod logs.

The "S3 Endpoint" Trap

If you see failed to get object: NoSuchBucket or 403 Forbidden despite having the right keys, check if Velero is actually hitting your MinIO server. Run:

kubectl logs -n velero deployment/velero

If you see requests going to s3.amazonaws.com, your --s3-url flag was ignored or overridden. This often happens when using Helm charts where the configuration block isn't properly mapped to the deployment arguments.

Restic Metadata Corruption

I hit a specific wall when I changed the bucket name in MinIO. I updated the Velero config, but my file-level backups (using Restic) started failing with:
error: repository is not initialized

Restic stores metadata in the bucket itself. If you move buckets, you can't just point Velero to the new one; you have to migrate the restic repository or re-initialize it. I learned the hard way that Restic is less flexible than CSI snapshots for backend migration.

CSI Snapshot Timeouts

In a multi-node Proxmox setup, I noticed that some backups would hang at the "snapshotting" phase. After digging into the Longhorn logs, I found that the snapshot was being created, but the CSI driver was timing out while waiting for the volume to reach a consistent state. The fix was increasing the snapshotTimeout in the Velero configuration to 10 minutes, giving the storage layer enough breathing room to finalize the snapshot on larger volumes.

Deep Dive: Why This Architecture Works

This architecture works because it acknowledges the reality of bare metal: things fail in ways the cloud hides from you.

By using MinIO as an S3-compatible layer, I get the industry-standard API that Velero expects, but I keep the data on my own hardware. This removes the egress costs and latency associated with pushing terabytes of snapshot data to a public cloud provider.

By separating the ETCD backups from the Velero backups, I've created two different recovery paths. If the Velero operator is broken, I can still restore ETCD to bring the API server back online. If the ETCD data is corrupted but the API is alive, I can use Velero to restore specific namespaces without nuking the entire cluster.

The decision to use snapshotVolumes: false on specific schedules is a pragmatic trade-off. I'd rather have a "successful" backup of my YAML manifests and secrets than a "partially failed" backup that tries (and fails) to snapshot a read-only NFS mount. I handle the NFS data separately via a simple tar and rsync pipeline.

Operational Lessons

If I were to do this again from scratch, I would change a few things:

Avoid MinIO in the same rack. I have my MinIO nodes in a different physical power circuit. If a PDU fails, I don't want my backup target to go dark at the same time as my cluster.
Use Kopia over Restic. Velero has started supporting Kopia, which is generally faster and handles deduplication more efficiently. If you're starting fresh, go with Kopia.
Automate Restore Tests. A backup is just a theoretical exercise until you've successfully restored it. I now run a monthly "fire drill" where I spin up a temporary single-node cluster and attempt to restore a single non-critical namespace from the MinIO bucket.

The biggest surprise was how much the "small things" matter. A missing s3-url flag or a slightly misconfigured systemd timer can be the difference between a 10-minute recovery and a weekend spent rebuilding a cluster from Git manifests.

For those building complex AI agent pipelines or industrial IoT systems, this level of redundancy is mandatory. When your agents are managing state across multiple databases and vector stores, a simple "git clone" of your manifests isn't a backup strategy. You need a consistent snapshot of the entire state, and Velero + MinIO is the most reliable way to achieve that on bare metal.

Agent Glass-Break Patterns: Controlled Escalation for Production

Guatu — Wed, 10 Jun 2026 10:15:27 +0000

I watched an autonomous ops agent attempt to "fix" a failing deployment by recursively deleting pods in a loop because it misinterpreted a CrashLoopBackOff as a transient networking glitch. The agent had the permissions to do it, the logic to justify it, and absolutely no circuit breaker to stop it from taking down the entire namespace. It was a classic case of giving a tool a hammer and watching it treat the entire infrastructure like a nail.

If you're running agents in production, you've probably realized that the standard "system prompt" safety is a joke. Telling an LLM "please be careful with the production database" is not a security boundary. You need a glass-break pattern: a way for agents to operate within a strict sandbox, but with a controlled, audited path to escalate privileges when a human approves it or a specific condition is met.

What I tried first

My first instinct was to lean on centralized identity. I tried routing every agent tool call through an Authentik-protected gateway. The idea was simple: the agent requests a tool, the gateway checks the session, and the action is authorized.

It was a nightmare. The latency added by the OIDC handshake for every single tool call made the agent feel sluggish, and the integration overhead for low-sensitivity observability tools was absurd. I spent more time debugging JWT expiration and redirect loops than actually building agent capabilities. I was treating a low-sensitivity internal tool like a public-facing enterprise application.

Then I tried the "Super-User" approach. I gave the agent a high-privilege service account but wrapped it in a complex set of Python decorators that checked for "safe" keywords in the arguments. This failed immediately. LLMs are too good at prompt injection and parameter manipulation. A simple --force flag or a clever string concatenation bypassed my "safety" filters in minutes.

The Actual Solution: Controlled Escalation

The fix was to move the security boundary from the application layer to the infrastructure and execution layer. I implemented a three-pronged approach: Network-level isolation for internal tools, safeBins for execution control, and a manual escalation trigger.

1. Infrastructure-Level Isolation

Instead of forcing every internal tool through a heavy auth layer, I shifted to a LAN-only access model using Kubernetes NetworkPolicy. This ensures that only the agent orchestrator can talk to the tool, and only from a specific subnet.

For a tool like Agent Quest, I stripped out the Authentik dependency and locked it down at the pod level:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: traefik-allow-egress-to-agentquest
spec:
  podSelector:
    matchLabels:
      app: traefik
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 10.0.0.140/32 # The specific IP of the Agent Quest service
    ports:
    - protocol: TCP
      port: 4444

This removes the auth overhead while ensuring that no one outside the cluster (or even in other namespaces) can trigger the tool. It aligns with the privacy-routed inference pattern of keeping sensitive traffic off the open wire.

2. Execution Control with safeBins

For tools that actually execute shell commands, like mcporter, I stopped relying on regex filters. I implemented a safeBins pattern. This is essentially an allowlist of binaries and the specific flags they are permitted to use.

If the agent tries to pass a flag not in the allowedValueFlags list, the execution engine kills the process before it ever hits the shell.

{
  "safeBins": {
    "mcporter": {
      "allowedValueFlags": ["--config", "--timeout"],
      "forbiddenFlags": ["--force", "--recursive", "--delete-all"]
    },
    "kubectl": {
      "allowedValueFlags": ["--dry-run=client", "-n"],
      "restrictedCommands": ["delete", "patch"]
    }
  }
}

This forces the agent to operate in a "read-only" or "safe-write" mode by default. If the agent needs to do something destructive, it cannot simply "decide" to do it; it must trigger the glass-break.

3. The Glass-Break Escalation

When the agent hits a safeBins restriction or a NetworkPolicy block, it triggers an escalation event. I integrated this with an n8n workflow that sends a Slack notification to me with the exact command the agent wants to run and the reasoning behind it.

The workflow looks like this:

Agent fails a safeBins check.
The error is caught by the orchestrator and pushed to an n8n webhook.
n8n sends a message: "Agent X wants to run mcporter --force. Reason: 'Pod is stuck in Terminating'. Approve?"
I click "Approve," which updates a temporary Redis key granting the agent a 5-minute window of escalated privileges.

Why it works

This works because it acknowledges that the LLM is an unreliable narrator. You cannot trust the agent to follow safety guidelines, but you can trust the Linux kernel and the Kubernetes API.

By moving the constraints to the binary level (safeBins) and the network level (NetworkPolicy), we create a hard boundary. The agent can hallucinate all it wants, but it cannot execute a --force flag if the execution wrapper doesn't allow it.

Combining this with the two-tier service account model ensures that even if the agent escalates, it's using a token with a strictly defined TTL. The "glass-break" isn't just a permission change; it's a temporary shift in the security posture of the system.

For the MSAM (Model State Management) integration, I had to rewrite the server-side tools using FastMCP to support this. I used a specific IngressRoute to ensure that the escalation triggers only came from trusted internal IPs:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: msam-ingress
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`msam.example.com`)
      kind: Rule
      services:
        - name: msam-service
          port: 443
          tls:
            insecureSkipVerify: true

Lessons Learned

The biggest surprise was how much the agent actually prefers these constraints. When the agent knows exactly what the boundaries are (because the error messages from safeBins are explicit), it stops trying to guess and starts asking for help. It turns a "failure" into a "collaboration."

If I were doing this again, I'd automate the memory index rebuilds more aggressively. I found that when I escalated an agent to fix a model registry mismatch (like the OpenClaw v2026.3.12 issue where codex-5.4 wasn't recognized), the agent often forgot that it had already tried a specific fix. I had to implement a rebuild-memory-index.py script to ensure the agent's long-term memory was synced with the actual state of the registry after a glass-break event.

A few caveats:

Latency: The human-in-the-loop part of the glass-break is a bottleneck. If you're in a high-availability environment, you'll need to define "Auto-Escalation" rules for low-risk tasks.
Complexity: You're adding a layer of middleware between the agent and the tool. If your middleware crashes, your agent is blind. I run my orchestration layer with a strict Recreate strategy on Kubernetes to avoid the split-brain issues I've seen with Ollama deployments.

Ultimately, production AI isn't about building the smartest agent. It's about building the most reliable cage for that agent to live in. The glass-break pattern allows the agent to be useful without giving it the keys to the kingdom.

Grafana Dashboards: Information Density vs Readability

Guatu — Mon, 08 Jun 2026 10:15:13 +0000

I spent three hours staring at a "Global Infrastructure" dashboard that took 12 seconds to load, only to realize I couldn't actually tell if my GPU nodes were throttling. I had roughly 40 panels on a single page, ranging from CPU steal percentages to disk IOPS and temperature sensors. It looked like a NASA control room, but it functioned like a legacy database query from 1998.

If you're managing a multi-node cluster or a complex AI pipeline, the temptation is to put every single metric you can possibly scrape into one view. The logic is: "If it's on the screen, I can't miss it." In reality, when everything is highlighted, nothing is. You end up with a dashboard that is visually noisy and computationally expensive.

The Performance Wall

Most people treat Grafana like a static webpage, but every panel is a live query. If you have 40 panels, you're hitting your Prometheus or VictoriaMetrics instance with 40 separate requests every time you refresh the page or change the time range.

Grafana has internal concurrency limits. It doesn't just fire all 40 queries at once; it batches them. When you hit a certain density, you start seeing the "loading" spinners stagger. You'll see the top row pop in, then a three-second gap, then the middle row. This isn't just an annoyance. It's a signal that your dashboard design is fighting the underlying data source.

I've seen this happen most often when people deploy a "thorough" community dashboard from a JSON export without pruning it. You get a beautiful layout, but it's querying metrics you don't even have exporters for, leading to a sea of "No Data" panels that still cost query time.

Information Density vs. Cognitive Load

There is a difference between a "dense" dashboard and a "cluttered" one.

A dense dashboard uses a high ratio of data to pixels. It uses small, efficient visualizations (like Stat panels or Gauges) to show current state, and reserves large Time Series panels for trends.

A cluttered dashboard is just a collection of every graph the engineer thought was "interesting" at the time of creation.

The goal is to reduce the time between looking at the screen and understanding the state of the system. If I have to squint to see if a line is crossing a threshold because there are six other lines in the same color palette, the dashboard has failed.

The Solution: Hierarchical Monitoring

Instead of one "God Dashboard," I moved to a three-tier hierarchy. This separates the "Is it broken?" view from the "Why is it broken?" view.

Tier 1: The Heartbeat (High Density, Low Detail)

This is a single screen. No time series graphs. Only Stat panels and Gauges.

Goal: Binary state. Green = OK, Red = Action Required.
Metrics: Cluster-wide CPU/RAM usage, number of Pending pods, GPU temperature peaks, and API latency.
Behavior: I keep this on a wall monitor. I don't want to see the "wiggle" of a graph; I want to see a red box if a node disappears.

Tier 2: The Service View (Medium Density)

This is where I use variables to filter by namespace or node.

Goal: Identify the specific component failing.
Metrics: Per-pod memory usage, network throughput, and request rates.
Behavior: I use Grafana variables ($node, $namespace) so that one dashboard template serves 20 different services.

Tier 3: The Deep Dive (Low Density, High Detail)

These are specialized dashboards for specific hardware or software.

Goal: Root cause analysis.
Metrics: GPU SM clock speeds, PCIe bus errors, or Longhorn volume replication lag.
Behavior: I only open these when Tier 1 or Tier 2 tells me something is wrong.

Implementing the Architecture

To make this work without manual overhead, I use a combination of Prometheus ServiceMonitors for auto-discovery and ConfigMaps for dashboard versioning.

If you're running GPUs, you shouldn't be manually adding every GPU to a dashboard. Use the nvidia-gpu-exporter and let Prometheus handle the labels.

Here is how I deploy the exporter to ensure the metrics are clean and available for the hierarchical dashboards:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-gpu-exporter
spec:
  selector:
    matchLabels:
      app: nvidia-gpu-exporter
  template:
    spec:
      containers:
        - name: nvidia-gpu-exporter
          image: ghcr.io/your-org/nvidia-gpu-exporter:v1.4.1
          ports:
            - containerPort: 9835
          env:
            - name: NVIDIA_VISIBLE_DEVICES
              value: all
      runtimeClassName: nvidia

To avoid the "manual update" nightmare, I store my dashboard JSONs in Git and deploy them via ConfigMaps. This allows me to prune unnecessary panels across the entire cluster at once.

apiVersion: v1
kind: ConfigMap
metadata:
  name: gpu-monitoring-dashboard
  labels:
    grafana_dashboard: "1"
data:
  dashboard.json: |
    {
      "id": null,
      "title": "GPU Health - Tier 2",
      "panels": [
        {
          "type": "stat",
          "title": "GPU Memory Usage",
          "datasource": "Prometheus",
          "targets": [
            {
              "expr": "sum(dcgm_fb_used) by (instance)"
            }
          ]
        },
        {
          "type": "timeseries",
          "title": "GPU Temperature Trend",
          "targets": [
            {
              "expr": "dcgm_temp"
            }
          ]
        }
      ]
    }

And to ensure Prometheus is actually picking up these metrics without me having to hardcode IPs, I use a ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: nvidia-gpu-exporter
  labels:
    release: monitoring
spec:
  selector:
    matchLabels:
      app: nvidia-gpu-exporter
  endpoints:
    - port: metrics
      interval: 30s

The Gotchas of High-Density Design

Even with a hierarchy, there are a few traps I fell into.

The "Too Many Variables" Trap

I once built a dashboard with six different dropdown variables (Cluster, Namespace, Pod, Container, Disk, GPU). Every time I changed one, Grafana had to re-evaluate every single panel. It felt like the browser was hanging.
The Fix: Limit your top-level variables. Use "chained" variables where the Pod dropdown only shows pods for the selected Namespace.

The Color Palette Problem

When you have 10 lines on one graph, Grafana's default colors start to repeat or become indistinguishable.
The Fix: Use "Overwrites" in the panel settings. Explicitly map a specific metric (e.g., node_cpu_seconds_total{mode="iowait"}) to a specific color like bright orange. This removes the cognitive load of checking the legend every five seconds.

The Refresh Rate Death Spiral

Setting a dashboard to "Auto-refresh: 5s" with 30 panels is a great way to DOS your own Prometheus instance.
The Fix: Tier 1 (Heartbeat) can refresh every 10-15 seconds. Tier 3 (Deep Dive) should be manual. There is no reason to auto-refresh a detailed GPU memory leak analysis every few seconds.

Lessons Learned

The most important thing I learned is that a dashboard is a tool for decision-making, not a data dump.

If you can't look at a dashboard for 5 seconds and tell me exactly what is wrong, it's too dense. I've spent too much time building "cool" dashboards that were useless in a 3 AM outage because I had to hunt through 15 panels to find the one metric that actually mattered.

I've applied this same philosophy to my other infrastructure. For example, when dealing with Longhorn volume health, I stopped trying to track every single replica's sync state on one page. Instead, I created a "Health Score" (a single Stat panel) that only turns red when the aggregate health of the volume drops below 100%.

If you're building out your own monitoring, start with the "Heartbeat" view. Ask yourself: "What is the one number that tells me I need to wake up?" Build that first. Everything else is just a deep dive for when things actually break.

For those managing high-performance AI workloads, this becomes even more critical. Monitoring GPU power states and memory fragmentation requires a different level of granularity than monitoring a web server. If you're struggling to balance the noise of bare-metal Kubernetes with the need for precision, I've dealt with these exact trade-offs in my infrastructure consulting.

Stop adding panels. Start deleting them.

Edge Computing for IIoT: When to Process at the Source

Guatu — Fri, 05 Jun 2026 16:15:13 +0000

My first attempt at a remote vibration monitoring system ended with a network switch that couldn't handle the throughput and a cloud bill that made me question my life choices. I was streaming raw high-frequency accelerometer data from several machines directly to a central cluster, thinking that "centralized visibility" was the gold standard. It wasn't. I had created a massive bottleneck where a 100ms network spike would cause gaps in the data, making it impossible to detect the very transient faults I was looking for.

If you're building industrial systems, the temptation is to push everything to a central dashboard as fast as possible. But in IIoT, the distance between the sensor and the compute is where most projects fail. You either drown in noise or you lose the signal because the network dropped a packet.

I spent a few months thinking that more bandwidth was the answer. I upgraded switches, tweaked MTU settings, and tried to optimize the MQTT payloads. I assumed the problem was the pipe. The reality was that I was trying to move the mountain to the geologist instead of just sending the geologist to the mountain.

The shift happened when I stopped treating the edge as a "dumb relay" and started treating it as a first-class compute node. I moved the FFT (Fast Fourier Transform) and initial anomaly detection to the source. Instead of sending 10kHz of raw voltage, I started sending a health score and a set of peak frequencies every few seconds.

The Architecture: Local Inference and the Privacy Hard-Wall

Once I moved basic signal processing to the edge, the next challenge was intelligence. I wanted an operator to be able to ask a local terminal, "Why is the XYZ-7000 vibrating?" without that query, and the sensitive machine telemetry attached to it, leaving the factory floor.

This is where the "privacy hard-wall" comes in. I implemented a system where the edge node handles the data synthesis and uses a local LLM to generate the answer. The raw telemetry never leaves the local subnet; only the synthesized natural language answer goes to the central log.

For this to work, I had to move away from the "cloud-first" mindset. I deployed local inference on the edge nodes using Ollama, but I quickly hit a wall with model capability. I tried qwen2.5:14b-instruct for tool calling to fetch documentation and real-time stats. It failed miserably. It would hallucinate flags, forget the JSON structure, or simply loop.

I found that for reliable tool calling in an industrial context, where a wrong command could theoretically trigger a physical action or a security breach, you need a larger context window and better reasoning. I bumped the requirements to qwen3:30b (or equivalent) as the minimum for any node handling autonomous tool orchestration.

Implementation: Securing the Edge Agent

If you're putting an AI agent at the edge to interact with industrial hardware, you cannot give it a raw shell. You need a strict allowlist and a way to ensure that the model doesn't accidentally execute rm -rf / because it misinterpreted a "cleanup" request.

I use a configuration-driven approach for tool restriction. In my openclaw.json (or similar agent config), I define safeBinProfiles. This ensures the agent can only use specific flags for specific binaries.

{
  "safeBinProfiles": {
    "knowledge.sh": {
      "minPositional": 0,
      "maxPositional": 2,
      "allowedValueFlags": ["--query", "--list"],
      "deniedFlags": ["--raw", "--export"]
    }
  }
}

By denying --raw and --export, I prevent the agent from dumping the entire local knowledge base into the chat context, which is a primary vector for data exfiltration.

Another practical hurdle was PATH resolution. I noticed the agent would often fail to call tools because it didn't have the full environment context of my user shell. The allowlist would reject the call because the binary wasn't in a "trusted" directory. I solved this by symlinking my industrial toolset into a dedicated, read-only bin directory.

# Create a trusted bin directory for the agent
sudo mkdir -p /opt/iiot-tools/bin

# Symlink the specific tool to ensure PATH resolution passes the allowlist
sudo ln -s /home/operator/scripts/knowledge.sh /opt/iiot-tools/bin/knowledge.sh

# Update the agent's environment to point here
export PATH="/opt/iiot-tools/bin:$PATH"

Routing and Fallbacks

In a production environment, hardware fails. If the GPU on the edge node dies, you can't just have the system stop working. However, you also can't just failover to GPT-4, because that violates the privacy hard-wall I mentioned earlier.

I implemented a tiered fallback strategy. If the primary high-performance model (running on a dedicated GPU) is unavailable, the system falls back to a smaller, CPU-bound model on the same node.

{
  "model.fallbacks": [
    "ollama/qwen3:30b", 
    "ollama/qwen2.5:14b-instruct", 
    "ollama/phi3:mini"
  ]
}

The trade-off here is that the phi3:mini fallback won't be able to do complex tool calling. I handle this by having the agent detect which model is currently active. If it's on a fallback model, it switches from "Autonomous Mode" (tool calling) to "Read-Only Mode" (answering based on cached data).

For the actual data retrieval, I use a query-based system rather than a search-based system. Instead of letting the LLM search through files, I use a wrapper script:

# The agent calls this instead of reading files directly
knowledge.sh query "What is the warranty period for the XYZ-7000?"

This script handles the RAG (Retrieval-Augmented Generation) internally and returns a synthesized answer. This keeps the raw documents hidden from the LLM's direct sight, adding another layer of security. This approach is similar to how I handle Privacy-Routed LLM Inference in my other projects.

Why This Works

The reason this beats the "cloud-central" approach is simple: physics.

Latency: Processing a vibration spike at the edge takes microseconds. Sending it to the cloud, waiting for a trigger, and sending a command back takes hundreds of milliseconds. In a CNC machine, that's the difference between a controlled stop and a broken tool.
Bandwidth: A single 3-axis accelerometer sampling at 20kHz generates a massive amount of data. By performing the FFT at the source, I reduce the data footprint by 99%, sending only the magnitudes of the significant frequency bins.
Security: By keeping the "intelligence" local, the attack surface is limited to the local network. There's no API key sitting in a cloud environment that can be leaked to grant access to the factory floor.

This architecture also makes Condition-Based Maintenance actually viable. You can't do true condition-based maintenance if your "condition" is dependent on the stability of your WAN connection.

Lessons Learned

If I had to do this again, I'd spend more time on the hardware abstraction layer. I spent too long writing scripts for specific sensor models. I should have implemented a standardized data format (like Sparkplug B) from day one.

I also learned that "Edge" is a spectrum. Some things belong on the microcontroller (interrupts, basic filtering), some on the gateway (FFT, local LLM routing), and some in the cluster (long-term trend analysis, fleet-wide health scoring). Trying to put everything on the gateway just creates a different kind of bottleneck.

The biggest surprise was the model capability gap. I really thought the 14B models would be enough for simple tool calling. They aren't. If you're building an agent that actually controls things or fetches critical data, don't skimp on the VRAM. Get the 30B+ models or you'll spend more time debugging hallucinations than actually monitoring your equipment.

Finally, the "privacy hard-wall" isn't just about security, it's about trust. Operators are hesitant to use AI tools when they think their every mistake is being uploaded to a corporate cloud for review. When they know the data stays on the machine, they actually use the tools.

This local-first approach is what allows for a clean Equipment Health Score. Instead of a dashboard with 500 blinking lights, the edge node calculates the score locally and sends one single integer to the cloud. The operator sees a "72," knows it's trending down, and asks the local agent for the reason—all without a single packet of raw telemetry ever leaving the building.