DEV Community: Guatu

Your Vector DB Snapshots Are Landing on the Same Disk That Will Fail

Guatu — Wed, 15 Jul 2026 16:15:47 +0000

Qdrant's default snapshot path is ./snapshots, relative to the storage directory. That one config default means every snapshot you create through the API lands on the same filesystem, same volume, same physical SSD as the live collection data. You run POST /collections/memory/snapshots, you get a 200 back, you feel responsible and safe. You are neither. When that disk fails, the snapshot dies in the same instant as the data it was supposed to protect.

What I expected

A snapshot API implies a safety mechanism. The mental model most of us carry is: live data is fragile, snapshots are the recovery path, so by definition they must live somewhere safer. Proxmox reinforces this instinct because VM snapshots and backups feel like separate things, and Proxmox Backup Server actually does ship your VM backups to a different box.

Vector databases don't work that way out of the box. Qdrant, Chroma, and Milvus all treat snapshots as a serialization feature, not a disaster recovery feature. The snapshot endpoint answers the question "give me a portable copy of this collection" and writes the answer to local disk. Where that local disk lives is your problem.

The expectation gap is worse in a homelab because the deployment is usually one container on one node with one bind mount:

# The setup that feels fine until it isn't
services:
  qdrant:
    image: qdrant/qdrant:v1.13.4
    volumes:
      - /mnt/nvme0/qdrant:/qdrant/storage
    ports:
      - "6333:6333"

Everything under /qdrant/storage sits on nvme0. Collections, WAL, and the snapshots/ directory the API writes into. One mount point, one blast radius.

What actually happens

Two separate failures hide in this setup, and the second one is sneakier than the first.

Failure one: physical co-location

Consumer NVMe drives in homelabs run hot, run constantly, and accumulate writes fast when a vector DB is doing HNSW index maintenance and WAL flushes underneath an agent that stores memories on every session. When the drive goes (wear-out, a flaky controller, or the classic SATA cable that worked fine for two years), you lose the collection and every snapshot of it simultaneously.

The trap is that snapshots make you stop worrying. You've automated a nightly snapshot job, the API returns success, ls shows a growing list of .snapshot files. Every observable signal says "protected." The only signal that says otherwise is df output showing that the snapshots share a device with the data, and nothing ever prompts you to look at that. It's the same gap I wrote about with Longhorn volume health: the system reports healthy because it's checking the wrong property. A snapshot that exists is not the same as a snapshot that survives the failure you're snapshotting against.

Failure two: logical incompatibility

Suppose you dodge the physical problem. You copy snapshots to your NAS, the disk fails, you rebuild the node, and you restore. The restore succeeds. Then your MCP server starts throwing errors on every search:

UnexpectedResponse: Status code 400 (Bad Request)
"Wrong input: Not existing vector name error: fast-nomic-embed-text-v1.5-q"

This is the second trap: the snapshot faithfully preserved a schema your tooling no longer speaks. mcp-server-qdrant moved from unnamed (default) vectors to named vectors around v0.6.0. Collections created by older versions store points with a bare vector; newer versions query by a specific vector name derived from the embedding model, like fast-nomic-embed-text-v1.5-q. If you run the server with uvx, which pulls the latest release on invocation, the client upgrades itself silently while your data stays frozen at the old schema. The restored collection is byte-perfect and functionally useless.

The difference in the collection config looks like this:

// Old collection: unnamed default vector
{ "params": { "vectors": { "size": 768, "distance": "Cosine" } } }

// What the current client expects: named vector
{ "params": { "vectors": {
    "fast-nomic-embed-text-v1.5-q": { "size": 768, "distance": "Cosine" }
} } }

Qdrant has no in-place rename for this. You can't patch a collection from unnamed to named vectors. The only path is a migration: read everything out, write everything into a new collection with the right schema.

So the full failure mode is a chain. Snapshots co-located with data cover you against nothing physical. Snapshots moved off-disk cover you against hardware but not against schema drift. Real recovery needs both physical separation and a tested restore path against the current client version.

The fix

Three layers, in order of how much pain they remove.

1. Put snapshots on a different device

Qdrant exposes the snapshot path as config. Point it at a mount backed by a different physical disk, or better, network storage:

services:
  qdrant:
    image: qdrant/qdrant:v1.13.4
    environment:
      - QDRANT__STORAGE__SNAPSHOTS_PATH=/qdrant/snapshots
    volumes:
      - /mnt/nvme0/qdrant:/qdrant/storage        # live data: fast local NVMe
      - /mnt/nas/qdrant-snapshots:/qdrant/snapshots  # snapshots: NFS to the NAS

One caveat: Qdrant builds snapshots with hard links where possible and needs a writable temp area, so snapshot creation over NFS is slower and briefly doubles disk usage during packaging. For collections in the single-digit-gigabyte range typical of agent memory, this is a non-issue. If it becomes one, snapshot to a second local disk and rsync from there.

If you're on Kubernetes with Longhorn instead of a single container, the equivalent move is a separate PVC on a different storage class (or at minimum different replica placement) for the snapshot path. Same principle: the recovery artifact must not share a failure domain with the thing it recovers.

2. Ship snapshots off the node entirely

A different disk in the same machine still shares a power supply, a mainboard, and your tendency to do risky things to that machine at 11pm. Get snapshots off the node:

#!/usr/bin/env bash
# nightly-qdrant-snapshot.sh
set -euo pipefail

COLLECTION="memory"
QDRANT="http://10.0.0.50:6333"

# Trigger snapshot, capture the generated name
SNAP=$(curl -sf -X POST "$QDRANT/collections/$COLLECTION/snapshots" \
  | jq -r '.result.name')

# Pull it over HTTP and push to object storage
curl -sf "$QDRANT/collections/$COLLECTION/snapshots/$SNAP" \
  -o "/tmp/$SNAP"
rclone copy "/tmp/$SNAP" "minio:qdrant-snapshots/$COLLECTION/"
rm "/tmp/$SNAP"

# Keep the node's snapshot dir from filling up
curl -sf -X DELETE "$QDRANT/collections/$COLLECTION/snapshots/$SNAP"

The detail worth noticing: Qdrant lets you download snapshots over the REST API, so the copy job doesn't need filesystem access to the Qdrant host at all. It can run anywhere that can reach port 6333, which makes it trivial to run as a Kubernetes CronJob or a container on a different node. Restore works the same way in reverse, with PUT /collections/{name}/snapshots/recover accepting a URL.

3. Handle the schema, not just the bytes

For the named-vector migration, snapshots don't help. You need to move the data through the API so it gets re-indexed under the new vector name. The scroll endpoint does the export side:

from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct

client = QdrantClient(url="http://10.0.0.50:6333")
VECTOR_NAME = "fast-nomic-embed-text-v1.5-q"

client.create_collection(
    collection_name="memory-v2",
    vectors_config={VECTOR_NAME: VectorParams(size=768, distance=Distance.COSINE)},
)

offset = None
while True:
    points, offset = client.scroll(
        collection_name="memory",
        limit=100,
        with_payload=True,
        with_vectors=True,   # omitting this silently drops the vectors
        offset=offset,
    )
    client.upsert(
        collection_name="memory-v2",
        points=[
            PointStruct(id=p.id, payload=p.payload, vector={VECTOR_NAME: p.vector})
            for p in points
        ],
    )
    if offset is None:
        break

This works when the embedding model hasn't changed, only the vector naming. If the model changed too (different dimensions, different quantization), the vectors themselves are stale and you have to re-embed from the payloads instead of copying vectors across. That's the strongest argument for always storing the source text in the payload: payloads are the durable asset, vectors are a derived cache you can rebuild.

Afterward, verify the schema instead of assuming:

curl -s http://10.0.0.50:6333/collections/memory-v2 \
  | jq '.result.config.params.vectors | keys'
# ["fast-nomic-embed-text-v1.5-q"]  <- named vector present, client will be happy

And pin your MCP server version so the client can't drift out from under the data again. uvx mcp-server-qdrant pulls latest; uvx mcp-server-qdrant==0.7.1 pulls what you tested against. Upgrade deliberately, with a migration plan, not as a side effect of a session starting.

Why this matters

For a lot of homelab services, losing a disk means re-running an Ansible playbook and moving on. Vector databases are different because the data is accumulated, not declared. If your agents write session learnings, decisions, and episodic memory into Qdrant (the pattern behind my Karpathy-style LLM wiki and the recall layer in vector search vs activation-based memory), then months of that accumulation lives in one collection. There's no Git repo to re-clone. The GitOps reflex of "the cluster is disposable, the repo is truth" fails exactly here, because the vector DB is the truth for that data. Anyone building agent memory systems for real workloads (something I help teams with) hits this eventually: the memory layer quietly becomes the most irreplaceable state in the whole stack, while getting the least backup attention.

The general lessons:

Check where your snapshots physically land, today. Not where you think they land. df -h on the snapshot directory, compare the device against the data directory. This takes thirty seconds and is the whole first half of this post.

A backup you haven't restored is a hypothesis. The 400 error above only surfaces when you actually recover a snapshot and run the current client against it. A quarterly restore drill into a scratch collection would catch schema drift months before it matters. Nobody does quarterly restore drills for their homelab vector DB, which is why pinning client versions is the practical substitute.

Snapshots and backups answer different questions. A snapshot answers "give me a copy of this collection's current state." A backup answers "can I get my data back after the worst realistic failure." The first is a feature the database gives you. The second is a property of your whole setup, and no single API call provides it.

Payloads outlive vectors. Embedding models change, quantization schemes change, vector naming conventions change. The source text in your payloads survives all of it. Design your collections so that a full re-embed from payloads is always possible, and half of these migration problems downgrade from "data loss" to "an afternoon of compute."

The fix costs one config line and a cron job. The failure costs every memory your agents ever stored. That's about as lopsided as risk math gets.

Eviction Without Deletion: Running an ACT-R Decay Policy for Agent Memory

Guatu — Wed, 15 Jul 2026 14:15:48 +0000

An agent's recall gets worse as its memory grows. Not slower, worse. Past a certain store size, top-K retrieval starts handing back memories that are semantically near the query and useless for the task, because a vector index has no idea whether something is relevant now or just looked relevant six weeks ago. The store fills up, the noise floor rises, and the agent quietly gets dumber while your dashboard says "memory: healthy."

If you run a long-lived agent with a persistent memory store, anything backed by a vector DB you keep appending to, this is coming for you. The append-only log feels safe. You never lose anything. But "never lose anything" and "always retrieve the right thing" pull in opposite directions, and past a few thousand entries the second one loses. I wrote about the theory side of this in Cognitive Memory for Agents: Vector Search vs Activation-Based Recall; this post is the production plumbing for the same idea.

The thesis is simple to state and annoying to accept: you should evict memories from the active retrieval path without deleting them. Reduce their activation so they stop surfacing, but keep the row so a future query can bring them back. Cognitive scientists have a model for exactly this, and it's older than most of the databases you'd run it on.

What deletion actually costs you

My first instinct was the obvious one. The store is too big, so prune it. I reached for the three standard tools and each one failed in its own way.

TTL expiry was first. Delete anything older than N days. This is fine for genuine ephemera and a disaster for everything else, because "old" is not the same as "irrelevant." An agent debugging a rare storage failure wants the note it wrote about that failure four months ago, precisely because the failure is rare. TTL deletes exactly the memories whose value is that they're old and specific. Once the row is gone, the agent can't re-learn it from retrieval. It's just gone.

LRU eviction was next, the classic cache move: drop the least-recently-used entries when you hit a size cap. Better, because at least it keys on access instead of raw age. It still hard-deletes, though, and it has no notion of importance. A memory accessed once but load-bearing (a security constraint, a hard-won config fix) gets evicted the same as idle chatter, just because nothing happened to touch it this week. LRU treats a critical fact and a stale log line identically.

Size caps were the third and worst. "Keep the newest 5,000 memories" turns your agent into a goldfish with a fixed horizon. The moment the store saturates, every new memory silently costs you an old one, and you have no control over which. You've built forgetting with no model of what's worth forgetting.

The pattern underneath all three: they conflate storage management with memory management. Deleting a row frees disk. It also destroys the option to ever retrieve that context again. In an agent, retrieval is re-learning. If you delete, you don't just evict from context, you amputate the ability to recover. Storage is cheap. That option is not.

So the real requirement is a way to make a memory stop showing up in normal retrieval while keeping the row intact, and a way for it to come back on its own if it becomes relevant again. That's decay, not deletion.

Borrowing the forgetting curve from ACT-R

ACT-R (Adaptive Control of Thought, Rational) is a cognitive architecture from John Anderson's group at CMU that models how human memory decides what's accessible. The part I care about is the base-level activation equation, which predicts how retrievable a memory is from nothing but its history of use:

B_i = ln( Σ  t_j^(-d) )   for j = 1..n uses

Read it slowly. For each time the memory was used, you take the time since that use, raise it to a negative power d (the decay rate, canonically 0.5), and sum. Recent uses contribute large terms; ancient uses contribute tiny ones. Then take the log. What falls out is a single scalar that rises with frequency (more uses, more terms) and with recency (recent uses dominate the sum), and decays as a power law when a memory sits untouched.

The power law matters. A pure exponential (e^(-λt), the form you'll see in half the "memory decay" blog snippets) forgets on a single timescale and drives old memories to zero fast. Anderson and Schooler showed back in 1991 that human forgetting, and the actual statistics of when information is needed again in the real world, follow a power law with a long tail. Old-but-frequently-used memories keep a floor of activation instead of dropping off a cliff. That long tail is the whole point: it's what lets a rare-but-important memory stay reachable for months.

Storing every access timestamp per memory doesn't scale, so ACT-R has an optimized form that assumes uses are spread roughly evenly across the memory's lifetime:

B_i ≈ ln( n / (1 - d) ) - d * ln(L)

Now you only need two numbers per memory: n, the use count, and L, the age since creation. Two scalar columns. That's it. This is the version that survives contact with a production database, because you can compute it in SQL with ln() and an epoch subtraction, no per-access history table required.

Retrieval activation is base level plus a term for how well the memory matches the current query. In ACT-R that's spreading activation; in our world it's the vector similarity you already compute:

A_i = B_i + importance_i + w * sim(query, memory_i)

importance_i is a static base weight you assign when writing the memory (a safety rule gets a boost, a passing observation gets zero). w tunes how much semantic match counts against usage history. Set w high and you're close to plain vector search; set it low and history dominates. The knob exists precisely so stale-but-similar junk can't win on similarity alone.

Putting it in Postgres

I run this on pgvector inside Postgres (CloudNativePG on the cluster), because I want the decay math and the ANN search in the same query engine. Here's the table:

-- pgvector store with ACT-R decay metadata; rows are never deleted
CREATE TABLE agent_memory (
    id           bigserial PRIMARY KEY,
    content      text        NOT NULL,
    embedding    vector(1024) NOT NULL,
    importance   real        NOT NULL DEFAULT 0.0,  -- static base weight
    created_at   timestamptz NOT NULL DEFAULT now(),
    last_used_at timestamptz NOT NULL DEFAULT now(),
    use_count    integer     NOT NULL DEFAULT 1,
    tier         text        NOT NULL DEFAULT 'hot'  -- 'hot' | 'cold'
);
CREATE INDEX ON agent_memory USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON agent_memory (tier);

Four metadata fields carry the whole policy: importance, created_at, use_count, and tier. No history table, no separate cache. Retrieval computes activation inline and re-ranks by it, filtering to the hot tier:

SELECT id, content,
       ln(use_count / (1 - 0.5))
         - 0.5 * ln(greatest(extract(epoch FROM now() - created_at), 1))
         + importance
         + 2.0 * (1 - (embedding <=> $1)) AS activation   -- $1 = query vector
FROM agent_memory
WHERE tier = 'hot'
ORDER BY activation DESC
LIMIT 8;

<=> is pgvector's cosine distance, so 1 - (embedding <=> $1) is cosine similarity, weighted here by 2.0. The decay constant 0.5 is inlined for clarity; parameterize it in real code. HNSW does the approximate search, the arithmetic re-ranks the candidates by activation. One round trip.

If you'd rather keep the math in the app, the Python is short and easier to unit-test:

import math

DECAY = 0.5          # ACT-R d; 0.5 is the canonical default
SIM_WEIGHT = 2.0     # semantic match vs. usage history

def base_level(use_count: int, age_seconds: float, decay: float = DECAY) -> float:
    # ACT-R optimized base-level learning (Anderson, 1993)
    if use_count <= 0:
        return float("-inf")
    return math.log(use_count / (1 - decay)) - decay * math.log(max(age_seconds, 1.0))

def activation(mem, similarity: float, now) -> float:
    age = (now - mem.created_at).total_seconds()
    return base_level(mem.use_count, age) + mem.importance + SIM_WEIGHT * similarity

The one rule that makes or breaks this: reinforce on use, not on retrieval. When the agent actually uses a memory in its reasoning for a turn, bump its counters. If you bump every candidate the ANN search returned, everything stays hot forever and you've rebuilt the append-only log with extra steps.

-- called with the ids the agent actually acted on this turn, not every hit
UPDATE agent_memory
SET use_count = use_count + 1,
    last_used_at = now()
WHERE id = ANY($1);

That single distinction, use versus retrieval, is what lets activation carry real signal. Retrieval is cheap and noisy; use is the agent voting with its behavior.

Soft eviction: cold, not gone

Eviction is a nightly sweep, not a delete. A Kubernetes CronJob recomputes base-level activation for every row and flips the tier based on a threshold stricter than the retrieval one:

UPDATE agent_memory
SET tier = CASE
    WHEN ln(use_count / (1 - 0.5))
       - 0.5 * ln(greatest(extract(epoch FROM now() - created_at), 1))
       + importance < -3.0            -- eviction threshold
    THEN 'cold' ELSE 'hot' END;

Cold memories keep their row, their embedding, their history. They just drop out of the default hot query. Your working retrieval set stays small and clean, which is the entire goal: less clutter for the vector search to trip over, fewer tokens shipped to the model, lower latency and cost per turn.

Re-activation is where "without deletion" earns its keep. When a hot search comes back thin (too few candidates clear the retrieval threshold τ), the agent runs a fallback query that drops the tier = 'hot' filter and searches cold storage too. Any cold memory that surfaces and actually gets used gets its use_count incremented by the same update above. That bumps its base-level activation, and the next nightly sweep flips it back to hot on its own. The memory came back from the cold, because you never destroyed it. No human re-import, no restore from backup, no lost context.

This is the layer where a decay policy plugs into a bigger design. In the six-layer memory architecture I run for Claude Code, ACT-R decay governs the lower, high-churn layers where most of the volume and most of the noise live, while the top layers (pinned instructions, explicit skills) stay outside the decay path entirely. And a memory that keeps re-activating is a signal in its own right: something the agent reaches for repeatedly is a candidate to promote into a durable agent skill rather than leaving it to fight for space in the general store.

Why decay beats deletion

The reason this works isn't just "we kept the rows." It's that activation is a better relevance model than similarity alone, and a better eviction signal than age alone.

Similarity answers "is this memory about the query." Activation adds "and has it earned a place in working memory," folding in how often and how recently the agent found it useful. That second question is what filters the semantically-close-but-stale results that pure vector search can't distinguish. You're not fighting the noise floor by making the store smaller through deletion, you're lowering it by making irrelevant memories quiet. Same rows on disk, cleaner signal at query time.

The power-law tail is what makes non-destructive eviction viable. Because old-but-used memories decay slowly instead of falling to zero, a memory can sit cold for weeks and still be one good re-activation away from hot. You get to run a tight working set (cheap, fast, low-noise) without paying the deletion tax of losing rare knowledge. High precision on the hot path, high recall available on demand through the cold path. You stop trading one for the other.

There's a reliability angle too. An agent stuck re-deriving the same wrong conclusion is often being fed the same stale context every loop. Decay gives outdated memories a way to fade out of the retrieval set instead of hammering the model every turn, which pairs with the deliberate stopping conditions I covered in Three-Layer Safety for Autonomous Agents. Fresh, relevant context in; quiet, decayed context out; fewer self-reinforcing loops. This kind of memory hygiene is exactly the sort of thing I end up building for clients running agents past the prototype stage, and it's part of what the agent infrastructure work at GuatuLabs is about.

What I'd tune, and what bit me

Start with d = 0.5. It's the canonical ACT-R value and a sane default, but decay rate is workload-dependent. An agent whose facts churn fast (live system state, market data) wants faster forgetting, a higher d. An agent accumulating durable knowledge wants slower. Change one variable at a time and watch what falls out of the hot tier; it's easy to over-forget and not notice until recall quietly degrades.

SIM_WEIGHT is the knob people get wrong. Set it too high and you're back to plain vector search with decorative math on top. Set it too low and a memory the agent hasn't touched recently can't surface even when it's a dead-on semantic match, which defeats re-activation. I keep similarity dominant enough that a strong match can pull a cold memory back, but not so dominant that history stops mattering. Tune it against real retrieval logs, not intuition.

The two thresholds are separate on purpose. The retrieval τ decides what surfaces in a live query; the eviction threshold decides what goes cold overnight. Keep eviction stricter (more negative) than retrieval, so there's a buffer band of memories that are dim but not yet cold. If the two thresholds are equal, memories flap between hot and cold on every sweep and your working set thrashes.

Multi-agent setups need one more decision: shared decay or isolated. If several agents write to one store, one agent's heavy use of a memory keeps it hot for everyone, which is either exactly what you want (a swarm converging on shared knowledge) or a privacy and relevance leak (one agent's context polluting another's working set). I default to per-agent use_count and a shared embedding, so activation is personal but the underlying content is deduplicated. The tradeoffs there deserve their own treatment, and they line up with the boundaries in Multi-Agent AI Systems.

The thing that genuinely surprised me: cold storage barely costs anything and having it changes how you write memories. Once eviction is reversible, the pressure to be clever about what to keep evaporates. You stop agonizing over "is this worth remembering" and just write it, because a bad memory doesn't rot the store, it decays out and sits harmless in cold until it's relevant or never. Deletion made every write a permanent commitment. Decay made writes cheap and forgetting safe, which is the right way around. The one real cost is discipline on reinforcement: the day you start bumping counters on retrieval instead of use, the whole policy collapses into an append-only log wearing a lab coat. Keep that line clean and the rest holds.

Prometheus Alerting Rules That Don't Cry Wolf

Guatu — Mon, 29 Jun 2026 16:15:49 +0000

The fastest way to get a team to ignore Prometheus is to point an alert at up == 0 and ship it. Week one it catches a real outage and everyone's impressed. By week three it has fired a couple dozen times for scrape blips that healed themselves before anyone looked, and people have quietly built a mental filter that drops every page from that channel. The alert still fires. Nobody reads it.

That's the failure mode I want to talk about: alerts that are technically correct and operationally useless. An alert you've trained yourself to ignore is worse than no alert, because it gives you the feeling of coverage without the substance.

If you're running kube-prometheus-stack or plain Prometheus and you've reached the point where your alert channel is mostly noise, this is for you. If you've ever gotten a page that said "GPU temperature high" followed by a 36-character UUID and zero indication of which physical card was cooking, you're exactly the audience. The fix isn't fewer alerts. It's alerts that know what they're talking about.

What I reached for first (and why it didn't hold up)

The instinct when you stand up monitoring is to alert on everything you can see. Scrape is failing? Page. CPU over 80%? Page. Memory climbing? Page. Disk filling? Page. It feels responsible. You're "covering your bases."

What you actually build is a slot machine that pays out anxiety. Two specific mistakes show up over and over, and I've made both.

The first is alerting without a for: clause. A bare expression like up == 0 evaluates on every rule interval, and the instant a single scrape fails the alert goes active. Scrapes fail for boring reasons all the time: a pod gets rescheduled, a node has a half-second of network weather, an exporter restarts during a Helm upgrade. None of those are incidents. All of them page you.

The second mistake is alerting on raw hardware identifiers with no human context. Most low-level exporters emit metrics keyed by opaque IDs. nvidia_smi_temperature_gpu carries a uuid label and nothing friendly. So your alert fires, you read it half-asleep, and now you're SSHing into nodes running nvidia-smi -L to figure out which UUID maps to which card in which chassis. By the time you've found it, the alert has either resolved or escalated. Either way you've spent ten minutes doing a join in your head that Prometheus should have done for you.

Both mistakes share a root cause. The alert describes a metric, not a situation. "up is zero" and "the GPU is at 89 degrees" are facts. "The monitoring for Node A's backup exporter has been unreachable for ten minutes, and that node is the only one tagged backup-critical" is a situation. Situations are actionable. Facts are homework.

The fixes, in order of how much noise they kill

1. Make every alert survive five minutes before it fires

This is the single highest-use change and it costs one line. The for: clause tells Prometheus to hold an alert in the pending state until the condition has been continuously true for that duration. Only then does it transition to firing and hit your notification path.

groups:
  - name: targets.rules
    rules:
      - alert: ExporterDown
        expr: up{job=~"pve-exporter|unpoller|nvidia-gpu-exporter"} == 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Exporter {{ $labels.job }} is unreachable"
          description: "{{ $labels.job }} on {{ $labels.instance }} has had failed scrapes for 10m+."

A scrape blip that recovers in 30 seconds never leaves pending. You never hear about it. A genuine outage that persists past ten minutes pages you, with the job and instance already in the message. The number isn't sacred: use for: 2m for things that genuinely matter fast, for: 15m for slow-burn conditions like disk fill. The point is that almost nothing on your infrastructure justifies paging on a single failed evaluation.

If you take only one thing from this post, take this. Most "Prometheus is too noisy" complaints are really "we forgot the for: clause" complaints.

2. Join hardware IDs to human names inside the alert expression

PromQL can do the UUID-to-name lookup for you at evaluation time, so the name lands in the alert annotation instead of in your debugging session. The trick is a vector match with group_left, multiplying your metric by an _info series whose value is always 1.

nvidia_smi_temperature_gpu
  * on(uuid) group_left(name) nvidia_smi_gpu_info
  > 85

nvidia_smi_gpu_info is a metric whose only job is to carry labels: it's 1 for every GPU, tagged with uuid and name. Multiplying your temperature reading by 1 doesn't change the value, but group_left(name) copies the name label from the info series onto the result. Now the alert knows it's a "Tesla P40" and not just a hex string.

Wired into a rule:

      - alert: GPUTemperatureHigh
        expr: |
          nvidia_smi_temperature_gpu
            * on(uuid) group_left(name) nvidia_smi_gpu_info
            > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "GPU {{ $labels.name }} is running hot"
          description: "{{ $labels.name }} ({{ $labels.uuid }}) has been above 85°C for 5m+."

The page now reads "GPU Tesla P40 is running hot" with the temperature and the UUID still available for the deep dive. That one annotation turns a cry-wolf alert into one you can act on without leaving the notification. This same pattern works for any exporter that splits readings from metadata: node info series, disk SMART data, network interface names. If you're already wrestling with the realities of monitoring passthrough hardware, the GPU passthrough field guide covers why those IDs are so unstable in the first place.

When you put these joined values on a dashboard, the same context discipline applies; I wrote about not drowning the signal in chrome in Grafana dashboards: information density vs readability.

3. Suppress alerts for failures you already expect

Not every failure is a problem. Velero is a good example: snapshot a Longhorn volume and it works, snapshot an NFS-backed volume and the CSI snapshot fails because the driver doesn't support it. That second failure is expected. It will happen on every backup run, forever, and alerting on it teaches people that backup alerts are noise, which is precisely the day your actual backup breaks and nobody notices.

The clean fix is to scope the alert to the targets you actually care about using a label match instead of trying to enumerate exceptions.

      - alert: CriticalVolumeSnapshotFailed
        expr: |
          increase(velero_volume_snapshot_failure_total[1h]) > 0
          and on(namespace, name)
            kube_persistentvolumeclaim_info{storageclass="longhorn"}
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "Snapshot failed for {{ $labels.namespace }}/{{ $labels.name }}"
          description: "A backup-critical Longhorn volume failed to snapshot for 30m+."

The and on(namespace, name) ...{storageclass="longhorn"} clause filters the failures down to volumes whose storage class is one you expect snapshots to work on. NFS volumes drop out of the result set entirely, so they can't fire. You've encoded "I know NFS snapshots fail and I don't care" directly into the rule, which is far better than a Slack message someone wrote in 2024 saying "just ignore the NFS ones."

You can push the same idea further with explicit labels. Relabel your backup targets to carry backup_critical="true" and only alert on those, so a dev-scratch volume failing to back up never wakes anyone. Storage alerts are their own rabbit hole, and the gap between "Longhorn says healthy" and "actually recoverable" is worth understanding before you decide what to alert on; that's Longhorn volume health in detail.

4. Separate "the service is down" from "the monitoring is broken"

This is the distinction that fixes the worst class of false positive. The up metric only tells you whether Prometheus could scrape the exporter. It says nothing about whether the exporter could reach the thing it's monitoring.

pve-exporter makes this concrete. It scrapes the Proxmox API on your behalf and returns a pve_up metric per target. If the exporter's API token expires or loses permissions, the exporter itself is healthy and scrapeable (up == 1), but it returns pve_up == 0 because it can't authenticate upstream. Alert on up alone and you'll either miss the auth failure entirely or, worse, misread it as the node being down and start a recovery you don't need.

Split it into two alerts that say two different things:

      - alert: PVEExporterUnreachable
        expr: up{job="pve-exporter"} == 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Prometheus cannot reach pve-exporter"
          description: "The monitoring path is broken, not necessarily the node."

      - alert: PVEAPIAuthFailing
        expr: pve_up == 0 and on(instance) up{job="pve-exporter"} == 1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "pve-exporter is up but cannot talk to the Proxmox API"
          description: "Token expired or lost permissions. Check the exporter credentials."

The first alert means "I'm blind." The second means "my eyes work but the door is locked." Those need different responses, and naming them differently is what stops you from running the wrong runbook at the wrong hour. Designing alerts around the credential boundary like this is the same instinct behind keeping monitoring identities scoped and rotatable, which is part of why I treat alerting trust as a consulting problem for production systems, not an afterthought.

One trap to avoid here: don't write a thermal alert against pve-exporter. As of v3.5.5 it doesn't expose temperature metrics at all, so a "node too hot" rule against that job produces a no data state forever, which Grafana will happily render as a permanently confusing yellow tile. Alert on what the exporter actually emits, not what you wish it did.

5. Make sure your alert rules are even being scraped

This is the silent killer, and it's specific to the Prometheus Operator world. You write a perfect ServiceMonitor, your exporter is running, and no data ever appears. No alert ever fires. Everything looks deployed and nothing works.

The cause is almost always a label mismatch. The Prometheus custom resource has a serviceMonitorSelector, and kube-prometheus-stack defaults it to match release: <your-helm-release-name>. A ServiceMonitor without that label is invisible to Prometheus. It exists, kubectl get servicemonitor shows it, and Prometheus ignores it completely.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: pve-exporter
  labels:
    release: monitoring   # MUST match the Prometheus serviceMonitorSelector
spec:
  selector:
    matchLabels:
      app: pve-exporter
  endpoints:
    - port: metrics
      interval: 60s

Check your Prometheus CR with kubectl get prometheus -o yaml and look at spec.serviceMonitorSelector. Whatever it matches, your ServiceMonitor labels have to satisfy it. The same applies to PrometheusRule objects via ruleSelector: get the label wrong and your beautifully written alert rules are never loaded. Verify they made it by opening the Prometheus UI under Status → Rules. If your group isn't listed, the selector dropped it.

While you're in there, one format gotcha. If you copied alert syntax from an old StackOverflow answer that uses ALERT FooBar IF expr FOR 5m, that's the Prometheus 1.x DSL and it will not load in 2.x or 3.x. The current format is the YAML groups: structure shown throughout this post. Mixing the two is a common reason rules silently fail validation.

Why these changes actually move the needle

Every one of these fixes pushes alerts in the same direction: from describing a metric to describing a situation a human can act on.

The for: clause works because it converts instantaneous facts into sustained conditions. Infrastructure is noisy at the millisecond scale and calm at the multi-minute scale. By requiring a condition to persist, you're matching the alert's time horizon to the time horizon a human actually responds on. Nobody pages on a 200-millisecond blip because nobody can act on one.

The PromQL joins work because the cost of context should be paid once, at rule-evaluation time, not every time a tired engineer reads the page. Prometheus already has the mapping from uuid to name sitting in an info metric. Doing the join in the rule means the expensive lookup happens automatically and the result is baked into the annotation. You move work from the human to the query engine, which is the whole point of having a query engine.

Label-based suppression works because it encodes intent. "Ignore NFS snapshot failures" is a decision, and decisions that live in someone's head rot the moment that person goes on vacation. Putting the decision in the rule expression makes it durable, reviewable, and visible in the same place as the alert it governs.

The target-versus-exporter split works because "I can't see" and "the thing is broken" demand different responses, and an alert that conflates them sends you down the wrong path under pressure. Naming the two states separately is cheap insurance against the expensive mistake of treating a permissions error like an outage.

A note on the Grafana alerting overlap

One source of duplicate pages worth flagging: you can define alerts as Prometheus/Mimir rules (the Ruler) and also define them in Grafana's unified alerting, and both can route to the same contact points. People wire up both without realizing it and then wonder why every incident pages twice.

Pick one home for alert definitions. I keep mine as PrometheusRule objects in Git alongside the rest of the GitOps config, because that's where the rest of the platform lives and code review catches bad expressions before they ship. Grafana then handles dashboards and visualization, not alert routing. If you prefer Grafana-managed alerts, that's a defensible choice, but run one or the other for any given signal. Two systems firing on the same metric is how you rebuild the noise problem you just spent a weekend fixing.

What I'd do differently and what still bites

If I were starting a monitoring stack from scratch, I'd write the first alert with a for: clause and a join already in place, before I ever shipped a bare up == 0. The bad habits are easy to form and annoying to unwind once a team has learned to distrust the channel.

A few things that still catch me:

Exporters that get OOMKilled look like target outages. nvidia-gpu-exporter (v1.4.1) needs more memory than you'd guess, around 256Mi rather than the 64Mi you might set by reflex, because the NVIDIA runtime injects libraries into the container at start. Set the limit too low and it restarts in a loop, your GPU alerts flap, and you blame the GPU. The behavior ties back to how the runtime mounts things; the NVIDIA Container Toolkit post explains why that injection happens.
no data is not OK. An alert that can't evaluate because its metric vanished is a failure state, not a passing one. Use absent() rules to catch metrics that should always exist and have gone missing, otherwise a deleted exporter reads as eternal calm.
Severity is a routing decision, not a vibe. Reserve paging severity for things that need a human awake right now. Everything else is a warning that lands in a channel you read during business hours. If every alert is critical, none of them are.

The goal was never zero alerts. It's that when one fires, you believe it. An alert channel people trust is worth more than one that's technically complete, because the complete-but-ignored channel catches nothing the moment it matters. Spend the extra line on for:, the extra clause on the join, and the extra label on suppression. That's the difference between a monitoring system and a smoke detector with a dead battery that beeps at random.

SealedSecrets: Storing Secrets in Git Without the Risk

Guatu — Sat, 27 Jun 2026 14:15:48 +0000

A base64-encoded string in a Git repo is not a secret. It's plaintext with one extra step, and echo "cGFzc3dvcmQ=" | base64 -d undoes that step in about a second. Plenty of teams discover this the hard way when a secret scanner flags their carefully "encoded" data: block as a live database password sitting in the commit history, readable by anyone who ever cloned the repo.

That's the core tension of GitOps. You want Git to be the single source of truth for everything in the cluster, but "everything" includes API keys, TLS private keys, and database passwords, and committing those in the clear is how breaches start. SealedSecrets resolves that tension with a small controller and a CLI, and it's the lightest-weight tool I know of that does the job properly. If you run ArgoCD or Flux and you're still keeping a secrets/ folder out of version control "for now," this is the post for you.

The false starts I'd warn you about

Before SealedSecrets clicks, most people cycle through the same set of wrong answers. I did too.

The first instinct is to just base64-encode the value and call it encrypted. Kubernetes stores secret data as base64, so it looks scrambled, and that visual scramble fools a surprising number of code reviews. Base64 is an encoding, not encryption. There is no key. Anyone with the repo has the secret. Cross this one off immediately.

The second instinct, at least mine, was to reach for Vault. HashiCorp Vault (or its OpenBazaar-era forks) is a genuinely good piece of software, and if you're running a platform team with dynamic secrets, lease revocation, and audit requirements, it earns its keep. For a homelab or a small cluster, it's a second stateful system to run, unseal, back up, and babysit. You're now protecting your secrets manager more carefully than the secrets it holds. That's a lot of operational surface area to avoid committing a few API keys to Git.

Then there's the git history problem, which trips people who try to fix this reactively. Say you already committed a plaintext secret last month. Deleting it from the current file does nothing. It lives forever in the history, retrievable with git log -p. You'd have to rewrite history with git filter-repo or BFG, force-push, and rotate the secret anyway because you have to assume it leaked. The lesson there is that secret management is a preventative discipline. The tool has to make the wrong thing hard to do in the first place.

SOPS with age or KMS keys is a legitimate alternative, and I'd put it second on my list. It encrypts files before they hit Git and decrypts them at apply time. The reason I land on SealedSecrets for Kubernetes-native GitOps is that it lives inside the cluster as a CRD, the decryption never leaves the controller, and the developer workflow is a single command. Different tradeoffs, both valid.

How SealedSecrets actually works

The model is asymmetric encryption, and once that clicks, everything else follows.

You install a controller (Bitnami's sealed-secrets-controller, typically into kube-system). On startup it generates an RSA key pair. The public key is freely distributable. The private key never leaves the cluster. You then use the kubeseal CLI to encrypt a normal Kubernetes Secret against that public key, producing a SealedSecret custom resource. That resource is safe to commit, safe to put in a public repo, safe to paste in a Slack channel. The only thing in the universe that can decrypt it is the controller holding the matching private key.

When the SealedSecret lands in the cluster, the controller watches for it, decrypts the payload, and produces a regular Secret with the same name and namespace. Your workloads consume that Secret exactly as they always have. Nothing downstream knows the difference.

Here's the workflow loop. Start with a normal secret manifest you generate locally and never commit:

# Generate a plain Secret manifest (stays on your machine, never committed)
kubectl create secret generic db-credentials \
  --namespace=app-prod \
  --from-literal=password='example-not-a-real-password' \
  --dry-run=client -o yaml > secret.yaml

Then seal it:

# kubeseal fetches the controller's public cert and encrypts against it
kubeseal --format yaml < secret.yaml > sealed-secret.yaml

# Delete the plaintext, commit the sealed version
rm secret.yaml
git add sealed-secret.yaml

The output is a SealedSecret whose encryptedData is an opaque blob:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: db-credentials
  namespace: app-prod
spec:
  encryptedData:
    password: AgBy3i4OJSWK+PiTySYZ...truncated...rO43cGDEq
  template:
    metadata:
      name: db-credentials
      namespace: app-prod

That blob is what goes in Git. ArgoCD syncs the SealedSecret, the controller unseals it into a real Secret, and your app starts. The person writing the manifest never has to see the production value in plaintext, which is the zero-trust angle I'll come back to.

One operational detail worth setting up early: seal offline using a saved public cert instead of hitting the cluster every time. Pull the cert once, commit it (it's public), and point kubeseal at it:

# Fetch the public cert once
kubeseal --fetch-cert > pub-cert.pem

# Seal against the saved cert, no cluster connection needed
kubeseal --cert pub-cert.pem --format yaml < secret.yaml > sealed-secret.yaml

This matters because it lets you seal secrets in CI without giving the pipeline cluster access. A GitHub Actions job can encrypt a new secret during a PR using only the public cert, which pairs nicely with the kind of automated pipeline I described in Infrastructure as Code, but Automated: OpenTofu and GitHub Actions.

The scopes gotcha

By default, a SealedSecret is bound to both its name and its namespace. This is "strict" scope, and it's the right default, but it bites people who don't expect it. If you seal a secret for app-prod and then try to reuse that same sealed blob in app-staging, the controller refuses to decrypt it. Same if you rename it. The name and namespace are mixed into the encryption, so changing either invalidates the seal.

You can loosen this when you genuinely need to:

# Namespace-wide: any name within the namespace can decrypt it
kubeseal --scope namespace-wide --format yaml < secret.yaml > sealed.yaml

# Cluster-wide: any name in any namespace can decrypt it
kubeseal --scope cluster-wide --format yaml < secret.yaml > sealed.yaml

My advice is to stay on strict scope unless you have a concrete reason not to. Loosening the scope trades blast radius for convenience. Cluster-wide in particular means anyone who can create a Secret object anywhere can pull your value out, which quietly undoes part of the boundary you set up with least-privilege service accounts.

The part nobody warns you about: this is a backup problem

Here's the reframe that took me a while to internalize. SealedSecrets makes your encryption essentially bulletproof. What it does instead is concentrate all your risk into one tiny object: the controller's private key. Lose that key, and every SealedSecret in every repo becomes permanently undecryptable. The blobs are fine. The cluster that can read them is gone.

Picture the disaster cleanly. Your cluster dies (a failed etcd restore, a wiped control plane, a homelab node that won't come back). You rebuild from scratch, install a fresh SealedSecrets controller, and point ArgoCD at your repo. ArgoCD dutifully applies every SealedSecret. The new controller generated a new key pair on install, so it cannot decrypt a single one of the blobs sealed against the old public key. Every secret-dependent workload fails to start. Your Git repo is intact, your manifests are perfect, and none of it works.

The fix is to treat the private key as the single most important artifact in your infrastructure and back it up off-cluster. The keys live as Secrets in the controller's namespace, labeled so you can grab the full set in one shot:

# Back up the active sealing keys (THIS FILE CONTAINS THE PRIVATE KEYS)
kubectl get secret -n kube-system \
  -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o yaml > sealed-secrets-master.key

That output is the actual private key material, so it never goes near Git. It goes into a password manager or an encrypted, off-cluster backup. Restoring is the inverse: apply the key backup before the controller fully reconciles, then restart it so it picks up the old keys:

# On the rebuilt cluster, restore keys then restart the controller
kubectl apply -f sealed-secrets-master.key
kubectl rollout restart deployment sealed-secrets-controller -n kube-system

Now the controller holds both the old and new private keys, decrypts all your historical blobs, and your secrets come back to life. I wrote a dedicated walkthrough on exactly this lifecycle in SealedSecrets Key Backup: Don't Lose Your Encryption Keys, because it's the one step that separates a working SealedSecrets setup from a time bomb.

One thing that helps here: the controller rotates its key every 30 days by default, generating a fresh pair while keeping all the old ones for decryption. That's good for forward security, but it also means your key backup is never "done." A backup from two months ago is missing two rotations of keys. Either automate the backup on a schedule or pin the key and disable rotation deliberately, knowing the tradeoff. Stale backups are how people think they're safe right up until they aren't.

The GitOps cleanup reality check

The other thing tutorials skip is what happens when you remove or rotate a secret in a GitOps world, because ArgoCD's relationship with deleted resources is more complicated than "delete from Git, it disappears from the cluster."

By default, the controller sets an owner reference on the Secret it generates, pointing back to the SealedSecret. That's the well-behaved case: delete the SealedSecret and Kubernetes garbage-collects the generated Secret along with it. Clean. The trouble starts when that chain is broken, or when you've turned off ArgoCD's pruning.

If your ArgoCD Application doesn't have automated pruning on, removing a SealedSecret from Git leaves the object sitting in the cluster as an orphan. ArgoCD shows the app as synced because everything in Git exists in the cluster; it just won't remove the things that used to be there. You end up with ghost SealedSecrets, and sometimes the Secrets they spawned, lingering long after you deleted them from the repo. To get the behavior most people actually want, turn on prune and self-heal:

spec:
  syncPolicy:
    automated:
      prune: true      # delete cluster resources removed from Git
      selfHeal: true   # revert manual drift back to Git state

Be deliberate about prune: true, though, because it's the same flag that creates the scarier failures. When ArgoCD prunes a whole application, it deletes resources in dependency order and respects finalizers, and that's where things get stuck. I've watched deletions hang on a PVC whose finalizer waits on a volume that's still attached, or an IngressRoute that won't go because something downstream holds a reference. The SealedSecret deletes instantly; it's the stateful neighbors (PVCs, certificates, anything with a finalizer) that turn a clean delete into a half-removed application you have to untangle by hand. None of that is SealedSecrets' fault directly, but rotating or removing a secret often means touching the surrounding app, and that's when the orphans surface. The App-of-Apps structure I described in GitOps for Homelabs makes this more predictable by keeping deletion boundaries clear, but you still want to watch a prune happen the first few times rather than trusting it blindly.

Why this is the right shape of tool

The reason SealedSecrets earns a place in a GitOps stack comes down to where the trust boundary sits. The developer, the CI pipeline, and the Git host all handle only the encrypted blob. None of them can decrypt it. The only component that ever sees plaintext is the controller, which already runs inside the trust boundary of the cluster and is governed by RBAC. You've moved the secret out of every system that shouldn't have it and into the one place that already has to.

That's a genuine zero-trust property, not marketing. A new engineer can submit a PR that adds a production secret without ever knowing its value, sealing it against the public cert from CI. Review it, merge it, and the controller is the only thing that ever decrypts it. Compare that to a model where someone has to know the real password to put it in the right place, and you can see why this is the better default for a team.

It also stays lightweight in a way that matters operationally. One controller, one CRD, one CLI. No second clustered datastore to keep alive, no unseal ritual, no separate HA story. For homelabs and small-to-medium clusters, the cost-to-value ratio is hard to beat, and it composes cleanly with the rest of a Kubernetes-native stack like cert-manager handling your TLS. If you're designing a GitOps platform for a team and want the secret lifecycle and disaster-recovery story built in from the start rather than bolted on later, that's the kind of thing I help with through infrastructure consulting.

Lessons learned

If I were setting this up fresh on a new cluster, here's what I'd do in order and what I'd watch for.

Back up the private key on day one, before you seal anything. The temptation is to install the controller, seal a few secrets, and move on, treating the backup as a later task. That's exactly backwards. The key is most valuable and most likely to be forgotten right at the start. Automate the backup to an off-cluster, encrypted destination and verify the restore at least once, because an untested backup is a guess.

Stay on strict scope and resist the urge to go cluster-wide for convenience. Every loosening of scope widens who can extract the value, and you rarely get a reminder of that decision until it matters.

Treat removal as carefully as creation. Adding a SealedSecret is the easy, well-lit path. Removing or rotating one drags in ArgoCD pruning, finalizers, and owner references, and that's where the orphaned resources hide. Turn on prune and selfHeal so Git stays authoritative, but watch the first few prunes and keep an eye on anything stateful nearby.

Validate sealed manifests in CI before they reach main. A malformed SealedSecret won't decrypt, and you'd rather catch that in a PR check than at sync time when a workload won't start. The same manifest-validation habit from Stop Merging Broken YAML applies cleanly here.

The thing that surprised me most was how completely SealedSecrets shifts the problem. You go in worried about encryption strength and whether someone can crack the blob in your repo. You come out realizing the encryption was never the weak point. The weak point is a single RSA private key that, if you lose it, turns your perfectly version-controlled secrets into perfectly version-controlled garbage. Get the backup right, and the rest of the tool more or less takes care of itself.

MetalLB on Bare Metal: LoadBalancer Without a Cloud Provider

Guatu — Sat, 27 Jun 2026 12:15:48 +0000

Deploy a Service of type LoadBalancer on a bare-metal cluster and kubectl get svc hands you a permanent <pending> in the EXTERNAL-IP column. No error, no event, no hint. The Service just sits there forever, waiting for an external IP that will never arrive, because nothing in your cluster is responsible for assigning one.

That gap is the entire reason MetalLB exists. On AWS or GCP, the cloud controller manager watches for LoadBalancer Services, calls the provider API, provisions an actual load balancer, and writes the IP back into the Service status. On bare metal there is no API to call. Kubernetes ships the abstraction but not the implementation, and you're left holding a Service type that does nothing until you supply the missing piece yourself.

Who should care

If you run Kubernetes on Proxmox VMs, on a stack of mini PCs, or on anything that isn't a managed cloud, you've hit this. The moment you try to expose ingress-nginx, Traefik, a database, or an LLM inference endpoint to the rest of your LAN, you need a stable IP that lives outside the cluster and routes in. MetalLB is the most common answer, and the config has changed enough across versions that a 2021 tutorial will actively mislead you. This walks through MetalLB v0.15.x, the L2-versus-BGP decision, and the connectivity traps that make an IP look assigned while traffic quietly goes nowhere.

What I reached for first (and why it was wrong)

Before MetalLB, the obvious-looking options all have a catch.

NodePort. It works, technically. You get a port in the 30000-32767 range on every node, and you can hit http://node-ip:31234. Then you try to use it for real and remember why nobody runs production this way: the ports are ugly, they're not 80/443, and you've now hardcoded a specific node's IP into whatever's calling the service. If that node reboots or drains, your "load balancer" is a single point of failure pointing at a machine that's gone.

hostNetwork on the ingress controller. Bind ingress-nginx directly to ports 80 and 443 on the host. This also works, and it's genuinely fine for a single-node setup. On a multi-node cluster it falls apart: now you need an external load balancer or DNS round-robin in front of the nodes to spread traffic, and you've moved the problem one layer up the stack instead of solving it. You also lose the clean LoadBalancer Service abstraction that everything else in the ecosystem expects.

Assuming spec.loadBalancerIP would just work. I set spec.loadBalancerIP: 10.0.0.210 on a Service, expecting Kubernetes to honor it. Nothing happened. That field assigns an IP only if a controller is watching for it and acts on the request. Without MetalLB or a cloud provider, it's an inert string. Worse, in newer MetalLB versions that field is deprecated in favor of an annotation, so even after installing MetalLB the old syntax silently does nothing.

The mistake underneath all three: treating type: LoadBalancer as a feature Kubernetes provides, when it's really a contract Kubernetes defines and expects something else to fulfill.

Installing MetalLB

MetalLB has two pieces. A controller Deployment handles IP allocation, watching Services and writing the assigned address into their status. A speaker DaemonSet runs on every node and is responsible for actually announcing the IP to the network, either via ARP (L2 mode) or BGP.

I install it with Helm so it fits a GitOps flow. If you run ArgoCD app-of-apps, this slots in as one more application.

helm repo add metallb https://metallb.github.io/metallb
helm install metallb metallb/metallb \
  --namespace metallb-system \
  --create-namespace \
  --version 0.15.3

Installing the chart does nothing visible on its own. MetalLB ships with no IP pool by default, which is deliberate: handing out addresses you didn't explicitly authorize would be a great way to start an IP conflict war with your DHCP server. You have to tell it which addresses it owns.

That configuration moved to CRDs back in v0.13. If a guide tells you to edit a ConfigMap named config in the metallb-system namespace, it predates that change and the syntax no longer applies. The two objects you need are IPAddressPool and L2Advertisement.

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: lan-pool
  namespace: metallb-system
spec:
  addresses:
    - 10.0.0.200-10.0.0.250  # carve this OUT of your DHCP range
  autoAssign: true

apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: lan-l2
  namespace: metallb-system
spec:
  ipAddressPools:
    - lan-pool

The pool defines which addresses MetalLB may hand out. The L2Advertisement says "announce these addresses using Layer 2." Without the advertisement object, MetalLB will assign an IP from the pool but never tell the network it exists, which produces one of the most confusing failure states: a Service with an EXTERNAL-IP that nothing on the LAN can reach. The pool without the advertisement is half a configuration.

The address range matters. Those IPs must live in the same subnet as your nodes, and they must be outside whatever your DHCP server hands out. Pick a slice your router will never lease. A .200-.250 block at the top of a /24 is a common choice precisely because most DHCP pools stop well before there.

Apply both, deploy a test Service, and the <pending> finally resolves:

$ kubectl get svc -n ingress-nginx
NAME                 TYPE           EXTERNAL-IP   PORT(S)
ingress-nginx        LoadBalancer   10.0.0.200    80:31...,443:32...

L2 versus BGP: pick the simple one until you can't

MetalLB runs in one of two modes, and most homelab and small-cluster setups want Layer 2.

L2 mode uses ARP for IPv4 (NDP for IPv6). One speaker pod becomes the leader for a given Service IP and answers ARP requests for it. When a host on the LAN asks "who has 10.0.0.200?", that node replies with its own MAC, traffic arrives at the node, and kube-proxy forwards it to a backend pod. The whole thing needs zero cooperation from your router or switch. It looks like a normal host claiming an IP, which is exactly what it is.

BGP mode has each speaker peer with your router and advertise the Service IPs as routes. The router then load-balances across nodes via ECMP. This gives you real multi-node traffic distribution and faster failover, but it requires a router that speaks BGP and a willingness to manage routing config. That's a reasonable trade in a rack with a proper top-of-rack switch. In a homelab where the "router" is a consumer box, it's usually more than you want to take on.

	L2 (ARP/NDP)	BGP
Router requirement	None	Must speak BGP
Traffic distribution	One node per IP at a time	ECMP across nodes
Failover speed	Seconds (ARP re-announce)	Sub-second
Setup complexity	Low	Moderate to high
Best for	Homelabs, small clusters	Datacenter, high throughput

The honest caveat about L2: it is not true load balancing. For any single Service IP, all traffic lands on one node, the elected leader, and that node forwards internally. You get failover (if the leader dies, another speaker takes over and re-announces via gratuitous ARP) but not bandwidth aggregation. For a homelab serving an ingress controller or an Ollama API, that's completely fine. A single 10G node will saturate most home internet connections long before it becomes the bottleneck. If you genuinely need to spread inbound traffic across nodes, that's the line where BGP earns its complexity.

Why L2 mode actually works

The mechanism is worth understanding, because it explains every weird failure you'll hit later.

When the leader speaker announces a Service IP, it doesn't add the IP to a network interface in the usual sense. It answers ARP requests at the protocol level. A client on the LAN broadcasts an ARP query, the leading node responds with its MAC address, and the client's ARP table now maps the Service IP to that node's hardware address. Traffic flows to the node, hits iptables/IPVS rules installed by kube-proxy, and gets DNAT'd to a pod endpoint.

When the leader node goes down, the other speakers notice through MetalLB's memberlist-based election, a new leader is chosen, and it sends a gratuitous ARP. Gratuitous ARP is an unsolicited "hey, this IP is at my MAC now" broadcast that updates every device's ARP cache without them asking. That's the failover. It takes a few seconds for caches to update, which is why L2 failover is measured in seconds rather than milliseconds.

The reason this is all Layer 2 and not Layer 3 is also the reason for the biggest constraint: ARP doesn't cross subnet boundaries. A router won't forward an ARP request from one VLAN into another. So the Service IP, the nodes, and the client all have to sit on the same broadcast domain. The instant you want a Service IP that lives on a different VLAN than your nodes, plain L2 mode stops being enough, and you're into either BGP or careful per-interface advertisement.

The port-ownership trap

Here's the failure that eats the most debugging time, and it's the one tutorials never warn you about. The IP gets assigned. kubectl get svc shows the EXTERNAL-IP. You telnet the port and it connects. Everything says "working." But the actual application returns connection resets or timeouts, and you can't figure out why a connection that succeeds is also broken.

A successful TCP handshake to a MetalLB IP proves exactly one thing: a node owns the IP and accepted the connection. It says nothing about whether a healthy backend pod exists behind it. The classic cause is externalTrafficPolicy.

apiVersion: v1
kind: Service
metadata:
  name: ollama
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local  # the trap
  selector:
    app: ollama
  ports:
    - port: 11434
      targetPort: 11434

With externalTrafficPolicy: Local, traffic is only delivered to pods on the same node that received it, and the source IP is preserved. The problem: MetalLB elects a leader node to announce the IP, but if your single Ollama pod is scheduled on a different node, the leader has no local endpoint. Traffic arrives, finds nothing local to forward to, and gets dropped. The TCP connect still succeeds because the node owns the IP; the request just dies after that.

MetalLB is supposed to handle this by only announcing from nodes that have a ready endpoint, but the interaction with a single-replica workload pinned to one node is genuinely confusing the first time you meet it. The fix, when you don't need to preserve the client source IP, is externalTrafficPolicy: Cluster. That lets any node accept the traffic and forward it internally to wherever the pod actually runs, at the cost of an extra hop and a SNAT'd source address.

The lesson generalizes past MetalLB: stop treating "the port is open" as "the service works." Verify the full path.

# 1. Did MetalLB assign an IP from the pool?
kubectl get svc -A | grep LoadBalancer

# 2. Which node is announcing it? (look for the assigned IP)
kubectl logs -n metallb-system -l component=speaker | grep 10.0.0.200

# 3. From ANOTHER host on the same VLAN, does the IP answer ARP?
arping -I eth0 10.0.0.200

# 4. Does the backend actually respond, not just accept the connection?
curl -v http://10.0.0.200:11434/api/tags

That step 3 is the one people skip. arping confirms the IP is genuinely claimed at Layer 2 on your network, independent of anything Kubernetes thinks. If arping gets no reply but kubectl swears the IP is assigned, your speaker isn't announcing on the interface you expect, and you've found your problem before wasting an hour in application logs.

The hybrid VLAN reality

Most tutorials assume a flat network where every node has one NIC on one subnet. Real labs are segmented. You might want ingress traffic on one VLAN and management on another, with nodes carrying multiple interfaces.

MetalLB v0.15.x handles this through interface and node selectors on the L2Advertisement. You can scope a pool so it only ever announces on a specific NIC, which keeps an ingress IP from accidentally being advertised on your management network.

apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: ingress-vlan
  namespace: metallb-system
spec:
  ipAddressPools:
    - ingress-pool
  interfaces:
    - eth1          # only announce on the ingress-facing NIC
  nodeSelectors:
    - matchLabels:
        node-role: ingress

This is where L2's same-subnet constraint bites hardest. Each pool's addresses still have to be reachable on the broadcast domain of the interface announcing them. You can segment which IPs go where, but you can't make ARP magically route across VLANs. If your design genuinely needs Service IPs that live on subnets your nodes don't touch, that's the signal to either bring those VLANs to the nodes as tagged interfaces or move to BGP and let the router do the routing it's built for.

MetalLB as the foundation, not the finish line

A MetalLB IP is rarely the whole story. It's the stable entry point everything else builds on. The typical flow looks like this:

External client → MetalLB IP → ingress controller Service → Ingress/IngressRoute → backend Service → pod.

MetalLB hands ingress-nginx or Traefik a single, stable LAN IP. Your ingress controller terminates TLS and routes by hostname to dozens of backends. You point a wildcard DNS record at that one MetalLB IP, and now *.lab.example.com resolves to your cluster. This is also the precondition for automated certificates: cert-manager with a DNS-01 solver needs your ingress reachable at a known address before it can prove ownership and issue certs. MetalLB provides the address; cert-manager provides the identity.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ollama
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  ingressClassName: nginx
  tls:
    - hosts: [ollama.lab.example.com]
      secretName: ollama-tls
  rules:
    - host: ollama.lab.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: ollama
                port: { number: 11434 }

That Ingress doesn't reference a MetalLB IP directly, and that's the point. It targets the ingress controller's Service, which is the MetalLB-assigned IP. The same pattern extends upward to the Gateway API: a Gateway's listener address comes from a LoadBalancer Service, so gatewayAddresses in tools layered on top ultimately resolve to an IP MetalLB allocated. Get the bottom layer right and everything above inherits a stable address.

If you want a Service to keep a fixed IP across restarts, annotate it (the deprecated spec.loadBalancerIP is the trap here):

metadata:
  annotations:
    metallb.io/loadBalancerIPs: 10.0.0.210

What I'd do differently

A few things I wish I'd internalized before the first install.

Carve the IP pool out of DHCP on day one. The single most common MetalLB incident is an address conflict because the pool overlapped the DHCP lease range, and your router handed the same IP to a laptop. Decide the block, exclude it in your DHCP server, and document it. This is boring and it prevents a class of intermittent failures that are miserable to diagnose.

Default to externalTrafficPolicy: Cluster unless you have a specific reason to preserve source IPs. Source IP preservation matters for some applications, geo-logic, and rate limiting, but if you don't need it, Local only buys you the single-node-endpoint trap. Reach for Cluster first and switch to Local deliberately when a workload demands the real client IP.

Treat the arping check as a permanent part of your toolkit, not a one-time debug step. Separating "is the pod running" from "is the network path open" is the difference between an hour of log-reading and a thirty-second confirmation. MetalLB sits exactly at the seam between Kubernetes and your physical network, so when it misbehaves, you have to test both sides independently.

And keep the whole config in Git. The pool, the advertisement, and the per-Service annotations are small, declarative, and exactly the kind of thing that drifts when you edit it by hand at 11pm. Folding MetalLB into a GitOps repo alongside the rest of the cluster, the same way I treat Longhorn for bare-metal storage, means the networking layer is as reproducible as everything else. Bare-metal Kubernetes is mostly the work of rebuilding the conveniences a cloud provider gave you for free, one controller at a time. If you're standing up this kind of infrastructure and would rather not rediscover every trap yourself, that's the kind of work I do.

The surprise, looking back, was how little of the difficulty was MetalLB itself. The install is two Helm commands and two small CRDs. The hard part was everything around it: which subnet, which VLAN, which traffic policy, and the stubborn assumption that a connection accepting means a service working. Get those right and the <pending> that started all of this never comes back.

OpenTofu State in MinIO: An S3-Compatible Backend for IaC

Guatu — Sun, 21 Jun 2026 10:15:26 +0000

A .tfstate file on your laptop is a single point of failure with no redundancy and no audit trail. The first time a GitHub Actions runner applies against the same infrastructure you just touched locally, you get drift, a lock error, or the failure mode nobody warns you about: a perfectly clean apply that quietly tears down a container. None of that is a state-storage problem in isolation, but remote state is where you fix the storage half and where you have to start thinking about the destruction half.

If you run infrastructure-as-code against a homelab or any self-hosted environment, you already have a perfectly good S3 endpoint sitting there: MinIO. You probably stood it up for backups. It can hold your OpenTofu state too, inside your own perimeter, with no Terraform Cloud account and no real AWS bill. The catch is that "S3-compatible" hides a few sharp edges, and pointing CI at your real compute changes the blast radius of a typo.

Who should care

This is for people running OpenTofu (or Terraform) against on-prem providers like Proxmox, where the resources under management are real boxes you'd rather not lose. If your state already lives in AWS S3 with DynamoDB locking and you're happy, you don't need this. The interesting territory is self-hosted: you want cloud-native state management without renting the cloud, and the provider you're driving can replace resources in ways a managed cloud rarely does.

What I tried first

The obvious starting point is local state. It works, right up until it doesn't. You write your .tf files, run tofu apply, and the state lands in terraform.tfstate next to your code. Then you add a CI pipeline, the runner starts from an empty checkout with no state, and OpenTofu happily plans to create everything you already created. Now you've got two sources of truth and a plan that wants to make duplicates.

The next instinct is to gitignore the state and commit it anyway "just for CI," or to scp it into the runner. Both are bad. State holds resource IDs and, depending on your providers, secrets in plaintext. It does not belong in Git. So remote state it is, and since MinIO is already running, the S3 backend looks like a five-minute job.

It is not a five-minute job the first time. My first MinIO backend block was copied from an AWS example with the endpoint swapped in:

terraform {
  backend "s3" {
    bucket   = "opentofu-state"
    key      = "proxmox/terraform.tfstate"
    region   = "us-east-1"
    endpoint = "https://minio.example.com"
  }
}

tofu init rejected endpoint as deprecated, and once I got past that, the AWS SDK underneath tried to reach opentofu-state.minio.example.com. That hostname does not exist and the wildcard cert doesn't cover it, so I got TLS and DNS errors that looked like a MinIO problem and were actually an addressing-style problem. Then OpenTofu tried to call the AWS instance-metadata endpoint to figure out a region and credentials, which obviously goes nowhere on a bare-metal runner. Every error pointed at a different layer. None of them said "you forgot path-style addressing," which is what every single one of them meant.

The actual solution

Here's the backend block that actually works against MinIO with current OpenTofu. The shape of this changed across versions, so the keys matter:

terraform {
  backend "s3" {
    bucket = "opentofu-state"
    key    = "proxmox/terraform.tfstate"
    region = "us-east-1" # arbitrary, but required by the SDK

    endpoints = {
      s3 = "https://minio.example.com"
    }

    use_path_style              = true # bucket in the path, not the hostname
    use_lockfile                = true # S3-native locking, no DynamoDB
    skip_credentials_validation = true
    skip_requesting_account_id  = true
    skip_metadata_api_check     = true
    skip_region_validation      = true
  }
}

A few of those lines are the whole game. endpoints = { s3 = ... } replaced the old top-level endpoint attribute. use_path_style = true is the rename of what older guides call force_path_style. The four skip_* flags stop the AWS SDK from trying to validate credentials, look up an account ID, hit the metadata service, or sanity-check the region against AWS's real region list. On MinIO every one of those checks either fails or wastes time.

Credentials do not go in the block. Keep them out of .tf files entirely and feed them through the standard AWS environment variables, which the S3 backend reads natively:

export AWS_ACCESS_KEY_ID="$(op read 'op://Infra/minio-opentofu/access-key')"
export AWS_SECRET_ACCESS_KEY="$(op read 'op://Infra/minio-opentofu/secret-key')"
tofu init

That access key should not be your MinIO root user. Create a dedicated user with a policy scoped to exactly the one bucket OpenTofu touches. Least privilege here is the difference between "a leaked CI secret can read one state file" and "a leaked CI secret owns your entire object store." This is the same mindset I wrote about for agent service accounts: give the automation the smallest grant that lets it do its job.

The MinIO policy looks like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
      "Resource": ["arn:aws:s3:::opentofu-state"]
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": ["arn:aws:s3:::opentofu-state/*"]
    }
  ]
}

ListBucket and GetBucketLocation are on the bucket itself. The object actions live on the /* path. DeleteObject is non-optional even though that surprises people: the lockfile is written before an operation and deleted after, so without delete you can acquire a lock and never release it. Apply it with the MinIO client:

mc admin policy create local opentofu-state ./opentofu-state-policy.json
mc admin user add local opentofu-ci "$(op read 'op://Infra/minio-opentofu/secret-key')"
mc admin policy attach local opentofu-state --user opentofu-ci

Wiring it into CI

In GitHub Actions, the MinIO credentials come from repository secrets and get injected as the AWS env vars the backend expects. The job stays small:

- name: OpenTofu Init
  env:
    AWS_ACCESS_KEY_ID: ${{ secrets.MINIO_ACCESS_KEY }}
    AWS_SECRET_ACCESS_KEY: ${{ secrets.MINIO_SECRET_KEY }}
  run: tofu init

- name: OpenTofu Plan
  env:
    AWS_ACCESS_KEY_ID: ${{ secrets.MINIO_ACCESS_KEY }}
    AWS_SECRET_ACCESS_KEY: ${{ secrets.MINIO_SECRET_KEY }}
  run: tofu plan -detailed-exitcode -out=tfplan

-detailed-exitcode is the piece most pipelines skip. It returns 0 for no changes, 2 for a non-empty plan, and 1 for an error. That lets you gate apply on human approval whenever the plan is non-empty, instead of letting every push to main run straight through to apply -auto-approve. If your runner can't reach MinIO over the public internet, this is also where a Tailscale step earns its keep; I covered that access pattern in Tailscale subnet routers.

The safety net that actually matters

Storing state remotely solves the "where does the truth live" problem. It does nothing for the "a successful apply destroyed something" problem, and on Proxmox that problem is real. The bpg and Telmate providers both have attributes marked ForceNew. Change one of those on an LXC, the network bridge or the rootfs storage, and the provider cannot update in place. It plans a destroy followed by a create. Replacing an LXC means deleting the container and its filesystem, then building a fresh one. The data that lived on it does not come back.

Read that plan by hand and you'd catch it instantly: -/+ destroy and then create replacement. Run it under -auto-approve in CI off a push to main, and the box is gone before anyone looks at the diff. Remote state didn't cause that, but remote state is what made unattended applies easy, so it's the moment to add guards. The first guard is a lifecycle block on anything stateful:

resource "proxmox_virtual_environment_container" "app" {
  # ... container config ...

  lifecycle {
    prevent_destroy = true
  }
}

With prevent_destroy = true, any plan that would delete or replace this resource fails at plan time with an error instead of proceeding. That's the behavior you want: the pipeline stops and shouts rather than quietly doing the destructive thing. When you genuinely need to replace the resource, you remove the block deliberately, in a reviewed commit, with your eyes open. The same defensive instinct shows up in provider-driven recreation elsewhere; I hit a version of it with hardware, where a card got replaced out from under me, in the GPU passthrough field guide.

Why it works

Two mechanics are worth understanding instead of cargo-culting, because both will bite you again in other contexts.

The path-style flag comes down to URL construction. AWS S3 prefers virtual-hosted-style addressing: the bucket becomes a subdomain, bucket.s3.amazonaws.com. MinIO serves buckets path-style by default: minio.example.com/bucket. When the SDK defaults to virtual-hosted style against MinIO, it builds opentofu-state.minio.example.com, which has no DNS record and no matching certificate. Setting use_path_style = true tells the SDK to keep the bucket in the path. Every confusing TLS-and-DNS error I got was this and only this, wearing four different costumes.

State locking is the other one. The classic Terraform/OpenTofu S3 backend needed a DynamoDB table to hold a lock item, because plain S3 historically had no way to do an atomic "create this only if it doesn't already exist." Self-hosters without DynamoDB faked it with sidecar services or just turned locking off and prayed. Recent OpenTofu (1.10+) added use_lockfile = true, which writes a .tflock object next to your state and relies on S3 conditional writes (PutObject with If-None-Match) to make lock acquisition atomic. MinIO implements conditional writes, so the lock is honest: a second apply that arrives while the first holds the lock gets refused at the object-store level, not by a polite convention. No extra database, no extra moving part. That's why the same MinIO instance you use for Velero Kubernetes backups can back your IaC state without bolting anything else on. One S3 layer, two jobs.

prevent_destroy works because it intercepts the plan graph before execution. OpenTofu walks the dependency graph, decides each resource needs no-op, update, create, or replace, and a lifecycle guard converts any planned delete on that resource into a hard error. It is a plan-time veto, not a runtime one, which is exactly why it's safe in CI: nothing has touched real infrastructure yet when it fires.

Lessons learned

The version churn on the S3 backend is the thing that wasted the most of my time, and most of it is invisible until init complains. endpoint became endpoints.s3. force_path_style became use_path_style. DynamoDB locking became use_lockfile. If you're following a guide written for an older OpenTofu or for Terraform pre-1.10, half the keys are renamed and you'll burn an afternoon on deprecation warnings that don't clearly explain the replacement. Check the backend docs for your exact version before copying anyone's block, including mine.

prevent_destroy is a seatbelt, not an airbag, and it has a real gap: it only protects a resource that's still in your configuration. Delete the resource block from your .tf files and the lifecycle rule goes with it, so the next plan cheerfully schedules the destroy with nothing left to veto it. Treat removing a stateful resource from config as a destructive operation in its own right, and review those diffs as carefully as you'd review an apply.

The destruction risk reframed how I think about remote state entirely. Moving state into MinIO felt like a storage decision, but the real change is that you've made unattended applies cheap, and cheap unattended applies against real compute need guardrails that local-state workflows never forced you to build. -detailed-exitcode to gate on human review, prevent_destroy on anything holding data, and a habit of actually reading replacement plans matter more than the backend block itself. This is the part of the workflow that complements the GitHub Actions automation I described in automating OpenTofu with GitHub Actions: the automation gets you speed, the guards keep the speed from being a liability.

Credentials were the last lesson, and an easy one to get lazy about. The MinIO key OpenTofu uses should be scoped to one bucket, stored in a secret manager, and pulled into env vars at runtime rather than parked in a .tf file or, worse, a shell history. If you've ever fought your shell over special characters in an automation token, you already know how these leak; I wrote about one such trap in Proxmox API tokens and the ! character. A scoped key plus a lifecycle guard turns the worst-case CI accident from "lost the cluster" into "lost one state file I can restore from a versioned bucket."

That last point is the quiet upside of running this on MinIO instead of a SaaS backend. Turn on bucket versioning and your state file gets the same history your backups do, inside the same perimeter, under the same retention you already control. If you want help designing that kind of self-hosted IaC and storage layer for a production environment, that's the sort of thing I do through GuatuLabs. The state of your infrastructure is some of the most sensitive data you have. Keeping it on hardware you own, behind credentials you scoped, with guards that refuse to delete the wrong box, is a reasonable thing to want, and it's maybe a weekend of work once you know which four flags actually matter.

CloudNativePG: Running PostgreSQL in Kubernetes Without the Pain

Guatu — Tue, 16 Jun 2026 00:15:32 +0000

A CloudNativePG cluster that sits in Setting up primary forever, with zero error events on the Cluster resource and a perfectly healthy operator, is one of the more frustrating ways to spend an afternoon. The operator says it's working. The pods never appear. And the actual cause has nothing to do with the database at all.

Running stateful databases on Kubernetes used to be the thing everyone told you not to do. CloudNativePG (CNPG) changed that calculus for a lot of people, including me. It's a proper operator: it handles failover, backups, connection routing, and rolling upgrades through native Kubernetes primitives instead of bolting Postgres onto a StatefulSet and praying. If you run a hardened cluster with admission controllers, network policies, and least-privilege RBAC, this post is about the friction you'll hit that the quickstart never mentions.

Who should care

If your cluster is vanilla, kubectl apply the operator and a Cluster manifest, and you're done in ten minutes. The CNPG docs are genuinely good for that path. This is for the rest of us: people running Kyverno or OPA Gatekeeper, self-signed cert chains, and the kind of policy-as-code setup where every workload has to justify its existence. That's where CNPG stops being a ten-minute install and starts being an integration project.

What I tried first

The first instinct, when a CNPG cluster hangs, is to assume you got the database config wrong. So you go read your Cluster manifest line by line. You check the storage class. You check that the PVC bound. You bump the operator log level and watch it cheerfully report that it's reconciling, over and over, with no complaints.

Here's the trap: the CNPG operator doesn't run initdb itself. It creates a Kubernetes Job to bootstrap the primary. That Job spawns a Pod. And in a hardened cluster, the Pod is where everything dies, because your admission controller is judging it against policies the operator's own Pods were exempted from but the bootstrap Job was not.

The mistake I see constantly is reading the wrong resource. People kubectl describe cluster and kubectl describe pod on the operator, find nothing, and conclude CNPG is broken. The events you need are on the Job and on the Pod the Job tries to create. A blocked Pod creation shows up as an event on the Job's owning controller, not on the Cluster:

# The Cluster looks stuck here, but says nothing useful
kubectl get cluster -n databases
# NAME       AGE   INSTANCES   READY   STATUS                    PRIMARY
# pg-main    8m    3           0       Setting up primary

# The real story is on the bootstrap Job's events
kubectl describe job -n databases pg-main-1-initdb

If a policy is the culprit, that describe output is where you'll finally see something like admission webhook "validate.kyverno.svc" denied the request: validation error: every container must define resource limits. The bootstrap Job's Pod template didn't set CPU/memory limits, your require-resource-limits policy rejected it, and the operator quietly retries forever because, from its perspective, it asked Kubernetes nicely and Kubernetes said no.

I spent longer than I'd like to admit assuming the storage layer was at fault before I went and looked at the Job. The lesson stuck: when an operator hangs, find the resource the operator creates, not the resource it manages.

The actual solution

1. Exempt CNPG lifecycle resources from blocking policies

CNPG generates Jobs and Pods on your behalf, and you can't directly edit their pod templates the way you would a Deployment you wrote. So the fix isn't to add resource limits to the Job. It's to teach your policy engine that CNPG-owned resources are allowed to skip the rule that's blocking them.

Every resource CNPG creates carries the cnpg.io/cluster label. That label is your exclusion key. For Kyverno, add an exclude block to the rule that's firing:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
spec:
  validationFailureAction: Enforce
  rules:
    - name: validate-resources
      match:
        any:
          - resources:
              kinds: ["Pod"]
      exclude:
        any:
          - resources:
              # CNPG-managed Pods (instances + bootstrap Jobs) carry this label
              selector:
                matchLabels:
                  cnpg.io/cluster: "*"
      validate:
        message: "Every container must define CPU and memory limits."
        pattern:
          spec:
            containers:
              - resources:
                  limits:
                    memory: "?*"
                    cpu: "?*"

This is a deliberately narrow exclusion. You're not disabling the policy. You're carving out resources that match a specific operator-owned label, which means a developer can't accidentally smuggle a limitless Pod past the gate by slapping a random label on it. If you want to be stricter, scope the exclusion to the databases namespace as well so the label only grants an exemption where CNPG is actually allowed to run.

The same idea applies to OPA Gatekeeper, just expressed differently: add the label to the constraint's match.excludedNamespaces or write a labelSelector exclusion in the constraint spec. The principle doesn't change. Match the operator's label, exempt the lifecycle resources, leave everything else under enforcement. I wrote about the general shape of this in Kyverno Admission Controllers: Policy-as-Code That Actually Works, and CNPG's initdb Job is the cleanest real-world example I've found of policy breaking infrastructure in a way that's invisible until you know where to look.

2. Give the operator the RBAC it actually needs

If you provision service accounts by hand instead of trusting the operator's defaults, remember that CNPG needs to manage Jobs, Pods, PVCs, Secrets, and Services on your behalf. A read-only or overly-scoped account will fail in the same silent way a policy block does: the reconcile loop runs, the create call gets a 403, and nothing visible happens.

The operator's ClusterRole covers this out of the box. If you're tightening it, the non-obvious permissions are the ability to create and delete Jobs (for initdb and restores) and to manage PVCs (for volume expansion and replica provisioning). Strip those and your cluster bootstraps fine until the first time it needs to scale or recover, then breaks. I go deeper on scoping accounts like this in Kubernetes RBAC: Building Least-Privilege Service Accounts.

3. Pin your PostgreSQL minor version away from 16.4

There's a known regression in PostgreSQL 16.4 where the server can hit a segmentation fault under certain memory conditions on nodes with large amounts of RAM available. If you're running CNPG on beefy worker nodes (16GB+ of available memory is the trigger zone), this is exactly the kind of thing that looks like a CNPG bug, a storage bug, or a kernel OOM, when it's actually upstream Postgres.

The fix is boring and effective: pin the image to a known-good minor and don't float the tag.

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: pg-main
  namespace: databases
spec:
  instances: 3
  # Pin explicitly. Do not use a floating major-version tag in production.
  imageName: ghcr.io/cloudnative-pg/postgresql:16.6
  storage:
    size: 20Gi
    storageClass: longhorn
  resources:
    requests:
      memory: "2Gi"
      cpu: "500m"
    limits:
      memory: "2Gi"
      cpu: "1"

Note the memory requests and limits are set to the same value. For a database, you almost never want Postgres getting throttled or evicted because a noisy neighbor ballooned and the scheduler decided your requests were a polite suggestion. Equal requests and limits put the Pod in the Guaranteed QoS class, which is what you want for a stateful workload you can't afford to lose to memory pressure.

4. Understand the three Services CNPG hands you

This is the part that pays off long after install. For a cluster named pg-main, CNPG creates a set of Services automatically, and each one routes to a different role:

Service	Routes to	Use it for
`pg-main-rw`	Current primary	Writes, migrations, anything that mutates
`pg-main-ro`	Replicas only	Read-only queries, reporting, analytics
`pg-main-r`	Any instance (primary or replica)	Reads where you don't care which node

The -rw Service is the important one: when CNPG fails over, it repoints -rw at the new primary. Your application doesn't need to know a failover happened. It keeps connecting to pg-main-rw.databases.svc.cluster.local and the operator handles the rest. That's the entire value proposition of running Postgres under an operator instead of as a hand-rolled StatefulSet.

For read/write splitting, point your app at two connection strings instead of one. Most ORMs and connection libraries support a primary/replica config:

# In your app's config or Secret
env:
  - name: DATABASE_URL_PRIMARY
    value: "postgresql://app:$(PGPASSWORD)@pg-main-rw.databases.svc.cluster.local:5432/appdb"
  - name: DATABASE_URL_REPLICA
    value: "postgresql://app:$(PGPASSWORD)@pg-main-ro.databases.svc.cluster.local:5432/appdb"

Send SELECTs that tolerate slight replication lag to -ro, and send everything else to -rw. The catch worth stating plainly: replicas are asynchronous by default, so a read immediately after a write can return stale data. If you need read-your-writes consistency for a given query, send it to -rw. Don't blanket-route all reads to replicas and then act surprised when a user doesn't see the row they just created.

5. Connection SSL: the untrusted-certificate wall

CNPG enables TLS by default and issues its own certificates through an internal CA. That's good for in-cluster security and annoying the first time a client refuses to connect because it doesn't trust the CA.

The error you'll see from a client is some flavor of SSL error: certificate verify failed or self-signed certificate in certificate chain. The wrong reaction is to globally disable TLS on the cluster. The right reaction depends on who's connecting:

# In-cluster clients: trust CNPG's CA. The operator publishes it as a Secret.
kubectl get secret pg-main-ca -n databases -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
# Then point the client at it:
# postgresql://...?sslmode=verify-full&sslrootcert=/etc/pg/ca.crt

For clients that genuinely can't do certificate verification (some managed platforms and serverless backends only support a binary "SSL on/off" toggle and can't be handed a custom CA), you have two honest options. Either set sslmode=require on the client, which encrypts the connection but skips CA verification, or terminate trust at a proxy you control. sslmode=require is the pragmatic middle ground: you keep encryption in transit and drop only the identity check. It's not as strong as verify-full, but it's a deliberate, documented tradeoff rather than turning TLS off entirely.

Here's the quick reference I keep around for the sslmode ladder:

`sslmode`	Encrypted?	Verifies CA?	Verifies hostname?
`disable`	No	No	No
`require`	Yes	No	No
`verify-ca`	Yes	Yes	No
`verify-full`	Yes	Yes	Yes

Aim for verify-full for anything in-cluster, where you control the CA distribution. Drop to require only for external clients that can't be handed the CA, and never to disable. If you're already running cluster-wide TLS automation, the CA-distribution problem is the same one cert-manager solves for ingress; I covered that workflow in cert-manager + Cloudflare DNS-01: Automated TLS for Everything.

6. Exposing pgAdmin without poking a hole in the cluster

You'll eventually want a GUI to poke at the database. The pattern I'd reach for is pgAdmin4 in its own namespace, reachable through your existing ingress controller, never exposed directly. Keep it in a separate namespace from the database so your network policies can treat it as an external-ish client that's explicitly allowed to reach the -rw/-ro Services, rather than something that lives inside the data tier.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: pgadmin
  namespace: pgadmin
  annotations:
    # Force HTTPS and lean on cert-manager for the cert
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    # pgAdmin needs a bigger body size for imports/exports
    nginx.ingress.kubernetes.io/proxy-body-size: "16m"
spec:
  ingressClassName: nginx
  tls:
    - hosts: ["pgadmin.example.com"]
      secretName: pgadmin-tls
  rules:
    - host: pgadmin.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: pgadmin
                port:
                  number: 80

Put authentication in front of it. pgAdmin's own login is fine, but I'd add an ingress-level auth layer (OAuth proxy or basic auth) so a leaked pgAdmin password isn't a direct line to your database. And lock down the NetworkPolicy so only the pgAdmin namespace can reach the database Services. A database admin GUI on the public internet with default credentials is how clusters become someone else's crypto miner.

Why it works

The thing that finally made CNPG click for me is that it's not pretending Postgres is stateless. It embraces the fact that a database has a primary and replicas, that failover is a real event, and that bootstrapping is a one-time Job rather than a steady-state process. Every piece of the design maps a Postgres concept onto a native Kubernetes object you can inspect with kubectl.

That's also why the failure modes are sneaky. The operator delegates the actual work to Jobs and Pods, so when an admission controller or RBAC rule blocks one of those, the operator has no good way to surface it beyond a stalled status. There's no exception thrown into your terminal. The reconcile loop is doing exactly what it's designed to do, which is keep trying, and "keep trying against a wall" looks identical to "working" until you go read the Job's events.

The Service abstraction works because CNPG owns the failover decision and the endpoint update atomically. When it promotes a replica, it updates the -rw Service's selector in the same control loop. There's no DNS TTL to wait out, no client-side failover logic to get wrong, no floating VIP to manage. Kubernetes Service routing was already solving "send traffic to whichever Pod currently has this role," and CNPG just plugs the primary/replica roles into that existing machinery. Running databases reliably on Kubernetes is the kind of platform-engineering work that separates a homelab toy from production infrastructure, and it's a chunk of what I do in consulting engagements.

Lessons learned

The biggest shift was learning to debug the resources the operator creates, not the ones it manages. kubectl describe cluster will lie to you by omission. The Job and its Pod tell the truth. If a CNPG cluster hangs in Setting up primary, my first move now is straight to the bootstrap Job's events, and nine times out of ten it's a policy or RBAC denial, not a database problem.

What surprised me was how much the hardened-cluster setup matters. Every CNPG tutorial assumes a permissive cluster, so the exact features that make a cluster production-grade (enforced resource limits, least-privilege RBAC, default-deny network policies) are the features that break the install. None of them are CNPG's fault. They're the cost of doing security right, and the fix is always a narrow, labeled exclusion rather than a blanket exception. If you run CNPG via GitOps, put those policy exclusions in the same ArgoCD app as the operator so they're never out of sync; the App-of-Apps pattern handles this cleanly.

If I were starting over, I'd pin the PostgreSQL minor version from day one and treat floating tags as a production smell, set Guaranteed QoS on the database Pods before the first incident rather than after, and write the read/write split into the application from the start instead of routing everything at the primary and refactoring later. None of those are hard. They're just the kind of decision that's cheap to make early and expensive to retrofit once you have data and uptime to protect.

CNPG genuinely delivers on running Postgres in Kubernetes without the pain, but only if you account for the cluster you actually have, not the empty one the docs assume. The operator is excellent. The integration with your security posture is the part you own.

Proxmox Backup Server: Incremental Backups for Your Whole Cluster

Guatu — Mon, 15 Jun 2026 18:15:32 +0000

A full Proxmox cluster rebuild from scratch takes somewhere between a weekend and a week, depending on how much of your config lives in Git versus your head. The VMs and LXCs themselves, the ones with actual state in them, those are the things you can't reconstruct from memory. Proxmox Backup Server (PBS) exists specifically for this problem: deduplicated, incremental backups of your entire virtualization layer, with verification built in.

If you're running a multi-node Proxmox cluster and your backup strategy is still "I'll just snapshot it manually before I do anything scary," this is the upgrade path. PBS slots into an existing cluster with surprisingly little friction, but the authentication model and a few operational quirks will trip you up if you don't know they're coming.

Why Not Just Use vzdump to NFS?

The built-in vzdump tool works. You can schedule backups to an NFS share and call it a day. I've seen plenty of homelabs run this way for years. The problem is what happens at scale.

With vzdump to a plain NFS target, every backup is a full copy. A 50 GB VM backed up daily for 30 days is 1.5 TB of storage, most of it identical data. PBS changes this fundamentally. It chunks the data, deduplicates across all backups (and across all VMs), and only transfers the changed chunks on subsequent runs. That 1.5 TB becomes something closer to 80-120 GB depending on churn rate.

The other thing vzdump alone doesn't give you is backup verification. PBS can mount and verify the integrity of every backup after it completes, checking that the data is actually restorable. That matters more than most people think. A backup you've never tested is just a hope.

I initially tried running vzdump backups to a Synology NFS share. It worked, but retention management was manual, dedup was nonexistent, and I had zero confidence that any given backup was actually restorable until I tried. PBS replaced all of that with a single integration point.

The PBS Placement Decision

Before installing anything, you need to answer one question: where does PBS run?

There are two reasonable options for a homelab:

Option 1: PBS as a VM on the cluster itself. Quick to set up, uses existing hardware, but your backup server lives on the infrastructure it's backing up. If you lose the node hosting PBS, you lose your backup target at the exact moment you need it most.

Option 2: PBS on a dedicated machine, physically separate from the cluster. This is the correct answer for anything you actually care about. A small mini-PC with a large spinning disk, or even an old desktop with a few terabytes of storage, is enough. The key property is that it's not on the same failure domain as your cluster.

I'd go with option 2 every time. A used mini-PC with a 4 TB drive costs less than the time you'll spend rebuilding a cluster from scratch. PBS itself is lightweight. It doesn't need much CPU or RAM. What it needs is disk space and network connectivity to your Proxmox nodes.

If you're running PBS on a NAS via NFS (mounting the NAS storage into a PBS VM), be aware that deduplication performance degrades over NFS compared to local storage. PBS's chunked dedup store does a lot of random I/O, and NFS adds latency to every operation. Local disk or direct-attached storage is preferable.

Installing PBS

PBS installs like any other Debian-based system. Download the ISO from the Proxmox site, boot it, run through the installer. The whole process takes about 10 minutes.

After installation, you'll access the web UI on port 8007:

https://10.0.0.50:8007

First thing to configure is a datastore, which is just a directory path where PBS will store backup chunks:

# On the PBS host, create the datastore directory
mkdir -p /mnt/backups/pbs-store

# Add it via the CLI (or through the web UI under Storage > Datastore)
proxmox-backup-manager datastore create main-store /mnt/backups/pbs-store

The datastore is where all the deduplicated chunks live. PBS handles the internal structure. You don't need to think about the file layout.

Adding PBS as Storage in Proxmox VE

On each Proxmox VE node (or once in a cluster, since storage config is shared), you add the PBS instance as a storage target. This is where the first gotcha lives.

In the PVE web UI, go to Datacenter > Storage > Add > Proxmox Backup Server. You'll need:

Server address (the IP of your PBS host)
Username and password (or API token)
Datastore name
Fingerprint (PBS uses a self-signed cert by default)

The fingerprint is available on the PBS dashboard or via:

# On the PBS host
proxmox-backup-manager cert info | grep Fingerprint

For a basic setup with username/password, this works out of the box. But if you're automating backup jobs or integrating with scripts, you'll want API tokens. And that's where things get interesting.

The API Token Authentication Trap

If you've worked with Proxmox API tokens before, you know PVE uses the format user@realm!tokenname with the secret passed as a separate header or parameter. PBS uses a similar but subtly different format, and the distinction will cost you hours if you don't catch it early.

The token format for PBS authentication:

# PVE token format (for reference)
user@realm!tokenname    (secret passed separately)

# PBS token format in storage config
user@realm!tokenname    (same structure, but permission model differs)

The real trap isn't the format. It's privilege separation.

When you create an API token in PBS, there's a checkbox labeled "Privilege Separation" that defaults to on. With privsep enabled, the token has its own independent permission set, completely separate from the user it belongs to. This means if your user backup@pbs has DatastoreBackup and DatastoreAudit roles on the datastore, but you created the token with privsep on and didn't assign those same roles to the token specifically, the token will authenticate successfully but return empty results or 403 errors on actual operations.

The fix:

# Create a user for backups
proxmox-backup-manager user create backup@pbs

# Create a token WITHOUT privilege separation
proxmox-backup-manager user generate-token backup@pbs pve-integration --privsep 0

# If you want privsep on (recommended for production), assign roles to the token directly
proxmox-backup-manager acl update / DatastoreBackup --auth-id backup@pbs!pve-integration

The --privsep 0 flag is the quick path for homelabs. The token inherits all permissions from its parent user. For a more locked-down setup, keep privsep on and explicitly grant the token the roles it needs. Either way, test the token before you walk away:

# Verify the token can actually list datastore contents
proxmox-backup-client list --repository 'backup@pbs!pve-integration@10.0.0.50:main-store'

If this returns an empty list (for a new datastore) or your existing backups, you're good. If it returns a 403 or permission error, check the privsep settings.

Scheduling Backup Jobs

With PBS added as a storage target in PVE, you schedule backups the same way you would any other vzdump job. Datacenter > Backup > Add. Select your PBS storage, pick the VMs and LXCs to include, set the schedule.

A reasonable starting configuration:

Schedule:     daily at 02:00
Selection:    all VMs and LXCs
Mode:         snapshot (for running machines)
Retention:    keep-last=7, keep-weekly=4, keep-monthly=3
Compression:  zstd

This gives you a week of daily recovery points, a month of weekly snapshots, and three months of monthly archives. Because PBS deduplicates, the storage cost of this retention policy is a fraction of what you'd expect.

The snapshot mode is important. It creates a consistent point-in-time snapshot without stopping the VM. For most workloads this is fine. If you're running a database directly in a VM (not in Kubernetes), consider using the stop mode or pre-freeze hooks to ensure filesystem consistency.

# You can also trigger a one-off backup via CLI
vzdump 100 --storage pbs-target --mode snapshot --compress zstd

The Stale Lock File Problem

Backup jobs will occasionally fail with an error like:

ERROR: backup of VM 101 failed - can't acquire lock '/var/lock/pve-manager/vzdump-101.lck'

This happens when a previous vzdump process was interrupted (killed, node rebooted during backup, OOM, etc.) and didn't clean up its lock file. The fix is straightforward:

# Check for stale lock files
ls -la /var/lock/pve-manager/vzdump-*.lck

# Remove the stale lock (only if no vzdump process is actually running)
ps aux | grep vzdump
# If no vzdump is running for that VMID:
rm -f /var/lock/pve-manager/vzdump-101.lck

There's a less obvious variant of this problem. On some nodes, the /var/lock/pve-manager/ directory itself can disappear after a reboot. This directory lives on a tmpfs and should be recreated by systemd-tmpfiles on boot. If it's missing:

# Recreate the lock directory
mkdir -p /var/lock/pve-manager

To make this persistent, verify that the tmpfiles configuration includes it:

# Check if the config exists
cat /usr/lib/tmpfiles.d/pve-manager.conf
# Should contain a line like:
# d /var/lock/pve-manager 0755 root root -

If that file is missing or doesn't include the lock directory, create a drop-in:

echo 'd /var/lock/pve-manager 0755 root root -' > /etc/tmpfiles.d/pve-manager.conf
systemd-tmpfiles --create

Backup Verification

PBS has a built-in verification system that reads back every chunk in a backup and checks its integrity. This is the feature that separates "I have backups" from "I have backups I can actually restore from."

Schedule verification jobs in the PBS web UI under Datastore > Verify Jobs. A good cadence is to verify the most recent backup daily and do a full verification of all backups weekly. Verification is I/O intensive but doesn't affect PVE operations since it runs on the PBS host.

# Manual verification via CLI
proxmox-backup-client verify --repository 'backup@pbs@10.0.0.50:main-store'

If verification fails for a specific snapshot, PBS will flag it in the UI. Don't ignore these warnings. A failed verification means that backup may not be restorable.

PBS in the Context of a Full 3-2-1 Strategy

PBS handles one layer of your backup stack: the hypervisor layer. VMs and LXCs, their disks, their configs. But if you're running Kubernetes on top of those VMs, there's application-level state that PBS backs up only indirectly.

Consider the layers:

Layer	What It Contains	Backup Tool	Recovery Speed
Hypervisor	VM disks, LXC rootfs, configs	PBS	Full VM restore in minutes
Kubernetes	PV data, etcd, secrets	Velero + MinIO	Namespace-level restore
GitOps	Manifests, Helm values, configs	Git (ArgoCD)	Re-sync from repo

PBS gives you the "bare metal to running VMs" recovery path. If a node dies, you restore the VMs to another node and they come up exactly as they were. But the Kubernetes workloads inside those VMs have their own state (persistent volumes, databases, application data) that benefits from Velero-level backups running in parallel.

The combination is what makes 3-2-1 actually work:

Three copies: live data + PBS backup + offsite copy (Synology, cloud bucket, second PBS instance)
Two media types: local SSD/NVMe (live) + HDD (PBS datastore)
One offsite: PBS supports built-in sync to a remote PBS instance, or you can replicate the datastore to a NAS for geographic separation

For the GitOps layer, ArgoCD already handles the "config as code" part. You don't need to back up Kubernetes manifests the traditional way because they're already in Git. What you need to back up is the state that isn't in Git: persistent volumes, database contents, secrets.

Garbage Collection and Datastore Maintenance

PBS deduplicates by storing data as content-addressed chunks. When you prune old backups, the chunks aren't immediately deleted. They become unreferenced. Garbage collection (GC) is the process that identifies and removes unreferenced chunks to reclaim disk space.

GC runs on a schedule within PBS. The default is usually fine, but keep an eye on the "Deduplication Factor" metric in the PBS dashboard. For a homelab with similar VMs (same base OS, similar packages), you'll typically see dedup factors between 3x and 8x. That means your backups are using 3-8x less space than the raw data size.

# Check datastore status including dedup factor
proxmox-backup-manager datastore list

# Manually trigger garbage collection
proxmox-backup-manager garbage-collection start main-store

If your dedup factor is close to 1x, something is off. Either your VMs have very little data in common (unlikely if they're running the same distro), or the chunk size configuration isn't optimal for your workload.

Monitoring Backup Health

PBS exposes metrics that you can pull into Grafana or any monitoring stack. The key things to watch:

Last backup timestamp per VM/LXC: if a backup hasn't run in 24+ hours, something is broken
Backup duration trends: a backup that used to take 10 minutes and now takes 60 suggests disk issues or unexpected data growth
Verification status: any failed verifications need immediate attention
Datastore usage: track the growth rate to predict when you'll need more storage

A simple monitoring approach is a cron job that checks for recent backups:

#!/bin/bash
# Check that every VM has a backup from the last 24 hours
CUTOFF=$(date -d '24 hours ago' +%s)

proxmox-backup-client list \
  --repository 'backup@pbs@10.0.0.50:main-store' \
  --output-format json | \
  jq -r '.[] | select(.backup_time < '$CUTOFF') | .backup_id' | \
  while read vm; do
    echo "WARNING: $vm has no backup in the last 24 hours"
  done

Lessons Learned

Test restores, not just backups. At least once a quarter, pick a VM and restore it to a temporary location. Verify it boots, verify the data is intact. A backup system you've never restored from is a hypothesis, not a strategy.

Privilege separation on API tokens is the silent killer. If your automated backups authenticate fine but return empty data or permission errors on operations, check privsep. This one issue probably accounts for half the "PBS isn't working" posts on the Proxmox forums.

Separate your failure domains. PBS running as a VM on the cluster it's backing up is better than no backups, but only barely. The whole point of backups is surviving hardware failure. A dedicated, physically separate PBS host (even a cheap one) fundamentally changes your recovery posture.

PBS handles the hypervisor layer, not the application layer. If you're running Kubernetes, you still need something like Velero for PV snapshots and namespace-level restores. PBS gives you "get back to running VMs." Velero gives you "get back to running applications." Both are necessary. Building a production homelab is only half the work if you don't have a plan for when things go wrong.

Deduplication makes aggressive retention policies cheap. Don't be stingy with retention. The marginal cost of keeping an extra month of weekly snapshots is tiny after dedup. The value of having that three-month-old snapshot when you discover slow data corruption is enormous.

Lock file issues are operational, not architectural. They're annoying, but they're just stale state from interrupted processes. Know where the lock files live, know how to check if a vzdump is actually running, and clean up when needed. Don't let a stuck lock file make you think PBS itself is broken.

When Agents Should Stop: Designing Safety Boundaries That Work

Guatu — Mon, 15 Jun 2026 12:38:16 +0000

An agent in my homelab posted "HEARTBEAT_OK" to the ops channel 47 times over one weekend. Every message was technically correct. The scheduled jobs were healthy, the agent verified them, and it reported in exactly like it was told to. By Monday morning I had muted the channel, which meant the one message that mattered (a failed backup verification) scrolled past unread sometime around 3 AM.

That incident wasn't an alignment problem or a runaway loop. It was a stopping problem. The agent had no concept of "nothing to say," so it said something every time it woke up. Most agent safety writing focuses on preventing harmful actions. In practice, the boundary I've had to engineer most carefully is more mundane: teaching agents when to do nothing and exit quietly.

If you run scheduled agents, autonomous loops, or anything where an LLM makes decisions on a timer, this is for you. The patterns below come from running multi-agent pipelines on my own infrastructure, and from the specific ways they've failed. I covered the theory in Three-Layer Safety for Autonomous Agents; this post is the operational follow-up, the part where theory meets a crash-looping gateway at 2 AM.

Stopping is a feature, not a failure state

The core mistake I made early on: treating an agent that stops as an agent that failed. My first orchestration scripts retried everything. Agent exits without completing the task? Retry. Agent says it's blocked? Rephrase the prompt and retry. The result was agents that burned tokens grinding against problems they'd already correctly identified as unsolvable from inside the loop.

What fixed it was giving agents a vocabulary for stopping. Mine boils down to three boundary types, all enforced outside the model:

Budget boundaries cap what an agent can spend: iterations, tokens, wall-clock time. These are the easy ones, and most frameworks give you something here. The mistake is setting them as emergency brakes (high enough that they never trigger) instead of as scoping decisions. If a task should take 3 iterations, cap it at 5, not 50. A cap that triggers at 50 means you've already wasted 45 iterations of spend before learning anything. I also set budgets per stage rather than per pipeline: a global 30-minute cap on a five-stage pipeline tells you nothing about which stage ran away.

Progress boundaries detect when the agent is still spending but no longer changing anything. This is the infinite-loop killer, and it's the one almost nobody implements. An agent can stay under every budget cap while making zero progress: rewriting the same file back and forth, re-running the same failing test with cosmetic tweaks. You detect this by hashing the observable state between iterations and stopping when the hash stops changing.

Reporting boundaries define when the agent is allowed to speak. This is the HEARTBEAT_OK lesson: an agent that reports success on every run trains humans to ignore it. Silence on success, noise on failure. The inversion matters more than it looks.

The configs

Progress detection is the highest-value boundary, so start there. The wrapper below runs an agent task in a loop and kills it when two consecutive iterations produce identical state:

#!/usr/bin/env bash
# agent-loop.sh: run an agent task with hard stop conditions
MAX_ITERATIONS=5
previous_state=""

for i in $(seq 1 "$MAX_ITERATIONS"); do
  run_agent_iteration "$TASK_FILE"   # your agent invocation here

  # Hash everything the agent can change: working tree + state dir
  current_state=$(
    { git diff; git status --porcelain; cat state/*.json 2>/dev/null; } \
    | sha256sum | cut -d' ' -f1
  )

  if [[ "$current_state" == "$previous_state" ]]; then
    echo "iteration $i produced no state change, stopping" >&2
    exit 2   # stopped at boundary, not failed
  fi
  previous_state="$current_state"

  task_complete && exit 0
done

echo "hit iteration cap ($MAX_ITERATIONS) without completing" >&2
exit 2

Exit code 2 is doing real work there. I use a three-value contract for every agent wrapper:

0 = done: task complete, verified
1 = failed: something broke, a human needs to look
2 = stopped: hit a boundary with partial progress, safe to resume

The distinction between 1 and 2 is the whole point. A failure pages someone. A boundary stop writes a state file and waits for the next scheduled run, which picks up where the last one left off. Collapsing those into one exit code gives you either alert fatigue or silent data loss, depending on which direction you collapse them.

Notice that all of this lives in the wrapper, not in the prompt. You can (and should) tell the agent about its budget in the prompt, because a model that knows it has two iterations left plans differently. But the prompt is advice. The wrapper is the boundary.

Reporting boundaries live in the scheduler config. Here's the shape I use for scheduled agent jobs after the heartbeat incident:

{
  "name": "nightly-health-check",
  "schedule": "0 6 * * *",
  "task": "Verify backup jobs completed and volumes are healthy.",
  "notify": {
    "on_success": "silent",
    "on_failure": "channel:#ops",
    "on_boundary_stop": "channel:#ops-low"
  },
  "deadman": "https://hc.example.com/ping/nightly-health"
}

Two things to notice. Success is silent: the channel only gets a message when something needs a human. And the deadman URL replaces the heartbeat message entirely: instead of the agent telling humans "I'm alive," it pings a dead-man's-switch endpoint (Healthchecks.io, or any self-hosted equivalent) that alerts only when the ping stops arriving. Machines are good at noticing absence. Humans are terrible at it. Route the liveness signal to the machine and the failure signal to the human.

Gotcha 1: silence can hide breakage

About a month after I made my agents quiet on success, a memory MCP server's tools started failing silently. Calls returned empty results instead of errors. The agents treated "no results" as "nothing to report" and exited cleanly, status 0, for eleven days. From the outside everything looked healthy: exit codes were green and the dead-man pings kept arriving, because the agent itself was running fine. Only the tools inside it were broken.

The lesson: "silence on success" requires verifying success, not just the absence of an exception. My health-check agents now end every run with an assertion phase that demands positive evidence:

# Don't trust "no errors". Demand proof of work.
results=$(query_memory_store "test-canary-record")
if [[ -z "$results" ]]; then
  echo "canary record missing: memory store is lying to us" >&2
  exit 1
fi

Plant a canary record you know exists, and fail loudly if the tooling can't find it. A tool that fails silently turns every downstream stop condition into a lie, because the agent is deciding "nothing to do" based on data it never received.

Gotcha 2: validate config before the gateway eats it

Stop conditions usually live in config files, which means they inherit every config-deployment failure mode. I learned this when I added a plausible-looking concurrency cap to an agent gateway's config. The key didn't exist in the schema. Older versions ignored unknown keys; the version I was running had switched to strict validation and rejected the whole file. The gateway crash-looped on restart, taking every scheduled agent down with it, including the ones whose job was to report that things were down.

Strict validation is the right behavior (a typo'd max_iteratons silently ignored is a budget cap that doesn't exist), but it means you treat agent config like any other production config: validate before reload, never after.

# Never restart a gateway on unvalidated config
agentctl validate --config /etc/agent/gateway.json || {
  echo "config invalid, refusing to restart" >&2
  exit 1
}
systemctl restart agent-gateway

If your agent platform ships a doctor or validate subcommand, wire it into the deploy path and make the restart conditional on it passing. If it doesn't ship one, a JSON Schema check in CI is twenty minutes of work and saves you a crash-looped orchestrator. Same idea as validating Kubernetes manifests before merge, just pointed at your agent stack.

Gotcha 3: a stopped agent must leave a note

Early versions of my boundary stops just exited. The next scheduled run started from scratch, re-derived the same context, hit the same boundary, and exited again. Functionally an infinite loop, just with a 24-hour period and a cron job in the middle.

Now every boundary stop writes a handoff file before exiting:

{
  "stopped_at": "2026-06-08T03:12:44Z",
  "reason": "no_progress",
  "iterations_used": 4,
  "progress_summary": "Identified failing PVC, replica rebuild blocked on node disk pressure",
  "blocking_on": "needs human: node disk cleanup or replica eviction",
  "resume_hint": "check node disk usage before retrying"
}

The next run reads the handoff first. If blocking_on names a human action and nothing in the environment has changed, it exits immediately at near-zero cost instead of re-deriving the same dead end. When the blocker clears, it resumes from the summary instead of from nothing. This one file turned boundary stops from an expensive pause into an actual checkpoint mechanism.

What I considered and rejected

Letting the model decide when to stop. Tempting, because the model often knows it's stuck. But a stop condition that lives inside the thing being bounded isn't a boundary, it's a suggestion. Models are also systematically optimistic that one more iteration will help. I let agents request an early stop (which short-circuits the loop), but enforcement stays in the wrapper, outside the model's reach.

Confidence thresholds. Some frameworks stop when the model's self-reported confidence drops below a cutoff. I tried it; self-reported confidence was noise, uncorrelated with whether the next iteration helped. The state-hash check costs one sha256sum and doesn't depend on the model grading its own homework.

Watchdog agents. A second agent that monitors the first and decides whether to kill it. This works, and for high-stakes pipelines I still use a reviewer stage (the pattern shows up in Multi-Agent AI Systems: Architecture Patterns That Actually Work). But as a stop mechanism it's expensive and introduces a new question: who stops the watchdog? Deterministic boundaries in the wrapper give you 90% of the value at roughly zero marginal cost.

Where this lands

Stopping is the cheapest safety mechanism you have, and it's the one most agent deployments skip because it doesn't feel like a feature. Nobody demos an agent exiting cleanly. But the boundaries above have prevented more incidents on my cluster than any prompt-engineering guardrail I've written: budget caps treated as scoping decisions instead of emergency brakes, state-hash progress detection, the 0/1/2 exit contract, silent success paired with loud failure and a machine-checked dead-man switch, and handoff files so a stop is a checkpoint instead of a discard.

Reach for this the moment any agent runs without a human watching: scheduled jobs, overnight batch pipelines, CI agents. Building agent systems that run unattended against real infrastructure is part of what I help teams with at GuatuLabs, and stop-condition design is reliably the piece nobody thought about before the first incident. If your ops channel has a recurring message in it right now that everyone has learned to scroll past, that's not a reporting feature. That's a stop condition nobody designed.

Network Policies with Calico: Default Deny and Namespace Isolation

Guatu — Mon, 15 Jun 2026 12:38:04 +0000

A default-deny NetworkPolicy is five lines of spec. Those five lines will also kill DNS resolution for every pod they select, because an egress deny blocks UDP packets to kube-dns just as happily as it blocks the traffic you were actually worried about. The distance between "I understand network policies" and "I rolled out default deny without an outage" is mostly three blind spots: DNS, your ingress controller, and admission webhooks.

Out of the box, Kubernetes runs a flat pod network. Every pod can open a connection to every other pod in the cluster, across namespaces, no questions asked. If you've already done the work of building least-privilege service accounts, a flat network is the same problem one layer down: identity is locked tight while the network is wide open. This post is about closing that gap with Calico on a bare-metal cluster (K8s 1.31, Calico 3.x), in an order that doesn't take the cluster down while you do it.

One prerequisite worth stating plainly: the NetworkPolicy API objects exist in every cluster, but they do nothing unless your CNI enforces them. Calico does. If you're on a CNI without policy support, you can apply these manifests all day and traffic flows anyway, which is its own special category of false confidence.

The rollout that looks right and isn't

The tempting approach goes like this: write one default-deny policy, template it across every namespace, apply, done. Security checkbox ticked before lunch.

Here's the policy everyone starts with:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: team-a
spec:
  podSelector: {}        # selects every pod in the namespace
  policyTypes:
    - Ingress
    - Egress

The empty podSelector selects all pods, and listing both policy types makes them isolated in both directions. Correct, minimal, and the moment it lands cluster-wide, three things break in a predictable order.

Failure one: DNS dies first, and it dies slowly

Every pod in a selected namespace loses the ability to resolve names, because queries to kube-dns in kube-system are egress traffic like any other. The nasty part is the failure mode. Connections to a denied endpoint fail fast with a timeout you'll notice. DNS failures look different: each lookup waits out a 5-second timeout per attempt, multiplied by the search domain list your ndots config generates. Apps get slow before they get broken, which sends you debugging application performance instead of network policy. I wrote about how the search domain expansion amplifies this in the ndots:5 post; default deny turns every one of those expanded lookups into a 5-second black hole.

Failure two: your ingress controller can't reach anything

Traffic from Traefik or ingress-nginx to your backend pods is just pod-to-pod traffic crossing a namespace boundary. Default deny on the application namespace blocks it, and every service behind the ingress starts returning 502s and 504s. The application pods are healthy, the Service endpoints are populated, readiness probes pass (kubelet probes come from the node, and Calico permits them). Everything looks green except the part where users reach it. This also bites cert-manager: an HTTP-01 challenge needs the ingress controller to reach the temporary solver pod, so default deny can silently stall certificate issuance long after the initial rollout.

Failure three: the webhook deadlock

This is the one that turns a degraded cluster into a stuck one. Admission webhooks (Kyverno, cert-manager's webhook, anything with a ValidatingWebhookConfiguration) receive calls from the API server. Deny ingress to the webhook pod and those calls time out. With failurePolicy: Fail, the API server now rejects the operations that webhook gates, and the trap closes: the NetworkPolicy you're trying to apply to fix the problem is itself an API operation that flows through admission. You're locked out of the fix by the thing you broke.

It gets worse if the policies are managed by automation. With a Kyverno generate rule or a GitOps controller syncing the policy, deleting the offending NetworkPolicy by hand buys you a few seconds before it's regenerated. You end up playing whack-a-mole against your own reconciliation loop while the cluster burns. The escape hatch is to pause the automation first (scale down Kyverno, disable ArgoCD auto-sync for that app), then remove the policy.

A detail that matters here: API server traffic to webhooks often originates from the control plane host network, not from a pod you can match with a podSelector. Allowing it means an ipBlock rule for your control plane CIDR, or excluding webhook namespaces from default deny entirely. I do the latter.

A rollout order that works

The fix for all three failures is the same discipline: never apply a deny you haven't already written the allows for, and never apply it wider than you can watch.

Step 1: one namespace, not the cluster

Pick a single application namespace with low blast radius. Resist the urge to start cluster-wide; the whole point of the first namespace is to discover the flows you forgot existed. kubectl get networkpolicy -A should stay boring while you learn.

Step 2: the baseline trio

Default deny ships as a set of three policies applied together, in one kubectl apply -f of one directory. The deny:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: team-a
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]

The DNS allow, which goes everywhere the deny goes, no exceptions:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: team-a
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Both protocols matter. DNS falls back to TCP for large responses, and an egress rule that only allows UDP produces intermittent failures that are miserable to track down.

The intra-namespace and ingress-controller allow:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-baseline-ingress
  namespace: team-a
spec:
  podSelector: {}
  policyTypes: [Ingress]
  ingress:
    # any pod in this same namespace
    - from:
        - podSelector: {}
    # everything in the ingress controller's namespace
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress

That kubernetes.io/metadata.name label is the load-bearing trick here. Since K8s 1.22, every namespace carries it automatically with its own name as the value, which gives you a stable way to select namespaces without inventing and maintaining your own labeling scheme.

With the trio applied, check behavior from inside the namespace before moving on:

# throwaway pod inside the locked-down namespace
kubectl -n team-a run probe --rm -it --image=busybox:1.36 --restart=Never -- sh
# inside the pod:
nslookup kubernetes.default                                  # should answer instantly
wget -qO- -T 2 http://api.team-b.svc.cluster.local           # should time out

Fast DNS plus a slow, eventually-failing cross-namespace connection is the signature of a healthy baseline. Instant DNS failure means the allow-dns policy didn't land; an instant cross-namespace success means the deny didn't.

Step 3: log before you deny

Calico's Log rule action is the visibility tool the vanilla NetworkPolicy API doesn't have. Before tightening further, I put a logging policy behind the allows so I can see what the deny is about to catch:

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: log-unmatched
spec:
  order: 4000                    # evaluated after everything else
  namespaceSelector: projectcalico.org/name == 'team-a'
  types: [Ingress, Egress]
  ingress:
    - action: Log
  egress:
    - action: Log

With the iptables dataplane, Log uses the kernel LOG target, so dropped-candidate packets show up in the kernel log with a calico-packet: prefix (configurable via logPrefix in FelixConfiguration):

journalctl -k --grep calico-packet

Two caveats. Kernel logging is noisy, so treat this as a diagnostic you enable for hours, not a permanent fixture. And the eBPF dataplane doesn't support the Log action, so if you've switched dataplanes this tool isn't available.

This step is where "set and forget" turns into something closer to auditing. Run a logging policy for a day against a namespace before enforcing, and you find the flows nobody documented: the metrics scraper, the backup job, the sidecar that phones a service in another namespace.

One class of flow deserves special mention: anything running with hostNetwork: true. Node-level monitoring agents and some bare-metal ingress deployments source their traffic from the node's IP, not a pod IP, so podSelector and namespaceSelector rules never match them. If scraping or health checks break only after enforcement, this is usually why, and the fix is an ipBlock rule covering your node CIDR rather than another selector you'll fight with.

Step 4: the cluster-wide backstop

Once the per-namespace pattern is proven, Calico's GlobalNetworkPolicy enforces namespace isolation as a guardrail across every tenant namespace at once, with infrastructure explicitly carved out:

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: tenant-isolation-backstop
spec:
  order: 3000
  namespaceSelector: >-
    projectcalico.org/name not in
    {"kube-system", "calico-system", "calico-apiserver",
     "ingress", "argocd", "cert-manager", "kyverno"}
  types: [Ingress, Egress]
  egress:
    # DNS keeps working even where namespace policies are missing
    - action: Allow
      protocol: UDP
      destination:
        selector: k8s-app == 'kube-dns'
        ports: [53]
    - action: Allow
      protocol: TCP
      destination:
        selector: k8s-app == 'kube-dns'
        ports: [53]

No explicit Deny rule, and that's deliberate. In Calico, when at least one policy selects an endpoint and no rule allows the packet, the packet is dropped at the end of evaluation. The backstop selects everything outside the exclusion list, allows DNS, and lets the implicit deny do the rest.

The order: 3000 is doing real work. Calico assigns Kubernetes NetworkPolicies an order of 1000, and lower order means earlier evaluation. An allow in a namespace's own policy terminates evaluation before the backstop is ever consulted. The backstop only catches traffic nothing else has claimed, which means namespaces with proper policies behave per their policies, and namespaces without any get isolation by default instead of the flat network.

That exclusion list is the "infrastructure exclusion" pattern, and I'd argue it's the single most important decision in the whole rollout. The namespaces that run your CNI, your ingress, your GitOps controller, and your admission webhooks are the namespaces where a policy mistake costs you the ability to fix policy mistakes. Leave them out of automated enforcement. Write their policies by hand, later, one at a time, with the logging step in between.

Step 5: automate generation, with the same exclusions

For new namespaces, a Kyverno generate rule stamps the baseline trio in automatically:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: generate-default-deny
spec:
  rules:
    - name: default-deny
      match:
        any:
          - resources:
              kinds: [Namespace]
      exclude:
        any:
          - resources:
              kinds: [Namespace]
              names: ["kube-system", "kube-public", "kube-node-lease",
                      "calico-system", "ingress", "argocd", "kyverno"]
      generate:
        apiVersion: networking.k8s.io/v1
        kind: NetworkPolicy
        name: default-deny
        namespace: "{{request.object.metadata.name}}"
        synchronize: true
        data:
          spec:
            podSelector: {}
            policyTypes: [Ingress, Egress]

Two operational notes. synchronize: true is what creates the regeneration loop from failure three: hand-deleting the generated policy gets it recreated within seconds, so during an incident you pause the ClusterPolicy before touching its output. And Kyverno treats generate rules as effectively immutable: if the generated resource definition is wrong, plan on deleting and recreating the ClusterPolicy rather than patching it in place.

Why this works

The mental model that makes all of this predictable: Kubernetes NetworkPolicies are additive allow-lists with an implicit deny that activates the moment any policy selects a pod. There is no deny rule in the vanilla API. A pod selected by zero policies accepts everything; a pod selected by any policy accepts only what the union of matching policies allows. That's why the baseline trio works as a set: the deny policy flips the pod into isolated mode, and the other two define the allowed surface.

Calico layers an ordered evaluation model on top. Policies are sorted by order, rules within a policy run top to bottom, and the first Allow or Deny terminates evaluation. Kubernetes-native policies slot in at order 1000 (you can see the converted versions with calicoctl get networkpolicy --all-namespaces, prefixed knp.default.). Pods matched by no policy at all fall through to Calico's per-namespace profiles, which default to allow. That layering is exactly what makes the backstop-at-3000 pattern safe: specific intent at 1000 wins, the guardrail catches the remainder, and the logging policy at 4000 sees only what's about to die.

Felix, Calico's per-node agent, also quietly saves you from the worst self-own. Its failsafe port list (SSH on 22, the API server on 6443, BGP on 179, etcd, Typha) is exempt from policy on host endpoints by default, so a bad policy can break your workloads without also locking you out of the nodes you need to fix it from. Don't shrink that list without a very specific reason.

Lessons learned

The failure modes are knowable in advance. DNS, ingress, and webhooks fail in that order every time, and writing the allows before the deny is cheaper in every way than discovering them from a monitoring graph. If a rollout plan doesn't mention kube-dns, port 53, or failurePolicy, it isn't done.

Namespace-by-namespace beats cluster-wide, even though it feels slower. The first namespace takes a day because you're discovering undocumented flows. The tenth takes ten minutes because there's nothing left to discover. Going cluster-wide first inverts that: you discover everything at once, in production, with automation re-applying the breakage faster than you can remove it.

Exclude infrastructure from automation permanently, not temporarily. Every system that can generate or sync policies (Kyverno, ArgoCD, your own scripts) should carry the same exclusion list for kube-system, the CNI namespace, ingress, GitOps, and webhook namespaces. The asymmetry is stark: a missing policy in those namespaces costs you some security posture, while a wrong policy there costs you the control plane's ability to accept the fix.

Logging is the difference between policy as guesswork and policy as engineering. The Log action is crude (kernel log lines, iptables dataplane only), but it converts "why is this connection failing" from a hypothesis into a grep. I'd take crude visibility over elegant blindness in any network debugging session. This pattern, restrict by default and watch the boundary, is the same shape as the guardrails I build around autonomous agent infrastructure: the deny is easy, and the engineering is in the observability that tells you what the deny will cost before you pay it.

The thing the docs undersell is that default deny is a migration, not a manifest. The YAML is trivial. The work is the inventory of flows your cluster actually depends on, and you only get that inventory by watching one namespace at a time with the logs on.

Velero + MinIO: Kubernetes Backup Strategy for Bare Metal

Guatu — Wed, 10 Jun 2026 12:15:14 +0000

I spent three hours staring at a PartiallyFailed status in Velero, wondering why my backups were failing despite the logs claiming the S3 connection was healthy. The culprit wasn't the network or the credentials. It was a handful of NFS-backed persistent volumes that Velero was trying to snapshot using a CSI driver that didn't support them.

If you're running Kubernetes on bare metal, you don't have the luxury of a "managed" backup service. You have to build the storage backend, the orchestration layer, and the recovery path yourself. Most of the documentation assumes you're pushing to AWS S3, but when you're running your own hardware, that's usually not the goal. You want your data on your own disks, under your own control.

The False Starts

My first attempt was naive. I thought I could just install Velero, point it at a MinIO instance running inside the same cluster, and call it a day. This was a mistake for two reasons.

First, backing up a cluster to a storage provider running inside that same cluster is a circular dependency. If the cluster goes down, your backups are gone. I quickly moved MinIO to a separate set of machines to ensure the backup target lived outside the blast radius of the Kubernetes API.

Second, I relied entirely on the "happy path" of CSI snapshots. I assumed that because I was using Longhorn for most of my stateful workloads, everything would just work. I forgot that I had a few legacy NFS mounts for shared configuration files. Velero tried to trigger a CSI snapshot on those NFS volumes, failed, and marked the entire backup as PartiallyFailed. I spent an hour chasing "S3 timeout" errors when the real issue was a storage class mismatch.

I also tried using the default Velero installation without specifying the S3 URL explicitly in the environment variables of the pod. I assumed the plugin would magically find MinIO if the credentials were correct. It didn't. I ended up with a loop of 403 Forbidden errors because Velero was trying to hit the actual AWS S3 endpoints instead of my local MinIO instance.

The Actual Solution

To get a reliable bare-metal backup strategy, you need three distinct layers: the S3-compatible target (MinIO), the orchestrator (Velero), and the control plane safety net (ETCD snapshots).

1. The Storage Backend (MinIO)

I run MinIO on a separate set of bare-metal nodes. For the sake of this setup, I've created a dedicated bucket called k8s-backups and a specific service account with read/write access to that bucket.

Running MinIO outside the cluster is non-negotiable. If you have a power failure on your K8s rack and your backups are on the same rack, you haven't built a backup system: you've just built a very expensive way to lose your data twice.

2. Installing Velero with MinIO

The trick here is the AWS plugin. Since MinIO uses the S3 API, we use the AWS provider but override the endpoint to point to the local MinIO server.

I used the following command to deploy Velero 1.14 on K8s 1.31:

velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.14.0 \
  --bucket k8s-backups \
  --secret-file ./credentials-velero \
  --use-volume-snapshots=true \
  --backup-destination-type=s3 \
  --s3-url http://minio.example.com:9000 \
  --namespace velero

The credentials-velero file is a standard AWS credentials format. To keep these secure and avoid committing them to Git, I use SealedSecrets to manage the secrets across my environments.

If you're deploying this via GitOps, I highly recommend using the official Helm chart but overriding the configuration.s3Url value. This ensures that when you scale your cluster or move nodes, the backup configuration remains consistent.

3. Handling the "PartiallyFailed" Nightmare

To stop Velero from trying to snapshot volumes that don't support it (like NFS), I had to be explicit. Labeling volumes to exclude them is a start, but the most effective way to handle a mixed-storage environment is to patch the backup schedule to ignore volume snapshots for specific workloads or to use Restic/Kopia for file-level backups.

If you have a schedule that keeps failing due to incompatible PVs, you can disable snapshot volumes for that specific schedule:

kubectl patch schedule daily-cluster-backup -n velero \
  --type=merge \
  -p '{"spec":{"template":{"snapshotVolumes":false}}}'

For the volumes that actually need backing up (like my Longhorn volumes), I rely on the Longhorn integration, which allows Velero to trigger native Longhorn snapshots.

4. The ETCD Safety Net

Velero is great for resources and PVs, but if your ETCD cluster completely collapses, you're in for a bad time. I don't trust a single tool for the control plane. I implemented a systemd timer on the control plane nodes to take raw ETCD snapshots every 24 hours.

I use this unit file to handle the snapshot and a basic retention policy:

[Unit]
Description=ETCD Snapshot Backup
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/bin/etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /var/backups/etcd/etcd-snapshot-$(date +%Y%m%d).db
ExecStartPost=/bin/sh -c '/usr/bin/find /var/backups/etcd -type f -name "etcd-snapshot-*.db" -mtime +7 -exec rm -f {} \;'

I then use a simple cron job to rsync these .db files to the MinIO server. This gives me a raw binary backup of the cluster state that is completely independent of the Velero operator.

Troubleshooting the Gap

When things go wrong with Velero and MinIO, the errors are rarely helpful. You'll see Backup failed in the high-level status, but the real gold is in the pod logs.

The "S3 Endpoint" Trap

If you see failed to get object: NoSuchBucket or 403 Forbidden despite having the right keys, check if Velero is actually hitting your MinIO server. Run:

kubectl logs -n velero deployment/velero

If you see requests going to s3.amazonaws.com, your --s3-url flag was ignored or overridden. This often happens when using Helm charts where the configuration block isn't properly mapped to the deployment arguments.

Restic Metadata Corruption

I hit a specific wall when I changed the bucket name in MinIO. I updated the Velero config, but my file-level backups (using Restic) started failing with:
error: repository is not initialized

Restic stores metadata in the bucket itself. If you move buckets, you can't just point Velero to the new one; you have to migrate the restic repository or re-initialize it. I learned the hard way that Restic is less flexible than CSI snapshots for backend migration.

CSI Snapshot Timeouts

In a multi-node Proxmox setup, I noticed that some backups would hang at the "snapshotting" phase. After digging into the Longhorn logs, I found that the snapshot was being created, but the CSI driver was timing out while waiting for the volume to reach a consistent state. The fix was increasing the snapshotTimeout in the Velero configuration to 10 minutes, giving the storage layer enough breathing room to finalize the snapshot on larger volumes.

Deep Dive: Why This Architecture Works

This architecture works because it acknowledges the reality of bare metal: things fail in ways the cloud hides from you.

By using MinIO as an S3-compatible layer, I get the industry-standard API that Velero expects, but I keep the data on my own hardware. This removes the egress costs and latency associated with pushing terabytes of snapshot data to a public cloud provider.

By separating the ETCD backups from the Velero backups, I've created two different recovery paths. If the Velero operator is broken, I can still restore ETCD to bring the API server back online. If the ETCD data is corrupted but the API is alive, I can use Velero to restore specific namespaces without nuking the entire cluster.

The decision to use snapshotVolumes: false on specific schedules is a pragmatic trade-off. I'd rather have a "successful" backup of my YAML manifests and secrets than a "partially failed" backup that tries (and fails) to snapshot a read-only NFS mount. I handle the NFS data separately via a simple tar and rsync pipeline.

Operational Lessons

If I were to do this again from scratch, I would change a few things:

Avoid MinIO in the same rack. I have my MinIO nodes in a different physical power circuit. If a PDU fails, I don't want my backup target to go dark at the same time as my cluster.
Use Kopia over Restic. Velero has started supporting Kopia, which is generally faster and handles deduplication more efficiently. If you're starting fresh, go with Kopia.
Automate Restore Tests. A backup is just a theoretical exercise until you've successfully restored it. I now run a monthly "fire drill" where I spin up a temporary single-node cluster and attempt to restore a single non-critical namespace from the MinIO bucket.

The biggest surprise was how much the "small things" matter. A missing s3-url flag or a slightly misconfigured systemd timer can be the difference between a 10-minute recovery and a weekend spent rebuilding a cluster from Git manifests.

For those building complex AI agent pipelines or industrial IoT systems, this level of redundancy is mandatory. When your agents are managing state across multiple databases and vector stores, a simple "git clone" of your manifests isn't a backup strategy. You need a consistent snapshot of the entire state, and Velero + MinIO is the most reliable way to achieve that on bare metal.

Agent Glass-Break Patterns: Controlled Escalation for Production

Guatu — Wed, 10 Jun 2026 10:15:27 +0000

I watched an autonomous ops agent attempt to "fix" a failing deployment by recursively deleting pods in a loop because it misinterpreted a CrashLoopBackOff as a transient networking glitch. The agent had the permissions to do it, the logic to justify it, and absolutely no circuit breaker to stop it from taking down the entire namespace. It was a classic case of giving a tool a hammer and watching it treat the entire infrastructure like a nail.

If you're running agents in production, you've probably realized that the standard "system prompt" safety is a joke. Telling an LLM "please be careful with the production database" is not a security boundary. You need a glass-break pattern: a way for agents to operate within a strict sandbox, but with a controlled, audited path to escalate privileges when a human approves it or a specific condition is met.

What I tried first

My first instinct was to lean on centralized identity. I tried routing every agent tool call through an Authentik-protected gateway. The idea was simple: the agent requests a tool, the gateway checks the session, and the action is authorized.

It was a nightmare. The latency added by the OIDC handshake for every single tool call made the agent feel sluggish, and the integration overhead for low-sensitivity observability tools was absurd. I spent more time debugging JWT expiration and redirect loops than actually building agent capabilities. I was treating a low-sensitivity internal tool like a public-facing enterprise application.

Then I tried the "Super-User" approach. I gave the agent a high-privilege service account but wrapped it in a complex set of Python decorators that checked for "safe" keywords in the arguments. This failed immediately. LLMs are too good at prompt injection and parameter manipulation. A simple --force flag or a clever string concatenation bypassed my "safety" filters in minutes.

The Actual Solution: Controlled Escalation

The fix was to move the security boundary from the application layer to the infrastructure and execution layer. I implemented a three-pronged approach: Network-level isolation for internal tools, safeBins for execution control, and a manual escalation trigger.

1. Infrastructure-Level Isolation

Instead of forcing every internal tool through a heavy auth layer, I shifted to a LAN-only access model using Kubernetes NetworkPolicy. This ensures that only the agent orchestrator can talk to the tool, and only from a specific subnet.

For a tool like Agent Quest, I stripped out the Authentik dependency and locked it down at the pod level:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: traefik-allow-egress-to-agentquest
spec:
  podSelector:
    matchLabels:
      app: traefik
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 10.0.0.140/32 # The specific IP of the Agent Quest service
    ports:
    - protocol: TCP
      port: 4444

This removes the auth overhead while ensuring that no one outside the cluster (or even in other namespaces) can trigger the tool. It aligns with the privacy-routed inference pattern of keeping sensitive traffic off the open wire.

2. Execution Control with safeBins

For tools that actually execute shell commands, like mcporter, I stopped relying on regex filters. I implemented a safeBins pattern. This is essentially an allowlist of binaries and the specific flags they are permitted to use.

If the agent tries to pass a flag not in the allowedValueFlags list, the execution engine kills the process before it ever hits the shell.

{
  "safeBins": {
    "mcporter": {
      "allowedValueFlags": ["--config", "--timeout"],
      "forbiddenFlags": ["--force", "--recursive", "--delete-all"]
    },
    "kubectl": {
      "allowedValueFlags": ["--dry-run=client", "-n"],
      "restrictedCommands": ["delete", "patch"]
    }
  }
}

This forces the agent to operate in a "read-only" or "safe-write" mode by default. If the agent needs to do something destructive, it cannot simply "decide" to do it; it must trigger the glass-break.

3. The Glass-Break Escalation

When the agent hits a safeBins restriction or a NetworkPolicy block, it triggers an escalation event. I integrated this with an n8n workflow that sends a Slack notification to me with the exact command the agent wants to run and the reasoning behind it.

The workflow looks like this:

Agent fails a safeBins check.
The error is caught by the orchestrator and pushed to an n8n webhook.
n8n sends a message: "Agent X wants to run mcporter --force. Reason: 'Pod is stuck in Terminating'. Approve?"
I click "Approve," which updates a temporary Redis key granting the agent a 5-minute window of escalated privileges.

Why it works

This works because it acknowledges that the LLM is an unreliable narrator. You cannot trust the agent to follow safety guidelines, but you can trust the Linux kernel and the Kubernetes API.

By moving the constraints to the binary level (safeBins) and the network level (NetworkPolicy), we create a hard boundary. The agent can hallucinate all it wants, but it cannot execute a --force flag if the execution wrapper doesn't allow it.

Combining this with the two-tier service account model ensures that even if the agent escalates, it's using a token with a strictly defined TTL. The "glass-break" isn't just a permission change; it's a temporary shift in the security posture of the system.

For the MSAM (Model State Management) integration, I had to rewrite the server-side tools using FastMCP to support this. I used a specific IngressRoute to ensure that the escalation triggers only came from trusted internal IPs:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: msam-ingress
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`msam.example.com`)
      kind: Rule
      services:
        - name: msam-service
          port: 443
          tls:
            insecureSkipVerify: true

Lessons Learned

The biggest surprise was how much the agent actually prefers these constraints. When the agent knows exactly what the boundaries are (because the error messages from safeBins are explicit), it stops trying to guess and starts asking for help. It turns a "failure" into a "collaboration."

If I were doing this again, I'd automate the memory index rebuilds more aggressively. I found that when I escalated an agent to fix a model registry mismatch (like the OpenClaw v2026.3.12 issue where codex-5.4 wasn't recognized), the agent often forgot that it had already tried a specific fix. I had to implement a rebuild-memory-index.py script to ensure the agent's long-term memory was synced with the actual state of the registry after a glass-break event.

A few caveats:

Latency: The human-in-the-loop part of the glass-break is a bottleneck. If you're in a high-availability environment, you'll need to define "Auto-Escalation" rules for low-risk tasks.
Complexity: You're adding a layer of middleware between the agent and the tool. If your middleware crashes, your agent is blind. I run my orchestration layer with a strict Recreate strategy on Kubernetes to avoid the split-brain issues I've seen with Ollama deployments.

Ultimately, production AI isn't about building the smartest agent. It's about building the most reliable cage for that agent to live in. The glass-break pattern allows the agent to be useful without giving it the keys to the kingdom.