DEV Community: Guatu

PCIe Device Passthrough: NIC Name Instability and MAC Pinning

Guatu — Fri, 08 May 2026 04:15:19 +0000

My Proxmox node rebooted, and suddenly the host was unreachable via SSH. I had to plug in a physical monitor and keyboard only to find that my primary network interface, which had been enp4s0 for months, had decided to rename itself to enp5s0.

Because my /etc/network/interfaces file was explicitly tied to enp4s0, the bridge didn't come up, the IP wasn't assigned, and I was locked out of my own hardware.

What I expected

I expected the Linux kernel to consistently enumerate my PCIe devices. In a static hardware environment where nothing has moved, the PCI bus address should be deterministic. If the NIC is plugged into the same slot and the BIOS hasn't changed, enp4s0 should stay enp4s0 forever. This is the "happy path" most documentation assumes.

What actually happened

The reality is that PCIe enumeration is not always a constant. I'm using a mix of onboard NICs and a PCIe expansion card. I also have a GPU passed through to a VM.

The surprise here is how the kernel's predictable network interface naming (systemd-udevd) interacts with the PCIe topology. When I added a new PCIe device and tweaked some BIOS settings for IOMMU, the way the kernel mapped the physical slots to the virtual naming changed. A slight shift in how the PCIe switch reported the devices caused the index to jump.

This isn't just a "one-time fluke." If you're running a multi-node cluster or using GPUs that might move addresses (something I've documented before in GPU PCI Address Instability), you'll find that the kernel is surprisingly flexible with where it puts things.

The root cause is that enp4s0 is a name derived from the PCI location. If the location changes—even by one digit—the name changes. If your network config depends on that name, your system is one reboot away from a blackout.

The Fix: MAC Pinning

The only way to stop this is to stop relying on the PCI slot location and start relying on the hardware's unique identifier: the MAC address.

I decided to use systemd .link files. This allows me to tell the kernel: "I don't care where this device is on the PCIe bus; if it has this MAC address, call it eth0."

1. Identify the MAC address

First, I had to find the actual MAC of the problematic NIC while I had console access.

ip link show

I looked for the interface that was currently named enp5s0 (the "wrong" name) and copied the link/ether value.

2. Create the .link file

I created a custom link file in /etc/systemd/network/. I chose the name 10-lan.link to ensure it loads early in the boot process.

# /etc/systemd/network/10-lan.link
[Match]
MACAddress=00:11:22:33:44:55

[Link]
Name=eth0

(Note: I've anonymized the MAC address above. Use your actual hardware MAC here.)

3. Update the network configuration

Once the interface is pinned to eth0, I had to update the Proxmox network configuration to match. I edited /etc/network/interfaces to replace the volatile enp4s0 with the stable eth0.

# Example snippet from /etc/network/interfaces
auto eth0
iface eth0 inet manual

auto vmbr0
iface vmbr0 inet static
    address 10.0.0.x/24
    gateway 10.0.0.1
    bridge-ports eth0
    bridge-stp off
    bridge-fd 0

4. Apply and verify

I ran systemd-networkd-restart (or just rebooted, since I was already at the console) and verified the name with ip a. The NIC was now consistently eth0, regardless of whether the PCIe bus shifted.

Why this matters

If you're just running a single VM on a desktop, this is a minor annoyance. But if you're building a production-grade homelab, this is a critical failure point.

You'll hit this specifically in these scenarios:

Adding/Removing PCIe Hardware: Adding a new NVMe drive or a GPU can shift the enumeration of other devices on the same root complex.
BIOS Updates: A BIOS update often resets PCIe lane bifurcation or IOMMU settings, which can completely reorder how the kernel sees your NICs.
Using PCIe Switches: Some high-end motherboards or riser cables use PCIe switches that can report different topologies depending on the power state of the devices.

The Tradeoff

The tradeoff here is that you're moving away from the "modern" predictable naming convention back to the "old" ethX style. Some people find eth0 ugly or outdated, but in a headless server environment, "ugly" is better than "unreachable."

I've also seen people try to fix this using udev rules in /etc/udev/rules.d/. While that works, .link files are the native systemd way to handle this and are generally cleaner to maintain.

Lessons Learned

The biggest lesson here is that documentation for Proxmox and Debian assumes your hardware topology is a constant. It isn't.

When you're doing complex things like PCIe passthrough—which I've detailed in my GPU Passthrough Gotcha Guide—you are intentionally messing with the PCI bus. You're telling the host kernel to ignore certain devices so the VM can claim them. This volatility is a side effect of that power.

If you are passing through NICs or GPUs, do not trust the default interface names. Pin your critical management interfaces to their MAC addresses immediately. It takes five minutes to set up and saves you from a midnight trip to the server rack because a reboot decided your network card now lives at enp6s0.

For those of you managing larger fleets or complex AI agent infrastructure, this kind of hardware-level stability is the foundation. You can't build a reliable multi-agent AI pipeline if the underlying Kubernetes worker nodes are randomly losing their network identity.

Next time you're configuring a new node, don't just copy the enpXsX name from the GUI. Take the extra step to pin it. Your future self will thank you when the next BIOS update doesn't break your entire cluster.

GPU PCI Address Instability: When Your Card Moves Between Reboots

Guatu — Thu, 07 May 2026 00:15:04 +0000

I spent an entire afternoon debugging a VM that refused to boot, only to find out my GPU had decided to change its PCI address. One reboot and the device that lived at 01:00.0 suddenly migrated to 02:00.0. Because my Proxmox VM configuration was pinned to the old address, the VM crashed with a QEMU assertion error, and the GPU simply vanished from the guest.

This usually happens because of how the BIOS handles PCIe enumeration during POST. If you have multiple PCIe devices or a complex motherboard topology, the bus numbering isn't always deterministic. This is compounded by AMD Ryzen C-states or weird UMA frame buffer settings that can delay device initialization, causing the kernel to assign addresses in a different order than the previous boot. If you've already dealt with AMD iGPU RAM theft, you know how sensitive these BIOS settings are.

If you're on Proxmox 8.4+, the "happy path" is to use the q35 machine type. The older i440fx is more prone to these PCI mapping failures and IRQ conflicts. I also found that preventing the card from entering deep power states helps avoid the "zombie GPU" scenario where the card is physically there but logically dead.

To stabilize this, I switched the VM to q35 and explicitly enabled PCIe mode for the passthrough device. I also added a kernel parameter to stop the CPU from entering deep sleep states, which I've found reduces the randomness of the PCIe bus scan.

# 1. Change VM to q35 machine type for better PCIe support
qm set <VMID> --machine q35

# 2. Pass through the GPU with pcie=1 to ensure it's treated as a PCIe device
# Replace <PCI_ADDRESS> with your current address (e.g., 0000:01:00.0)
qm set <VMID> -hostpci0 <PCI_ADDRESS>,pcie=1

# 3. To stop the GPU from entering D3cold (which can cause boot-time instability)
# Run this on the Proxmox host
echo 0 > /sys/bus/pci/devices/0000:<PCI_BUS>:<PCI_SLOT>.0/d3cold_allowed

If the addresses keep shifting despite these changes, you're fighting your motherboard's firmware. At that point, I stopped fighting the VM abstraction and moved the NVIDIA drivers directly onto the Proxmox host. I then used the NVIDIA Container Toolkit to expose the GPU to my Kubernetes worker. It removes the PCI address fragility entirely because the host driver handles the hardware mapping, and the containers just see the device.

The lesson here is that PCI addresses are not constants; they are suggestions. If your workload requires 100% uptime and you can't guarantee a static PCI map, stop using VM passthrough and move the driver to the host.

Cognitive Memory for Agents: Vector Search vs Activation-Based Recall

Guatu — Wed, 06 May 2026 22:15:04 +0000

I spent a few weeks trying to build an agent that could remember specific user preferences across sessions without bloating the context window to a point where latency became unbearable. The standard advice is always "just use a vector database." But as the memory store grew, I noticed a weird gap: the agent could find a document about "user prefers dark mode" via cosine similarity, but it couldn't "recall" the immediate emotional state or the nuance of the last three turns of conversation unless they were explicitly mirrored in the embedding.

The problem is that vector search is a retrieval mechanism, not a cognitive memory system. When you move from simple RAG to actual agentic memory, you have to choose between external vector search and internal activation-based recall.

The Decision Point

You face this choice when your agent's "short-term" memory (the context window) is full, and your "long-term" memory (the database) is returning results that are mathematically similar but contextually irrelevant.

If you need your agent to remember a 500-page technical manual, you need a vector store. If you need your agent to exhibit a consistent "personality" or recall a specific pattern of behavior that isn't easily summarized into a string of text for an embedding model, you need something closer to activation-based recall.

Option A: Vector Search (The External Archive)

Vector search is the industry standard for a reason: it's easy to scale and the tooling is mature. You turn a piece of text into a vector using an embedding model (like text-embedding-3-small), shove it into a store like FAISS or Milvus, and query it with another vector.

Strengths:

Scale: You can store billions of vectors.
Cold Storage: It doesn't eat VRAM. It lives on disk or in a dedicated database.
Interpretability: I can literally query the database and see exactly which chunk of text was retrieved.

Weaknesses:

The "Semantic Gap": Cosine similarity is a blunt instrument. If a user says "That's not what I meant," a vector search might retrieve a passage about "meaning" or "intent" rather than understanding the correction.
Latency: You have to embed the query, hit the DB, and then stuff the results into the prompt.

Here is a basic implementation using FAISS. I use this for the "knowledge base" layer of my agents:

import faiss
import numpy as np

# Dimension depends on your embedding model (e.g., 1536 for OpenAI)
dimension = 128 
nb = 1000  # number of memory chunks
index = faiss.IndexFlatL2(dimension) 

# Mocking embeddings of agent experiences
vectors = np.random.random((nb, dimension)).astype('float32')
index.add(vectors) 

# Querying for the top 4 most similar memories
queries = np.random.random((1, dimension)).astype('float32')
distances, indices = index.search(queries, 4) 
print(f"Retrieved memory indices: {indices}")

Option B: Activation-Based Recall (The Internal Intuition)

Activation-based recall is more akin to how biological memory works. Instead of searching a database, the "memory" is stored in the weights or the hidden states of the model. In modern agent architectures, this often involves using activation hooks or specialized memory layers (like Memory Transformers) that allow the model to trigger a recall based on the current internal state of the network.

Strengths:

Speed: There is no external API call or DB lookup. The recall happens during the forward pass.
Nuance: It captures "how" something was said, not just "what" was said. It's an associative trigger rather than a keyword search.

Weaknesses:

The Black Box: Debugging this is a nightmare. You can't just "look" at the database to see why the agent recalled a specific memory.
VRAM Pressure: Storing these activations or maintaining a dynamic memory network consumes precious GPU memory.

I've experimented with simple activation hooks in PyTorch to track which "states" trigger certain behaviors. It's not a full-blown Memory Transformer, but it's a start:

import torch
from torch import nn

class AgentModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.memory_buffer = []

    def forward(self, x):
        # In a real system, this would be a specific layer's activation
        # that represents a 'concept' or 'state'
        activation = torch.tanh(x) 

        # Store the activation state for later recall/analysis
        self.memory_buffer.append(activation.detach().cpu().numpy())
        return activation

model = AgentModel()
input_tensor = torch.rand(1, 128)
output = model(input_tensor)
print(f"Stored state vector: {model.memory_buffer[-1]}")

Decision Framework

Criteria	Vector Search	Activation-Based Recall
Data Volume	Massive (TB+)	Small (MB to GB)
Retrieval Speed	Milliseconds (Network/Disk)	Microseconds (GPU)
Precision	Semantic/Keyword	Associative/Pattern
Debugging	Easy (Query the DB)	Hard (Analyze Tensors)
Resource Cost	CPU/Disk/API	VRAM/Compute

My Pick and Why

I don't pick one. I use a hybrid.

If you're building a production agent, relying solely on vector search leads to that "robotic" feeling where the agent repeats the same retrieved snippet regardless of the conversation flow. Relying solely on activations is a recipe for a system you can't debug when it starts hallucinating.

I implement a tiered system. I use a vector store for the "Library" (hard facts, documentation) and a sliding window of activations for the "Working Memory" (current mood, immediate goals, recent corrections). This mirrors the 6-layer memory architecture I've used for my own tools.

For those building multi-agent systems, I recommend offloading the vector search to a shared service and keeping the activation-based recall local to the agent's specific instance. This prevents the "shared memory" from becoming a noisy mess of conflicting embeddings. You can see how this fits into larger patterns in my post on multi-agent architecture patterns.

If you're still struggling with agents that forget things every five minutes, you might be hitting a safety loop. I've written about three-layer safety for autonomous agents which often solves the "infinite loop" problem that people mistake for a memory issue.

If you need help designing a memory architecture that doesn't melt your GPU or your budget, check out my AI agent consulting services.

Lessons learned:
The docs for vector DBs make it sound like they replace the need for cognitive memory. They don't. They replace the need for a filing cabinet. If you want an agent that actually "feels" like it's learning from a conversation in real-time, you have to move closer to the activations.

Vibration Monitoring Architecture: From Sensor to Dashboard

Guatu — Wed, 06 May 2026 16:15:04 +0000

The first time I tried to stream raw vibration data to a dashboard, I managed to crash my MQTT broker in under ten minutes. I had a high-frequency accelerometer spitting out samples at 5kHz, and I thought I'd just wrap those values in JSON and send them over the wire. The result wasn't a pretty graph; it was a series of Connection refused errors and a broker that had completely locked up under the weight of thousands of tiny packets per second.

If you're building a vibration monitoring system, you're not just dealing with "IoT data." You're dealing with signal processing. There is a massive difference between reporting a temperature every 30 seconds and capturing the harmonic frequencies of a motor bearing. If you treat vibration data like any other telemetry, your network will choke, your database will bloat, and your dashboards will be useless.

What I tried first (The wrong way)

My initial assumption was that the "modern stack" (Sensor $\rightarrow$ MQTT $\rightarrow$ Time Series DB $\rightarrow$ Grafana) would handle everything. I used a cheap industrial sensor that output raw voltage via a 4-20mA loop, fed into a PLC, which then pushed data to a Python script on a Raspberry Pi.

I wrote a simple loop that read the sensor and published to a topic:

# DO NOT DO THIS
while True:
    val = sensor.read() 
    client.publish("factory/machine1/vibration", json.dumps({"value": val}))

I quickly hit three walls:

Network Saturation: Sending one MQTT packet per sample is an architectural sin. The overhead of the TCP/IP stack and MQTT headers is larger than the actual payload. I was spending 90% of my bandwidth on headers.
Database Explosion: InfluxDB is great, but inserting 5,000 points per second per sensor is a recipe for a disk space crisis. My cardinality exploded, and queries that should have taken milliseconds started taking 30 seconds.
The "Noise" Problem: The raw data was a jagged mess. I couldn't see the actual vibration patterns because the high-frequency electrical noise from the nearby VFDs (Variable Frequency Drives) was masking the mechanical signal.

I realized that the gap between the sensor and the dashboard isn't a straight line. It's a funnel. You have to aggressively reduce the data volume at the edge before it ever touches the network.

The Actual Solution: The Edge-Heavy Pipeline

To make this work, I shifted the intelligence to the edge. The goal is to move from "streaming raw samples" to "streaming features." Instead of sending every single point, I calculate the RMS (Root Mean Square), Peak-to-Peak, and FFT (Fast Fourier Transform) bins locally.

1. Signal Conditioning and Edge Processing

I moved the processing to a dedicated edge gateway. I used a Python-based service that buffers samples in memory, applies a digital filter to remove electrical noise, and calculates the metrics.

Here is the implementation of the signal conditioning and feature extraction:

import numpy as np
from scipy.signal import butter, filtfilt
import paho.mqtt.client as mqtt
import time

# Configuration for a 10kHz sampling rate
FS = 10000 
CUTOFF = 2000 # Remove noise above 2kHz
ORDER = 4

def butter_lowpass_filter(data, cutoff, fs, order=5):
    nyq = 0.5 * fs
    normal_cutoff = cutoff / nyq
    b, a = butter(order, normal_cutoff, btype='low', analog=False)
    return filtfilt(b, a, data)

def calculate_features(buffer):
    # Filter the raw signal to remove high-frequency noise
    filtered = butter_lowpass_filter(buffer, CUTOFF, FS, ORDER)

    # Calculate RMS - the primary indicator of overall vibration level
    rms = np.sqrt(np.mean(filtered**2))

    # Calculate Peak-to-Peak
    ptp = np.ptp(filtered)

    # Perform FFT to find the dominant frequency
    fft_vals = np.abs(np.fft.rfft(filtered))
    freqs = np.fft.rfftfreq(len(filtered), 1/FS)
    dominant_freq = freqs[np.argmax(fft_vals)]

    return {
        "rms": float(rms),
        "ptp": float(ptp),
        "dom_freq": float(dominant_freq)
    }

# Main loop: Buffer 1000 samples, then send 1 summary packet
client = mqtt.Client()
client.connect("mqtt-broker.example.com", 1883)

buffer = []
while True:
    val = read_sensor_raw() # Mock function for ADC read
    buffer.append(val)

    if len(buffer) >= 1000:
        features = calculate_features(buffer)
        # Send summary instead of 1000 raw points
        client.publish("iiot/machine1/vibration/features", str(features))
        buffer = [] # Clear buffer

2. The Transport Layer (MQTT 5.0)

For the broker, I shifted from a basic Mosquitto setup to a more controlled configuration. Since vibration data is critical for predictive maintenance, I needed to ensure that the "heartbeat" of the machine was always known.

I used MQTT 5.0 "Will Messages" to detect if a gateway went offline. If the gateway crashes, the broker immediately publishes a "disconnected" status to the health topic, so the dashboard doesn't just show a flat line (which could be mistaken for a stopped machine).

# mosquitto.conf snippet
listener 1883
allow_anonymous false
password_file /etc/mosquitto/passwd
# Prevent the broker from being overwhelmed by slow consumers
max_queued_messages 1000

I've written more about choosing the right broker in my MQTT Broker Selection post, but for vibration, the priority is low latency and high reliability over massive scale.

3. Storage and Visualization

I used InfluxDB 2.x for storage because of its native handling of time-series data. Instead of storing the raw waveform, I store the calculated features. This reduces the storage requirement by 1000x.

In Grafana, I set up a dashboard that monitors the RMS value. However, looking at a raw line graph of vibration is usually useless for operators. They don't know if 0.5g is "bad" or "normal."

I integrated this with a health scoring system. I used a Flux query in InfluxDB to compare the current RMS against a baseline (the average of the last 7 days).

// InfluxDB Flux Query for Relative Vibration
from(bucket: "iiot_data")
  |> range(start: -1h)
  |> filter(fn: (r) => r["_measurement"] == "vibration_sensor")
  |> filter(fn: (r) => r["_field"] == "rms")
  |> aggregateWindow(every: 1m, fn: mean)
  |> map(fn: (r) => ({ r with value: r._value / 0.15 })) // Normalize against threshold 0.15g

This feeds directly into the concept of Equipment Health Scoring, where the goal is to give the operator a single "Health %" rather than a complex spectrum analysis.

Why this architecture works

The reason this works is that it respects the laws of physics and networking.

The Nyquist-Shannon Theorem tells us we need to sample at twice the frequency of the signal we want to capture. If you want to detect a bearing fault at 2kHz, you must sample at 4kHz+. Trying to do this over WiFi or Ethernet using standard JSON-over-MQTT is impossible because the packet overhead kills the throughput.

By calculating the RMS and FFT at the edge, we are performing Data Reduction. We transform a high-bandwidth signal (time domain) into a low-bandwidth set of descriptors (frequency domain).

The edge processing also acts as a mechanical filter. By using a Butterworth low-pass filter, I can strip out the 60Hz hum from the power lines and the high-frequency spikes from the VFDs. If you do this in the cloud, you've already wasted the bandwidth sending noise.

Lessons learned and caveats

If I had to build this again, I'd change a few things:

1. Hardware-level filtering: I spent too much time in Python trying to fix signal noise. In a real industrial environment, you should use an analog anti-aliasing filter (a physical capacitor/resistor circuit) before the signal ever hits the ADC. Software filters are great, but they can't fix aliasing if the signal was already corrupted during sampling.

2. The "Buffer" Trap: My Python script used a simple list for the buffer. At very high sampling rates, Python's list appending becomes slow. I had to switch to numpy arrays with pre-allocated memory to avoid garbage collection pauses that caused gaps in the data.

3. Provisioning the Edge: Managing these Python scripts across five different gateways was a nightmare. I eventually moved the deployment to a GitOps flow, using OpenTofu and GitHub Actions to manage the underlying VM configurations on my Proxmox cluster, ensuring every gateway had the exact same version of scipy and numpy.

4. The Dashboard Paradox: The more data I put on the dashboard, the less the operators used it. The final version of the system only shows three things: a Green/Yellow/Red light for health, the current RMS value, and a "Time to Maintenance" estimate. Everything else (the FFT bins, the raw waveforms) is hidden in a "Deep Dive" tab that only the reliability engineer ever opens.

Vibration monitoring is a classic example of where "more data" is actually "less information." The value isn't in the sensor; it's in the reduction process that happens between the sensor and the screen.

Unprivileged LXC + Docker: The runc Sysctl Permission Trap

Guatu — Tue, 05 May 2026 00:15:20 +0000

sysctl: setting key "net.ipv4.ip_local_port_range": Permission denied

I saw this error while trying to tune the network stack for a high-concurrency service running in Docker, which itself was hosted inside an unprivileged LXC container on Proxmox. The weird part? I was root inside the container.

I expected that since I had already enabled nesting=1 and keyctl=1 in the LXC configuration, Docker would have the necessary permissions to modify kernel parameters via runc. In a standard VM, this is trivial. In a privileged container, it just works. But in an unprivileged container, the user namespace mapping creates a wall that runc cannot climb.

What actually happened is a collision between systemd (v243+), runc, and the Linux kernel's security model for unprivileged user namespaces. When you run an unprivileged LXC, the root user inside the container is actually a non-privileged user on the Proxmox host (usually UID 100000).

The kernel prevents these mapped users from modifying sysctl settings because those settings are often global or namespace-specific in ways that could allow a container to crash the host or leak information. runc, the runtime Docker uses, tries to apply these settings during container creation, but the kernel returns a permission denied error. Because of how some Docker versions handle this, the error is sometimes swallowed, and your app just runs with the wrong defaults.

If you're building a production-grade homelab, you probably don't want to just switch to a privileged container. That's a security nightmare. Instead, you have to move the configuration "up" the chain.

The fix is to apply the sysctl settings at the LXC level before the container fully initializes, or directly on the host if the parameter isn't namespaced. Since we want to keep the host clean, using an LXC pre-start hook is the cleanest way to inject these settings.

On the Proxmox host, you can add a hook to the container's configuration file (usually in /etc/pve/lxc/ID.conf).

# Add this to your LXC .conf file on the Proxmox host
lxc.hook.pre-start = /usr/bin/echo "net.ipv4.ip_local_port_range = 1024 65535" >> /etc/sysctl.d/99-lxc.conf

However, for most users, the most reliable method is to define the parameter in the host's sysctl.conf if it's a global setting, or use the lxc.sysctl directive in the config file:

# Example Proxmox LXC config snippet
arch: amd64
cores: 2
memory: 2048
net0: name=eth0,bridge=vmbr0,ip=10.0.0.x/24,gw=10.0.0.1
ostype: ubuntu
unprivileged: 1
features: nesting=1,keyctl=1
# Inject the sysctl here
lxc.sysctl.net.ipv4.ip_local_port_range = 1024 65535

After adding this, you have to restart the container. If you just restart the Docker daemon inside the LXC, the kernel parameter won't update because the LXC boundary is where the restriction lives.

This trap is common when you're trying to optimize networking or memory management (like vm.max_map_count for Elasticsearch) inside a nested environment. If you've dealt with the headache of GPU passthrough on Proxmox, you know that the gap between "it's a container" and "it's an unprivileged container" is where most of the pain lives.

One last thing to watch out for: UID shifts. If you're mounting NFS shares into these containers to provide storage for your Docker volumes, you'll hit the UID mismatch. The container thinks it's root (UID 0), but the host sees UID 100000. I've spent hours debugging "Permission Denied" on volumes only to realize I needed to chmod 0777 the host directory or properly map the IDs in the .conf file.

If you're scaling this into a larger cluster, I highly recommend moving these workloads to bare-metal Kubernetes. I wrote about my experience with Longhorn for bare-metal storage, and while the initial setup is heavier than an LXC, you stop fighting the Proxmox container permission war and start dealing with standard K8s primitives.

AdGuard Home: Network-Wide DNS Filtering with Failover

Guatu — Mon, 04 May 2026 22:15:20 +0000

DNS is the single point of failure that makes everyone in the house complain that "the internet is down" when, in reality, your DNS container just crashed. I've spent too much time as the sole admin of my network having to manually flip DNS settings on my router because a single AdGuard Home instance decided to stop responding. If you're running this in a homelab, you can't just set it and forget it. You need a failover strategy that doesn't require you to touch a CLI while your family is staring at you.

The mistake most people make is trusting the default upstream behavior. They add three upstream servers and assume AdGuard Home will magically route around a dead one instantly. In practice, depending on your version and config, you can still hit timeouts that feel like a total outage. I've moved my setup to a Kubernetes deployment using MetalLB to give it a static IP, but the real win is the explicit failover logic in the adguard-home.yaml.

I prefer using a combination of Cloudflare and Quad9 for the primary upstreams, with a dedicated fallback. This ensures that if my primary DNS providers have a routing issue, the system pivots to a tertiary option without dropping the request.

# adguard-home.yaml snippets
upstream_dns:
  - "1.1.1.1"
  - "1.0.0.1"
  - "9.9.9.9"

dns:
  # Use parallel requests to find the fastest response
  upstream_mode: parallel 

failover:
  enabled: true
  health_check_interval: 30
  health_check_timeout: 10
  fallback_upstream: "8.8.8.8"

For those running this on K8s, don't skimp on memory limits. I initially set my memory request too low and saw the OOM killer terminate the pod every time I updated a large blocklist. I now pin my resources to ensure stability, especially when integrated with cert-manager for automated TLS to secure the dashboard.

helm install adguard-home k8s-at-home/adguard-home \
  --namespace network \
  --create-namespace \
  --set image.tag=latest \
  --set resources.limits.memory=1Gi \
  --set resources.requests.memory=256Mi

The biggest lesson here is that "high availability" for DNS isn't just about having two pods. It's about how the system handles the gap between a server being "up" and a server actually returning a valid record. If you're building out larger infrastructure, I've found that combining this with a strict manifest validation pipeline prevents the kind of YAML typos that can take your entire network offline.

Keep your upstreams diverse and your memory limits realistic.

Three-Layer Safety for Autonomous Agents: Stopping the Infinite Loop

Guatu — Thu, 30 Apr 2026 22:15:29 +0000

I watched an autonomous agent spend three hours and 40,000 tokens trying to close a GitHub issue that had an open dependency, only to fail because it kept hallucinating a force_close flag that didn't exist in the API. It didn't just fail; it entered a perfect infinite loop: it would call the tool, get a 400 error, interpret the error as a "temporary network glitch," and try again with the exact same payload.

If you've built agents that actually touch production systems, you know this feeling. Prompting the agent to "be careful" or "follow the schema" is a placebo. When you move from a chat window to an autonomous loop, the gap between the LLM's intent and the system's reality becomes a canyon where agents go to die (and burn through your API credits).

For anyone running agent orchestration in a homelab or production environment, you need a safety architecture that doesn't rely on the model's "good behavior." I've moved to a three-layer safety model: Token-Level Enforcement, Pre-Execution Gates, and Execution Isolation.

What I tried first

My first instinct was to lean heavily on PydanticAI. The idea of using Pydantic for type-safe tool calling seemed like the silver bullet. I spent a week building out complex schemas, thinking that if the code validated the output, the agent would simply "learn" to provide the correct format.

I was wrong. I hit a wall where the agent would produce a JSON object that was almost correct, but it would miss a closing brace or add a trailing comma. Pydantic would throw a ValidationError, the agent would see that error in its history, and then it would attempt to "fix" the JSON by adding even more commentary around the code block. This created a feedback loop of ValidationError $\rightarrow$ Apology $\rightarrow$ Broken JSON.

Then I tried adding a "supervisor" agent to review the actions of the "worker" agent. This just doubled my latency and doubled my token cost without actually solving the root cause. The supervisor often hallucinated the same API capabilities as the worker because they were using the same base model.

The real problem wasn't the logic; it was the lack of deterministic boundaries. I was treating the LLM as a reliable software component when it's actually a probabilistic engine. To make it safe, I had to stop trying to "convince" the model to be safe and start forcing it to be safe at the infrastructure level.

Layer 1: Token-Level Schema Enforcement

The first layer of safety happens before the agent even finishes its sentence. If you're using Ollama v0.5.0 or newer, you can stop relying on the model to "try its best" with JSON.

Most people use the OpenAI-compatible API layer provided by frameworks, but that often just wraps the prompt in "Please return JSON." Ollama now supports a native format parameter that enforces the schema at the token-sampling level. This means the model physically cannot sample a token that violates the JSON schema.

Here is how I implemented this for my homelab health reports using qwen2.5:14b-instruct. I switched from the 32B model to the 14B variant because the 32B was causing 502 timeouts on my Tesla P40s due to VRAM pressure.

import httpx
from pydantic import BaseModel, Field

# Define the strict structure we want
class HomelabHealthReport(BaseModel):
    node_status: dict[str, str]
    critical_alerts: list[str]
    storage_utilization: float = Field(description="Percentage 0-100")

# Extract the JSON schema for Ollama
schema = HomelabHealthReport.model_json_schema()

def get_safe_report():
    # We bypass the high-level wrappers and hit the API directly
    # to ensure the 'format' parameter is actually passed.
    response = httpx.post(
        "http://ollama:11434/api/chat",
        json={
            "model": "qwen2.5:14b-instruct",
            "stream": False,
            "format": schema, # This is the magic: token-level enforcement
            "prompt": "Generate a health report for the homelab based on current metrics."
        },
        timeout=30.0
    )

    if response.status_code != 200:
        print(f"API Error: {response.status_code}")
        return None

    return response.json()["message"]["content"]

# Result is guaranteed to be valid JSON matching HomelabHealthReport

By moving the constraint to the sampler, I eliminated the ValidationError loops entirely. The model no longer "guesses" the JSON; it is constrained by the grammar of the schema.

Layer 2: The Pre-Execution Gate (ActionGate)

Even with perfect JSON, an agent can still decide to do something stupid. Token-level safety ensures the format is right, but it doesn't ensure the intent is safe.

I implemented an ActionGate. This is a deterministic middleware layer that sits between the agent's tool-call and the actual execution. It doesn't use an LLM. It uses hard-coded business logic and state checks.

If an agent tries to close a ticket, the ActionGate checks if there are open dependencies. If it tries to reboot a node, it checks if that node is currently the only one running a critical service.

class SafetyException(Exception):
    pass

def check_action_safety(action_name, params, context):
    """
    Deterministic safety check. 
    No LLMs allowed here.
    """
    # Prevent closing issues that have blocking dependencies
    if action_name == "close_issue":
        issue_id = params.get("issue_id")
        if context.get(f"issue_{issue_id}_has_dependency"):
            raise SafetyException(
                f"Safety Violation: Cannot close issue {issue_id} while dependencies are open."
            )

    # Prevent destructive actions on production nodes during peak hours
    if action_name == "reboot_node":
        node_id = params.get("node_id")
        if context.get("is_production") and context.get("peak_hours"):
            raise SafetyException(
                f"Safety Violation: Reboot of {node_id} forbidden during peak hours."
            )

    return True

# Usage in the agent loop
try:
    if check_action_safety(tool_call.name, tool_call.args, current_context):
        result = execute_tool(tool_call)
except SafetyException as e:
    # We feed the specific error back to the agent so it can pivot
    result = f"Action rejected by Safety Gate: {str(e)}"

This prevents the "infinite loop of failure" I mentioned earlier. Instead of the agent getting a generic 400 error from an API and thinking it's a network glitch, it gets a clear, human-readable explanation: "You cannot do this because X." This forces the agent to change its strategy rather than just retrying the same failed request.

Layer 3: Execution Isolation and Shell Safety

The final layer is where the rubber meets the road. I've spent too many hours debugging "quoting hell."

When you have an agent generating a command that needs to run over SSH, inside a Proxmox container (pct exec), as a specific user (su), and then executing a Python script, you have four layers of shell interpretation. If you use f-strings to build these commands, a single single-quote in the agent's output will break the entire pipeline.

I saw this happen when an agent tried to pass a complex JSON string as an argument to a script. The shell interpreted the quotes, the su command stripped another layer, and by the time it hit Python, the syntax was mangled.

The fix is to stop passing code as shell arguments. Instead, pipe the code directly into the stdin of the remote process.

The wrong way (prone to quoting errors):

# This will break the moment the agent adds a ' or " to the payload
ssh node-a "pct exec 101 -- su - user -c 'python3 -c \"print(\"Hello World\")\"'"

The right way (Shell-safe piping):
I wrote a helper that writes the agent's intended Python logic to a temporary file or pipes it directly. This avoids the shell's interpretation of the string entirely.

# We pipe the actual script content into the remote shell
cat ~/bin/helpers/scout-ideas-helper.py | \
  ssh node-a "pct exec 101 -- su - user -c 'python3 -'"

In this setup, python3 - tells Python to execute the code coming from stdin. The shell only sees the command to start Python, not the code itself. This completely eliminates the quoting nightmare.

To manage the tools themselves, I've moved away from custom boilerplate and started using FastMCP. It allows me to wrap my MSAM (Multi-Agent System Architecture) tools into a standardized server that the agents can discover and use without me having to manually update the tool definitions every time I add a new function. I've detailed the setup for this in my post on Building MCP Servers with FastMCP.

Why this works

This architecture works because it acknowledges that the LLM is the most unreliable part of the system.

Token-level enforcement removes the "formatting" problem. The agent can no longer fail because it forgot a comma.
The ActionGate removes the "logic" problem. The agent can no longer perform an action that is fundamentally unsafe, regardless of how confident it is.
Execution Isolation removes the "infrastructure" problem. The agent's output is treated as data (stdin) rather than as a command (shell argument).

When you combine these, you move from a system that is "mostly working" to one that is "predictably bounded."

Lessons Learned

The biggest surprise was how much the format parameter in Ollama reduced the need for complex prompt engineering. I spent weeks refining a "System Prompt" to ensure JSON compliance, only to find that a single API parameter did the job better than 500 words of instructions.

If I were to do this over again, I would have implemented the ActionGate much sooner. I spent too much time trying to make the agent "smarter" when I should have just made the environment "stricter."

A few caveats:

Latency: Each layer adds a small amount of overhead. The ActionGate is negligible (milliseconds), but the token-level enforcement can slightly increase the time to first token because the sampler has to do more work.
VRAM: As I noted, model size matters. Qwen 2.5 14B is the sweet spot for my hardware. If you're running on limited VRAM, don't chase the 32B or 70B models just for the sake of "intelligence" if it leads to 502 timeouts and unstable inference.
Memory Drift: Ensure your agent's memory is cleaned up. I use a six-layer memory architecture to prevent the agent from getting confused by outdated context, which is often the root cause of why it tries to perform unsafe actions in the first place.

Building autonomous agents isn't about finding the perfect model; it's about building the perfect cage for that model to operate in.

Stop Merging Broken YAML: Kubernetes Manifest Validation in CI

Guatu — Sat, 25 Apr 2026 22:15:35 +0000

Pushing a broken manifest to your main branch is a rite of passage, but it's one that becomes significantly more painful when you're running a GitOps workflow with ArgoCD. I've spent far too many late nights staring at a "Sync Failed" status in ArgoCD, only to realize I had a typo in a Traefik IngressRoute or a missing resource limit that Kyverno was blocking. The problem isn't just the error itself; it's the feedback loop. If the error only surfaces during deployment, your CI pipeline has failed its primary job.

The goal is to move validation as far left as possible. I started integrating kubeconform into my GitHub Actions workflow to catch structural errors—like invalid API versions or malike fields—before the code even reaches a pull request review. However, structural validation is only half the battle. You also have to deal with policy enforcement. I recently ran into a situation where a Kyverno policy enforcing resource limits on all Jobs was breaking my CloudNativePG (CNPG) deployments. The CNPG operator creates Jobs that don't always follow the standard resource pattern, and because the policy was too broad, the cluster refused to provision the primary.

The fix involves two parts: using kubeconform for schema validation in CI and using targeted exclusions in your Kyverno policies. For the CI side, you don't need a complex setup. A simple action step can scan your entire manifests directory.

# GitHub Action snippet for manifest validation
jobs:
  validate-manifests:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Validate Kubernetes manifests
        uses: yannh/kubernetes-manifest-validate@v1.11
        with:
          manifests: |
            kubernetes/workloads/**/*.yaml
            kubernetes/infrastructure/**/*.yaml

On the cluster side, when you have a legitimate reason to bypass a policy—like the CNPG example—don't just disable the policy globally. Use labels to create an exclusion scope. This keeps your GitOps for Homelabs workflow clean without sacrificing security for the rest of your workloads.

apiVersion: kyverno.io/v1
kind: Policy
metadata:
  name: require-resource-limits
spec:
  rules:
    - name: enforce-limits-on-jobs
      match:
        resources:
          kinds:
            - Job
      # Exclude CNPG clusters so the operator can manage its own jobs
      exclude:
        resources:
          labels:
            cnpg.io/cluster: "*"
      validate:
        message: "All containers must have resource limits defined."
        pattern:
          spec:
            template:
              spec:
                containers:
                  - resources:
                      limits:
                        cpu: "?*"
                        memory: "?*"

Validating at the PR stage catches the "dumb" mistakes, while smart policy exclusions prevent the "smart" tools from breaking your legitimate infrastructure.

GPU D3cold Power States: How to Brick Your Card Without Trying

Guatu — Fri, 24 Apr 2026 18:15:49 +0000

THE SYMPTOM: My NVIDIA Tesla P40 would stop responding after a VM shutdown. No error messages, just a dead GPU that required a full host reboot to recover.

WHAT I EXPECTED: A clean shutdown of a VM with GPU passthrough should leave the GPU in a ready state. I assumed the host would handle power states gracefully.

WHAT ACTUALLY HAPPENED: The GPU went into D3cold, a low-power state that it couldn't exit without a full host reboot. This happened even after proper VM shutdowns. The issue was especially prevalent on Proxmox 8.4 with kernel 6.8.x and QEMU 8.0.1, where the lack of FLR support on the P40 made it impossible to reset the GPU from the host.

THE FIX: I disabled D3cold before passthrough by writing 0 to /sys/bus/pci/devices/0000:08:00.0/d3cold_allowed. I also pinned the GPU’s PCI address using a udev rule to prevent it from shifting on reboot. Here's the rule I used:

ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{device}=="0x1b80", ATTR{bus}=="0000:08", SYMLINK+="gpu-passthrough"

This ensured the GPU stayed on the same PCIe bus and avoided the D3cold trap entirely. For Proxmox 8.4 users, I also had to explicitly set -machine q35 in the VM config to prevent QEMU from asserting on boot.

WHY THIS MATTERS: If you're running non-FLR GPUs like the P40 on Proxmox 8.4 or later, you're likely to hit this issue. It's not just a matter of setting up passthrough — you need to actively prevent the GPU from entering D3cold and lock its PCI address. If you skip either step, you're asking for a bricked GPU. I've seen this happen on more than one occasion, and the fix is always the same: disable D3cold and pin the address.

This isn't just a Proxmox-specific gotcha. Any system that doesn't support FLR on the GPU and relies on the kernel to manage power states is at risk. If you're using a Tesla P40, T4, or any other non-FLR GPU and you're seeing GPU failures after VM shutdown or reboot, this is the fix you need. I've also seen this issue surface with AMD GPUs under certain conditions, though the fix is slightly different.

If you're running AI workloads on Kubernetes or any other system that depends on GPU passthrough, this is a critical detail. You don't want to be the one who has to power cycle a node just to get a GPU working again. I've had to do it more than once. It's not fun. The key is to prevent the GPU from ever getting into a state where it can't reset itself.

For those who are considering moving away from GPU passthrough entirely, I've also found that running the NVIDIA driver directly on the host can be a much more stable option. It avoids all the PCIe bus instability and power state issues. I've tested this with the NVIDIA Container Toolkit and it's worked well for me in production environments.

I've written this post not because I want to scare you — but because I want to save you from the frustration of a bricked GPU. If you're running Proxmox, using older GPUs, and you've had this issue, you're not alone. I've been there, and I've found a way to avoid it.

cert-manager + Cloudflare DNS-01: Automated TLS for Everything

Guatu — Fri, 24 Apr 2026 04:15:49 +0000

I spent two days chasing a cert-manager error that looked like it was coming from the future. The message was clean: Error: failed to solve challenge: failed to update DNS record: 403 Forbidden. I had followed the docs, created the API token, set up the ClusterIssuer, and even double-checked the zone. But the error wouldn’t go away. Turns out, the token didn’t have the right scope, and I had no idea. That’s the kind of thing that happens when you skip the part of the documentation that says "make sure your token has these exact permissions."

If you’re running Kubernetes on bare metal, in a homelab, or in a production environment, you need TLS. You need it for ingress, for internal services, for anything exposed to the internet. cert-manager is the go-to tool for this, and Cloudflare is the go-to DNS provider for a lot of us. But the setup isn’t as straightforward as the docs make it look. I’m going to walk through what I tried first, what actually worked, and why it matters.

I’m not here to sell you on cert-manager or Cloudflare. I’m here to tell you what I did when I tried to make TLS work for everything in my cluster , and what I had to fix when it didn’t.

What I Tried First

I started with the standard cert-manager installation, using the Helm chart. I followed the example for Cloudflare DNS-01, set up the API token, and created the ClusterIssuer. I used a Kubernetes Secret to store the token, and referenced it in the cloudflare provider block of the issuer configuration.

That’s where it went wrong.

The first error was 403 Forbidden, and I had no idea why. I checked the token’s scope again. I double-checked the zone name. I even created a new token with all the permissions I could think of. Nothing worked. The cert-manager logs just said the same thing again and again.

I tried looking for similar issues on GitHub, Stack Overflow, and the cert-manager forums. The most common answers were things like:

"Make sure your token has the DNS:Edit scope"
"Check that the zone name is correct"
"Ensure that the API token is not expired"

I had done all of those. And still, it didn’t work.

Then I thought: maybe the token was created for the wrong zone. I went to the Cloudflare dashboard, created a new token for the exact zone name I was using. I gave it Zone:Read and DNS:Edit permissions, and then I re-deployed the ClusterIssuer.

Still nothing.

It was at this point that I realized the issue wasn’t the token or the zone , it was the email field in the provider configuration. I had used the same email that I used to register the domain. But cert-manager requires a Cloudflare account email, not the email used to register the domain. That’s not something you see in the documentation, and that’s exactly the kind of thing that breaks your day.

The Actual Solution

Let’s get specific. Here’s what I ended up with:

Step 1: Create a Cloudflare API Token

Go to the Cloudflare API Tokens dashboard, and create a new token. Give it the following permissions:

Zone:Read (for reading the zone information)
DNS:Edit (for updating DNS records during the DNS-01 challenge)

Make sure the token is scoped to the exact zone you're using, not the entire account.

Step 2: Store the API Token in a Kubernetes Secret

Create a Kubernetes Secret to store your Cloudflare API token:

kubectl create secret generic cloudflare-api-token \
  --from-literal=api-token="your-cloudflare-api-token-here" \
  --namespace=cert-manager

Step 3: Configure cert-manager with the Cloudflare DNS-01 Provider

Here’s the complete configuration for the ClusterIssuer:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: cloudflare
spec:
  acme:
    email: "your-cloudflare-account-email@example.com"
    server: https://acme-v02.api.sandbox.cloudfla.re
    privateKeySecretRef:
      name: cloudflare-acme-account-key
    solvers:
      - selector:
          dnsZones:
            - "example.com"
        dns01:
          cloudflare:
            apiTokenSecretRef:
              name: cloudflare-api-token
              key: api-token

Make sure to replace your-cloudflare-account-email@example.com with the actual email address associated with your Cloudflare account, not the one you used to register the domain.

Step 4: Deploy a Sample Ingress to Test

Create a simple Ingress to test if cert-manager can issue a certificate:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: "cloudflare"
spec:
  tls:
    - hosts:
        - "example.com"
      secretName: example-com-tls
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: nginx
                port:
                  number: 80

Once this is deployed, cert-manager should automatically request and issue a certificate for example.com.

Step 5: Use SealedSecrets for Secure Credential Management (Optional)

If you want to store your Cloudflare API token securely, you can use SealedSecrets. This is especially useful if you're using GitOps or want to ensure that secrets aren't stored in plain text in your version control system.

First, install SealedSecrets:

kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/latest/download/controller.yaml

Then, seal your secret:

kubeseal --cert ./sealed-secrets/public-key.pem < cloudflare-api-token.yaml > cloudflare-api-token-sealed.yaml

Apply the sealed secret:

kubectl apply -f cloudflare-api-token-sealed.yaml

And update your ClusterIssuer to reference the sealed secret:

apiTokenSecretRef:
  name: cloudflare-api-token-sealed
  key: api-token

Why It Works

cert-manager uses the ACME protocol to issue certificates. When you use the DNS-01 challenge, cert-manager needs to modify a DNS record to prove that it has control over the domain. This is where Cloudflare comes in , it allows cert-manager to temporarily update a DNS record on your behalf.

The key part here is the cloudflare provider configuration in the ClusterIssuer. It tells cert-manager how to authenticate with Cloudflare using the API token. The email field must be the email associated with the Cloudflare account, not the domain's registration email.

Cloudflare's API is rate-limited, and if you're not careful, you can hit these limits and cause certificate issuance to fail. That’s why it's important to ensure your API token has the correct scopes and that you're using the right email.

Also, cert-manager requires that the zone in the selector.dnsZones field matches the domain exactly. If you're using a wildcard, like *.example.com, make sure to include that in your configuration.

Lessons Learned

Use the right email. The Cloudflare email is not the same as the domain's registration email. This is the first thing I missed, and it took me hours to figure out.
Token scopes matter. If your token doesn’t have the right permissions, cert-manager can't update the DNS records. Always double-check that your token has Zone:Read and DNS:Edit permissions.
Secure your secrets. Using SealedSecrets is a great way to keep your Cloudflare API token safe. It adds an extra layer of security and ensures that your secrets aren't exposed in your version control.
Test everything. Don’t assume that just because the configuration looks right, it will work. Create a test Ingress and watch the cert-manager logs to see what’s happening under the hood.
Watch for API limits. Cloudflare's API has rate limits, and if you’re issuing a lot of certificates, you could hit those limits. It’s a good idea to monitor your usage and set up alerts if needed.

I also found that cert-manager v1.13+ has stricter validation for DNS01 providers. Older versions might not show errors clearly, which can make debugging a lot harder. I ended up using v1.14.0, which had better error messages and worked more reliably with Cloudflare.

If you're using a dynamic IP setup, like with a home network or a cloud provider that assigns dynamic IPs, you’ll need to set up a DDNS update mechanism. I used a CronJob with GitOps to ensure that my Cloudflare A record was always up to date. That way, even if my IP changes, the certificate can still be issued and renewed.

I also had to deal with a bug in my Helm chart that was using an old version of cert-manager. The Helm chart I was using had a misconfigured targetRevision, which was causing the issuer to fail silently. I had to manually update the targetRevision to match the version I was using.

Final Thoughts

Automating TLS with cert-manager and Cloudflare DNS-01 is a powerful combination. It saves you from manually issuing certificates and keeps your cluster secure without the overhead of managing them by hand. But it’s not without its gotchas , and I’ve had my fair share of them.

If you're new to cert-manager or Cloudflare, I recommend starting small. Create a test Ingress, watch the logs, and make sure everything works before rolling it out to production. And if you hit a roadblock, don’t be afraid to check the cert-manager logs , they often give you the exact error you need to fix the problem.

You can also find more information on how to set up cert-manager with Cloudflare in the official documentation, or in some of the other posts on guatulabs.dev about Kubernetes and infrastructure. If you're interested in more advanced topics, like setting up a GitOps pipeline with ArgoCD or using SealedSecrets for secure credential management, those are also worth reading.

In the end, the goal is to make TLS work for everything , and that’s what cert-manager is all about.

SealedSecrets Key Backup: Don't Lose Your Encryption Keys

Guatu — Wed, 22 Apr 2026 22:15:49 +0000

I lost access to a SealedSecrets key once , not because I deleted it, but because I didn't know where it was stored. The cluster kept running, the apps kept deploying, but the moment I tried to rotate the key or redeploy a sealed secret, I hit a wall. The controller couldn't decrypt anything. The only way out was to find the original key, and I had to dig through old manifests and cluster logs to get it back. That’s when I learned the hard way: SealedSecrets keys aren’t magical. They’re just Kubernetes secrets, and they can be lost if you don’t back them up.

The SealedSecrets controller uses a single key to encrypt and decrypt secrets. If that key is lost, all your sealed secrets become unusable. You can’t just regenerate it , the encryption is tied to that specific key. The key is stored as a Kubernetes secret in the sealed-secrets namespace. If you don’t back it up, and it gets deleted or corrupted, you're out of luck.

Here’s the command I use to back it up. It exports the key to a YAML file, which I store off-cluster in version control or a secure backup system:

kubectl get secret sealed-secrets-key -n sealed-secrets -o yaml > sealed-secrets-key-backup.yaml

This is the only way to ensure you can recover from a key loss. If you're using GitOps tools like ArgoCD, make sure this backup is part of your repo and included in your CI/CD pipeline. Otherwise, the moment you redeploy the sealed-secrets controller, the key could be lost if it's not versioned.

If you lose the key, the only way to recover is to restore it from a backup. You can do that by applying the YAML file back into the cluster. Just make sure the namespace and secret name match the original. If you're using ArgoCD, you may need to disable the sealed-secrets app, apply the key, and then re-enable it to avoid reconciliation conflicts.

Don’t assume the key is safe just because it's in the cluster. Back it up, version it, and keep it somewhere you can get to when you need it. That’s the only way to stay ahead of a potential outage.

Ollama on Kubernetes: Recreate Strategy and Single-GPU Deadlock

Guatu — Tue, 21 Apr 2026 20:16:02 +0000

I deployed Ollama on Kubernetes, and the GPU worker node locked up mid-rollout. No logs, no error, just a dead pod that wouldn’t terminate and a new one that wouldn’t schedule. It wasn’t a crash. It wasn’t a timeout. It was a deadlock I’d never seen before.

I expected a smooth rollout. Ollama is a single-container, single-GPU workload. I set up a Deployment with a single replica, used a PersistentVolumeClaim for model storage, and assumed Kubernetes would manage the rest. That’s what the documentation says.

What actually happened was a scheduling deadlock. The old pod was still running, using the GPU, but the new pod couldn’t schedule because the GPU was in use. Kubernetes’ default RollingUpdate strategy tried to keep one pod running while replacing the other, but the GPU couldn’t be shared. The new pod waited for the old one to terminate, and the old pod waited for the new one to start. Deadlock.

The fix was switching the Deployment strategy from RollingUpdate to Recreate. That way, the old pod terminates before the new one starts. No GPU contention. No deadlock. It’s a simple change , just set type: Recreate in the Deployment spec.

Here’s what the Deployment looks like with the fix:

spec:
  replicas: 1
  strategy:
    type: Recreate

I also had to configure the NVIDIA runtime correctly. Ollama needs the NVIDIA_VISIBLE_DEVICES=all environment variable set, and the NVIDIA container runtime must be properly mounted. Otherwise, the container fails to initialize, and the pod stays in a CrashLoopBackOff state.

spec:
  containers:
    - name: ollama
      env:
        - name: NVIDIA_VISIBLE_DEVICES
          value: all
      volumeMounts:
        - name: nvidia-driver
          mountPath: /usr/local/nvidia
  volumes:
    - name: nvidia-driver
      hostPath:
        path: /usr/local/nvidia

Why does this matter? If you’re running any GPU workload on Kubernetes , especially ones that require exclusive access to the GPU , you need to understand the limitations of RollingUpdate. It’s not a one-size-fits-all strategy. For GPU workloads, Recreate is the only safe option. Otherwise, you’ll hit deadlocks that leave your pods in limbo.

Another gotcha I ran into was PVC sizing. Ollama models can be large , some of the larger ones require over 100Gi of storage. I initially set the PVC to 50Gi, and the pod wouldn’t schedule. The PVC couldn’t be bound because the node didn’t have enough storage capacity. I had to bump the PVC size and make sure the underlying storage class (Longhorn in my case) had enough available space.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ollama-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi

If you’re running Ollama on Kubernetes, you’ll need to be mindful of these details. A single misconfigured PVC or deployment strategy can bring your whole system to a halt. I’ve seen it happen more than once. It’s not just about getting the container to start , it’s about making sure it stays up and doesn’t lock the system in the process.

If you’re building AI agent workloads or running large models in production, this is a common pitfall. The documentation doesn’t always highlight the GPU-specific constraints of Kubernetes. But when you’re working with real hardware, those constraints are real. And when they bite, you’ll wish you’d read about them before it’s too late.

For more on GPU workloads and Kubernetes, check out NVIDIA Container Toolkit: Why the Default Runtime Matters. If you’re using Longhorn for storage, Kubernetes Storage on Bare Metal: Longhorn in Practice is a good next step.