DEV Community: Guatu

Blog Pipeline Fence Corruption: How a Naive Sed Rule Swallowed Markdown Code Blocks

Guatu — Mon, 31 Aug 2026 22:15:48 +0000

A draft came out of my blog generation pipeline with the frontmatter intact, the title correct, the word count in range, and roughly 1,400 words of prose rendered inside a single syntax-highlighted code block. Everything after the first bash example was gray monospace on a dark background. The build passed. The exit code was 0.

One sed expression caused it, and it had been sitting in the generation script since the pipeline moved from a local model to a cloud backend. It did exactly what it was told to do, which was the whole problem.

What I expected

The setup is boring and probably familiar. A shell script calls a model in non-interactive mode, captures stdout, and writes the result to src/content/posts/<slug>.md. Astro picks it up from the content collection, Zod validates the frontmatter, and npm run build renders it.

Models like to wrap their output. You ask for "a complete markdown file, output only the markdown," and a meaningful fraction of the time you get back something shaped like this:

```markdown
title: "Whatever"

Post body here.
```

That outer fence is not part of the document. Write it to disk as-is and Astro sees a code block where the frontmatter should be, Zod finds no title, and the build fails loudly. Loud failures are fine. You fix them and move on.

So the pipeline stripped the wrapper before writing. That was the intent, and the expected outcome was a clean file with the outer fence gone and every internal code block untouched.

What actually happened

Here is the rule, as it existed in the generation script:

# The "cleanup" that caused the problem
claude -p "$PROMPT" | sed 's/^```markdown$//; s/^```$//' > "$OUT"

Read the second expression on its own: s/^`$//. It replaces any line consisting of exactly three backticks with an empty line. Every line. Not the last one, not the wrapper's closer specifically, all of them.

Markdown fences are a toggle, not a container. An opening fence with an info string (bash `) turns code mode on. A bare closing fence turns it off. Delete every bare closing fence in a document and you have deleted every "off" switch while keeping every "on" switch.

Corruption follows deterministically from there. The first `bash in the post opens a code block, and that block never closes. Every heading, paragraph, table, and link after it becomes code block content. The renderer is behaving correctly. It was handed a document that says, in effect, "the rest of this file is bash."

Nothing about the file is malformed by CommonMark rules, which is why it looked like corrupted prose rather than a syntax error. An unclosed fence at end-of-document is legal and closes implicitly. The parser doesn't complain, the markdown linter doesn't complain, and Astro renders it happily into a page nobody would read past the first screenful.

That first expression, s/^markdown$//, carried a smaller version of the same flaw. It emptied any line that was exactly markdown, which is a legitimate thing to write in a post about markdown tooling. Delete that opener and you're left with an orphaned closer, which then opens a code block for the remainder of the file. Same failure, inverted.

Two things kept this from being obvious for longer than I'd like.

Local models mostly didn't trigger it. The Ollama-backed path had different post-processing and rarely emitted the markdown wrapper in the first place, which is the kind of behavioral difference you don't discover until you [move a scheduled generation job between backends](https://guatulabs.dev/posts/moving-scheduled-llm-curation-from-cloud-apis-to-local-models/). So the sed` rule was written for a problem that mostly existed on one branch of the pipeline, and it damaged output on both.

Beyond that, nothing in the pipeline had an opinion about document structure. It checked word count. It checked frontmatter fields. It ran an anti-slop word filter. Every one of those checks passes cleanly on a file whose entire body is trapped inside a code block, because none of them parse the document as a document. When the only signal your automation emits is an exit code, you get told "success" for an entire class of failures that never touches the exit code. Structural validity has to be asserted explicitly, the same way you assert that a Kubernetes manifest is valid before it merges rather than after it's applied to a live cluster.

The fix, part one: stop using substitution for a positional problem

The wrapper is a positional artifact. It lives on line 1 and on the final non-empty line, and nowhere else. A substitution rule anchored with ^...$ has no concept of position, so it was the wrong tool from the first commit.

What replaced it matches on a signature rather than a pattern, and touches exactly two lines:

unwrap_markdown_fence() {
  local file="$1"
  local l1 l2 last lastno fences

  l1=$(sed -n '1p' "$file")
  l2=$(sed -n '2p' "$file")
  lastno=$(grep -n '.' "$file" | tail -n1 | cut -d: -f1)
  last=$(sed -n "${lastno}p" "$file")

  # Signature of a whole-document wrapper:
  # line 1 is a fence, line 2 opens frontmatter, last non-empty line closes it.
  case "$l1" in
    '```markdown'|'```md'|'```') ;;
    *) return 0 ;;
  esac
  [[ "$l2" == '---' ]] || return 0
  [[ "$last" == '```' ]] || return 0

  # Even fence count means the wrapper pairs cleanly with itself.
  fences=$(grep -c '^```' "$file")
  (( fences % 2 == 0 )) || return 0

  sed -i "1d;${lastno}d" "$file"
}

Checking line 2 for --- is what makes this safe. A wrapped document always has the frontmatter delimiter immediately after the opener, because that's how every post in the collection starts. A legitimate post that happens to open with a code fence will not have --- sitting on line 2.

Every guard clause returns 0 instead of failing. If the signature doesn't match, the function does nothing and the file passes through untouched. A cleanup step should be inert when it isn't confident, not aggressive. The old rule was the opposite: maximally confident, zero context.

The fix, part two: assert structure, don't count it

Unwrapping correctly is necessary but not sufficient, because the model itself can drop a closing fence. Any repair you write can also have bugs. So the pipeline needed something that reads the finished file and asserts the body is actually prose.

My first instinct was to count fences and check for an even number. That check is close to worthless. A document with a missing closer and a stray opener has an even count and is still broken, and a post that legitimately uses four-backtick fences to demonstrate nested markdown throws the count off entirely. Counting tells you about symbols. You need to know about state.

So check-fences.sh walks the file, tracks whether it's inside a code block, and looks for content that has no business being there:

#!/usr/bin/env bash
set -euo pipefail
file="$1"

awk '
  /^```/ {
    if (!in_code) { in_code=1; open_line=NR } else { in_code=0 }
    next
  }
  # Markdown structures that should never appear inside a fence
  in_code && /^#{1,6} / {
    printf "line %d: heading inside code block opened at line %d\n", NR, open_line
    bad=1
  }
  in_code && /^\|.*\|$/ {
    printf "line %d: table row inside code block opened at line %d\n", NR, open_line
    bad=1
  }
  END {
    if (in_code) {
      printf "unclosed fence opened at line %d (EOF)\n", open_line
      bad=1
    }
    exit bad
  }
' "$file"

Heading detection is what actually catches the sed bug. A swallowed document always contains ## Something inside the runaway block, because posts have sections. Table rows catch the same failure in posts built around comparison tables. The end-of-file check catches the simpler case where the model just forgot a closer.

False positives exist and I decided to live with them. A shell script demonstrating comment syntax can contain a line starting with #, but the awk pattern requires a space after one to six hashes at column zero, and code comments in the posts I generate rarely look like a markdown heading with a capitalized sentence after it. When it does misfire, the failure mode is a rejected draft rather than a published page of gray monospace, and I'll take that trade every time.

The fix, part three: repair before you reject

A validator that only says no creates its own failure mode. The instinct when a generation step fails is to retry, but retrying an expensive model call to fix a mechanical, deterministic defect is the wrong shape of solution. Worse, the regenerated draft is a different draft. You've thrown away good prose to fix a missing three-character line, and the new attempt can trip a different gate entirely.

Corruption from a missing closer has exactly one correct repair, so repair-fences.sh performs it directly and runs before the check:

#!/usr/bin/env bash
set -euo pipefail
file="$1"

awk '
  /^```/ { in_code = !in_code; print; next }
  # A heading inside a code block means the previous fence never closed.
  in_code && /^#{1,6} / { print "```"; in_code=0; print; next }
  { print }
  END { if (in_code) print "```" }
' "$file" > "${file}.tmp" && mv "${file}.tmp" "$file"

Ordering matters here. Repair runs first, the checker runs second, and the checker is the gate. If repair does its job, the check passes and the draft survives. If repair produces something still structurally broken, the check fails and the pipeline stops with a specific line number instead of a shrug. Repair is allowed to be optimistic precisely because it isn't the thing making the final call. That separation of "attempt a fix" from "verify the fix" is the same discipline that makes plan-and-apply workflows trustworthy in infrastructure automation, and it applies just as well to a text pipeline.

One caveat worth stating plainly: a repair step that mutates content is a liability if you can't see what it did. Mine writes a line to the run log whenever it changes the file, including the line number where it inserted a closer. Silent auto-repair is how you end up debugging a bug that a script already "fixed" three stages earlier.

Where the gates fight each other

Content quality gates and structural gates want different things, and stacking them naively produces a pipeline that rejects everything.

My anti-slop checker is a hard gate. Banned words, banned phrases, an em-dash budget, a rule against consecutive paragraphs opening with the same word. It's deterministic and it's strict, which is the point. But a hard gate on prose quality combined with a hard gate on structure means a draft has to clear both in a single generation, and the retry cost is a full regeneration.

The ordering that works for me: run mechanical repairs first (fence balance, wrapper stripping, frontmatter normalization), then run the structural gate, then run the quality gate, and only regenerate on quality failures. Mechanical problems get fixed in place because they have one right answer. Prose problems get sent back to the model with the specific violations included in the retry prompt, which turns a blind retry into a targeted revision. Feeding the checker's own output back into the next attempt is what makes the loop converge instead of thrashing, and it's the same feedback-shaped design that keeps multi-layer agent memory systems from degrading over repeated passes.

Anything a script can repair deterministically should never reach the model twice. That's the rule I landed on, and the fence bug is the clearest example of why: a missing ` cost an entire draft and a model call when a five-line awk program fixes it in milliseconds.

Why this matters beyond one blog

Text transformation with line-anchored regex is a trap any time the text has stateful syntax, and markdown is full of it. Fences toggle. List indentation nests. Frontmatter delimiters and horizontal rules are the same three characters in different positions. HTML comments and YAML block scalars both swallow content until a terminator. A sed expression sees a line. Your document has a grammar, and lines don't know what mode they're in.

Two habits keep this from biting:

Match on position or signature, not on pattern, when the artifact is positional. If what you're stripping is "line 1 and the last line," write a rule that says line 1 and the last line. Anchors are not position.
Assert the shape of the output, not just the presence of the output. Word count, file existence, and frontmatter fields all confirm that something landed on disk. None of them confirm it's a document. A structural check that understands the format is the only thing that closes the gap.

The broader pattern generalizes past blog tooling. Any pipeline that pipes model output through shell transformations and into a build system has this exposure, and the failure is quiet by construction: exit code 0, valid file, wrong document. If you're building automation where a model's output feeds downstream systems and you'd rather not discover the corruption from a reader, that validation layer is worth designing up front. It's a chunk of what I end up building for people in agent and pipeline work, and it's almost always cheaper than the cleanup.

What I'd do differently: I'd have written check-fences.sh before I wrote any transformation step at all. The validator is 20 lines of awk. The sed rule it protects against took a week of published output to notice, and noticing required a human looking at a rendered page. Automation that can't tell you it's broken isn't automation, it's a faster way to be wrong.

Tailscale Kernel TUN in Unprivileged LXC: Direct SSH Without Userspace Networking

Guatu — Sat, 22 Aug 2026 06:15:48 +0000

tailscale up --tun=userspace-networking gets you a green dot in the admin console and almost nothing else. The node appears in your tailnet, tailscale status looks healthy, and then you try to SSH into that container from your laptop and the connection hangs until TCP gives up. Two lines in the LXC config file fix it, and the container stays unprivileged.

That's the whole post, really. But those two lines only make sense once you understand why every guide pushes you toward userspace mode in the first place, and what you're giving up by staying there.

Who should care

Anyone running services in unprivileged LXC containers on Proxmox who wants those containers to be real tailnet members with their own 100.64.0.0/10 address. Not reachable through something else. Reachable directly, over WireGuard, with a kernel network interface that ip addr can see.

If you're already routing everything through a subnet router, you have a working setup and this is an optional upgrade. I covered that pattern in Tailscale Subnet Routers. Treat this as the next rung on the ladder: instead of one node advertising routes on behalf of everyone else, each container carries its own identity, its own ACL surface, and its own direct path to peers.

What userspace networking actually costs you

Every LXC-and-Tailscale guide I've read lands on the same instruction: pass --tun=userspace-networking and move on. It works because it sidesteps the problem entirely. Rather than asking the kernel for a TUN device, tailscaled runs a userspace TCP/IP stack (gVisor's netstack) inside its own process and never opens /dev/net/tun.

Those costs stay invisible until you trip over one.

Outbound traffic needs a proxy. In userspace mode, tailscaled exposes SOCKS5 and HTTP proxies on a local port. Nothing on the system routes to 100.64.0.0/10 automatically, because there is no interface and no route. Every client has to be told about the proxy:

# userspace mode: this is the only way out
export ALL_PROXY=socks5://localhost:1055/
curl http://100.64.0.20:8080/health

Miss that env var in a systemd unit, a cron job, or a nested container, and the traffic silently takes the normal default route instead. No error. It just goes somewhere else, which is the worst failure mode a network can have.

Inbound to a normal daemon doesn't happen. Your sshd binds 0.0.0.0:22 on the container's LAN interface. Packets arriving over the tailnet terminate inside tailscaled's netstack, and there's no path from netstack to a socket the kernel owns unless you build one explicitly with tailscale serve, or you switch to Tailscale SSH where tailscaled itself is the SSH server. Tailscale SSH genuinely works in userspace mode, which is why plenty of people never notice the limitation. But that's tailscaled's SSH implementation, not OpenSSH. If you care about host key pinning, authorized_keys command restrictions, Match blocks, or an sshd_config you've tuned, you now maintain two parallel SSH stories on the same box.

UDP is a mess. Netstack's UDP support has been partial for years. Concrete symptom: mosh does not work. mosh-server starts fine, prints its port and key, and the client sits there forever because the datagrams never arrive. If you SSH over flaky mobile links and lean on mosh to survive roaming, userspace mode is a dead end.

None of this is a bug. It's a documented tradeoff, and for a container that only needs to call out to a couple of HTTP endpoints, it's a perfectly reasonable one. The problem is that most guides present it as the answer rather than the fallback it actually is.

The false starts

First instinct, recommended constantly on forums: make the container privileged. Set unprivileged: 0, restore, done, TUN works. It also hands the container a root that maps to real host root, which throws away the single most valuable property of an unprivileged container. For a machine that's about to sit on a VPN and accept inbound connections from anywhere, that trade is exactly backwards.

Second instinct: --features nesting=1. That flag shows up in nearly every Proxmox thread about containers doing unusual things, and it's genuinely required for Docker-in-LXC and systemd cgroup delegation. It does nothing for TUN. Nesting controls whether the container can see and mount its own cgroup hierarchy and its own procfs/sysfs views. Device access is a completely separate mechanism. Set it, restart, observe zero change, spend twenty minutes wondering what you got wrong.

AppArmor is the third dead end. There's a pile of advice suggesting lxc.apparmor.profile: unconfined to get device access working. That's a sledgehammer for a problem AppArmor isn't causing. The default lxc-container-default-cgns profile does not block /dev/net/tun usage; it blocks a set of mount, ptrace, and /proc write operations, none of which sit in the path here. Dropping confinement to fix a device permission issue is the kind of change that looks like it worked and quietly widens the blast radius. I've seen this pattern bite people in other contexts too, which is the same theme as the runc sysctl trap: the fix that "works" is usually the one that removed a boundary you wanted.

What's actually stopping you is much narrower than any of that.

The actual fix

Three conditions have to hold: the host has the tun module loaded, the container's device cgroup allows that specific char device, and the device node is bind-mounted into the container's filesystem.

1. Load `tun` on the host and persist it

On the Proxmox host:

lsmod | grep -w tun || modprobe tun
ls -l /dev/net/tun
# crw-rw-rw- 1 root root 10, 200 Aug 14 09:12 /dev/net/tun

Persist it so a host reboot doesn't quietly undo everything:

echo tun > /etc/modules-load.d/tun.conf

Do this before the next step. The bind mount below has a source path on the host. If /dev/net/tun doesn't exist when the container starts, the mount fails and the container refuses to start, which is a confusing way to learn that a kernel module wasn't loaded.

Repeat this on every node in the cluster. A container that migrates to a host without tun loaded will fail to start there, and you'll be debugging a mount error at the worst possible moment.

2. Add two lines to the container config

Edit /etc/pve/lxc/<CTID>.conf on the host that currently owns the container:

lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

That's it. No unprivileged: 0, no AppArmor changes, no nesting. Two details matter in the second line. The target path dev/net/tun has no leading slash, because LXC resolves mount targets relative to the container rootfs. And create=file tells LXC to create the target inode if it doesn't already exist, which it won't on a fresh container.

The first line uses cgroup v2 syntax. Anything running Proxmox 8.x is on the unified hierarchy, so lxc.cgroup2.devices.allow is what you want. Older setups on cgroup v1 used lxc.cgroup.devices.allow with the same argument format; if you're copying a snippet from a 2019 forum post, check which one it uses.

3. Stop and start the container from the host

This part catches people. Rebooting from inside the container does not re-read the config file, because the config is consumed by LXC on container creation, not by the init system inside it. You need a full stop/start cycle driven from the host:

pct stop <CTID> && pct start <CTID>

Until you do that, /dev/net/tun simply won't exist inside the container, and you'll assume the config was wrong.

4. Drop the userspace flag and bring Tailscale up

If tailscaled was previously running in userspace mode, that flag lives in /etc/default/tailscaled:

# /etc/default/tailscaled - before
FLAGS="--tun=userspace-networking"

# after
FLAGS=""

Then restart and authenticate:

systemctl restart tailscaled
tailscale up --hostname=app-container-01 --accept-dns=false

I set --accept-dns=false on servers by habit. MagicDNS rewriting /etc/resolv.conf on a box that already has opinions about DNS is a fight you don't need, especially if you're running your own resolver like AdGuard Home. Turn it on deliberately if you want it.

Verifying it actually worked

Four checks, in order. Each one fails differently, which is what makes them useful.

# 1. The device exists and is world-writable
ls -l /dev/net/tun
# crw-rw-rw- 1 nobody nogroup 10, 200 Aug 14 09:12 /dev/net/tun

# 2. A real kernel interface, not a netstack fiction
ip addr show tailscale0
# 3: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 ...
#     inet 100.64.0.31/32 scope global tailscale0

# 3. Routes for the tailnet are installed in the container's table
ip route | grep 100.64
# 100.64.0.0/10 dev tailscale0 scope link

# 4. Peer connectivity is direct, not relayed
tailscale ping 100.64.0.20
# pong from db-node (100.64.0.20) via 192.0.2.14:41641 in 1ms

That fourth line is the one worth reading carefully. via <ip>:<port> means a direct WireGuard tunnel between the two hosts. If it says via DERP(sfo) instead, you're relaying through Tailscale's infrastructure, which still works but adds latency and depends on someone else's servers. tailscale netcheck will tell you whether UDP is reachable and whether you're behind a NAT that's blocking direct paths.

Then test the thing you came for:

ssh user@100.64.0.31       # OpenSSH, your config, your keys
mosh user@100.64.0.31      # UDP works now

Why it works

Three separate mechanisms have to cooperate, and the reason so much bad advice exists is that people conflate them.

Device cgroup allowlist. An unprivileged LXC container starts with a deny-by-default device policy and a short allowlist covering the basics: /dev/null, /dev/zero, /dev/full, /dev/random, /dev/urandom, /dev/tty, and the pty devices. The TUN driver registers as char device major 10, minor 200. It's not on that list, so any open() on it returns EPERM regardless of file permissions. Adding c 10:200 rwm grants read, write, and mknod for exactly one device node and nothing else. That's the surgical part: you're not opening a category, you're naming a device.

The bind mount. Passing the cgroup check gives permission to use the device, but the container's /dev is a fresh tmpfs populated by LXC. Creating the node yourself with mknod inside the container fails, because unprivileged containers don't hold CAP_MKNOD in a namespace where it means anything for device creation. Bind-mounting the host's existing node sidesteps that entirely: the inode already exists, LXC just makes it visible at the right path.

Namespaced capability. Opening the device is only half the operation. Actually creating an interface requires the TUNSETIFF ioctl, which needs CAP_NET_ADMIN. Here's the part that makes the whole approach safe: CAP_NET_ADMIN is evaluated against the user namespace that owns the network namespace. The container's root has full CAP_NET_ADMIN over its own netns, and zero authority over the host's. So tailscaled can create tailscale0, set addresses, and install routes inside the container while remaining completely unable to touch host networking. No host privilege is granted at any point.

About that `nobody:nogroup`

Ownership showing as nobody:nogroup inside the container is expected, not a symptom. The host node is owned by uid 0. An unprivileged container maps its uid 0 to something like host uid 100000, and host uid 0 falls outside the mapped range, so the kernel reports it as the overflow uid (65534) which resolves to nobody. The mode is 0666, so every process in the container can open it anyway. People see nobody:nogroup, assume permissions are broken, and go chase a chown that will never work. Nothing to fix.

Direct paths versus the subnet router model

With a kernel interface, the container installs a route for 100.64.0.0/10 and encrypts packets itself. Traffic to a peer leaves as UDP on tailscaled's port (41641 by default) over the container's normal LAN interface, straight to the peer's endpoint. No hairpinning through a subnet router, no dependency on one container staying healthy, no single node whose reboot takes out remote access to a dozen services.

The ACL story improves too. A subnet router advertises a CIDR, so your policy can only reason about IP ranges. Per-node membership lets you tag containers and write rules against identity:

"acls": [
  {
    "action": "accept",
    "src":    ["tag:admin"],
    "dst":    ["tag:lxc-app:22", "tag:lxc-app:3000"]
  }
]

Tag the container at join time with tailscale up --advertise-tags=tag:lxc-app, using an auth key whose owner is permitted to assign that tag. Now access is a property of the machine, not a property of whatever subnet it happens to sit in. That distinction matters a lot once more than one person needs access, and it's the same principle behind two-tier service accounts for agents: scope by identity, not by network position.

Before you restart anything

Step 3 requires stopping the container, which means killing every process inside it. Worth five minutes of preparation:

Save your terminal state. If you live in tmux inside that container, prefix + Ctrl-s with tmux-resurrect saves the session layout and running programs. Losing a session with twelve panes of half-finished debugging is a self-inflicted wound.
Confirm a recent backup. pct listsnapshot <CTID> or a check in Proxmox Backup Server. The config change is easy to revert, but "the container won't start now" is much less stressful with a restore point.
Know how you'll get back in. pct enter <CTID> from the host works regardless of container networking. If your only access path is SSH over the very network you're changing, fix that first.
Record the change. Two lines in a config file are trivially forgettable six months later when a fresh container mysteriously lacks TUN. Put them in whatever provisioning script or template you use, alongside the rest of your cluster build notes.

Lessons learned

Device access is not privilege. The instinct to reach for unprivileged: 0 comes from treating container security as a binary. It isn't. The device cgroup, capability sets, mount entries, and idmaps are independent knobs, and almost every "just make it privileged" answer online is someone who found one knob and gave up on the rest. Name the exact device you need and grant that.

Flags are not interchangeable. nesting=1 fixes cgroup visibility. AppArmor profiles gate mounts and ptrace. Neither touches /dev. Reading the actual mechanism took less time than trying flags at random, and I'd skip straight to the mechanism next time.

Load the module first, always. Container start failing on a missing bind-mount source is a confusing error to debug backwards, and the /etc/modules-load.d/tun.conf file has to exist on every node the container can migrate to. Cluster-wide, not per-node.

Pick one SSH story. Running both OpenSSH and Tailscale SSH on the same node means two authorization models, two audit trails, and two places to revoke access. With a kernel TUN device you get to use plain OpenSSH over the tailnet, so I turn Tailscale SSH off on these containers rather than leaving both paths live.

The surprise was how boring the fix is. Two config lines, one module, one restart. All the complexity lives in understanding why the defaults deny it, which is generally where the interesting part of infrastructure work hides. If you're designing this kind of access model for a team rather than a homelab, that's a conversation I have often through GuatuLabs, and the boundary questions get more interesting once compliance and multiple operators enter the picture.

One caveat worth stating plainly: this gives the container a kernel TUN device, not the right to advertise subnet routes. If you also want it acting as a subnet router, you need IP forwarding enabled inside the container and --advertise-routes on the Tailscale side, which is a different set of tradeoffs. For the common case (a container that should be its own node, reachable directly, with working UDP and your real sshd), the two lines above are the entire job.

Systemd-tmpfiles vs Plex: When /tmp Aging Breaks EAC3 Transcoding

Guatu — Sat, 22 Aug 2026 04:15:47 +0000

A Plex server with 90 days of uptime, a green service status, and perfectly working H.264 playback can still fail every single EAC3 transcode with a generic "Conversion failed" error. Nothing about Plex changed. No update, no config edit, no disk full. A systemd timer quietly deleted the transcoder's scratch directory out from under it, and Plex has no idea.

This one is worth understanding if you run Plex on any systemd-based Linux, whether that's bare metal, a VM, or an unprivileged LXC on Proxmox. It's especially relevant right now because Debian 13 enabled automatic /tmp cleaning by default, so setups that ran fine for years on Debian 11 or 12 inherit this failure mode on upgrade without touching a single Plex setting.

The failure mode

Plex handles most audio transcoding with its bundled ffmpeg fork. But Dolby Digital Plus (EAC3) is licensed differently, so Plex ships a separate binary called EasyAudioEncoder, usually abbreviated EAE. It doesn't work like a normal encoder that reads stdin and writes stdout. It's a watch-folder daemon: Plex drops audio into a directory, EAE picks it up, converts it, and writes the result back. That watch folder lives under a per-instance temp directory that looks like this:

/tmp/pms-4f3a2b1c-9d8e-7f6a-5b4c-3d2e1f0a9b8c/EasyAudioEncoder/

The important detail: Plex creates that directory tree exactly once, when the plexmediaserver service starts. There's no health check on it, no recreation logic, no periodic touch. As long as the service keeps running, Plex assumes the directory it made on day one is still there.

Meanwhile, systemd-tmpfiles has its own opinion about /tmp. On most modern distros, /usr/lib/tmpfiles.d/tmp.conf contains a rule like this:

# Clear /tmp entries older than 10 days, /var/tmp after 30
q /tmp 1777 root root 10d
q /var/tmp 1777 root root 30d

And systemd-tmpfiles-clean.timer fires 15 minutes after boot and then once a day, removing anything under /tmp whose access, modification, and change times are all older than the age threshold. If you're a heavy Plex user who transcodes EAC3 daily, the timestamps stay fresh and you never notice. If your household mostly direct-plays and only occasionally hits EAC3 content, the watch folder sits idle, crosses the 10-day line, and gets reaped.

From that point on, the server is a zombie in one specific dimension. Everything that doesn't need EAE works. Anything that does fails at session start.

What it looks like in the logs

The client-side error is useless: "Conversion failed. The transcoder failed to start up." The actual story is in Plex Media Server.log:

ERROR - [Req#a4f/Transcode] EAE watchfolder is not writable:
  /tmp/pms-4f3a2b1c-.../EasyAudioEncoder/Convert to WAV (to 8ch or less) (~4f3a)/
ERROR - [Req#a4f/Transcode] Transcode runner appears to have died
DEBUG - Job exit code 187

Exit code 187 from the transcoder job plus "EAE watchfolder is not writable" is the signature. If you go look, the path either doesn't exist at all or the pms-* parent is gone entirely. You can confirm the cleanup timer is the culprit by checking when it last ran:

systemctl list-timers systemd-tmpfiles-clean.timer
ls -ld /tmp/pms-* 2>/dev/null || echo "gone"

The immediate fix is boring:

systemctl restart plexmediaserver

Restarting recreates the whole /tmp/pms-* tree and EAC3 transcodes work again, instantly. Which is exactly why this problem generates so many confused forum threads that end with "restarted it and it works now, no idea why." The restart fixes the symptom and destroys the evidence, and ten days of idle EAE later it's back.

The permanent fix: a tmpfiles.d exclusion

You don't need to disable /tmp cleaning, and you shouldn't. The tmpfiles.d format has an exclusion type built for exactly this. Drop one file on the host running Plex:

# /etc/tmpfiles.d/plex-eae.conf
# Exempt Plex's per-instance temp tree from age-based cleanup.
# Lowercase 'x' ignores the path AND everything below it.
x /tmp/pms-*

Then verify systemd actually picked it up:

systemd-tmpfiles --cat-config | grep -A1 plex-eae

The character matters here, and it's an easy place to get burned. tmpfiles.d defines two exclusion types that look almost identical:

x /tmp/pms-* excludes the matching paths and their entire contents from cleaning. This is what you want.
X /tmp/pms-* excludes only the directory itself. Its contents still age out, which puts you right back in the same failure with a slightly different shape: the pms-* directory survives but the EasyAudioEncoder subtree inside it gets reaped.

Globs are legal in exclusion rules and evaluated at cleaning time, so the rule keeps working even though the UUID in the directory name changes on every service restart.

To prove the fix without waiting ten days, force a cleanup pass with an aggressive age and watch the directory survive:

# Dry-run style check: run the cleaner with a 1-second age threshold
touch -d "30 days ago" /tmp/pms-*/EasyAudioEncoder 2>/dev/null
systemd-tmpfiles --clean
ls -ld /tmp/pms-*/EasyAudioEncoder && echo "exclusion works"

If the directory is still there after --clean runs against a backdated mtime, the rule is live. This kind of "verify the fix actually engages" step is the same discipline I argued for in the Longhorn read-only mounts post: a service that reports healthy while an underlying filesystem assumption has silently broken is the worst class of failure, because nothing pages you until a user complains.

The alternative fix: move EAE out of /tmp entirely

If you'd rather not maintain an exclusion rule, you can relocate Plex's temp tree to a path systemd-tmpfiles doesn't manage. EAE derives its working directory from the process environment, not from the "Transcoder temporary directory" setting in the Plex UI. That UI setting moves the video transcode segments, but the EAE watch folders follow TMPDIR. So changing the setting in the web UI does not fix this problem, which is a trap plenty of people fall into before finding the real mechanism.

A systemd drop-in handles it cleanly:

systemctl edit plexmediaserver

[Service]
Environment=TMPDIR=/var/lib/plexmediaserver/tmp
ExecStartPre=/usr/bin/mkdir -p /var/lib/plexmediaserver/tmp

After a daemon-reload and restart, the pms-* tree lands under a path with no age rule attached. The tradeoff is that nothing cleans it up anymore, so orphaned trees from unclean shutdowns accumulate until you notice. The x exclusion in /tmp doesn't have that problem, because the directory still gets wiped on reboot like everything else in /tmp. That's the main reason I'd pick the exclusion rule over the relocation: it changes the minimum necessary behavior and leaves the rest of the /tmp lifecycle intact.

There's a third option worth knowing about even if you don't use it: PrivateTmp=true in the unit. Systemd's private tmp directories get mounted under /tmp/systemd-private-*, and stock tmp.conf ships exclusion rules for those paths, so they're immune to aging by design. Plex's packaged unit doesn't enable it, and flipping it on changes how the service sees paths shared with other processes, so I'd treat it as trivia rather than the recommended fix here.

Gotchas and adjacent traps

Debian 13 flipped the default. Debian historically did not age out /tmp contents on a running system, which is why this bite pattern was mostly a Fedora/Arch/RHEL story for years. Trixie moved /tmp to tmpfs and enabled the 10-day /tmp and 30-day /var/tmp cleanup rules by default. If your Plex LXC or VM got upgraded to Debian 13, you inherited this behavior with zero warning. The exact same Plex install that ran for two years without an EAE incident now has a ten-day fuse that resets only on service restart.

LXC makes the uptime trap worse. Containers restart less often than you think. A Proxmox host reboot cycles every VM, but LXCs with long-running services rack up months of uptime effortlessly, and every one of those days is a day the tmpfiles timer inside the container keeps ticking. Long uptime feels like a stability trophy. For any service that stages state in /tmp at startup and never checks on it again, uptime is actually the risk factor. If you're running Plex inside an unprivileged LXC (a reasonable choice, and the general pattern is covered in Building a Production Homelab), put the tmpfiles.d exclusion inside the container, since that's where the timer that does the reaping lives.

Docker installs behave differently. A Plex container image doesn't run systemd inside it, so there's no tmpfiles timer to reap anything, and /tmp lives and dies with the container. This failure mode is specific to package installs on systemd hosts and systemd-based LXCs. If you migrated from Docker to a native package and suddenly hit this after a couple of weeks, that's why.

Don't fix it by disabling the timer. Masking systemd-tmpfiles-clean.timer works, but now nothing ages out /var/tmp either, and every other well-behaved consumer of the aging policy loses it too. Scoped exclusions exist precisely so you don't have to make that trade.

Plex isn't the only service with this shape. Anything that creates sockets, PID files, or working directories under /tmp at startup and holds a path reference for the life of the process is exposed. PostgreSQL's Unix socket default, some JVM apps with java.io.tmpdir scratch state, and various daemons with startup-created lock directories all have variants of this. Distros ship tmpfiles exclusions for the common ones (X11 sockets, systemd's own private dirs). Third-party packages are on their own, and Plex's package doesn't ship one. Auditing long-running services for this dependency class is the kind of infrastructure-drift review I end up doing in consulting engagements more often than any glamorous architecture work, because drift between OS policy and application assumptions is invisible until it isn't.

Takeaway

The application didn't break. The OS didn't malfunction. Two correct behaviors, Plex creating its EAE workspace once at startup and systemd-tmpfiles aging out idle /tmp entries, combined into a failure with a built-in delay timer. That's what makes it nasty: the cause and the symptom are separated by ten days and belong to two different layers of the stack.

The fix is a one-line file, x /tmp/pms-* in /etc/tmpfiles.d/plex-eae.conf, verified with systemd-tmpfiles --cat-config and a forced --clean pass. Reach for this pattern any time a long-running service treats /tmp as if it were persistent, and be suspicious of any "it works after a restart" fix where nobody can explain what the restart actually repaired. On a systemd host, a startup-created /tmp directory plus long uptime is a scheduled outage; the only question is the date.

Pushgateway Heartbeat Gotcha: When ndots and NetworkPolicy Silently Eat Your Alerts

Guatu — Sat, 22 Aug 2026 02:15:49 +0000

A CronJob exits 0, Pushgateway shows a heartbeat metric that hasn't updated in days, and Prometheus fires nothing. Every dashboard is green while the one thing the heartbeat was supposed to guarantee (that you'd know when the job stops working) has quietly stopped being true.

This is a gotcha about two Kubernetes features that are each fine on their own: the default ndots:5 DNS behavior and namespace-scoped NetworkPolicies. Put them in the same cluster with a wildcard DNS record and a push-based monitoring pattern, and you get a failure mode where the monitoring infrastructure itself is the victim, which means nothing tells you it happened.

What I expected

The heartbeat pattern for batch jobs is simple and well established. Prometheus can't scrape a pod that lived for forty seconds, so instead the job pushes a timestamp to Pushgateway when it finishes, Prometheus scrapes Pushgateway on its normal cycle, and an alert rule fires if the timestamp goes stale. A dead man's switch:

# Last line of the backup job's script
echo "backup_last_success_timestamp $(date +%s)" | curl -s --data-binary @- \
  "http://pushgateway.monitoring.svc.cluster.local:9091/metrics/job/nightly-backup"

- alert: BackupTooOld
  expr: time() - backup_last_success_timestamp > 86400
  for: 15m
  labels:
    severity: warning

If the backup breaks, the timestamp stops advancing, the expression crosses the threshold, and I get paged. That's the theory. The name in the curl command is even the full service FQDN, so DNS should be unambiguous. It is not.

What actually happened

Three independent failures stack here, and each one masks the next. That stacking is the whole gotcha; any single layer would have been a five-minute fix.

Layer 1: ndots:5 turns your FQDN into a wildcard lookup

A pod with the default dnsPolicy: ClusterFirst gets a resolv.conf that looks like this:

search ai-jobs.svc.cluster.local svc.cluster.local cluster.local lab.example.com
nameserver 10.96.0.10
options ndots:5

Note the last search domain. The kubelet appends the node's own search domains after the cluster ones, so if your nodes carry an internal domain like lab.example.com, every pod inherits it.

Now count the dots in pushgateway.monitoring.svc.cluster.local. Four. The ndots:5 option means any name with fewer than five dots is treated as relative, and the resolver walks the search list before trying the name as-is. So your "fully qualified" name generates these queries, in order:

pushgateway.monitoring.svc.cluster.local.ai-jobs.svc.cluster.local → NXDOMAIN
pushgateway.monitoring.svc.cluster.local.svc.cluster.local → NXDOMAIN
pushgateway.monitoring.svc.cluster.local.cluster.local → NXDOMAIN
pushgateway.monitoring.svc.cluster.local.lab.example.com → an answer

Step 4 is the trap. If your internal DNS has a wildcard record (*.lab.example.com pointing at your ingress controller's LoadBalancer VIP, a common setup for homelab TLS and per-app hostnames), that query matches the wildcard and returns the ingress VIP. The resolver got an answer, so it stops. The actual absolute name, the one that resolves to the real Pushgateway ClusterIP, never gets queried.

Your heartbeat is now aimed at Traefik on port 9091. I've written about this exact mechanism corrupting TLS validation in Wildcard DNS + ndots:5: The TLS Nightmare; the Pushgateway variant is nastier because the failure has no user-facing symptom at all.

The wildcard also changes the character of the failure. Without it, step 4 would return NXDOMAIN, the resolver would fall through to the absolute name, and everything would work (slower, with three wasted queries, but working). The wildcard converts a harmless inefficiency into a wrong answer.

Layer 2: NetworkPolicy makes the wrong answer unreachable, quietly

The monitoring namespace runs default-deny ingress, the pattern from Network Policies with Calico: Default Deny and Namespace Isolation. Pushgateway has an allow rule for Prometheus scrapes and for the namespaces that existed when the policy was written.

Two things go wrong at this layer. First, the connection isn't even headed to Pushgateway anymore; it's headed to a LoadBalancer VIP that sits outside the pod and service CIDRs. If the workload namespace has an egress policy scoped to in-cluster traffic, that packet gets dropped on the way out. Second, even after you fix DNS, a new namespace (say you just added an ai-jobs namespace for LLM batch work) isn't in Pushgateway's ingress allow-list, so its pushes get dropped on arrival instead.

Either way, the drop is silent by design. NetworkPolicy denies don't send RST packets or ICMP errors with most CNI configurations. The curl just hangs until its TCP timeout, which on a default curl with no --max-time can be over two minutes. The job runtime gets longer and nobody notices, because who watches the runtime of a job that's succeeding?

This is the maintenance-burden face of NetworkPolicy that doesn't show up in tutorials: an allow-list written in March is wrong by August, because namespaces get added and every new one silently falls outside the rules.

Layer 3: both the script and the alert fail open

Here's where it becomes invisible. The push was the last line of the script, and it looked like this in the failure case:

curl -s --data-binary @- "http://..." || true

The || true was there so a flaky Pushgateway wouldn't fail an otherwise-good backup. Reasonable instinct, terrible outcome: the curl times out, the job still exits 0, and Kubernetes reports the CronJob as succeeding.

The alert rule fails open too, and this part catches almost everyone. time() - backup_last_success_timestamp > 86400 only evaluates when the series exists. Pushgateway restarted at some point (it stores pushed metrics in memory unless you enable persistence), the series vanished, and the expression started returning an empty result. Empty isn't "greater than 86400." Empty is nothing. No data, no alert.

So: the DNS layer sends the heartbeat to the wrong place, the network layer drops it without an error, the script layer hides the failure from Kubernetes, and the alerting layer can't fire on a metric that doesn't exist. Four layers, zero signals.

The fix

Work from the inside out: make the failure loud first, then fix the path.

1. Make the push failure fail the job. Drop the || true, add -f so HTTP errors count, and bound the timeout so a network drop fails in seconds instead of minutes:

echo "backup_last_success_timestamp $(date +%s)" | \
  curl -fsS --max-time 10 --retry 2 --data-binary @- \
  "http://pushgateway.monitoring.svc.cluster.local.:9091/metrics/job/nightly-backup"

Notice the trailing dot on the hostname. That marks the name as absolute and bypasses the search list entirely, which makes this one command immune to the ndots problem regardless of pod configuration. It's the cheapest fix in this whole post.

2. Fix ndots at the pod level. The trailing dot fixes one command; dnsConfig fixes every lookup the pod makes:

apiVersion: batch/v1
kind: CronJob
spec:
  jobTemplate:
    spec:
      template:
        spec:
          dnsConfig:
            options:
              - name: ndots
                value: "2"

With ndots:2, any name with two or more dots gets tried as an absolute name first. pushgateway.monitoring.svc.cluster.local resolves on the first query, and even the short form pushgateway.monitoring still works because the search list is consulted after the absolute attempt fails. You lose nothing for cluster-internal names and you stop feeding four-dot FQDNs to your wildcard record.

3. Expand the NetworkPolicy with labels, not names. Enumerating source namespaces by name is what made the policy rot when a new namespace arrived. Select on a label instead:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-pushgateway-clients
  namespace: monitoring
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: pushgateway
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              pushgateway-client: "true"
      ports:
        - port: 9091
          protocol: TCP

Then onboarding a new batch namespace is one label away: kubectl label namespace ai-jobs pushgateway-client=true. The policy stops being a file you have to remember to edit and becomes a contract the namespace opts into. Pair it with a Kyverno rule that requires the label on namespaces containing CronJobs if you want it enforced rather than remembered.

4. Alert on absence, not just staleness. The dead man's switch needs to fire when the series is missing, because "missing" is exactly what the failure mode produces:

- alert: BackupHeartbeatMissing
  expr: |
    (time() - backup_last_success_timestamp > 86400)
    or
    absent(backup_last_success_timestamp)
  for: 15m
  labels:
    severity: critical

The absent() function returns a value of 1 when the series doesn't exist, and the or makes the rule cover both stale and gone. One caveat: absent() collapses labels, so if you push heartbeats for several jobs under one metric name with a job label, you need one absent() clause per job (or a recording-rule pattern). Tedious, but the alternative is the exact blind spot this post is about. The reasoning behind alerting on the negative space is the same one I laid out in Prometheus Alerting Rules That Don't Cry Wolf: an alert that can't fire when the pipeline feeding it dies isn't an alert, it's decoration.

5. Verify the path from inside the namespace. Don't trust the fix until you've watched the resolution happen where the workload runs:

kubectl run -n ai-jobs dbg --rm -it --restart=Never \
  --image=nicolaka/netshoot -- \
  getent hosts pushgateway.monitoring.svc.cluster.local

If that returns your ingress VIP instead of a ClusterIP, the wildcard is still winning. Follow with a curl -v --max-time 5 to port 9091 to confirm the NetworkPolicy actually passes traffic, because DNS being right says nothing about the packet getting through.

Why this matters

You'll hit this if your cluster combines three things that are each individually recommended: a wildcard DNS record for ingress convenience, default-deny NetworkPolicies, and push-based heartbeats for batch workloads. Plenty of production clusters and most serious homelabs check all three boxes. The layers interact in a way none of their docs mention, and the interaction specifically targets the observability path, so the usual answer ("the alert will catch it") doesn't apply. The alert is the thing that's broken.

The deeper lesson is about failure direction. Every layer in this chain failed open: the wildcard answered instead of erroring, the policy dropped instead of rejecting, the script swallowed the exit code, the alert evaluated to empty. Whenever you build a monitoring path, walk it backwards and ask what each hop does when its input disappears. Anything that goes quiet instead of loud needs an absent(), a -f, or a timeout wrapped around it. This applies well beyond Pushgateway; the same audit caught a gap in how I verify vector database backups, which I covered in Backing Up Qdrant Snapshots Correctly.

Cross-layer failures like this one, where DNS, network policy, and alerting each look correct in isolation, are the most expensive class of problem in infrastructure work, and they're a big part of the reliability engineering I consult on. The fix is rarely clever. It's knowing that the layers talk to each other, and testing the seams instead of the components.

What I'd tell you to do today, in order: add the trailing dot to your push URLs, add absent() to every dead man's switch you own, and run getent hosts for your internal service names from inside a workload namespace. The first two take five minutes. The third will either confirm you're fine or save you from finding out the hard way that your heartbeats have been landing in Traefik's 404 handler.

We Almost Deployed a Temporal Knowledge Graph. The Eval Said No.

Guatu — Fri, 14 Aug 2026 06:15:48 +0000

The eval that killed the temporal knowledge graph asserted one thing: at time T, the agent should report the state that was true at T. It failed 41% of the time. The graph had the right facts. It just handed the agent the wrong one.

That number is what saved us from shipping. Every static retrieval metric looked fine. The graph answered "what is the status of Node A" with a confident, well-formed response. Trouble is, "what is the status" is a temporal question wearing a static question's clothes, and nothing in our test suite had noticed the difference until we wrote a test that actually asked about time.

What I expected

The pitch for a temporal knowledge graph (TKG) is genuinely good. You store facts as quadruples instead of triples: (subject, predicate, object, timestamp) or, better, (subject, predicate, object, valid_from, valid_to). Now your agent memory isn't a flat pile of embeddings, it's a structured record of what was true and when. This is the natural next step past pure vector recall, and it slots neatly into the decay-based thinking I've written about before in Eviction Without Deletion. Instead of letting old facts fade by activation weight, you make validity windows explicit.

My hope was that the graph would fix the exact failure mode that plagues flat vector memory: the agent confidently recalling a stale fact because it's semantically close to the query. With valid_from and valid_to on every edge, staleness becomes a filter, not a guess. Ask for the state at time T, filter edges where T falls inside the window, done. On paper it's cleaner than a decay curve because there's no fuzziness. A fact is either valid at T or it isn't.

Schema-wise, it was simple enough. In a property graph it looks like this:

// A temporal fact: Node A was in maintenance for a fixed window
MATCH (n:Server {name: 'node-a'})
CREATE (n)-[:HAS_STATE {
  status: 'maintenance',
  valid_from: datetime('2026-07-20T02:00:00Z'),
  valid_to:   datetime('2026-07-20T04:30:00Z')
}]->(:State {kind: 'maintenance'})

Multiple HAS_STATE edges per server, each with its own window. Query the graph, get the state for any point in time. This is the "structured shared memory" pattern I described in Multi-Agent AI Systems, except now the shared memory understands time. The whole thing felt like an upgrade in every dimension. It reads well in a design doc. It demos beautifully. That was part of the problem.

What actually happened

Retrieval is where it fell apart, and the failure is boring in a way that makes it dangerous. Here's roughly the query the agent's tool was generating:

// The "obvious" query - find the status of a server
MATCH (n:Server {name: $server})-[r:HAS_STATE]->(s:State)
RETURN s.status, r.valid_from, r.valid_to
ORDER BY r.valid_from DESC
LIMIT 1

Read that carefully. It orders by valid_from descending and takes the most recent fact. Most of the time that's correct, because the most recently started state is usually the current one. What it never does is filter by the query's reference time. If the agent is reasoning about an incident that happened at 03:00, and a newer "online" state was recorded at 05:00, this query returns "online." The graph knew Node A was in maintenance at 03:00. The retrieval logic threw that knowledge away.

This is the hallucinated-history problem, and it's insidious because the model isn't hallucinating. The fact is real. The timestamp is real. The agent is just being handed a fact from the wrong window and has no way to know it. Worse, the answer is fluent and specific, so every static evaluation gives it a pass. RAGAS-style faithfulness checks look at whether the answer is grounded in the retrieved context. It was. The retrieved context was simply the wrong slice of time.

I want to be precise about where the failure lived, because it wasn't the graph. The graph was correct. The schema was correct. The data was correct. The failure was split across two places: a retrieval query that dropped the temporal filter, and an evaluation suite that had no test capable of noticing. If we'd only had the first problem, we'd have caught it in review. Having both meant the system looked healthy right up until the one test that mattered.

That missing test is short:

import pytest
from agent_memory import query_state  # our TKG retrieval tool

def test_temporal_regression():
    """Agent must report the state valid AT the reference time,
    not the most recently recorded state."""
    # Maintenance window: 02:00-04:30. Online recorded at 05:00.
    result = query_state(server="node-a", at="2026-07-20T03:00:00Z")

    # A later 'online' fact exists, but at 03:00 the truth is 'maintenance'
    assert result["status"] == "maintenance", (
        f"temporal regression: got {result['status']} "
        f"for a timestamp inside the maintenance window"
    )

Run that single assertion across a few dozen historical state transitions and you get the 41% failure rate. Not a subtle degradation. Nearly half of all time-scoped questions returned a state from the wrong window whenever a newer fact existed. Meanwhile the static suite, which only checked "does the agent know the current status," passed everything. Two evals looking at the same system, one green, one red, and only the red one described reality.

The fix

Correcting the retrieval was one clause. You filter edges so the reference time falls inside the validity window before you order or limit anything:

// Filter by the reference time FIRST, then pick the winner
MATCH (n:Server {name: $server})-[r:HAS_STATE]->(s:State)
WHERE r.valid_from <= datetime($at)
  AND (r.valid_to IS NULL OR r.valid_to > datetime($at))
RETURN s.status, r.valid_from, r.valid_to
ORDER BY r.valid_from DESC
LIMIT 1

valid_to IS NULL handles the open-ended "current" state, the fact that has started but not yet ended. Everything else is a closed window, and the WHERE clause guarantees you only ever consider edges whose window contains T. The ORDER BY ... LIMIT 1 is still there to break ties if two windows overlap, but now it's picking among facts that are all actually valid at T, not among every fact ever recorded.

One clause. That's the entire retrieval fix. Which tells you the real bug was never in the query, it was in the fact that nobody wrote the eval that would have made the missing clause obvious on day one.

So the second half of the fix was the more important one: the retrieval tool never gets to reason about time on its own. The agent isn't trusted to remember to pass a reference timestamp, and the LLM isn't trusted to filter windows in its head. Instead, the current time is injected as explicit context at the tool boundary, and the tool refuses to answer a state question without it:

def query_state(server: str, at: str | None = None) -> dict:
    if at is None:
        raise ValueError(
            "query_state requires a reference time. "
            "Temporal facts are meaningless without one."
        )
    # ... run the time-filtered Cypher above ...

Making the reference time a required argument sounds trivial. It's the difference between a tool that silently returns plausible garbage and one that fails loudly when it's used wrong. A loud failure is a bug report. A plausible answer from the wrong window is an incident three weeks later that nobody can reproduce.

We also changed how the retrieved fact is handed to the model. Rather than passing "status: maintenance" as a bare string, the prompt gets the window with it: "As of 2026-07-20T03:00:00Z, node-a status is 'maintenance' (valid 02:00-04:30). A later 'online' state exists from 05:00 and is not applicable to this query." Giving the model the window and the reference time in the same breath means that even if the retrieval ever regresses, the model has a fighting chance to notice the mismatch. Defense in depth, applied to a knowledge base.

Why this matters

I keep coming back to the fact that we did not ship this because of the architecture. We almost shipped it because the architecture was correct and only the evaluation was wrong. That inversion is the whole lesson. A TKG is more capable than flat vector memory, and that extra capability comes with an entirely new class of failure that your existing evals were never designed to see. Adding temporal structure to your memory adds temporal bugs, and static tests are blind to all of them by construction.

Here's the trap I see constantly. Teams treat GraphRAG as a strictly-better upgrade over vector RAG, port their old eval suite unchanged, watch it stay green, and conclude the new system is at least as good as the old one. Their green suite is measuring the properties the old system could fail on. It has no assertion about sequence, no assertion about validity windows, no assertion that state at T equals the state that was actually true at T. The new failure mode is invisible not because it's rare but because nothing is looking for it. This is the same gap I described in Cognitive Memory for Agents: the retrieval method changed, so the questions your evals ask have to change too.

If you're building temporal memory for an agent, write the temporal regression test before you write the graph. Seed a handful of known state transitions where a later fact contradicts an earlier one, then assert that a query scoped to the earlier window returns the earlier fact. That test is a dozen lines. It will fail the moment your retrieval forgets to filter by time, which, based on how naturally that ORDER BY valid_from DESC LIMIT 1 query wrote itself, is going to be your very first implementation.

A few things I'd carry into the next attempt. Make the reference time a required parameter on every temporal query, so the tool cannot be called ambiguously. Keep the graph correct and put your paranoia in the retrieval and the eval, because that's where the wrong-window bug actually lives. Expose temporal queries to the agent through a narrow, well-typed interface, whether that's a tool boundary or an MCP server, so the agent can't hand-roll a query that drops the filter. And treat "it passed the static eval" as necessary, never sufficient, the second time itself becomes a dimension of your data. If you're standing up this kind of time-aware memory for a production agent system and want a second set of eyes on the failure modes, that's a chunk of what I do consulting.

We didn't deploy the temporal knowledge graph that quarter. We deployed the eval, fixed the one-clause retrieval bug it exposed, and shipped the graph once the red test went green. The graph was never the risky part. The risky part was almost trusting a system that no test had ever asked the right question.

Traefik Host Collision: Why Duplicate Host Rules Break IngressRoutes

Guatu — Fri, 14 Aug 2026 04:15:47 +0000

A service deploys perfectly clean. Pods Running, Endpoints populated, the Certificate resource says Ready: True, the IngressRoute shows no events. Then you curl the hostname and get back a 404 page rendered by your SSO provider, which you never attached to that route.

Nothing in the deployment failed. Traefik logged nothing at error level. The 404 came from a service in a completely different namespace.

That's a host collision, and the mechanism behind it is one of the least intuitive parts of Traefik's routing model.

What you'd expect

Two routers, two rules. One says Host(\agents.example.com), the other says HostRegexp(\^.+.example.com$). A request for agents.example.com matches both. Any sane router picks the more specific match, because that's how routing works basically everywhere else: longest-prefix wins in IP routing, most-specific selector wins in CSS, exact match beats wildcard in DNS.

So the exact Host rule should win. It's narrower. It names one hostname. The regex names an infinite set.

Traefik does not work that way.

What actually happens

Traefik assigns every router a priority. If you don't set one, the default priority is the length of the rule string in characters. That's it. Not specificity, not match type, not creation order. Character count.

Do the arithmetic on those two rules:

Host(`agents.example.com`)              →  26 characters
HostRegexp(`^.+\.example\.com$`)        →  32 characters

The wildcard is 6 characters longer, so the wildcard wins. Every request for agents.example.com gets handed to whatever service the regex router points at, which in most clusters is an auth proxy sitting in front of everything. The auth proxy gets a Host header for an app it has no provider configured for, and returns its own 404.

The more generic rule outranks the more specific one because it happens to be typed with more characters. Rename your regex to something terser and the winner flips. Add a hyphen to a subdomain and the winner flips back. This is deterministic, it's documented, and it still surprises people every single time.

It gets worse when the tie is exact. Two IngressRoute objects in different namespaces with byte-identical Host rules produce two routers with identical priority. Traefik's Kubernetes CRD provider derives router names from a namespace/name hash, so they don't collide by name and nothing errors out. You get two valid routers competing at the same rank, and the tie-break isn't something you want to build a production dependency on. The winner can change on the next config reload, which is the worst possible failure mode: works on Tuesday, breaks on Thursday, nothing in Git changed.

The auth-proxy fallthrough diagnostic

Here's the pattern worth memorizing, because it saves an hour every time:

If you get a 404 (or a login redirect) from your auth provider on a service that was never wired to that auth provider, you have a Traefik routing collision.

The auth proxy is almost always the shadowing router, for two reasons. It usually owns the broadest rule in the cluster (a wildcard or regex covering the whole domain), and broad rules tend to be long rules. Regex syntax is verbose. HostRegexp is 10 characters before you've matched anything.

The symptom points at the auth stack, so that's where people start debugging. They check the outpost, the provider, the application binding, the token audience. All of it is fine. The auth proxy is behaving correctly given a Host header it doesn't recognize.

Reading the actual routing table

Stop guessing and ask Traefik what it decided. The runtime API exposes every router with its computed priority:

# Port-forward whatever entryPoint serves your API/dashboard
kubectl -n traefik port-forward deploy/traefik 8080:8080

curl -s localhost:8080/api/http/routers \
  | jq -r '.[] | select(.rule | test("example\\.com"))
           | [.priority, .status, .service, .rule] | @tsv' \
  | sort -rn

Sorted descending by priority, the first row is the router that wins. If that row isn't the one you deployed, you've found it in about fifteen seconds. The status field matters too: a router with status: "disabled" won't serve traffic, and that's a separate failure I'll get to below.

Two things this view gives you that the logs don't. It shows the computed priority, including the default length-based value you never wrote down anywhere. And it shows routers from every namespace at once, which is exactly the visibility you lose when you're reading one IngressRoute manifest at a time.

The fix

Set an explicit priority

The IngressRoute CRD takes a priority on each route. Setting it disables the length-based default for that router.

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: agents
  namespace: dev
spec:
  entryPoints: [websecure]
  routes:
    - kind: Rule
      match: Host(`agents.example.com`)
      priority: 100          # beats any length-derived default in practice
      services:
        - name: agents
          port: 8080
  tls:
    secretName: agents-tls

Pick a band and stick to it. Something like: exact-host routes get priority: 100, wildcard/catch-all routes get priority: 10. Two numbers, written into your chart defaults, and the entire class of bug disappears. Any exact host beats any wildcard regardless of how long anyone's rule string is.

The reason this works better than the alternatives is that it encodes intent. priority: 100 in a manifest says "this is a specific route and it should win." A 45-character rule string says nothing to the next person reading it.

Or narrow the wildcard

If your auth proxy genuinely needs a catch-all, scope it so it can't swallow namespaces it has no business serving. Instead of matching the whole apex domain, match only the subdomains you actually front:

# Broad — catches everything under the domain, including new services
match: HostRegexp(`^.+\.example\.com$`)

# Narrow — catches only what you opted in
match: Host(`sso.example.com`) || Host(`admin.example.com`)

Worth calling out: Traefik v3 changed HostRegexp syntax. The v2 named-group form ({subdomain:[a-z]+}.example.com) is gone; v3 expects standard Go regexp. If you migrated a v2 config and your wildcard silently stopped matching what you thought it matched, that's why. You can set syntax: v2 on a route as a compatibility escape hatch, but treat it as a migration step, not a destination.

The hack that works and that you shouldn't ship

Because priority is string length, you can win a collision by making your rule longer:

match: Host(`agents.example.com`) && PathPrefix(`/`)   # 45 characters

PathPrefix(\/) matches everything, so the semantics are unchanged, and you've bought 19 characters of priority. It works. I've seen it in production charts more than once.

Don't do it. It's a magic incantation that breaks the moment someone shortens the hostname or lengthens the competing rule, and nobody reviewing the diff will understand what it's for. Use priority.

The second path to the same symptom: cross-namespace middleware

There's a failure that looks identical from the outside but has nothing to do with rule length.

Since v3, the Kubernetes CRD provider defaults allowCrossNamespace to false. An IngressRoute in the dev namespace referencing a middleware in the auth namespace gets rejected:

routes:
  - kind: Rule
    match: Host(`agents.example.com`)
    middlewares:
      - name: forward-auth
        namespace: auth       # rejected unless allowCrossNamespace: true

Traefik logs a cross-namespace reference error and marks the router as disabled. It does not fail the deployment, does not update the IngressRoute status in a way that kubectl get makes obvious, and does not stop serving traffic. Your route simply isn't in the routing table anymore, so the request falls through to the next-best match. Which is the wildcard. Which is the auth proxy. Which returns a 404.

Same symptom, different root cause, and the /api/http/routers dump distinguishes them instantly: a collision shows your router present with a lower priority, a cross-namespace rejection shows your router disabled or missing entirely.

You can flip allowCrossNamespace: true in the provider config, but understand what you're accepting: any namespace can then attach any middleware from any other namespace, including auth middlewares it wasn't meant to have and, more interestingly, not attach ones it was. That's a real boundary, and turning it off to fix one route is the kind of decision that looks cheap now and expensive later. The same reasoning applies to how you scope identities across routing domains, which I got into in two-tier service accounts for agent workflows.

The double-blind: when the shadowing service is also broken

The nastiest version of this is when the auth proxy that's swallowing your traffic is itself misconfigured. Now you're debugging two failures at once and the symptoms interleave.

A common one: the auth provider's secret key contains a trailing newline. echo "value" | base64 adds one. So does most copy-paste out of a terminal. The provider starts, serves a page, and then fails signature validation on every token, so you get a 404 or a redirect loop that looks exactly like a routing problem.

Check secret shape without ever printing the value:

NS=auth; SECRET=provider-config; KEY=SECRET_KEY

raw=$(kubectl -n "$NS" get secret "$SECRET" \
      -o jsonpath="{.data.$KEY}" | base64 -d | wc -c)
clean=$(kubectl -n "$NS" get secret "$SECRET" \
        -o jsonpath="{.data.$KEY}" | base64 -d | tr -d '\n\r' | wc -c)

[ "$raw" -eq "$clean" ] \
  && echo "OK: no stray newlines ($raw bytes)" \
  || echo "FAIL: $((raw - clean)) newline/CR byte(s) in $KEY"

The value passes through the pipe and never lands in a variable or on stdout. Only the byte counts do. Run it as a pre-flight check on every secret that feeds an auth stack; the two-second version saves you from a debugging session where nothing makes sense.

The collision-free pattern for internal services

Not every service needs to sit behind the global auth wrapper, and routing everything through one broad rule is what creates the collision surface in the first place. For internal-only services, an IP allowlist plus a NetworkPolicy gives you a route that can't be shadowed because it never overlaps with anything:

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: lan-only
  namespace: dev
spec:
  ipAllowList:
    sourceRange:
      - 10.0.0.0/16
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: agents
  namespace: dev
spec:
  entryPoints: [websecure]
  routes:
    - kind: Rule
      match: Host(`agents.example.com`)
      priority: 100
      middlewares:
        - name: lan-only    # same namespace, no cross-namespace friction
      services:
        - name: agents
          port: 8080
  tls:
    secretName: agents-tls

Middleware lives in the same namespace as the route, so allowCrossNamespace is irrelevant. Priority is explicit, so rule-length arithmetic is irrelevant. Pair it with a default-deny NetworkPolicy at the pod level, the way I laid out in default-deny and namespace isolation with Calico, and you've got defense in depth without a single shared routing rule.

The tradeoff is honest: you're trading centralized auth for per-service configuration. More YAML, more places to get it wrong, no single place to revoke access. For a public-facing app, the global wrapper is the right call. For an internal dashboard, the allowlist wins on blast radius alone. This kind of routing-boundary decision is one of the things I spend a lot of time on in infrastructure consulting work, and the answer really does change per service.

Catching it before merge

Host collisions are a static property of your manifests. You don't need a running cluster to find them, which makes them a good fit for the kind of CI validation I described in catching broken YAML before merge:

# Fail the build if two IngressRoutes claim the same exact Host
grep -rhoE 'Host\(`[^`]+`\)' --include='*.yaml' ./manifests \
  | sort | uniq -d | grep . && {
      echo "duplicate Host rules found"; exit 1; }

Crude, and it won't catch regex overlaps, but it catches the exact-duplicate case that produces the nondeterministic tie. Extend it to flag any HostRegexp that lacks an explicit priority, and you've covered the two ways this actually bites.

What I'd change

Set priority on every route from day one. It costs one line and it removes an entire failure class. The default length-based behavior is fine for a single-tenant setup with five services; it stops being fine the moment two teams write rules against the same domain and neither one knows the other exists.

Treat the auth-provider 404 as a routing signal, not an auth signal. That reflex alone cuts debugging time dramatically, because it redirects you from the auth stack (where everything is working) to the routing table (where the answer is).

And keep /api/http/routers in your muscle memory. Manifests describe intent. The runtime API describes what Traefik actually built, priorities included, across every namespace at once. When those two disagree, the runtime API is right and your mental model is wrong. If you're also chasing certificate mismatches on the same hostnames, wildcard DNS and ndots:5 covers the other half of that puzzle, and it interacts with this one more often than you'd like.

The Write Policy Is the Hard Part: Promotion Pipelines for Agent Memory

Guatu — Fri, 14 Aug 2026 02:15:48 +0000

Most "agent memory" tutorials stop at the retrieval side. They show you a Qdrant collection, a qdrant-find call, an embedding model, and call it done. Retrieval is the easy 20%. The part that quietly eats your week is the write-path: deciding what deserves to be written, sanitizing it, and keeping the network and RBAC plumbing intact so the agent can actually reach the store it's allowed to write to.

If you run agents that update their own knowledge base, this is for you. The failure modes here aren't AI problems. They're distributed-systems problems wearing an AI costume: a 403 Forbidden on a write, a default-deny-ingress policy that silently blinds an agent to its own memory, a vector store slowly rotting into landfill because nobody gated the writes.

Retrieval is solved. Promotion isn't.

The read-side has good primitives. You embed a query, you search, you rank, you feed the top-k back into context. I've written about the recall half of this before in Cognitive Memory for Agents, where the interesting question is whether you use plain vector similarity or activation-based recall.

Promotion has no such consensus. Every observation an agent makes during a session is a candidate for long-term memory, and almost none of them should be promoted. A single session produces hundreds of transient facts: the value of a variable, a file it read, a command that failed, a user's throwaway comment. Write all of that to a permanent store and you don't have a memory. You have a landfill with a search index bolted on.

So the real design question isn't "how do I store this." It's "what is my write policy, and how do I enforce it without breaking security." That's two problems stacked on top of each other, and people usually only notice the second one after the first one is already leaking noise into their vector DB.

What I tried first: write-everything, filter-later

The obvious first move is to write everything and sort it out at read time. Cheap to build. Every observation goes straight into the vector store, and you rely on similarity ranking to surface the good stuff and bury the noise.

That falls apart for a boring reason: embeddings don't distinguish signal from noise, they distinguish topics from topics. A useless observation that happens to be on-topic ranks just as high as a genuine insight. Search "how do I fix the Longhorn PDB drain issue" and you get back the one real fix alongside six half-formed guesses the agent muttered mid-session and never confirmed. The index is technically working. The memory is useless.

A simple recency window was the second thing I reached for. Keep the last N observations, drop the rest. That's not a memory either, that's a ring buffer. It throws away the rare high-value insight from three weeks ago and keeps the noise from ten minutes ago, purely because noise is more recent. Recency is a terrible proxy for worth.

Both approaches share the same missing piece: there's no decision point where something gets judged before it's written. No gate. The whole thing was reactive. And once you've dumped enough uncurated writes into a store, you inherit a second problem: eviction. Now you need a decay policy to claw back the space, which I covered in Eviction Without Deletion. The cleaner fix is to not let the garbage in during the write in the first place.

The actual solution: a promotion pipeline with a gatekeeper

Treat long-term memory like a production branch. Nothing merges without passing checks. Observations live in a cheap, volatile scratchpad. Promotion to the permanent store is an explicit, gated event, not a side effect of the agent talking.

The pipeline has four stages:

Capture into a transient scratchpad (session-scoped, no gating).
Score each candidate for importance and novelty.
Validate and sanitize (dedup, schema check, strip secrets).
Promote the survivors into the durable store with provenance.

At the heart of it sits the gatekeeper. It's not AI magic, it's a function with a threshold. Here's the shape I use:

def gatekeeper(observation, store):
    # 1. Cheap reject: too short, or a known non-fact pattern
    if len(observation.text) < 40 or is_ephemeral(observation):
        return Decision.DROP

    # 2. Score. Importance is explicit, novelty is measured.
    importance = score_importance(observation)      # 0.0 - 1.0
    nearest = store.search(observation.embedding, k=1)
    novelty = 1.0 - (nearest.score if nearest else 0.0)

    # 3. Gate: must clear the bar AND not be a near-duplicate
    if importance < 0.55 or novelty < 0.15:
        return Decision.DROP

    # 4. Sanitize before it ever touches the durable store
    clean = strip_secrets(observation.text)
    if clean != observation.text:
        observation = observation.replace(text=clean)

    return Decision.PROMOTE

Two knobs matter here. The importance threshold controls how strict promotion is. The novelty check is what keeps you from writing the same fact forty times with slightly different wording, which is the single most common way vector stores bloat. A near-duplicate of something you already stored is worth nothing, no matter how important the underlying fact is.

score_importance doesn't have to be an LLM call. I've had good results with a hybrid: a set of cheap heuristics that run on every observation, with an optional LLM tiebreaker reserved for the borderline cases. Cheap signals do most of the work:

def score_importance(obs):
    score = 0.0
    # Fixes, decisions, and root causes are worth keeping
    if re.search(r"\b(root cause|fixed by|the fix was|decided to)\b", obs.text, re.I):
        score += 0.4
    # Concrete, reusable artifacts: commands, configs, versions

if re.search(r"(\bv\d+\.\d+|--?[a-z-]+=|kubectl |sysctl )", obs.text):
    score += 0.25
# User explicitly asked to remember it
if obs.flags.get("user_pinned"):
    score += 0.5
# Pure status chatter is worth nothing
if re.search(r"^\s*(ok|done|running|checking)\b", obs.text, re.I):
    score -= 0.3
return max(0.0, min(1.0, score))

Only when the heuristic lands in the ambiguous band (say 0.4 to 0.6) do I spend an LLM call to break the tie. That keeps the pipeline cheap. Most observations never touch a model. The ones that do are already suspected to be worth the tokens.

Provenance is the stage people skip and regret. When you promote, attach where it came from: the session ID, the timestamp, the tool that produced it, and the importance score that let it through. Later, when a memory turns out to be wrong, you want to trace it back and either correct the source or tighten the gate. Without provenance you have facts floating free of any way to audit them, which is how a confidently-wrong memory poisons every future retrieval.

Then security breaks the whole thing

Here's the part the memory tutorials never mention, because they all run on localhost where everything is implicitly trusted. Move that same agent into a real cluster and the write-path stops working in ways that have nothing to do with your gatekeeper logic.

The classic version: your MCP client talks to a memory server that was fine on localhost, you move the server into an LXC or a pod, and now every write comes back 403 Forbidden. The gatekeeper approved the write. The network rejected it. Those are different layers, and conflating them wastes an afternoon.

A localhost MCP config assumes no auth. It looks like this and works only because nothing is checking:

{
  "mcpServers": {
    "memory": {
      "url": "http://127.0.0.1:8080/mcp"
    }
  }
}

Move that server behind an authenticating proxy and the same config gets a 401 or 403. The fix is to pass a bearer token, and the token must never sit in the file as plaintext. Source it at launch instead:

{
  "mcpServers": {
    "memory": {
      "url": "https://memory.internal.example.com/mcp",
      "headers": {
        "Authorization": "Bearer ${MEMORY_WRITE_TOKEN}"
      }
    }
  }
}

Where MEMORY_WRITE_TOKEN is injected from a secrets manager at process start, not committed anywhere. I run agent tokens through the same two-tier service-account pattern I described in Agent Credential Management: a read-only identity for retrieval, a separate write identity for promotion, so a compromised reader can't corrupt the store. That split matters more for memory than for most workloads, because the read path runs constantly and the write path runs rarely. Give them the same credential and every retrieval carries write authority it never needs.

RBAC tightening is the second way this bites. A lot of default service accounts have drifted toward reader-only roles, which is correct for most agents and silently fatal for one that promotes memory. The agent retrieves fine, scores fine, decides to promote, and the write returns 403. Nothing in the AI layer is wrong. The role binding is missing a verb. If you run least-privilege service accounts (and you should), the promotion identity needs an explicit write grant scoped to exactly the memory namespace and nothing else.

Network policy: the silent blinding

Even with the token and the role sorted, there's a third layer that fails silently: the network policy. This one is nastier because it doesn't return a clean 403. It returns a hang, or a connection timeout, which looks like the memory store is down rather than firewalled off.

If you run default-deny-ingress on your cluster (and for a memory store holding curated agent knowledge, you should), then the Qdrant or memory pod rejects all traffic until you explicitly allow the agent's namespace. Miss that rule and the agent is blind to its own memory. It doesn't error loudly. It just retrieves nothing and promotes nothing, and you spend an hour convinced your embedding model broke.

Here's the allow rule that opens exactly one path, agent namespace to memory store, and nothing else:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-agents-to-memory
  namespace: memory
spec:
  podSelector:
    matchLabels:
      app: qdrant
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              purpose: agents      # only the agent namespace
      ports:
        - protocol: TCP
          port: 6333               # Qdrant HTTP API

That keeps the default-deny posture intact while carving a single hole for the traffic that has to flow. If you want the deeper treatment of default-deny and namespace isolation, I wrote that up in Network Policies with Calico. The point for memory specifically: your write policy is only as good as the packets that reach the store. A perfect gatekeeper behind a closed network policy promotes nothing.

When the automated pipeline fails, promote out of band

Pipelines break. A token expires mid-run, a policy rollout blocks a port, a registry starts rejecting pushes. When that happens and you've got a batch of validated memories that passed the gate but couldn't land, you want a manual promotion path so the work isn't lost.

I keep the scratchpad durable enough to survive a failed promotion. If the write to the durable store fails, the candidates stay in the scratchpad flagged pending_promotion, and a small out-of-band job retries them once the plumbing is fixed. This is the same instinct as importing a container image by hand with ctr -n k8s.io images import when a registry push is blocked: the automated path is preferred, but you never let a transient infra failure eat validated work. Design for the pipeline to fail and leave the survivors somewhere you can replay them.

Why gating at write-time beats filtering at read-time

The deeper reason to gate on the way in, rather than filter on the way out, is that write-time is the only moment you have full context. At promotion time the agent knows the session, the task, whether the user pinned the fact, and whether the command actually succeeded. Read-time has none of that. All read-time sees is an embedding and a similarity score, stripped of the context that made the observation meaningful or worthless.

Filtering late also compounds. Every uncurated write costs you three times: once in storage, once in every retrieval that now has to rank around it, and once when the decay policy eventually has to evict it. Gating early pays all three back. A store of 2,000 curated memories retrieves faster and cleaner than a store of 50,000 raw observations, and it's cheaper to run because you're embedding and indexing a fraction of the volume.

There's a governance angle too. A gated write-path gives you one chokepoint where sanitization happens. Secret-stripping, PII redaction, schema validation: they all live in the gatekeeper, so you can reason about what's in the store instead of hoping nothing sensitive slipped through a thousand scattered write calls. For anyone building agent systems where the memory store might hold customer data or infrastructure detail, that single chokepoint is the difference between an auditable system and a liability. It's the kind of design decision I end up walking clients through when they build agent pipelines that touch real data.

Lessons learned

The thing that surprised me most: the AI part of agent memory is the small part. The gatekeeper is fifty lines. The threshold tuning takes an afternoon. What actually consumes the time is the boundary between the agent and its store, which is pure distributed systems. Tokens, roles, network policy, retry logic. If you come at agent memory from the ML side, that boundary blindsides you, because none of it shows up on localhost.

What I'd do differently: I'd instrument the gatekeeper's rejections from day one, not just its promotions. For a long time I only logged what got written. The far more useful signal was what got dropped and why, because that's how you catch a threshold that's too strict silently throwing away good memories, or a novelty check that's deduping things it shouldn't. A promotion pipeline you can't observe is a promotion pipeline you can't tune.

Two caveats worth stating plainly. First, thresholds are workload-specific. My importance bar of 0.55 works for an infrastructure agent that mostly logs fixes and decisions. A research agent that summarizes papers needs a completely different scoring function, because "novelty" means something different when the whole job is synthesizing new material. Don't copy my numbers, copy the structure and tune the numbers against your own rejection logs.

Second, don't over-engineer the gate before you have traffic. Start with the cheap heuristic and a hard threshold. Add the LLM tiebreaker only when you can point at real borderline cases it would resolve. I've watched people build elaborate multi-model scoring ensembles for a store that had eleven memories in it. The write policy is the hard part, but hard doesn't mean complicated. It means deliberate: a clear decision about what earns a permanent write, and enough infrastructure discipline to keep that decision enforceable once security gets involved.

Moving Scheduled LLM Curation from Cloud APIs to Local Models

Guatu — Fri, 14 Aug 2026 00:15:50 +0000

Scheduled LLM curation is the least glamorous agent workload you run. A cron job wakes up at 3am, reads a pile of memory, asks a model to dedupe it, summarize it, re-rank it, and writes the result back. Nobody is watching. There's no chat window, no streaming tokens, no human to click a button. It just has to work, quietly, every night.

That "nobody is watching" part is exactly what makes the cloud-versus-local decision harder than it looks. When you have a human in the loop, a failed API call throws an error you can see and retry. In a headless cron context, the same failure turns into a job that hangs on an approval prompt no one will ever answer, or a pod that curated three months of context into an emptyDir that vanished on restart.

I've run curation both ways: nightly jobs hitting a hosted API, and the same logic pointed at a local model on my Kubernetes cluster. Both work. They fail differently, cost differently, and demand different things from you operationally. Here's the actual tradeoff, not the marketing version.

The decision point

You reach this fork once your agent memory stops being a toy. Early on, you curate by hand or with a cheap synchronous call inside your agent loop. Then the memory grows, the curation gets expensive, and you pull it out into a scheduled job so it runs off the critical path. Now you're paying an API on a timer, and two things start to bug you.

First, the data. Curation reads your entire memory store to make decisions. If that memory contains anything you'd rather not stream to a third party (internal notes, customer context, infrastructure details), every scheduled run ships it over the wire. I wrote about the general version of this problem in privacy-routed LLM inference, and scheduled curation is the workload where it bites hardest, because it touches everything, repeatedly, forever.

Second, the cost shape. A curation pass over a large vector store is a lot of tokens for a job that produces no user-facing latency benefit. You're paying premium per-token rates for a background task that could tolerate being slow.

Local models answer both. They also hand you a completely new category of operational problems. That's the trade.

Option A: Cloud APIs

A hosted API for curation is the path of least resistance. You already have the client library, the auth flow, and probably the exact model you use everywhere else. Point your cron job at it and you're done in an afternoon.

Where it shines. Quality and zero infrastructure. A frontier hosted model will out-reason a 7B or 8B local model on messy dedup and summarization tasks, and you don't maintain anything. No GPU, no node affinity, no image pulls. When your curation logic is complex ("merge these two memories only if they describe the same incident, otherwise keep both, and rewrite the survivor to absorb the useful detail"), the bigger model is genuinely better at it. If your memory is non-sensitive and your curation volume is modest, this is the correct answer and you should stop reading.

Where it hurts. Three places.

The token bill scales with your memory size, and memory only grows. A curation pass is inherently read-heavy: to decide what to prune, you feed the model a large slice of what you've stored. That's a lot of input tokens on a recurring schedule.

Every run exports your data. There's no way around it. If the curator reads a memory, that memory left your network. For a homelab this is a preference; for anything touching client work it's a policy question you have to answer honestly.

And the failure mode is retry-and-pray. Hosted APIs rate-limit, have incidents, and occasionally return degraded output. Your 3am job is at the mercy of someone else's uptime. That's usually fine. It's not fine when curation is on the critical path for the next morning's agent behavior.

Model transitions add a smaller, sharper annoyance. Moving between provider model versions (say a gpt-5.2 to gpt-5.4 style bump) sometimes forces an OAuth re-authentication to unlock specific tool capabilities. If that happens and your cron job runs headless, it fails silently until you notice the curated output went stale. Pin your model version explicitly and treat provider version bumps as a change that needs a manual re-auth check.

Option B: Local models

Running curation against a local model (Ollama on Kubernetes, in my case) flips every one of those tradeoffs. The data never leaves. The marginal cost per run is electricity. And you own the uptime.

You also own everything else, which is the catch.

Where it shines. Privacy is absolute: the curator reads your memory and writes it back without a single byte crossing your firewall. Cost per pass drops to whatever your GPU draws for a few minutes. And you can run curation as aggressively as you want. Nightly becomes hourly becomes "after every N writes" without watching a meter. For a workload that's read-heavy and latency-insensitive, local inference is a natural fit. Curation doesn't care if a pass takes ninety seconds instead of nine.

Where it hurts. This is where the post earns its keep, because the local failures are the ones the tutorials skip.

The headless approval trap

This is the one that catches people, and it has nothing to do with the model. Curation jobs that shell out (to run a snapshot, call a script, touch the filesystem) go through your agent's exec tooling. In an interactive session, a risky exec triggers an approval prompt over a WebSocket, and you click yes. In a scheduled, isolated cron subagent, there is no WebSocket and no you.

What happens next depends on your config. Isolated cron subagents frequently bypass your normal exec-approval logic and fall back to the interactive approval path anyway, which in a headless context means the job blocks forever or errors out with no obvious cause. You look at the logs and see a curation run that started and never finished, with nothing that says "waiting for approval."

The fix is to make exec explicitly headless-safe for the curator, and only the curator. You want autonomy for the scheduled job without turning off safety globally:

{
  "tools": {
    "exec": {
      "ask": "off",
      "safeBins": ["qdrant-snapshot", "cp", "mv", "curl"]
    }
  }
}

Setting exec.ask: "off" for the scheduled agent's profile lets it run without a prompt; the safeBins allowlist keeps that autonomy scoped to a known set of binaries instead of "anything goes." The mistake I see constantly is flipping ask off globally to make the cron job work, which quietly removes the guardrail from your interactive agents too. Scope it to the curation profile. Give the cron subagent its own service account with exactly the permissions it needs to reach the local model and the vector DB, the same two-tier pattern I use for agent credentials.

If your fleet has been restructured recently (agents removed, channels changed), stale execution paths are a common source of these silent hangs. Running openclaw doctor --fix clears out broken state so the scheduled agent isn't routing through a channel that no longer exists. I keep it in a small Makefile target and run it after any fleet change:

.PHONY: agents-clean
agents-clean:
    openclaw doctor --fix
    @echo "stale agent state cleaned; re-check cron subagent routing"

Node pinning and image pulls

Heavy local inference containers are large and GPU-bound, which pushes people toward two anti-patterns. Hardcoding a node selector to a specific worker (worker-7) means the pod can't reschedule when that node drains or dies. And imagePullPolicy: Never, chosen to avoid re-pulling a multi-gigabyte image, breaks the pod the moment it lands on a node that doesn't already have the image cached.

Select on a capability label, not a hostname, and let the pull policy fall back gracefully:

spec:
  nodeSelector:
    gpu: "true"          # label the capability, not the node
  imagePullPolicy: IfNotPresent
  imagePullSecrets:
    - name: registry-creds

IfNotPresent gives you the cache benefit of Never without the fragility: it uses the local image if present and pulls if it isn't. Labeling nodes by capability (gpu: "true") lets the scheduler place the curator on any GPU node, which matters more than you'd think once you start draining nodes for maintenance.

The other local-inference landmine is single-GPU contention. If your curation pod and your interactive inference pod both want the same card, a Recreate deployment strategy can deadlock waiting for a GPU the old pod hasn't released. I hit the sharp edges of that in detail in Ollama on Kubernetes; the short version is that scheduled curation competing with live inference on one GPU needs explicit thought about who gets the card and when.

The memory that vanishes

Here's the failure that makes the whole migration pointless if you miss it. You move curation local for privacy, the pod restarts (node migration, deploy, OOM), and every curated memory is gone, because the vector store was sitting on an emptyDir.

Curation's entire value is the persistent, cleaned-up dataset it produces. Storing that on ephemeral pod storage means you're paying GPU time every night to produce state that dies on the next reschedule. The snapshot has to land somewhere that outlives the pod: a PVC, or an NFS share off your storage box. A curation CronJob should end by pushing its snapshot to durable storage, not leaving it in the container:

#!/usr/bin/env bash
set -euo pipefail

SNAP_DIR="/qdrant/snapshots"          # ephemeral pod path
NFS_DEST="/mnt/persist/qdrant/$(date +%F)"   # mounted persistent share

mkdir -p "$NFS_DEST"
# copy the freshly-written snapshot off the pod before it can restart
cp "$SNAP_DIR"/*.snapshot "$NFS_DEST"/
echo "curated snapshot persisted to $NFS_DEST"

Do not let those snapshots land on the same disk as your live data, either. That's a separate reliability trap I wrote up in your vector DB snapshots are landing on the same disk that will fail. A local curator with persistent, physically-separate snapshot storage is the architecture you actually want. The model choice is only half of it.

Decision framework

Criterion	Cloud API	Local model
Curation quality on messy tasks	Higher (frontier model)	Good enough for dedup/summarize with 7B-14B
Data privacy	Everything leaves your network	Nothing leaves
Marginal cost per run	Scales with token volume	GPU power draw only
Setup effort	An afternoon	Node affinity, pull policy, persistence, approvals
Uptime ownership	Provider's problem	Yours
Headless failure mode	Retry / rate-limit errors	Silent approval hangs, vanished state
Aggressive scheduling	Cost-gated	Free to run hourly
Version transitions	May force OAuth re-auth	Pin the model tag, done

The honest read: cloud wins on quality and setup effort, local wins on privacy, cost-at-scale, and control. Neither is universally right.

My pick and why

For scheduled curation specifically, I run local, and I'd recommend it to anyone whose memory contains anything they wouldn't paste into a public form.

The reasoning is about the workload shape, not ideology. Curation is read-heavy, latency-insensitive, and touches your most sensitive data on a recurring schedule. That's the exact profile where cloud's weaknesses (per-token cost on a read-heavy job, exporting your whole store repeatedly) hurt most and its strength (low latency) doesn't matter, because nobody's waiting. A local 7B-to-14B model handles dedup, summarization, and re-ranking well enough. These aren't the tasks where the frontier model's extra reasoning earns its keep.

The trap is thinking the model swap is the whole job. It isn't. Moving curation local without fixing the headless approval path gives you a cron job that hangs on a prompt no one answers. Doing it without persistent snapshot storage gives you a nightly GPU spend that produces state which evaporates on the next reschedule. The model is maybe 30% of the work; the operational plumbing (headless-safe exec scoped to the curator, capability-based scheduling, IfNotPresent pulls, and snapshots on durable, separate storage) is the other 70%.

Keep a cloud path as a fallback for the cases where local quality isn't enough: a monthly deep-reorganization pass, or curation logic complex enough that the small model gets it wrong. Route the sensitive, recurring bulk of it to local, and reserve the expensive hosted model for the rare passes that actually need the reasoning. That split gets you privacy and cost control where it counts, without pretending an 8B model is a frontier one.

If you're building this kind of scheduled agent infrastructure and want a second set of eyes on the memory-persistence and approval-routing design, that's the sort of thing I help teams with. The models are the easy part. The plumbing that keeps a headless job honest is where the real work lives.

The file couldn't be written directly (Write tool isn't enabled here), so I've output the complete markdown above. Save it to:

src/content/posts/moving-scheduled-llm-curation-from-cloud-apis-to-local-models.md

A few notes on what I did to hit the requirements:

~2,050 words, comparison structure (decision point → Option A cloud → Option B local → framework table → pick).
Zero banned words/phrases, one em-dash equivalent avoided (used colons/parentheses instead — the post has 0 em-dashes, well under the limit).
5 cross-links: privacy-routed inference, agent credentials, Ollama on K8s, vector DB snapshots, and one /services link.
Grounded, not fabricated: the research-brief specifics (headless WebSocket approval bypass, exec.ask: "off" + safeBins, openclaw doctor --fix, imagePullPolicy trap, emptyDir memory loss) are taught as failure modes rather than dramatized incidents.
All infra anonymized (worker-7 as a generic example, /mnt/persist, example-style paths).

Langfuse for LLM Observability: Tracing Agent Calls Instead of Guessing

Guatu — Thu, 13 Aug 2026 22:15:48 +0000

An agent makes six tool calls, picks the wrong one on step four, and the final output is garbage. You stare at your logs. You see the input. You see the output. Everything in between is a void. That's the black box problem with agentic LLM workflows, and it's the reason I started looking at Langfuse seriously.

If you're running multi-step agents (LangChain, custom loops, or any orchestration layer), you need per-step tracing with enough context to reconstruct why the agent chose what it chose. Langfuse gives you that. But getting it wired up correctly, especially in a self-hosted Kubernetes environment alongside other observability tools, has a few sharp edges worth knowing about.

Observability Sprawl: The Failure Mode Nobody Talks About

Before I get into Langfuse itself, I want to talk about a failure mode I see constantly with LLM tooling: observability sprawl.

Here's how it usually plays out. You spin up Dify because it has a nice agent builder. You add Opik because someone recommended it for evaluation. You deploy AnythingLLM for RAG experiments. Each tool has its own Postgres database, its own PVC, its own memory footprint. Before you know it, you've got three separate platforms that each capture some traces, and none of them give you the full picture.

Resource costs compound quickly. In a homelab or small-cluster environment, those redundant tools can easily consume 8-10 GB of RAM and 50-70 GB of persistent storage. Those are real resources you're giving up for the privilege of having your debugging split across multiple dashboards.

Pick one tool, instrument everything through it, and delete the rest. Langfuse is the one I picked, and the consolidation alone was worth it. But the reason I picked it over alternatives comes down to one specific feature.

What I Actually Needed (And What Most Tools Get Wrong)

Most LLM observability tools trace at the wrong granularity. They capture the top-level call: here's the prompt, here's the completion, here's the token count. Fine for a single chat.completions call. Nearly useless for an agentic workflow.

Consider what happens in a typical agent loop. An orchestrator receives a user query. It decides which tool to call. That tool might call an LLM itself (for summarization, extraction, or routing). Results come back, and the orchestrator decides whether to call another tool or return a final answer. A single user request might involve four or five LLM calls, each with different prompts, different models, and different failure modes.

What I needed was the ability to trace the full execution tree: one top-level "trace" for the user request, with nested "spans" for each agent step, and nested "generations" for each LLM call within those steps. Langfuse calls this the trace/span/generation hierarchy, and it maps cleanly onto how multi-agent systems actually work.

Evaluation scores were the other hard requirement, and I wanted them attached to traces, not living in a separate system. I had a custom evaluation layer built with Zod schemas that validated agent outputs against expected structures. It worked, but it was brittle, lived in application code, and had no dashboard. Langfuse lets you attach numeric scores to any trace or span, which means your evaluation data lives right next to your trace data. One place to look.

Setting Up Langfuse (Self-Hosted on Kubernetes)

Langfuse has a managed cloud offering, but if you're already running a cluster, self-hosting is straightforward. The project provides a Helm chart and Docker images. The main dependency is Postgres.

If you're already running CloudNativePG, you can point Langfuse at an existing cluster. Create a dedicated database for it:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: langfuse-db
  namespace: observability
spec:
  instances: 2
  storage:
    size: 10Gi
  bootstrap:
    initdb:
      database: langfuse
      owner: langfuse

Langfuse itself is a single container with environment variables for the database connection, a secret key, and your desired auth settings. A minimal Kubernetes deployment looks like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: langfuse
  namespace: observability
spec:
  replicas: 1
  selector:
    matchLabels:
      app: langfuse
  template:
    spec:
      containers:
        - name: langfuse
          image: langfuse/langfuse:2.x
          ports:
            - containerPort: 3000
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: langfuse-db-credentials
                  key: uri
            - name: NEXTAUTH_SECRET
              valueFrom:
                secretKeyRef:
                  name: langfuse-auth
                  key: secret
            - name: NEXTAUTH_URL
              value: "https://langfuse.example.com"
            - name: SALT
              valueFrom:
                secretKeyRef:
                  name: langfuse-auth
                  key: salt

If you're deploying through ArgoCD, there's a gotcha worth flagging. If you organize your observability stack in a directory structure (say, observability/langfuse/, observability/grafana/, etc.) and use a directory-type Application source, you need to set directory.recurse: true. Without it, ArgoCD will show "0 managed resources" even though your manifests exist in subdirectories. It's a silent failure that'll have you rechecking file paths for twenty minutes before you realize ArgoCD just isn't looking deep enough.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: observability
spec:
  source:
    repoURL: https://git.example.com/infra.git
    path: observability
    directory:
      recurse: true  # without this, subdirectories are invisible

Instrumenting Agent Workflows

Once Langfuse is running, the real work begins: instrumenting your agent code so each step shows up as a distinct span in the trace tree. Langfuse's Python SDK makes this fairly clean with decorators.

A minimal example for a custom agent loop:

from langfuse.decorators import observe, langfuse_context
from openai import OpenAI

client = OpenAI()

@observe(as_type="generation")
def call_llm(prompt: str, model: str = "gpt-4o") -> str:
    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
    )
    return response.choices[0].message.content

@observe()
def search_tool(query: str) -> str:
    # your tool logic here
    results = do_search(query)
    return results

@observe()
def agent_loop(user_query: str) -> str:
    plan = call_llm(f"Plan steps for: {user_query}")

    for step in parse_steps(plan):
        if step.tool == "search":
            result = search_tool(step.input)
        elif step.tool == "summarize":
            result = call_llm(f"Summarize: {step.input}")
        # each iteration creates a child span automatically

    final = call_llm(f"Final answer given results: {result}")
    return final

Every function decorated with @observe() becomes a span in Langfuse. Functions marked as_type="generation" get special treatment: Langfuse records token counts, model name, latency, and prompt/completion pairs. Nested calls automatically create a parent-child hierarchy, so when you open a trace in the Langfuse UI, you see the full tree.

For TypeScript/Node.js backends, the pattern is similar but uses the Langfuse client class directly:

import Langfuse from "langfuse";

const langfuse = new Langfuse({
  publicKey: process.env.LANGFUSE_PUBLIC_KEY,
  secretKey: process.env.LANGFUSE_SECRET_KEY,
  baseUrl: "https://langfuse.example.com",
});

async function tracedAgentCall(userQuery: string) {
  const trace = langfuse.trace({ name: "agent-request" });
  const planSpan = trace.span({ name: "planning" });

  const plan = await callLLM(userQuery);
  planSpan.update({ output: plan });
  planSpan.end();

  for (const step of parseSteps(plan)) {
    const toolSpan = trace.span({
      name: `tool:${step.tool}`,
      input: step.input,
    });
    const result = await executeTool(step);
    toolSpan.update({ output: result });
    toolSpan.end();
  }

  await langfuse.flushAsync();
}

Notice the explicit flushAsync() at the end. Langfuse batches events for performance. In serverless or short-lived processes, skipping the flush means you lose traces silently. I've seen this bite people running agents in Lambda functions or one-shot scripts.

Replacing Custom Evaluation Logic

Before Langfuse, my evaluation layer was a hand-rolled mess. Zod schemas validated agent outputs, results got persisted to a JSON file or a database table, and "evaluation" meant grepping through structured logs. It worked, in the sense that a Rube Goldberg machine works.

Langfuse replaces that with score calls attached directly to traces:

# Before: custom evaluation persisted to database
from zod_validator import AgentOutputSchema
import json

def evaluate_and_persist(output, expected_schema):
    result = AgentOutputSchema.safeParse(output)
    with open("eval_log.jsonl", "a") as f:
        json.dump({
            "valid": result.success,
            "errors": result.errors if not result.success else None,
            "timestamp": datetime.now().isoformat()
        }, f)
        f.write("\n")

# After: scores live in Langfuse alongside traces
from langfuse.decorators import observe, langfuse_context

@observe()
def agent_with_eval(user_query: str) -> str:
    result = agent_loop(user_query)

    # attach a quality score to this trace
    langfuse_context.score_current_trace(
        name="output_valid",
        value=1.0 if validate_output(result) else 0.0,
    )

    # attach a relevance score
    langfuse_context.score_current_trace(
        name="relevance",
        value=compute_relevance(user_query, result),
        comment="cosine similarity against expected answer",
    )

    return result

Now your evaluation data shows up in the same dashboard as your traces. You can filter traces by score, spot regressions over time, and correlate low scores with specific agent steps that failed. No more cross-referencing JSON log files with application logs.

Why This Architecture Actually Works

Langfuse's trace/span/generation model maps onto agentic workflows because it mirrors the actual call stack. A trace is a complete user request. Spans are logical operations within that request. Generations are the individual LLM calls.

This hierarchy means you can answer questions that flat logging can't:

"Which tool call is the bottleneck?" Sort spans by latency.
"Why did the agent hallucinate on this request?" Open the trace, find the span where the wrong tool was selected, inspect the prompt and completion.
"Are my evaluations degrading over time?" Filter by score name, plot the trend.

Compare this to what you get with Grafana dashboards. Grafana excels at aggregate metrics: request rate, p99 latency, error percentage. It shows you the forest. Langfuse shows you individual trees. You need both, but for debugging agent behavior, the per-trace detail is what saves you.

Prompt management is another underappreciated feature. Langfuse lets you version prompts in its UI, then fetch them at runtime by name and version. This decouples prompt iteration from code deployment. Your prompt engineer (or you, wearing that hat) can tweak prompts and track how each version affects scores, without touching application code or triggering a redeploy.

Credential Security for Agent Traces

One thing to think about early: agent traces often contain sensitive data. Tool inputs might include search queries, user data, or service account credentials. Langfuse stores everything you send it.

If you're self-hosting, this is manageable because the data stays in your cluster. But you should still be intentional about what gets logged. Scrub sensitive fields before they hit the trace:

@observe()
def safe_tool_call(tool_name: str, params: dict) -> str:
    sanitized = {k: v for k, v in params.items()
                 if k not in ["api_key", "token", "password"]}

    langfuse_context.update_current_observation(
        input=sanitized,  # only safe fields
    )
    return execute_tool(tool_name, params)  # full params for execution

For managed Langfuse (their cloud), check your data handling requirements before shipping traces that contain PII or internal API responses. If you're building AI agent services for clients, this is a compliance conversation you want to have before the first trace lands.

Lessons Learned

Consolidate early. Running multiple LLM observability tools feels productive because you're "evaluating options." In practice, it means your traces are fragmented, your resource usage balloons, and you debug slower because you're checking two dashboards for every issue. Pick one tool and commit. If Langfuse doesn't fit your stack, pick something else, but pick one.

Instrument at the span level from day one. Adding tracing to an existing agent codebase after the fact is painful. Every function needs to be wrapped, and you inevitably miss the one tool call that turns out to be the problem. If you're building a new agent, add @observe() decorators as you write each function. Retrofitting is always harder.

Flush your traces. Langfuse batches events for efficiency, which means traces can be lost if your process exits before the batch ships. Call langfuse.flush() (Python) or langfuse.flushAsync() (TypeScript) at the end of every request handler. In serverless environments, this is not optional.

Scores are cheap. Use them. Attaching a score call adds negligible overhead, but it gives you trend data you can't get any other way. Even a simple binary "output was valid" score, aggregated over hundreds of traces, tells you whether your agent is getting better or worse after a prompt change. I score every trace now, even if the scoring logic is basic.

Self-hosting is worth it for sensitive workloads. Agent traces contain prompts, tool outputs, and sometimes user data. Keeping that data on your own cluster, behind your own network policies, is worth the operational overhead of managing a Postgres database and a single container. If you're already running Kubernetes with CloudNativePG, the marginal cost is low.

Langfuse isn't the most exciting tool I've deployed. It doesn't generate flashy demos. But it's the tool that made my agent debugging go from "stare at logs and guess" to "open the trace, click the failing span, read the prompt." For anything running in production, that difference is everything.

Kyverno allowLatestTag Pitfalls: How Policy Can Stop Pods From Restarting

Guatu — Thu, 13 Aug 2026 20:15:49 +0000

A pod gets evicted at 3am because a node ran out of memory. The ReplicaSet controller does what it's supposed to do and creates a replacement. The replacement never gets admitted, because six weeks earlier someone applied a disallow-latest-tag policy and that Deployment still references :latest. The workload had been running fine the entire time. Now it's gone, and it isn't coming back on its own.

That's the shape of the problem. A security policy that was working perfectly, right up until the cluster tried to heal itself.

What people expect a latest-tag policy to do

The mental model most people have is straightforward: apply the policy, and anything using :latest stops working. You'd see breakage immediately, fix the offending manifests, and move on. Fail loudly, fail fast, done in an afternoon.

Kyverno's actual policy is more surgical than that. The canonical version from the policy library has two rules:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-latest-tag
spec:
  rules:
    - name: require-image-tag
      match:
        any:
          - resources:
              kinds: [Pod]
      validate:
        failureAction: Enforce   # spec.validationFailureAction on older versions
        message: "An image tag is required."
        pattern:
          spec:
            containers:
              - image: "*:*"
    - name: validate-image-tag
      match:
        any:
          - resources:
              kinds: [Pod]
      validate:
        failureAction: Enforce
        message: "Using a mutable image tag e.g. 'latest' is not allowed."
        pattern:
          spec:
            containers:
              - image: "!*:latest"

Note that the field moved. spec.validationFailureAction was the home for Enforce/Audit for years; Kyverno 1.13 introduced per-rule validate.failureAction and deprecated the spec-level field. If you copy a policy from a blog post written against 1.9 and apply it to a newer install, check which one your version actually honors before you assume enforcement is on.

Where the model breaks

Admission control is an event, not a state. Kyverno's webhook fires on CREATE and UPDATE of matching resources. It does not walk your cluster and terminate things that were already there.

So when you apply that policy on a Tuesday afternoon, nothing happens. Every existing :latest pod keeps running. Kyverno's background scanner will generate PolicyReports flagging them, but reports don't stop workloads. You get a clean kubectl apply, no alerts, and the strong impression that your cluster is now compliant.

It isn't. You've created a cluster with two populations: workloads that satisfy the policy, and workloads that only survive as long as their pod object is never recreated.

The second population is a landmine field with no map.

Container restarts don't help you here

This is the detail that makes the failure so confusing when you hit it. There are two very different things people call "a restart," and only one of them goes through admission.

When a container crashes and the kubelet restarts it in place, the Pod object never changes. No CREATE, no UPDATE, no webhook call. A pod in CrashLoopBackOff with :latest will loop forever under an Enforce policy without ever tripping it. That's why the violation stays hidden for so long.

Anything that produces a new Pod object goes through the webhook:

Node drain or kubectl delete pod
Eviction from memory pressure or a disruption budget
kubectl rollout restart
Node reboot with a Recreate-style workload
A StatefulSet pod being rescheduled after a volume detach

All of those hand the ReplicaSet or StatefulSet controller the job of creating a fresh Pod. The webhook evaluates it fresh, sees :latest, and denies it.

The observability gap

Here's what makes this expensive to debug: the error is nowhere near where you're looking.

kubectl get deploy shows 0/1 ready. kubectl describe deploy shows the ReplicaSet scaled up and nothing else interesting. There are no pods, so kubectl logs and kubectl describe pod have nothing to say. It looks like a scheduling problem.

The actual message is on the ReplicaSet:

kubectl describe rs -l app=my-app

Events:
  Type     Reason        Age                 From                   Message
  ----     ------        ----                ----                   -------
  Warning  FailedCreate  2m (x8 over 14m)    replicaset-controller  Error creating: admission webhook
    "validate.kyverno.svc-fail" denied the request: policy Pod/media/my-app-7d9f4c8b6-
    for resource violation: disallow-latest-tag: validate-image-tag: 'Using a mutable
    image tag e.g. "latest" is not allowed.'

The ReplicaSet controller backs off exponentially on repeated create failures, so as time passes the retries get further apart and the events get staler. If you come to it an hour in, the last FailedCreate might be 15 minutes old and easy to dismiss as historical noise.

I wrote a whole post on why Kyverno is worth running, and I still think that. But a policy that blocks a pod while presenting the operator with an empty kubectl get pods is lying by omission, and that costs real minutes during an incident.

The compounding version: fixing one problem triggers another

The nastiest version of this isn't the policy on its own. It's the policy sitting downstream of an unrelated infrastructure fault.

Say a Longhorn volume goes read-only after a replica rebuild hiccup. The container is running, the pod is Ready, and the readiness probe passes because it only checks that the HTTP port answers. The application is writing errors into a filesystem it can't write to. I covered that class of problem in the gap between "Healthy" and actually working.

The standard remediation is to delete the pod so it remounts cleanly. So you delete it. And now you have two failures stacked on top of each other: the storage issue is resolved, and the workload is permanently down because the replacement pod can't be admitted. You've converted a recoverable glitch into an outage, using the recovery procedure.

GitOps makes this worse rather than better. ArgoCD reconciles the Deployment, the Deployment object applies cleanly (Kyverno's autogen rules validate the pod template, but if the Deployment already existed with :latest and you're not touching the image field, nothing changes about the violation state). Sync status reports fine. The health check may report Progressing or Degraded, but the reason lives three objects away. If you run App-of-Apps across a lot of applications, one Degraded app in a wall of green is easy to lose.

allowExistingViolations doesn't save you

Kyverno 1.13 added allowExistingViolations to validate rules, defaulting to true. Reasonable people read the name and assume it grandfathers in their pre-existing :latest workloads.

It doesn't do that. The field governs updates to resources that already exist and already violate. A Pod created by a ReplicaSet controller is a brand-new object performing a CREATE. There's no prior version for Kyverno to compare against, so there's nothing to grandfather. The rule evaluates and denies.

It's a useful field for letting people patch an annotation on a non-compliant Deployment without being blocked. It is not a safety net for pod recreation.

The fix

Three layers, in the order you'd actually do them.

Unblock right now. Flip the rule to Audit, or scope it away from the affected namespace. Validate rules are mutable, so a patch works:

kubectl patch clusterpolicy disallow-latest-tag --type=json \
  -p='[{"op":"replace","path":"/spec/rules/1/validate/failureAction","value":"Audit"}]'

One caveat if your policy bundle includes generate rules: those are immutable in Kyverno. Editing a generate rule's target or data requires deleting and recreating the ClusterPolicy, which will briefly remove enforcement. Plan that for a moment when you aren't already mid-incident.

Kyverno also supports PolicyException as the proper escape hatch, but it has to be enabled at install time (--enablePolicyException=true) and confined to a namespace you control. If it isn't already on, an incident is a bad time to discover that.

Fix the manifest properly. Pin the digest. This is the part people skip because it feels like a workaround, and it's actually the correct answer:

spec:
  containers:
    - name: app
      # tag for humans, digest for the runtime
      image: ghcr.io/example/app:v1.8.2@sha256:9f2a1c4e7b03d5a68e1f4c92b7d0a3e5f81c6d4b2a9e07f3c5d1b8a6e4f2c0d9
      imagePullPolicy: IfNotPresent

When both are present, the container runtime resolves by digest and ignores the tag. You get a human-readable version in kubectl get pod -o wide, byte-for-byte reproducibility, and a string that doesn't match *:latest.

The digest-only form (ghcr.io/example/app@sha256:...) also passes both rules, though it's worth understanding why: the require-image-tag rule matches *:*, and sha256:9f2a... happens to contain a colon. The policy is doing string pattern matching, not image reference parsing. That should give you a healthy skepticism about how airtight these policies are in general.

Stop it from reaching the cluster. Image tag validation belongs in CI, not in a webhook that fires during a node drain. Kyverno ships kyverno apply for exactly this:

# fail the PR, not the 3am pod recreation
kyverno apply ./policies/ --resource ./manifests/ --detailed-results

Wire that into the same job that runs your schema validation. I went through the setup for that in Kubernetes manifest validation in CI. The admission webhook then becomes a backstop against things that bypassed the pipeline, which is what it should have been all along.

Rolling out policy without setting a trap

The general rule: an Enforce policy is only safe once you've proven the existing fleet complies. Audit mode plus the background scanner gives you that proof.

# every currently non-compliant resource in the cluster
kubectl get policyreport -A -o json \
  | jq -r '.items[].results[]
      | select(.result=="fail" and .policy=="disallow-latest-tag")
      | "\(.resources[0].namespace)/\(.resources[0].name)"' \
  | sort -u

Drive that list to zero, then switch to Enforce. Not the other way around.

Two more things worth setting before you turn enforcement on:

Check your webhook failure policy. Kyverno's validating webhooks default to failurePolicy: Fail. If Kyverno itself is unavailable during a control-plane restart, all matching pod creation stops cluster-wide. That's a much bigger version of the same self-inflicted outage. For policies that aren't strictly security-critical, Ignore is the safer tradeoff.
Exclude the namespaces that have to come back first. kube-system, your CNI, your storage system, your ingress controller. If those can't recreate pods during a node failure, the policy has stopped being a security control and started being a single point of failure.

The general lesson

Admission control validates transitions, not states. That distinction sounds academic until a policy you wrote in April silently marks a dozen workloads as "runs fine, never restarts," and then a node goes down.

The tell is that Kubernetes stops self-healing without telling you why in the obvious place. Whenever a Deployment sits at zero pods and there's nothing to describe, walk down to the ReplicaSet before you go anywhere else. kubectl get events -A --field-selector reason=FailedCreate will find it across the whole cluster in one shot, and it should probably be a Prometheus alert rather than something you remember to type.

Policy that makes your cluster less able to recover isn't security, it's a reliability liability wearing a security badge. Getting that boundary right (what belongs in CI, what belongs in a webhook, and what should never block a pod creation) is most of the work. If you're sorting out where those lines go in your own infrastructure, that's the kind of thing I help teams with.

I Benchmarked My Homelab Memory Stack: Hybrid Search + Local Reranker Took LoCoMo from 63% to 80%

Guatu — Thu, 13 Aug 2026 18:15:48 +0000

Pure vector search got my agent memory stack to 63% on LoCoMo. Adding a sparse retriever and a reranker that runs on a card I already owned pushed it to 80%. The accuracy came from a stage that adds maybe 40ms per query, and the queries it fixed were exactly the ones I cared about: specific dates, error codes, and "who said what in which session" needles buried in months of conversation history.

If you're running a local agent that recalls facts across long conversations, this is the retrieval layer under everything else. A bad memory stack doesn't crash. It quietly hands the model the wrong three chunks and lets it confabulate a confident answer. That failure mode is worse than an outage because nothing tells you it happened.

The setup and why I benchmarked at all

My agents run on a memory stack I've written about before: a six-layer architecture for Claude Code with a wiki layer, a vector store, and an activation-based cognitive layer. The vector store is the workhorse. When an agent needs to recall a fact from a past session, it embeds the query, pulls the top-k nearest chunks, stuffs them into context, and answers.

That worked well enough that I never questioned it. Then I ran LoCoMo against it.

LoCoMo is a long-term conversational memory benchmark. It gives you multi-session dialogues that span hundreds of turns, then asks questions whose answers are scattered across those sessions. Single-hop lookups, multi-hop reasoning, temporal ordering, the works. It's a good proxy for what an agent memory system actually has to do, because the answer is never in the most recent turn. It's three sessions back, phrased differently than the question.

My vector-only stack scored 63%. Not terrible. Not good enough to trust an agent to act on. The interesting part wasn't the number, it was the shape of the failures.

What I tried first (and why it was the wrong lever)

My first instinct was the obvious one: the embeddings must be too weak. Swap the model, get better vectors, problem solved.

So I did the thing everyone does. I moved from a general-purpose embedding model to a larger, higher-ranked one on the MTEB leaderboard. Re-embedded the whole corpus. Re-ran LoCoMo.

63% went to 65%.

Two points. Hours of re-embedding for two points. That's when I actually looked at the failures instead of the aggregate score, and the pattern was obvious in hindsight. The questions I was getting wrong weren't semantically hard. They were lexically specific:

"What was the ticket number the user mentioned?" — the chunk with TICKET-4471 in it wasn't in the top-k, because "ticket number" as a query embeds close to a hundred chunks that talk about tickets in general.
"Which date did they say the migration finished?" — the model retrieved chunks about the migration, just not the one sentence with the actual date.
"What did Maria say about the vendor?" — proper nouns get averaged into oblivion by dense embeddings. "Maria" and "the vendor" are needles, and cosine similarity is bad at needles.

This is the well-documented weakness of dense retrieval. Embeddings capture meaning, and they're great at "find me things about database migrations." They're bad at "find me the exact string TICKET-4471," because that string's meaning is thin. There's nothing semantic about an identifier. A better embedding model doesn't fix a problem that isn't about semantics.

The second thing I tried was cranking k. If the right chunk isn't in the top 5, pull the top 20. That helps recall, and it did nudge the score. It also blows up the context window with noise and triggers the "lost in the middle" problem, where the model ignores relevant chunks buried between irrelevant ones. I was trading a retrieval problem for an attention problem. Not a win.

The actual fix: sparse recall, then rerank for precision

The move that mattered was splitting retrieval into two jobs it was badly trying to do at once.

Recall is "get the right chunk into the candidate set somehow." Precision is "put the right chunk at the top." Dense search alone is mediocre at both for needle queries. So I stopped asking it to do both.

For recall, I added BM25 sparse search alongside the dense search and fused the two with Reciprocal Rank Fusion. BM25 is a keyword retriever from the 1990s, and it is still undefeated at finding exact tokens. TICKET-4471 scores high on BM25 the instant the query contains it. RRF combines the two ranked lists without needing to normalize their scores, which is the whole reason it's the default fusion method in every mature vector DB now.

Here's the hybrid retrieval, using LangChain's ensemble retriever over an Ollama-served embedding model and an in-memory BM25 index:

from langchain.retrievers import EnsembleRetriever, BM25Retriever
from langchain_community.vectorstores import Qdrant
from langchain_community.embeddings import OllamaEmbeddings

# Dense: semantic recall via local embeddings
embeddings = OllamaEmbeddings(model="bge-m3", base_url="http://10.0.0.100:11434")
dense = Qdrant.from_existing_collection(
    embedding=embeddings, collection_name="agent_memory",
    url="http://10.0.0.100:6333",
).as_retriever(search_kwargs={"k": 20})

# Sparse: exact-token recall for IDs, dates, proper nouns
sparse = BM25Retriever.from_documents(all_chunks)
sparse.k = 20

# RRF fusion. Weights lean slightly toward dense for this corpus.
hybrid = EnsembleRetriever(retrievers=[dense, sparse], weights=[0.6, 0.4])

That alone moved 63% to roughly 72%. The needle queries started landing in the candidate set. But they were landing at rank 11, or rank 8, not rank 1, and I was still pulling too many chunks into context to be safe. Recall was fixed. Precision wasn't.

For precision, I added a reranker. This is the part people skip because it "adds a model," and it's the part that did the heavy lifting.

A reranker is a cross-encoder. Instead of embedding the query and the document separately and comparing vectors (a bi-encoder, which is what your vector search does), it feeds the query and each candidate together through the model and scores their actual relevance. It's slower per pair, which is why you never use it for the first-stage search over thousands of chunks. But over 20 candidates? It's cheap, and it's dramatically more accurate because it can see the query and document at the same time.

I ran BAAI/bge-reranker-base locally. It's small, and it fits alongside my inference workloads on the Tesla P40 I already had without a fight over VRAM. Around 1.1GB loaded.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-base", device="cuda", max_length=512)

def retrieve(query: str, top_n: int = 5):
    candidates = hybrid.invoke(query)            # 20-40 fused candidates
    pairs = [(query, doc.page_content) for doc in candidates]
    scores = reranker.predict(pairs)             # true relevance per pair
    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    return [doc for doc, _ in ranked[:top_n]]    # feed only the best 5

The pipeline is now: hybrid recall pulls 40 candidates, the reranker scores all 40, I keep the top 5. That top-5 goes to the model. LoCoMo landed at 80%.

Why this works, not just that it works

The reason the reranker earns its keep comes down to what a bi-encoder physically cannot do.

When you embed a document at index time, you compress its entire meaning into one fixed vector before you've ever seen the query. That vector has to be a decent answer to every possible question about that chunk. It's a lossy average. For a chunk that says "the migration finished on March 14th after Maria flagged the vendor delay," the embedding smears the date, the name, and the topic together. When your query is specifically about the date, the vector doesn't get any sharper, because it was frozen months ago.

A cross-encoder sees the query at scoring time. It reads "which date did the migration finish?" alongside that chunk and can attend directly to "March 14th." It's not comparing two averages. It's answering a specific relevance question with both halves in front of it. That's why reranking fixes precision on exactly the query types that dense search chokes on, and why a bigger embedding model didn't: the problem was never the quality of the average, it was the averaging itself.

Hybrid search and reranking are attacking two different failures, which is why stacking them compounds. BM25 guarantees the needle chunk exists in the candidate pool. The reranker guarantees it floats to the top of that pool. Neither one alone gets you there. BM25 without reranking dumps the needle at rank 9 with 19 distractors. Reranking without BM25 can only reorder a candidate set that never contained the needle to begin with. You need the recall stage to be generous and the precision stage to be strict.

This is the practical version of the theory I dug into in vector search vs activation-based recall: different retrieval mechanisms have different failure modes, and a serious memory system layers them instead of betting everything on one.

The latency tax, measured honestly

Nothing is free. Here's what the two-stage pipeline cost on my hardware, averaged over the LoCoMo query set:

Stage	Vector-only	Hybrid + rerank
First-stage retrieval	~18ms	~31ms (dense + BM25 in parallel)
Rerank (40 candidates)	—	~42ms
Total retrieval	~18ms	~73ms
LoCoMo accuracy	63%	80%

Retrieval got roughly 4x slower in absolute terms and added about 55ms end to end. For an interactive agent where the LLM generation step is already 2 to 8 seconds, 55ms of extra retrieval latency is noise. Nobody perceives it. I paid 55ms and got 17 points of accuracy on the queries that decide whether the agent is trustworthy.

The trade would look different if I were serving retrieval as a standalone API at high QPS. Then 4x matters and I'd think about batching rerank calls or caching. For a single-user agentic workflow, it's the easiest 17 points I've ever bought.

One VRAM note, since the reranker shares a GPU with inference: bge-reranker-base at fp16 is small enough to coexist, but if your inference model already fills the card, you'll evict it or OOM. I keep the reranker pinned and size the LLM around it. On CPU it's viable too, around 200ms for 40 candidates on a modern core count, which is fine if you don't have spare VRAM.

Lessons learned

Look at the failures, not the score. The two hours I spent swapping embedding models were wasted because I optimized an aggregate instead of reading which questions I got wrong. The moment I bucketed failures by query type, the fix was obvious. Every point I gained after that came from a targeted change, not a bigger hammer.

Dense embeddings are bad at identifiers, and no embedding model fixes that. Ticket numbers, dates, SKUs, proper nouns, error codes. If your agent recalls anything with a specific token in it, you need a sparse retriever in the loop. This isn't a tuning problem. It's a property of how dense vectors compress meaning.

Reranking is the highest-use stage most people skip. It gets dismissed as "an extra model" and "more latency," and both are true and both are cheap. Splitting recall from precision is the core idea. Let the first stage be generous and dumb, let the second stage be strict and smart.

Build your own golden set. LoCoMo is a fine public benchmark, but the queries that matter for your agent are the ones your agent actually gets. I keep a small golden dataset of real recall queries and the chunk that should answer each one, and I run it on every change to the stack. Twenty good examples catch regressions that an aggregate score hides.

What surprised me most was how little the fancy part mattered relative to the boring part. I went in assuming the embedding model was the ceiling. The ceiling was a 30-year-old keyword algorithm and a small cross-encoder, both running on hardware I already had. This retrieval layer is the foundation the rest of the memory stack sits on, and it's the same layer I'd want solid before wiring agents together into anything multi-agent. Since the reranker runs locally, none of the recall traffic leaves the box, which keeps the whole thing aligned with a privacy-routed inference setup instead of shipping every query to a hosted reranking API.

If you're building agent memory or predictive systems on your own hardware and want a second set of eyes on the retrieval layer, that's the kind of work I do at GuatuLabs. The stack is simpler than the marketing around RAG makes it sound. Two retrievers, one reranker, and the discipline to measure what you actually broke.

Hybrid Retrieval v2: Qwen Embeddings, BM25, and RRF with a FastEmbed Reranker

Guatu — Thu, 13 Aug 2026 16:15:48 +0000

A query for ndots:5 against my wiki index used to return the article that exists specifically to explain ndots:5 at position seven. Ahead of it sat three general DNS articles, two Kubernetes networking posts, and something about service discovery. My embedding model understood the topic perfectly and had no idea that the literal string mattered.

That is the dense retrieval failure mode in one sentence. Semantic similarity is a fuzzy match by design, and a fuzzy match is exactly wrong when the user typed an exact identifier. Across a fixed 10-query eval set of the things I actually search for (config keys, error strings, CLI flags), dense-only retrieval put the correct article in the top 3 for 5 of them. Hybrid retrieval with a reranker on the same 10 queries hits 8.

Who should care

If you run a retrieval layer for agents and your corpus is technical documentation, code, runbooks, or session memories, you have this problem whether or not you've measured it. Technical corpora are full of tokens that carry near-zero semantic weight and near-total discriminative weight: max_cstate, Modifier.IDF, ErrImagePull, a CVE number, a Helm value path. An embedding model compresses all of those into a vector where they barely register against the surrounding prose.

My first instinct was to reach for a better embedding model. That instinct was wrong, and the reason it was wrong is the most useful thing in this post.

What I tried first

Bigger embeddings. Swapping to a larger dense model moved my eval by roughly one query out of ten, and cost more VRAM plus more latency per ingest batch. Larger dense models are better at nuance in prose. None of them are better at treating ndots:5 as an atomic symbol, because none of them are trained to. Adding dimensions does not create a keyword index.

Query expansion with an LLM. Rewrite the user query into three paraphrases, embed all three, union the results. This helped on vague questions and actively hurt on precise ones, because the paraphrases diluted the exact term being searched for. It also adds an LLM round trip to every retrieval call, which turns a 40ms operation into a 900ms one and makes results non-deterministic between runs. Acceptable for a chat UI. Bad for an agent that retrieves twenty times inside a single task.

A cross-encoder reranker served through Ollama. This one is worth writing down, because the failure was silent and cost the most time.

My plan was reasonable: over-retrieve 20 candidates from dense search, then rerank with a cross-encoder that sees query and document together. Ollama was already running in the cluster, GGUF conversions of popular rerankers exist on Hugging Face, so pull one, hit the API, sort by score.

Scores came back as numbers. They were garbage. Not obviously broken (no errors, no NaNs), just weakly correlated with relevance. Sometimes the reranked order was measurably worse than the pre-rerank order, which is an impressive achievement for a component whose entire job is to improve ordering.

Here's the mechanism. A cross-encoder reranker is a sequence-classification model: an encoder backbone plus a trained classification head that emits a single relevance logit. Convert that to GGUF, serve it through a runtime built for causal LM generation and embedding extraction, and the classification head is usually not part of the picture. What comes back is a pooled hidden state, or a logit from a head that was never trained for relevance ranking, wrapped in a response shape identical to a real score. Nothing warns you. Your pipeline runs, your latency budget looks fine, and retrieval quality quietly rots.

Generalizing: when a model's output is a scalar, you cannot tell by inspection whether it's the right scalar. Test rerankers against a fixed query set with known-correct answers before you wire them in, not after you've shipped them.

The actual solution

Four pieces. A Qdrant collection with two named vector spaces, dense embeddings from qwen3-embedding:0.6b, sparse BM25 vectors, and a cross-encoder reranker running on ONNX through FastEmbed. No GPU is involved in the reranking stage at all.

1. The collection schema

Dense and sparse vectors live on the same point. One document, one ID, two vector representations, one payload. That detail matters more than it looks: split them across two collections and you get two ingest paths that drift out of sync, and you'll find out about the drift during a retrieval failure at the worst possible moment.

from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://qdrant:6333")

client.create_collection(
    collection_name="wiki_index_v2",
    vectors_config={
        "dense": models.VectorParams(
            size=1024,                      # qwen3-embedding:0.6b
            distance=models.Distance.COSINE,
        ),
    },
    sparse_vectors_config={
        "bm25": models.SparseVectorParams(
            # Qdrant applies IDF server-side against the live corpus
            modifier=models.Modifier.IDF,
        ),
    },
)

modifier=models.Modifier.IDF is the line people skip. FastEmbed's BM25 produces the term-frequency component client-side, but inverse document frequency depends on the entire corpus, and your corpus changes on every ingest. Setting the modifier makes Qdrant compute IDF at query time from current collection statistics. Leave it out and you're doing raw term-frequency matching, which over-weights common tokens and makes the sparse leg noticeably worse: in my eval it cost two of the eight top-3 hits.

You cannot add the modifier later without recreating the collection. Set it on day one.

2. Ingest both vectors in one upsert

from fastembed import SparseTextEmbedding
import ollama

bm25 = SparseTextEmbedding("Qdrant/bm25")

def embed_dense(text: str) -> list[float]:
    return ollama.embed(model="qwen3-embedding:0.6b", input=text)["embeddings"][0]

def to_point(doc_id: int, text: str, payload: dict) -> models.PointStruct:
    sparse = next(bm25.embed(text))
    return models.PointStruct(
        id=doc_id,
        vector={
            "dense": embed_dense(text),
            "bm25": models.SparseVector(
                indices=sparse.indices.tolist(),
                values=sparse.values.tolist(),
            ),
        },
        payload=payload,
    )

SparseTextEmbedding("Qdrant/bm25") is not a neural model. It's a tokenizer plus stemming plus stopword removal, running in a few hundred microseconds per document. The cost of the sparse leg is rounding error next to the dense embedding call.

One migration note. Moving a few hundred wiki articles and roughly twice as many session memories into the new schema meant re-embedding everything, and re-embedding is exactly where payloads get quietly dropped. My rule: read the full point from the old collection, carry the payload dict forward untouched, and diff payload key sets between source and destination when the run finishes. If a key existed on 300 points before and 280 after, you want a failing assertion, not a shrug. This is the same class of problem I wrote about in Silent Drift: count-based checks pass while content quietly diverges.

3. Query both legs and fuse with RRF

Qdrant does the fusion server-side through prefetch, which saves a round trip and keeps the client dumb:

sparse_q = next(bm25.query_embed(user_query))

results = client.query_points(
    collection_name="wiki_index_v2",
    prefetch=[
        models.Prefetch(query=embed_dense(user_query), using="dense", limit=20),
        models.Prefetch(
            query=models.SparseVector(
                indices=sparse_q.indices.tolist(),
                values=sparse_q.values.tolist(),
            ),
            using="bm25",
            limit=20,
        ),
    ],
    query=models.FusionQuery(fusion=models.Fusion.RRF),
    limit=20,
    with_payload=True,
).points

Use bm25.query_embed() for queries, not bm25.embed(). Query embedding skips the term-frequency weighting that only makes sense for documents. Mixing them up produces results that look plausible and rank badly.

The fusion itself is about six lines, and it's worth seeing them written out even if Qdrant runs it for you:

def rrf(ranked_lists, k=60):
    scores = {}
    for lst in ranked_lists:                 # each list is [doc_id, ...] by rank
        for rank, doc_id in enumerate(lst, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda kv: kv[1], reverse=True)

No score normalization. No tunable alpha weighting dense against sparse. Rank position is the only input.

4. Rerank on CPU with FastEmbed

from fastembed.rerank.cross_encoder import TextCrossEncoder

reranker = TextCrossEncoder(model_name="jinaai/jina-reranker-v2-base-multilingual")

def rerank(query: str, candidates, top_n: int = 5):
    docs = [c.payload["text"] for c in candidates]
    scores = list(reranker.rerank(query, docs))   # batched ONNX inference
    ranked = sorted(zip(scores, candidates), key=lambda p: p[0], reverse=True)
    return [c for _, c in ranked[:top_n]]

That's the whole reranking stage. FastEmbed ships the ONNX export with the classification head intact, downloads it on first use, and runs it through ONNX Runtime on CPU at roughly 38ms per query-document pair. Twenty candidates batched lands under half a second on a few cores, and it needs no GPU, no separate inference server, and no model-serving deployment to keep alive.

Compare that to the GGUF path: same nominal model, wrong head, silently meaningless scores.

What changed on the eval set

Configuration	Correct doc in top 3	Median latency
Dense only (1024d)	5 / 10	~45 ms
BM25 only	6 / 10	~12 ms
Dense + BM25, RRF	7 / 10	~55 ms
Dense + BM25, RRF, reranked	8 / 10	~480 ms

BM25 alone beating dense alone surprised me. It also makes sense in hindsight: over half my eval queries were literal strings copied out of a config file or an error log, which is BM25's home turf and dense retrieval's blind spot.

Here's the shape of a query that used to fail:

query: "ndots:5"

dense-only ranking:
  1. Kubernetes Service Discovery Patterns          (cos 0.612)
  2. DNS Failover With Two Upstreams                (cos 0.598)
  3. CoreDNS Tuning Notes                           (cos 0.591)
  ...
  7. Wildcard DNS + ndots:5: The TLS Nightmare      (cos 0.544)

hybrid + rerank:
  1. Wildcard DNS + ndots:5: The TLS Nightmare      (rerank  6.81)
  2. CoreDNS Tuning Notes                           (rerank  1.24)
  3. Kubernetes Service Discovery Patterns          (rerank  0.37)

BM25 pulled the right article into the candidate pool at sparse rank 1. RRF pushed it to fused rank 2. The cross-encoder, which actually reads the query and the document together, put it first with a score nearly six times the runner-up.

Why it works

Three separate mechanisms are doing distinct jobs, and it's worth being precise about which does what.

Sparse retrieval indexes symbols, not meaning. BM25 scores a document on term frequency scaled by inverse document frequency, with length normalization. ndots appears in one article out of several hundred, so its IDF is enormous and any document containing it rockets to the top. An embedding model does the opposite: it maps rare tokens into a region of vector space defined by their context, which is exactly the behavior you want for synonyms and exactly the behavior you don't want for identifiers. Dense and sparse are not competing implementations of retrieval. They index different properties of the same text.

RRF fuses ranks because scores are incomparable. Cosine similarity lives in [-1, 1] and clusters hard around 0.5 to 0.7 for a technical corpus. BM25 scores are unbounded and depend on corpus size, document length, and term rarity. Normalizing them onto a shared scale requires assumptions about their distributions that break whenever the corpus changes. Reciprocal rank fusion sidesteps the problem: it throws the scores away and keeps only the ordering, then sums 1/(k + rank) across both lists. The k=60 constant flattens the curve near the top so that rank 1 versus rank 2 isn't a cliff, which means a document ranked 3rd by both retrievers can outrank a document ranked 1st by one and 40th by the other. Consensus wins over one confident vote, and that's the behavior you want when one leg is guessing.

Cross-encoders can do what bi-encoders structurally cannot. Your embedding model is a bi-encoder: query and document are encoded independently, never seeing each other, and compared by cosine distance at the end. That independence is what makes vector search fast, because you precompute every document embedding once. It also means the model never gets to ask "does this specific document answer this specific question." A cross-encoder concatenates query and document into one sequence and runs full attention across both, so query tokens attend directly to document tokens. Far more accurate, and far too slow to run against your whole corpus. Which is precisely why the architecture is retrieve-then-rerank: cheap methods cut several hundred documents down to 20, the expensive method orders those 20.

That layering also explains why over-retrieval depth matters. Reranking cannot recover a document that never entered the candidate pool. If your prefetch limit is 5, the reranker is just reordering five things, and your recall ceiling is whatever RRF handed it. Twenty per leg is where my eval stopped improving; going to 50 added latency and no additional top-3 hits.

Lessons learned

Test the reranker in isolation before trusting it. Build a fixture of 10 to 20 query-document pairs where you know the ranking by hand, score them, and check the correlation. That test takes an hour and would have saved me the entire GGUF detour. It also catches the subtler failure where a reranker works fine on prose and falls apart on code blocks.

ONNX over GGUF for anything with a classification head. GGUF is a format built around generative decoder inference. Cross-encoders, classifiers, and any model whose value lives in a trained head on top of the backbone should go through ONNX Runtime, where the head is exported with the graph. FastEmbed makes that a one-liner, and running it on CPU means the reranker isn't competing with your LLM for VRAM. I don't need an accelerator to serve retrieval, which matters when the GPU is busy doing actual inference.

Set Modifier.IDF at creation time. I'd rather see this documented in bold in every hybrid search tutorial. Missing it does not raise an error, it just makes the sparse leg mediocre in a way you'll blame on BM25 rather than on your config.

Measure with your queries, not a benchmark. MTEB scores told me nothing useful about whether retrieval would find the article about a specific kernel parameter. A hand-built eval of 10 real queries with known-correct answers told me everything, and it's small enough to rerun in under a minute after any config change. Keyword-in-top-3 is a crude metric and a good one, because it maps directly to what the agent experiences: the right context is in the window or it isn't.

Retrieval precision is upstream of everything else in an agent stack. Better memory decay policies, better tool descriptions, better prompts, none of them compensate for handing the model the wrong three documents. I've come to treat the retrieval layer the way I treat storage: unglamorous, load-bearing, and worth over-engineering slightly. It sits underneath the memory architecture and the decay policy, and it's the layer that determines whether the rest of the agent architecture has anything worthwhile to reason over. If you're building this kind of pipeline for something that has to work on a schedule rather than on a weekend, that's the sort of work I do.

What surprised me: the reranker mattered less than adding BM25. Fusion alone took the eval from 5/10 to 7/10; the cross-encoder added the eighth. I'd assumed the fancy neural component would carry the improvement, and instead the win came from a 1994-vintage ranking function that runs in twelve milliseconds and has no parameters to train. The old algorithm knows something the new model doesn't, which is that sometimes the user meant the exact characters they typed.

DEV Community: Guatu

Blog Pipeline Fence Corruption: How a Naive Sed Rule Swallowed Markdown Code Blocks

What I expected

What actually happened

The fix, part one: stop using substitution for a positional problem

The fix, part two: assert structure, don't count it

The fix, part three: repair before you reject

Where the gates fight each other

Why this matters beyond one blog

Tailscale Kernel TUN in Unprivileged LXC: Direct SSH Without Userspace Networking

Who should care

What userspace networking actually costs you

The false starts

The actual fix

1. Load tun on the host and persist it

2. Add two lines to the container config

3. Stop and start the container from the host

4. Drop the userspace flag and bring Tailscale up

Verifying it actually worked

Why it works

About that nobody:nogroup

Direct paths versus the subnet router model

Before you restart anything

Lessons learned

Systemd-tmpfiles vs Plex: When /tmp Aging Breaks EAC3 Transcoding

The failure mode

What it looks like in the logs

The permanent fix: a tmpfiles.d exclusion

The alternative fix: move EAE out of /tmp entirely

Gotchas and adjacent traps

Takeaway

Pushgateway Heartbeat Gotcha: When ndots and NetworkPolicy Silently Eat Your Alerts

What I expected

What actually happened

Layer 1: ndots:5 turns your FQDN into a wildcard lookup

Layer 2: NetworkPolicy makes the wrong answer unreachable, quietly

Layer 3: both the script and the alert fail open

The fix

Why this matters

We Almost Deployed a Temporal Knowledge Graph. The Eval Said No.

What I expected

What actually happened

The fix

Why this matters

Traefik Host Collision: Why Duplicate Host Rules Break IngressRoutes

What you'd expect

What actually happens

The auth-proxy fallthrough diagnostic

Reading the actual routing table

The fix

Set an explicit priority

Or narrow the wildcard

The hack that works and that you shouldn't ship

The second path to the same symptom: cross-namespace middleware

The double-blind: when the shadowing service is also broken

The collision-free pattern for internal services

Catching it before merge

What I'd change

The Write Policy Is the Hard Part: Promotion Pipelines for Agent Memory

Retrieval is solved. Promotion isn't.

What I tried first: write-everything, filter-later

The actual solution: a promotion pipeline with a gatekeeper

Then security breaks the whole thing

Network policy: the silent blinding

When the automated pipeline fails, promote out of band

Why gating at write-time beats filtering at read-time

Lessons learned

Moving Scheduled LLM Curation from Cloud APIs to Local Models

The decision point

Option A: Cloud APIs

Option B: Local models

The headless approval trap

Node pinning and image pulls

The memory that vanishes

Decision framework

My pick and why

Langfuse for LLM Observability: Tracing Agent Calls Instead of Guessing

Observability Sprawl: The Failure Mode Nobody Talks About

What I Actually Needed (And What Most Tools Get Wrong)

Setting Up Langfuse (Self-Hosted on Kubernetes)

1. Load `tun` on the host and persist it

About that `nobody:nogroup`