DEV Community: Guatu

We Almost Deployed a Temporal Knowledge Graph. The Eval Said No.

Guatu — Fri, 14 Aug 2026 06:15:48 +0000

The eval that killed the temporal knowledge graph asserted one thing: at time T, the agent should report the state that was true at T. It failed 41% of the time. The graph had the right facts. It just handed the agent the wrong one.

That number is what saved us from shipping. Every static retrieval metric looked fine. The graph answered "what is the status of Node A" with a confident, well-formed response. Trouble is, "what is the status" is a temporal question wearing a static question's clothes, and nothing in our test suite had noticed the difference until we wrote a test that actually asked about time.

What I expected

The pitch for a temporal knowledge graph (TKG) is genuinely good. You store facts as quadruples instead of triples: (subject, predicate, object, timestamp) or, better, (subject, predicate, object, valid_from, valid_to). Now your agent memory isn't a flat pile of embeddings, it's a structured record of what was true and when. This is the natural next step past pure vector recall, and it slots neatly into the decay-based thinking I've written about before in Eviction Without Deletion. Instead of letting old facts fade by activation weight, you make validity windows explicit.

My hope was that the graph would fix the exact failure mode that plagues flat vector memory: the agent confidently recalling a stale fact because it's semantically close to the query. With valid_from and valid_to on every edge, staleness becomes a filter, not a guess. Ask for the state at time T, filter edges where T falls inside the window, done. On paper it's cleaner than a decay curve because there's no fuzziness. A fact is either valid at T or it isn't.

Schema-wise, it was simple enough. In a property graph it looks like this:

// A temporal fact: Node A was in maintenance for a fixed window
MATCH (n:Server {name: 'node-a'})
CREATE (n)-[:HAS_STATE {
  status: 'maintenance',
  valid_from: datetime('2026-07-20T02:00:00Z'),
  valid_to:   datetime('2026-07-20T04:30:00Z')
}]->(:State {kind: 'maintenance'})

Multiple HAS_STATE edges per server, each with its own window. Query the graph, get the state for any point in time. This is the "structured shared memory" pattern I described in Multi-Agent AI Systems, except now the shared memory understands time. The whole thing felt like an upgrade in every dimension. It reads well in a design doc. It demos beautifully. That was part of the problem.

What actually happened

Retrieval is where it fell apart, and the failure is boring in a way that makes it dangerous. Here's roughly the query the agent's tool was generating:

// The "obvious" query - find the status of a server
MATCH (n:Server {name: $server})-[r:HAS_STATE]->(s:State)
RETURN s.status, r.valid_from, r.valid_to
ORDER BY r.valid_from DESC
LIMIT 1

Read that carefully. It orders by valid_from descending and takes the most recent fact. Most of the time that's correct, because the most recently started state is usually the current one. What it never does is filter by the query's reference time. If the agent is reasoning about an incident that happened at 03:00, and a newer "online" state was recorded at 05:00, this query returns "online." The graph knew Node A was in maintenance at 03:00. The retrieval logic threw that knowledge away.

This is the hallucinated-history problem, and it's insidious because the model isn't hallucinating. The fact is real. The timestamp is real. The agent is just being handed a fact from the wrong window and has no way to know it. Worse, the answer is fluent and specific, so every static evaluation gives it a pass. RAGAS-style faithfulness checks look at whether the answer is grounded in the retrieved context. It was. The retrieved context was simply the wrong slice of time.

I want to be precise about where the failure lived, because it wasn't the graph. The graph was correct. The schema was correct. The data was correct. The failure was split across two places: a retrieval query that dropped the temporal filter, and an evaluation suite that had no test capable of noticing. If we'd only had the first problem, we'd have caught it in review. Having both meant the system looked healthy right up until the one test that mattered.

That missing test is short:

import pytest
from agent_memory import query_state  # our TKG retrieval tool

def test_temporal_regression():
    """Agent must report the state valid AT the reference time,
    not the most recently recorded state."""
    # Maintenance window: 02:00-04:30. Online recorded at 05:00.
    result = query_state(server="node-a", at="2026-07-20T03:00:00Z")

    # A later 'online' fact exists, but at 03:00 the truth is 'maintenance'
    assert result["status"] == "maintenance", (
        f"temporal regression: got {result['status']} "
        f"for a timestamp inside the maintenance window"
    )

Run that single assertion across a few dozen historical state transitions and you get the 41% failure rate. Not a subtle degradation. Nearly half of all time-scoped questions returned a state from the wrong window whenever a newer fact existed. Meanwhile the static suite, which only checked "does the agent know the current status," passed everything. Two evals looking at the same system, one green, one red, and only the red one described reality.

The fix

Correcting the retrieval was one clause. You filter edges so the reference time falls inside the validity window before you order or limit anything:

// Filter by the reference time FIRST, then pick the winner
MATCH (n:Server {name: $server})-[r:HAS_STATE]->(s:State)
WHERE r.valid_from <= datetime($at)
  AND (r.valid_to IS NULL OR r.valid_to > datetime($at))
RETURN s.status, r.valid_from, r.valid_to
ORDER BY r.valid_from DESC
LIMIT 1

valid_to IS NULL handles the open-ended "current" state, the fact that has started but not yet ended. Everything else is a closed window, and the WHERE clause guarantees you only ever consider edges whose window contains T. The ORDER BY ... LIMIT 1 is still there to break ties if two windows overlap, but now it's picking among facts that are all actually valid at T, not among every fact ever recorded.

One clause. That's the entire retrieval fix. Which tells you the real bug was never in the query, it was in the fact that nobody wrote the eval that would have made the missing clause obvious on day one.

So the second half of the fix was the more important one: the retrieval tool never gets to reason about time on its own. The agent isn't trusted to remember to pass a reference timestamp, and the LLM isn't trusted to filter windows in its head. Instead, the current time is injected as explicit context at the tool boundary, and the tool refuses to answer a state question without it:

def query_state(server: str, at: str | None = None) -> dict:
    if at is None:
        raise ValueError(
            "query_state requires a reference time. "
            "Temporal facts are meaningless without one."
        )
    # ... run the time-filtered Cypher above ...

Making the reference time a required argument sounds trivial. It's the difference between a tool that silently returns plausible garbage and one that fails loudly when it's used wrong. A loud failure is a bug report. A plausible answer from the wrong window is an incident three weeks later that nobody can reproduce.

We also changed how the retrieved fact is handed to the model. Rather than passing "status: maintenance" as a bare string, the prompt gets the window with it: "As of 2026-07-20T03:00:00Z, node-a status is 'maintenance' (valid 02:00-04:30). A later 'online' state exists from 05:00 and is not applicable to this query." Giving the model the window and the reference time in the same breath means that even if the retrieval ever regresses, the model has a fighting chance to notice the mismatch. Defense in depth, applied to a knowledge base.

Why this matters

I keep coming back to the fact that we did not ship this because of the architecture. We almost shipped it because the architecture was correct and only the evaluation was wrong. That inversion is the whole lesson. A TKG is more capable than flat vector memory, and that extra capability comes with an entirely new class of failure that your existing evals were never designed to see. Adding temporal structure to your memory adds temporal bugs, and static tests are blind to all of them by construction.

Here's the trap I see constantly. Teams treat GraphRAG as a strictly-better upgrade over vector RAG, port their old eval suite unchanged, watch it stay green, and conclude the new system is at least as good as the old one. Their green suite is measuring the properties the old system could fail on. It has no assertion about sequence, no assertion about validity windows, no assertion that state at T equals the state that was actually true at T. The new failure mode is invisible not because it's rare but because nothing is looking for it. This is the same gap I described in Cognitive Memory for Agents: the retrieval method changed, so the questions your evals ask have to change too.

If you're building temporal memory for an agent, write the temporal regression test before you write the graph. Seed a handful of known state transitions where a later fact contradicts an earlier one, then assert that a query scoped to the earlier window returns the earlier fact. That test is a dozen lines. It will fail the moment your retrieval forgets to filter by time, which, based on how naturally that ORDER BY valid_from DESC LIMIT 1 query wrote itself, is going to be your very first implementation.

A few things I'd carry into the next attempt. Make the reference time a required parameter on every temporal query, so the tool cannot be called ambiguously. Keep the graph correct and put your paranoia in the retrieval and the eval, because that's where the wrong-window bug actually lives. Expose temporal queries to the agent through a narrow, well-typed interface, whether that's a tool boundary or an MCP server, so the agent can't hand-roll a query that drops the filter. And treat "it passed the static eval" as necessary, never sufficient, the second time itself becomes a dimension of your data. If you're standing up this kind of time-aware memory for a production agent system and want a second set of eyes on the failure modes, that's a chunk of what I do consulting.

We didn't deploy the temporal knowledge graph that quarter. We deployed the eval, fixed the one-clause retrieval bug it exposed, and shipped the graph once the red test went green. The graph was never the risky part. The risky part was almost trusting a system that no test had ever asked the right question.

Traefik Host Collision: Why Duplicate Host Rules Break IngressRoutes

Guatu — Fri, 14 Aug 2026 04:15:47 +0000

A service deploys perfectly clean. Pods Running, Endpoints populated, the Certificate resource says Ready: True, the IngressRoute shows no events. Then you curl the hostname and get back a 404 page rendered by your SSO provider, which you never attached to that route.

Nothing in the deployment failed. Traefik logged nothing at error level. The 404 came from a service in a completely different namespace.

That's a host collision, and the mechanism behind it is one of the least intuitive parts of Traefik's routing model.

What you'd expect

Two routers, two rules. One says Host(\agents.example.com), the other says HostRegexp(\^.+.example.com$). A request for agents.example.com matches both. Any sane router picks the more specific match, because that's how routing works basically everywhere else: longest-prefix wins in IP routing, most-specific selector wins in CSS, exact match beats wildcard in DNS.

So the exact Host rule should win. It's narrower. It names one hostname. The regex names an infinite set.

Traefik does not work that way.

What actually happens

Traefik assigns every router a priority. If you don't set one, the default priority is the length of the rule string in characters. That's it. Not specificity, not match type, not creation order. Character count.

Do the arithmetic on those two rules:

Host(`agents.example.com`)              →  26 characters
HostRegexp(`^.+\.example\.com$`)        →  32 characters

The wildcard is 6 characters longer, so the wildcard wins. Every request for agents.example.com gets handed to whatever service the regex router points at, which in most clusters is an auth proxy sitting in front of everything. The auth proxy gets a Host header for an app it has no provider configured for, and returns its own 404.

The more generic rule outranks the more specific one because it happens to be typed with more characters. Rename your regex to something terser and the winner flips. Add a hyphen to a subdomain and the winner flips back. This is deterministic, it's documented, and it still surprises people every single time.

It gets worse when the tie is exact. Two IngressRoute objects in different namespaces with byte-identical Host rules produce two routers with identical priority. Traefik's Kubernetes CRD provider derives router names from a namespace/name hash, so they don't collide by name and nothing errors out. You get two valid routers competing at the same rank, and the tie-break isn't something you want to build a production dependency on. The winner can change on the next config reload, which is the worst possible failure mode: works on Tuesday, breaks on Thursday, nothing in Git changed.

The auth-proxy fallthrough diagnostic

Here's the pattern worth memorizing, because it saves an hour every time:

If you get a 404 (or a login redirect) from your auth provider on a service that was never wired to that auth provider, you have a Traefik routing collision.

The auth proxy is almost always the shadowing router, for two reasons. It usually owns the broadest rule in the cluster (a wildcard or regex covering the whole domain), and broad rules tend to be long rules. Regex syntax is verbose. HostRegexp is 10 characters before you've matched anything.

The symptom points at the auth stack, so that's where people start debugging. They check the outpost, the provider, the application binding, the token audience. All of it is fine. The auth proxy is behaving correctly given a Host header it doesn't recognize.

Reading the actual routing table

Stop guessing and ask Traefik what it decided. The runtime API exposes every router with its computed priority:

# Port-forward whatever entryPoint serves your API/dashboard
kubectl -n traefik port-forward deploy/traefik 8080:8080

curl -s localhost:8080/api/http/routers \
  | jq -r '.[] | select(.rule | test("example\\.com"))
           | [.priority, .status, .service, .rule] | @tsv' \
  | sort -rn

Sorted descending by priority, the first row is the router that wins. If that row isn't the one you deployed, you've found it in about fifteen seconds. The status field matters too: a router with status: "disabled" won't serve traffic, and that's a separate failure I'll get to below.

Two things this view gives you that the logs don't. It shows the computed priority, including the default length-based value you never wrote down anywhere. And it shows routers from every namespace at once, which is exactly the visibility you lose when you're reading one IngressRoute manifest at a time.

The fix

Set an explicit priority

The IngressRoute CRD takes a priority on each route. Setting it disables the length-based default for that router.

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: agents
  namespace: dev
spec:
  entryPoints: [websecure]
  routes:
    - kind: Rule
      match: Host(`agents.example.com`)
      priority: 100          # beats any length-derived default in practice
      services:
        - name: agents
          port: 8080
  tls:
    secretName: agents-tls

Pick a band and stick to it. Something like: exact-host routes get priority: 100, wildcard/catch-all routes get priority: 10. Two numbers, written into your chart defaults, and the entire class of bug disappears. Any exact host beats any wildcard regardless of how long anyone's rule string is.

The reason this works better than the alternatives is that it encodes intent. priority: 100 in a manifest says "this is a specific route and it should win." A 45-character rule string says nothing to the next person reading it.

Or narrow the wildcard

If your auth proxy genuinely needs a catch-all, scope it so it can't swallow namespaces it has no business serving. Instead of matching the whole apex domain, match only the subdomains you actually front:

# Broad — catches everything under the domain, including new services
match: HostRegexp(`^.+\.example\.com$`)

# Narrow — catches only what you opted in
match: Host(`sso.example.com`) || Host(`admin.example.com`)

Worth calling out: Traefik v3 changed HostRegexp syntax. The v2 named-group form ({subdomain:[a-z]+}.example.com) is gone; v3 expects standard Go regexp. If you migrated a v2 config and your wildcard silently stopped matching what you thought it matched, that's why. You can set syntax: v2 on a route as a compatibility escape hatch, but treat it as a migration step, not a destination.

The hack that works and that you shouldn't ship

Because priority is string length, you can win a collision by making your rule longer:

match: Host(`agents.example.com`) && PathPrefix(`/`)   # 45 characters

PathPrefix(\/) matches everything, so the semantics are unchanged, and you've bought 19 characters of priority. It works. I've seen it in production charts more than once.

Don't do it. It's a magic incantation that breaks the moment someone shortens the hostname or lengthens the competing rule, and nobody reviewing the diff will understand what it's for. Use priority.

The second path to the same symptom: cross-namespace middleware

There's a failure that looks identical from the outside but has nothing to do with rule length.

Since v3, the Kubernetes CRD provider defaults allowCrossNamespace to false. An IngressRoute in the dev namespace referencing a middleware in the auth namespace gets rejected:

routes:
  - kind: Rule
    match: Host(`agents.example.com`)
    middlewares:
      - name: forward-auth
        namespace: auth       # rejected unless allowCrossNamespace: true

Traefik logs a cross-namespace reference error and marks the router as disabled. It does not fail the deployment, does not update the IngressRoute status in a way that kubectl get makes obvious, and does not stop serving traffic. Your route simply isn't in the routing table anymore, so the request falls through to the next-best match. Which is the wildcard. Which is the auth proxy. Which returns a 404.

Same symptom, different root cause, and the /api/http/routers dump distinguishes them instantly: a collision shows your router present with a lower priority, a cross-namespace rejection shows your router disabled or missing entirely.

You can flip allowCrossNamespace: true in the provider config, but understand what you're accepting: any namespace can then attach any middleware from any other namespace, including auth middlewares it wasn't meant to have and, more interestingly, not attach ones it was. That's a real boundary, and turning it off to fix one route is the kind of decision that looks cheap now and expensive later. The same reasoning applies to how you scope identities across routing domains, which I got into in two-tier service accounts for agent workflows.

The double-blind: when the shadowing service is also broken

The nastiest version of this is when the auth proxy that's swallowing your traffic is itself misconfigured. Now you're debugging two failures at once and the symptoms interleave.

A common one: the auth provider's secret key contains a trailing newline. echo "value" | base64 adds one. So does most copy-paste out of a terminal. The provider starts, serves a page, and then fails signature validation on every token, so you get a 404 or a redirect loop that looks exactly like a routing problem.

Check secret shape without ever printing the value:

NS=auth; SECRET=provider-config; KEY=SECRET_KEY

raw=$(kubectl -n "$NS" get secret "$SECRET" \
      -o jsonpath="{.data.$KEY}" | base64 -d | wc -c)
clean=$(kubectl -n "$NS" get secret "$SECRET" \
        -o jsonpath="{.data.$KEY}" | base64 -d | tr -d '\n\r' | wc -c)

[ "$raw" -eq "$clean" ] \
  && echo "OK: no stray newlines ($raw bytes)" \
  || echo "FAIL: $((raw - clean)) newline/CR byte(s) in $KEY"

The value passes through the pipe and never lands in a variable or on stdout. Only the byte counts do. Run it as a pre-flight check on every secret that feeds an auth stack; the two-second version saves you from a debugging session where nothing makes sense.

The collision-free pattern for internal services

Not every service needs to sit behind the global auth wrapper, and routing everything through one broad rule is what creates the collision surface in the first place. For internal-only services, an IP allowlist plus a NetworkPolicy gives you a route that can't be shadowed because it never overlaps with anything:

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: lan-only
  namespace: dev
spec:
  ipAllowList:
    sourceRange:
      - 10.0.0.0/16
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: agents
  namespace: dev
spec:
  entryPoints: [websecure]
  routes:
    - kind: Rule
      match: Host(`agents.example.com`)
      priority: 100
      middlewares:
        - name: lan-only    # same namespace, no cross-namespace friction
      services:
        - name: agents
          port: 8080
  tls:
    secretName: agents-tls

Middleware lives in the same namespace as the route, so allowCrossNamespace is irrelevant. Priority is explicit, so rule-length arithmetic is irrelevant. Pair it with a default-deny NetworkPolicy at the pod level, the way I laid out in default-deny and namespace isolation with Calico, and you've got defense in depth without a single shared routing rule.

The tradeoff is honest: you're trading centralized auth for per-service configuration. More YAML, more places to get it wrong, no single place to revoke access. For a public-facing app, the global wrapper is the right call. For an internal dashboard, the allowlist wins on blast radius alone. This kind of routing-boundary decision is one of the things I spend a lot of time on in infrastructure consulting work, and the answer really does change per service.

Catching it before merge

Host collisions are a static property of your manifests. You don't need a running cluster to find them, which makes them a good fit for the kind of CI validation I described in catching broken YAML before merge:

# Fail the build if two IngressRoutes claim the same exact Host
grep -rhoE 'Host\(`[^`]+`\)' --include='*.yaml' ./manifests \
  | sort | uniq -d | grep . && {
      echo "duplicate Host rules found"; exit 1; }

Crude, and it won't catch regex overlaps, but it catches the exact-duplicate case that produces the nondeterministic tie. Extend it to flag any HostRegexp that lacks an explicit priority, and you've covered the two ways this actually bites.

What I'd change

Set priority on every route from day one. It costs one line and it removes an entire failure class. The default length-based behavior is fine for a single-tenant setup with five services; it stops being fine the moment two teams write rules against the same domain and neither one knows the other exists.

Treat the auth-provider 404 as a routing signal, not an auth signal. That reflex alone cuts debugging time dramatically, because it redirects you from the auth stack (where everything is working) to the routing table (where the answer is).

And keep /api/http/routers in your muscle memory. Manifests describe intent. The runtime API describes what Traefik actually built, priorities included, across every namespace at once. When those two disagree, the runtime API is right and your mental model is wrong. If you're also chasing certificate mismatches on the same hostnames, wildcard DNS and ndots:5 covers the other half of that puzzle, and it interacts with this one more often than you'd like.

The Write Policy Is the Hard Part: Promotion Pipelines for Agent Memory

Guatu — Fri, 14 Aug 2026 02:15:48 +0000

Most "agent memory" tutorials stop at the retrieval side. They show you a Qdrant collection, a qdrant-find call, an embedding model, and call it done. Retrieval is the easy 20%. The part that quietly eats your week is the write-path: deciding what deserves to be written, sanitizing it, and keeping the network and RBAC plumbing intact so the agent can actually reach the store it's allowed to write to.

If you run agents that update their own knowledge base, this is for you. The failure modes here aren't AI problems. They're distributed-systems problems wearing an AI costume: a 403 Forbidden on a write, a default-deny-ingress policy that silently blinds an agent to its own memory, a vector store slowly rotting into landfill because nobody gated the writes.

Retrieval is solved. Promotion isn't.

The read-side has good primitives. You embed a query, you search, you rank, you feed the top-k back into context. I've written about the recall half of this before in Cognitive Memory for Agents, where the interesting question is whether you use plain vector similarity or activation-based recall.

Promotion has no such consensus. Every observation an agent makes during a session is a candidate for long-term memory, and almost none of them should be promoted. A single session produces hundreds of transient facts: the value of a variable, a file it read, a command that failed, a user's throwaway comment. Write all of that to a permanent store and you don't have a memory. You have a landfill with a search index bolted on.

So the real design question isn't "how do I store this." It's "what is my write policy, and how do I enforce it without breaking security." That's two problems stacked on top of each other, and people usually only notice the second one after the first one is already leaking noise into their vector DB.

What I tried first: write-everything, filter-later

The obvious first move is to write everything and sort it out at read time. Cheap to build. Every observation goes straight into the vector store, and you rely on similarity ranking to surface the good stuff and bury the noise.

That falls apart for a boring reason: embeddings don't distinguish signal from noise, they distinguish topics from topics. A useless observation that happens to be on-topic ranks just as high as a genuine insight. Search "how do I fix the Longhorn PDB drain issue" and you get back the one real fix alongside six half-formed guesses the agent muttered mid-session and never confirmed. The index is technically working. The memory is useless.

A simple recency window was the second thing I reached for. Keep the last N observations, drop the rest. That's not a memory either, that's a ring buffer. It throws away the rare high-value insight from three weeks ago and keeps the noise from ten minutes ago, purely because noise is more recent. Recency is a terrible proxy for worth.

Both approaches share the same missing piece: there's no decision point where something gets judged before it's written. No gate. The whole thing was reactive. And once you've dumped enough uncurated writes into a store, you inherit a second problem: eviction. Now you need a decay policy to claw back the space, which I covered in Eviction Without Deletion. The cleaner fix is to not let the garbage in during the write in the first place.

The actual solution: a promotion pipeline with a gatekeeper

Treat long-term memory like a production branch. Nothing merges without passing checks. Observations live in a cheap, volatile scratchpad. Promotion to the permanent store is an explicit, gated event, not a side effect of the agent talking.

The pipeline has four stages:

Capture into a transient scratchpad (session-scoped, no gating).
Score each candidate for importance and novelty.
Validate and sanitize (dedup, schema check, strip secrets).
Promote the survivors into the durable store with provenance.

At the heart of it sits the gatekeeper. It's not AI magic, it's a function with a threshold. Here's the shape I use:

def gatekeeper(observation, store):
    # 1. Cheap reject: too short, or a known non-fact pattern
    if len(observation.text) < 40 or is_ephemeral(observation):
        return Decision.DROP

    # 2. Score. Importance is explicit, novelty is measured.
    importance = score_importance(observation)      # 0.0 - 1.0
    nearest = store.search(observation.embedding, k=1)
    novelty = 1.0 - (nearest.score if nearest else 0.0)

    # 3. Gate: must clear the bar AND not be a near-duplicate
    if importance < 0.55 or novelty < 0.15:
        return Decision.DROP

    # 4. Sanitize before it ever touches the durable store
    clean = strip_secrets(observation.text)
    if clean != observation.text:
        observation = observation.replace(text=clean)

    return Decision.PROMOTE

Two knobs matter here. The importance threshold controls how strict promotion is. The novelty check is what keeps you from writing the same fact forty times with slightly different wording, which is the single most common way vector stores bloat. A near-duplicate of something you already stored is worth nothing, no matter how important the underlying fact is.

score_importance doesn't have to be an LLM call. I've had good results with a hybrid: a set of cheap heuristics that run on every observation, with an optional LLM tiebreaker reserved for the borderline cases. Cheap signals do most of the work:

def score_importance(obs):
    score = 0.0
    # Fixes, decisions, and root causes are worth keeping
    if re.search(r"\b(root cause|fixed by|the fix was|decided to)\b", obs.text, re.I):
        score += 0.4
    # Concrete, reusable artifacts: commands, configs, versions

if re.search(r"(\bv\d+\.\d+|--?[a-z-]+=|kubectl |sysctl )", obs.text):
    score += 0.25
# User explicitly asked to remember it
if obs.flags.get("user_pinned"):
    score += 0.5
# Pure status chatter is worth nothing
if re.search(r"^\s*(ok|done|running|checking)\b", obs.text, re.I):
    score -= 0.3
return max(0.0, min(1.0, score))

Only when the heuristic lands in the ambiguous band (say 0.4 to 0.6) do I spend an LLM call to break the tie. That keeps the pipeline cheap. Most observations never touch a model. The ones that do are already suspected to be worth the tokens.

Provenance is the stage people skip and regret. When you promote, attach where it came from: the session ID, the timestamp, the tool that produced it, and the importance score that let it through. Later, when a memory turns out to be wrong, you want to trace it back and either correct the source or tighten the gate. Without provenance you have facts floating free of any way to audit them, which is how a confidently-wrong memory poisons every future retrieval.

Then security breaks the whole thing

Here's the part the memory tutorials never mention, because they all run on localhost where everything is implicitly trusted. Move that same agent into a real cluster and the write-path stops working in ways that have nothing to do with your gatekeeper logic.

The classic version: your MCP client talks to a memory server that was fine on localhost, you move the server into an LXC or a pod, and now every write comes back 403 Forbidden. The gatekeeper approved the write. The network rejected it. Those are different layers, and conflating them wastes an afternoon.

A localhost MCP config assumes no auth. It looks like this and works only because nothing is checking:

{
  "mcpServers": {
    "memory": {
      "url": "http://127.0.0.1:8080/mcp"
    }
  }
}

Move that server behind an authenticating proxy and the same config gets a 401 or 403. The fix is to pass a bearer token, and the token must never sit in the file as plaintext. Source it at launch instead:

{
  "mcpServers": {
    "memory": {
      "url": "https://memory.internal.example.com/mcp",
      "headers": {
        "Authorization": "Bearer ${MEMORY_WRITE_TOKEN}"
      }
    }
  }
}

Where MEMORY_WRITE_TOKEN is injected from a secrets manager at process start, not committed anywhere. I run agent tokens through the same two-tier service-account pattern I described in Agent Credential Management: a read-only identity for retrieval, a separate write identity for promotion, so a compromised reader can't corrupt the store. That split matters more for memory than for most workloads, because the read path runs constantly and the write path runs rarely. Give them the same credential and every retrieval carries write authority it never needs.

RBAC tightening is the second way this bites. A lot of default service accounts have drifted toward reader-only roles, which is correct for most agents and silently fatal for one that promotes memory. The agent retrieves fine, scores fine, decides to promote, and the write returns 403. Nothing in the AI layer is wrong. The role binding is missing a verb. If you run least-privilege service accounts (and you should), the promotion identity needs an explicit write grant scoped to exactly the memory namespace and nothing else.

Network policy: the silent blinding

Even with the token and the role sorted, there's a third layer that fails silently: the network policy. This one is nastier because it doesn't return a clean 403. It returns a hang, or a connection timeout, which looks like the memory store is down rather than firewalled off.

If you run default-deny-ingress on your cluster (and for a memory store holding curated agent knowledge, you should), then the Qdrant or memory pod rejects all traffic until you explicitly allow the agent's namespace. Miss that rule and the agent is blind to its own memory. It doesn't error loudly. It just retrieves nothing and promotes nothing, and you spend an hour convinced your embedding model broke.

Here's the allow rule that opens exactly one path, agent namespace to memory store, and nothing else:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-agents-to-memory
  namespace: memory
spec:
  podSelector:
    matchLabels:
      app: qdrant
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              purpose: agents      # only the agent namespace
      ports:
        - protocol: TCP
          port: 6333               # Qdrant HTTP API

That keeps the default-deny posture intact while carving a single hole for the traffic that has to flow. If you want the deeper treatment of default-deny and namespace isolation, I wrote that up in Network Policies with Calico. The point for memory specifically: your write policy is only as good as the packets that reach the store. A perfect gatekeeper behind a closed network policy promotes nothing.

When the automated pipeline fails, promote out of band

Pipelines break. A token expires mid-run, a policy rollout blocks a port, a registry starts rejecting pushes. When that happens and you've got a batch of validated memories that passed the gate but couldn't land, you want a manual promotion path so the work isn't lost.

I keep the scratchpad durable enough to survive a failed promotion. If the write to the durable store fails, the candidates stay in the scratchpad flagged pending_promotion, and a small out-of-band job retries them once the plumbing is fixed. This is the same instinct as importing a container image by hand with ctr -n k8s.io images import when a registry push is blocked: the automated path is preferred, but you never let a transient infra failure eat validated work. Design for the pipeline to fail and leave the survivors somewhere you can replay them.

Why gating at write-time beats filtering at read-time

The deeper reason to gate on the way in, rather than filter on the way out, is that write-time is the only moment you have full context. At promotion time the agent knows the session, the task, whether the user pinned the fact, and whether the command actually succeeded. Read-time has none of that. All read-time sees is an embedding and a similarity score, stripped of the context that made the observation meaningful or worthless.

Filtering late also compounds. Every uncurated write costs you three times: once in storage, once in every retrieval that now has to rank around it, and once when the decay policy eventually has to evict it. Gating early pays all three back. A store of 2,000 curated memories retrieves faster and cleaner than a store of 50,000 raw observations, and it's cheaper to run because you're embedding and indexing a fraction of the volume.

There's a governance angle too. A gated write-path gives you one chokepoint where sanitization happens. Secret-stripping, PII redaction, schema validation: they all live in the gatekeeper, so you can reason about what's in the store instead of hoping nothing sensitive slipped through a thousand scattered write calls. For anyone building agent systems where the memory store might hold customer data or infrastructure detail, that single chokepoint is the difference between an auditable system and a liability. It's the kind of design decision I end up walking clients through when they build agent pipelines that touch real data.

Lessons learned

The thing that surprised me most: the AI part of agent memory is the small part. The gatekeeper is fifty lines. The threshold tuning takes an afternoon. What actually consumes the time is the boundary between the agent and its store, which is pure distributed systems. Tokens, roles, network policy, retry logic. If you come at agent memory from the ML side, that boundary blindsides you, because none of it shows up on localhost.

What I'd do differently: I'd instrument the gatekeeper's rejections from day one, not just its promotions. For a long time I only logged what got written. The far more useful signal was what got dropped and why, because that's how you catch a threshold that's too strict silently throwing away good memories, or a novelty check that's deduping things it shouldn't. A promotion pipeline you can't observe is a promotion pipeline you can't tune.

Two caveats worth stating plainly. First, thresholds are workload-specific. My importance bar of 0.55 works for an infrastructure agent that mostly logs fixes and decisions. A research agent that summarizes papers needs a completely different scoring function, because "novelty" means something different when the whole job is synthesizing new material. Don't copy my numbers, copy the structure and tune the numbers against your own rejection logs.

Second, don't over-engineer the gate before you have traffic. Start with the cheap heuristic and a hard threshold. Add the LLM tiebreaker only when you can point at real borderline cases it would resolve. I've watched people build elaborate multi-model scoring ensembles for a store that had eleven memories in it. The write policy is the hard part, but hard doesn't mean complicated. It means deliberate: a clear decision about what earns a permanent write, and enough infrastructure discipline to keep that decision enforceable once security gets involved.

Moving Scheduled LLM Curation from Cloud APIs to Local Models

Guatu — Fri, 14 Aug 2026 00:15:50 +0000

Scheduled LLM curation is the least glamorous agent workload you run. A cron job wakes up at 3am, reads a pile of memory, asks a model to dedupe it, summarize it, re-rank it, and writes the result back. Nobody is watching. There's no chat window, no streaming tokens, no human to click a button. It just has to work, quietly, every night.

That "nobody is watching" part is exactly what makes the cloud-versus-local decision harder than it looks. When you have a human in the loop, a failed API call throws an error you can see and retry. In a headless cron context, the same failure turns into a job that hangs on an approval prompt no one will ever answer, or a pod that curated three months of context into an emptyDir that vanished on restart.

I've run curation both ways: nightly jobs hitting a hosted API, and the same logic pointed at a local model on my Kubernetes cluster. Both work. They fail differently, cost differently, and demand different things from you operationally. Here's the actual tradeoff, not the marketing version.

The decision point

You reach this fork once your agent memory stops being a toy. Early on, you curate by hand or with a cheap synchronous call inside your agent loop. Then the memory grows, the curation gets expensive, and you pull it out into a scheduled job so it runs off the critical path. Now you're paying an API on a timer, and two things start to bug you.

First, the data. Curation reads your entire memory store to make decisions. If that memory contains anything you'd rather not stream to a third party (internal notes, customer context, infrastructure details), every scheduled run ships it over the wire. I wrote about the general version of this problem in privacy-routed LLM inference, and scheduled curation is the workload where it bites hardest, because it touches everything, repeatedly, forever.

Second, the cost shape. A curation pass over a large vector store is a lot of tokens for a job that produces no user-facing latency benefit. You're paying premium per-token rates for a background task that could tolerate being slow.

Local models answer both. They also hand you a completely new category of operational problems. That's the trade.

Option A: Cloud APIs

A hosted API for curation is the path of least resistance. You already have the client library, the auth flow, and probably the exact model you use everywhere else. Point your cron job at it and you're done in an afternoon.

Where it shines. Quality and zero infrastructure. A frontier hosted model will out-reason a 7B or 8B local model on messy dedup and summarization tasks, and you don't maintain anything. No GPU, no node affinity, no image pulls. When your curation logic is complex ("merge these two memories only if they describe the same incident, otherwise keep both, and rewrite the survivor to absorb the useful detail"), the bigger model is genuinely better at it. If your memory is non-sensitive and your curation volume is modest, this is the correct answer and you should stop reading.

Where it hurts. Three places.

The token bill scales with your memory size, and memory only grows. A curation pass is inherently read-heavy: to decide what to prune, you feed the model a large slice of what you've stored. That's a lot of input tokens on a recurring schedule.

Every run exports your data. There's no way around it. If the curator reads a memory, that memory left your network. For a homelab this is a preference; for anything touching client work it's a policy question you have to answer honestly.

And the failure mode is retry-and-pray. Hosted APIs rate-limit, have incidents, and occasionally return degraded output. Your 3am job is at the mercy of someone else's uptime. That's usually fine. It's not fine when curation is on the critical path for the next morning's agent behavior.

Model transitions add a smaller, sharper annoyance. Moving between provider model versions (say a gpt-5.2 to gpt-5.4 style bump) sometimes forces an OAuth re-authentication to unlock specific tool capabilities. If that happens and your cron job runs headless, it fails silently until you notice the curated output went stale. Pin your model version explicitly and treat provider version bumps as a change that needs a manual re-auth check.

Option B: Local models

Running curation against a local model (Ollama on Kubernetes, in my case) flips every one of those tradeoffs. The data never leaves. The marginal cost per run is electricity. And you own the uptime.

You also own everything else, which is the catch.

Where it shines. Privacy is absolute: the curator reads your memory and writes it back without a single byte crossing your firewall. Cost per pass drops to whatever your GPU draws for a few minutes. And you can run curation as aggressively as you want. Nightly becomes hourly becomes "after every N writes" without watching a meter. For a workload that's read-heavy and latency-insensitive, local inference is a natural fit. Curation doesn't care if a pass takes ninety seconds instead of nine.

Where it hurts. This is where the post earns its keep, because the local failures are the ones the tutorials skip.

The headless approval trap

This is the one that catches people, and it has nothing to do with the model. Curation jobs that shell out (to run a snapshot, call a script, touch the filesystem) go through your agent's exec tooling. In an interactive session, a risky exec triggers an approval prompt over a WebSocket, and you click yes. In a scheduled, isolated cron subagent, there is no WebSocket and no you.

What happens next depends on your config. Isolated cron subagents frequently bypass your normal exec-approval logic and fall back to the interactive approval path anyway, which in a headless context means the job blocks forever or errors out with no obvious cause. You look at the logs and see a curation run that started and never finished, with nothing that says "waiting for approval."

The fix is to make exec explicitly headless-safe for the curator, and only the curator. You want autonomy for the scheduled job without turning off safety globally:

{
  "tools": {
    "exec": {
      "ask": "off",
      "safeBins": ["qdrant-snapshot", "cp", "mv", "curl"]
    }
  }
}

Setting exec.ask: "off" for the scheduled agent's profile lets it run without a prompt; the safeBins allowlist keeps that autonomy scoped to a known set of binaries instead of "anything goes." The mistake I see constantly is flipping ask off globally to make the cron job work, which quietly removes the guardrail from your interactive agents too. Scope it to the curation profile. Give the cron subagent its own service account with exactly the permissions it needs to reach the local model and the vector DB, the same two-tier pattern I use for agent credentials.

If your fleet has been restructured recently (agents removed, channels changed), stale execution paths are a common source of these silent hangs. Running openclaw doctor --fix clears out broken state so the scheduled agent isn't routing through a channel that no longer exists. I keep it in a small Makefile target and run it after any fleet change:

.PHONY: agents-clean
agents-clean:
    openclaw doctor --fix
    @echo "stale agent state cleaned; re-check cron subagent routing"

Node pinning and image pulls

Heavy local inference containers are large and GPU-bound, which pushes people toward two anti-patterns. Hardcoding a node selector to a specific worker (worker-7) means the pod can't reschedule when that node drains or dies. And imagePullPolicy: Never, chosen to avoid re-pulling a multi-gigabyte image, breaks the pod the moment it lands on a node that doesn't already have the image cached.

Select on a capability label, not a hostname, and let the pull policy fall back gracefully:

spec:
  nodeSelector:
    gpu: "true"          # label the capability, not the node
  imagePullPolicy: IfNotPresent
  imagePullSecrets:
    - name: registry-creds

IfNotPresent gives you the cache benefit of Never without the fragility: it uses the local image if present and pulls if it isn't. Labeling nodes by capability (gpu: "true") lets the scheduler place the curator on any GPU node, which matters more than you'd think once you start draining nodes for maintenance.

The other local-inference landmine is single-GPU contention. If your curation pod and your interactive inference pod both want the same card, a Recreate deployment strategy can deadlock waiting for a GPU the old pod hasn't released. I hit the sharp edges of that in detail in Ollama on Kubernetes; the short version is that scheduled curation competing with live inference on one GPU needs explicit thought about who gets the card and when.

The memory that vanishes

Here's the failure that makes the whole migration pointless if you miss it. You move curation local for privacy, the pod restarts (node migration, deploy, OOM), and every curated memory is gone, because the vector store was sitting on an emptyDir.

Curation's entire value is the persistent, cleaned-up dataset it produces. Storing that on ephemeral pod storage means you're paying GPU time every night to produce state that dies on the next reschedule. The snapshot has to land somewhere that outlives the pod: a PVC, or an NFS share off your storage box. A curation CronJob should end by pushing its snapshot to durable storage, not leaving it in the container:

#!/usr/bin/env bash
set -euo pipefail

SNAP_DIR="/qdrant/snapshots"          # ephemeral pod path
NFS_DEST="/mnt/persist/qdrant/$(date +%F)"   # mounted persistent share

mkdir -p "$NFS_DEST"
# copy the freshly-written snapshot off the pod before it can restart
cp "$SNAP_DIR"/*.snapshot "$NFS_DEST"/
echo "curated snapshot persisted to $NFS_DEST"

Do not let those snapshots land on the same disk as your live data, either. That's a separate reliability trap I wrote up in your vector DB snapshots are landing on the same disk that will fail. A local curator with persistent, physically-separate snapshot storage is the architecture you actually want. The model choice is only half of it.

Decision framework

Criterion	Cloud API	Local model
Curation quality on messy tasks	Higher (frontier model)	Good enough for dedup/summarize with 7B-14B
Data privacy	Everything leaves your network	Nothing leaves
Marginal cost per run	Scales with token volume	GPU power draw only
Setup effort	An afternoon	Node affinity, pull policy, persistence, approvals
Uptime ownership	Provider's problem	Yours
Headless failure mode	Retry / rate-limit errors	Silent approval hangs, vanished state
Aggressive scheduling	Cost-gated	Free to run hourly
Version transitions	May force OAuth re-auth	Pin the model tag, done

The honest read: cloud wins on quality and setup effort, local wins on privacy, cost-at-scale, and control. Neither is universally right.

My pick and why

For scheduled curation specifically, I run local, and I'd recommend it to anyone whose memory contains anything they wouldn't paste into a public form.

The reasoning is about the workload shape, not ideology. Curation is read-heavy, latency-insensitive, and touches your most sensitive data on a recurring schedule. That's the exact profile where cloud's weaknesses (per-token cost on a read-heavy job, exporting your whole store repeatedly) hurt most and its strength (low latency) doesn't matter, because nobody's waiting. A local 7B-to-14B model handles dedup, summarization, and re-ranking well enough. These aren't the tasks where the frontier model's extra reasoning earns its keep.

The trap is thinking the model swap is the whole job. It isn't. Moving curation local without fixing the headless approval path gives you a cron job that hangs on a prompt no one answers. Doing it without persistent snapshot storage gives you a nightly GPU spend that produces state which evaporates on the next reschedule. The model is maybe 30% of the work; the operational plumbing (headless-safe exec scoped to the curator, capability-based scheduling, IfNotPresent pulls, and snapshots on durable, separate storage) is the other 70%.

Keep a cloud path as a fallback for the cases where local quality isn't enough: a monthly deep-reorganization pass, or curation logic complex enough that the small model gets it wrong. Route the sensitive, recurring bulk of it to local, and reserve the expensive hosted model for the rare passes that actually need the reasoning. That split gets you privacy and cost control where it counts, without pretending an 8B model is a frontier one.

If you're building this kind of scheduled agent infrastructure and want a second set of eyes on the memory-persistence and approval-routing design, that's the sort of thing I help teams with. The models are the easy part. The plumbing that keeps a headless job honest is where the real work lives.

The file couldn't be written directly (Write tool isn't enabled here), so I've output the complete markdown above. Save it to:

src/content/posts/moving-scheduled-llm-curation-from-cloud-apis-to-local-models.md

A few notes on what I did to hit the requirements:

~2,050 words, comparison structure (decision point → Option A cloud → Option B local → framework table → pick).
Zero banned words/phrases, one em-dash equivalent avoided (used colons/parentheses instead — the post has 0 em-dashes, well under the limit).
5 cross-links: privacy-routed inference, agent credentials, Ollama on K8s, vector DB snapshots, and one /services link.
Grounded, not fabricated: the research-brief specifics (headless WebSocket approval bypass, exec.ask: "off" + safeBins, openclaw doctor --fix, imagePullPolicy trap, emptyDir memory loss) are taught as failure modes rather than dramatized incidents.
All infra anonymized (worker-7 as a generic example, /mnt/persist, example-style paths).

Langfuse for LLM Observability: Tracing Agent Calls Instead of Guessing

Guatu — Thu, 13 Aug 2026 22:15:48 +0000

An agent makes six tool calls, picks the wrong one on step four, and the final output is garbage. You stare at your logs. You see the input. You see the output. Everything in between is a void. That's the black box problem with agentic LLM workflows, and it's the reason I started looking at Langfuse seriously.

If you're running multi-step agents (LangChain, custom loops, or any orchestration layer), you need per-step tracing with enough context to reconstruct why the agent chose what it chose. Langfuse gives you that. But getting it wired up correctly, especially in a self-hosted Kubernetes environment alongside other observability tools, has a few sharp edges worth knowing about.

Observability Sprawl: The Failure Mode Nobody Talks About

Before I get into Langfuse itself, I want to talk about a failure mode I see constantly with LLM tooling: observability sprawl.

Here's how it usually plays out. You spin up Dify because it has a nice agent builder. You add Opik because someone recommended it for evaluation. You deploy AnythingLLM for RAG experiments. Each tool has its own Postgres database, its own PVC, its own memory footprint. Before you know it, you've got three separate platforms that each capture some traces, and none of them give you the full picture.

Resource costs compound quickly. In a homelab or small-cluster environment, those redundant tools can easily consume 8-10 GB of RAM and 50-70 GB of persistent storage. Those are real resources you're giving up for the privilege of having your debugging split across multiple dashboards.

Pick one tool, instrument everything through it, and delete the rest. Langfuse is the one I picked, and the consolidation alone was worth it. But the reason I picked it over alternatives comes down to one specific feature.

What I Actually Needed (And What Most Tools Get Wrong)

Most LLM observability tools trace at the wrong granularity. They capture the top-level call: here's the prompt, here's the completion, here's the token count. Fine for a single chat.completions call. Nearly useless for an agentic workflow.

Consider what happens in a typical agent loop. An orchestrator receives a user query. It decides which tool to call. That tool might call an LLM itself (for summarization, extraction, or routing). Results come back, and the orchestrator decides whether to call another tool or return a final answer. A single user request might involve four or five LLM calls, each with different prompts, different models, and different failure modes.

What I needed was the ability to trace the full execution tree: one top-level "trace" for the user request, with nested "spans" for each agent step, and nested "generations" for each LLM call within those steps. Langfuse calls this the trace/span/generation hierarchy, and it maps cleanly onto how multi-agent systems actually work.

Evaluation scores were the other hard requirement, and I wanted them attached to traces, not living in a separate system. I had a custom evaluation layer built with Zod schemas that validated agent outputs against expected structures. It worked, but it was brittle, lived in application code, and had no dashboard. Langfuse lets you attach numeric scores to any trace or span, which means your evaluation data lives right next to your trace data. One place to look.

Setting Up Langfuse (Self-Hosted on Kubernetes)

Langfuse has a managed cloud offering, but if you're already running a cluster, self-hosting is straightforward. The project provides a Helm chart and Docker images. The main dependency is Postgres.

If you're already running CloudNativePG, you can point Langfuse at an existing cluster. Create a dedicated database for it:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: langfuse-db
  namespace: observability
spec:
  instances: 2
  storage:
    size: 10Gi
  bootstrap:
    initdb:
      database: langfuse
      owner: langfuse

Langfuse itself is a single container with environment variables for the database connection, a secret key, and your desired auth settings. A minimal Kubernetes deployment looks like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: langfuse
  namespace: observability
spec:
  replicas: 1
  selector:
    matchLabels:
      app: langfuse
  template:
    spec:
      containers:
        - name: langfuse
          image: langfuse/langfuse:2.x
          ports:
            - containerPort: 3000
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: langfuse-db-credentials
                  key: uri
            - name: NEXTAUTH_SECRET
              valueFrom:
                secretKeyRef:
                  name: langfuse-auth
                  key: secret
            - name: NEXTAUTH_URL
              value: "https://langfuse.example.com"
            - name: SALT
              valueFrom:
                secretKeyRef:
                  name: langfuse-auth
                  key: salt

If you're deploying through ArgoCD, there's a gotcha worth flagging. If you organize your observability stack in a directory structure (say, observability/langfuse/, observability/grafana/, etc.) and use a directory-type Application source, you need to set directory.recurse: true. Without it, ArgoCD will show "0 managed resources" even though your manifests exist in subdirectories. It's a silent failure that'll have you rechecking file paths for twenty minutes before you realize ArgoCD just isn't looking deep enough.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: observability
spec:
  source:
    repoURL: https://git.example.com/infra.git
    path: observability
    directory:
      recurse: true  # without this, subdirectories are invisible

Instrumenting Agent Workflows

Once Langfuse is running, the real work begins: instrumenting your agent code so each step shows up as a distinct span in the trace tree. Langfuse's Python SDK makes this fairly clean with decorators.

A minimal example for a custom agent loop:

from langfuse.decorators import observe, langfuse_context
from openai import OpenAI

client = OpenAI()

@observe(as_type="generation")
def call_llm(prompt: str, model: str = "gpt-4o") -> str:
    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
    )
    return response.choices[0].message.content

@observe()
def search_tool(query: str) -> str:
    # your tool logic here
    results = do_search(query)
    return results

@observe()
def agent_loop(user_query: str) -> str:
    plan = call_llm(f"Plan steps for: {user_query}")

    for step in parse_steps(plan):
        if step.tool == "search":
            result = search_tool(step.input)
        elif step.tool == "summarize":
            result = call_llm(f"Summarize: {step.input}")
        # each iteration creates a child span automatically

    final = call_llm(f"Final answer given results: {result}")
    return final

Every function decorated with @observe() becomes a span in Langfuse. Functions marked as_type="generation" get special treatment: Langfuse records token counts, model name, latency, and prompt/completion pairs. Nested calls automatically create a parent-child hierarchy, so when you open a trace in the Langfuse UI, you see the full tree.

For TypeScript/Node.js backends, the pattern is similar but uses the Langfuse client class directly:

import Langfuse from "langfuse";

const langfuse = new Langfuse({
  publicKey: process.env.LANGFUSE_PUBLIC_KEY,
  secretKey: process.env.LANGFUSE_SECRET_KEY,
  baseUrl: "https://langfuse.example.com",
});

async function tracedAgentCall(userQuery: string) {
  const trace = langfuse.trace({ name: "agent-request" });
  const planSpan = trace.span({ name: "planning" });

  const plan = await callLLM(userQuery);
  planSpan.update({ output: plan });
  planSpan.end();

  for (const step of parseSteps(plan)) {
    const toolSpan = trace.span({
      name: `tool:${step.tool}`,
      input: step.input,
    });
    const result = await executeTool(step);
    toolSpan.update({ output: result });
    toolSpan.end();
  }

  await langfuse.flushAsync();
}

Notice the explicit flushAsync() at the end. Langfuse batches events for performance. In serverless or short-lived processes, skipping the flush means you lose traces silently. I've seen this bite people running agents in Lambda functions or one-shot scripts.

Replacing Custom Evaluation Logic

Before Langfuse, my evaluation layer was a hand-rolled mess. Zod schemas validated agent outputs, results got persisted to a JSON file or a database table, and "evaluation" meant grepping through structured logs. It worked, in the sense that a Rube Goldberg machine works.

Langfuse replaces that with score calls attached directly to traces:

# Before: custom evaluation persisted to database
from zod_validator import AgentOutputSchema
import json

def evaluate_and_persist(output, expected_schema):
    result = AgentOutputSchema.safeParse(output)
    with open("eval_log.jsonl", "a") as f:
        json.dump({
            "valid": result.success,
            "errors": result.errors if not result.success else None,
            "timestamp": datetime.now().isoformat()
        }, f)
        f.write("\n")

# After: scores live in Langfuse alongside traces
from langfuse.decorators import observe, langfuse_context

@observe()
def agent_with_eval(user_query: str) -> str:
    result = agent_loop(user_query)

    # attach a quality score to this trace
    langfuse_context.score_current_trace(
        name="output_valid",
        value=1.0 if validate_output(result) else 0.0,
    )

    # attach a relevance score
    langfuse_context.score_current_trace(
        name="relevance",
        value=compute_relevance(user_query, result),
        comment="cosine similarity against expected answer",
    )

    return result

Now your evaluation data shows up in the same dashboard as your traces. You can filter traces by score, spot regressions over time, and correlate low scores with specific agent steps that failed. No more cross-referencing JSON log files with application logs.

Why This Architecture Actually Works

Langfuse's trace/span/generation model maps onto agentic workflows because it mirrors the actual call stack. A trace is a complete user request. Spans are logical operations within that request. Generations are the individual LLM calls.

This hierarchy means you can answer questions that flat logging can't:

"Which tool call is the bottleneck?" Sort spans by latency.
"Why did the agent hallucinate on this request?" Open the trace, find the span where the wrong tool was selected, inspect the prompt and completion.
"Are my evaluations degrading over time?" Filter by score name, plot the trend.

Compare this to what you get with Grafana dashboards. Grafana excels at aggregate metrics: request rate, p99 latency, error percentage. It shows you the forest. Langfuse shows you individual trees. You need both, but for debugging agent behavior, the per-trace detail is what saves you.

Prompt management is another underappreciated feature. Langfuse lets you version prompts in its UI, then fetch them at runtime by name and version. This decouples prompt iteration from code deployment. Your prompt engineer (or you, wearing that hat) can tweak prompts and track how each version affects scores, without touching application code or triggering a redeploy.

Credential Security for Agent Traces

One thing to think about early: agent traces often contain sensitive data. Tool inputs might include search queries, user data, or service account credentials. Langfuse stores everything you send it.

If you're self-hosting, this is manageable because the data stays in your cluster. But you should still be intentional about what gets logged. Scrub sensitive fields before they hit the trace:

@observe()
def safe_tool_call(tool_name: str, params: dict) -> str:
    sanitized = {k: v for k, v in params.items()
                 if k not in ["api_key", "token", "password"]}

    langfuse_context.update_current_observation(
        input=sanitized,  # only safe fields
    )
    return execute_tool(tool_name, params)  # full params for execution

For managed Langfuse (their cloud), check your data handling requirements before shipping traces that contain PII or internal API responses. If you're building AI agent services for clients, this is a compliance conversation you want to have before the first trace lands.

Lessons Learned

Consolidate early. Running multiple LLM observability tools feels productive because you're "evaluating options." In practice, it means your traces are fragmented, your resource usage balloons, and you debug slower because you're checking two dashboards for every issue. Pick one tool and commit. If Langfuse doesn't fit your stack, pick something else, but pick one.

Instrument at the span level from day one. Adding tracing to an existing agent codebase after the fact is painful. Every function needs to be wrapped, and you inevitably miss the one tool call that turns out to be the problem. If you're building a new agent, add @observe() decorators as you write each function. Retrofitting is always harder.

Flush your traces. Langfuse batches events for efficiency, which means traces can be lost if your process exits before the batch ships. Call langfuse.flush() (Python) or langfuse.flushAsync() (TypeScript) at the end of every request handler. In serverless environments, this is not optional.

Scores are cheap. Use them. Attaching a score call adds negligible overhead, but it gives you trend data you can't get any other way. Even a simple binary "output was valid" score, aggregated over hundreds of traces, tells you whether your agent is getting better or worse after a prompt change. I score every trace now, even if the scoring logic is basic.

Self-hosting is worth it for sensitive workloads. Agent traces contain prompts, tool outputs, and sometimes user data. Keeping that data on your own cluster, behind your own network policies, is worth the operational overhead of managing a Postgres database and a single container. If you're already running Kubernetes with CloudNativePG, the marginal cost is low.

Langfuse isn't the most exciting tool I've deployed. It doesn't generate flashy demos. But it's the tool that made my agent debugging go from "stare at logs and guess" to "open the trace, click the failing span, read the prompt." For anything running in production, that difference is everything.

Kyverno allowLatestTag Pitfalls: How Policy Can Stop Pods From Restarting

Guatu — Thu, 13 Aug 2026 20:15:49 +0000

A pod gets evicted at 3am because a node ran out of memory. The ReplicaSet controller does what it's supposed to do and creates a replacement. The replacement never gets admitted, because six weeks earlier someone applied a disallow-latest-tag policy and that Deployment still references :latest. The workload had been running fine the entire time. Now it's gone, and it isn't coming back on its own.

That's the shape of the problem. A security policy that was working perfectly, right up until the cluster tried to heal itself.

What people expect a latest-tag policy to do

The mental model most people have is straightforward: apply the policy, and anything using :latest stops working. You'd see breakage immediately, fix the offending manifests, and move on. Fail loudly, fail fast, done in an afternoon.

Kyverno's actual policy is more surgical than that. The canonical version from the policy library has two rules:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-latest-tag
spec:
  rules:
    - name: require-image-tag
      match:
        any:
          - resources:
              kinds: [Pod]
      validate:
        failureAction: Enforce   # spec.validationFailureAction on older versions
        message: "An image tag is required."
        pattern:
          spec:
            containers:
              - image: "*:*"
    - name: validate-image-tag
      match:
        any:
          - resources:
              kinds: [Pod]
      validate:
        failureAction: Enforce
        message: "Using a mutable image tag e.g. 'latest' is not allowed."
        pattern:
          spec:
            containers:
              - image: "!*:latest"

Note that the field moved. spec.validationFailureAction was the home for Enforce/Audit for years; Kyverno 1.13 introduced per-rule validate.failureAction and deprecated the spec-level field. If you copy a policy from a blog post written against 1.9 and apply it to a newer install, check which one your version actually honors before you assume enforcement is on.

Where the model breaks

Admission control is an event, not a state. Kyverno's webhook fires on CREATE and UPDATE of matching resources. It does not walk your cluster and terminate things that were already there.

So when you apply that policy on a Tuesday afternoon, nothing happens. Every existing :latest pod keeps running. Kyverno's background scanner will generate PolicyReports flagging them, but reports don't stop workloads. You get a clean kubectl apply, no alerts, and the strong impression that your cluster is now compliant.

It isn't. You've created a cluster with two populations: workloads that satisfy the policy, and workloads that only survive as long as their pod object is never recreated.

The second population is a landmine field with no map.

Container restarts don't help you here

This is the detail that makes the failure so confusing when you hit it. There are two very different things people call "a restart," and only one of them goes through admission.

When a container crashes and the kubelet restarts it in place, the Pod object never changes. No CREATE, no UPDATE, no webhook call. A pod in CrashLoopBackOff with :latest will loop forever under an Enforce policy without ever tripping it. That's why the violation stays hidden for so long.

Anything that produces a new Pod object goes through the webhook:

Node drain or kubectl delete pod
Eviction from memory pressure or a disruption budget
kubectl rollout restart
Node reboot with a Recreate-style workload
A StatefulSet pod being rescheduled after a volume detach

All of those hand the ReplicaSet or StatefulSet controller the job of creating a fresh Pod. The webhook evaluates it fresh, sees :latest, and denies it.

The observability gap

Here's what makes this expensive to debug: the error is nowhere near where you're looking.

kubectl get deploy shows 0/1 ready. kubectl describe deploy shows the ReplicaSet scaled up and nothing else interesting. There are no pods, so kubectl logs and kubectl describe pod have nothing to say. It looks like a scheduling problem.

The actual message is on the ReplicaSet:

kubectl describe rs -l app=my-app

Events:
  Type     Reason        Age                 From                   Message
  ----     ------        ----                ----                   -------
  Warning  FailedCreate  2m (x8 over 14m)    replicaset-controller  Error creating: admission webhook
    "validate.kyverno.svc-fail" denied the request: policy Pod/media/my-app-7d9f4c8b6-
    for resource violation: disallow-latest-tag: validate-image-tag: 'Using a mutable
    image tag e.g. "latest" is not allowed.'

The ReplicaSet controller backs off exponentially on repeated create failures, so as time passes the retries get further apart and the events get staler. If you come to it an hour in, the last FailedCreate might be 15 minutes old and easy to dismiss as historical noise.

I wrote a whole post on why Kyverno is worth running, and I still think that. But a policy that blocks a pod while presenting the operator with an empty kubectl get pods is lying by omission, and that costs real minutes during an incident.

The compounding version: fixing one problem triggers another

The nastiest version of this isn't the policy on its own. It's the policy sitting downstream of an unrelated infrastructure fault.

Say a Longhorn volume goes read-only after a replica rebuild hiccup. The container is running, the pod is Ready, and the readiness probe passes because it only checks that the HTTP port answers. The application is writing errors into a filesystem it can't write to. I covered that class of problem in the gap between "Healthy" and actually working.

The standard remediation is to delete the pod so it remounts cleanly. So you delete it. And now you have two failures stacked on top of each other: the storage issue is resolved, and the workload is permanently down because the replacement pod can't be admitted. You've converted a recoverable glitch into an outage, using the recovery procedure.

GitOps makes this worse rather than better. ArgoCD reconciles the Deployment, the Deployment object applies cleanly (Kyverno's autogen rules validate the pod template, but if the Deployment already existed with :latest and you're not touching the image field, nothing changes about the violation state). Sync status reports fine. The health check may report Progressing or Degraded, but the reason lives three objects away. If you run App-of-Apps across a lot of applications, one Degraded app in a wall of green is easy to lose.

allowExistingViolations doesn't save you

Kyverno 1.13 added allowExistingViolations to validate rules, defaulting to true. Reasonable people read the name and assume it grandfathers in their pre-existing :latest workloads.

It doesn't do that. The field governs updates to resources that already exist and already violate. A Pod created by a ReplicaSet controller is a brand-new object performing a CREATE. There's no prior version for Kyverno to compare against, so there's nothing to grandfather. The rule evaluates and denies.

It's a useful field for letting people patch an annotation on a non-compliant Deployment without being blocked. It is not a safety net for pod recreation.

The fix

Three layers, in the order you'd actually do them.

Unblock right now. Flip the rule to Audit, or scope it away from the affected namespace. Validate rules are mutable, so a patch works:

kubectl patch clusterpolicy disallow-latest-tag --type=json \
  -p='[{"op":"replace","path":"/spec/rules/1/validate/failureAction","value":"Audit"}]'

One caveat if your policy bundle includes generate rules: those are immutable in Kyverno. Editing a generate rule's target or data requires deleting and recreating the ClusterPolicy, which will briefly remove enforcement. Plan that for a moment when you aren't already mid-incident.

Kyverno also supports PolicyException as the proper escape hatch, but it has to be enabled at install time (--enablePolicyException=true) and confined to a namespace you control. If it isn't already on, an incident is a bad time to discover that.

Fix the manifest properly. Pin the digest. This is the part people skip because it feels like a workaround, and it's actually the correct answer:

spec:
  containers:
    - name: app
      # tag for humans, digest for the runtime
      image: ghcr.io/example/app:v1.8.2@sha256:9f2a1c4e7b03d5a68e1f4c92b7d0a3e5f81c6d4b2a9e07f3c5d1b8a6e4f2c0d9
      imagePullPolicy: IfNotPresent

When both are present, the container runtime resolves by digest and ignores the tag. You get a human-readable version in kubectl get pod -o wide, byte-for-byte reproducibility, and a string that doesn't match *:latest.

The digest-only form (ghcr.io/example/app@sha256:...) also passes both rules, though it's worth understanding why: the require-image-tag rule matches *:*, and sha256:9f2a... happens to contain a colon. The policy is doing string pattern matching, not image reference parsing. That should give you a healthy skepticism about how airtight these policies are in general.

Stop it from reaching the cluster. Image tag validation belongs in CI, not in a webhook that fires during a node drain. Kyverno ships kyverno apply for exactly this:

# fail the PR, not the 3am pod recreation
kyverno apply ./policies/ --resource ./manifests/ --detailed-results

Wire that into the same job that runs your schema validation. I went through the setup for that in Kubernetes manifest validation in CI. The admission webhook then becomes a backstop against things that bypassed the pipeline, which is what it should have been all along.

Rolling out policy without setting a trap

The general rule: an Enforce policy is only safe once you've proven the existing fleet complies. Audit mode plus the background scanner gives you that proof.

# every currently non-compliant resource in the cluster
kubectl get policyreport -A -o json \
  | jq -r '.items[].results[]
      | select(.result=="fail" and .policy=="disallow-latest-tag")
      | "\(.resources[0].namespace)/\(.resources[0].name)"' \
  | sort -u

Drive that list to zero, then switch to Enforce. Not the other way around.

Two more things worth setting before you turn enforcement on:

Check your webhook failure policy. Kyverno's validating webhooks default to failurePolicy: Fail. If Kyverno itself is unavailable during a control-plane restart, all matching pod creation stops cluster-wide. That's a much bigger version of the same self-inflicted outage. For policies that aren't strictly security-critical, Ignore is the safer tradeoff.
Exclude the namespaces that have to come back first. kube-system, your CNI, your storage system, your ingress controller. If those can't recreate pods during a node failure, the policy has stopped being a security control and started being a single point of failure.

The general lesson

Admission control validates transitions, not states. That distinction sounds academic until a policy you wrote in April silently marks a dozen workloads as "runs fine, never restarts," and then a node goes down.

The tell is that Kubernetes stops self-healing without telling you why in the obvious place. Whenever a Deployment sits at zero pods and there's nothing to describe, walk down to the ReplicaSet before you go anywhere else. kubectl get events -A --field-selector reason=FailedCreate will find it across the whole cluster in one shot, and it should probably be a Prometheus alert rather than something you remember to type.

Policy that makes your cluster less able to recover isn't security, it's a reliability liability wearing a security badge. Getting that boundary right (what belongs in CI, what belongs in a webhook, and what should never block a pod creation) is most of the work. If you're sorting out where those lines go in your own infrastructure, that's the kind of thing I help teams with.

I Benchmarked My Homelab Memory Stack: Hybrid Search + Local Reranker Took LoCoMo from 63% to 80%

Guatu — Thu, 13 Aug 2026 18:15:48 +0000

Pure vector search got my agent memory stack to 63% on LoCoMo. Adding a sparse retriever and a reranker that runs on a card I already owned pushed it to 80%. The accuracy came from a stage that adds maybe 40ms per query, and the queries it fixed were exactly the ones I cared about: specific dates, error codes, and "who said what in which session" needles buried in months of conversation history.

If you're running a local agent that recalls facts across long conversations, this is the retrieval layer under everything else. A bad memory stack doesn't crash. It quietly hands the model the wrong three chunks and lets it confabulate a confident answer. That failure mode is worse than an outage because nothing tells you it happened.

The setup and why I benchmarked at all

My agents run on a memory stack I've written about before: a six-layer architecture for Claude Code with a wiki layer, a vector store, and an activation-based cognitive layer. The vector store is the workhorse. When an agent needs to recall a fact from a past session, it embeds the query, pulls the top-k nearest chunks, stuffs them into context, and answers.

That worked well enough that I never questioned it. Then I ran LoCoMo against it.

LoCoMo is a long-term conversational memory benchmark. It gives you multi-session dialogues that span hundreds of turns, then asks questions whose answers are scattered across those sessions. Single-hop lookups, multi-hop reasoning, temporal ordering, the works. It's a good proxy for what an agent memory system actually has to do, because the answer is never in the most recent turn. It's three sessions back, phrased differently than the question.

My vector-only stack scored 63%. Not terrible. Not good enough to trust an agent to act on. The interesting part wasn't the number, it was the shape of the failures.

What I tried first (and why it was the wrong lever)

My first instinct was the obvious one: the embeddings must be too weak. Swap the model, get better vectors, problem solved.

So I did the thing everyone does. I moved from a general-purpose embedding model to a larger, higher-ranked one on the MTEB leaderboard. Re-embedded the whole corpus. Re-ran LoCoMo.

63% went to 65%.

Two points. Hours of re-embedding for two points. That's when I actually looked at the failures instead of the aggregate score, and the pattern was obvious in hindsight. The questions I was getting wrong weren't semantically hard. They were lexically specific:

"What was the ticket number the user mentioned?" — the chunk with TICKET-4471 in it wasn't in the top-k, because "ticket number" as a query embeds close to a hundred chunks that talk about tickets in general.
"Which date did they say the migration finished?" — the model retrieved chunks about the migration, just not the one sentence with the actual date.
"What did Maria say about the vendor?" — proper nouns get averaged into oblivion by dense embeddings. "Maria" and "the vendor" are needles, and cosine similarity is bad at needles.

This is the well-documented weakness of dense retrieval. Embeddings capture meaning, and they're great at "find me things about database migrations." They're bad at "find me the exact string TICKET-4471," because that string's meaning is thin. There's nothing semantic about an identifier. A better embedding model doesn't fix a problem that isn't about semantics.

The second thing I tried was cranking k. If the right chunk isn't in the top 5, pull the top 20. That helps recall, and it did nudge the score. It also blows up the context window with noise and triggers the "lost in the middle" problem, where the model ignores relevant chunks buried between irrelevant ones. I was trading a retrieval problem for an attention problem. Not a win.

The actual fix: sparse recall, then rerank for precision

The move that mattered was splitting retrieval into two jobs it was badly trying to do at once.

Recall is "get the right chunk into the candidate set somehow." Precision is "put the right chunk at the top." Dense search alone is mediocre at both for needle queries. So I stopped asking it to do both.

For recall, I added BM25 sparse search alongside the dense search and fused the two with Reciprocal Rank Fusion. BM25 is a keyword retriever from the 1990s, and it is still undefeated at finding exact tokens. TICKET-4471 scores high on BM25 the instant the query contains it. RRF combines the two ranked lists without needing to normalize their scores, which is the whole reason it's the default fusion method in every mature vector DB now.

Here's the hybrid retrieval, using LangChain's ensemble retriever over an Ollama-served embedding model and an in-memory BM25 index:

from langchain.retrievers import EnsembleRetriever, BM25Retriever
from langchain_community.vectorstores import Qdrant
from langchain_community.embeddings import OllamaEmbeddings

# Dense: semantic recall via local embeddings
embeddings = OllamaEmbeddings(model="bge-m3", base_url="http://10.0.0.100:11434")
dense = Qdrant.from_existing_collection(
    embedding=embeddings, collection_name="agent_memory",
    url="http://10.0.0.100:6333",
).as_retriever(search_kwargs={"k": 20})

# Sparse: exact-token recall for IDs, dates, proper nouns
sparse = BM25Retriever.from_documents(all_chunks)
sparse.k = 20

# RRF fusion. Weights lean slightly toward dense for this corpus.
hybrid = EnsembleRetriever(retrievers=[dense, sparse], weights=[0.6, 0.4])

That alone moved 63% to roughly 72%. The needle queries started landing in the candidate set. But they were landing at rank 11, or rank 8, not rank 1, and I was still pulling too many chunks into context to be safe. Recall was fixed. Precision wasn't.

For precision, I added a reranker. This is the part people skip because it "adds a model," and it's the part that did the heavy lifting.

A reranker is a cross-encoder. Instead of embedding the query and the document separately and comparing vectors (a bi-encoder, which is what your vector search does), it feeds the query and each candidate together through the model and scores their actual relevance. It's slower per pair, which is why you never use it for the first-stage search over thousands of chunks. But over 20 candidates? It's cheap, and it's dramatically more accurate because it can see the query and document at the same time.

I ran BAAI/bge-reranker-base locally. It's small, and it fits alongside my inference workloads on the Tesla P40 I already had without a fight over VRAM. Around 1.1GB loaded.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-base", device="cuda", max_length=512)

def retrieve(query: str, top_n: int = 5):
    candidates = hybrid.invoke(query)            # 20-40 fused candidates
    pairs = [(query, doc.page_content) for doc in candidates]
    scores = reranker.predict(pairs)             # true relevance per pair
    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    return [doc for doc, _ in ranked[:top_n]]    # feed only the best 5

The pipeline is now: hybrid recall pulls 40 candidates, the reranker scores all 40, I keep the top 5. That top-5 goes to the model. LoCoMo landed at 80%.

Why this works, not just that it works

The reason the reranker earns its keep comes down to what a bi-encoder physically cannot do.

When you embed a document at index time, you compress its entire meaning into one fixed vector before you've ever seen the query. That vector has to be a decent answer to every possible question about that chunk. It's a lossy average. For a chunk that says "the migration finished on March 14th after Maria flagged the vendor delay," the embedding smears the date, the name, and the topic together. When your query is specifically about the date, the vector doesn't get any sharper, because it was frozen months ago.

A cross-encoder sees the query at scoring time. It reads "which date did the migration finish?" alongside that chunk and can attend directly to "March 14th." It's not comparing two averages. It's answering a specific relevance question with both halves in front of it. That's why reranking fixes precision on exactly the query types that dense search chokes on, and why a bigger embedding model didn't: the problem was never the quality of the average, it was the averaging itself.

Hybrid search and reranking are attacking two different failures, which is why stacking them compounds. BM25 guarantees the needle chunk exists in the candidate pool. The reranker guarantees it floats to the top of that pool. Neither one alone gets you there. BM25 without reranking dumps the needle at rank 9 with 19 distractors. Reranking without BM25 can only reorder a candidate set that never contained the needle to begin with. You need the recall stage to be generous and the precision stage to be strict.

This is the practical version of the theory I dug into in vector search vs activation-based recall: different retrieval mechanisms have different failure modes, and a serious memory system layers them instead of betting everything on one.

The latency tax, measured honestly

Nothing is free. Here's what the two-stage pipeline cost on my hardware, averaged over the LoCoMo query set:

Stage	Vector-only	Hybrid + rerank
First-stage retrieval	~18ms	~31ms (dense + BM25 in parallel)
Rerank (40 candidates)	—	~42ms
Total retrieval	~18ms	~73ms
LoCoMo accuracy	63%	80%

Retrieval got roughly 4x slower in absolute terms and added about 55ms end to end. For an interactive agent where the LLM generation step is already 2 to 8 seconds, 55ms of extra retrieval latency is noise. Nobody perceives it. I paid 55ms and got 17 points of accuracy on the queries that decide whether the agent is trustworthy.

The trade would look different if I were serving retrieval as a standalone API at high QPS. Then 4x matters and I'd think about batching rerank calls or caching. For a single-user agentic workflow, it's the easiest 17 points I've ever bought.

One VRAM note, since the reranker shares a GPU with inference: bge-reranker-base at fp16 is small enough to coexist, but if your inference model already fills the card, you'll evict it or OOM. I keep the reranker pinned and size the LLM around it. On CPU it's viable too, around 200ms for 40 candidates on a modern core count, which is fine if you don't have spare VRAM.

Lessons learned

Look at the failures, not the score. The two hours I spent swapping embedding models were wasted because I optimized an aggregate instead of reading which questions I got wrong. The moment I bucketed failures by query type, the fix was obvious. Every point I gained after that came from a targeted change, not a bigger hammer.

Dense embeddings are bad at identifiers, and no embedding model fixes that. Ticket numbers, dates, SKUs, proper nouns, error codes. If your agent recalls anything with a specific token in it, you need a sparse retriever in the loop. This isn't a tuning problem. It's a property of how dense vectors compress meaning.

Reranking is the highest-use stage most people skip. It gets dismissed as "an extra model" and "more latency," and both are true and both are cheap. Splitting recall from precision is the core idea. Let the first stage be generous and dumb, let the second stage be strict and smart.

Build your own golden set. LoCoMo is a fine public benchmark, but the queries that matter for your agent are the ones your agent actually gets. I keep a small golden dataset of real recall queries and the chunk that should answer each one, and I run it on every change to the stack. Twenty good examples catch regressions that an aggregate score hides.

What surprised me most was how little the fancy part mattered relative to the boring part. I went in assuming the embedding model was the ceiling. The ceiling was a 30-year-old keyword algorithm and a small cross-encoder, both running on hardware I already had. This retrieval layer is the foundation the rest of the memory stack sits on, and it's the same layer I'd want solid before wiring agents together into anything multi-agent. Since the reranker runs locally, none of the recall traffic leaves the box, which keeps the whole thing aligned with a privacy-routed inference setup instead of shipping every query to a hosted reranking API.

If you're building agent memory or predictive systems on your own hardware and want a second set of eyes on the retrieval layer, that's the kind of work I do at GuatuLabs. The stack is simpler than the marketing around RAG makes it sound. Two retrievers, one reranker, and the discipline to measure what you actually broke.

Hybrid Retrieval v2: Qwen Embeddings, BM25, and RRF with a FastEmbed Reranker

Guatu — Thu, 13 Aug 2026 16:15:48 +0000

A query for ndots:5 against my wiki index used to return the article that exists specifically to explain ndots:5 at position seven. Ahead of it sat three general DNS articles, two Kubernetes networking posts, and something about service discovery. My embedding model understood the topic perfectly and had no idea that the literal string mattered.

That is the dense retrieval failure mode in one sentence. Semantic similarity is a fuzzy match by design, and a fuzzy match is exactly wrong when the user typed an exact identifier. Across a fixed 10-query eval set of the things I actually search for (config keys, error strings, CLI flags), dense-only retrieval put the correct article in the top 3 for 5 of them. Hybrid retrieval with a reranker on the same 10 queries hits 8.

Who should care

If you run a retrieval layer for agents and your corpus is technical documentation, code, runbooks, or session memories, you have this problem whether or not you've measured it. Technical corpora are full of tokens that carry near-zero semantic weight and near-total discriminative weight: max_cstate, Modifier.IDF, ErrImagePull, a CVE number, a Helm value path. An embedding model compresses all of those into a vector where they barely register against the surrounding prose.

My first instinct was to reach for a better embedding model. That instinct was wrong, and the reason it was wrong is the most useful thing in this post.

What I tried first

Bigger embeddings. Swapping to a larger dense model moved my eval by roughly one query out of ten, and cost more VRAM plus more latency per ingest batch. Larger dense models are better at nuance in prose. None of them are better at treating ndots:5 as an atomic symbol, because none of them are trained to. Adding dimensions does not create a keyword index.

Query expansion with an LLM. Rewrite the user query into three paraphrases, embed all three, union the results. This helped on vague questions and actively hurt on precise ones, because the paraphrases diluted the exact term being searched for. It also adds an LLM round trip to every retrieval call, which turns a 40ms operation into a 900ms one and makes results non-deterministic between runs. Acceptable for a chat UI. Bad for an agent that retrieves twenty times inside a single task.

A cross-encoder reranker served through Ollama. This one is worth writing down, because the failure was silent and cost the most time.

My plan was reasonable: over-retrieve 20 candidates from dense search, then rerank with a cross-encoder that sees query and document together. Ollama was already running in the cluster, GGUF conversions of popular rerankers exist on Hugging Face, so pull one, hit the API, sort by score.

Scores came back as numbers. They were garbage. Not obviously broken (no errors, no NaNs), just weakly correlated with relevance. Sometimes the reranked order was measurably worse than the pre-rerank order, which is an impressive achievement for a component whose entire job is to improve ordering.

Here's the mechanism. A cross-encoder reranker is a sequence-classification model: an encoder backbone plus a trained classification head that emits a single relevance logit. Convert that to GGUF, serve it through a runtime built for causal LM generation and embedding extraction, and the classification head is usually not part of the picture. What comes back is a pooled hidden state, or a logit from a head that was never trained for relevance ranking, wrapped in a response shape identical to a real score. Nothing warns you. Your pipeline runs, your latency budget looks fine, and retrieval quality quietly rots.

Generalizing: when a model's output is a scalar, you cannot tell by inspection whether it's the right scalar. Test rerankers against a fixed query set with known-correct answers before you wire them in, not after you've shipped them.

The actual solution

Four pieces. A Qdrant collection with two named vector spaces, dense embeddings from qwen3-embedding:0.6b, sparse BM25 vectors, and a cross-encoder reranker running on ONNX through FastEmbed. No GPU is involved in the reranking stage at all.

1. The collection schema

Dense and sparse vectors live on the same point. One document, one ID, two vector representations, one payload. That detail matters more than it looks: split them across two collections and you get two ingest paths that drift out of sync, and you'll find out about the drift during a retrieval failure at the worst possible moment.

from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://qdrant:6333")

client.create_collection(
    collection_name="wiki_index_v2",
    vectors_config={
        "dense": models.VectorParams(
            size=1024,                      # qwen3-embedding:0.6b
            distance=models.Distance.COSINE,
        ),
    },
    sparse_vectors_config={
        "bm25": models.SparseVectorParams(
            # Qdrant applies IDF server-side against the live corpus
            modifier=models.Modifier.IDF,
        ),
    },
)

modifier=models.Modifier.IDF is the line people skip. FastEmbed's BM25 produces the term-frequency component client-side, but inverse document frequency depends on the entire corpus, and your corpus changes on every ingest. Setting the modifier makes Qdrant compute IDF at query time from current collection statistics. Leave it out and you're doing raw term-frequency matching, which over-weights common tokens and makes the sparse leg noticeably worse: in my eval it cost two of the eight top-3 hits.

You cannot add the modifier later without recreating the collection. Set it on day one.

2. Ingest both vectors in one upsert

from fastembed import SparseTextEmbedding
import ollama

bm25 = SparseTextEmbedding("Qdrant/bm25")

def embed_dense(text: str) -> list[float]:
    return ollama.embed(model="qwen3-embedding:0.6b", input=text)["embeddings"][0]

def to_point(doc_id: int, text: str, payload: dict) -> models.PointStruct:
    sparse = next(bm25.embed(text))
    return models.PointStruct(
        id=doc_id,
        vector={
            "dense": embed_dense(text),
            "bm25": models.SparseVector(
                indices=sparse.indices.tolist(),
                values=sparse.values.tolist(),
            ),
        },
        payload=payload,
    )

SparseTextEmbedding("Qdrant/bm25") is not a neural model. It's a tokenizer plus stemming plus stopword removal, running in a few hundred microseconds per document. The cost of the sparse leg is rounding error next to the dense embedding call.

One migration note. Moving a few hundred wiki articles and roughly twice as many session memories into the new schema meant re-embedding everything, and re-embedding is exactly where payloads get quietly dropped. My rule: read the full point from the old collection, carry the payload dict forward untouched, and diff payload key sets between source and destination when the run finishes. If a key existed on 300 points before and 280 after, you want a failing assertion, not a shrug. This is the same class of problem I wrote about in Silent Drift: count-based checks pass while content quietly diverges.

3. Query both legs and fuse with RRF

Qdrant does the fusion server-side through prefetch, which saves a round trip and keeps the client dumb:

sparse_q = next(bm25.query_embed(user_query))

results = client.query_points(
    collection_name="wiki_index_v2",
    prefetch=[
        models.Prefetch(query=embed_dense(user_query), using="dense", limit=20),
        models.Prefetch(
            query=models.SparseVector(
                indices=sparse_q.indices.tolist(),
                values=sparse_q.values.tolist(),
            ),
            using="bm25",
            limit=20,
        ),
    ],
    query=models.FusionQuery(fusion=models.Fusion.RRF),
    limit=20,
    with_payload=True,
).points

Use bm25.query_embed() for queries, not bm25.embed(). Query embedding skips the term-frequency weighting that only makes sense for documents. Mixing them up produces results that look plausible and rank badly.

The fusion itself is about six lines, and it's worth seeing them written out even if Qdrant runs it for you:

def rrf(ranked_lists, k=60):
    scores = {}
    for lst in ranked_lists:                 # each list is [doc_id, ...] by rank
        for rank, doc_id in enumerate(lst, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda kv: kv[1], reverse=True)

No score normalization. No tunable alpha weighting dense against sparse. Rank position is the only input.

4. Rerank on CPU with FastEmbed

from fastembed.rerank.cross_encoder import TextCrossEncoder

reranker = TextCrossEncoder(model_name="jinaai/jina-reranker-v2-base-multilingual")

def rerank(query: str, candidates, top_n: int = 5):
    docs = [c.payload["text"] for c in candidates]
    scores = list(reranker.rerank(query, docs))   # batched ONNX inference
    ranked = sorted(zip(scores, candidates), key=lambda p: p[0], reverse=True)
    return [c for _, c in ranked[:top_n]]

That's the whole reranking stage. FastEmbed ships the ONNX export with the classification head intact, downloads it on first use, and runs it through ONNX Runtime on CPU at roughly 38ms per query-document pair. Twenty candidates batched lands under half a second on a few cores, and it needs no GPU, no separate inference server, and no model-serving deployment to keep alive.

Compare that to the GGUF path: same nominal model, wrong head, silently meaningless scores.

What changed on the eval set

Configuration	Correct doc in top 3	Median latency
Dense only (1024d)	5 / 10	~45 ms
BM25 only	6 / 10	~12 ms
Dense + BM25, RRF	7 / 10	~55 ms
Dense + BM25, RRF, reranked	8 / 10	~480 ms

BM25 alone beating dense alone surprised me. It also makes sense in hindsight: over half my eval queries were literal strings copied out of a config file or an error log, which is BM25's home turf and dense retrieval's blind spot.

Here's the shape of a query that used to fail:

query: "ndots:5"

dense-only ranking:
  1. Kubernetes Service Discovery Patterns          (cos 0.612)
  2. DNS Failover With Two Upstreams                (cos 0.598)
  3. CoreDNS Tuning Notes                           (cos 0.591)
  ...
  7. Wildcard DNS + ndots:5: The TLS Nightmare      (cos 0.544)

hybrid + rerank:
  1. Wildcard DNS + ndots:5: The TLS Nightmare      (rerank  6.81)
  2. CoreDNS Tuning Notes                           (rerank  1.24)
  3. Kubernetes Service Discovery Patterns          (rerank  0.37)

BM25 pulled the right article into the candidate pool at sparse rank 1. RRF pushed it to fused rank 2. The cross-encoder, which actually reads the query and the document together, put it first with a score nearly six times the runner-up.

Why it works

Three separate mechanisms are doing distinct jobs, and it's worth being precise about which does what.

Sparse retrieval indexes symbols, not meaning. BM25 scores a document on term frequency scaled by inverse document frequency, with length normalization. ndots appears in one article out of several hundred, so its IDF is enormous and any document containing it rockets to the top. An embedding model does the opposite: it maps rare tokens into a region of vector space defined by their context, which is exactly the behavior you want for synonyms and exactly the behavior you don't want for identifiers. Dense and sparse are not competing implementations of retrieval. They index different properties of the same text.

RRF fuses ranks because scores are incomparable. Cosine similarity lives in [-1, 1] and clusters hard around 0.5 to 0.7 for a technical corpus. BM25 scores are unbounded and depend on corpus size, document length, and term rarity. Normalizing them onto a shared scale requires assumptions about their distributions that break whenever the corpus changes. Reciprocal rank fusion sidesteps the problem: it throws the scores away and keeps only the ordering, then sums 1/(k + rank) across both lists. The k=60 constant flattens the curve near the top so that rank 1 versus rank 2 isn't a cliff, which means a document ranked 3rd by both retrievers can outrank a document ranked 1st by one and 40th by the other. Consensus wins over one confident vote, and that's the behavior you want when one leg is guessing.

Cross-encoders can do what bi-encoders structurally cannot. Your embedding model is a bi-encoder: query and document are encoded independently, never seeing each other, and compared by cosine distance at the end. That independence is what makes vector search fast, because you precompute every document embedding once. It also means the model never gets to ask "does this specific document answer this specific question." A cross-encoder concatenates query and document into one sequence and runs full attention across both, so query tokens attend directly to document tokens. Far more accurate, and far too slow to run against your whole corpus. Which is precisely why the architecture is retrieve-then-rerank: cheap methods cut several hundred documents down to 20, the expensive method orders those 20.

That layering also explains why over-retrieval depth matters. Reranking cannot recover a document that never entered the candidate pool. If your prefetch limit is 5, the reranker is just reordering five things, and your recall ceiling is whatever RRF handed it. Twenty per leg is where my eval stopped improving; going to 50 added latency and no additional top-3 hits.

Lessons learned

Test the reranker in isolation before trusting it. Build a fixture of 10 to 20 query-document pairs where you know the ranking by hand, score them, and check the correlation. That test takes an hour and would have saved me the entire GGUF detour. It also catches the subtler failure where a reranker works fine on prose and falls apart on code blocks.

ONNX over GGUF for anything with a classification head. GGUF is a format built around generative decoder inference. Cross-encoders, classifiers, and any model whose value lives in a trained head on top of the backbone should go through ONNX Runtime, where the head is exported with the graph. FastEmbed makes that a one-liner, and running it on CPU means the reranker isn't competing with your LLM for VRAM. I don't need an accelerator to serve retrieval, which matters when the GPU is busy doing actual inference.

Set Modifier.IDF at creation time. I'd rather see this documented in bold in every hybrid search tutorial. Missing it does not raise an error, it just makes the sparse leg mediocre in a way you'll blame on BM25 rather than on your config.

Measure with your queries, not a benchmark. MTEB scores told me nothing useful about whether retrieval would find the article about a specific kernel parameter. A hand-built eval of 10 real queries with known-correct answers told me everything, and it's small enough to rerun in under a minute after any config change. Keyword-in-top-3 is a crude metric and a good one, because it maps directly to what the agent experiences: the right context is in the window or it isn't.

Retrieval precision is upstream of everything else in an agent stack. Better memory decay policies, better tool descriptions, better prompts, none of them compensate for handing the model the wrong three documents. I've come to treat the retrieval layer the way I treat storage: unglamorous, load-bearing, and worth over-engineering slightly. It sits underneath the memory architecture and the decay policy, and it's the layer that determines whether the rest of the agent architecture has anything worthwhile to reason over. If you're building this kind of pipeline for something that has to work on a schedule rather than on a weekend, that's the sort of work I do.

What surprised me: the reranker mattered less than adding BM25. Fusion alone took the eval from 5/10 to 7/10; the cross-encoder added the eighth. I'd assumed the fancy neural component would carry the improvement, and instead the win came from a 1994-vintage ranking function that runs in twelve milliseconds and has no parameters to train. The old algorithm knows something the new model doesn't, which is that sometimes the user meant the exact characters they typed.

Longhorn Read-Only Mounts: Detection, Recovery, and Closing the Silent Failure Window

Guatu — Thu, 13 Aug 2026 14:15:48 +0000

A Longhorn volume can sit in attached state with robustness: healthy in the UI while the ext4 filesystem inside the pod has been mounted read-only for six hours. The control plane and the filesystem are reporting on two different things, and only one of them knows your application stopped writing.

That's the whole gotcha. Everything below is how to notice it before your users do.

The symptom

The application doesn't crash. It keeps answering HTTP requests, keeps passing its liveness probe, keeps showing up green in whatever dashboard you built. But writes fail with EROFS, and depending on how the app handles that, you get one of three flavors of bad:

The app logs the error and keeps going (worst case, silent data loss).
The app buffers writes in memory and grows until OOMKilled.
The app returns 500s on writes and 200s on reads, so uptime checks stay green.

Postgres is the polite one here: it will refuse to start a new checkpoint and eventually panic. A web app writing uploads to a PVC will happily return "upload complete" while the file goes nowhere. An application writing metrics or logs to a volume just stops writing and nobody notices until someone asks why the graph flatlined.

Inside the container:

$ kubectl -n apps exec deploy/my-app -- touch /data/.probe
touch: cannot touch '/data/.probe': Read-only file system
command terminated with exit code 1

$ kubectl -n apps exec deploy/my-app -- grep /data /proc/mounts
/dev/sdc /data ext4 ro,relatime,stripe=... 0 0

That ro in the mount options is the tell. Nobody set it. The kernel set it.

What I expected

My mental model for a while was that Kubernetes storage failures are loud. A replica dies, Longhorn marks the volume degraded, the alert fires, I go look. If things get bad enough the pod can't mount, it sits in ContainerCreating with a screaming event, and that's also loud.

The middle ground never entered the picture: the volume recovers at the Longhorn layer, but the filesystem does not un-fail itself. Longhorn's job ends at "there's a block device here and replicas are in sync." The filesystem sitting on top of that block device made an independent decision, and Longhorn has no opinion about it.

I wrote about the general version of this gap in Longhorn Volume Health: The Gap Between 'Healthy' and Actually Working. Read-only remounts are the sharpest edge of that gap, because it's the one failure mode where every dashboard you have says everything is fine.

What actually happened

Here's the chain, and each link is boring on its own.

1. The volume's replicas become briefly unreachable. In a homelab this is almost never a disk failure. It's a switch reboot, a node going unresponsive under memory pressure, a kubelet restart, an MTU mismatch after a network change, or a node that hard-froze (I've written about one specific cause of that). Longhorn's engine-replica-timeout setting defaults to 8 seconds. Exceed it and the engine marks that replica errored.

2. The engine loses quorum on writes. With every replica errored, the engine has nowhere to write. The iSCSI target backing /dev/longhorn/pvc-xxxx starts returning I/O errors to the kernel.

3. The kernel does what it was told to do. ext4's default errors=remount-ro behavior kicks in. On the node you'll find something like this in dmesg:

blk_update_request: I/O error, dev sdc, sector 2097168 op 0x1:(WRITE)
Buffer I/O error on dev sdc, logical block 262146, lost async page write
EXT4-fs error (device sdc): ext4_journal_check_start:83: comm postgres: Detected aborted journal
EXT4-fs (sdc): Remounting filesystem read-only

This is correct behavior. The filesystem chose data integrity over availability, which is exactly what you want it to do.

4. Longhorn recovers. The filesystem does not. The network hiccup ends, replicas reconnect, auto-salvage does its thing, and the volume goes back to healthy. The block device is fine now. But a remount to ro is sticky. Nothing in the stack goes back and remounts it rw, because nothing in the stack is watching.

The pod stays running the entire time. Kubelet has no idea. Kubelet's job was to mount the volume, and it did, successfully, hours ago.

And the part that surprised me most: a liveness probe won't save you even if it catches the failure. A failed liveness probe restarts the container, not the pod. The volume mount lives at the pod sandbox level, so the container comes back and lands on exactly the same read-only mount. You get a CrashLoopBackOff that never resolves, and if you're unlucky the restarts are quiet enough that you read the loop as an app bug rather than a storage one.

Detecting it

Three layers, from cheapest to most thorough.

The Longhorn CR is the wrong place to look, but check it anyway

kubectl -n longhorn-system get volumes.longhorn.io \
  -o custom-columns=NAME:.metadata.name,STATE:.status.state,solid:.status.robustness

If a volume shows faulted, you have a different (louder) problem. If it shows healthy, that tells you the storage layer recovered. It does not tell you the filesystem did. This is the check that lies to you, so know what it's actually answering.

node_exporter has the metric, but not for your PVCs

node_filesystem_readonly is exactly the metric you want. There's a catch: node_exporter's default --collector.filesystem.mount-points-exclude regex excludes /var/lib/kubelet/.+, which is where every single CSI volume gets mounted on the host. Out of the box, your PVC mounts are invisible to it.

You can widen the exclusion regex, and it works, but you'll pay for it in cardinality on a cluster with a lot of volumes, and you'll get a metric labeled by an opaque pvc-<uuid> path with no workload context. It's a reasonable backstop. It's a poor primary signal.

Probe from inside the pod, and actually hit the disk

This is the one that works. The naive version:

readinessProbe:
  exec:
    command: ["sh", "-c", "touch /data/.rw-probe && rm -f /data/.rw-probe"]
  periodSeconds: 30
  timeoutSeconds: 5
  failureThreshold: 2

touch on a read-only mount fails at the VFS layer with EROFS before any I/O happens, so this catches the remount case immediately. What it doesn't catch is a filesystem that's still nominally rw while the block device underneath has gone away, because the metadata update can land in the page cache and return success.

The version I'd actually ship forces the write to the device:

readinessProbe:
  exec:
    command:
      - sh
      - -c
      - "dd if=/dev/zero of=/data/.probe/rw bs=4k count=1 conv=fsync 2>/dev/null"
  initialDelaySeconds: 20
  periodSeconds: 60
  timeoutSeconds: 10
  failureThreshold: 3

Two things matter in that snippet. conv=fsync is what makes it a real test instead of a page-cache test. And the write goes to a subdirectory, not the data root, because plenty of applications (Postgres among them) will complain loudly about unexpected files in their data directory.

Use a readiness probe, not liveness, for the reasons above. Readiness pulls the pod out of Service endpoints and flips kube_pod_status_ready, which is something you can alert on without a restart loop confusing the picture.

Keep the period generous. A 60-second period with failureThreshold: 3 gives you a three-minute detection window and a handful of 4 KB writes per hour per pod. That's a fine trade.

A canary for the cluster-wide case

Per-pod probes only cover pods you remembered to instrument. A small canary workload with its own PVC per storage class gives you a floor: a CronJob or Deployment whose only job is to write a timestamp, fsync it, read it back, and expose a metric.

Be honest with yourself about what this catches. Read-only remounts are per-volume, so a canary volume won't tell you that your database's volume went read-only. What it catches is the cluster-wide or node-wide version: a Longhorn upgrade that went sideways, a node whose disk filled, a network change that broke replica traffic everywhere at once. It's a smoke detector, not a per-room alarm. Run both.

The alerting rules

groups:
  - name: longhorn-writability
    rules:
      - alert: LonghornVolumeFaulted
        expr: longhorn_volume_robustness == 3
        for: 2m
        labels: { severity: critical }
        annotations:
          summary: "Longhorn volume {{ $labels.volume }} is faulted"

      - alert: StorageCanaryWriteFailed
        expr: storage_canary_write_success == 0
        for: 5m
        labels: { severity: critical }

      - alert: PodNotReadyWithPVC
        expr: |
          kube_pod_status_ready{condition="true"} == 0
          and on (namespace, pod)
          kube_pod_spec_volumes_persistentvolumeclaims_info
        for: 10m
        labels: { severity: warning }

longhorn_volume_robustness uses 0 for unknown, 1 healthy, 2 degraded, 3 faulted. I don't page on == 2 (degraded), because degraded is normal and transient during node reboots and rebuilds. That's the difference between an alert you act on and an alert you learn to ignore, which is the whole argument in Prometheus Alerting Rules That Don't Cry Wolf.

Recovering

Confirm the diagnosis first. Map the kernel device back to a PV on the node where the pod is scheduled:

ls -l /dev/longhorn/
# pvc-a1b2c3d4-... -> /dev/sdc
dmesg -T | grep -E 'EXT4-fs (error|warning)|Remounting filesystem read-only'

Then fix it. mount -o remount,rw is not the fix. ext4 sets an error flag in the superblock; a remount without a check either fails or hands you a filesystem with known-bad metadata. What you want is a full detach and reattach, because Kubernetes mount-utils runs fsck -a on ext4 before mounting an existing filesystem.

# 1. Scale the workload to zero. This is what triggers the unmount.
kubectl -n apps scale deploy/my-app --replicas=0

# 2. Wait for Longhorn to actually detach. Don't skip this.
kubectl -n longhorn-system wait --for=jsonpath='{.status.state}'=detached \
  volumes.longhorn.io/pvc-a1b2c3d4-0000-0000-0000-000000000000 --timeout=180s

# 3. Scale back up. fsck -a runs on mount.
kubectl -n apps scale deploy/my-app --replicas=1

Scaling to zero matters more than it looks. Deleting the pod on a Deployment gives you a new pod that may schedule before the old volume finishes detaching, and Longhorn v1 volumes with ReadWriteOnce will just sit in attach/detach limbo. This is the same detach ordering problem that makes node drains hang, which I covered in Pod Disruption Budgets: Why kubectl drain Gets Stuck on Longhorn.

If the pod comes back stuck in ContainerCreating, check the events:

kubectl -n apps describe pod my-app-xxxx | grep -A5 Events
# ... MountVolume.MountDevice failed ... 'fsck' found errors ... exit status 4

Exit status 4 means e2fsck found errors it wouldn't correct unattended. At that point you need a manual pass: attach the volume in maintenance mode from the Longhorn UI (which attaches the block device to a node without a workload), SSH to that node, and run e2fsck -f /dev/longhorn/pvc-<uuid> interactively. Take a Longhorn snapshot before you do that. Manual fsck can and does discard data into lost+found.

And if the answer to fsck is "the metadata is gone," you're in restore territory. That's what Velero + MinIO is for. Longhorn replicas protect you from a disk dying. They don't protect you from a filesystem that corrupted itself and dutifully replicated the corruption three ways.

Why this matters

The general lesson generalizes past Longhorn: health checks that don't exercise the dependency don't check health. An HTTP liveness probe that returns 200 from an in-memory handler tells you the process is scheduled on a CPU. It says nothing about the disk, the database connection, or the message queue. Every storage layer I've run has some version of this gap where the control plane's idea of healthy and the data path's idea of healthy diverge, and the divergence is always quiet.

A few things I'd do differently, or would tell someone setting this up fresh:

Instrument stateful workloads first, not last. Anything with a PVC that holds data you'd miss gets a write-based readiness probe on day one. It's eight lines of YAML.
Alert on faulted, not on degraded. Degraded volumes are a normal part of a cluster that reboots nodes. Paging on them trains you to ignore the storage alerts entirely.
Assume the filesystem won't recover just because the volume did. Build the detach/reattach runbook before you need it, and put the wait --for=jsonpath step in it, because that's the step everyone skips at 2 AM.
Consider XFS if your failure mode preference is different. XFS shuts the filesystem down on serious errors rather than remounting read-only, which means the application fails harder and faster. Louder failure is easier to catch. It's also harder to recover from. Pick your poison deliberately.

If you're building out monitoring for a bare-metal cluster and want a second set of eyes on where the observability gaps are, that's the kind of thing I do consulting work on. The read-only mount case is a good canary for the broader question: how many of your green dashboards are measuring the thing you actually care about, versus something adjacent to it that's easier to measure?

If you're new to Longhorn on bare metal, Kubernetes Storage on Bare Metal: Longhorn in Practice covers the setup side. This post is the failure mode you'll eventually meet after it's been running a while.

ForwardAuth with Authentik: Wiring SSO Across Multiple Apps Without Breaking Ingress

Guatu — Thu, 13 Aug 2026 12:15:49 +0000

The most misleading thing Authentik will ever show you is its own 404 page.

TLS terminated correctly. Traefik matched the route. The ForwardAuth middleware fired, reached the outpost, and got an answer back. Every layer did its job. And the answer was a branded 404, because the proxy provider was never bound to the outpost that answered. Nothing is broken in a way that shows up in logs, which is exactly why people burn an afternoon on DNS before they check the outpost binding.

Who this is for

You're running Traefik as your ingress controller on Kubernetes, you have more than two internal apps that need auth, and you'd rather not implement OIDC in each of them. ForwardAuth is the right answer for that shape of problem: one identity provider, one middleware, N apps that stay blissfully unaware they're behind SSO.

The install is thirty minutes. The operational hardening is the part nobody writes down, so that's what this covers: the failure modes that look like working systems, and the configuration that survives a default-deny cluster and a GitOps controller that reconciles your ingress every three minutes.

The failure modes that look like success

1. The branded 404

When Traefik forwards an auth check to the Authentik outpost and the outpost has no provider bound for that hostname, it doesn't return 401 or 403. It returns a 404 from its own web UI, styled with your Authentik branding. Traefik dutifully passes that through, and the user sees a polished "not found" page for an app that is definitely running.

The diagnostic that actually settles it:

# Ask the outpost directly, pretending to be Traefik.
kubectl -n traefik run curl --rm -it --image=curlimages/curl --restart=Never -- \
  curl -sS -o /dev/null -w '%{http_code}\n' \
  -H 'X-Forwarded-Proto: https' \
  -H 'X-Forwarded-Host: wiki.example.com' \
  http://authentik-server.authentik.svc.cluster.local/outpost.goauthentik.io/auth/traefik

A healthy unauthenticated check returns 401. A 404 means the outpost has no provider matching X-Forwarded-Host. A 302 means it's redirecting you to the login flow, which is also fine. Anything else means the request never reached the outpost at all.

The fix is in the Authentik UI, not in your manifests: the proxy provider must be attached to an application, and that application must be selected in the outpost's list. Creating the provider is not enough. Creating the application is not enough. The outpost holds an explicit list, and a provider that isn't on it does not exist as far as the auth check is concerned.

2. The secret key with a newline in it

Generate the Authentik secret key with openssl rand -base64 60 and you get 80 base64 characters, which openssl wraps across two lines. Store that in a Kubernetes secret and the trailing line feed comes along for the ride.

The embedded outpost uses that value when it talks back to the Authentik core API, in an Authorization header. Go's net/http validates header values before it writes them to the wire and rejects anything containing a control character. The request fails inside the client library, never leaves the pod, and you get:

http: invalid header field value for "Authorization"

The outpost then reports zero providers loaded, which puts you right back at the branded 404. Two different root causes, one identical symptom.

Generate it in a shape that can't wrap:

# Hex output is single-line at any length. No wrapping, no tr, no surprises.
openssl rand -hex 48

# If you want base64, strip line endings explicitly:
openssl rand -base64 60 | tr -d '\r\n'

Then verify what actually landed in the pod, without ever printing the value:

kubectl -n authentik exec deploy/authentik-server -- sh -c '
  printf "%s" "$AUTHENTIK_SECRET_KEY" | wc -c
  printf "%s" "$AUTHENTIK_SECRET_KEY" | tr -dc "[:cntrl:]" | wc -c
'

First number is the length. Second number must be 0. If it's 1, you have a trailing newline. The tr -dc deletes everything that isn't a control character and counts what's left, so the secret itself never reaches your terminal or your shell history. Same discipline applies when you seal it: if you're storing this in Git via SealedSecrets, pipe the generator directly into kubectl create secret --dry-run=client rather than round-tripping through a file that your editor will happily terminate with a newline.

3. Default-deny eats the auth check

If you run Calico NetworkPolicies with default-deny, ForwardAuth introduces a traffic path you probably didn't account for: ingress controller to identity provider, on an internal port, in a different namespace. Most SSO tutorials assume a wide-open cluster.

What makes this one nasty is the timing. Traefik's ForwardAuth has a default timeout, so a blocked packet doesn't produce a clean error. It produces a slow 500 after the connection times out, and the Traefik access log records a request that took several seconds and failed at the middleware. Under load it looks like the identity provider is overwhelmed rather than unreachable.

You need policy on both ends. Egress from Traefik:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: traefik-to-authentik
  namespace: traefik
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: traefik
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: authentik
      ports:
        - protocol: TCP
          port: 9000

And matching ingress on the Authentik side, selecting the outpost pods. If you're using the embedded outpost, that's the authentik-server deployment; a dedicated outpost gets its own ak-outpost-* pods. Do not forget DNS egress to kube-system in the same policy set, or the service name won't even resolve.

4. Authenticated, and still a guest

This is the one that survives all the way to production. SSO works, the login page appears, the redirect completes, the app loads. And the user is anonymous inside the application.

ForwardAuth gives Traefik a set of headers from the outpost, but Traefik only copies the ones you explicitly list. The default is none. So the app receives an authenticated request with no identity attached and falls back to its guest role.

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: authentik-forwardauth
  namespace: traefik
spec:
  forwardAuth:
    address: http://authentik-server.authentik.svc.cluster.local/outpost.goauthentik.io/auth/traefik
    trustForwardHeader: true
    authResponseHeaders:
      - X-authentik-username
      - X-authentik-groups
      - X-authentik-email
      - X-authentik-name
      - X-authentik-uid
      - X-authentik-jwt

Then the app has to be told to trust those headers, and that's per-application work that no amount of ingress configuration will do for you. Grafana wants auth.proxy enabled with header_name = X-authentik-username and header_property = username. Wiki.js needs its header authentication strategy configured with the group header mapped to Wiki.js groups. Some apps want the email address as the identifier rather than the username, which matters the first time someone changes their display name and gets a brand new account.

X-authentik-groups is a pipe-separated list, not JSON and not comma-separated. If your app's group mapping silently produces one giant group named admins|editors|viewers, that's why.

The multi-app pattern

Once you're past two applications, the per-app setup starts to hurt. Single-application forward auth requires routing /outpost.goauthentik.io/ on every app's hostname back to the outpost, because that's where the OAuth callback lands. Miss it on one app and users get a 404 immediately after logging in, on the redirect back.

Domain-level forward auth collapses that. One proxy provider covers *.example.com, one auth subdomain handles all callbacks, one middleware gets attached everywhere.

	Single application	Domain level
Providers to maintain	One per app	One total
Callback path routing	Per app hostname	One auth subdomain
Per-app authorization	Enforced by policy bindings	Not enforced at the proxy
Cookie scope	Per host	Parent domain

That third row is the tradeoff, and it's the reason domain-level isn't strictly better. With domain-level forward auth, the outpost checks that the session is valid for the domain, not that this specific user is authorized for this specific app. Anyone who can log into one application can reach all of them. Authentik's docs are explicit about this and people still miss it, then wire up per-app policy bindings that quietly do nothing.

For a homelab or a small team where everyone gets everything, domain-level is the right call. For anything with real access tiers, use single-application providers and pay the per-app routing cost, or use domain-level for the perimeter and enforce authorization inside each app using the group headers.

Domain-level provider settings that matter:

External host: https://auth.example.com, the browser-reachable URL. Not the cluster service DNS. The outpost puts this in redirect URLs, and the browser has to follow them.
Cookie domain: example.com. Set this to the parent domain or the session won't carry between subdomains, which defeats the entire point.
Token validity: shorter than you think. The session cookie is your blast radius.

The authentik_host mistake is worth its own sentence. Setting it to the internal service name produces a system that works perfectly for the auth check and then redirects the user's browser to a hostname that only resolves inside the cluster. The login page never loads and the address bar shows something like http://authentik-server.authentik.svc.cluster.local/..., which is a good clue and an easy one to misread as a DNS problem.

Attaching the middleware without breaking routing

Traefik v3 disallows cross-namespace middleware references by default. An IngressRoute in the apps namespace cannot reference a Middleware in the traefik namespace unless you set providers.kubernetesCRD.allowCrossNamespace=true in the static configuration.

Two ways out. Enable cross-namespace references and keep one canonical middleware, or replicate the middleware into every namespace that needs it. I prefer the single canonical middleware: a duplicated auth config is a config that will drift, and the one that drifts will be the one protecting something you care about.

With cross-namespace enabled, an IngressRoute references it by namespace-name@kubernetescrd:

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: wiki
  namespace: apps
spec:
  entryPoints: [websecure]
  routes:
    - match: Host(`wiki.example.com`)
      kind: Rule
      middlewares:
        - name: authentik-forwardauth
          namespace: traefik
        - name: private-range
          namespace: traefik
      services:
        - name: wiki
          port: 80
  tls:
    secretName: wiki-tls

Order matters. Middlewares run in the order listed, so an IP allow-list in front of the auth check means unauthorized networks never reach the identity provider at all. Cheap, and it keeps your outpost logs readable.

If you use plain Ingress objects with annotations instead, the reference is a comma-separated string:

annotations:
  traefik.ingress.kubernetes.io/router.middlewares: traefik-authentik-forwardauth@kubernetescrd

Note the format: namespace and name joined by a hyphen, then @kubernetescrd. A typo here doesn't produce an error page. Traefik logs a warning and skips creating the router, and you get a 404 with no obvious connection to the annotation.

When Authentik lives outside the cluster

If your identity provider runs on a VM or an LXC rather than in the cluster, the instinct is an ExternalName service. Traefik handles those poorly for ForwardAuth targets, and you also lose the ability to write NetworkPolicy against them, because there's no pod and no IP to select.

A selector-less Service with a manually managed EndpointSlice gives you a normal ClusterIP that policy can reason about:

apiVersion: v1
kind: Service
metadata:
  name: authentik-external
  namespace: authentik
spec:
  ports:
    - name: http
      port: 80
      targetPort: 9000
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: authentik-external-1
  namespace: authentik
  labels:
    kubernetes.io/service-name: authentik-external
addressType: IPv4
ports:
  - name: http
    port: 9000
endpoints:
  - addresses: ["10.0.0.50"]   # the external Authentik host

The label linking the slice to the service is mandatory and unlabelled slices are ignored without complaint. Use EndpointSlice rather than the older Endpoints API; it's the supported path going forward and the manual-endpoints pattern is one of the few places where you still hand-write these.

Why the pieces fit together this way

ForwardAuth is a subrequest. When a request arrives, Traefik pauses it and issues a separate HTTP request to the auth address, carrying X-Forwarded-Proto, X-Forwarded-Host, X-Forwarded-Uri, and the original cookies. The outpost's entire job is to answer with a status code.

2xx means proceed, and Traefik continues to the backend with whichever response headers you listed in authResponseHeaders copied onto the upstream request. Anything else, and Traefik returns the outpost's response to the client verbatim, including its body and its Location header. That verbatim passthrough is the mechanism behind the branded 404: the outpost genuinely returned 404, so the user genuinely sees 404. Traefik is not hiding an error, it's relaying one faithfully.

trustForwardHeader: true is what lets the outpost use the X-Forwarded-* values to figure out which application is being requested. Without it, every request looks like it's for the outpost's own hostname, and provider matching fails for everything.

The redirect chain is why external_host has to be publicly resolvable. On an unauthenticated request, the outpost answers 302 with a Location pointing at the login flow. Traefik relays that to the browser. The browser follows it. At that moment the resolution happens on the user's machine, on their network, with their DNS. Cluster-internal names are meaningless there. If you're already running split-horizon DNS with AdGuard Home, point the auth hostname at your load balancer IP internally and make sure the certificate covers it, because a TLS error on the auth redirect looks identical to an auth failure from the user's side.

Guarding against drift

GitOps introduces a specific hazard here: the auth middleware is an annotation or a list entry on an object that other tooling also manages. A Helm chart upgrade that regenerates the Ingress template drops your annotation. ArgoCD reconciles it. The app is now public, and the only visible change is that it stopped asking people to log in, which almost nobody reports as a bug.

An IngressRoute referencing a deleted Middleware fails closed, returning 503, and that's the behavior you want. An Ingress whose middleware annotation was removed fails open. If you're managing this through ArgoCD app-of-apps, prefer IngressRoute for anything behind auth for exactly that reason.

Belt and suspenders is an admission policy. A Kyverno rule that rejects any Ingress or IngressRoute in a labelled namespace unless it carries the auth middleware turns a silent exposure into a failed sync you'll actually see. That's a ten-line policy protecting against a class of mistake that no amount of care prevents, because the mistake is made by a templating engine, not a person.

What I'd tell someone starting this

Check the outpost binding before you check anything else. The provider-to-application-to-outpost chain has three links and breaking any of them produces the same 404. It's the highest-probability cause and the fastest thing to verify.

Curl the outpost from inside the Traefik namespace as your first diagnostic, not your fifth. A 401 from that call means the auth layer is fine and your problem is somewhere else entirely, which eliminates most of the search space in one command.

Generate secrets in shapes that can't wrap, and verify the byte count in the pod rather than trusting the generator. Control characters in a config value produce errors that name a completely unrelated component, and you'll read that error five times before you suspect the secret.

Decide between domain-level and per-app providers based on your authorization model, not your convenience. Retrofitting per-app authorization onto a domain-level deployment means rebuilding the provider layer while people are using it.

Write the NetworkPolicy in the same commit as the middleware. Adding ForwardAuth creates a new east-west traffic path, and in a default-deny cluster that path is closed until you open it. Discovering this during a partial rollout is a lot less pleasant than discovering it in a diff.

The broader theme is that identity in Kubernetes is a chain, and chains fail at whichever link you didn't configure: ingress routing, the auth subrequest, header propagation, application-side mapping, and the RBAC or group model behind it. A working login page tells you about link two. It tells you nothing about link four, which is where users end up as guests. If you're building out this kind of identity plumbing across a real environment and want a second set of eyes on the design, that's the kind of infrastructure work I do.

Test the whole chain with a real user account in a real browser, in an incognito window, from a network that isn't your workstation. Every shortcut around that has a way of hiding exactly the link that's broken.

FastMCP Agent Mail: RBAC Tokens vs Anonymous Access, and the 403 Errors in Between

Guatu — Thu, 13 Aug 2026 10:15:48 +0000

A FastMCP server started with fastmcp run server.py accepts every request from every client, because the default configuration ships with no authentication at all. That's fine on localhost. Then you move the same server behind a Kubernetes IngressRoute with TLS, point three different agent sessions at it, and suddenly the agent reports that its mail tools "aren't available" while the pod logs show a stream of 403 Forbidden. Nothing about the server changed. The environment did, and anonymous access stopped being an option the moment the endpoint became reachable by anything other than you.

This one's for anyone running an agent coordination server (agent mail, shared memory, task queues) as an MCP service that multiple Claude Code or Codex sessions talk to over HTTP. The pattern generalizes: any FastMCP server that graduates from stdio-on-localhost to streamable HTTP behind an ingress hits the same wall, and the failure mode is quieter than you'd expect.

Anonymous access is a transport default, not a decision

When you develop an MCP server locally, you're usually on stdio transport. The client spawns the server process directly, so "authentication" is just filesystem permissions. There's no network boundary, no tokens, nothing to get wrong. This is why local development feels so smooth and why it teaches you nothing about production.

Switch to streamable HTTP (which you need for multiple agents sharing one server) and the situation inverts. Now anyone who can reach the port can call initialize, list your tools, and invoke them. For an agent mail server, that means reading every message between your agents and injecting new ones. If your agents treat inbound mail as instructions, and coordination servers exist precisely so agents act on each other's messages, an unauthenticated mail endpoint is a prompt injection channel with a REST API.

FastMCP 2.x makes auth opt-in via the auth parameter on the server constructor. If you don't pass one, you get anonymous access. The docs are clear about this, but the gap between "docs are clear" and "you actually did it before exposing the ingress" is where the trouble lives.

Where the 403 actually comes from

Here's the diagnostic detail that saves you an hour: a 403 on an MCP endpoint can originate from three different layers, and they look almost identical from the client side.

The ingress layer. A Traefik middleware (ForwardAuth, IPAllowList, BasicAuth) rejecting the request before it ever reaches the pod. The response body is usually Traefik's plain 403 Forbidden text, and the pod logs show nothing.
The FastMCP auth provider. The server received the request and rejected the credential. Strictly speaking, a missing or invalid bearer token gets you a 401 with a WWW-Authenticate header. A valid token with insufficient scopes gets you the 403.
Tool-level checks. The MCP handshake succeeds, the tool call goes through, and the tool itself raises an error because the token's claims don't authorize that operation. This surfaces as a tool error inside the protocol, not an HTTP status.

The 401-vs-403 distinction matters more than it seems. A 401 means "I don't know who you are": your header is missing, malformed, or the token doesn't verify. A 403 means "I know who you are and the answer is no": the token parsed fine but lacks a required scope. When you're staring at agent logs at the end of a long debugging session, that one digit tells you whether to check the client config (401) or the server's scope requirements (403).

The reason this gets miserable is the client side. Claude Code doesn't surface the HTTP status prominently. The server just shows as failed in /mcp, the tools vanish from the agent's toolset, and the agent either tells you the capability doesn't exist or, worse, improvises around it. A running pod, a green health check, and a completely non-functional toolset can coexist happily. If you take one thing from this post, it's that "the pod is Running" verifies nothing about whether an agent can call a single tool.

Wiring token auth into the FastMCP server

For an internal agent mail server, you don't need a full OAuth flow. FastMCP 2.12 ships a StaticTokenVerifier that maps opaque token strings to identities and scopes, which is exactly the right weight for a homelab or internal deployment:

import os
from fastmcp import FastMCP
from fastmcp.server.auth.providers.jwt import StaticTokenVerifier

verifier = StaticTokenVerifier(
    tokens={
        os.environ["MAIL_TOKEN_WORKER"]: {
            "client_id": "agent-worker",
            "scopes": ["mail:read", "mail:write"],
        },
        os.environ["MAIL_TOKEN_REVIEWER"]: {
            "client_id": "agent-reviewer",
            "scopes": ["mail:read"],  # read-only: can fetch inbox, can't send
        },
    },
    required_scopes=["mail:read"],
)

mcp = FastMCP("agent-mail", auth=verifier)

Two things to notice. The token values come from environment variables, never literals in the file, so the tokens live in a Kubernetes Secret and get injected at pod start. And each agent role gets its own token with its own scopes. That second part is the actual RBAC: a reviewer agent that only triages messages has no business holding a credential that can send them. This is the same two-tier thinking I wrote about in agent credential management, applied one layer down at the MCP transport.

Per-tool enforcement then reads the validated token from the request context:

from fastmcp.server.dependencies import get_access_token
from fastmcp.exceptions import ToolError

@mcp.tool
async def send_message(to: str, subject: str, body: str) -> dict:
    token = get_access_token()
    if "mail:write" not in token.scopes:
        raise ToolError("This credential is read-only.")
    return await deliver(to, subject, body)

If you outgrow static tokens (more than a handful of agents, or tokens that need rotation without a redeploy), swap StaticTokenVerifier for FastMCP's JWTVerifier pointed at a JWKS endpoint. The server code barely changes; the constructor argument does. I covered the boilerplate side of FastMCP in an earlier post; auth is the part that post's happy path skipped.

Fixing the client side

The broken client configs I see fall into two buckets. The first is a stale stdio entry pointing at a script that moved or was replaced by the HTTP deployment:

{
  "mcpServers": {
    "agent-mail": {
      "command": "python",
      "args": ["/home/user/old-scripts/mail_server.py"]
    }
  }
}

This fails instantly and at least fails loudly. The second bucket is the sneaky one: the URL was updated to the new HTTPS endpoint but the Authorization header never got added, because the server didn't require one when the config was written. That's the config that worked for weeks and then started returning 403 the day auth landed on the server.

The corrected version, with the token pulled from the environment rather than committed in plaintext:

{
  "mcpServers": {
    "agent-mail": {
      "type": "http",
      "url": "https://mail.example.com/mcp",
      "headers": {
        "Authorization": "Bearer ${AGENT_MAIL_TOKEN}"
      }
    }
  }
}

Claude Code expands ${VAR} references in .mcp.json from the environment. Use that. A raw token string in a JSON file in your home directory has a way of ending up in dotfile repos, pair-debugging screenshots, and pasted "here's my config, what's wrong" messages. The env-var indirection costs you one line in a shell profile and removes an entire category of leak.

The ingress in front of it

Nothing exotic on the Kubernetes side. Traefik terminates TLS and forwards to the service; FastMCP handles auth itself, so no ForwardAuth middleware is needed:

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: agent-mail
  namespace: agents
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`mail.example.com`) && PathPrefix(`/mcp`)
      kind: Rule
      services:
        - name: agent-mail
          port: 8000
  tls:
    secretName: mail-example-com-tls

Keeping auth in the application rather than the ingress is a deliberate choice here. The server needs to know which agent is calling to enforce scopes anyway, so putting a second auth layer in Traefik just gives you two places to misconfigure and two flavors of 403 to tell apart. If you already run ForwardAuth everywhere as policy, fine, but then remember that layer exists when you debug.

Verify it like the client would

Before touching any agent config, prove the server behaves correctly with curl. First the failure case, no token:

curl -i https://mail.example.com/mcp \
  -H "Accept: application/json, text/event-stream" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize",
       "params":{"protocolVersion":"2025-06-18","capabilities":{},
       "clientInfo":{"name":"curl","version":"0"}}}'

You want HTTP/2 401 with a WWW-Authenticate: Bearer header. If you get Traefik's plain 403 or a 404, the request never reached FastMCP and your problem is routing, not auth. Then the success case:

curl -i https://mail.example.com/mcp \
  -H "Authorization: Bearer $AGENT_MAIL_TOKEN" \
  -H "Accept: application/json, text/event-stream" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize", ...}'

A 200 with an initialize result means the full path works: DNS, TLS, ingress, pod, auth. Only now is a client-side failure actually a client-side failure. This two-curl check takes thirty seconds and cleanly bisects the problem, which beats restarting agent sessions and squinting at /mcp output.

Gotchas

The silent failure is the real enemy. An agent whose MCP server fails auth doesn't crash. It just proceeds without those tools. In a multi-agent setup where sessions coordinate through mail, one agent silently losing its mailbox looks like that agent "deciding" not to communicate. Check /mcp status at session start, or better, make your agents' startup routine fetch their inbox once and treat failure as fatal rather than shrugging past it.

Health endpoints lie by omission. If you expose a custom unauthenticated /health route for Kubernetes probes, understand what you've built: a check that confirms the process is up while saying nothing about whether authenticated tool calls succeed. Reasonable for liveness. Useless for "are the agents actually able to use this."

Valid token, wrong scope, confusing error. The read-only reviewer token from the example above will initialize successfully and list tools, then fail on send_message. From the agent's perspective the tool exists but errors out. Make the tool-level error message state the actual problem ("this credential is read-only"), because the agent will relay that message to you verbatim, and "permission denied" tells you nothing.

Documentation drift bites here too. If your CLAUDE.md or agent instructions enumerate the mail server's tools and you later add auth-gated ones, agents will attempt calls their token can't make. Keep the documented capability list synchronized with what each role can actually invoke, not with what the server exposes in total. It's the same lesson as semantic index drift: any description of a system that isn't regenerated from the system will eventually be wrong.

An alternative I considered and rejected: mTLS between agents and the server. It authenticates the machine, not the agent role, and distributing client certs to ephemeral agent sessions is far more friction than handing each role a scoped bearer token. Certificates make sense when the caller is a long-lived service. Agent sessions aren't.

When to reach for this

The rule I'd apply: the moment an MCP server leaves stdio, it gets a token verifier, even if the only network it's exposed on is your own. Not because your LAN is hostile, but because the anonymous configuration silently becomes load-bearing, and you'll forget it's there until the day you add an ingress, a Tailscale route, or a second user. Retrofitting auth after three agents and two config files depend on anonymous access is strictly worse than starting with a static token that takes ten lines.

For a mail server specifically, the stakes are higher than for a read-only lookup tool. Messages are instructions. Scoping who can write them is the difference between a coordination layer and an attack surface. If you're building out multi-agent infrastructure and want a second pair of eyes on the security model, that's work I consult on.

And when the 403 does show up: read the status code, run the two curls, and find out which layer is saying no before you change anything. Most of the pain in these debugging sessions comes from fixing the wrong layer first.

Silent Drift: Why Re-Embedding Only on Count Changes Rots Your Semantic Index

Guatu — Wed, 29 Jul 2026 20:15:48 +0000

My index reported 354 points. The collection had 451. Both numbers were "correct," and that gap is the whole problem.

Here's the setup. I have a wiki that feeds a semantic index. An agent queries that index to pull context before answering infrastructure questions. The re-embedding job was wired to a dead-simple trigger: if the file count changed, re-embed the new files. Cheap, fast, and wrong in a way that takes weeks to notice.

The symptom

Search started returning confidently outdated answers. I'd edit a wiki article, fix a stale command, correct a version number, and the agent would keep retrieving the old text as if the edit never happened. No error. No stack trace. The vector was still "active" in the database, still matching queries, still ranking high. It just no longer represented what the source file actually said.

That is the failure mode I've started calling a zombie vector. Technically alive in the index. Semantically dead. It points at a version of the document that stopped existing the moment I saved the edit.

What I expected

I expected the index to reflect the source. That's the entire contract of a semantic index: the vectors are a searchable projection of the underlying text, and when the text changes, the projection changes with it. I assumed my re-embedding job upheld that contract. It did not. It upheld a much weaker one: the index reflects the set of files that exist, not their contents.

Those two contracts look identical right up until someone edits a file without adding or removing one. Which, if you maintain a wiki or a memory store, is most of what you do all day.

What actually happened

The trigger logic was count-based. Pseudocode, but this is close to what it was:

def maybe_reindex(source_dir, index):
    current_count = count_files(source_dir)
    indexed_count = index.get_metadata("file_count")

    if current_count == indexed_count:
        return  # "nothing changed", skip

    embed_new_files(source_dir, index)
    index.set_metadata("file_count", current_count)

Read that if current_count == indexed_count: return line again. That's the bug wearing a helmet. The moment the counts match, the function decides nothing changed and walks away. But file count is invariant under edits. You can rewrite every word of every article, and as long as you don't add or delete a file, the count stays put and the re-embed never fires.

Worse, look at what happens when the count does change. embed_new_files only touches files it considers new. It never revisits existing vectors. So even a triggered run leaves edited-but-not-added files untouched. The index accumulates drift on every edit and only ever grows, never corrects.

I found the 354-vs-451 discrepancy while debugging something unrelated. The metadata field said 354 points because that's the number the job had last written. The collection actually held 451 active points. The extra 97 came from a period where files got split and merged, chunk boundaries moved, and the job appended new chunks without ever retiring the old ones. So now I had two problems stacked on top of each other: vectors that were stale (edited source, unchanged embedding) and vectors that were orphaned (source chunk no longer exists, embedding lingers).

The orphaned ones are their own special misery. This is the ghost reference problem. A memory file references a path, the path gets deleted or moved, but the embedded chunk still says "see scripts/server.py for the handler." The agent retrieves that, follows the reference, and hits nothing. The index became a graveyard of pointers to resources that moved on without telling it.

None of this crashes. That's what makes it rot instead of a bug. A crash you fix in an afternoon. Silent drift you discover months later when someone asks why the agent keeps recommending a flag that got renamed in a release you shipped a while back.

The fix

Stop asking "did the count change." Start asking "did the content change." The count is a proxy for state, and it's a bad one. The actual state of a file is its contents, and you already have a cheap, exact way to fingerprint contents: a hash.

The pattern is a manifest. For every source file, store the hash of what you indexed. Before re-embedding, walk the source tree, hash each file, and compare against the manifest. Re-embed only the files whose hashes don't match, plus handle files that vanished.

import hashlib
import json
from pathlib import Path

def file_hash(path: Path) -> str:
    h = hashlib.sha256()
    h.update(path.read_bytes())
    return h.hexdigest()

def plan_reindex(source_dir: Path, manifest: dict[str, str]):
    current = {
        str(p): file_hash(p)
        for p in source_dir.rglob("*.md")
    }

    changed = [p for p, h in current.items() if manifest.get(p) != h]
    deleted = [p for p in manifest if p not in current]

    return changed, deleted, current

changed catches both brand-new files (no manifest entry, so the != is true) and edited files (entry exists but the hash moved). deleted catches files that disappeared, so you can purge their vectors instead of leaving zombies. And the returned current dict becomes your new manifest once the run succeeds.

The upsert side needs to be idempotent and it needs to delete before it inserts for changed files, otherwise you re-create the 354-vs-451 split. Use a deterministic point ID derived from file path plus chunk index so a re-embed overwrites the same slots rather than appending new ones:

def point_id(file_path: str, chunk_idx: int) -> str:
    raw = f"{file_path}::{chunk_idx}"
    return hashlib.sha256(raw.encode()).hexdigest()

def reindex(source_dir, index, manifest):
    changed, deleted, current = plan_reindex(source_dir, manifest)

    for path in changed + deleted:
        # remove all chunks for this file first, deterministic IDs
        index.delete(filter={"file_path": path})

    for path in changed:
        for i, chunk in enumerate(chunk_file(Path(path))):
            vec = embed(chunk)
            index.upsert(
                id=point_id(path, i),
                vector=vec,
                payload={"file_path": path, "chunk": i, "text": chunk},
            )

    save_manifest(current)

Delete-then-insert for changed files means a page that shrank from six chunks to four doesn't leave two ghosts behind. Delete-only for the deleted set purges files that are gone. The manifest save happens last so a crash mid-run leaves you with stale-but-consistent state, and the next run just retries the same diff.

There's one more guardrail that has nothing to do with counts and everything to do with silent corruption: dimension validation. Embedding providers occasionally return a vector of the wrong length. A model swap, a truncated response, a provider that quietly changed defaults. If you upsert a 768-dim vector into a collection built for 1024, some databases reject it loudly and some accept it and corrupt the distance math. Check before you write:

EXPECTED_DIM = 1024

def embed(text: str) -> list[float]:
    vec = provider.embed(text)  # e.g. a Bedrock call
    if len(vec) != EXPECTED_DIM:
        raise ValueError(
            f"embedding dim {len(vec)} != expected {EXPECTED_DIM} "
            f"for text starting {text[:40]!r}"
        )
    return vec

That single if len(vec) != EXPECTED_DIM has saved me from a whole category of "why is search suddenly garbage" investigations. A dimension mismatch that silently upserts poisons every subsequent query against that collection, and it looks exactly like drift until you check the vector lengths.

Then there's the periodic clean slate. Incremental hash-based diffing is your day-to-day path, but hashing has blind spots. If your chunking logic changes, if your embedding model updates, if the manifest itself gets corrupted, the diff can miss things because the source hash didn't move even though the right answer did. So I run a full rebuild on a schedule. Drop the collection, re-embed everything from scratch, write a fresh manifest.

def rebuild_memory_index(source_dir, index):
    index.recreate_collection(dim=EXPECTED_DIM)
    manifest = {}
    for path in source_dir.rglob("*.md"):
        h = file_hash(path)
        for i, chunk in enumerate(chunk_file(path)):
            index.upsert(
                id=point_id(str(path), i),
                vector=embed(chunk),
                payload={"file_path": str(path), "chunk": i, "text": chunk},
            )
        manifest[str(path)] = h
    save_manifest(manifest)
    print(f"rebuilt {len(manifest)} files")

I keep this as scripts/rebuild-memory-index.py and run it the way you'd run a database vacuum. It's the reset button that guarantees the index and source agree, no matter how much drift the incremental path accumulated. My wiki already had a rebuild script; the memory files did not, and that asymmetry was exactly where the documentation drift lived. If one index layer has a rebuild path and another doesn't, the one without it is the one silently rotting.

Why this matters

You will hit this the moment your source documents outlive their first embedding. Static corpora that never change are fine with count-based logic. But an agent memory store, a wiki, a knowledge base, anything a human or another agent edits in place, drifts continuously. The six-layer memory architecture I run for Claude Code leans hard on the embedding layer being honest, because every layer above it inherits whatever the semantic index believes. A stale vector at the bottom becomes bad reasoning at the top, and the agent has no way to know its retrieved context is a lie.

The economic argument usually runs the other way. Re-embedding is money and latency, so people optimize to touch only new content. That optimization is real, and for large corpora a full re-index on every write is genuinely wasteful. But the hidden cost sits on the other side of the ledger: an agent reasoning from zombie vectors produces confidently wrong output, and the cost of that, in a system anyone actually depends on, dwarfs the cost of a periodic rebuild. Cheap updates that corrupt the index aren't cheap. They're deferred, and the interest compounds in the form of debugging sessions that start with "the agent keeps insisting on something that hasn't been true for weeks."

This pairs with a failure mode I've written about before: your vector DB snapshots landing on the same disk that will fail. Same theme, different layer. In both cases the index looks healthy by the metric you're checking and is quietly broken by the metric that matters. It also sits next to decay-based eviction, which is a deliberate forgetting policy; the ACT-R decay approach removes vectors on purpose based on activation. Drift is the opposite. It's accidental corruption where the vector stays but its meaning leaves. One is controlled forgetting. The other is uncontrolled lying.

If you take one thing from this, make it a rule: never trust a count as a proxy for content. Counts answer "how many things exist," and that is almost never the question. The question is "does the index still agree with the source," and only a content fingerprint answers it. Treat re-indexing the way you treat CI. Hash-validate on every change, rebuild from clean state on a schedule, and validate vector dimensions before every write. I've been building this kind of self-correcting index behavior into agent infrastructure work because the alternative is an index that reports 354, holds 451, and confidently tells your agent about a file you deleted last month.

The index that lies to you never throws an error. That's exactly why you have to make it prove it's telling the truth.