DEV Community: Victor García

Seven agents per user: delegation, and every way it broke

Victor García — Wed, 29 Jul 2026 13:42:55 +0000

Here's a bug report we got from our own system, and it took us longer to understand than we'd like to admit:

"Ask Francis to generate a voice recording. He says he'll delegate it to Dali. Then nothing happens."

Nothing happened. No error, no timeout, no failed tool call. The coordinator agent said it was handing the job to the specialist, and then the conversation just… stopped. Dali never spoke.

We built a seven-agent topology because one agent with sixty tools is worse than seven agents with ten each. That part turned out to be right. What we underestimated is that delegation is the whole product. Every interesting failure in the last four months has been at the seam between two agents, not inside one.

This is a tour of that seam, including the parts still broken.

THE TOPOLOGY

Every user gets seven agents, provisioned atomically when their account is created. They are not shared. They are not a pool. Here's the actual configuration, straight out of auto-provision.service.ts:

Agent	Role	Model	Scopes
Francis	Coordinator — notes, calendar, mail, contacts, search, diary, bookmarks	`claude-sonnet-4-6`	`notes:* events:* emails:* contacts:* diary:* bookmarks:* search:read graph:read`
Atlas	Productivity — files, photos, office docs, projects, diagrams	`claude-sonnet-4-6`	`files:* photos:* search:read`
Sentinel	Infrastructure — storage, network, reverse proxy, DNS, VPN, Docker	`deepseek-chat`	`storage:* hal:* proxy:* dns:* vpn:* docker:*`
Dali	Creative — image generation, media, voice (TTS/STT), web browsing	`claude-sonnet-4-6`	`audio:* media:* photos:read files:read`
Ledger	Money — bank accounts, transactions, budgets, CRM	`deepseek-chat`	`finance:* crm:* search:read`
Darwin	Intelligence — patterns, preferences, cross-domain search	`claude-sonnet-4-6`	`search:read`
HASS	Smart home — IoT devices, automations, scenes, sensors	`deepseek-chat`	`ha:*`

Three things worth pointing out, because they're the parts people get wrong when they copy this shape.

The models are deliberately different. Sonnet for the agents that need judgment (coordination, creative work, pattern-finding); DeepSeek for the agents that mostly run deterministic infrastructure and finance operations. This isn't cost optimization dressed up as architecture — Sentinel restarting a container doesn't need the same reasoning as Francis deciding which specialist should handle a request.

The topology is flat. Depth-1, siblings only, no nested sub-agents. Francis is the chief but not a bottleneck: you can talk to any agent directly from the chat. The coordinator exists to route the requests where you didn't say who should handle it.

Two of them are nearly empty now, and we left it that way. Darwin ships with exactly one skill (claw-search). There's a comment in the config explaining why: claw-graph and claw-insights were retired in June because the knowledge graph and the derived insights turned out to be human features of the dashboard, consumed through REST, not something an agent needed to poke at. Ledger is in the same shape. The honest version of a multi-agent design is that some agents shrink after you watch what people actually ask for.

There's also a hidden eighth agent, crawl, dedicated to the browser extension. It's marked hidden: true and never shows up in the agents module — it exists so that the extension's chat doesn't compete with Francis for attention in the user's real browser session.

Namespacing, or: why every agent has a double dash in its name

Agents are per-user, and the runtime that executes them is shared. So the identifier of an agent is {prefix}--{name} — paco--atlas, admin--francis. The prefix is the local part of the user's email, normalized to [a-z0-9-], with a numeric suffix on collision, and it lives in users.agent_prefix as UNIQUE NOT NULL.

Each agent-user pair gets its own workspace directory:

~/.openclaw/workspaces/paco--atlas/
├── SOUL.md      # personality (atlas = methodical, dali = creative...)
├── USER.md      # user profile, propagated from the chief on bootstrap
├── TOOLS.md     # the catalog of skills this agent can reach
├── MEMORY.md    # persistent memory
└── skills/      # physical copies of the SKILL.md files it's allowed to read

The display name stays "Atlas". The namespacing is invisible to the user — right up until it isn't, which we'll get to.

FAILURE 1: THE CHILD THAT INHERITED THE WRONG TOOLS

The runtime gives an agent two ways to start work somewhere else. sessions_spawn creates a child. sessions_send hands a message to another agent's session and waits for the reply.

We used sessions_spawn first, because "spawn a Dali to do the audio" reads like exactly what you want. It isn't. A spawned child inherits the parent's tools, not the target agent's. So sessions_spawn(agentId: "paco--dali") produces something that is named Dali, is addressed as Dali, and does not have claw_audio, because Francis doesn't have claw_audio.

The delegation guide now says it in a box, in capitals, at the top:

sessions_spawn creates a subagent that INHERITS YOUR tools, not the specialist's.
If you do sessions_spawn(agentId: "<PREFIX>--dali", ...), that child "Dali" ends up
WITHOUT claw_audio (because you don't have it) and the task fails.
sessions_spawn is only for YOUR OWN sub-tasks (a worker of yours running in parallel
with your same tools). To delegate to a specialist, ALWAYS sessions_send.

Scenario	Tool	Why
Ask Dali for audio, Atlas for files, Sentinel for infra	`sessions_send`	The specialist runs as itself, with its tools (synchronous, you wait for the reply)
Run a sub-task of your own in parallel	`sessions_spawn`	It's your worker; inheriting your tools is correct here

The second-order lesson is more useful than the fix: when two operations differ only in whose capabilities they carry, a model will pick the wrong one. The names don't encode the difference. Only the documentation does, and documentation is a prompt, and prompts get compacted.

FAILURE 2: AN OPTIONAL PARAMETER WITH NO DESCRIPTION

This is the one that produced the bug report at the top, and it is still open.

sessions_spawn takes an agentId. In the runtime's schema, that parameter is declared as:

agentId: Type.Optional(Type.String())

Optional. And — this is the part that matters — with no description. Meanwhile taskName and thread both have descriptions. The tool's own summary is one line: "Spawn subagent or ACP session."

Put yourself in the model's position. You get a tool with several described parameters and one undescribed optional one. You fill in what's described: task, taskName, label, model, runtime, sandbox, mode. You omit the one nobody explained. And because it's optional, the runtime doesn't complain — it spawns you. Francis delegates to Francis, does the work badly or not at all, and reports back that the specialist has been notified.

We verified this against a real trace (session ea2c4895): the call carried seven parameters and no agentId.

It's the same class of bug as another one we hit in the MCP facade — a parameter the model can't fill because the schema never told it the parameter was load-bearing. A tool call is a contract between a schema and a language model, and an undocumented optional field is a contract clause written in invisible ink.

This one is still in our open debt list. The fix we've scoped is a before_tool_call hook that rejects sessions_spawn without an agentId and returns the specialist map in the error, so the model can retry correctly in the same turn — the same hook mechanism we already use for agent identity. We haven't shipped it yet, and until we do, delegation via spawn is unreliable. Saying otherwise would be more comfortable and less true.

FAILURE 3: THE PARENT THAT WAITED FOREVER

When a parent delegates and yields, it's waiting for an announce-back from the child. That notification is best-effort. So:

Children can abort (timeout, reset, bug) and the notification to the parent is best-effort. If Core/Gateway restarts while a parent is yielded, the children's announce-back is lost and the parent hangs forever — there's no sweeper to detect it.

A conversation that is neither finished nor running, with no error anywhere, is the worst possible state: nothing to retry, nothing to report, nothing in the logs.

The detector we built reads the last 8 KB of each parent's session JSONL looking for a sessions_yield not followed by an assistant message, and writes a row into stuck_yield_sessions (migration 0187, idempotent via a unique key on (user_id, session_key)). Two design details earned their keep:

A 3-minute grace window (STUCK_GRACE_MS) before flagging anything. The comment in the code explains why: "so we don't false-positive during legitimately long delegations (Atlas with thinking + tool calls can take ~30 s before the first assistant text)." Without it, the detector's main output is noise about work that's going fine.
Auto-resolution. If the parent un-yields on its own, the row clears without anyone intervening. A recovery mechanism that requires a human to close its own tickets is a second job, not a fix.

Recovery is a POST /managed-agents/:id/stuck-yield/resume that injects a synthetic message into the parent, plus a button in the dashboard.

FAILURE 4: THE CONVERSATIONS NOBODY COULD SEE

The runtime runs children internally and never emits WS events for their messages to Core. Result: in the dashboard's Conversations tab you see the chief ("I delegated to Atlas") but you don't see Atlas's conversation. To audit a council or a multi-agent workflow you have to open the raw JSONL in the workspace.

This is an observability gap that becomes a correctness gap. If you can't see what the delegated agent did, you can't tell "Dali refused" from "Dali was never asked" from "Dali did it and the reply got lost" — which are three completely different bugs that present identically.

The mirror tails each subagent's session JSONL with byte-offset watermarks (migration 0188, dedup index 0189, 1 MB read cap per pass) and writes the messages into agent_conversations in two modes: type='delegation' for child runs, and type='webchat' for the parent's reply to the announce-back — which, as the code notes, "happens inside the Gateway after chat-bridge.callGateway has already returned", and was therefore invisible to Core by construction.

FAILURE 5: THE POLICY WITH NOWHERE TO PLUG IN

We built a DelegationPolicy. Per-agent defaults, retry with backoff, an in-memory circuit breaker, storage for all of it. It was good work. We reverted it the same day, with migration 0180_drop_delegation_policy.sql.

The reason is in the doc, and it is the most useful paragraph in this post:

Delegation between agents happens via sessions_spawn, which is a native LLM tool inside the runtime — not an RPC that Core can intercept. With that API, the DelegationPolicy we built has no insertion point:
✅ Storage · ✅ Per-agent defaults · ✅ withRetry helper with backoff · ✅ In-memory CircuitBreaker
❌ Nobody invokes it — Francis still delegates via sessions_spawn directly.
Keeping it alive was infrastructure without a consumer.

Four components, all working, all correct, all unreachable. We had built a governance layer for a call path that never passes through us.

We wrote down the three conditions that would unblock it — the runtime exposing sessions.spawn as an RPC, changing the skill's contract to an HTTP delegate_to against Core, or building our own spawning runtime ("several sprints, high risk") — and deleted the code. The conditions are the valuable artifact. The code was the expensive way to discover them.

FAILURE 6: THE ALLOW-LIST THAT ALLOWED TOO MUCH

Agent-to-agent messaging is gated by an allow-list. Ours was a flat list of wildcards, one per user: paco--*, pepito--*, admin--*, ten entries. And the guard's formula was matchesAllow(requester) && matchesAllow(target).

Read that twice. Both sides match the list. paco--francis matches. pepito--atlas matches. So paco--francis → pepito--atlas is allowed. One user's coordinator could send work to another user's specialist, and everything downstream — per-user workspaces, per-user scopes, isolated memory — would be doing its job correctly on the wrong person's data.

What was actually preventing it, as we noted at the time, is that the models don't know other users' agent IDs. That is not a security boundary. That is a fact about vocabulary.

It's now closed deny-by-default in the identity plugin (crossUserSendBlock): an agent can only send to agents sharing its own prefix. Legitimate cross-user delegation, when we build it, will be an explicit grant — not the absence of a check.

WHAT I'D DO DIFFERENTLY

Find the insertion point before writing the policy. The DelegationPolicy revert cost us a day and taught us something we could have learned in twenty minutes by asking "what function will call this?" before writing the first line. When you're building governance for a system you don't fully own, the first deliverable isn't the policy — it's the proof that the policy can be enforced.

Treat tool schemas as prompts, because they are. Two of the six failures above (the spawn/send confusion, the undescribed agentId) are the model correctly following an under-specified contract. We spent weeks looking at model behavior when the bug was in a type declaration. The parameter descriptions in a tool schema are not documentation for humans reading the code; they are the only instructions the model receives at the moment it has to decide.

Build the observability at the same time as the delegation, not after. We shipped delegation, then spent a month unable to diagnose it because the child conversations were invisible. The mirror should have been part of the first version. A multi-agent system where you can only see one agent isn't a multi-agent system, it's a black box with a friendly name.

Assume best-effort notifications will be lost, and write the sweeper first. The stuck-yield detector isn't clever; it's 8 KB of tail-read and a grace window. The mistake was shipping the yield before shipping the thing that notices when the yield never ends.

ONE MORE THING: THE AGENTS SHARE A BRAIN

A rule from the delegation guide that surprises everyone, including us:

While you wait, do NOTHING else: no sessions_history, no process, no exec, no
sessions_yield, no re-sending the task, no trying it yourself with Python/shell.
sessions_send already waits; the result comes back on its own. On this team the
agents SHARE an AI model: if you keep acting (e.g. polling their history), you hog
the model and the specialist CAN'T work → the task hangs. Stay still.

An agent that anxiously checks on its colleague's progress prevents its colleague from making progress. Seven agents, one model instance, and the failure mode is that helpfulness starves the thing it's trying to help.

THE TAKEAWAY

The topology was the easy decision. Seven specialists and a coordinator, flat, per-user, with their own scopes and their own workspaces — that shape has held up for four months without a single revision.

Every hard problem lived in the handoff: a child with the wrong tools, a parameter the model couldn't see, a parent waiting on a message that got lost in a restart, a conversation nobody could read, a policy with nowhere to attach, and an allow-list held together by the fact that nobody knew the right names.

If you're building multi-agent anything: the agents are not the system. The seams are the system.

Next up: #16 — fitting seven agents into a model with an 80k context window, which is a story about a 45% cut, two conclusions I was wrong about, and one bug that turned out not to exist.

Deterministic agent identity: a before_tool_call hook fills the token the model kept getting wrong

Victor García — Sun, 26 Jul 2026 10:42:27 +0000

Francis is the main agent — the one you talk to in the dashboard. Dalí is the creative one, the sub-agent you hand an audio or an image task to. One day we asked Francis to have Dalí generate a short audio clip, and nothing came back. No error in the dashboard, no red toast, no timeout the user could see. Just silence.

We logged the bug as "Dalí doesn't answer," and it sat in the tracker for a while looking like a flaky model or a bad prompt — the kind of thing you're tempted to blame on a small local model having a bad day.

It was neither. It was an authentication hole with a very specific shape, and closing it meant taking a 64-character secret out of the language model's hands entirely.

The problem the model couldn't see

Our agents don't talk to Core with curl. They call typed tools — claw_notes, claw_calendar, claw_audio — served by an MCP facade that runs host-side at 127.0.0.1:7250 (that's the previous post's subject). Every one of those tool calls has to carry an auth_token argument. Core takes the token, resolves it to a userId plus a set of scopes, and dispatches the call through its full pipeline via app.inject() — auth, scope check, semantic-scope check, approval, handler, audit. The facade reimplements none of that security; the token is the single thing that lets the pipeline know who is asking.

So the whole design rests on one question: where does that token come from?

Before this fix, it came from the prompt. Two bridges injected it — ws/chat-bridge.ts for webchat and ws/voice-bridge.ts for voice. When you typed to Francis in the dashboard, the chat bridge dropped Francis's token into the prompt, and Francis was instructed to copy it into every claw_* call. That works — mostly — for a main agent talking through a bridge.

A subagent spawned with sessions_spawn passes through neither bridge. When Francis delegates to Dalí, there is no chat bridge and no voice bridge in the path. So Dalí received no token at all. Every claw_* call it made failed auth, silently, and the delegated work just stopped. That was "Dalí doesn't answer": not a model refusing, a model that was never handed the credential it needed to do anything.

And even the main agent, on the happy path, was fragile. We were asking a language model to reproduce a 64-hex-character secret, verbatim, on every single call. A capable model does it — most of the time. But a secret the model has to retype is a secret the model will eventually get wrong, and when it improvised a tool it hadn't been shown properly, it improvised the token too.

Fact	Value
Real runtime token	64 hex characters
Copied correctly on a capable model	6 / 6 (Qwen3.6-35B IQ2_M, ~27K-token context)
Value emitted when it improvised a wrong tool	42 characters — not in the database
Subagent success rate before the fix	0% (no token was ever injected)

The 6/6 number is the trap. On rails, a strong model looks perfectly reliable, and you conclude the design is fine. Then the same model, one wrong turn later, emits a 42-character string that matches nothing in the database — and the subagents, which you can't watch as closely, get it wrong 100% of the time because they get nothing at all. Fragile by construction.

Why not just a header

The obvious fix is a header. MCP servers usually authenticate with a static header — so put the token there, stop asking the model to carry it, done.

It doesn't work here. In OpenClaw, the mcp.servers config is global. The headers are static and shared across every agent — there is no per-agent identity per session. If we hard-coded a token into the server's headers, every agent on the box would authenticate as the same identity. That destroys the entire point of per-agent scopes: Dalí is supposed to have audio:*/media:* and nothing else; Francis has its full grant. A shared header can't express "this call is Dalí, that call is Francis."

That global-static constraint is exactly why ADR-10 put identity in a tool argument in the first place. ADR-11 doesn't undo that decision — the facade still dispatches Authorization: Bearer <auth_token> through app.inject(). All that changes is who fills the argument. Instead of the model, it's the host.

The hook

OpenClaw lets a native plugin register a before_tool_call hook — a function that runs on the host, sees every tool call before it executes, and can rewrite its params. We wrote one: openclaw-plugins/claw-identity/index.js, plain ESM JavaScript, no build step. The core of it is about five lines.

// The core of openclaw-plugins/claw-identity/index.js — the before_tool_call hook
api.on("before_tool_call", async (event, ctx) => {
  const toolName = event?.toolName ?? "";
  if (!toolName.startsWith("claw-os__")) return;      // only the MCP facade tools
  const token = loadTokenMap(mapPath)[ctx?.agentId];  // map keyed by ctx.agentId
  if (!token) return;                                  // no entry -> don't touch the call
  return { params: { ...event.params, auth_token: token } }; // override host-side
}, { priority: 100 });

That's the whole idea. For any tool whose name starts with the facade prefix claw-os__, the hook looks up the calling agent by ctx.agentId in a token map and rewrites params.auth_token to that agent's token. Whatever value the model put there is overwritten. If there's no map entry for the agent, the hook returns undefined and leaves the call untouched — it never breaks a call it doesn't recognize.

The model never sees the token, never types it, never hallucinates it. And because the key is ctx.agentId, a subagent spawned via sessions_spawn gets its own token injected host-side, with its own scopes — the exact case that used to fail silently. Determinism, keyed on who is actually running the tool.

The per-agent token and the map

Where does the map come from? Core mints it. syncUserRuntimeTokens(userId) in core/src/services/agent-runtime-token.service.ts creates exactly one canonical agent_tokens row per agent, with name='runtime'. The database stores only the sha256 hash; the plaintext exists only in an on-disk map.

That map is ~/.config/micelclaw/agent-tokens.json, and its shape is deliberately dull:

{
  "paco--francis": "…64 hex chars…",
  "paco--dali":    "…64 hex chars…"
}

The keys are exactly the ctx.agentId values the hook looks up — <prefix>--<agent>, one per user-agent. Writing that file is where we were careful, because the plugin reads it live:

// Atomic map write — core/src/services/agent-runtime-token.service.ts
async function writeMapAtomic(map) {
  const path = runtimeTokenMapPath();               // ~/.config/micelclaw/agent-tokens.json
  await mkdir(configDir(), { recursive: true, mode: 0o700 });
  const tmp = `${path}.tmp`;
  await writeFile(tmp, JSON.stringify(map, null, 2), { mode: 0o600 });
  await rename(tmp, path);                           // atomic swap: the plugin never reads a half file
}

mkdir with mode 0o700, the file itself 0o600, and a temp-file-plus-rename so the swap is atomic. The plugin reads through an mtime cache (loadTokenMap): it only re-reads from disk when Core actually rewrites the file, and on any read error it falls back to the last-known map and never throws. That mtime-triggered re-read is exactly why the write has to be atomic — if we wrote the file in place, the plugin could catch it half-written and hand out a truncated token.

The scopes matter as much as the token. The desired scope set is derived from the agent's live role grant:

deriveScopesFromSkills(skillScopeDomains(skills), roleScopesFor(agentDbId), perms)

roleScopesFor returns the agent's broadest base token — so Francis keeps its full grant (~17 scopes) and Dalí keeps audio:*/media:*. Nothing is downgraded, nothing is escalated: the runtime token carries exactly the identity the agent already had. And it is re-minted only when the scopes actually change — a scopesEqual guard keeps it stable otherwise, so we're not churning credentials on every sync.

Here's the whole thing in four pieces:

Piece	File	Role
The hook	`openclaw-plugins/claw-identity/index.js`	`before_tool_call` → injects `auth_token`, priority 100
Token service	`core/src/services/agent-runtime-token.service.ts`	Mints one `runtime` token per agent, writes the map
Registration	`core/src/services/openclaw-bootstrap.service.ts`	Idempotent `configPatch` at boot + on regenerate-tools
The facade	`core/src/mcp/claw-os-mcp-server.ts`	Serves `claw_*` at `127.0.0.1:7250`, dispatches via `app.inject()`

Rollout and verification

The hook only helps if the plugin actually loads. Registration is an idempotent configPatch — plugins.load.paths plus plugins.entries.claw-identity.enabled — done by ensureClawIdentityPluginRegistered() in openclaw-bootstrap.service.ts, both at boot and from regenerate-tools. The plugin then loads on the next Gateway reload.

We didn't trust that it worked until the logs said so. Two lines in the Gateway log (/tmp/openclaw/openclaw-YYYY-MM-DD.log) tell the whole story:

[claw-identity] plugin cargado; mapa de tokens = /home/victor/.config/micelclaw/agent-tokens.json
[claw-identity] before_tool_call tool=claw-os__claw_notes agent=paco--francis token=ok

The first line says the plugin loaded and resolved its map. The second says the hook fired for a real MCP tool call, for the right agent, and injected a token that Core accepted (token=ok).

Then the end-to-end proof — the one that actually closed "Dalí doesn't answer." We sent Dalí a direct message: "genera un audio que diga Hola caracola." The log showed before_tool_call tool=claw-os__claw_audio agent=paco--dali token=ok. Dalí called the real claw_audio tool — not a shell workaround, the actual tool — and an .ogg file landed in Drive with a real file_id. The subagent that had never received a token now authenticates like any other agent.

The gotchas

None of this was clean the first time, and the credibility of a fix like this is in the sharp edges we hit getting there:

plugins.entries.<id> rejects env. The entry schema only accepts enabled, hooks, and config. A configPatch with an env key returns INVALID_REQUEST: Unrecognized key: "env" — and the whole patch is dropped, not just the bad key. So the plugin resolves the map path by its default (~/.config/micelclaw/agent-tokens.json) rather than reading it from the entry's env.
The MCP SDK strips undeclared params. Any tool-call argument not declared in the tool's inputSchema disappears silently. We host-inject an internal _conversation_id (so Core can route L2 approval cards back to the chat they came from) — and it must be declared in the schema, or the SDK drops it on the floor and the routing quietly breaks. Same class of trap as the token itself.
The plugin needs a local SDK symlink in dev. index.js imports openclaw/plugin-sdk/plugin-entry; from its own directory, Node needs a local node_modules/openclaw symlink pointing at /usr/lib/node_modules/openclaw (gitignored). A packaged deploy installs the plugin with its own dependencies; dev has to fake it.
Boot registration can lose the WS race. If the Gateway WebSocket isn't up yet when Core tries to register the plugin, the configPatch is dropped and retried. That's why POST /managed-agents/regenerate-tools re-ensures registration once the WS is reliably up — it's the belt to the boot-time suspenders.
tsx watch doesn't hot-reload reliably on WSL. Code changes need a full Core restart to take effect, and the plugin itself only loads on a Gateway reload. Edit-and-refresh does not apply here; you verify against the real, restarted runtime or you verify nothing.

Where it grew

Worth being honest about how this file aged. It started as ~40 lines that did one thing: fill auth_token. The current index.js is 327 lines, because the hook turned out to be the natural place to enforce anything that has to be true host-side, keyed by agent identity.

It now also redirects sessions_spawn→sessions_send for cross-agent delegation; enforces a deny-by-default cross-user sessions_send guard (crossUserSendBlock); injects a wb_<prefix> boardId into the five board-scoped workboard tools; ensures a dedicated delegation session per (conversation × specialist); and injects the _conversation_id mentioned above. Each of those is a place where the model shouldn't be trusted to get identity right, and the hook is already sitting on the one chokepoint every tool call passes through — so it kept accreting responsibilities.

That's powerful and it's a smell at the same time. What began as "the auth_token hook" is really "the per-agent host-side identity layer" now, and the name on the tin no longer describes it.

What I'd do differently

Ship identity out of the prompt from day one. Having the model copy its own token "works" on a strong model and fails silently for subagents and weak ones. The 6/6 copy rate lulled us into thinking the design was sound; the host-side hook should have been the first design, not the fix that arrived after "Dalí doesn't answer." If a component has to reproduce a secret to function, that's the bug, no matter how reliable the reproduction looks in a demo.

Decide up front whether the hook is single-purpose or the identity chokepoint. It has since absorbed delegation redirection, a cross-user deny guard, workboard board-scoping, and conversation-id routing. That's a coherent set of responsibilities — but nobody decided it should be one file, it just grew from ~40 to ~330 lines one commit at a time. I'd name and document it as "the per-agent host-side identity layer" from the start, so its scope is a choice rather than an accident.

Finish the cleanup. Two follow-ups are still open in the source doc. The chat and voice bridges still inject the identity token into the prompt as a fallback — redundant now that the hook overwrites it — and auth_token is still a declared field on the tool input schema. Once the hook is proven in production, both should go: drop auth_token from the schema, and slim the now-pointless token out of the bridge prompts. Leaving a fallback in place feels safe, but a fallback the model can still fill is a value the model can still get wrong.

The takeaway

If your model has to copy a secret, it will eventually get it wrong — and the calls you never see, the ones subagents make, will get it wrong 100% of the time. Move identity out of the prompt and into a host-side before_tool_call hook: deterministic, per-agent, each agent keeps its own scopes, and the model can't hallucinate what it never handles.

Next up: #15 — Multi-agent topology: 7 agents per user, delegation, and the failure modes.

The MCP facade: how agents talk to the backend without curl

Victor García — Thu, 23 Jul 2026 06:09:03 +0000

An agent skill runs curl http://127.0.0.1:7200/api/v1/notes from inside its Docker sandbox. It fails with exit code 7 — "couldn't connect" — before authentication even runs, because the sandbox is launched with network: none. There is no loopback to reach. There is no network at all.

Meanwhile, the SKILL.md files were confidently telling the model to authenticate with a $CLAW_API_KEY environment variable that didn't exist inside the sandbox in the first place.

So the picture was: our agents had been instructed to talk to a backend they could never reach, using a token that wasn't there. Every write to a note, every calendar lookup, every "remember this" — all of it was a curl into a wall. This post is about the MCP facade that replaced that model, and the one finding that made the whole design fall out cleanly.

# inside the agent's Docker sandbox (network: none)
$ curl http://127.0.0.1:7200/api/v1/notes
# exit code 7 — couldn't connect, before auth even runs

The fix we almost built

The first instinct was to punch a hole through the wall without taking it down. Keep network: none entirely — the isolation is a feature, not an accident — and hand the sandbox a filesystem socket instead of a network address. Bind-mount a Unix socket into the container and the agent could do curl --unix-socket /run/claw.sock http://core/api/v1/notes: no TCP, no network namespace, just a file descriptor mounted into the container. Call it option C.

Option C was sound. A bind-mounted socket genuinely sidesteps the network: none wall that killed exit-7 curl, because a Unix socket isn't a network — it's a file. It would have worked.

And then we found something that made it irrelevant.

The finding that changed everything

The Model Context Protocol has a client and a server. We had been assuming the MCP client — the thing that actually issues the tool call — would live wherever the agent's code lives: inside the sandbox.

It doesn't. The MCP client runs host-side, in the OpenClaw runtime process, not inside the agent's Docker container. When a model emits a tool call, the runtime — outside the sandbox — is what dispatches it.

That single fact rewrites the problem. A tool call never touches the sandbox's network namespace, because it never originates inside the sandbox. The network: none wall — the exact wall that killed curl with exit 7 — is simply not in the path. There is nothing to tunnel through, no socket to bind-mount, no bridge to build.

Option C became obsolete overnight. It would have solved a real problem — reach Core without a network — but host-side MCP solved the same problem more cleanly, with no second listener to run and secure. We shelved it before writing a line of it.

The shape was now obvious. Core would run its own MCP server — the facade — and expose each domain of the API as a typed tool.

Facade:  core/src/mcp/claw-os-mcp-server.ts
         streamable-HTTP MCP server on 127.0.0.1:7250
         (Core itself is on 127.0.0.1:7200)

Each domain gets one tool named claw_<domain> with an action enum. claw_notes takes action: create | list | get | update | delete. claw_calendar, claw_contacts, claw_finance, and the rest follow the same mould. The model never learns a URL. It learns a vocabulary of typed tools.

A translator, not a second backend

Here was the decision that mattered most, and the one we kept coming back to: the facade is a thin translator, not a reimplementation.

The tempting version of this project is a second backend — a service that knows how to create a note, check a scope, run an approval, sign an audit record. That version is a disaster. It's a duplicate of the real Core that drifts out of sync the first time someone changes a validation rule and forgets the copy.

We refused to write it. Every facade tool call does exactly three things:

Look up the capability registry → (method, url, body, query).
Build an internal HTTP request carrying Authorization: Bearer <auth_token> plus x-claw-capability: <domain>.<action>.
Dispatch that request through the full Fastify pipeline via app.inject(): auth → scope-validation → semantic-scope → approval-check → handler → audit.

app.inject() is Fastify's in-process request injector — it runs a request through every plugin and middleware the real HTTP server runs, without a socket. So a claw_notes call and a human's POST /api/v1/notes walk the same hallway. The facade adds the Bearer prefix, names the capability, and gets out of the way.

// what the agent actually calls (facade tool, host-side on :7250)
claw_notes {
  "action": "create",
  "params": { "title": "...", "content": "...", "tags": ["..."] },
  "auth_token": "<per-conversation session token>"  // bare value, NO "Bearer"
}

// what the facade builds internally and dispatches via app.inject()
Authorization: Bearer <auth_token>   // the facade adds the "Bearer"
x-claw-capability: notes.create
// → auth → scope-validation → semantic-scope → approval-check → handler → audit

Because every call routes through the real pipeline, everything the pipeline already enforces stays enforced, with nothing rewritten:

Preserved for free	Where it lives
Deny-by-default scopes	`scope-validation.ts`
L0–L3 approval levels	`approval-check.ts`
Ed25519 audit signing	audit stage of the pipeline
Tier gating (Free/Pro/Plus)	the handlers
CRUD hooks (embed / heat / extract)	post-write pipeline
Credits accounting	the handlers

The agent's identity travels in the auth_token argument; auth.ts resolves the fixed userId from it, which is what makes impersonation impossible — a model cannot claim to be someone else by editing a field, because the token is the identity.

The capability registry

The (method, url, body, query) lookup in step 1 comes from a declarative registry: one file per domain under core/src/mcp/registry/domains/*.ts. Each Capability maps a (domain, action, params) triple to exactly one internal HTTP request.

// registry entry: (domain, action, params) -> one internal request
// core/src/mcp/registry/domains/notes.ts
{ domain: 'notes', action: 'create', method: 'POST', url: '/notes',
  scope: 'write:notes', approval: 'L0' }

This registry is the single source of truth, and it earns its keep by feeding two consumers at once. The same declarations generate:

The facade's tool catalog (what the model sees as callable tools).
The GET /managed-agents/available-tools endpoint that powers dynamic Tool Access (what an admin toggles per agent in the dashboard).

One file, two surfaces, no chance of them disagreeing. Add a capability and it shows up in both the model's vocabulary and the permission UI in the same commit.

Security had to come first

There's an uncomfortable truth buried in this whole design. For as long as the agents lived behind network: none, a lot of latent vulnerabilities didn't matter — they were contained by a wall that happened to block everything. The moment we connected agents host-side, that wall came down, and those vulnerabilities became live.

So the actual first phase of this project (Fase 0) wasn't the facade at all. It was closing three holes before exposing anything:

Guard	What it closed
C1	`POST/PATCH /agent-tokens` now returns `403` when `keyScope === 'agent'` — an agent can't mint or widen its own token (no self-escalation).
C2	The same `403` on `requireAdmin` routes — an agent token can never reach an admin surface.
C3	`scope-validation` flipped to deny-by-default — an unmapped resource is denied, not allowed.

C3 is the one that changes the posture of the whole system. Before, "we didn't map this resource" meant "anything goes." After, it means "no." Sensitive domains had to be re-exposed deliberately and carefully on top of that floor — the kind of thing that, done in the wrong order, quietly ships a hole.

Identity: the model stopped copying the token

The first version of the auth model (ADR-10) made the model responsible for copying its own auth_token — a 64-hex secret — into every single claw_* call. On a capable model, on rails, this works. But it's fragile by construction: the moment the model improvises a wrong tool call, it also mangles or hallucinates the token, and worse, subagents got no token at all. A child spawned via sessions_spawn doesn't pass through the chat bridge that injected the token, so delegated work silently failed. This is the real reason behind the months-old complaint that "Dali doesn't answer" when you delegate to her.

ADR-11 fixed it by taking the model out of the loop entirely. A native OpenClaw plugin, openclaw-plugins/claw-identity, registers a before_tool_call hook. For any tool named claw-os__*, the hook overwrites params.auth_token with the correct per-agent token, resolved host-side from ctx.agentId against a token map at ~/.config/micelclaw/agent-tokens.json (chmod 600). The model's value, if it bothered to supply one, is ignored.

Deterministic, and — crucially — it covers subagents. Any agent, main or spawned, gets its own token injected host-side. It can't hallucinate a token it never has to write. (That hook is a whole story of its own; it's next week's post.)

Skills stopped being API docs

Once the mechanics — URL, headers, method, request shape — moved into the typed tool and its schema, the SKILL.md files had nothing left to document. So they became something better.

Each skill shrank from a full API reference to a ~25–45 line judgment guide: not how to call the endpoint (the schema captures that) but when and how — the taste and context a schema can't encode. "Prefer list with a tag filter before get." "Don't create a duplicate contact; search first." The skill teaches the decision, the tool carries the call.

Auditing the real surface

One rule we set early: never mirror the capability registry to describe the API. Audit the real REST surface, because the registry is a deliberate subset of it, and pretending otherwise hides what's exposed.

So on 2026-07-03 we cross-audited both:

Surface	Count
REST endpoints in `routes/`	1456
Capabilities across code domains	424
Code domains	31
REST surface deliberately not exposed to agents	~60%

That ~60% is intentional, not a backlog. It's admin, config, binary, and UI-only endpoints that agents have no business touching. The audit's value is precisely in making the gap explicit — you can only reason about what an agent can do if you've measured what the surface actually is, endpoint by endpoint, rather than trusting the shorter list to speak for the longer one.

The gotchas that cost us

Wiring typed tools into a model turned out to have sharp edges that had nothing to do with the design and everything to do with the plumbing.

The MCP SDK silently strips undeclared args. Any argument not declared in a tool's inputSchema is dropped before it reaches the facade. Since the identity hook injects auth_token (and _conversation_id) host-side, those fields must still be declared in the schema — otherwise the SDK strips the hook's injected value and it dies silently on the way in. The thing you inject must be a thing you declared.

A z.record schema sent a weak model into an 84-call loop. Our first cut emitted the tool's params as z.record → JSON Schema properties: {} — no declared fields. A small local model (Francis, a 35B-A3B at IQ2_M) was asked for "contacts starting with a" and made 84 calls with params: {}, completely unable to construct params: { search: "a" } because the schema gave it nothing to fill in. We fixed it in tool-builder.ts by generating params as the union of every action's .shape, .partial().passthrough() — real, named fields the model can reason about.

One .email() took down every Groq chat. zod v4's .email() emits a regex with lookaheads. Groq compiles every tool schema with an RE2 engine, and RE2 has no lookarounds — so it returned a 400 and rejected the entire chat, for every agent that happened to carry the files domain. One field, one lookahead, whole conversation dead. Fixed with a generic lookaround-stripping sanitizer in tool-builder.ts (the real validation still runs against the original zod, so nothing loosens).

And a preset gotcha: bundle-mcp — the tool key that hands the agent the facade at all — must be in tools.sandbox.tools.alsoAllow, or the sandbox prunes it and the agent sees none of the facade tools. It's in the coding, messaging, and full presets for exactly this reason.

For completeness, a handful of skills were not migrated and still curl, on purpose: claw-pdf-tools (binary endpoints), claw-bind (lives under /api/v1/auth, uses a JWT/system token), and the four claw-app-* meta-skills (app authoring, an admin flow).

What I'd do differently

I'd ship the Fase 0 hardening as the foundation, before wiring any agent host-side. Connecting agents lit up vulnerabilities that network: none had been quietly containing. Deny-by-default and the C1/C2/C3 guards should have been the first commit, not a prerequisite we scrambled to close. Ordering security after exposure is how you ship a window.

I'd inject identity host-side from day one (ADR-11), not make the model copy a 64-hex token (ADR-10). The model-copies-the-token design was fragile against weak models and broke subagents entirely — sessions_spawn children got no token, which is the whole reason delegated work silently failed. A before_tool_call hook is both more robust and simpler. We took the long way there.

I'd declare the params schema shape up front. The empty-properties z.record schema sent a small model into a loop it could not escape — 84 calls, all params: {}. It read like a model failure and was a schema-shape bug. Give the model named fields and it fills them.

I'd sanitize tool schemas for RE2 from the start. A single .email() in one domain 400'd every Groq chat for every agent carrying that domain. A one-line validator emitting a lookahead took down whole conversations. Cheap to prevent, expensive to diagnose.

The takeaway

The agent never learned an API. It learned a vocabulary of typed tools — and every call still walks the same auth → scope → approval → audit hallway a human's request does. That's the entire trick: the facade is a translator, not a second backend. It reuses the whole pipeline via app.inject() and rewrites nothing. The AI can only touch what you allow, and everything it touches passes the exact permission checks and audit log as if you'd done it yourself — because, mechanically, it did.

Next up: #14 — Deterministic agent identity: injecting the per-agent token with a before_tool_call hook

Building a process manager for 40 services on 8GB of RAM

Victor García — Wed, 22 Jul 2026 07:56:55 +0000

At some point during development, I ran docker ps and counted 16 containers. Then there were the systemd services — PostgreSQL, SSH, the core server itself. Plus internal workers: the job scheduler, the async queue, the WebSocket server, the OpenClaw gateway. And Ollama, sitting quietly with a 4B model eating 5.4GB of VRAM.

40 processes. On a machine with 8GB of RAM.

Not all of them need to run simultaneously. The mail server needs to be always on. The PDF converter only needs to run when someone opens a document. Jellyfin only needs to run in the evenings. The voice STT model can sleep unless someone is talking.

This post is about the process manager that orchestrates all of that — a unified view of everything running on the system, a RAM budget engine that decides what fits, and lifecycle policies that start and stop services based on actual usage.

The unified view

The first problem was visibility. Docker containers report stats through the Docker API. Systemd services report through systemctl. Internal Node.js workers (scheduler, queue, WebSocket) only exist in the Claw Core process. Ollama has its own API for model management. Four different systems, four different interfaces.

We unified them into a single table with a consistent model:

interface ClawProcess {
  id: string;           // 'docker:claw-whisper', 'systemd:postgresql', 'internal:scheduler'
  name: string;         // 'Speech-to-Text (Whisper)', 'PostgreSQL', 'Job Scheduler'
  type: 'docker' | 'systemd' | 'internal';
  status: 'running' | 'stopped' | 'failed' | 'starting';
  cpu_percent: number | null;
  memory_bytes: number | null;
  gpu_percent: number | null;
  vram_bytes: number | null;
  uptime_seconds: number | null;
  restartable: boolean;
  stoppable: boolean;
  has_logs: boolean;
}

The id format type:name makes every process globally addressable. docker:claw-mailu-imap is the IMAP container. systemd:postgresql is the database. internal:async-queue is the background job processor. One namespace, one table, one API.

Three collectors feed the table:

Collector	Source	Method
Docker	Docker Engine API	`GET /containers/json` via Unix socket
systemd	systemctl	`list-units --type=service --output=json`
Internal	Claw Core process	In-memory state from scheduler, queue, WS server

Plus a special case: Ollama. It's a systemd service, but it also has loaded models that consume VRAM independently. The process manager queries Ollama's API (GET /api/ps) to show which models are loaded, their parameter count, and VRAM usage. You can unload a model directly from the process table to free VRAM.

The summary bar

The top of the process manager shows system-wide stats at a glance:

PROCESSES 16/40    CPU 0.5%    MEMORY 3205 MB    GPU 4%    VRAM 5579 / 16311 MB

16 out of 40 registered processes are currently running. Total CPU across all processes. Total memory. GPU utilization and VRAM usage (critical when you're running AI models locally).

The VRAM indicator is particularly useful. When Ollama loads qwen3.5:4b, it consumes 5.4GB of VRAM. If someone tries to load a second model and there isn't enough VRAM, the process manager shows exactly why. No more mysterious "out of memory" errors — the dashboard tells you what's using what.

Lifecycle policies: not everything needs to be always-on

The real power isn't the table view — it's deciding what runs and when. Each managed service has a lifecycle policy:

Policy	When it runs	Example services
Always-on	24/7, never auto-stopped	PostgreSQL, Claw Core, mail server, WireGuard, Tailscale
On-demand	Starts when the user navigates to the module, stops after idle timeout	ONLYOFFICE, Stirling PDF, Portainer, SolidInvoice
Scheduled	Runs during configured time windows or intervals	Jellyfin (08:00-01:00), RSS fetcher (every 60 min)
Triggered	Starts on a specific event, stops when the task completes	yt-dlp (download requested), claw-vision (photo uploaded)

In a real deployment, the breakdown looks like this: 5 always-on services, 22 on-demand, 8 scheduled. The 22 on-demand services are the key — they represent ~70% of total potential memory usage, but most of the time only 2-3 of them are actually running.

When you open the Office module in the dashboard, the system starts ONLYOFFICE (if it's not already running). The loading state shows a progress spinner. After 30 minutes of no document activity, ONLYOFFICE stops automatically. You never think about it.

Each service's policy is configurable in Settings → Services. Click on a service to expand its configuration: policy selector (always/scheduled/on-demand/disabled), schedule windows, timeout settings, RAM limits. Presets cover common patterns ("Diurnal" for services that run during working hours, "Periodic fetch" for services that sync on intervals).

The RAM budget engine

Here's the core problem: 40 services can't all run on 8GB of RAM. Something has to decide.

The RAM budget engine works with hardware profiles:

Profile	RAM	Budget for services	Max concurrent on-demand
Lite	≤ 6 GB	~1.5 GB	2
Standard	8-14 GB	~3.3 GB	4
Performance	16+ GB	~8 GB	8

The profile is auto-detected from system RAM and determines the budget — how much memory is available for managed Docker services (excluding always-on core services like PostgreSQL and Claw Core, which are non-negotiable).

Before starting any on-demand service, the engine checks:

Can this service fit in the remaining budget?
├── Yes → Start it
└── No → Is there an idle on-demand service we can evict?
    ├── Yes → Stop the longest-idle service, then start the new one
    └── No → Tell the user: "Not enough memory. Stop X to start Y."

Eviction targets the longest-idle on-demand service. Always-on services are never evicted — the system won't kill your mail server to make room for a PDF converter. The user always has the final say: the engine suggests what to evict, but it doesn't force it.

Under memory pressure (free RAM drops below 10%), the engine gets more aggressive: it progressively stops idle on-demand services until pressure is relieved, notifying the user via WebSocket for each one.

The RAM budget view in Settings shows this live:

RAM Budget: 260 / 3335 MB (8%)
├── wg-easy: 11 MB
├── mailu: 57 MB
├── wyoming-whisper: 177 MB
└── wyoming-piper: 15 MB

260MB used out of a 3,335MB budget. Room for several more on-demand services before the budget gets tight.

Drain guards: stopping safely

You can't just docker stop a service and hope for the best. ONLYOFFICE might have a document open. Qbittorrent might be mid-download. The terminal might have an active SSH session.

Drain guards check safety before stopping:

Service	Guard	What it checks
ONLYOFFICE	Document sessions	Are there open documents? → Auto-save first
Qbittorrent	Active downloads	Is anything downloading? → Wait or warn
Terminal	SSH sessions	Are there active sessions? → Warn user
Jellyfin	Active streams	Is anyone watching? → Warn user

If a drain guard blocks a stop operation, the user gets a notification: "ONLYOFFICE has 2 documents open. Save and close them, or force stop." Force stop is always available — but the system tries to prevent data loss by default.

Ollama: the special case

Ollama deserves its own section because it's neither a simple Docker container nor a systemd service — it's both. The systemd service runs the Ollama daemon. The models loaded into it consume VRAM independently of the daemon's RAM usage.

The process manager handles Ollama differently:

Shows the Ollama daemon as a systemd process with its own CPU/RAM stats
Below it, shows loaded models in an expandable section: model name, parameter count, RAM usage, VRAM usage
Each model has an "Unload" button that calls DELETE /api/generate to free VRAM
The summary bar includes VRAM as a separate metric from RAM

In our current setup, qwen3.5:4b sits loaded with 5.43GB of VRAM. When we need to load a different model (say, for a vision task), the process manager shows the VRAM impact before loading. If VRAM is full, you can unload the current model first.

The Ollama client in Claw Core (the priority queue singleton from post 2) manages model loading automatically — it loads models on demand and unloads them based on idle time. The process manager gives you visibility into what it's doing and manual override when you need it.

The log panel

Every process with has_logs: true gets a log viewer. Click the logs icon on any process row and a side panel opens with the last 200 lines, auto-scrolling, with a filter input for grep-style searching.

Docker logs come from the Docker API (GET /containers/{id}/logs). Systemd logs come from journalctl -u {service}. Internal workers log to pino (the Fastify logger) and their output is captured from the running process.

The filter is surprisingly useful for debugging. When Mailu's IMAP container is acting up, filtering for "error" or "failed" in the log panel is faster than SSH-ing into the server and running docker logs | grep.

Approval integration

Process lifecycle operations integrate with the approval system (post 11):

Operation	Approval level
View processes, logs	Level 0 (auto)
Start a service	Level 0 (auto)
Restart a service	Level 2 (confirm)
Stop a service	Level 2 (confirm)

Starting is unrestricted because the RAM budget engine already gates it — if there's no budget, it won't start. Restart and stop require confirmation because they affect running services. The agent can restart a crashed container, but only after asking "claw-mailu-imap is failing, should I restart it?"

What the numbers look like in practice

On our development machine (8GB RAM, RTX 2060):

Category	Count	Typical RAM
Always-on (core)	5	~400 MB
Always-on (mail)	6	~400 MB
Always-on (network)	3	~50 MB
Always-on (AI)	3	~200 MB
On-demand (active)	2-3	~300-500 MB
Systemd	3	~100 MB
Internal workers	5	~50 MB

Total running at any given time: ~15-20 processes, ~1.5-2 GB. Well within the Standard profile's 3.3GB budget. The other 20+ on-demand services wait in Docker's stopped state, consuming zero RAM.

Peak usage (everything on-demand activated simultaneously): ~4.5 GB. This never actually happens — it would require opening every module in the dashboard at the same time. The lifecycle policies ensure only what's needed is running.

What I'd do differently

I'd build the RAM budget engine before adding any managed services. We added services first (mail, PDF, office, media) and the budget engine later. During the gap, the machine would occasionally OOM because too many containers were running. The budget engine should have been the foundation, not an afterthought.

I'd add a "service groups" concept. Right now, Mailu is 6 separate containers (IMAP, SMTP, antispam, admin, Redis, frontend). They start and stop together, but the process table shows them individually. A group like "Mail Server" that expands to show its 6 components would reduce visual noise from 40 rows to ~20.

I'd expose the process table via the agent. Currently, the agent can restart and stop services via the HAL API, but it can't see the process table the way the dashboard shows it. A claw-processes skill with ?format=compact would let the agent answer "what's using all my RAM?" with actual data instead of guessing.

The takeaway

A self-hosted personal OS isn't one service — it's 40 services pretending to be one. The user should never think about which containers are running, how much RAM they're using, or when to start and stop them. That's the process manager's job.

Three components make it work: a unified process table (Docker + systemd + internal + Ollama, one namespace), a RAM budget engine (hardware profiles, eviction logic, pressure detection), and lifecycle policies (always-on, on-demand, scheduled, triggered).

The result: 40 services on 8GB of RAM, with the user seeing a single coherent system. The process manager is the plumbing that makes "personal OS" feel like an OS instead of a collection of Docker containers.

Next up: InsightFace face recognition on a mini-PC — how we built a face detection pipeline that clusters people across your photo library using a 2B model and a Python sidecar, all running on CPU.

Designing an AI approval system: when should your agent ask for permission?

Victor García — Tue, 07 Apr 2026 10:38:11 +0000

An AI agent that can only read data is safe but useless. An AI agent that can send emails, delete files, format disks, and configure VPNs is useful but terrifying. The entire value of a personal AI operating system comes from the agent acting on your behalf — and the entire risk comes from the same thing.

We needed a system that says "yes" fast to everyday operations and "are you sure?" to dangerous ones. Not a blanket confirmation on everything (that just trains the user to click "approve" without reading). Not unrestricted access either (one prompt injection away from rm -rf /).

This post is about the 4-level approval system we built, the dual-layer architecture (shell + API), and the surprisingly difficult design decision of where to draw the line between "just do it" and "ask me first."

The two attack surfaces

An AI agent in our system can cause damage in two completely different ways:

Shell execution. The agent uses the runtime's exec tool to run commands on the host machine. This is raw power — curl, ls, grep, but also rm, dd, python3 -c 'import os; os.system("...")'. The attack surface is the entire operating system.

API operations. The agent calls our REST API via curl. The curl command itself is harmless — it's the endpoint that's dangerous. POST /storage/pools creates a RAID array. DELETE /files/:id removes a file. POST /emails/send sends an email you can't unsend. The business logic is the attack surface.

These need separate control mechanisms because they have different risk profiles and different mitigation strategies.

Layer 1: Shell control

By default, the agent can only execute a small set of safe binaries:

Safe bins: curl, jq, cat, echo, date, wc, head, tail, grep

That's it. The agent talks to the outside world through curl to our REST API. Everything else — file manipulation, package installation, network commands, scripting — is blocked at the runtime level.

There's an "Unrestricted Shell Mode" toggle in Settings → Security. It's deliberately scary: the toggle is marked in red, requires the user's password (not just a click), and shows a warning explaining that this allows the agent to execute any command on the system.

Even in unrestricted mode, destructive commands (rm, dd, mkfs, fdisk) always require per-operation confirmation from the user. Full freedom doesn't mean no guardrails — it means the agent can attempt anything, but the user decides on dangerous operations.

The key design principle: most users never enable unrestricted mode. The agent does everything it needs through the API. Shell access is a power-user feature for people who know what they're doing.

Layer 2: Operation approvals

This is where it gets interesting. Every API operation has an approval level:

Level	Name	What happens
0	Auto	Execute immediately, no record. Reads, searches, listings.
1	Logged	Execute immediately, log to audit trail. Creates, updates, non-destructive writes.
2	Confirm	Pause and ask the user "Are you sure?" before executing. Sends, deletes, infrastructure changes.
3	Secure	Pause, ask for confirmation AND a numeric PIN. Format disk, delete volume, system reset.

The approval level is checked by a Fastify middleware (preHandler) that runs before the route handler. If the request comes from an agent token and the operation requires Level 2+, the middleware returns a 202 Accepted with an approval_id instead of executing the operation. The agent then asks the user for confirmation through whatever channel they're chatting on — Telegram, WhatsApp, or the web dashboard.

How it looks in practice

Level 0 — Auto (reading notes):

User: "What notes do I have about the project?"
Agent: GET /notes?search=project&format=compact
→ Executes immediately. User sees results.

No friction. Reads are always auto.

Level 1 — Logged (creating a note):

User: "Save a note about today's meeting decisions"
Agent: POST /notes {title: "Meeting decisions", content: "..."}
→ Executes immediately. Logged to audit trail.

Still no friction — the user asked for it. But the audit trail records that agent francis created note abc123 at 14:32. If something goes wrong, there's accountability.

Level 2 — Confirm (sending an email):

User: "Send Ana the budget update"
Agent: POST /emails/send {to: "ana@techcorp.com", subject: "Budget Q3", ...}
→ 202 Accepted {approval_id: "req_xyz"}

Agent: "I'm about to send this email to Ana García:
        Subject: Budget Q3
        [Preview of the body]
        Should I send it?"

User: "Yes"
Agent: POST /approvals/req_xyz/approve
→ Email sent.

One confirmation step. The agent shows what it's about to do. The user says yes or no. This catches the common case where the agent misunderstood the intent — "I said Ana, not María" — without making every email a five-step process.

Level 3 — Secure (deleting a volume):

User: "Delete the old backup volume"
Agent: POST /storage/volumes/vol_old/delete
→ 202 Accepted {approval_id: "req_abc", level: 3}

Agent: "⚠️ This will permanently delete volume vol_old (2.3TB).
        This cannot be undone.
        Please confirm with your security PIN."

User: "4829"
Agent: POST /approvals/req_abc/approve {pin: "4829"}
→ Volume deleted.

Two-factor: the user confirms AND enters their PIN. The PIN is a 4-6 digit numeric code set during initial setup, stored hashed with bcrypt. In messaging channels, the PIN message is deleted from chat history after verification (when the channel supports it).

The approval lifecycle

Every Level 2+ operation creates an approval_request record:

CREATE TABLE approval_requests (
    id              UUID PRIMARY KEY,
    user_id         UUID NOT NULL,
    requested_by    VARCHAR(100) NOT NULL,    -- 'agent:francis'
    operation       VARCHAR(200) NOT NULL,    -- 'POST /emails/send'
    level           SMALLINT NOT NULL,        -- 2 or 3
    summary         TEXT NOT NULL,            -- "Send email to Ana: Budget Q3"
    params          JSONB,                    -- Request body snapshot
    status          VARCHAR(20) DEFAULT 'pending',
    resolved_at     TIMESTAMPTZ,
    pin_verified    BOOLEAN DEFAULT false,
    expires_at      TIMESTAMPTZ NOT NULL,
    created_at      TIMESTAMPTZ NOT NULL
);

The lifecycle is:

Agent triggers operation
    ↓
Middleware: level >= 2? → Create approval_request (status: pending)
    ↓
Notify user via WebSocket (Dash) + Gateway RPC (Telegram/WhatsApp)
    ↓
User approves, rejects, or ignores
    ↓
├── Approved → Execute operation, status: approved
├── Rejected → Return error to agent, status: rejected
└── Timeout (30min) → Auto-expire, status: expired

Three timeout stages prevent approvals from hanging forever:

Stage	Default	What happens
Reminder	5 minutes	Send a reminder to the user
Escalation	15 minutes	Notify the system owner (if different from the user)
Expiry	30 minutes	Auto-reject the request

All three are configurable in Settings → Security → Approval Timeouts.

What's configurable and what isn't

The default levels are sensible but not everyone agrees on what's "dangerous." A power user who sends 50 emails a day wants email sending at Level 1 (logged, no confirmation). A cautious user wants it at Level 2 (confirm every one).

Settings → Security shows a table of operations with dropdown selectors:

Operation                    Default    Your Level
─────────────────────────────────────────────────
Create note                  Logged     [1 - Logged ▼]
Send email                   Confirm    [2 - Confirm ▼]
Delete files (permanent)     Confirm    [2 - Confirm ▼]
Enable VPN                   Confirm    [2 - Confirm ▼]
Add VPN peer                 Confirm    [2 - Confirm ▼]
Delete volume                Secure     [3 - Secure  ▼]
Format disk                  Secure     [3 - Secure  ▼]
System reset                 Secure     [3 - Secure  ▼]

Two constraints prevent dangerous configurations:

Level 3 operations can't go below Level 2. You can downgrade "Delete volume" from Secure (3) to Confirm (2), but not to Logged (1) or Auto (0). Destructive, irreversible operations always require at least one confirmation.
Level 0 operations can't be upgraded. Read operations are always auto. Making GET /notes require confirmation would break the system — the agent would need approval to answer "what notes do I have?"

Changing approval levels is itself a Level 2 operation — the system asks for confirmation before letting you change the security settings.

Why agents can't approve their own requests

This sounds obvious but it's the most important security decision in the system: an agent API key cannot approve an approval request. Only JWT tokens (human login via Dash) or system tokens can call POST /approvals/:id/approve.

If the agent could approve its own requests, a prompt injection attack could chain: trigger the operation → intercept the approval request → approve it. The human-in-the-loop is only meaningful if the human is the one doing the approving.

In messaging channels (Telegram, WhatsApp), the approval flows through the agent — the user says "yes" in the chat, and the agent calls the approve endpoint. But the approve endpoint verifies that the approval was triggered by a user message, not by the agent itself. The requested_by field records which agent requested the operation, and the same agent cannot resolve it.

The middleware: 15 lines that matter

The approval check is a Fastify preHandler that runs on every route with an assigned level:

// Simplified — the real version handles edge cases
async function approvalMiddleware(req, reply) {
  // Skip for human (JWT) requests — the Dash IS the confirmation
  if (req.authType === 'jwt') return;

  const level = getOperationLevel(req.method, req.url, req.userId);

  if (level <= 1) return; // Auto or Logged — proceed

  // Level 2 or 3: create approval request and pause
  const approval = await createApprovalRequest({
    userId: req.userId,
    requestedBy: `agent:${req.apiKeyName}`,
    operation: `${req.method} ${req.url}`,
    level,
    summary: buildSummary(req),
    params: req.body,
  });

  // Notify user
  await notifyApproval(approval);

  // Return 202 — the agent knows to ask the user
  reply.code(202).send({
    data: { approval_id: approval.id, level, summary: approval.summary },
    hint: level === 3 
      ? "Ask user to confirm with PIN" 
      : "Ask user to confirm"
  });
}

The hint field in the 202 response tells the agent's skill what kind of confirmation to request. Level 2: ask for a yes/no. Level 3: ask for the PIN.

Human requests from the Dash skip the middleware entirely. When you click "Send" in the email composer, you ARE the confirmation. The approval system only gates agent-initiated operations.

The skill: teaching the agent to ask

The claw-approvals skill teaches the agent how to handle the approval flow:

## Approval Protocol

When you receive a 202 response with an approval_id:
1. Show the user what you're about to do (use the summary field)
2. For Level 2: ask "Should I proceed?"
3. For Level 3: ask "Please confirm with your security PIN"
4. On "yes" or PIN: POST /approvals/{id}/approve (with pin if Level 3)
5. On "no" or "cancel": POST /approvals/{id}/reject
6. Never approve your own requests — always wait for user input

The skill also handles /pending (show pending approvals), /history (show past approvals), and edge cases like expired requests.

What I learned about where to draw the line

The hardest part wasn't building the system — it was deciding which operations go at which level. Some were obvious:

Read anything → Level 0. No debate.
Format disk → Level 3. No debate.

The middle ground is where every conversation happened:

Email sending: Level 1 or Level 2? We went with Level 2 (Confirm) as default. Email is irreversible — you can't unsend it. A misunderstood intent ("send Ana the budget" when you meant "draft Ana the budget") has real consequences. But we made it configurable because power users find the confirmation annoying.

Creating a note: Level 0 or Level 1? We went with Level 1 (Logged). Creating a note is harmless — but logging it means the audit trail shows everything the agent did. If the agent starts creating garbage notes due to a bug, the log tells you when it started. Zero friction, full accountability.

VPN operations: Level 2. Enabling or disabling VPN changes network topology. Adding a peer grants network access to a device. These aren't destructive (you can undo them) but they affect security posture. Confirmation is appropriate.

Deleting files: Level 2, not Level 3. Files have soft delete with restore capability. "Delete" doesn't mean "gone forever" — it means "moved to trash." If deletion were permanent (bypassing soft delete), it would be Level 3.

The general heuristic: reversible actions need less approval than irreversible ones. Creating is Level 1 (you can delete it). Sending is Level 2 (you can't unsend it). Destroying is Level 3 (it's gone forever).

The dual-layer interaction

The two layers (shell + API) work together:

User: "Check my disk space"

Agent: exec "df -h"
→ Layer 1: "df" is in safe bins → Execute immediately

Agent: GET /storage/overview?format=compact
→ Layer 2: Level 0 (read) → Execute immediately

Agent: "You have 450GB free on /data. The backup volume is 78% full."

No approvals needed — both layers allowed the operations silently.

User: "Set up a RAID1 mirror with the two new drives"

Agent: POST /storage/pools {name: "mirror1", level: "raid1", devices: ["sda", "sdb"]}
→ Layer 2: Level 2 (Confirm) → 202 Accepted

Agent: "I'm about to create a RAID1 pool using sda and sdb. 
        This will erase all data on both drives. Confirm?"

User: "Yes"
→ Pool created.

And in unrestricted shell mode:

User: "Install htop on the server"

Agent: exec "apt-get install htop"
→ Layer 1: Not in safe bins + unrestricted mode → Ask for confirmation

Agent: "I want to run: apt-get install htop
        This will install a package on your system. OK?"

User: "Go ahead"
→ Layer 1: Approved → Execute

Both layers enforce independently. An operation that passes Layer 1 (shell) can still be blocked by Layer 2 (API). An operation that bypasses Layer 2 (because it's a direct shell command) is still caught by Layer 1.

What I'd do differently

I'd add batch approvals from the start. When the agent needs to send 15 emails from a mail merge, asking for 15 individual confirmations is absurd. A "batch approve" mechanism ("Send these 15 emails? Here's the list.") should have been in v1. It's now on the backlog.

I'd make the approval history more visible. The audit trail exists (GET /approvals/history), and there's an "Approvals History" section in the sidebar, but it should be more prominent. A weekly summary — "Your agent executed 340 operations this week: 280 auto, 55 logged, 5 confirmed" — would build trust and help users understand their agent's activity.

I'd reconsider the messaging channel PIN flow. Sending a PIN via Telegram is not ideal — it's visible in chat history even if the agent tries to delete the message. For Level 3 operations, maybe the system should redirect to the Dash where a secure input modal exists, rather than accepting PINs in plain text through messaging.

What's next: the sandbox

There's a missing piece between "restricted mode" (curl only) and "unrestricted mode" (everything, with confirmations). What if the agent could have a place to go completely wild — install packages, run services, break things — without any risk to your real system?

We're planning sandbox environments: Docker containers that the agent can create from Settings → Security. Not one — as many as you need. Each sandbox is an isolated machine with full root access. apt install, pip install, systemctl, custom scripts, databases, web servers — anything goes. Zero approval, zero restrictions, zero risk to the host.

The workflow we're designing around:

Sandbox "deploy-test"     → experimenting with nginx + certbot config
Sandbox "ml-pipeline"     → building a data processing pipeline with pandas
Sandbox "new-skill"       → developing and testing a new agent skill

Each one: independent, disposable, unrestricted.

The agent knows which sandbox it's working in. Commands routed to a sandbox go to that container. Commands on the real system go through normal approval layers. The two worlds don't touch — no shared volumes, no network bridge to internal services, no mount to /data.

The key is the promote-to-production flow. Once you've got something working in the sandbox — a configuration, a script, a service setup — you tell the agent "promote this to production." At that point, and only at that point, normal approval rules kick in. The agent needs Level 2 confirmation to copy files to the host, Level 2 to install a package on the real system, Level 3 to modify infrastructure. The sandbox is the drafting table; production is the real thing.

If a sandbox gets trashed, nuke it and spawn a fresh one in seconds. The sandboxes are cheap — a base Debian image with internet access and a persistent volume for the workspace. Multiple sandboxes can run simultaneously for different experiments without interfering with each other or with the host.

This isn't implemented yet, and there are open design questions: should sandboxes have read-only access to the real API (for testing skills against real data)? Should there be resource limits per sandbox (CPU, RAM, disk)? What's the UX for promoting — file-by-file or snapshot the whole container? We'd love input from anyone who's built agent sandboxing — this is genuinely uncharted territory for personal AI systems.

The takeaway

An AI approval system needs two properties: it must be fast for safe operations (no friction on reads, minimal friction on writes) and deliberate for dangerous ones (explicit confirmation, PIN for irreversible actions, timeout for stale requests).

Four levels handle this: auto (reads), logged (writes), confirm (irreversible), secure (destructive + PIN). Two layers: shell control for host-level commands, API control for business operations. One principle: agents cannot approve their own requests.

The system processes ~95% of operations at Level 0 or 1 — invisible to the user. The 5% that require confirmation are the operations where a mistake actually matters: sending an email to the wrong person, deleting a volume, configuring network access. Those 5% are where trust is built or broken.

Next up: building a process manager — how we manage Docker containers, systemd services, and Ollama models from a single dashboard with auto-start, health monitoring, and graceful shutdown.

PII-aware routing: how to use cloud AI and keep your sensitive data local

Victor García — Fri, 27 Mar 2026 14:23:21 +0000

Here's the tension at the heart of every personal AI system: cloud models are better at reasoning, but your data is private. A self-hosted system can run everything locally — but a 2B parameter model on a mini-PC isn't going to draft a nuanced email response or analyze a complex financial situation the way a frontier model can.

The naive solutions are both bad. "Send everything to the cloud" means your diary entries, medical notes, and financial records pass through someone else's servers. "Run everything locally" means accepting worse reasoning on tasks where model quality actually matters.

We built a third option: a PII-aware routing layer that classifies every piece of data by sensitivity, routes it to the right model, and pseudonymizes anything sensitive that needs cloud reasoning power.

The classification: four levels, zero LLM calls

Every record in the system gets a sensitivity level. The classification is entirely deterministic — regex patterns and domain rules. No LLM in the classification loop, because sending data to an LLM to decide if the data is too sensitive to send to an LLM is a circular problem.

Level	What it means	Example domains
`low`	Public or low-risk data	Events, bookmarks
`normal`	Common personal data	Notes, contacts, files, diary
`high`	Sensitive personal data	Emails, financial transactions
`critical`	Never leaves the device	Medical/health data

Each domain has a default sensitivity level. Events are low — knowing you have a meeting at 3pm isn't particularly sensitive. Emails are high — they contain names, addresses, business context, and sometimes confidential information. Health entries are critical — always local, no exceptions.

But domains are just the baseline. The classifier also scans content for PII patterns that override the default:

Email addresses     → elevate to high minimum
Phone numbers       → elevate to high minimum
Credit card numbers → elevate to high minimum
IBAN codes          → elevate to high minimum
SSN / DNI / NIE     → elevate to high minimum
Medical terminology → elevate to critical

A note titled "Grocery list" stays at normal. A note containing "Dr. García prescribed 20mg omeprazole" gets elevated to critical because the regex matched medical terminology. The content drives the classification, not just the domain.

This is deliberately conservative. The regex patterns over-match — "Dr." triggers medical detection even if it's "Dr. Pepper." False positives mean data gets routed locally when it could have gone to the cloud. False negatives mean sensitive data leaks. Over-matching is the correct failure mode.

The routing decision

Once classified, the router decides where each piece of data goes:

low / normal  → Cloud LLM — best reasoning
high          → Cloud LLM WITH pseudonymization — good reasoning, protected data
critical      → Local model only (Ollama) — or skip if Ollama unavailable

The decision isn't binary "local vs cloud." There's a middle path: pseudonymize the sensitive parts, send to the cloud for reasoning, and de-pseudonymize the response before the user sees it.

This matters because most tasks involving sensitive data don't need the sensitive parts for reasoning. "Summarize this email thread" needs the content structure and topic — not the actual names and email addresses. "What's the sentiment of this diary entry?" needs the emotional content — not the specific people mentioned.

The pseudonymizer

When a high sensitivity record needs cloud processing, the pseudonymizer replaces PII with consistent tokens:

Entity type	Pseudonym format	Example
Person	`Person_XXXX`	"Ana García" → `Person_A3F2`
Email	`email_XXXX@example.com`	"ana@techcorp.com" → `email_7B1C@example.com`
Phone	`+00-XXXX-0000`	"+34 612 345 678" → `+00-E5D9-0000`
Organization	`Org_XXXX`	"TechCorp" → `Org_4C8A`
Location	`Location_XXXX`	"Calle Sagasta 15" → `Location_B2E1`

Three properties make this work:

Consistency. The same value always produces the same pseudonym (SHA-256 of the original value, truncated). "Ana García" is always Person_A3F2, in every record, in every session. This means the cloud model can reason about relationships: "Person_A3F2 sent 3 emails to Person_B7D1 about Org_4C8A" preserves the structure even though the names are hidden.

Reversibility. The pseudonym_map table stores every mapping. When the cloud model's response comes back, the system replaces all pseudonyms with real values before storing or displaying the result. The user never sees Person_A3F2 — they see "Ana García."

Persistence. Mappings survive across sessions. If "Ana García" was pseudonymized yesterday and appears again today, she gets the same pseudonym. This means the cloud model can build consistent context across multiple interactions without ever learning the real name.

The detection itself uses regex — no LLM call. It's the same NER-lite approach as the sensitivity classifier: pattern matching for emails, phones, card numbers, and named entity patterns for persons and organizations. Not perfect, but fast and deterministic.

What this looks like in practice

Scenario 1: Calendar event (low sensitivity)

User asks: "What's on my calendar tomorrow?"

The system fetches tomorrow's events. Events are low sensitivity. The full data — titles, locations, attendees — goes straight to the cloud model. No pseudonymization needed. The model reasons about the schedule and responds with a natural summary.

Cost: one cloud API call. Privacy: no sensitive data exposed.

Scenario 2: Email analysis (high sensitivity)

User asks: "Summarize the email thread about the partnership."

The email thread is high sensitivity (default for emails). Before sending to the cloud model:

Original: "Ana García <ana@techcorp.com> wrote: Hi Paco, 
regarding the TechCorp partnership with NexaTech..."

Pseudonymized: "Person_A3F2 <email_7B1C@example.com> wrote: 
Hi Person_0D4E, regarding the Org_4C8A partnership with Org_9F3B..."

The cloud model receives the pseudonymized version. It can still analyze the thread structure, identify that Person_A3F2 is negotiating with Person_0D4E, and summarize the key points. The reasoning quality is nearly identical — the model doesn't need to know the real names to understand the negotiation dynamics.

The response comes back with pseudonyms:

"Person_A3F2 proposed a revenue-sharing model with Org_9F3B. 
Person_0D4E agreed in principle but requested..."

The system de-pseudonymizes:

"Ana García proposed a revenue-sharing model with NexaTech. 
Paco agreed in principle but requested..."

Cost: one cloud API call + ~2ms pseudonymization. Privacy: no real names or emails left the device.

Scenario 3: Health data (critical sensitivity)

User asks: "What medications am I taking?"

Health entries are critical. They never leave the device, period. The system routes to the local Ollama model. If Ollama is unavailable, the query fails gracefully — it does NOT fall back to the cloud.

The local model's response might be less polished, but for medical data retrieval, the task is usually simple: find the records and list them. A 2B model handles that fine.

Cost: one local model call. Privacy: absolute — zero data exposure.

Scenario 4: Note with accidental PII (elevated sensitivity)

User creates a note: "Meeting with Dr. López about the lab results. Blood pressure 140/90."

The note's domain is normal, but the content contains medical terminology ("Dr.", "lab results", "blood pressure"). The classifier elevates it to critical. From this point on, this note is treated like health data — local only.

The user didn't tag it as medical. They didn't configure anything. The system caught it automatically. Conservative false positives are the design choice: if a note mentions "Dr. Pepper," it gets elevated too. That's a minor inconvenience (one note processed locally instead of on the cloud) with zero privacy risk.

The audit trail

Every routing decision is logged:

Field	What it records
`domain`	Which data domain (notes, emails, etc.)
`record_id`	Which specific record
`sensitivity`	Classified sensitivity level
`action`	What happened: `sent_pseudonymized`, `sent_plain`, `blocked`
`destination`	Where it went: `embeddings`, `contextual_retrieval`, `sleep_time`

The pii_routing_log table creates a complete audit of what data was exposed to which processing pipeline. If you ever need to answer "did my medical data ever touch a cloud service?", the answer is in the log.

This is also how we verify the system works correctly. The log shows every routing decision. If a critical record ever appears with action sent_plain and a cloud destination, that's a bug — and the log caught it.

Where routing applies

PII-aware routing isn't just for chat interactions. It applies everywhere the system sends data to an LLM:

Embeddings. When generating semantic embeddings, the text is classified before being sent to the embedding model. If you're using a cloud embedding API (future option), high and critical records get embedded locally via Ollama instead.

Contextual retrieval. The HyDE pipeline (generating hypothetical answers for better search) uses LLM calls. If the search touches sensitive domains, those calls route through the pseudonymizer.

Sleep-time compute. The background intelligence jobs process records during idle periods. The enrichment job (re-extracting entities from hot records) respects the same routing rules — a critical record only gets re-extracted if Ollama is available.

Entity extraction. When the CRUD hooks pipeline sends text to the LLM for entity extraction, the same classification applies. A health-related note gets extracted locally.

The routing layer sits between every LLM consumer in the system and the actual model call. It's middleware — invisible to the features that use it, enforced consistently everywhere.

The multi-agent dimension

With a multi-agent topology (7 agents in our system), PII routing gets another layer: agent scoping.

Each agent has a scoped token that defines what domains it can access:

Francis (main):     notes:*, events:*, emails:*, contacts:*, diary:*
Sentinel (infra):   storage:*, hal:*, network:*
Dalí (creative):    photos:read, files:read
Ledger (finance):   finance:*, crm:*
Darwin (analytics): graph:*, insights:*

Sentinel can't access emails. Dalí can't read the diary. This is enforced at the API level — even if a prompt injection tricks Dalí into requesting diary entries, the scoped token blocks it.

Combined with PII routing, this creates defense in depth:

Agent scoping prevents access to domains the agent shouldn't touch
Sensitivity classification catches PII regardless of domain
Pseudonymization protects data that needs cloud processing
Audit logging records everything for verification

A prompt injection attack would need to bypass all four layers to exfiltrate sensitive data. The scoping blocks the API call. The classification catches the content. The pseudonymizer strips the PII. The audit log records the attempt.

What we explicitly don't do

We don't use ML for classification. A fine-tuned classifier could be more accurate than regex patterns. But it would need to see the data to classify it — which means sending potentially sensitive data to a model before deciding if it's safe to send to a model. Regex is dumber but has zero data exposure during classification.

We don't redact — we pseudonymize. Redaction ([REDACTED]) destroys information the cloud model needs for reasoning. Pseudonymization preserves structure ("Person_A sent an email to Person_B") while hiding identity. The cloud model can still reason about relationships, quantities, and patterns.

We don't let the user override critical. You can change a record's sensitivity from normal to high manually. You cannot downgrade critical to anything else. Health data stays local regardless of user preferences. This is a deliberate paternalistic choice — the privacy risk of accidentally exposing medical data outweighs the convenience of sending it to a better model.

We don't route based on the LLM provider's privacy policy. Whether provider A's privacy policy is better than provider B's is irrelevant. The system treats all cloud LLMs identically: external services that should never see critical data and should only see high data in pseudonymized form. Trust the math, not the terms of service.

What I'd do differently

I'd add per-field sensitivity, not just per-record. Currently, a contact record is normal even though the phones field is arguably more sensitive than the company field. Per-field classification would let us pseudonymize just the phone number while sending the company name to the cloud. More precise, but also more complex — the pseudonymizer would need to understand JSON field structure.

I'd build a sensitivity dashboard earlier. The pii_routing_log has all the data, but there's no UI for it yet. A dashboard showing "this week: 450 records processed, 380 sent plain, 65 pseudonymized, 5 blocked" would build user trust and make the privacy system tangible.

I'd make the regex patterns configurable. Different users have different sensitivity needs. A doctor might want "aspirin" to be flagged as medical. A pharmacist might want it treated as normal. The current patterns are one-size-fits-all, which means they're too aggressive for some users and not aggressive enough for others.

The takeaway

The privacy problem in personal AI isn't "local vs cloud." It's "which data goes where." Most of your data is fine to send to a cloud model — your calendar events and bookmark titles aren't secrets. Some data needs protection but can still benefit from cloud reasoning — pseudonymize it and send the structure without the identity. A small fraction of data should never leave your device — route it locally and accept the quality trade-off.

Three components: a regex classifier (zero LLM calls, deterministic), a SHA-256 pseudonymizer (consistent, reversible, persistent), and a routing table (domain defaults + content elevation). No ML, no fine-tuning, no privacy policy trust assumptions.

The system processes your medical notes with a 2B local model and your calendar queries with a cloud model. It knows the difference because a regex matched "blood pressure" — not because it asked an AI what's sensitive.

Next up: designing an AI approval system — when should your agent ask for permission, and how do you build a confirmation workflow that doesn't slow everything down?

Sleep-time compute for personal data: what your AI should do while you sleep

Victor García — Thu, 26 Mar 2026 12:01:52 +0000

Your personal AI assistant sits idle most of the day. You send it a message, it responds, then it waits. For hours. Maybe all night. The compute is there — the model is loaded, the database is running, the server is warm. But nothing happens until you type the next message.

That's test-time compute: work done when the user asks for it. Letta's research (arXiv:2504.13171) showed that shifting processing to idle periods — sleep-time compute — achieves 5× fewer tokens at test time and 15% more correct answers. But their implementation only processes conversation memory. Nobody had applied it to structured personal data.

We did.

The idea

Instead of the agent doing all its thinking when you ask a question, it does most of the thinking in the background — during idle periods when you're not using the system. When you finally ask "what's going on with Project Tempest?", the answer is already half-assembled.

The system maintains four background jobs that run every 30 minutes, but only when you're idle:

Priority	Job	What it does
1	Enrich connections	Finds hot records with few graph links, re-runs entity extraction
2	Generate summary	Compiles a weekly overview from the change log
3	Detect patterns	Discovers entities that co-occur but aren't linked
4	Update preferences	Learns behavioral patterns from your data

Each job consumes tokens from a configurable budget (default: 5,000 tokens per execution). When the budget runs out, lower-priority jobs get skipped. This means enriching connections (the most impactful job) always runs, while preference learning (the least time-sensitive) gets skipped first if resources are tight.

The trigger: idle detection

The engine only runs when you're not using the system. If you're actively writing notes or reading emails, the background jobs wait. This matters for two reasons:

Resource contention. The LLM (whether local via Ollama or remote via API) is a shared resource. Background jobs competing with user requests for model access would add latency to the interactions you actually care about.
Relevance. Sleep-time processing works on data that has settled. Running entity extraction on a note you're still editing wastes tokens — the note will change again in 30 seconds. Waiting until you're idle means processing stable data.

The idle detector is simple: if no user activity (API requests from the dashboard, agent messages, WebSocket heartbeats) has occurred in the last N minutes, the user is idle. The scheduler checks this condition before executing each run.

Job 1: Enrich connections

The highest-priority job. It finds records that are "hot" (recently accessed, heat score > 0.3) but poorly connected in the knowledge graph (fewer than 3 entity links). These are records you care about but that the system doesn't fully understand yet.

Here's what the query returned on a real run:

Optimal Fuse Burn Rate Calculations v3    heat: 0.5  links: 2
Recipe: Rodney's Smoked Eyebrows Marinade heat: 0.5  links: 2
Banned Substances List (and Why Rodney…)  heat: 0.5  links: 2

Three notes flagged. Each has a warm heat score (recently accessed) but only 2 entity links where the average for this user is 5+. The initial extraction caught the obvious entities, but a second pass might find connections to people, projects, or locations that were mentioned implicitly.

The engine re-enqueues them to the async extraction pipeline at priority: low — they won't compete with real-time user actions. When the extraction worker picks them up, it sends the full note content to the LLM for a more thorough entity pass than the initial CRUD hook provides.

Why prioritize this job? Because the knowledge graph is the foundation of search ranking, the digest engine, and the agent's contextual awareness. A poorly connected hot record means the system is blind to something you're actively working on. Enriching it improves everything downstream.

Cost on this run: 600 tokens (3 records × ~200 tokens each). Execution time: 16ms (just the enqueue — the actual extraction happens later).

Job 2: Generate summary

Aggregates a week of change log activity and the most active entities from the knowledge graph into a single pre-computed insight.

On this run, the change log query returned:

files:        201 inserts, 5 updates, 152 deletes
rss:          141 inserts
emails:       61 inserts, 32 updates, 7 deletes
notes:        2 updates
kanban_cards: 1 update

And the top graph entities by recent activity:

Madrid              (location)      28 mentions
micelclaw           (organization)  12 mentions
Meta Platforms, Inc.(organization)   7 mentions
Instagram           (location)       6 mentions
Victoria            (person)         1 mention

Two things jump out. First: 201 file inserts and 152 file deletes in one week — that's a bulk operation or a sync cycle, not manual activity. The summary captures this so the agent can mention it if asked "what happened this week?" without scanning 600+ change log rows at query time.

Second: "Instagram" classified as a location is an entity extraction error — the kind of noise the enrichment job (Job 1) and the merge candidates system are designed to catch over time.

The summary gets stored as a weekly_summary insight with a 7-day TTL. No LLM call needed — it's pure SQL aggregation.

Cost on this run: 50 tokens (fixed). Execution time: 14ms.

Job 3: Detect patterns

The most interesting job. It self-joins entity_links to find pairs of entities that co-occur in 3 or more records but have no direct link between them — latent connections nobody made explicit.

On this run, five patterns emerged:

Rodney ↔ Dolores     co-occur in 12 records, no direct link
Rodney ↔ Linda       co-occur in 11 records, no direct link
Rodney ↔ BoomClaw    co-occur in 10 records, no direct link
Warehouse B ↔ Rodney co-occur in 10 records, no direct link
Benny ↔ Rodney       co-occur in 10 records, no direct link

Every pattern radiates from "Rodney" — he's a hub entity that appears alongside four other entities across 10-12 records without any direct graph edge connecting them. The extraction pipeline created links from each note/email to "Rodney" and to "Dolores" independently, but never linked Rodney to Dolores directly. The co-occurrence pattern reveals the relationship that was hiding in plain sight.

Each pair becomes a connection_discovered insight with a 14-day TTL. The next time you search for "Rodney," the graph traversal finds Dolores, Linda, BoomClaw, Warehouse B, and Benny — even though no single record ever says "Rodney works with Dolores."

This is the job that produces the "how did it know that?" moments. The answer is always the same: it counted co-occurrences while you weren't looking.

The query itself — a self-join on entity_links filtered by NOT EXISTS — took 138ms. That's the heaviest operation in the pipeline, and it runs during idle time where nobody notices. At query time, the connections are already in the graph.

Cost on this run: 30 tokens (fixed — pure SQL, no LLM).

Job 4: Update preferences

The system learns behavioral patterns by analyzing your data over time. On this run, two patterns were detected:

Writing time distribution (last 30 days): All 50 notes created at hour 12 UTC. That's not a preference — that's a signal so strong it maxed out confidence immediately. The system now knows that if you're going to write a note, it's probably at noon.

Tag frequency:

safety: 11, personal: 9, r-and-d: 6, strategy: 5, humor: 5

Both get persisted via UPSERT with incremental confidence — each observation nudges the score up by 0.05, capped at 0.95. After multiple runs, the preferences look like this:

scheduling / preferred_writing_hour = "12"
  confidence: 0.95 (max), evidence: 20,650 observations

organization / preferred_tags = ["safety","personal","r-and-d","strategy","humor"]
  confidence: 0.95 (max), evidence: 417 observations

The agent uses these when it needs to make decisions. Scheduling a reminder? It knows noon is when you're active. Suggesting tags for a new note? It offers your most-used tags first. Creating a diary entry template? It matches your writing style.

If a preference is wrong, you delete it via the API. The system may re-learn it later if the pattern persists, but with reduced confidence — the deletion counts as negative feedback.

Cost on this run: 20 tokens (fixed — pure SQL, no LLM).

The real numbers: 700 tokens, 184 milliseconds

Here's the actual pipeline summary from the run above:

┌────────────────────┬────────┬────────┬───────────────────────────────┐
│ Job                │ Tokens │ ms     │ Output                        │
├────────────────────┼────────┼────────┼───────────────────────────────┤
│ enrich_connections │ 600    │ 16     │ 3 notes re-enqueued           │
│ generate_summary   │ 50     │ 14     │ 1 weekly_summary insight      │
│ detect_patterns    │ 30     │ 138    │ 5 connection_discovered       │
│ update_preferences │ 20     │ 16     │ 2 preferences updated         │
├────────────────────┼────────┼────────┼───────────────────────────────┤
│ Total              │ 700    │ 184    │                               │
└────────────────────┴────────┴────────┴───────────────────────────────┘

700 out of the 5,000 token budget — 14%. The full pipeline completed in under 200 milliseconds. Three of the four jobs are pure SQL with zero LLM calls. Only enrich_connections queues work for the model, and even that just enqueues — the actual extraction runs later at low priority.

Every execution gets logged to sleep_time_jobs for auditability. If a job fails, the error is recorded and the next job still runs — the pipeline is fault-tolerant by design.

The token budget: sleep-time vs test-time

Every sleep-time execution has a capped budget — 5,000 tokens by default, configurable per user. This prevents runaway costs from background processing. The jobs run in priority order and stop when the budget is exhausted.

The insight from Letta's research holds: spending tokens during idle time dramatically reduces what you need to spend during active conversations. When the agent already knows that Rodney is connected to Dolores across 12 records (because Job 3 discovered it overnight), answering "who works with Rodney?" costs a graph traversal query (~5ms) instead of a full cross-domain LLM analysis (~3,000 tokens).

We track sleep-time and test-time token usage separately in the token metrics dashboard, so you can see the trade-off directly: more sleep-time tokens → fewer test-time tokens → faster, cheaper responses.

The three-stage digest

The sleep-time engine powers an evolved version of the Digest Engine — the system that tells the agent "here's what changed since you last checked."

The original digest was simple: scan the change log, format a markdown file, write it to the agent's workspace. The agent reads it on the next heartbeat.

The v2 digest is a three-stage pipeline:

Stage 1 — Selection. Filter changes by relevance using configurable rules stored in a digest_rules table. VIP emails (from your boss, from specific contacts) trigger immediate notification via PostgreSQL LISTEN/NOTIFY. Routine changes (a synced contact updated its phone number) get buffered for the periodic digest.

Stage 2 — Correlation. Use the LLM to discover cross-domain connections between changes. "You received an email from Ana García. Ana is attending tomorrow's meeting. You have 2 unfinished notes about the project she's working on." This stage is why sleep-time matters — the correlation discovery happens in the background, not when the agent is trying to respond to you.

Stage 3 — Scoring. Rate each item by urgency, cross-domain relevance, and historical feedback (did the user act on similar insights before?). The output shifts from "what changed" to "what matters and why."

The scored digest gets written to DIGEST.md in the agent's workspace. The agent reads it and decides what to surface. Urgent items might trigger an immediate notification. Low-score items accumulate for a daily summary.

The world model

One output of the sleep-time engine is a materialized "world model" — a living document that summarizes your current state:

Active projects and their status
Key people and recent interactions
Upcoming deadlines and events
Behavioral patterns and preferences

The world model is updated incrementally. Each sleep-time run only modifies the sections affected by recent changes. The agent references it as persistent context — a pre-computed summary of "what's going on in your life right now" that doesn't need to be recomputed every conversation.

This is inspired by Daniel Miessler's PAI framework (MISSION.md, GOALS.md, PROJECTS.md pattern), adapted to structured data. Instead of the user maintaining these documents manually, the system generates them from real data.

Zero cost when idle

The most important design decision: when nothing has changed, the engine does nothing. Zero tokens. Zero queries. The scheduler checks for pending changes in the change log before executing any job. No changes → skip the entire run.

This means the system's cost is proportional to your activity, not to time. A weekend where you don't use the system costs nothing. A busy Monday with 50 emails and 10 notes triggers multiple enrichment passes. The cost follows the value.

Similarly, the digest delivery to the agent is conditional. No changes → no DIGEST.md written → no system event → the agent doesn't wake up → zero tokens consumed. This was a deliberate choice over a heartbeat model where the agent would check for updates periodically regardless.

What I'd do differently

I'd add entity type validation in the summary job. The real run showed "Instagram" classified as a location — a clear extraction error that propagated into the weekly summary. A simple validation step (is this entity type plausible for this name?) would catch obvious misclassifications before they pollute insights. The data exists to fix this; we just haven't built the filter yet.

I'd build the pattern detection job first, not the enrichment job. Enrichment (Job 1) improves the knowledge graph incrementally. Pattern detection (Job 3) produces visible, surprising insights that users actually react to. "Ana García is connected to Project Tempest" is a moment of delight. "We added 2 more entity links to your note about Tempest" is invisible maintenance. Leading with delight would have made the sleep-time engine feel valuable sooner.

I'd make the token budget adaptive. Right now it's a flat 5,000 tokens per run. A smarter approach: scale the budget with the amount of pending work. 3 new records → 1,000 tokens. 50 new records after a sync → 10,000 tokens. The budget should match the opportunity, not be a fixed ceiling.

I'd add a "sleep-time log" visible in the dashboard. Currently, the only way to see what the engine did is through the insights API and the sleep_time_jobs table. A visible log ("Last night I discovered 3 new connections, updated 2 preferences, and generated your weekly summary") would build trust and make the background processing feel tangible.

The takeaway

A personal AI system that only works when you talk to it is wasting 95% of its available compute. The data is sitting in PostgreSQL. The model is loaded in Ollama. The knowledge graph has gaps that a 30-token LLM call could fill. Why wait for the user to ask?

Sleep-time compute shifts the work from "the user asked a question and now we scramble" to "we already know the answer because we connected the dots overnight." Four jobs, a token budget, an idle detector, and a three-stage digest pipeline. The system gets smarter while you sleep.

The insight that makes it all work: spending tokens when nobody is waiting for a response is categorically cheaper — in latency, in user experience, and in total cost — than spending them when someone is staring at a loading spinner.

Next up: PII-aware routing — how we send sensitive data to local models and everything else to the cloud, without the user having to think about it.

Hybrid search with RRF: combining pgvector, tsvector, and a knowledge graph in one query"

Victor García — Tue, 24 Mar 2026 12:04:03 +0000

Here's a search query: "beach trip."

Full-text search finds nothing — no record contains the word "beach." But there's a note that says "Qué calor en Valencia, el agua estaba perfecta." Semantic search finds it because the embedding for "beach trip" is close to the embedding for a hot day at the beach in Valencia.

Now a different query: "Ana García."

Semantic search returns a dozen vaguely related records. Full-text search returns the 3 records that literally contain "Ana García." But neither shows you that Ana attended last week's meeting, is CC'd on 5 email threads, and appears in tomorrow's calendar — connections that only the knowledge graph knows about.

No single search method is enough. We needed all three, plus a way to combine them that doesn't require manual tuning.

The four signals

Our search pipeline produces four independent scores for every candidate result:

Signal	Source	What it catches	Tier
Semantic	pgvector cosine similarity	Meaning-based matches ("beach" → "calor en Valencia")	Pro
Full-text	tsvector + GIN + ts_rank	Exact keyword matches, fast and precise	Free
Graph	entity_links overlap	Relational connections ("Ana García" → meetings she attended)	Pro
Heat	record_heat table	Temporal relevance (recently accessed records)	Free (display), Pro (in ranking)

Free tier users get full-text search only — which is still fast and well-ranked thanks to tsvector with weighted columns (title gets weight A, content gets weight B, tags get weight C). Pro users get all four signals fused together.

The pipeline

The search happens in seven steps:

Query: "Ana García project update"
    │
    ├── 1. Vector search ──→ top-50 by cosine similarity
    ├── 2. Full-text search ──→ top-N by ts_rank (UNION ALL across domains)
    └── 3. Graph discovery ──→ N candidates via entity_links
                │
                ▼
         4. Deduplicate by (domain, record_id)
                │
                ▼
         5. Rank-normalize each signal to [0, 1]
                │
                ▼
         6. Detect degenerate signals
                │
                ▼
         7. Weighted fusion + multi-signal bonus
                │
                ▼
            Final ranked results with provenance

Let me walk through each step.

Step 1: Vector search

The query text is embedded on-the-fly using the same model that embeds records (qwen3-embedding:0.6b, 1024 dimensions). Then a cosine similarity query runs against the embeddings table:

SELECT domain, record_id, 1 - (embedding <=> $query_embedding) AS similarity
FROM embeddings
WHERE 1 - (embedding <=> $query_embedding) >= 0.3
ORDER BY embedding <=> $query_embedding
LIMIT 50;

The 0.3 minimum threshold filters garbage. The top 50 candidates move to the next step. If Ollama is down and we can't embed the query, this signal is simply skipped — the other signals still work.

Step 2: Full-text search

A UNION ALL query across all domain tables, using PostgreSQL's native full-text search:

SELECT 'note' AS domain, id AS record_id, ts_rank(search_vector, query) AS score
FROM notes
WHERE search_vector @@ plainto_tsquery('simple', $q) AND deleted_at IS NULL
UNION ALL
SELECT 'event', id, ts_rank(search_vector, query)
FROM events
WHERE search_vector @@ plainto_tsquery('simple', $q) AND deleted_at IS NULL
UNION ALL
-- ... contacts, emails, files, diary, bookmarks, kanban_cards
ORDER BY score DESC
LIMIT 50;

We use plainto_tsquery('simple', ...) instead of language-specific configurations. The simple configuration doesn't stem words, which matters for multilingual data — Spanish and English records coexist, and stemming rules for one language would butcher the other.

Each domain table has a search_vector tsvector column maintained by a trigger (or GENERATED ALWAYS AS ... STORED for newer tables). The vectors are weighted: title gets 'A', description/content gets 'B', tags get 'C'. A match in the title ranks higher than a match in the body.

Step 3: Graph discovery

This signal is different — it doesn't match text, it matches relationships.

The query is matched against graph_entities.normalized_name. If "Ana García" matches a Person entity, we find all records linked to that entity via entity_links:

-- Find entities mentioned in the query
SELECT id FROM graph_entities
WHERE normalized_name ILIKE '%ana garcia%' AND deleted_at IS NULL;

-- Find all records linked to those entities
SELECT source_type AS domain, source_id AS record_id
FROM entity_links
WHERE target_type = 'graph_entity' AND target_id = ANY($entity_ids);

The graph_score for each result is the overlap ratio: how many of the query's entities appear in the result's connections, divided by the total entities found in the query.

Step 4: Deduplication

The three signals produce candidate sets that overlap. A note containing "Ana García" might appear in vector search (semantically similar), full-text search (exact keyword match), and graph search (linked to the Ana García entity). We deduplicate by (domain, record_id) and track which signals produced each candidate.

Step 5: Rank-based normalization

Here's where it gets interesting. We do NOT normalize by raw scores. We normalize by rank position:

normalized_value = (N - position) / N

The top result in a signal gets 1.0. The bottom gets nearly 0. A candidate absent from a signal gets 0.

Why rank-based instead of min-max normalization? Because cosine similarity scores cluster. In a typical query, the top-50 vector search results might have similarities between 0.54 and 0.64 — a 10-point range. Min-max normalization would stretch this to 0.0–1.0, making the difference between rank 1 and rank 50 look huge when it's actually tiny.

Rank-based normalization treats position as the signal, not magnitude. First place is first place, whether it scored 0.99 or 0.55.

Step 6: Degenerate signal detection

Sometimes a signal doesn't discriminate. If all 50 vector search results have cosine similarities within 5% of each other, the signal is noise — everything "looks the same" to the embedding model.

When this happens, the weight assigned to the degenerate signal gets redistributed proportionally to the other signals. The search doesn't fail — it just relies more on the signals that are actually informative.

Step 7: Weighted fusion + multi-signal bonus

The final score combines all four signals:

score = (α × heat + β × semantic + γ × fulltext + δ × graph) / (α + β + γ + δ)

Default weights: all equal at 0.25 each. But the user can adjust them via the dashboard's ranking sliders — crank heat to prioritize recent activity, drop graph to ignore relationship signals, etc.

Then comes the multi-signal bonus:

Signals that found this result	Multiplier
1 signal only	×1.00
2 signals	×1.25
3 signals	×1.50

A result that appears in vector search, full-text search, AND graph search gets a 50% bonus. This rewards results that are independently confirmed by multiple methods — they're almost certainly relevant.

Heat doesn't count for the bonus calculation. It's a temporal signal, not a relevance signal — a record being hot doesn't mean it matches the query.

Two search modes

The system exposes two search endpoints that use this pipeline differently:

Standard search: `GET /search`

The default. Uses Reciprocal Rank Fusion with fixed weights:

rrf_score = Σ 1/(K + rank_i)    where K = 60
final = rrf_score × (1 + 0.1 × heat_score)

RRF is elegant: it ignores raw scores entirely, only caring about rank position. A result that's #1 in vector search and #3 in full-text search gets 1/(60+1) + 1/(60+3) = 0.0164 + 0.0159 = 0.0323. The K=60 constant smooths the curve — the difference between rank 1 and rank 10 is meaningful but not extreme.

Heat acts as a post-fusion tiebreaker with a maximum 10% boost. It never dominates relevance. A cold but highly relevant result always beats a hot but marginally relevant one.

Advanced search: `GET /search/advanced`

The full power mode. Four independent sliders, degenerate signal detection, multi-signal bonus. Used by the Search module in the dashboard where users can see and control exactly how results are ranked.

Every result includes full provenance:

{
  "provenance": {
    "heat_score": 0.52,
    "vector_score": 0.68,
    "fulltext_score": 0.52,
    "graph_score": 0.11
  }
}

Complete transparency. The user can see that a result ranked high because of semantic similarity (0.68) despite low graph connectivity (0.11), and adjust weights accordingly.

Free tier: surprisingly good without vectors

Free users don't get semantic search, graph search, or heat-weighted ranking. They get tsvector + GIN full-text search across all domains, with ts_rank() scoring and weighted columns.

This sounds like a big downgrade. In practice, it's surprisingly solid:

Searches for names, titles, and specific terms work perfectly — full-text search is exact.
Weighted tsvector means a match in the title ranks above a match in the body.
plainto_tsquery('simple', ...) handles both Spanish and English without configuration.
The UNION ALL across domains means one search bar finds notes, emails, events, contacts, and files.
GIN indexes make it O(log n) — fast even with tens of thousands of records.

The meaningful Pro differentiator is semantic search: finding "calor en Valencia" when you search for "beach trip." That's genuinely impossible with keyword matching. But for the 80% of searches where people type exactly what they're looking for, free tier search works fine.

Temporal expansion: the "wow" moment

Here's a feature that surprised us with how useful it turned out to be.

If the query mentions an entity that has upcoming events (within ±7 days), temporally close results get a boost:

temporal_boost = 1 + 0.15 × proximity

In practice: you search for "Ana García." The system finds that Ana has a meeting with you tomorrow. Notes, emails, and contacts related to Ana that were created or accessed in the last week get boosted. The search results naturally organize around "here's everything relevant to Ana before your meeting tomorrow."

We didn't plan this as a feature — it fell out of having the knowledge graph and the calendar in the same database. But it consistently produces the kind of results that make people say "how did it know I needed that?"

The tuning knobs

Five parameters control the pipeline behavior:

Parameter	Default	What it does
`MIN_SIMILARITY`	0.3	Cosine sim threshold. Higher = less vector noise
`RRF_FETCH_SIZE`	50	Candidates per signal. Lower = faster, fewer candidates
`RRF_K`	60	RRF smoothing constant. Higher = smoother rank differences
Multi-signal factor	0.25	Bonus per additional signal (1 + 0.25 × (count-1))
Degenerate threshold	5%	Signal suppression when score range is too narrow

We've left these at their defaults since implementation. The multi-signal bonus and degenerate detection handle most edge cases automatically. If we ever need to tune, the provenance metadata on every result tells us exactly which signal is helping or hurting.

Performance

All of this runs inside a single PostgreSQL instance, on the same machine that serves the API:

Operation	Latency
Vector search (50 candidates)	~15-30ms
Full-text search (UNION ALL, 7 tables)	~5-15ms
Graph discovery	~5-10ms
Fusion + scoring	~2-5ms
Total (standard search)	~30-60ms
Total (advanced search, 4 signals)	~50-80ms

No Elasticsearch. No Solr. No separate search service. pgvector and tsvector are PostgreSQL extensions that run in the same process as the rest of the database. One backup strategy, one connection pool, one operational concern.

For a personal system with a few thousand records per domain, this is more than fast enough. If we ever hit scale problems (unlikely for single-user), the first optimization would be reducing RRF_FETCH_SIZE from 50 to 20 — cutting candidate generation in half with minimal quality loss.

What I'd do differently

I'd implement full-text search from the very beginning, not as a free-tier afterthought. We built vector search first (Phase 3) and added tsvector later as a "fallback for free users." Turns out full-text search is essential even for Pro users — it catches exact matches that embeddings miss. "Show me the email from patricia@work.com" is a full-text query, not a semantic one.

I'd add provenance to the standard search too, not just advanced. We initially only exposed the score breakdown in /search/advanced. When we added it to the standard /search endpoint (in the provenance field), debugging search quality became ten times easier. Every bug report went from "the search is bad" to "this result has vector_score 0.8 but graph_score 0, why?"

I'd explore reranking with a cross-encoder. Our pipeline does retrieval + fusion but no reranking. A small cross-encoder model (like ms-marco-MiniLM) could re-score the top 20 results for higher precision. We deferred this because the current quality is good enough and adding another model to the Ollama queue would increase latency. But for a future post-MVP iteration, it's the obvious next step.

The takeaway

The trick to hybrid search isn't the individual signals — pgvector and tsvector are well-documented, and knowledge graph traversal is just recursive CTEs. The trick is the fusion: how you combine signals with different scales, different failure modes, and different strengths.

Reciprocal Rank Fusion solves the scale problem — ranks instead of raw scores. Degenerate signal detection solves the failure mode problem — a noisy signal gets suppressed instead of poisoning results. Multi-signal bonus solves the confidence problem — results confirmed by multiple methods are almost certainly good.

Four signals, one UNION ALL, one PostgreSQL instance, under 100ms. The search that finds "calor en Valencia" when you type "beach trip" — and also shows you that Ana García has a meeting with you tomorrow.

Next up: sleep-time compute for personal data — what your AI should be doing while you sleep, and why idle cycles are the most valuable compute you have.

Your AI agent is wasting 90% of its tokens on field names"

Victor García — Fri, 20 Mar 2026 13:56:54 +0000

We built the compact API format (previous post) and felt good about ourselves. API responses were 78% smaller. Tokens saved. Problem solved.

Then we actually measured where our agent's tokens were going.

The API responses weren't the problem. The skills were.

The audit

We ran a token audit across all 31 skills in the system. Here's what we found:

Main agent (Francis) — 11 skills loaded:

always:true skills (injected on every single message): ~17,500 tokens
Total including on-demand skills: ~20,600 tokens

Global (all 31 skills across all agents):

always:true skills (12 total): ~27,500 tokens
All skills combined: ~50,000 tokens

That's 25% of Sonnet's context window consumed by skills alone. Before the user says a word. Before the agent reads a single note or email. A quarter of the available context is just instructions on how to call APIs.

Add the workspace identity files — SOUL.md, IDENTITY.md, USER.md, TOOLS.md, AGENTS.md, BOOTSTRAP.md — and you're looking at another 3-5K tokens. So the agent starts every conversation with roughly 20-22K tokens of system prompt. That's over 10% of the context window, gone.

The user's actual message? Usually 20-50 tokens. A rounding error.

The top 5 offenders

Skill	always	Tokens
claw-search	true	~5,100
claw-hal	true	~4,200
claw-approvals	true	~3,050
claw-files	true	~2,850
claw-mail	true	~2,780

claw-search alone burns 5,100 tokens every message. It's the biggest skill because it handles cross-domain search routing — deciding whether to query the user's data (notes, emails, contacts) or the agent's own workspace memory. That routing logic is complex and takes words to explain.

claw-hal (hardware abstraction — storage, docker, network) is second because it covers multiple subsystems. When someone asks "how's my disk?", HAL needs to know about volumes, SMART data, mount points, and Docker containers. That's a lot of endpoints.

Why this matters more than API response size

Think about the token flow of a typical interaction:

User: "What meetings do I have today?"

System prompt:  ~20,000 tokens (skills + identity)
User message:        ~8 tokens
API call:          ~300 tokens (compact format response)
Agent response:     ~50 tokens
─────────────────────────────────
Total:          ~20,358 tokens

The API response is 1.5% of the total. Even if we made it zero tokens, we'd save almost nothing. The system prompt is 98% of the cost.

This is why we say the title of the previous post was slightly misleading. Yes, compact format saves 78% on API responses. But API responses are the small slice. The real token budget is dominated by the system prompt — and within that, by the skills.

What we did about it

1. The `always:true` / `always:false` split

The most impactful decision: most skills don't need to be loaded on every message.

If you say "save a note about the meeting," the agent needs claw-notes. It does not need claw-photos, claw-diary, claw-bookmarks, claw-storage, or home-assistant. Loading all of them wastes context on instructions the agent will never use for this interaction.

OpenClaw's skill system supports an always flag in the skill metadata:

metadata: {"openclaw":{"always":true}}

Skills marked always:true are injected into every prompt. Skills marked always:false are only activated when the conversation context matches their description. The routing model (a fast, cheap classifier) reads the user's message and decides which skills to load.

Our split:

always:true (every message)	always:false (on demand)
claw-notes, claw-calendar, claw-mail, claw-contacts, claw-drive, claw-search	claw-diary, claw-photos, claw-bookmarks, claw-storage, claw-hal, claw-graph, home-assistant

The first group are things people expect to always work: "save a note," "what's on my calendar," "check my email." If these weren't always loaded, the agent would sometimes miss obvious requests.

The second group are contextual: "how's my disk?" activates storage. "Show me photos from last week" activates photos. The routing model triggers them based on keywords.

Result: the main agent's always-on cost dropped from ~20,600 to ~17,500 tokens. Still a lot, but 3,100 tokens saved on every single message adds up fast across hundreds of daily interactions.

2. Writing skills for tokens, not for humans

The SKILL.md file is not documentation. It's a prompt. Every word costs money.

Our early skills looked like documentation:

## Creating a note

To create a new note, send a POST request to the notes endpoint.
The request body should contain the title and content fields.
The title is optional — if not provided, the system will use the
first line of the content as the title.

### Example

POST /api/v1/notes
Content-Type: application/json

{
  "title": "Meeting notes from Q1 review",
  "content": "Discussed budget allocation...",
  "tags": ["work", "q1"]
}

### Response

201 Created
{
  "data": {
    "id": "550e8400-...",
    "title": "Meeting notes from Q1 review",
    ...
  }
}

That's ~150 tokens to say "POST /notes with title, content, and tags." After optimization:

### Create note
- `POST /notes` body: `{title?, content, tags?[]}`
- Response: `201` with created note

~30 tokens. Same information. The agent doesn't need prose explaining what a POST request is. It doesn't need example JSON responses — it knows what a 201 looks like. It needs the method, the path, the body fields, and which ones are optional.

The guidelines we adopted:

No full JSON response examples. The agent doesn't need them.
Include body fields for POST/PATCH — the agent does need those.
Use ? suffix for optional fields — title? instead of "title (optional)."
One line per operation when possible.
No prose connectors — "To create a note, you should..." becomes "Create: POST /notes"

3. The compact instruction in every skill

Every skill now instructs the agent to use ?format=compact for listings:

## API optimization
- List: always use `?format=compact`
- Detail: `GET /:id` (full JSON) only when needed
- Do NOT use `format=compact` on POST/PATCH/DELETE

This ensures the savings from the compact API format (post 6) are actually realized. Without this instruction, the agent defaults to full JSON responses — it doesn't know compact exists unless the skill tells it.

4. Multi-agent delegation

The nuclear option for token optimization: don't load skills you don't need because a different agent handles them.

Our multi-agent topology has 7 agents. The main agent (Francis) is a router — it handles common requests and delegates specialized ones:

Atlas handles search, knowledge graph, and research
Sentinel handles infrastructure, HAL, Docker, network
Dalí handles photos, media, creative tasks
Ledger handles finance, invoicing, crypto
Darwin handles analytics, insights, sleep-time intelligence

Francis keeps 11 skills. The heavy ones like claw-hal (4,200 tokens) move to Sentinel, who only loads when infrastructure questions come up. claw-photos and visual intelligence move to Dalí. The search skill stays with Francis because search is needed in almost every interaction.

The main agent's prompt drops from ~20K to ~17.5K tokens. Still significant, but the per-message cost is meaningful — especially when using cloud models billed per token.

Full disclosure: the multi-agent topology is still early. We've defined the roles and the skill distribution, but we haven't battle-tested delegation patterns, error propagation between agents, or the overhead of agent-to-agent communication. There are almost certainly optimizations we're missing — whether it's smarter skill chunking, dynamic skill loading based on conversation history, or something we haven't thought of at all. If you've built multi-agent systems and see room for improvement, we'd genuinely love to hear about it in the comments.

The counterintuitive insight: bigger models handle it better

Here's something we didn't expect: Sonnet (the larger model) processes large skill contexts more efficiently than Haiku (the smaller, supposedly faster model).

When the system prompt is ~20K tokens across 12 skills, the root cause of latency isn't API performance or network — it's the model processing the skill context. Haiku, despite being "faster" per token, takes longer to reason through a large, complex system prompt. Sonnet processes the same context and produces a better-routed response in less wall-clock time.

This means the intuition of "use the small model for simple routing" breaks down when the routing itself requires understanding a large skill corpus. The small model saves on per-token cost but loses on latency and accuracy. For our use case — personal OS with 12+ skills — Sonnet as the primary agent model is strictly better than Haiku, despite the higher per-token price.

The numbers after optimization

Metric	Before	After	Change
Main agent always-on skills	~20,600 tokens	~17,500 tokens	-15%
Tokens per skill (avg)	~1,700	~1,400	-18%
Skills always:true	15	6 (main) / 12 (global)	-60%
API response (10 events)	~4,200 tokens	~800 tokens	-81%
System prompt total	~25,000 tokens	~20,000 tokens	-20%

The 20% reduction in system prompt is nice, but the real win is architectural: understanding that skills are the dominant cost and designing the multi-agent topology, the always/on-demand split, and the skill writing guidelines around that reality.

What I'd do differently

I'd measure token consumption from day one. We built 12 skills before ever counting how many tokens they consumed together. If we'd measured after the third skill, we'd have adopted the concise writing style immediately instead of rewriting everything later.

I'd design the multi-agent topology earlier. The decision to split agents was driven by token costs, but it should have been driven by separation of concerns. Sentinel handling all infrastructure makes sense regardless of tokens — it's a different expertise domain. We arrived at the right architecture for the wrong reason.

I'd add a token budget per skill in the manifest. Right now there's no mechanism to warn when a skill exceeds a reasonable size. A max_tokens: 3000 field in the manifest would force skill authors (including us) to stay concise. If your skill is over budget, you need to split it or trim it.

The takeaway

When you're building an AI agent system, the optimization hierarchy is:

System prompt size (~20K tokens, 98% of most interactions) — reduce always-on skills, write concisely, use multi-agent delegation
Skill activation routing — load only what's needed for this specific message
API response format — compact, diff-aware, progressive disclosure
Model selection — sometimes the bigger model is faster because it handles context better

Most optimization guides start at #3. The actual money is at #1.

Your agent isn't wasting tokens on API responses. It's wasting them on instructions it doesn't need for this particular message. Fix the prompt, then fix the API.

Next up: hybrid search with Reciprocal Rank Fusion — how we combined pgvector, tsvector, the knowledge graph, and heat scoring into a single search pipeline.

From JSON to compact: reducing API payloads 60% for LLM consumption"

Victor García — Thu, 19 Mar 2026 12:17:24 +0000

Every time your AI agent calls your API, it pays for the response in tokens. Not just the useful data — every "id":, every "created_at":, every null field it didn't ask for. JSON is designed for humans reading documentation, not for LLMs processing structured data.

We measured our API responses before optimization. A typical "show me my events today" call returned 3 events in ~1,200 tokens. After compact format: ~280 tokens. Same information, 77% fewer tokens.

This post is about how we built that format, what we tried and killed, and the surprising insight that changed how we think about API design for agents.

The problem: JSON is expensive

Here's what a standard events response looks like:

{
  "data": [
    {
      "id": "550e8400-e29b-41d4-a716-446655440000",
      "title": "Standup",
      "description": null,
      "location": "Discord",
      "start_at": "2026-02-18T09:00:00Z",
      "end_at": "2026-02-18T09:30:00Z",
      "all_day": false,
      "recurrence": {"freq": "weekly", "days": ["mon","wed","fri"]},
      "status": "confirmed",
      "source": "google",
      "source_id": "abc123",
      "calendar_name": "Work",
      "reminders": null,
      "attendees": null,
      "custom_fields": null,
      "created_at": "2026-01-15T10:00:00Z",
      "updated_at": "2026-02-17T10:00:00Z",
      "synced_at": "2026-02-18T06:00:00Z",
      "deleted_at": null
    }
  ],
  "meta": {"total": 1, "limit": 20, "offset": 0},
  "tier": "free"
}

One event. 18 fields. Of those, the agent needs maybe 5 to answer "what's on my calendar?": title, start time, end time, location, and whether it recurs. The other 13 fields — nulls, internal IDs, sync timestamps — are noise.

Multiply this by 20 events and every API call burns tokens on data the agent will never use.

The solution: `?format=compact`

Add one query parameter and the response transforms entirely:

GET /api/v1/events?date=today&format=compact

{
  "format": "compact",
  "summary": "3 events (today)",
  "lines": [
    "09:00-09:30 Standup [Discord] 🔁 id:550e8400",
    "14:00-15:00 Comida con Ana [La Mar] id:660e8400",
    "📅 all-day: Cumple Mamá id:770e8400"
  ],
  "ids": [
    "550e8400-e29b-41d4-a716-446655440000",
    "660e8400-e29b-41d4-a716-446655440001",
    "770e8400-e29b-41d4-a716-446655440002"
  ],
  "meta": {"total": 3, "limit": 20, "offset": 0, "has_more": false}
}

Three events in under 300 tokens. The agent can read the summary line and respond "You have 3 events today" without parsing anything. If it needs to modify an event, the ids array has the UUIDs in the same order as the lines.

Each domain has its own line format:

Domain	Line format	Example
Events	`HH:MM-HH:MM title [location] 🔁`	`09:00-09:30 Standup [Discord] 🔁`
Notes	`📌 "title" (tags) [relative_date]`	`📌 "Meeting notes Q1" (trabajo) [2w ago]`
Emails	`from: "subject" [FOLDER] status 📎`	`Juan: "Budget Q3" [INBOX] unread 📎`
Contacts	`name — company — email`	`Ana García — TechCorp — ana@tech.com`
Files	`filename (size) [type]`	`report.pdf (2.3MB) [pdf]`
Diary	`date mood "preview"`	`2026-02-18 😊 "Great day at the office..."`

The line templates are fixed per domain. They include exactly the fields an agent needs for a conversational summary, and nothing else.

What the agent does with it

The key insight is progressive disclosure. Compact format is the first step; full JSON is the second.

User: "What's on my calendar today?"
Agent: GET /events?date=today&format=compact
       → reads summary: "3 events (today)"
       → responds: "You have 3 events today: Standup at 9, Comida con Ana at 2, and it's Mamá's birthday."

User: "Move the lunch to 3pm"
Agent: PATCH /events/660e8400-e29b-41d4-a716-446655440001
       → body: {"start_at": "2026-02-18T15:00:00Z", "end_at": "2026-02-18T16:00:00Z"}

The agent used compact for the listing (cheap) and a direct PATCH for the mutation (needs the UUID, which compact provides). It never needed to fetch the full JSON for any event.

Every skill instructs the agent to use ?format=compact for listings and GET /:id (full JSON) only when it needs details about a specific record. This pattern — compact for browse, full for drill-down — is progressive disclosure implemented through the API, not the UI.

What we tried and killed

Before arriving at compact format, we explored three ideas. All of them died.

TOON (Token-Oriented Object Notation)

TOON is a serialization format designed for LLMs. It defines headers once and streams rows — like a TSV with a schema preamble. Research claims 30-60% fewer tokens than JSON.

We evaluated it and dropped it. The savings over our compact format were marginal (~3-5 tokens per line), but TOON introduced real problems: special characters need escaping, the parser has to handle edge cases with newlines in content, and the agent needs to understand a custom format instead of reading natural-language lines.

Compact lines are readable in English (or Spanish). TOON rows are not. For an LLM, readability matters more than raw compression.

Token Budget (`?token_budget=N`)

The idea: tell the server "I have 500 tokens left, fit the response in that." The server would progressively reduce fields, then limit results, then summarize content to fit within the budget.

We killed it because it violates separation of responsibilities. The server doesn't know which fields matter to the agent in a given context. Sometimes the agent needs attendees and not descriptions. Sometimes the reverse. Making the server decide which fields to cut is a design smell.

The agent already has ?fields=, ?limit=, and ?format=compact to control granularity. It doesn't need the server to guess on its behalf.

Semantic cache (pgvector similarity)

The idea: cache API responses by query embedding. If a new query is semantically similar to a cached one, return the cached response. Research shows 31% of queries are semantically similar.

For a single-user personal system, the hit rate would be 5-15% at best. And the overhead of embedding each query (50-200ms via Ollama) is slower than just running the database query directly (5-20ms). We'd be adding latency to save latency.

We kept the SHA-256 exact-match cache as a possibility for the future (when the sleep-time engine starts making repetitive queries), but the semantic layer was pure overhead.

Diff-aware responses: `?since=`

Compact format reduces the size of each response. But there's another dimension: reducing how often the agent needs to ask at all.

The ?since= parameter lets the agent say "I last checked at 14:00 — what changed?"

GET /api/v1/events?since=2026-02-18T14:00:00Z

{
  "data": {
    "created": [],
    "updated": [
      {"id": "660e8400-...", "title": "Comida con Ana", "start_at": "2026-02-18T15:00:00Z"}
    ],
    "deleted": []
  },
  "meta": {
    "response_timestamp": "2026-02-18T14:35:00Z",
    "total_changes": 1
  }
}

The agent stores the response_timestamp from each call and passes it as ?since= next time. If nothing changed, the response is essentially empty. The response_timestamp is captured before the query executes — conservative by design, so changes are never lost (at worst, the agent sees a duplicate).

Combined with compact format, the pattern becomes:

First call: GET /events?format=compact → full compact listing
Subsequent calls: GET /events?since=<last_timestamp> → only changes
If changes exist and agent needs context: GET /events/:id → full detail on specific records

For a system where the agent polls every 30 minutes and maybe 2 records changed, this turns a 1,200-token response into a 150-token one.

Token counting: the `X-Token-Count` header

Every response includes an X-Token-Count header with the approximate token count of the response body. It's a heuristic (±15% accuracy), not an exact count — we use character-based estimation rather than running a tokenizer on every response.

The agent doesn't make binary decisions based on this number. It's informational — "this response cost me approximately 450 tokens" — so the agent can track its consumption over time and adjust strategy. If it notices that contacts responses are consistently heavy, it might switch to compact or reduce limits.

We briefly considered using tiktoken for exact counting and decided against it. The precision wasn't worth the overhead on every response, and the agent doesn't need exact numbers — it needs a signal.

The numbers

Measured across 7 domain endpoints with realistic data volumes:

Domain	JSON (tokens)	Compact (tokens)	Reduction
Events (10)	~4,200	~800	81%
Notes (10)	~5,800	~1,200	79%
Emails (10)	~8,500	~1,500	82%
Contacts (10)	~3,200	~900	72%
Files (10)	~2,800	~700	75%
Diary (10)	~4,000	~1,100	73%
Search (10)	~6,000	~1,000	83%

Average reduction: ~78%. The title says 60% because that's the conservative number for small result sets (3-5 records). As result sets grow, the savings compound.

Emails save the most because they have the most fields (30+ columns, most of which are null for any given record). Contacts save the least because they're already relatively compact in JSON.

Design decisions worth noting

Compact and ?fields= are mutually exclusive. Compact templates are fixed per domain. Allowing custom field selection within compact would mean building a dynamic template engine — complexity for a feature nobody asked for. If you need specific fields, use ?fields=. If you need minimal tokens, use ?format=compact. Not both.

Compact is ignored silently on non-list endpoints. GET /notes/:id?format=compact returns normal JSON. No error, no warning. The agent shouldn't need to remember which endpoints support which formats — it just adds format=compact to everything and the server does the right thing.

Errors are always full JSON. Even with ?format=compact, errors return the standard error envelope with code, message, details, and hint. The agent needs structured error information more than it needs minimal tokens in failure cases.

The ids array is the bridge. The compact lines are for the LLM to read and summarize. The ids array is for the LLM to act — it maps 1:1 to the lines, so "the second event" maps to ids[1]. This dual-track design (human-readable lines + machine-actionable IDs) is what makes compact format actually useful rather than just cheap.

What I'd do differently

I'd build compact format before anything else. We built it during the API Intelligence phase (Cluster A), but it should have been in Phase 1. Every skill we wrote before compact format included verbose JSON examples that the agent had to parse. The day we shipped compact, we updated every skill to use it and immediately saw lower latency and better agent responses.

I'd kill more ideas faster. TOON, token budget, and semantic cache each took a design session to spec out and a decision to kill. The design sessions weren't wasted — they clarified what we actually needed — but we could have killed them in 30 minutes of back-of-envelope calculation instead of writing full specs.

The takeaway

If you're building an API that LLM agents will consume, you're probably shipping too much data. JSON is great for browsers and SDKs. It's terrible for language models that pay per token.

The fix is embarrassingly simple: a query parameter that switches the response format from structured objects to human-readable lines with an ID array. Fixed templates per domain. No custom serializer, no protocol change, no content negotiation. Just a different if branch in your response handler.

The deeper insight: the agent doesn't need your data model. It needs a summary it can repeat to the user and IDs it can use to take action. Everything else is overhead.

Next up: your AI agent is wasting 90% of its tokens — and it's not the API's fault. It's the skills.

Heat scoring: teaching your data to forget (gracefully)"

Victor García — Wed, 18 Mar 2026 17:26:00 +0000

Here's something that bothered us early on: a search for "project meeting" returned every meeting note we'd ever written, sorted by relevance. The note from this morning and the one from eight months ago scored identically — because semantically, they're equally "about project meetings."

But they're not equally useful. The one from this morning matters right now. The one from eight months ago is a fossil. We needed a way to express that difference without breaking search when someone genuinely wants to find old records.

The solution was to give every record a temperature.

The concept

Every record in every domain table — notes, events, contacts, emails, files, diary entries — has a heat score. It's a number between 0 and 1 that answers one question: how relevant is this record right now?

Heat rises when you interact with a record (open it, edit it, create it). Heat decays exponentially over time when you don't. A note you wrote this morning is hot. A note you haven't touched in six months is cold. A contact you email every week stays permanently warm.

Three tiers classify records:

Tier	Heat score	What it means
Hot	> 0.7	Actively used right now
Warm	> 0.2	Used recently-ish
Cold	≤ 0.2	Hasn't been touched in a while

The idea is inspired by MemoryOS (EMNLP 2025), which applies memory tiers to conversational AI. We adapted it to structured personal data — records instead of chat messages, database rows instead of context windows.

The formula

The heat score combines two signals: how often you've accessed a record, and how recently.

effective_access = access_count × max(0, 1 - (hours_since_last_increment / 4383))
raw_heat = effective_access × e^(-λ × hours_since_last_access)

Where λ = ln(2) / 168 — a half-life of one week (168 hours).

The effective_access part handles frequency. A contact you've accessed 100 times counts for a lot — but that count itself decays linearly over a year. Six months of neglect and those 100 accesses are worth 50. A year of neglect and they're worth zero. This prevents records from staying hot forever just because they were used intensively once.

The exponential decay part handles recency. Something accessed right now has full heat. Something accessed a week ago has half. Two weeks ago, a quarter. The exponential curve means heat drops fast in the first few days, then slowly flattens out.

The combined effect is interesting:

Contact accessed 100 times, last access 6 months ago: effective_access ≈ 50, but the exponential decay from 6 months of recency drives heat_score close to 0. It's cold.
Note accessed 3 times, last access today: effective_access = 3, multiplied by nearly full recency. Heat_score is moderate. It's warm, trending hot.
Email accessed once, created this morning: effective_access = 1, recency = fresh. Low warm. One more access and it tips to solid warm.

Normalization: the 0-1 range

The raw formula produces unbounded values — a record accessed 50 times today gives a raw heat of ~50. That's useless for ranking. So we normalize:

normalized = raw_heat / SCALE_FACTOR    // SCALE_FACTOR = 10.0 (calibrated)
heat_score = normalized ≤ 0.8 ? normalized : 0.8 + (normalized - 0.8) × 0.5
heat_score = min(heat_score, 1.0)

The soft cap at 0.8 is deliberate. Getting from 0.0 to 0.8 takes normal usage. Getting from 0.8 to 1.0 takes double the activity. This prevents "thermal saturation" — a record you obsessively check doesn't permanently peg the meter at 1.0, leaving room for other records to be relatively hotter.

The hard cap at 1.0 is enforced in the database: CHECK (heat_score >= 0 AND heat_score <= 1).

The table

Heat data lives in its own table, not in the domain tables:

CREATE TABLE record_heat (
    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    domain          VARCHAR(30) NOT NULL,
    record_id       UUID NOT NULL,
    access_count    INTEGER NOT NULL DEFAULT 0,
    last_accessed   TIMESTAMPTZ NOT NULL DEFAULT now(),
    last_increment  TIMESTAMPTZ NOT NULL DEFAULT now(),
    heat_score      REAL NOT NULL DEFAULT 0.0,
    memory_tier     VARCHAR(10) NOT NULL DEFAULT 'cold',
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    UNIQUE(domain, record_id)
);

Same pattern as the embeddings table: (domain, record_id) instead of foreign keys. No cascading deletes to manage — when a note is deleted, the heat row becomes an orphan that the cleanup cron handles.

Why a separate table instead of adding columns to each domain table? Because heat is cross-cutting. Every domain behaves the same way. A single cron job, a single update function, a single query pattern. If heat lived in 7+ domain tables, every change to the formula would be 7+ ALTER statements.

When heat changes

On access (instant)

Every time you open a record — GET /notes/:id, GET /contacts/:id, whatever — a post-handler fires:

Increment access_count by 1
Set last_accessed to now
Recalculate heat_score and memory_tier
Upsert into record_heat

That's one SQL query. Under 1 millisecond. The user never notices.

Creating or editing a record also bumps heat. A record you just wrote or just modified reflects active intent — it should be hot.

Deleting doesn't bump heat. The decay cron will cool it naturally.

On cron (every 6 hours)

A scheduled job recalculates heat for every row in record_heat. This is necessary because the exponential decay is time-based — without periodic recalculation, a record that nobody accesses would stay at whatever heat it had when last touched. The cron ensures everything drifts toward cold over time.

The cron also applies the linear decay to access_count — proportionally reducing it based on time elapsed since the last recalculation.

Source tracking

Not all accesses are equal. We track the source of each heat event:

type HeatSource = 'user_dash' | 'agent_primary' | 'agent_creative' 
                | 'sleep_time' | 'sync' | 'unknown';

A user opening a contact in the dashboard is a strong signal of interest. The sleep-time engine scanning the same contact for background analysis is not. The heat system accepts the source parameter so that downstream consumers (like the proactive preference learner) can distinguish genuine human interest from automated access.

Sync imports get lower initial heat than user-created records, because the user didn't actively choose to create them.

How heat changes search

Heat never filters search results. If you search for "apartment lease" and the only match is a cold note from a year ago, it absolutely must appear. Hiding results because they're cold would be insane.

Instead, heat acts as a post-fusion tiebreaker in the search ranking algorithm:

final_score = rrf_score × (1 + 0.1 × heat_score)

This is step 4 in the Reciprocal Rank Fusion pipeline. The 0.1 multiplier means heat can boost a result by at most 10%. It breaks ties between equally relevant results — the hotter one wins — but it can never override genuine relevance. A cold but highly relevant result still beats a hot but marginally relevant one.

In the advanced search mode, heat becomes one of four independently weighted signals:

score = (α × heat + β × semantic + γ × fulltext + δ × graph) / (α + β + γ + δ)

The user can drag sliders to adjust the weight of each signal. Want to find recent-activity-first? Crank heat to 1.0. Want to find the most semantically relevant regardless of age? Drop heat to 0. The weights are transparent — every search result shows its raw scores for all four signals.

How heat changes the digest

The Digest Engine — the background job that compiles "here's what changed since you last checked" — uses heat in two ways:

Prioritization. When 50 things changed overnight, the digest needs to decide what to mention first. Heat-weighted changes surface near the top. "Ana García, who you've been emailing all week, sent a new message" ranks above "A contact you haven't opened in 4 months was synced."

Cold filtering. By default, the search endpoint excludes records in the "accessed but cold" zone (heat > 0 but < 0.2). These are records you interacted with once and then forgot. They still exist and are findable if you search explicitly — but they don't clutter routine results. A ?include_cold=true parameter brings them back for exhaustive searches.

How heat changes the API

Every list endpoint gains a ?tier= filter:

GET /notes?tier=hot          → only hot notes
GET /contacts?tier=warm      → warm + hot contacts
GET /events?tier=cold        → only cold events (rarely useful, but available)
GET /notes                   → all notes (no change from before)

The AI agent uses this heavily. When the user says "what's on my plate?", the agent queries ?tier=hot across all domains to get a quick snapshot of active items without wading through the entire database. When the user says "find that contract from last year", the agent searches without tier filtering because cold records are exactly what's needed.

The edge case: heat on edges

Records aren't the only thing with heat. Graph edges — the connections between entities in the knowledge graph — also have a heat score.

When you open a note and then immediately open a related email (within 60 seconds), the edge between those two records gets warmer. The dashboard tracks these co-navigation patterns automatically:

if (lastViewedRecord && Date.now() - lastViewedRecord.timestamp < 60_000) {
  api.post('/links/heat-edge', {
    from_type: lastViewedRecord.type,
    from_id: lastViewedRecord.id,
    to_type: currentRecord.type,
    to_id: currentRecord.id
  });
}

Hot edges become visible in the graph visualization — glowing connections between frequently co-accessed records. Cold edges fade almost invisible. It turns the knowledge graph into a heat map of your actual workflow, not just a static web of extracted entities.

What I'd do differently

I'd start with simpler thresholds and tune later. We went through three iterations of tier boundaries (the original design had unbounded scores with thresholds at 5.0 and 0.5, then we normalized to 0-1 with a soft cap, then adjusted thresholds to 0.7/0.2). Starting with "anything accessed in the last week is hot, anything in the last month is warm, everything else is cold" would have been fine for v1.

I'd add heat to the search provenance from day one. Heat was initially invisible in search results — you could feel its effect on ranking but couldn't see the raw score. Adding provenance (the breakdown of all four search signals) happened later, and it was immediately obvious it should have been there from the start. Transparency in ranking builds trust.

The takeaway

Heat scoring is maybe 150 lines of code: one table, one upsert function, one cron job, one post-fusion multiplier. But it fundamentally changes the system's relationship with time.

Without heat, every record is equally present. Your database is a flat archive — 2,000 notes, all with the same weight. With heat, the database has a sense of "now." Recent activity floats to the surface. Old activity sinks naturally. The AI agent, the search engine, and the digest all benefit from the same signal: what matters to you right now.

The formula doesn't need to be clever. Exponential decay with a weekly half-life and a soft cap. That's it. The cleverness is in where you apply it — as a tiebreaker in search, as a filter in the agent's queries, as a priority signal in the digest, as a visual indicator in the dashboard.

Data that knows how to forget is data that stays useful.

Next up: from JSON to compact — how we reduced API payloads by 60% for LLM consumption, and why your AI agent is wasting most of its tokens on field names.

Building a personal knowledge graph with just PostgreSQL (no Neo4j needed)"

Victor García — Wed, 18 Mar 2026 13:04:24 +0000

At some point during development, we needed to answer questions that search couldn't: "Who's connected to this project?" "What links this email to tomorrow's meeting?" "Which people keep appearing together across my notes?"

Semantic search finds records by meaning. Full-text search finds them by keywords. But neither can traverse relationships. For that, you need a graph.

The obvious choice was Neo4j, or at least Apache AGE (which adds Cypher queries to PostgreSQL). We evaluated both and went with... two tables and WITH RECURSIVE.

This post explains why, and shows the actual schema, queries, and API that power the knowledge graph.

Why not a graph database?

We seriously considered Apache AGE. It adds OpenCypher support directly inside PostgreSQL — no separate service, same database. The competitive analysis even had it as "Innovation #2: Bi-temporal personal knowledge graph via Apache AGE."

We rejected it for three reasons:

Scale. This is a personal system. One user, maybe a family. We're talking about fewer than 10,000 entity nodes and 50,000 edges. At that scale, a recursive CTE with depth 2-3 takes microseconds. The performance argument for a graph engine simply doesn't exist.

Dependencies. Apache AGE is a PostgreSQL extension that needs to be compiled and installed. On a bare-metal mini-PC with different architectures (x86, ARM), that's a maintenance headache. Our system already depends on two extensions (pgcrypto and pgvector). Adding a third for a feature that SQL handles natively felt wrong.

Type safety. We use Drizzle ORM for everything. Apache AGE queries return untyped results from Cypher strings embedded in SQL. Our recursive CTEs return typed rows that Drizzle understands. No impedance mismatch, no parsing layer, no surprise nulls.

The decision was documented as "ADR B-4: Apache AGE rejected in favor of SQL-native approach with recursive CTEs." The roadmap originally referenced AGE — we had to go back and update it.

The schema: two tables

The entire knowledge graph lives in two tables. One for nodes, one for edges.

Nodes: `graph_entities`

CREATE TABLE graph_entities (
    id                UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    entity_type       VARCHAR(30) NOT NULL,    -- person, project, location, topic
    name              TEXT NOT NULL,            -- "Ana García", "Micelclaw OS", "Zaragoza"
    normalized_name   TEXT NOT NULL,            -- "ana garcia", "micelclaw os", "zaragoza"
    properties        JSONB DEFAULT '{}',       -- {email, company, role, coordinates...}
    merge_history     JSONB DEFAULT '[]',       -- [{merged_from, date, reason}]
    created_at        TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at        TIMESTAMPTZ NOT NULL DEFAULT now(),
    deleted_at        TIMESTAMPTZ,
    UNIQUE(entity_type, normalized_name)
);

Four entity types: person, project, location, topic. That's it. We considered adding more (organization, event, document) and decided against it. Four types cover 95% of real-world connections in personal data. The properties JSONB handles the rest — a person can have an email and company, a location can have coordinates.

The normalized_name column is the key to entity resolution: lowercase, no accents, trimmed. When the extraction pipeline finds "Ana García" in a note and "ana garcia" in an email, the UNIQUE constraint ensures they map to the same node.

Edges: `entity_links` (extended)

We already had an entity_links table from the initial schema — it was one of the original 13 tables. It stored simple connections like "this note mentions this contact." For the knowledge graph, we extended it with three columns:

ALTER TABLE entity_links ADD COLUMN link_type VARCHAR(20) NOT NULL DEFAULT 'manual';
-- 'manual' | 'extracted' | 'inferred' | 'structural'

ALTER TABLE entity_links ADD COLUMN confidence REAL NOT NULL DEFAULT 1.0;
-- 0.0 to 1.0

ALTER TABLE entity_links ADD COLUMN created_by VARCHAR(20) NOT NULL DEFAULT 'system';
-- 'user' | 'llm' | 'system'

Three ALTER statements. That's the entire migration for turning a flat links table into a knowledge graph edge store.

The source_type and target_type columns (VARCHAR, no CHECK constraint) now accept graph_entity as a valid type. This means an edge can connect:

A note → a graph entity (Person mentioned in text)
A graph entity → a graph entity (Person works at Project)
An event → a graph entity (Event located in City)

The relationship taxonomy follows a subject → verb → object convention:

Relationship	Direction	Created by
`mentions`	record → entity	LLM extraction
`attended_by`	event → person	Sync (calendar attendees)
`located_in`	record → location	LLM extraction
`works_at`	person → project/org	LLM extraction
`collaborates_with`	person → person	Inferred from co-occurrence
`relates_to`	entity → entity	Manual or sleep-time engine

We also reserved three relationships for future use: contradicts, follows_up, supersedes — for a Zettelkasten-style auto-linking feature that isn't built yet but whose namespace is already protected.

The queries: recursive CTEs

Three query patterns cover every graph operation we need.

Expansion: "Who is connected to Ana García?"

Given a node, find all neighbors up to depth N:

WITH RECURSIVE graph_walk AS (
    -- Base: direct connections from the starting entity
    SELECT
        el.target_type,
        el.target_id,
        el.relationship,
        el.confidence,
        1 AS depth
    FROM entity_links el
    WHERE el.source_type = 'graph_entity'
      AND el.source_id = $entity_id
      AND el.confidence >= 0.5

    UNION ALL

    -- Recursive: follow edges from discovered nodes
    SELECT
        el.target_type,
        el.target_id,
        el.relationship,
        el.confidence,
        gw.depth + 1
    FROM graph_walk gw
    JOIN entity_links el
        ON el.source_type = gw.target_type
       AND el.source_id = gw.target_id
    WHERE gw.depth < $max_depth    -- default: 2
      AND el.confidence >= 0.5
)
SELECT DISTINCT ON (target_type, target_id) *
FROM graph_walk
ORDER BY target_type, target_id, depth;

At depth 2 with 10K nodes, this query returns in under 10ms. We set the default to 2 and the maximum to 3. Depth 3 is rarely useful for personal data — it usually returns the entire graph.

Path finding: "What connects this email to that meeting?"

BFS to find the shortest path between two nodes:

WITH RECURSIVE path_search AS (
    SELECT
        ARRAY[source_id] AS path,
        target_type,
        target_id,
        1 AS depth
    FROM entity_links
    WHERE source_id = $from_id

    UNION ALL

    SELECT
        ps.path || el.source_id,
        el.target_type,
        el.target_id,
        ps.depth + 1
    FROM path_search ps
    JOIN entity_links el
        ON el.source_id = ps.target_id
    WHERE ps.depth < $max_depth    -- default: 4
      AND NOT (el.source_id = ANY(ps.path))  -- cycle prevention
)
SELECT * FROM path_search
WHERE target_id = $to_id
ORDER BY depth
LIMIT 1;

The NOT (el.source_id = ANY(ps.path)) clause prevents infinite loops. The LIMIT 1 with ORDER BY depth gives us the shortest path.

Subgraph: "Show me everything around this project"

For the visualization view in the dashboard (a force-directed graph using react-force-graph-2d):

-- Get the top N entities by mention count, centered on an optional entity
SELECT
    ge.id, ge.name, ge.entity_type,
    COUNT(el.id) AS mention_count,
    rh.heat_score
FROM graph_entities ge
LEFT JOIN entity_links el
    ON (el.target_type = 'graph_entity' AND el.target_id = ge.id)
    OR (el.source_type = 'graph_entity' AND el.source_id = ge.id)
LEFT JOIN record_heat rh
    ON rh.domain = 'graph_entity' AND rh.record_id = ge.id
WHERE ge.deleted_at IS NULL
GROUP BY ge.id, rh.heat_score
ORDER BY mention_count DESC
LIMIT $limit;

Then a second query fetches all edges between the returned nodes. The dashboard renders nodes sized by mention count and colored by heat score — hot nodes glow, cold nodes fade.

How nodes get created

Nodes enter the graph through three pipelines, only one of which involves an LLM:

1. Contacts → Person nodes (no LLM)

When a contact is created or synced from Google, the CRUD hook creates a Person node directly from structured data:

Contact {display_name: "Ana García", company: "TechCorp", emails: [{address: "ana@techcorp.com"}]}
    ↓
graph_entities {entity_type: "person", name: "Ana García", normalized_name: "ana garcia",
                properties: {email: "ana@techcorp.com", company: "TechCorp"}}

No model needed. The data is already structured.

2. Event attendees → Person nodes (no LLM)

Calendar events have attendees as structured JSONB. Each attendee is resolved against existing Person nodes by email, or created as a new node:

Event attendee {email: "ana@techcorp.com", name: "Ana"}
    ↓
Match against graph_entities WHERE properties->>'email' = 'ana@techcorp.com'
    ↓
Found → reuse existing node (even if name differs slightly)
Not found → create new Person node

3. Text extraction → All entity types (LLM)

This is the async pipeline from the previous blog post. The 2B model extracts persons, projects, locations, and topics from notes, emails, diary entries, and file contents. Each extracted entity is upserted by (entity_type, normalized_name) — if it already exists, the properties are merged.

The key insight: pipelines 1 and 2 mean the graph has a solid foundation of real, structured Person nodes before the LLM ever runs. When the LLM extracts "Ana García" from a note, it matches against a node that already exists from the contacts sync. No orphan entities, no duplicates — just a new edge connecting the note to an existing person.

Entity resolution: the hard problem

Extraction creates nodes. Entity resolution decides whether two nodes are the same thing. We handle this in three levels:

Level 1 — Deterministic (automatic). The UNIQUE(entity_type, normalized_name) constraint handles exact matches. "Ana García" and "ana garcía" always map to the same node. For persons, email matching overrides name matching — if two nodes have the same email, they're the same person regardless of name differences.

Level 2 — Suggested merges (semi-automatic). An endpoint returns pairs of same-type entities with high name similarity (using pg_trgm's similarity() function, threshold > 0.4):

GET /graph/merge-candidates

[{
  entity_a: {name: "Ana García", mention_count: 15},
  entity_b: {name: "Ana G.", mention_count: 3},
  similarity: 0.87
}]

A merge redirects all edges from the absorbed node to the surviving node, records the event in merge_history, and hard-deletes the duplicate.

Level 3 — Sleep-time resolution (automatic, low-priority). A background job periodically reviews Person nodes, calculates cross-similarities, and proposes merge candidates. The AI agent can also trigger merges when context makes it obvious ("Juan" and "Juan Pérez" in the same conversation).

What we explicitly don't try to resolve: nicknames ("Pepe" = "José"), role-based references ("the accountant"), and ambiguous entities ("Santiago" — person or city?). Those are either handled by the agent in conversation or corrected manually. The graph needs to be conservative — an incorrect merge destroys data, while a duplicate is just noise that can be cleaned up later.

The API

Six endpoints expose the graph:

GET  /graph/entities              — Search by name, filter by type
GET  /graph/entities/:id          — Entity detail with direct connections
GET  /graph/connections           — Expansion traversal (depth 1-3)
GET  /graph/path                  — Shortest path between two entities
GET  /graph/subgraph              — Nodes + edges for visualization
GET  /graph/stats                 — Counts, orphans, pending queue
GET  /graph/merge-candidates      — Similar entity pairs
POST /graph/merge                 — Fuse two entities
POST /graph/cleanup               — Delete orphan nodes (0 connections)

The graph is a Pro feature. Free tier users still get entity_links (manual connections between records), but the automatic entity extraction, the graph visualization, and the traversal queries require Pro.

What the graph enables

Once you have a knowledge graph, things that were impossible become trivial:

"Who's involved in this project?" — expansion query from a Project entity, depth 1, filter by Person type. Returns everyone who's been mentioned in connection with the project across all domains.

"What connects this email to that meeting?" — path query. Returns: email → mentions → Person:Ana García → attended_by → Event:Sprint Review. Two hops, one shared person.

"Show me everything about Ana García" — entity detail. Returns: 15 mentions across notes and emails, 3 events attended, works at TechCorp, collaborates with Javier Losada. All discovered automatically.

Search ranking. The hybrid search uses graph connectivity as one of four signals (along with semantic similarity, full-text relevance, and heat score) in a Reciprocal Rank Fusion algorithm. A note that mentions entities connected to your recent activity ranks higher.

The graph also feeds the sleep-time engine (background jobs that discover cross-domain correlations), the proactive digest ("Ana García, who you're meeting tomorrow, sent you an email yesterday"), and the AI agent's contextual awareness.

The numbers

At the current scale of development:

Metric	Value
Graph entities	~150-300 per active user
Entity links	~500-1000
Expansion query (depth 2)	< 10ms
Path query (depth 4)	< 15ms
Subgraph for visualization (100 nodes)	< 20ms
Memory overhead	0 (it's just PostgreSQL)

No new service to deploy. No new port to manage. No new backup strategy. It's tables in the same database that holds everything else, queried with the same ORM, backed up with the same pg_dump.

What I'd do differently

I'd add pg_trgm from day one. We needed it for merge candidates (fuzzy name matching) and ended up adding it as a later migration. It's a small extension with zero downsides — should have been in the initial schema alongside pgcrypto and pgvector.

I'd index entity_links more aggressively. The default indexes cover the UNIQUE constraint, but expansion queries benefit from separate indexes on (source_type, source_id) and (target_type, target_id). We added them later when profiling showed sequential scans on the edges table.

I'd build the visualization earlier. The force-directed graph view in the dashboard was one of the last features built, but it should have been one of the first. Seeing the graph visually — nodes clustering around projects, people forming communities — was the moment the knowledge graph went from "interesting data structure" to "this actually understands my data." It would have motivated better extraction quality earlier.

The takeaway

A knowledge graph doesn't need a graph database. At personal scale (< 10K nodes, < 50K edges), PostgreSQL with recursive CTEs is fast, simple, and — critically — already there. No new infrastructure, no new dependency, no new operational burden.

Two tables (graph_entities + extended entity_links), three traversal patterns (expansion, path, subgraph), three entity creation pipelines (structured sync, calendar attendees, LLM extraction), and three levels of entity resolution (deterministic, suggested, sleep-time).

The graph is the connective tissue between everything else in the system — search, heat scoring, the AI agent, the digest engine. And it's all just rows in PostgreSQL.

Next up: heat scoring — how we taught records to fade like memories, and why a simple exponential decay formula changes how search works.