DEV Community: Jackson Ly

Our Rust file watcher ate 23.6 GB of RAM, and our ignore rules never had a chance

Jackson Ly — Mon, 13 Jul 2026 13:40:17 +0000

We build a local-first indexer for macOS. It watches the user's home folder so that when a file changes, we reindex just that file. Watching one project directory is a solved problem. Watching a real human's $HOME is a different sport, and it broke us in a way that took a heap profiler to see.

The symptom

The app sat at roughly 25 GB of resident memory. Not a slow leak either. It climbed there shortly after launch and stayed. One core also pinned itself for about ten minutes at every boot.

The obvious suspects were wrong. It was not the embedding model, and it was not our index. We had a careful ignore list (caches, node_modules, Library, build output), so surely we were not touching millions of files.

We were not. But something else was.

The finding

We use the excellent notify crate, specifically notify-debouncer-full, which is what you reach for when you want raw filesystem events collapsed into something sane. The setup you copy from the README uses RecommendedCache.

RecommendedCache resolves to FileIdMap. From the docs, the debouncer "can optionally keep track of the file system IDs all files and stitches rename events together". That is a genuinely useful feature: on macOS FSEvents and on Windows, a rename shows up as two unrelated events, and pairing them requires knowing that the file at the old path and the file at the new path are the same inode.

To do that, it has to know the file ID of every file under the watch root. So it walks the entire watch root and caches a (PathBuf, FileId) for every entry it finds.

Here is the part that mattered: that walk does not know about your ignore rules. Our ignore list filters events. The cache is built underneath that, from the watch root down. We told the debouncer to watch $HOME, so it faithfully walked all of $HOME, including everything we thought we had excluded, following symlinks as it went.

Under malloc_history, the damage was specific:

23.6 GB of live heap in the cache
4.16 GB of that in the hashbrown table itself
19.4 GB spread across 54.2 million path allocations
roughly 6.4 million files stat-ed at boot, which is the pinned core and the ten minutes

54 million allocations to remember the inode of files we had explicitly said we did not care about.

The fix, and what it costs

We switched the cache to NoCache.

That is not free, and it is worth being precise about the tradeoff rather than pretending we outsmarted the library. FileIdMap exists for a reason: with NoCache, the debouncer can no longer stitch rename events together. A rename stops arriving as "this moved from A to B" and degrades into an unrelated delete at A and a create at B.

We could absorb that, for two reasons that are specific to our design:

Our indexer is idempotent. Indexing the same file twice converges to the same row, so a spurious create is cheap and harmless.
We already run a rescan and reconcile pass that removes index entries whose files no longer exist. Deletions get collected there regardless of whether the watcher paired them correctly.

So a rename becomes delete-plus-create, and the system settles into the right state on its own. If your indexer is not idempotent, or you have no reconcile pass, this trade is not available to you and you should fix the scope instead.

Result: 24.8 GB down to 1.45 GB.

The second bug we found on the way

While measuring, we caught something else. When FSEvents drops events (it does this under load, by design, and tells you so), the debouncer emits a rescan. Our code fanned that out as one rescan per configured root.

Our roots were nested. We watched $HOME, and we also watched Documents, Downloads, and Desktop because they were configured separately. So a single dropped event triggered four concurrent, roughly $HOME-scale re-walks, and every file under the nested roots got embedded twice. You could see it plainly in the logs: two EmbeddingsGenerated lines per document.

The fix was to de-nest the roots into independent subtrees before emitting any rescan, so overlapping configuration collapses into one walk.

The lesson worth stealing

Library defaults are calibrated for the common case, and the common case is a project folder with a few thousand files. Nothing in the API warns you, because nothing is wrong with the API. FileIdMap is correct. It is correct at 5,000 files and it is a memory bomb at 6.4 million, and the difference is entirely in what you point it at.

So, two things:

Know what your watcher caches before you widen the scope. Filtering events is not the same as filtering what the library indexes internally. Ours sat below our filter, where we could not see it.

Measure the heap, do not reason about it. We would never have found 54.2 million path allocations by reading code. malloc_history found it in one pass. Every hour we spent theorizing about the embedding model was an hour we did not spend attaching a profiler.

If you are watching a user's whole home directory, assume every default in your stack was written by someone who was not.

Point any app at a local LLM on your Mac (OpenAI-compatible endpoints)

Jackson Ly — Fri, 10 Jul 2026 00:17:10 +0000

Most apps that grew an "AI" feature in the last two years talk to one of a handful of cloud APIs, and almost all of them speak the same dialect: the OpenAI Chat Completions format. That one detail is the reason you can pull the cloud out and run the whole thing locally on a Mac without the app ever noticing.

Here is the trick, why it works, and the gotchas that bite.

The one interface everything agrees on

OpenAI's /v1/chat/completions endpoint became the de facto standard. So when an app lets you "use your own key" or "set a custom base URL," it is almost always going to POST to {base_url}/chat/completions with a JSON body of messages and read back the same shape. It does not care what is on the other end, only that the response matches.

Local runners leaned into this. Both popular Mac ones expose exactly that endpoint:

Ollama serves an OpenAI-compatible API at http://localhost:11434/v1 (its native API lives on /api, but the /v1 path speaks the OpenAI dialect).
LM Studio has a built-in server you switch on from the Developer tab, serving on http://localhost:1234/v1.

So "make this app local" usually reduces to: point its base URL at one of those, put any non-empty string where it wants an API key, and pick a model you have pulled.

The 60-second version

Ollama:

brew install ollama        # or the .dmg from ollama.com
ollama serve &             # server on :11434
ollama pull llama3.1:8b    # pull a model once

Confirm it speaks OpenAI:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.1:8b",
    "messages": [{"role": "user", "content": "say hi in 3 words"}]
  }'

If that returns a choices[0].message.content, any OpenAI-compatible client can use it. In the app, set:

Base URL: http://localhost:11434/v1
API key: ollama (or literally anything; it is ignored)
Model: llama3.1:8b

LM Studio is the same idea with a GUI: load a model, toggle the server on, and use base URL http://localhost:1234/v1.

Pointing real tools at it

The pattern shows up everywhere once you look for it. The official OpenAI SDKs are the clearest example: change one field.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
r = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "summarize this in one line: ..."}],
)
print(r.choices[0].message.content)

Same code, cloud or local. Only the base_url changed. In JavaScript it is the baseURL option; in a lot of CLI and editor tools it is an OPENAI_BASE_URL environment variable. Some end-user apps expose it too: DEVONthink 4, for instance, lets its Chat point at Ollama or LM Studio directly, so search and summarize run over your own documents with nothing leaving the machine.

The honest gotchas

It is not always drop-in. The places it breaks, roughly in order of how often they catch people:

Context window. A local model often defaults to a small context (Ollama defaults to 2048 tokens unless you raise num_ctx). If an app sends a big document and the model only reads the last 2k tokens, it looks like the model "ignored" your content. Raise the context in the model config, or send smaller chunks.
Missing features. Some apps assume function calling, JSON mode, or vision. Not every local model or runner supports all of them, and the failure is often silent. Check the runner's compatibility notes for the specific feature you depend on.
Streaming quirks. Most local servers stream fine, but a few clients expect particular SSE framing. If a response hangs, turn streaming off in the client as a test to isolate it.
Model quality. A 7-8B model is not GPT-class. For classification, summarizing, tagging, and search over your own material it is often plenty; for long chains of reasoning it will disappoint. Match the job to the model rather than expecting parity.
RAM. Everything is bounded by unified memory on a Mac. 16GB comfortably runs a capable 7-8B model; below that, stay in the 3B range and keep the context modest.

Why bother

Two reasons that actually matter. It is free after the download, so there is no per-token meter running while you iterate on a prompt fifty times. And nothing leaves the machine, which for anything sensitive (client data, personal notes, a private codebase) is the difference between "I can use AI on this" and "I am not allowed to."

The best part is that because it is one interface, you do not have to commit. Keep a cloud base URL for the genuinely hard reasoning and a local one for the bulk, private, high-volume work, and switch between them by changing a single string.

Written with AI assistance and edited by a human. Endpoint details reflect public docs as of July 2026 and move quickly, so check each project's current docs for exact paths and defaults.

MLX vs GGUF on Apple Silicon: which local LLM format should you actually use?

Jackson Ly — Tue, 07 Jul 2026 01:16:53 +0000

If you run local models on a Mac, you eventually hit the same fork in the road: the same model is available as a GGUF file and as an MLX version, and something has to tell you which one to download. The short answer is that MLX is faster on Apple Silicon and GGUF goes everywhere. The useful answer is knowing when that trade actually matters, because for a lot of setups it does not.

Here is the practical version, with the reasons.

What the two formats actually are

GGUF is a single self-contained file. Weights, tokenizer, metadata, and quantization parameters are all bundled into one portable blob that llama.cpp (and everything built on it) can load anywhere: Mac, Linux, Windows, CPU, CUDA, Metal. That portability is the whole point of the format.

MLX is Apple's array framework, not a file. An MLX model is a directory of safetensors files plus a config that the MLX runtime reads directly. It is built to run on Apple Silicon and quantize natively against the unified memory pool. It does not leave Apple Silicon, full stop.

So this is not really "two file formats." It is one portable format and one Apple-native runtime that happen to ship the same model weights.

The performance gap is real but bounded

On the same Mac, at the same quantization level, MLX runs roughly 15 to 40 percent faster than GGUF and uses about 10 percent less memory. The speedup comes from MLX being compiled for Apple Silicon and operating directly on the unified memory pool, where the CPU and GPU share one block of RAM and every gigabyte is usable as model memory.

There is one quality nuance worth knowing. At 4-bit, GGUF's Q4_K_M uses mixed precision inside each layer, so it can hold quality slightly better than a naive 4-bit MLX quant. It is a small difference, but if you are quantizing aggressively on a memory-tight machine, it is the kind of thing that shows up in outputs.

	GGUF	MLX
What it is	Portable single file	Apple framework, safetensors dir
Runs on	Mac, Linux, Windows, CPU, CUDA, Metal	Apple Silicon only
Speed on M-series	Baseline	~15 to 40% faster
Memory	Baseline	~10% less
4-bit quality	Slightly better (Q4_K_M mixed precision)	Fine, marginally behind at 4-bit
Best for	Portability, ecosystem, cross-platform	Raw throughput on Apple Silicon

Tool support is the part people get wrong

The format you should pick is partly decided by the tool you already use.

LM Studio has had an MLX backend since late 2024 and also runs GGUF through its bundled llama.cpp. So on LM Studio you genuinely get to choose per model, and picking the MLX build on an M-series chip is usually free speed.

Ollama ran GGUF through llama.cpp's Metal path for a long time. Its 0.19 preview added an optional MLX backend that reports roughly double the old speed, but it targets Macs with 32GB or more of unified memory. On a 16GB machine you stay on the GGUF/Metal path, so the MLX question does not even come up for you yet.

That 32GB line matters more than the raw benchmark. A lot of "MLX is 2x faster" claims quietly assume a machine most people do not have.

The portability catch that bites later

MLX being Apple-only is fine until it is not. If you are building something that might need a CUDA fallback, a Linux server, or a Windows user, and you shipped MLX as your only build, you are re-quantizing under pressure the day that requirement lands. GGUF would have just run.

This is the real decision axis, and it is not about speed. It is about how long your setup has to live and where it has to run. A weekend project that only ever runs on your own MacBook has no reason not to take the MLX speedup. Infrastructure that will outlive any single runtime should default to GGUF, or ship both.

The decision, compressed

Personal use on an M-series Mac with 32GB or more: take MLX. It is faster and lighter and you lose nothing you will miss.
16GB Mac, or you just want the simplest path: GGUF. You are likely on the GGUF path anyway, and the portability is free insurance.
Cross-platform, a server, or anything that might need CUDA later: GGUF, or build both. Do not paint yourself into an Apple-only corner for a 20 percent speedup.
You are not sure: GGUF. It runs everywhere, and you can always add the MLX build later when you know you want it.

The honest summary is that both are good, and on a modern Mac you can usually try each in a couple of minutes with the same tool. Speed is the easy axis to measure and the wrong one to optimize first. Portability is the one you only notice when it is gone.

Written with AI assistance and edited by a human. Format details reflect public information as of July 2026 and move quickly (Ollama's MLX backend in particular is young), so check each tool's current release notes before you rely on a specific number.

What belongs in an agent's context window (and what to evict)

Jackson Ly — Mon, 06 Jul 2026 11:37:07 +0000

Most "agent memory" writeups are about getting things in: which vector store, how to chunk, how to embed. The harder question in practice is the opposite one. Your context window is a fixed budget. Every token you spend on a stale tool output is a token you did not spend on the thing that actually decides the next step. So the real engineering problem is not retrieval, it is eviction: what earns a place in context, and what gets dropped or compressed.

Here is the model that has held up for me across a few agent builds.

Context is a working set, not a memory

Borrow the term from operating systems. The working set is the small slice of memory a process actually touches right now. Everything else lives on disk and gets paged in on demand. An agent's context window is a working set, and treating it like long-term storage is the mistake that makes agents slow, expensive, and confused.

Durable memory (the vector store, the notes, the logs) is your disk. It can be huge. The context window is RAM: small, hot, and contested. The job of "agent memory" is mostly the pager: deciding what to bring into the working set for this step and what to let fall back to disk.

Once you see it this way, most context bloat is a pager bug. You retrieved 40 chunks because they matched, kept all 40 in context for the rest of the session, and now every subsequent step pays attention over 40 things that stopped being relevant 10 steps ago.

Two axes, not one

The thing that made my eviction decisions cleaner was realizing they run on two independent axes, and people collapse them.

Fidelity: must stay exact vs can be approximated. Some content breaks if you compress it. A commitment the agent made ("I will not touch prod"), a tool's exact error string, an ID, the user's literal instruction. Others degrade gracefully: a long document you only need the gist of, an old reasoning trace, the transcript of a resolved subtask. Summarize the second kind aggressively. Never summarize the first.

Recency vs salience. Recency is easy and mostly wrong on its own. The token that matters is not the most recent one, it is the most salient one for the current step: the retrieved fact this decision hinges on, the constraint that must not be violated, the goal. Recent context is one bucket of salient content, not the definition of it.

Cross the two axes and you get a priority you can actually implement. High-salience-must-stay-exact is sacred: keep it verbatim, evict last. Low-salience-can-approximate is the first thing you compress or drop. The dangerous quadrant is high-fidelity-but-low-salience: exact content you are paying full price to keep, that is not helping this step. That is where most of the waste lives.

A concrete eviction policy

You do not need anything fancy to start. A policy that beats "keep everything until the window overflows":

Pin the invariants. Goal, hard constraints, the user's literal ask. These never get evicted or summarized. Small, always present.
Keep the last N turns raw. Recency is a cheap proxy for salience for conversational coherence. N is small, like 3 to 6.
Demote retrieved evidence after it is used. A chunk that informed a step gets summarized to its one-line claim plus a pointer back to durable memory. If a later step needs the detail, page it back in.
Compress resolved subtasks to their outcome. The full transcript of "figured out the config" becomes "config is X, verified." The reasoning got you there, but it is not load-bearing anymore.
Evict on salience, not age. When you need room, drop the lowest-salience item, not the oldest. An old invariant outranks a fresh but irrelevant tool dump.

The pointer-back part is what makes this safe. You are not deleting information, you are moving it from RAM to disk with a way to fault it back. Summarization without a pointer is just lossy deletion, and that is how agents "forget" things they clearly should have known.

The failure mode to watch

The subtle bug is trusting your own summaries. When you compress retrieved evidence into a one-line claim, that claim is now a memory the agent will act on, and it was written by an LLM that may have smoothed over the caveat that mattered. Keep the pointer to the source, and when a step is high-stakes, page the raw evidence back in instead of trusting the summary. Fidelity and salience decide what to keep; stakes decide when to distrust your own compression.

None of this needs a new framework. It needs you to stop treating the context window as storage and start treating it as a scheduler: a small, contested resource where the real skill is deciding what to page out.

Written with AI assistance and edited by a human. I think about this because I build a local-first personal memory layer, where the "disk" is your own captured work and the pager runs on your machine.

What poisoning a RAG store taught us about agent memory

Jackson Ly — Mon, 06 Jul 2026 04:37:45 +0000

Most of the "AI memory" conversation is about getting things in: chunking, embeddings, which vector database, how to compress a long chat into something you can recall later. Almost none of it is about what happens when one of those memories is wrong on purpose. That is the part we cannot stop thinking about while building a local-first personal memory layer, because an assistant that remembers you is also an assistant that can be talked into remembering a lie.

Key takeaways

Do not try to detect a poisoned memory. Gate what an unproven memory is allowed to do.
"Two independent sources" is the retrieval spelling of an old, unsolved problem: permissionless membership. Independence is not free.
Standing earned from outcomes beats standing bought with identity, but the agent cannot grade its own homework.
A per-action cap does not stop a slow drip. You need a cumulative budget plus a reversible human gate on the irreversible tail.
Everything rests on one axis nobody bounds: provenance that survives transformation. That is the floor.

This is the writeup of a public thread on r/LangChain that turned into one of the best technical exchanges we have had in a while. It started with a builder (the maintainer of an open-source memory layer, mnemo / agora-mnemo) doing something more people should do: he tried to poison his own RAG store and reported honestly on what held and what did not. Over about a week he built each idea, measured it, tried to break it, and shipped the parts that survived. Credit for the experiments and the shipped code is his. Our part was mostly the framing.

Why do retrieval-time poison defenses fail?

The naive defense against a poisoned document is to detect the poison: compute perplexity, look for outliers, flag the weird chunk. This is the wall that keeps failing, and it fails for a boring reason. It lives in embedding space, so it breaks the moment you swap the encoder. His results showed exactly that. A defense tuned on one embedder fell over on the next.

The reframe that travels further: you did not build a poison detector, you rediscovered that retrieval and authority are two different privileges, and only the second one has to be earned. In most memory stacks they are the same thing. If a chunk comes back in the top-k, it gets to shape the answer. Separate them. Let untrusted content inform the model, and never let it authorize an action on its own. That is the same split that holds up best against prompt injection (the dual-LLM, capabilities pattern in Google DeepMind's CaMeL work): the privileged planner never touches raw untrusted text, and untrusted values only reach a sensitive tool call through a typed capability a policy has to allow.

Retrieval is access control's easy half. The hard half is information flow: a value admitted for a cheap read can still poison the context a trusted action later consumes. The fix is not a better authority tier, it is a low-water mark. An action inherits the minimum integrity of everything in its context, so the instant a single-source, unproven memory enters, the action's effective integrity drops and it fails closed. If one poisoned read does not drag the whole action down, your labels are not propagating.

Why are "independent sources" not actually free?

Once you gate on authority, the tempting rule is "trust a claim if at least two independent sources agree." The word doing all the work is independent. If an attacker can mint sources cheaply (new domains, new document ids, new ingest times), corroboration collapses into a Sybil attack: forge three records, forge the independence, graduate your own poison. He measured it. Two freshly registered domains supplying their own corroboration sail through the count gate.

This is not a RAG quirk. "Two independent sources" is the retrieval spelling of permissionless membership, and it has been open for decades. Byzantine agreement is safe up to a fraction of faulty replicas only if the replicas are actually independent, and that independence was always assumed out of band. The only ways anyone has ever gotten it without a trusted registrar are to make identity scarce: proof of work, stake, or a real-world attestation. So the design question stops being "is this chunk poisoned" (encoder-dependent, keeps breaking) and becomes "what makes a source costly to mint" (a membership-cost question people have studied for twenty years).

What actually survived the experiments?

He built four ways to make a corroborating source costly (an allowlist or registrar, signed attestation, proof of work, and staked-with-decay standing) and measured each against a forged multi-source poison. The honest result sorts cleanly:

Cost alone only prices the attack, it does not wall it. Proof of work and staking are symmetric taxes: the honest writer pays the same, and by a classic result (Cheng and Friedman, 2005) no symmetric reputation function can be Sybil-proof.
What helps is asymmetric and rooted outside the system. Standing earned from outcomes (did acting on this memory turn out well) is the only one of the four with an asymmetric root. A wrong poison cannot earn a good outcome.
But the agent cannot grade its own homework. If the outcome signal is endogenous (an attack like MINJA gets the agent to write its own corroboration), the whole thing collapses. The only unforgeable outcome is a real consequence the attacker actually pays, and you can only afford to observe that on actions you were willing to be wrong about. So outcome-learning and action-gating are the same boundary: you earn standing on the reversible set, and high-stakes actions never learn, they only get gated.

There is a density catch worth stating plainly: only re-used memory earns standing, so the cold long tail never accrues it and lives under the cap permanently. Standing-gating is a hot-path privilege, not a global one.

The attack that quietly defeats a per-action cap

The residual we thought we had was "one patient sleeper, single shot, under a blast-radius cap." That was wrong, and it is the correction we would most want you to take away. A per-item cap is silent on a run of individually reversible writes that sum past the reversibility threshold. That is salami-slicing, and he measured a version compounding into an irreversible outcome. The only scalar that bounds it is a cumulative per-source budget, not a per-action one. The only lever that bites a patient attacker (who just amortizes a small cap over a long horizon) is retroactive forfeiture: accrued standing becomes a bond you burn on a caught defection, so effective cost scales with one over the probability of getting caught, not with the cap.

Ship those and you get the honest shape of the whole problem: you do not remove the residual, you price it, bound it, and put a reversible human gate on the irreversible tail. Auto-forfeiture stays off by default, because on an attacker-influenceable oracle an automatic penalty is a framing weapon (feed bad outcomes attributed to a rival, get the rival auto-slashed). The human is not there to be the truth oracle. They are there to keep forfeiture rare and reversible, which is the only property that defangs the framing attack.

The floor everything stands on

Strip the layers down and they all quietly ride on one axis nobody had bounded: attribution. The detector, the budget, and the gate all assume you can correctly assign an action to its source and label whether it is irreversible. A single relabel does not degrade the other three, it voids all of them at once and silently, because they are all keyed on the same source id. Attribution is not a fourth control. It is the floor the other three stand on.

You can make attribution tamper-evident with textbook cryptography (hash-chained receipts, in the Haber and Stornetta lineage), so a post-hoc relabel stops being silent. What you cannot get from inside a single system is attribution you can trust at write time: if the attacker controls the labeling channel, the label's truth is exactly the thing they control. That only moves if you bind the label to something the attacker does not hold at write time (origin-signed provenance, or independent corroboration of the world), at which point "can I trust the label" becomes "can I trust the root," which is the identity question again. The tower terminates on exactly one thing you choose to trust from outside, and everything above it is mechanism.

Where this lands for a personal, local-first brain

We are building recal as a local-first personal memory layer: something that watches how you actually work and can act on it, on your own machine, behind an approve-gate you control. Personal memory raises the stakes on every point above. The "sources" are your own captured activity and notes, the "irreversible actions" are things done on your behalf, and there is no cloud moderation team to catch a bad memory for you. The takeaway we are designing around is the one this thread earned: do not detect bad memories, gate what an unproven memory is allowed to do, keep the destructive tail behind a reversible human gate, and treat provenance-that-survives-transformation as the substrate the whole thing rests on. "Solved" was always the wrong word. Turning an encoder-resetting arms race into a bounded, priced, auditable risk is the shippable win.

FAQ

Does a knowledge graph or better vector database fix RAG poisoning?
No. Both are retrieval improvements, and the failure is that retrieval is treated as authority. The fix is to separate the two: retrieve widely, but let only corroborated, provenance-tracked memory drive an action.

Is detecting AI-generated or poisoned text a viable defense?
On its own, no. Detection lives in embedding space and resets every time the encoder changes. Gating what a memory is allowed to do generalizes across encoders in a way detection does not.

What is the single most important primitive to build first?
Provenance that survives transformation (chunking, summarization, re-embedding). Every other control (budgets, standing, slashing) is keyed on knowing which source an action came from, so attribution is the floor.

Does this require a blockchain?
No. The useful cryptography here is boring and old: hash-chained, tamper-evident receipts. The hard, still-open part is not tamper-evidence, it is trusting a label at write time, which needs an external signing root, not a consensus network.

Written with AI assistance and edited by a human. This is the synthesis of a real public thread on r/LangChain; the experiments and shipped code are the mnemo / agora-mnemo maintainer's, and the framing was developed collaboratively in the open. Technical claims (CaMeL, Cheng and Friedman 2005, MINJA, Sybil / Douceur 2002, Haber and Stornetta) were checked against their primary sources.

Why I built my Mac assistant to run 100% on-device (and what local-first actually cost me)

Jackson Ly — Mon, 29 Jun 2026 06:44:55 +0000

I'm building a proactive personal assistant for the Mac called recal. It watches how I work, learns my patterns, and starts doing the repetitive busywork for me. The one constraint I refused to bend on: it runs entirely on-device. No cloud, no account, no server. 0 bytes of my activity leave the machine.

This is the honest engineering version of that decision: why I made it, what it bought me, and what it genuinely cost. I'm the founder, building in public, and the product is pre-launch, so this is about the architecture, not a sales pitch.

Privacy by architecture, not by policy

Every AI productivity tool I tried wanted the same thing: my screen, my files, my activity history, living on someone else's server. For a tool whose entire job is to watch how I work all day, "trust our privacy policy" was never going to be enough. A policy is a promise. Architecture is a guarantee.

If the data never leaves the device, there is no server to breach, no account to leak, no policy to quietly change next quarter. You can pull the ethernet cable and it still works. That is a fundamentally different security model than "we encrypt it in transit."

The catch is that you have to actually build for it, and "local-first" stops being a marketing word the moment you hit the real engineering.

Capture is the scariest part, so it is default-deny

A tool that observes your work is one bad decision away from being spyware. The Microsoft Recall backlash made that vivid for everyone. So the capture layer is built default-deny: it does not record unless a specific, allowed signal says it is safe to.

The model I kept coming back to is Helen Nissenbaum's contextual integrity: information is not simply private or public, it is appropriate or inappropriate to a context. A password field, a banking tab, an incognito window are contexts where capture is never appropriate, no matter how useful the data would be. So sensitivity is a first-class tag on every observed event, decided at capture time, and anything uncertain gets dropped rather than stored. The expensive, paranoid default is the correct one here.

Local retrieval without a cloud index

The fun objection to local-first AI is "but the good models live in data centers." True, and I am not pretending a Mac runs a frontier model. But most of what a personal assistant needs is not a giant model, it is your own context, retrievable fast.

That part runs entirely on-device: content gets embedded with a small ONNX model running locally, and retrieval is a mix of embeddings, recency, and plain filters. For personal re-finding ("what was I looking at last Tuesday before the meeting"), that combination beats a cloud round-trip, and it never ships your life to an index you do not control. The thing that surprised me: you need far less model than the industry implies, as long as you are retrieving over your own data instead of trying to reason about the whole world.

The part I care about most: it proposes, I approve

"It did something on its own" is terrifying the first time it is wrong. So recal never acts on its own. It does the work in the background, then surfaces a finished result for one tap: approve, or do not.

That approve-gate is not a UX afterthought, it is the trust contract. It keeps a human as the one deciding while the machine does the doing. It also makes a wrong proposal cheap (you reject it and move on) instead of catastrophic (it already emailed the wrong person). Building the gate first, before the autonomy, was the single best product decision I made.

What it cost

Being honest, because that is the point of building in public.

No giant cloud model to lean on. Anything that genuinely needs frontier-scale reasoning has to be designed around, deferred, or made opt-in, and you feel that constraint daily.
Capture you can trust is slow to build. Default-deny means you are constantly saying no to data you would love to have, and writing the rules that decide "appropriate" is most of the work.
You carry the whole stack. There is no server to hotfix. The observation engine, the index, and the agent all ship to the user's machine in Swift and Rust, and a bug is a bug on their hardware, not a deploy you roll back.

Would I do it again? Yes, without hesitation. The cost is real, but what I get back is a tool I would actually trust to watch my whole working life, which is the only kind worth building.

If the on-device, local-first direction resonates, I am building recal in public and there is a waitlist at recal.so. But mostly I wanted to write the architecture down honestly, because I think more personal software should be built this way, and I would genuinely like to hear how others are drawing the same lines.