basemind is my code-intelligence tool: a pure-Rust code map and scanner, tree-sitter across 300+ languages, a content-addressed blob store, a Fjall-backed inverted index. I harden it by pointing it at a repo big enough to hurt, a 20M-line, 200k-commit monorepo, and watching what it does.
I did this debugging session with Claude Code. It started as "which version is installed?" and ended with a 7x speedup, two upstream PRs, and a feature turned off. Here is how I worked, and the methods that made it effective.
The symptom is not the bug
The report: basemind's cache grew from 1.8 GB to 4 GB while CPU pinned for a long time. I did not open the scanner code. I looked at the live system. ps and lsof found a process at ~70% CPU holding the index open, and it was a serve process, not a scan. Different bug entirely.
That process was an old release. So I had Claude dig through git instead of the debugger: git log on the watcher, then git merge-base --is-ancestor and commit dates. A commit titled "fix: stop watcher CPU runaway" was dated four days after the running binary shipped. The bug (a watcher re-scanning its own writes forever) was already fixed. It was a stale process left over from another editor session.
Two rules, both applied before reading a line of source:
- Verify the running artifact, not a proxy. Homebrew, the PATH, and the editor plugin each reported a different version. Find the one that actually runs.
- "Live bug or old build?" is the first question. Dating the fix against the running version can end an investigation before it starts.
Measurement tells a scan from a leak
I killed the ghost, and a fresh scan appeared at 488% CPU, nearly five cores. That is rayon pegging every core, and it is indistinguishable from a runaway loop on a single reading.
So I sampled three signals over time: CPU%, cache size, blob count. Disk climbed then plateaued. Blob writes burst then stopped. CPU fell to one core. That is a one-time rebuild converging, not a leak. A leak keeps allocating, a loop keeps writing, a scan converges. The derivative is the diagnosis, not the number.
To watch without babysitting, I had Claude write background monitors: shell loops sampling every 30s, printing only on a state transition. One fired on "SCAN COMPLETE + IDLE," another on "0% CPU and no disk growth for 4 minutes." The alert encodes the hypothesis: a hang is low CPU and no progress, a healthy scan is high CPU or growing disk.
Isolate the confound
A multi-worktree test then hung: scans wedged at 0% CPU, and lsof on a stuck process itself hung for two minutes. A hanging lsof means a stuck socket. The last log line was Registering VLM OCR backend. The OCR tier made a network call with no timeout, a firewall dropped it, and the connection blocked forever.
I did not debug the hang in place. That tier is opt-in behind a cargo feature, so I built basemind with default features, a binary that cannot make the call, and studied the scanner in isolation. When two failures tangle, change one variable.
Read for the mechanism, then measure the fix
On a clean binary I took a baseline. A fresh worktree scan took 181s, re-parsed all 67,700 files though their blobs already existed, and rebuilt the git-history index per worktree at ~160s. Now I read code, to confirm the mechanism, not guess a patch.
- The "unchanged file" fast paths keyed off the per-worktree index, empty on a fresh worktree. So every file fell through to a full re-parse, despite its blob already sitting in the shared store.
- The git-history index opened at the worktree's own cache dir, though it derives entirely from the shared
.git.
Both fixes followed patterns already in the code, and each got a regression test written to fail first. Re-measured: 181s to 25s, ~7x, all 67,700 files reused, peak RSS about halved. The re-measurement is the point. It turns "should be faster" into "is 7x faster."
Follow the failure across the boundary
The network hang still deserved a fix. I traced it: the OCR and embedding models download through hf_hub, whose builder in 0.4 and 0.5 constructs a ureq agent with no timeout. A firewalled download blocks forever. The root cause was in the dependency.
I forked it and had Claude map every download call site across two crates, then add a watchdog: run the blocking fetch on a detached thread, error on a tunable deadline, degrade by skipping the model instead of hanging. The mechanical multi-file edit went to a subagent with a tight spec. Then I read the diff and reran the build and tests myself. The subagent even corrected my assumption about a type not being Clone. That shipped as an upstream PR plus an honest bug report. Delegate the mechanical work, verify with diffs and reruns, never a summary.
Sometimes the fix is deletion
Monitoring caught the embedding pass stuck on a general-purpose model, an empty model dir with a stale lock, spinning a core. The sharper question was not why it was slow, but whether code embeddings should exist at all.
I traced what gets embedded (symbol, signature, doc, body) against what embeds it (a general-English model), and basemind already builds a BM25 index over the same text. A prose model embeds code weakly, the one real win (natural language to symbol) is already covered by the keyword lane, and the cost is a 100 MB pull plus gigabytes of vectors. Verdict: code embeddings off by default, kept for docs and images. A feature that is off cannot hang, leak, or pin a core.
The methodology
- Observe the live system (ps, lsof, du) before reading source.
- Separate stale-artifact bugs from live bugs by dating the fix against the running version.
- Measure dynamics, not snapshots. The derivative separates converging from leaking.
- Encode the hypothesis into a monitor so the machine watches for you.
- Isolate confounds by changing one variable.
- Read code for the mechanism, then write the smallest fix.
- Prove it with a test that fails without the fix, and re-measure the metric you started from.
- Follow failures into dependencies, and report them honestly.
- Ask whether the costly feature should exist, not just why it broke.
- Delegate the mechanical work to an agent, but verify every diff yourself.
What shipped
- Worktree cache reuse: cold scan 181s to 25s (~7x), 67,700 files reused, peak RSS about halved.
- Code embeddings off by default, per-file
embed_exclude, safe preset swapping. - An upstream timeout guardrail in the dependency, plus a bug report.
The method I kept coming back to: verify the artifact, watch the live system, measure the change over time, isolate the confound, read for the mechanism, and prove the fix by the number you started from. basemind lives at github.com/Goldziher/basemind.
Top comments (10)
The monitors are the part I'd steal. Most people alert on a threshold (CPU > X, disk > Y) and then can't tell rayon pegging five cores from a runaway loop, because both trip the same number. You encoded the hypothesis into the predicate instead: hang is low CPU AND no progress, a healthy scan is high CPU OR growing disk. That's the derivative-is-the-diagnosis point made operational. The alert fires on a state transition that distinguishes the two shapes, not on a level either shape can reach.
The 'live bug or old build' check, dating the fix against the running binary, is the one I see skipped most and it's the cheapest. Half of 'I can't reproduce the bug' sessions are really 'you're running last week's binary.' Verifying the running artifact before reading a line of source should be step zero, and almost never is.
Exactly, the predicate is where the work happens. A level threshold can't separate the two because both hit the same number, so the alert has to fire on the transition: low CPU with flat progress is a hang, high CPU or growing disk is a healthy scan. And the running-artifact check being step zero is the one I see skipped most too. I stamp the build and diff it against the fix before reading a line, because half of "can't reproduce" is really "wrong binary."
The stale-process finding is the whole post in one bug: an old binary re-scanning its own writes was never in the source you were reading, because source is a map of the inputs it named, and the cause lived in the one it did not, a live actor nobody declared. That is exactly why rule one lands, observe the live system before the code. Reading harder would never have surfaced it, the file you were staring at was innocent and complete.
One I would add next to rule three: the derivative separates convergence from a leak right up until the process is feeding itself, and then a self-consuming loop reads as either depending on when you sample. The clean tell is not a metric, it is killing the suspected actor and watching the dynamics change. Growth survives the kill, it was passive; growth stops, it was a live actor. That one experiment splits them faster than any amount of sampling. Good writeup, and rule nine, whether the embeddings deserved to exist at all, is the one most people never reach.
Thank you, and I agree. Sampling tells you the shape of the dynamics; killing the suspected actor tells you the causation. I leaned on the derivative because it's passive and cheap, but you're right that a self-feeding loop can read as convergence depending on when you sample.
Right, and I would put them in series rather than pick one: the derivative is the cheap passive detector that fingers a suspect, the kill is the expensive confirmer you spend only on that one. You do not kill everything, just the thing the sampling already flagged. And the catch that keeps the derivative valuable is that the kill is a perturbation, in production you often cannot take the suspected actor down just to check, so the passive signal is what you lean on when killing is too costly, and the kill is the confirmation you earn your way to when the dynamics stay ambiguous and taking it down is safe. Cheap detector, expensive verdict.
In series is the right framing, and the perturbation point is the crux. In prod you usually can't take the suspect down just to check, so the passive signal carries you until a kill is cheap enough to be worth the verdict. Cheap detector, expensive confirmer, spent only on what the derivative already flagged. Good thread.
Thanks Na'aman. One last crumb for the drawer: when the kill never gets cheap because the suspect is never safe to take down, a shadow replica is the middle path. You perturb the clone, not prod, and the verdict costs you infra instead of uptime.
The "live system first, source second" discipline here is the right call, and the version-confusion angle is a trap I have fallen into myself more than once â the artifact that shipped and the artifact you are debugging are often not the same object.
The derivative-based diagnosis (CPU% + disk growth + blob count) is a clean mental model. I have used something similar for embedded storage systems where the equivalent signals are write amplification, compaction queue depth, and WAL head vs tail distance. Same principle: a converging scan has a directional derivative, a leak has zero mean, and a hang has zero both.
On the OCR tier blocking forever with no timeout â this is exactly the failure mode that bites in any async Rust service that calls out to a plugin ecosystem. The fix (build without the feature flag) is the right instinct, but a more general guard is wrapping every network call in a oneshot with a timeout channel, so the behavior is explicit and the default is "fail fast" rather than "hang indefinitely." A trait bound like T: WithTimeout would be too restrictive, but a helper that wraps any Future is a one-liner in tokio.
Did you end up instrumenting the network calls themselves for timeout, or just disabling the feature for the monorepo scans?
The "live system first, source second" discipline here is the right call, and the version-confusion angle is a trap I have fallen into myself more than once â the artifact that shipped and the artifact you are debugging are often not the same object.
The derivative-based diagnosis (CPU% + disk growth + blob count) is a clean mental model. I have used something similar for embedded storage systems where the equivalent signals are write amplification, compaction queue depth, and WAL head vs tail distance. Same principle: a converging scan has a directional derivative, a leak has zero mean, and a hang has zero both.
On the OCR tier blocking forever with no timeout â this is exactly the failure mode that bites in any async Rust service that calls out to a plugin ecosystem. The fix (build without the feature flag) is the right instinct, but a more general guard is wrapping every network call in a oneshot with a timeout channel, so the behavior is explicit and the default is "fail fast" rather than "hang indefinitely." A trait bound like T: WithTimeout would be too restrictive, but a helper that wraps any Future is a one-liner in tokio.
Did you end up instrumenting the network calls themselves for timeout, or just disabling the feature for the monorepo scans?
Some comments may only be visible to logged-in visitors. Sign in to view all comments.