I built a small MCP server called Codicil that indexes docs in a repo and lets an AI coding assistant search them instead of guessing. Runbooks, config notes, whatever I've written down about my homelab. It's just for me, one repo, one user, nobody else touches it.
For months it ran against a real Ollama box doing embeddings with nomic-embed-text, so every query was an actual semantic search over vectors. Then I decommissioned that box for unrelated reasons and the embedding endpoint went dark.
I didn't notice for something like two weeks. Queries still came back with reasonable answers so I just kept using it. Turns out it had fallen back to grepping the raw files the whole time, and it never said a word about it.
I don't remember exactly what tipped me off at this point, just that at some point I went looking and realized it had been running on keyword fallback for a while.
Honestly my first thought was that this is a little embarrassing. I'm the only person who uses this thing, so I'm also the only monitoring it has, and I went two weeks without checking. But then I actually looked at why it didn't just break, and that part I don't think is embarrassing at all.
Most tools like this have one failure mode when the embedding service goes away: they stop working. No embedding, no vector, no search. I didn't want that, so a bunch of the code is just "what happens if this specific thing isn't there" handling, scattered around instead of centralized:
- No embedding host reachable? Falls back to grepping files off disk directly.
- Index empty or never built? Same fallback.
- Reindex dies halfway through a file? New chunks get written before old ones are deleted, so a crash mid-run leaves you stale, not empty.
- Two processes hit the store at once? This one actually got me for real. A long-running
serveprocess and a one-offcodicil indexfrom a cron job raced each other. There's an exclusivefcntl.flockon the store now (fcntl.flock(fd, LOCK_EX | LOCK_NB)), so the second process gets refused outright instead of getting to touch anything.
None of that is exotic, it's just annoying to write and easy to skip. I skipped some of it the first time through, which is how the process-race thing happened.
Keyword search is worse than semantic search, for the record. It doesn't get synonyms, it just counts overlapping words. If I ran both side by side on the same queries, semantic wins most of the time. But it only wins when the embedding host is actually up, and mine wasn't, for two weeks, and I didn't know.
There's also a dumber bug I found while doing all this that's worth mentioning because it's the kind of thing that's obvious in hindsight: each embedding model gets its own Chroma collection now (docs_<sha256-of-model-name>[:12]), because Chroma fixes a collection's vector dimension the first time you write to it. Before that fix, switching embedding models against the same collection would've just broken silently the next time you tried to write a different-sized vector into it. I hadn't hit this one in practice, I just noticed it could happen and fixed it before it did.
I've been putting off publishing this because I kept trying to make the ending land on some bigger point about AI tooling and knowledge decay, and every version of that ending sounded like something off a conference slide. So I'm just going to leave it here: the embedding host died, I didn't notice, and the reason I didn't notice is the only part of this worth writing down.
Codicil's open source: github.com/colehellman/codicil.
Top comments (2)
I think there are two additional reasons the fallback remained convincing.
First, your corpus is unusually friendly to lexical search. Semantic search is most valuable when the query language differs from the source language. Here, the notes and queries come from the same person, so terminology and phrasing are likely to overlap.
I saw the opposite in message-recall work: lexical search performed much worse when the writers and searchers were different people. “Punctured wheel” does not find “flat tire,” even though they may describe the same event.
Second, the consumer is an AI coding assistant. Depending on how it invokes MCP tools, it can act as a query rewriter: extracting likely keywords, trying another phrasing, and synthesizing an answer from partial matches. That can mask a real decline in retrieval quality. Grep plus an agent over a small, self-authored corpus may genuinely work well enough much of the time.
Your ending captures the important failure mode: retrieval degradation is hard to notice because the missing results are invisible.
Two low-cost safeguards would help: expose or log which backend served every query, and add a direct retrieval canary—a fixed document and query that match only semantically. Run it below the LLM layer and alert when either the semantic backend or expected document is missing. That would catch the fallback within minutes rather than weeks.
That's a good point. My experiment definitely had two favorable conditions: I wrote both the code and the queries, and Claude could reformulate queries before calling the MCP tools. I hadn't really considered the latter as a source of resilience until you mentioned it. I especially like the canary idea.
My current health checks only verified that the embedding service was reachable, not that semantic retrieval was actually being used. A fixed semantic-only query below the LLM layer would have exposed the fallback almost immediately.