Developers are pushing back against cloud API billing and the privacy risks of sending proprietary codebases to third-party endpoints. A Hacker News thread from this year put it plainly: the problem isn't the price per token, it's the unpredictability of usage-based billing when AI agents are continuously polling APIs. On Reddit, the privacy concern is starker -- for enterprise and defense work, sending company IP to OpenAI or Anthropic is a hard no regardless of cost.
zerikai_memory has a local mode for exactly this: everything runs through Ollama, nothing leaves your machine. We shipped mistral:7b as the default local model. ornith:9b dropped in June 2026, trained specifically for agentic coding tasks, so we tested both. Here is what we found.
How zerikai_memory Uses a Local Model
zerikai_memory runs in three modes: cloud (DeepSeek), local (Ollama), and hybrid. Routing between them is handled by _should_use_cloud() at main.py:986, a 4-step priority chain: explicit override, keyword match, word count threshold, then MEMORY_MODE env var fallback. In local mode that function always returns false -- everything stays on-device.
In local mode, every synthesis call hits _query_ollama at main.py:1555. The model receives a project brief from _load_project_context plus structured ChromaDB entity payloads: function signatures, file paths, line ranges, docstrings. It returns an answer with inline #file:line citations. One call, no streaming, no tool loop.
The retrieval layer (ChromaDB + L2 distance + lexical rerank) is model-agnostic. Both models received identical context in every query test. The only variable was synthesis.
Hardware
- GPU: NVIDIA RTX 3050, 8GB GDDR6 dedicated VRAM
- CPU: Intel i7-12700
- RAM: 32GB
- OS: Windows 11
8GB dedicated VRAM is the hard ceiling. Windows offers shared system memory as overflow, but inference on shared RAM through PCIe is slow enough to matter in practice.
Models
mistral:7b (v0.3) -- 4.4GB, 32K context. Function calling via Ollama raw mode only: you construct [AVAILABLE_TOOLS] prompts by hand and parse [TOOL_CALLS] responses yourself. Last updated May 2024.
ornith:9b-dense -- roughly 5.5GB estimated, built on Gemma 4 and Qwen 3.5. RL-trained for agentic coding tasks with native tool calling. 69.4 on SWE-Bench Verified, 43.1 on Terminal-Bench 2.1, matching or beating models 3x its parameter count.
Test 1: Benchmark Script (Latency)
A standalone Python script hit Ollama directly with static ChromaDB payload samples constructed from real zerikai_memory workspace entities. 3 queries, 3 samples each per model, raw latency measured at the HTTP layer. This test measures inference speed on the hardware, not synthesis quality against a live codebase.
| Model | Mean | StdDev | Min | Max |
|---|---|---|---|---|
| mistral:7b | 6.14s | 3.58s | 2.92s | 14.57s |
| ornith:9b | 13.39s | 5.76s | 8.77s | 25.67s |
ornith's cold start on the first query hit 25.67s. It spilled into shared memory before Ollama pinned the model. Warmed samples settled at 9-17s. mistral fits clean in 8GB and runs at 3-7s warm.
The benchmark script is in the repo (in the blog/ directory). Run it against your own hardware before committing to a model swap.
Test 2: Live MCP Queries (Synthesis Quality)
Both models were tested live through the zerikai_memory universal-brain MCP layer against the real zerikai_memory workspace index. Same 5 queries, models switched manually between runs, Ollama restarted between each switch. This test measures what a developer actually experiences in Cursor or VS Code -- not synthetic payloads, real ChromaDB retrieval.
Same query, same ChromaDB context, same system prompt. Here is what came back.
Query 1: How does query routing decide between Ollama and DeepSeek?
ornith:9b:
Based on the retrieved codebase memory, here's how query routing works:
Routing Logic:
1. Explicit Override: The `_should_use_cloud` function first checks for an explicit
`use_cloud` parameter that can force cloud mode (DeepSeek) regardless of other
conditions #main.py:986-1005.
2. Keyword Detection: If the user query contains keywords like "architecture",
"explain", "how does", or "overview", it routes to DeepSeek's cloud mode
#config.py:43-44.
3. Word Count Threshold: Queries exceeding 40 words automatically route to
cloud/DeepSeek via `_should_use_cloud`'s word count check #main.py:992-994,
with this threshold defined in config as `CLOUD_ESCALATION_WORD_COUNT = 40`
#config.py:37.
4. Default Mode: Falls back to whatever is set as `DEFAULT_MEMORY_MODE`, which
defaults to "hybrid" unless explicitly changed via the `MEMORY_MODE` environment
variable #config.py:31-32, with this function returning based on whether that
mode equals "cloud" #main.py:996-998.
Important Note: The retrieved context does not contain explicit information about
Ollama's role in the routing decision or how local model queries are constructed
when cloud mode is NOT selected.
mistral:7b:
The query routing in zerikai_memory decides between using Ollama or DeepSeek based
on a 4-step priority chain. The priorities are as follows:
1. Explicit use_cloud override.
2. CLOUD_ESCALATION_KEYWORDS keyword match.
3. CLOUD_ESCALATION_WORD_COUNT length threshold.
4. DEFAULT_MEMORY_MODE fallback (recommended).
This decision-making process aims to balance privacy, speed, and API costs while
providing maximum context nuance for the user. For more detailed information on
modes, configurations, and usage, you can refer to the README.md file in the
project repository.
Sources: main.py:986, README.md:320, main.py:1495, main.py:1311, README.md:105
Query 2: Where is the .brain directory path defined and how is it resolved across operating systems?
ornith:9b:
The `.brain` directory path is defined and resolved through two key locations:
Definition: In `config.py`, line 1-2, there's a constant that defines the base
name for the brain directory [config.py:1]:
DB_PATH = '.brain/'
BRAIN_DIR = Path(DB_PATH)
Resolution: In `main.py`, lines 120-145, within the `init_workspace` function,
this path is resolved by joining it with the workspace's absolute path [main.py:130]:
path = Path(workspace_path).resolve()
db_path = path / DB_PATH
db_path.mkdir(parents=True, exist_ok=True)
The resolution process:
1. Takes the provided `workspace_path` parameter
2. Resolves it to an absolute path using `.resolve()`
3. Joins it with the `.brain/` directory constant
4. Creates the full directory structure if it doesn't exist
mistral:7b:
The `DB_PATH` constant, which is used for defining the storage root for all
workspace data, is defined in `config.py`:
DB_PATH = Path(__file__).parent / ".brain"
It resolves to `zerikai_memory/.brain/`. This path is platform-agnostic as the
`Path` object in Python handles different operating systems' file paths.
Sources: config.py:67, main.py:1034, main.py:818, README.md:375, main.py:1796
Query 3: How does background brief synthesis avoid MCP timeouts?
ornith:9b:
Background brief synthesis avoids MCP timeouts by running as a fire-and-forget
task launched with `asyncio.create_task`, meaning `_background_brief_synthesis()`
is called without awaiting or returning its result. This decouples the slow
LLM-based brief generation from the MCP request lifecycle, so the IDE agent
receives its response immediately rather than waiting for synthesis to complete.
Sources:
* main.py:784 - 1.96 (rerank)
* main.py:1827 - 0.99 (rerank)
* main.py:393 - 0.98 (rerank)
* main.py:2094 - 0.94 (rerank)
mistral:7b:
Background brief synthesis avoids MCP timeouts by using asynchronous task creation
with `asyncio.create_task` in the `_background_brief_synthesis` function. This
approach allows the function to run independently without blocking the main event
loop, ensuring timely responses and avoiding MCP timeouts. The synthesized brief is
saved as a markdown file in `.brain/contexts/<workspace_id>.md`.
Sources: main.py:784, main.py:1827, main.py:393, main.py:2094
Query 4: How does save_to_memory work with tree-sitter and what file types does it index?
ornith:9b:
`save_to_memory` routes by file extension: files with `.py`, `.js`, `.ts`, `.css`,
`.html`, or `.md` are processed via tree-sitter entity extraction (the
`extract_entities` flow in `code_indexer.py`). Other extensions fall through to
DeepSeek/Ollama LLM summarization.
For the tree-sitter path, tree-sitter parses the source and extracts one
`CodeEntity` per function/method/class -- each holding signature, docstring/JSDoc,
and metadata. These entities are upserted into ChromaDB with deterministic md5 IDs
(so re-scans overwrite duplicates), and token usage is logged to the SQLite tracker
(`zerikai.db`).
Sources: main.py:1072, README.md:279, code_indexer.py:169, code_indexer.py:204,
code_indexer.py:30
mistral:7b:
The `save_to_memory` function works by saving content to persistent vector memory
in ChromaDB. It uses tree-sitter for code parsing for supported file types:
.py, .js, .ts, .css, .html, and .md. Other formats fall back on DeepSeek/Ollama
LLM summarization. It uses deterministic md5 IDs so re-scans overwrite duplicates
and logs token usage to `zerikai.db`.
Sources: main.py:1072, README.md:279, code_indexer.py:169, code_indexer.py:204,
code_indexer.py:30
Query 5: How does asyncio semaphore control Ollama concurrency in local mode?
Neither model had enough indexed context to answer this fully. What each model did with that gap is the most important data point in this test.
ornith:9b:
Based on the available codebase memory context, I cannot determine how asyncio
semaphore specifically controls Ollama concurrency in local mode. The retrieved
entities show:
1. `_should_use_cloud` - Determines cloud usage #main.py:986
2. `MEMORY_MODE` configuration with cloud/hybrid modes #config.py:30-45
3. `_background_scan` - Uses Semaphore(4) for tree-sitter and Semaphore(2) for
LLM summarization during scanning #main.py:1827
None of these entities contain the specific concurrency logic for `_query_ollama`
at runtime. If this routing exists in the codebase, it's not present in the
retrieved memory entries.
mistral:7b:
The project uses asyncio semaphore for controlling Ollama concurrency in local mode.
The query can be answered by explaining that asyncio semaphore is used to manage
the number of concurrent processes for tree-sitter parsing and LLM summarization
during the scan loop, with a Semaphore(4) for tree-sitter parsing and Semaphore(2)
for LLM summarization in local mode.
Sources: config.py:62, config.py:53, main.py:1827, main.py:1555
mistral answered with confidence. The Semaphore(4) and Semaphore(2) values it cited are real -- but they belong to the background scan pipeline, not query-time Ollama concurrency. It applied the wrong context to a different question and presented the answer as fact. ornith said it didn't have the information and stopped. For an IDE assistant, a confident wrong answer costs more than an honest gap.
The Prerequisite: Docstring Quality
The order of operations matters and it is not obvious from the README.
After editing your .memignore file, run embedding-docstring on your codebase first. Then run scan_workspace. If you scan first with sparse or missing docstrings, ChromaDB indexes thin vectors. Re-scanning won't fix it unless you re-enrich first and scan again. The memory is only as good as what tree-sitter extracted, and tree-sitter only extracts what is there.
zerikai_memory ships with the embedding-docstring skill for this reason. It audits and rewrites docstrings, comment blocks, and inline documentation across an entire workspace for vector embedding quality, covering Python, JavaScript, TypeScript, and HTML. It writes missing documentation from scratch and respects a .memignore file at the workspace root. The correct workflow is:
.memignore → embedding-docstring → scan_workspace → query
Skip the first step and both models underperform. You will spend time blaming the model or the hardware when the real problem is what went into ChromaDB.
Current status: works well with pi.dev, VS Code support in progress due to large file size constraints in some editors. Update: as of 7/14/2026 VS Code now supports large files, so the skill is usable in both Cursor and VS Code.
Brief Generation: An Uncontrolled But Useful Data Point
As a secondary test, we compared briefs generated for the same workspace by DeepSeek (cloud, sparse docstrings) and ornith:9b (local, after embedding-docstring enrichment). This is not a controlled comparison -- the docstring density differed between runs, so the model is not the only variable.
What the comparison shows is that ornith:9b, given enriched ChromaDB context, produces dense, precise briefs: atomic overwrite semantics, naming convention breakdowns, explicit gap flags where documentation is missing. DeepSeek against sparse context produced thinner output with some inferred detail not present in the code.
The takeaway is not that ornith beats DeepSeek for brief generation. It is that embedding-docstring enrichment is visible and measurable in the output. When the context is rich, ornith produces briefs good enough to feed meaningful synthesis queries. When it is not, neither model can compensate.
Full Local Mode and Brief Synthesis: The Semaphore Fix
Before this release, full local mode had a GPU saturation problem. _synthesize_deep_brief at main.py:538 fired asyncio.gather across all 9 brief sections simultaneously with no concurrency gate. In local mode that meant 9 concurrent Ollama calls hitting the GPU at once -- guaranteed to saturate an 8GB card.
The fix shipped alongside this test. A global ollama_semaphore initialized in main.py after client setup gates _build_section calls through a _build_section_safe wrapper when use_cloud=False. Cloud and hybrid modes bypass the semaphore entirely -- DeepSeek handles its own rate limiting on the API side.
ollama_semaphore = asyncio.Semaphore(OLLAMA_MAX_CONCURRENCY)
async def _build_section_safe(name):
if not use_cloud:
async with ollama_semaphore:
return await _build_section(name, workspace_id, workspace_path)
return await _build_section(name, workspace_id, workspace_path)
OLLAMA_MAX_CONCURRENCY is configurable via .env, defaulting to 1 for 8GB hardware. Users on cards with more VRAM headroom can raise it. The ornith:9b brief in this post was generated with this fix in place -- full local mode brief synthesis is production-ready as of this release.
Hardware and Cost
If token pricing is the reason you are reading this, here is what a GPU upgrade costs against what you are spending on API calls:
- RTX 3060 12GB (recommended minimum for ornith:9b): $330-$470 new. ASUS Dual and Gigabyte WINDFORCE variants available at Newegg around $340-$440.
- RTX 4060 Ti 16GB: $400-$500. The extra VRAM lets you load larger 13B-14B quantized models without spilling to system RAM.
- RTX 4070 12GB: around $600. Faster Tensor cores, quicker token generation.
AMD cards (RX 6700 XT 12GB, refurbished from $380) offer equivalent VRAM but require ROCm configuration. Ollama's CUDA path is plug-and-play on NVIDIA. AMD works but adds setup overhead.
On 8GB (RTX 3050 class), ornith:9b runs but cold starts are painful and VRAM headroom is tight. The RTX 3060 12GB is the practical sweet spot for local zerikai_memory use.
Recommendation
ornith:9b is the new default local model recommendation, replacing mistral:7b.
On 8GB dedicated VRAM: ornith fits but runs tight. Cold start hits 25s when Ollama hasn't pinned the model. Warm synthesis at 9-17s is acceptable for a local-only workflow where you are not switching models or running concurrent GPU workloads. Set OLLAMA_MAX_CONCURRENCY=1 in .env.
On 10-12GB dedicated VRAM (RTX 3060 12GB or better): the model stays pinned, cold starts drop significantly, and citation precision is consistently better than mistral.
Under 8GB dedicated VRAM, or if synthesis latency matters more than citation precision, use mistral:7b. Set OLLAMA_MODEL=mistral:7b in .env. It handles synthesis correctly when context is dense. When context is thin, it will fill gaps with confident but wrong answers.
The query test was clean and controlled. The model difference is real and attributable to ornith's training on agentic coding tasks, not hardware or docstring quality. Use the benchmark script in the repo to validate on your own machine before switching.
📖 Original Publication: This engineering post-mortem was originally published on the Zerikai Tech Blog. Read the clean, formatted web version at https://zerikai.com.
Top comments (15)
Query 5 is the most valuable data point in the whole post and I'm glad you led the conclusion with it. "A confident wrong answer costs more than an honest gap" is exactly right for an IDE assistant, because the wrong
Semaphorecitation looks authoritative enough that a dev pastes it and moves on. The scary part is that both answers cite real code, so a shallow "does it have sources?" check passes on both. Makes me wonder if that honest-gap habit is stable for ornith across harder queries, or if it also starts making things up once the retrieved context is close but wrong rather than clearly missing. That's the line I'd want to know before trusting it in an editor.Good catch, but the risk is slightly different from what you're describing. The retrieval is nearest-neighbor search against deterministically indexed entities, tree-sitter AST extractions of real code, no summarization, no interpretation. ChromaDB can't return fabricated context because there is no fabricated context to return.
The real failure mode is the LLM synthesis becoming the source of truth instead of the retrieved findings. Query 5 is a clean example: mistral received real semaphore values from the scan pipeline, applied them to a different question, and presented the synthesis as fact. The retrieved entities were accurate; the synthesis betrayed them. zerikai_memory instructs the model to respond factually against what was retrieved, but a less capable model can't hold that boundary when the context is adjacent but not exact. ornith held it.
The last line of defense is the IDE agent itself. When it takes the synthesized answer and goes to verify the cited file and line, the mismatch surfaces immediately. The architecture assumes that verification step happens, which means the synthesis layer needs to be honest about gaps, not fill them. That's the behavior ornith demonstrated and mistral didn't.
thanks for sharing!
"A confident wrong answer costs more than an honest gap" is the whole thing, and picking ornith:9b over mistral:7b for that reason is the right call. The nuance I'd add: right now honesty is a property you selected by choosing a more disciplined model, which means it is only as reliable as that model's mood on any given query. mistral fabricated on Query 5 by applying real semaphore values to the wrong question, and the failure is diagnostic: the retrieval was correct, the synthesis just was not entailed by it. That is a gap the architecture can close independently of which model you run.
Since your tree-sitter chunks already carry deterministic #file:line identity, you can check the generated answer against its own cited spans before returning it: does every claim actually trace to a retrieved chunk, and does the citation resolve to code that supports it. When it does not, degrade to "insufficient context" instead of trusting the model to volunteer that. Then honesty stops being a model you hope stays humble and becomes a gate every answer passes, which also frees you to run the faster model where it is safe. Retrieval quality and synthesis honesty being independent defenses is exactly right, and the second one is enforceable, not just selectable.
the thread's zeroed in on synthesis honesty, which is right, but there's a second reason retrieval precision matters that's specific to the 8gb box: it's your latency lever too. loose retrieval means a fatter prompt, and on this hardware a fatter prompt is exactly what spills into shared memory (the ornith cold start at 25s is the tell). a reranker that returns 3 tight chunks instead of 8 loose ones cuts both the wrong-context risk mistral tripped on in query 5 and the tokens you pay to synthesize, and it's the one layer that's already model-agnostic so it compounds whichever model you run. did you ever measure retrieval precision separately from the latency runs, or only end to end?
That’s a sharp read of the data, and you're spot on that the 25.67s cold start was the model spilling into shared memory. However, tuning chunk counts was an optimization bottleneck we simply didn't need to break through for a few reasons.
First, the shared memory spill was strictly a cold-start pinning issue. Once the model was warm, synthesis settled comfortably into a 9–17 second window. Paired with gating execution via OLLAMA_MAX_CONCURRENCY=1, the VRAM overhead remained perfectly stable without requiring us to truncate our context window.
Second, while Mistral completely choked on the loose background context in Query 5 and hallucinated, Ornith:9b's agentic training natively handled the noise. It recognized the gap, stated what was missing, and stopped.
Because Ornith already achieved synthesis quality comparable to DeepSeek without confidently fabricating answers, over-engineering our chunking and retrieval precision layer wasn't necessary to get a highly reliable, local memory out the door.
It's also worth noting that we treated the RTX 3050 8GB strictly as a hard minimum baseline for our testing.
fair, and OLLAMA_MAX_CONCURRENCY=1 gating is the right call for keeping the 8gb box stable. where i'd still push back: a model recognizing a gap only works when retrieval hands it something to notice the mismatch against. the dangerous case isn't the loose noise ornith can flag, it's a single nearest-neighbor chunk that's coherent and on-topic but quietly answers a slightly different question. with nothing to contrast it against, even an honest model has no gap to see. so precision and model honesty aren't substitutes, they cover different halves. the 8gb box just tempts you to lean all-in on the model because context is scarce, which is exactly when a bad retrieval has nowhere to hide.
That is a completely valid concern, but it actually highlights why the pipeline is decoupled. The architecture separates the initial retrieval pool from the final LLM prompt budget to address exactly what you are describing.
The configuration (
.env) exposes this directly as a tunable variable:It is fully exposed as an environment variable, so anyone can dial it up or down.
Because we pull an initial pool, let the lexical reranker sort it, and then enforce a hard slice (relevant[:k]) before hitting the LLM, we don't have to choose between context isolation and VRAM safety. We protect the 8GB envelope at the prompt layer while letting users tune the retrieval net to their exact comfort level. Beyond exposing that knob for people to tweak, I'm happy with how the current baseline balances the two.
It is all in there; pretty happy with how it is performing at this point. This has been an evolutionary process; I have learned and grown the tool based on those findings. Always with the core goals in mind and practical application.
Thanks
fair, and the decoupling is the right shape for what it targets. the thing is a wider FETCH_CAP plus reranking fixes recall, the case where the right file was keyword-relevant but semantically distant so it'd have been missed. my worry was the other half: when the rank-1 chunk after reranking is already coherent, on-topic, and quietly answering a slightly different question. the hard slice keeps the winner, so a bigger pool just stacks more losers underneath it. reranking reorders, it doesn't notice the top result is confidently wrong. so the knob buys you recall, not precision on the winner. not saying tune it differently, just that those are two different failures.
To track what actually happened across this thread: you opened with model honesty being insufficient against a bad rank-1 result, and that precision and model honesty cover different halves. I responded with how the retrieval architecture handles both. You then said reranking reorders but can't catch a confidently wrong winner. That's a different claim than the first one, and it skips over what the rerank weight is actually doing.
LEXICAL_RERANK_WEIGHT=0.05 is calibrated to stay below the L2 semantic spread so a keyword hit can break a tie but cannot override a genuinely closer semantic result. The weight is documented in .env for exactly that reason.
Beyond that, the chunking granularity is the actual precision defense. Tree-sitter emits one entity per function or class, so a chunk's semantic scope is already narrow. The "coherent but quietly wrong" case you're describing has much less surface area when each vector covers one function, not a file.
You went from model honesty being the gap, to reranking can't catch a bad winner. Both are covered. Unfortunately, I can no longer engage in theoretical abstractions only vaguely related to this project.
The Ollama-local path is the right call for codebases that can't leave the machine, and the post-mortem framing is exactly what this space needs more of. The thing that tends to bite in production with local embedding models is embedding model drift: you ship with mistral:7b embeddings, a team member updates to a newer model six months later, and suddenly every existing vector in ChromaDB is from a different embedding space so retrieval silently degrades without an obvious error. Versioning the embedding model name alongside the chunk data in the store and refusing to mix-read across versions is unglamorous but saves a real class of bugs. Curious whether you ran into model-drift issues across the team during development or whether everyone stayed on the same Ollama version.
Good question, but zerikai_memory sidesteps the embedding drift problem by design. The indexing layer is deterministic -- tree-sitter parses the codebase and extracts discrete code entities (functions, classes, methods) via AST, not probabilistic LLM summarization. ChromaDB embeds those entities, and that embedding model doesn't change when you swap synthesis models. Swapping mistral:7b for ornith:9b doesn't touch a single vector in the store.
The synthesis LLM only enters at query time -- it reads the retrieved chunks and writes an answer. That's the only layer where model choice affects output, and it affects answer quality, not retrieval fidelity. Which is exactly what we measured in the post. The class of bugs you're describing is real in systems where the LLM drives indexing, but that's not the architecture here.
Nice post-mortem. The part that stood out to me was separating retrieval quality from synthesis quality.
One practical check I would add: log the retrieved entity set and the final cited answer independently. Then when you swap mistral:7b for ornith:9b, you can tell whether a bad answer came from Chroma/reranking or from the model going beyond the retrieved context.
Local mode is not just a privacy story. It also makes evals much more repeatable if you pin the model version, prompt, and index snapshot.
Thanks, and the logging separation is already in place. The server log tracks retrieval and synthesis as independent events. Here's a real example from the query 5 run in the post:
Retrieval completed at 00:17:02, synthesis returned at 00:17:26, 24 seconds of ornith:9b on an RTX 3050. The rerank line shows what ChromaDB surfaced and what scored top. The generate call shows when the model responded. When an answer goes wrong you already have both events timestamped and separated to know where to look. Your pinning point is solid, OLLAMA_MODEL=ornith:9b fixed in .env plus a versioned .brain/ snapshot gives you a reproducible eval loop that cloud APIs can't match the same way.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.