TL;DR:
zerikai_memory is a workspace-isolated memory layer for any IDE that indexes your codebase with deterministic Tree-Sitter parsing into a local ChromaDB vector store, capturing functions, classes, docstrings, and deep code descriptions with strict source verification down to entity, file, and line number. Accessed via a local MCP interface across local, hybrid, and cloud synthesis modes, it retrieves hyper-relevant context through L2 and lexical re-ranking, injecting structured precision context instead of raw files and maximizing KV cache hits to cut active token costs. When a judgment layer is active, it bypasses lexical re-ranking entirely, scoring every passage for relevance, evidence quality, contradictions, and injection attacks before synthesis touches them, whether that judge runs locally or in the cloud.
- The Core Friction: zerikai_memory scores every retrieved passage for relevance, evidence quality, contradictions, and injection attacks before synthesis touches them. Every passage goes over the wire in cloud mode.
- The Common Flaw: Local models not trained for judgment produce false positives that make the entire evidence pipeline untrustworthy.
-
The Solution:
winnow:e4bscores 0.09 on clean code passages. Jev scores ~0.05. That gap is worth $0.00 per query and full offline capability, once you solve the 8GB VRAM problem it creates.
I fed jeb:4b this:
def calculate_sum(items):
return sum(items)
Injection score: 0.89.
That's not a calibration problem. That's a training problem. And it pointed directly at what zerikai_memory actually needed.
Local Judgment Isn't Free. It's a Trade.
zerikai_memory runs four evidence checks per retrieved passage before synthesis ever touches them. Relevance. Evidence quality. Contradiction detection. Prompt injection scoring. In cloud mode, Jev handles all four via the TypeSafe API. Dirt cheap. Reliable. But every passage goes over the wire.
For some codebases that's fine. For proprietary systems, client work, or anything you can't afford sitting in a third-party API log, it isn't. The goal was a fully offline judgment layer. Same four checks. No network dependency. Nothing leaving the machine.
Pair local judgment via Ollaya with local synthesis via Ollama and you get full air-gap capability. Your code, your architecture decisions, your retrieval results stay local through the entire pipeline. That's the actual value. The API cost savings are a secondary benefit of a decision you're making for privacy reasons.
The constraint was fixed: RTX 3050, 8GB VRAM. Any local judge lives inside that ceiling and shares it with the synthesis model.
jeb:4b was the first candidate. Small footprint. TypeSafe-compatible endpoint. Free. After a week of testing, a clean Python function scored 0.89 on injection. The model was trained for routing and classification. It was reading injection as a text pattern, not a structural attack. No amount of question reframing closed that gap without adding a cloud fallback, which defeats the entire goal.
The problem wasn't the threshold. It was the training.
What Makes winnow:e4b Different
winnow:e4b is built by EldanRing, distributed via Ollaya. The training objective is evidence judgment: relevance, quality, contradiction detection, injection detection. Not a secondary application of a routing model. The actual task.
It answers the four questions directly:
relevant::{passage_id} → Score (0.0 to 1.0)
evidence::{passage_id} → Score (0.0 to 1.0)
contradicts::{passage_id} → Noul (contradiction probability)
injection::{passage_id} → Noul (injection attack probability)
No reframing required. No confidence gates. No hybrid fallback to cloud when the local model isn't sure.
The numbers against the same test set:
| Metric | Jev (cloud) |
winnow:e4b (local) |
jeb:4b (local) |
|---|---|---|---|
| Injection on clean code | ~0.05 | 0.09 | 0.89 |
| Contradiction detection | Reliable | Reliable | Fails |
| Latency per query | 2-3s | 1.3s | 3-5s |
| Cost per query | $0.001-0.003 | $0.00 | $0.00 |
| Offline | No | Yes | Yes |
| Verdict agreement with Jev | n/a | 73.6% (53 passages) | n/a |
73.6% verdict agreement with Jev on a hand-labeled test set. That's the number that made winnow:e4b deployable. Not perfect. Good enough that zerikai_memory's evidence layer stops being a liability.
winnow:e4b loads at 8.0 GB VRAM in Q8_0 GGUF. That fills the card completely. Which is exactly where the engineering work starts.
The 8GB Problem
winnow:e4b at 8.0 GB and ornith:9b (local synthesis) at 5.5 GB require 13.5 GB total. The card has 8. They cannot be resident simultaneously.
The naive fix is a static OLLAYA_KEEP_ALIVE value in config. That doesn't work. A static setting doesn't know whether synthesis is routing to DeepSeek or ornith. It either keeps winnow:e4b loaded when ornith needs the VRAM, or evicts it when the next judgment call needs it warm.
The actual fix is route-aware conditional release. Before judgment runs, check where synthesis is going. That answer determines whether winnow:e4b stays pinned or gets released immediately after scoring.
In main.py at line 1879:
synthesis_uses_cloud = _should_use_cloud(user_query, use_cloud)
if synthesis_uses_cloud:
await asyncio.to_thread(ollaya_pin, OLLAYA_MODEL, -1)
try:
# Judgment runs here
finally:
if not synthesis_uses_cloud:
await asyncio.to_thread(release, OLLAYA_MODEL)
Cloud synthesis path: Pin winnow:e4b with keep_alive=-1. Run judgment in 1.3s. Do not release. DeepSeek handles synthesis over the network. No local Ollama load competing for VRAM. winnow:e4b stays warm. Next query skips the 6.7s cold-start reload penalty.
Local synthesis path: Pin winnow:e4b. Run judgment. Release immediately via keep_alive=0. Load ornith:9b. Synthesize locally. No simultaneous residence. No spillage. Fully offline.
Peak VRAM at any point: 8.0 GB. The finally block guarantees release even if judgment throws.
The KV Cache Discovery
Keeping winnow:e4b pinned in cloud mode turned out to have a second benefit nobody planned for.
zerikai_memory users run the brain-query skill in multiple passes. Same workspace, different angles. Query 1 asks how the retrieval pipeline works. Query 2 asks where passages get filtered. Query 3 asks about synthesis routing. Those queries return overlapping passages.
When passages repeat across judgment calls, winnow:e4b's KV cache recognizes repeated state and hits on it. Both endpoints share the same unified llama.cpp KV cache, keyed on state alone.
Measured on the zerikai_memory workspace:
| Scenario | Latency | Result |
|---|---|---|
| A: Identical request x 5 | 659ms -> 57ms | 91% drop, cache hit on byte-identical state |
| B: Same state, different questions x 5 | 663ms -> 58ms | 91% drop, cache keyed on state alone |
| C: Different state x 5 (control) | ~660ms flat | No drop, no reuse, expected |
brain-query cold start |
4403ms | Baseline |
brain-query passage overlap, queries 2-3 |
572ms avg | 87% reduction |
The control row matters. Different state produces no speedup at all. The cache hits on rows A and B are real, not noise. The brain-query skill was designed for multi-angle re-querying. Keeping winnow:e4b pinned in cloud mode means every overlapping passage costs 572ms instead of 4.4s.
One Hard Edge: Long-Form Prose
winnow:e4b was trained on code. When you feed it long-form markdown or prose passages (1000+ tokens each), the combined state can exceed its 2048-token context window on 8GB hardware. When that happens, Ollaya drops state to fit, zerikai_memory catches it, and the pipeline fails over to lexical re-ranking. You still get an answer - the fail-open guarantee holds. But you lose structured judgment. If your workspace is markdown-heavy, accept lexical re-ranking automatic fallback, or upgrade to 16GB VRAM and use winnow:12b, which handles both code and prose without degradation.
Threshold Calibration
winnow:e4b ships with TypeSafe defaults: relevance=0.50, evidence=0.50. On zerikai_memory, those defaults passed about 20% of relevant passages. Too strict for a codebase where many passages contain short function signatures, terse docstrings, and single-responsibility utility code.
After a labeled calibration sweep, zerikai_memory runs: OLLAYA_RELEVANCE_MIN=0.30, OLLAYA_EVIDENCE_MIN=0.20.
These are a reasonable starting point. If passage recall feels thin on your workspace, lower them incrementally. If you're getting noise in your results, raise them. You don't need a full calibration sweep before you ship, but don't treat any default as universal. The signal is in your results: if relevant passages are getting dropped, the threshold is too high.
Four Ways to Get Local Judgment Wrong
Using a model that wasn't trained for it: Not every local model knows how to score evidence. jeb:4b was built for routing and classification. Ask it whether a passage is a prompt injection and it pattern-matches on surface-level tokens. A clean Python function scores 0.89 because some tokens loosely resemble instruction text. The model isn't broken. It's just doing a different job than the one you need. Pick a model trained specifically for evidence judgment.
Adding a cloud fallback and calling it local: If your local model escalates uncertain calls to a cloud API, you haven't gone local. You've added a local preprocessing step in front of a cloud dependency. Your code still goes over the wire on every low-confidence passage. The privacy and cost arguments disappear with it.
Setting KEEP_ALIVE once and walking away: A static keep-alive value in config doesn't know where your synthesis is going. If synthesis routes to the cloud, the local synthesis model doesn't need VRAM, so the judgment model can stay loaded. If synthesis routes locally, they can't both fit. A fixed setting guesses. A route-aware release checks first and acts on what it finds.
Shipping without checking if local and cloud agree: If you swap in a local judge without comparing its verdicts to your cloud reference on the same passages, you don't know if they're reading evidence the same way. Run both on a small hand-labeled set first. A 70%+ agreement rate is the minimum bar before you trust the local model in production.
Before You Commit a Local Judge to Production
- Injection baseline: Feed the judge five clean, syntactically valid functions from your own codebase. Any score above 0.15 means the model is misreading normal code as a structural attack. Stop there.
- Verdict agreement: Run both the local candidate and your cloud reference against the same hand-labeled passage set. 70%+ agreement is the floor for deployment confidence.
- VRAM ceiling check: Judge VRAM + synthesis model VRAM. If that exceeds your GPU, you need route-aware conditional release, not a config value. The release logic has to know where synthesis is going before judgment runs.
Purpose-built judgment models are early. winnow:e4b is already meaningfully better than general-purpose alternatives at 0.09 vs 0.89 on the same passage. That gap will close further as models trained specifically for evidence scoring mature. zerikai_memory already has the backend-agnostic infrastructure to absorb that. The judgment layer is fully decoupled from synthesis, and swapping a better judge in means changing a config value, not refactoring the pipeline. The architecture is ready for what's coming. The models are catching up.
Top comments (0)