Why my 0.8 cosine threshold failed in an offline AI memory

Manoj K — Fri, 06 Nov 2026 04:37:37 +0000

I'm a self-taught robotics engineering student in Bengaluru. I built Neo, a fully offline AI assistant that remembers facts about you across sessions. It runs llama3.1 through Ollama, stores vectors in LanceDB and facts in SQLite, and does web search through a self-hosted SearXNG. No cloud, no API keys.

The problem

When you tell the assistant "I stopped liking chess", it has to decide what that means. Is it a new fact, an update to an old one, or a retraction?

My first design used one rule: if cosine similarity with an old fact is above 0.8, treat it as a contradiction.

What I measured

I tested 10 real sentence pairs with my actual embedding model through Ollama. The 0.8 rule missed paraphrases and updates. Unrelated pairs scored only 0.29 to 0.41, so the useful range sat much lower than I had assumed.

I also found an earlier mistake. My retrieval eval was measuring L2 distance, not cosine. After fixing the metric, Recall@k came out at 0.95 and MRR at 0.925.

The fix

I stopped asking cosine to make the decision. Now it only finds candidates, using a low threshold of about 0.5. Then the flow is:

new fact
   ↓
find candidates (cosine ≥ 0.5, active facts only)
   ↓
no candidates → ADD
candidates found → LLM returns {action, target_id}
   ↓
code validates the answer
   ↓
write to SQLite + LanceDB

The LLM picks one of ADD, UPDATE, DELETE or NONE, plus a target id. Plain code then checks the answer:

An unknown or malformed target id falls back to ADD.
An UPDATE whose target text is identical to the new fact becomes NONE.

The model proposes and the code decides what is allowed. This mattered because llama3.1 sometimes returned UPDATE for a fact that already existed, or picked the wrong target. Prompt tweaks helped, but the guards in code are what made it reliable.

What I'd tell others

Test your threshold on real sentence pairs before trusting it.
Check which metric your eval actually uses (L2 vs cosine).
Keep the LLM's output narrow and validate it in code.
Write tests with fake LLM calls so you can check the gate logic without Ollama running.

Honest notes

v1.9 was mostly written with Claude and turned out unstable, so I rebuilt v1.9.5 by hand from v1.8.5 to understand every part. The README credits the AI help. Known limits are listed in KNOWN_ISSUES.md. For example, comparative preferences aren't extracted, and follow-up web queries lose context. It also needs hardware that can run an 8B model locally.

Repo: https://github.com/Neo-X7/neo-AI

How would you handle contradicting facts in a memory system? I'd like to hear other approaches and how can I improve things in the future.

How I fixed silent Ollama failures in my local AI Assistant

Manoj K — Wed, 10 Jun 2026 12:32:01 +0000

How I fixed silent Ollama failures in my local AI assistant

Neo-AI is an offline assistant with episodic memory, running entirely on-device using Ollama, SQLite, and LanceDB — no cloud, no data leaks.

The Failure

While testing before a release, the CLI showed a generic "An error occurred." I opened the log file and found the real error:
ERROR: Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible.
Ollama was installed but not running. Neo had no way to start it automatically.

The naive fix

def start_ollama():
    try:
        import httpx
        httpx.get("http://localhost:11434")
        return
    except Exception:
        console.print(Text("Starting OLlama server...",style="dim"))
        subprocess.Popen(["ollama","serve"],stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL)
        time.sleep(2)
        console.print(Text("Ollama server started.", style="dim"))

This uses a hardcoded health check — a GET request to localhost:11434. If Ollama responds, skip. If not, spawn it as a background process using subprocess.Popen, which launches Ollama silently without blocking Neo. stdout and stderr are discarded via DEVNULL so Ollama's internal logs don't pollute Neo's terminal.
The flaw: time.sleep(2) is a guess. On a cold WSL start, Ollama can take longer. Neo proceeds before Ollama is actually ready.

The Better Fix

def start_ollama():
    def is_running():
        try:
            httpx.get("http://localhost:11434", timeout=2)
            return True
        except Exception:
            return False
    if is_running():
        return
    console.print(Text("Starting Ollama server...", style="dim"))
    subprocess.Popen(["ollama", "serve"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
    for _ in range(10):  # wait up to 10 seconds
        time.sleep(1)
        if is_running():
            console.print(Text("Ollama server started.", style="dim"))
            return
    raise RuntimeError("Ollama failed to start after 10 seconds.")

Instead of sleeping blindly, the retry loop polls every second for up to 10 seconds. Neo only proceeds once Ollama confirms it's alive. If it never starts, a RuntimeError is raised and logged.

Takeaway

Local LLM reliability is an engineering problem, not a hardware problem. Don't sleep — poll.
Neo-AI on GitHub