In-process embeddings for a desktop AI app, or how a missing daemon silently broke my RAG

I'm building a desktop AI app for non-technical users — the kind of person who double-clicks an icon and expects search to work, and who will never open a terminal in their life. Under the hood it does local RAG: ingest documents, embed them, retrieve on each question. For a while, the embedding step went through Ollama.

The bug that didn't look like a bug

On my machine everything was fine, because on my machine Ollama is always running. Then I ran the flow a real user would hit — fresh login, Ollama not started — and watched the app quietly fall apart:

• /ask returned an answer with no retrieved context.
• /search returned nothing at all.
• Ingestion accepted documents and indexed none of them.

No stack trace. No red banner. The app looked healthy and did nothing useful.

To be clear, this wasn't Ollama being sneaky. Ollama fails loudly — you get connection refused the instant you hit localhost:11434 with nothing behind it. The failure was mine: my error handling caught that exception and turned it into an empty result, and every caller downstream treated "empty" as "no matches." A user asking a perfectly good question got a confident, sourceless answer.

Why it happens: the daemon is a dependency you can't see

The swallowed exception is the easy part to fix. The architectural lesson is the one worth keeping: by routing embeddings through Ollama, I had made a separate background process's liveness a hard requirement for my app to function correctly. That's a fine trade when you are the operator. It's a terrible trade when your user doesn't know a daemon exists, can't tell whether it's running, and certainly won't restart it after a reboot.

For a desktop app aimed at non-technical people, "keep this background service alive or your search silently breaks" is not a requirement you get to impose.

The fix

Load the same model in-process. bge-m3, 1024-dim, normalized — identical to what I was pulling from Ollama, just running inside my own Python backend via sentence-transformers.

Before:

import requests

def embed(texts: list[str]) -> list[list[float]]:
    resp = requests.post(
        "http://localhost:11434/api/embed",
        json={"model": "bge-m3", "input": texts},
    )
    resp.raise_for_status()
    return resp.json()["embeddings"]

After:

from sentence_transformers import SentenceTransformer

_model: SentenceTransformer | None = None

def _get_model() -> SentenceTransformer:
    global _model
    if _model is None:                       # lazy singleton
        _model = SentenceTransformer("BAAI/bge-m3")
    return _model

def embed(texts: list[str]) -> list[list[float]]:
    model = _get_model()
    return model.encode(texts, normalize_embeddings=True).tolist()

encode is blocking and CPU/GPU-heavy, so in a FastAPI backend you don't call it on the event loop. Push it to a worker thread:

import anyio

async def embed_async(texts: list[str]) -> list[list[float]]:
    return await anyio.to_thread.run_sync(embed, texts)

And make health deterministic instead of guessing. A tiny status enum beats a boolean, because "loading a 2 GB model" is a real state that lasts several seconds:

from enum import Enum

class ModelStatus(str, Enum):
    NOT_LOADED = "NOT_LOADED"
    LOADING = "LOADING"
    READY = "READY"
    FAILED = "FAILED"

Now /health can tell the UI exactly where it stands, and the UI can disable search until the answer is READY instead of returning empty nonsense.

Why not a hybrid fallback

The obvious next thought is: keep both. Try Ollama, fall back to in-process if the daemon is down. For embeddings, that is a bug — not a robustness feature.

Ollama serves a quantized GGUF. sentence-transformers runs fp32. Same model name, different numerics, and therefore a different vector space. The two backends do not produce interchangeable vectors.

That matters because your index is written once and queried many times. If some vectors were written by the GGUF path and you query with the fp32 path — or you re-ingest under a different backend than you started with — the cosine distances between them are quietly meaningless. Nothing throws. Retrieval just gets subtly, unpredictably worse, and you'll waste days blaming your chunking or your reranker.

So the rule is one backend per index. If you want to switch backends, you re-embed the whole corpus once, deliberately. You never mix them and hope.

Trade-offs

I'm not going to pretend this is free.

• PyTorch is heavy. sentence-transformers pulls in PyTorch — gigabytes on disk. You're trading a daemon dependency for a large Python dependency. This only makes sense if your app already ships a Python backend or sidecar, which mine does.
• The first run downloads ~2.2 GB. The bge-m3 weights come from Hugging Face on first use. For real users you bundle them with the app or self-host them, because anonymous HF downloads get throttled once you're past a few hundred megabytes, and "the app hangs on first launch" is a terrible first impression.
• fp32 eats RAM. In-process fp32 uses more memory than Ollama's quantized GGUF, and it competes with the LLM for that memory on the same machine. On an 8 GB laptop, that competition is real.

When Ollama is still the right call

If your users are developers running their own stack — people who already have Ollama up, who want bring-your-own-model, who treat the daemon as infrastructure they control — then routing through it is the right design. The daemon stops being a hidden liability and becomes a feature. My users aren't developers, so for me it wasn't.

The code and a longer write-up are here: https://github.com/JackYU96/embed-without-daemon

The app itself isn't launched yet — this is a pattern I settled on while building it, not a shipped product. If you've solved the desktop-embeddings problem a different way, I'd genuinely like to hear it.

Comments

[Luis]

Great debugging story. This is a perfect example of how distributed dependencies can quietly become the weakest link in an AI application. Moving embeddings in-process simplifies deployment, reduces operational complexity, and removes an entire class of failure modes.

It's also a good reminder that reliability often comes from reducing moving parts, not adding more infrastructure. For desktop AI apps especially, minimizing external services can improve startup time, offline support, and the overall user experience. Great engineering lesson!