When fastembed silently rotted my worker: anchor your caches, don't trust /tmp

My worker just crashed on a research job, and the traceback pointed at a file that should have been there:

WARNING  fastembed.common.model_management:download_files_from_huggingface
         Local file sizes do not match the metadata.
run_research failed, NoSuchFile: [ONNXRuntimeError] : 3 : NO_SUCHFILE :
Load model from C:\Users\reach\AppData\Local\Temp\fastembed_cache\
models--qdrant--bge-small-en-v1.5-onnx-q\snapshots\<hash>\model_optimized.onnx
failed. File doesn't exist

The stack was ordinary — arq picking up a run_research job, calling into my answer_question, which tries to embed the query. What was weird is that this had worked yesterday. Same code, same model, same machine. And it wasn't a corrupt-download error either — the warning said file sizes didn't match metadata, and then one specific .onnx file was flat-out gone.

The tell: it's in Temp

Look at the path again: C:\Users\reach\AppData\Local\Temp\fastembed_cache\....

That's the Windows OS temp directory. Nothing in my app pointed fastembed there — I just never told it where to cache. Here's what my init looked like:

# src/pacos/knowledge/index/embeddings.py
from functools import lru_cache
from pacos.shared.config import settings

Vector = list[float]

@lru_cache(maxsize=1)
def _model():
    from fastembed import TextEmbedding
    return TextEmbedding(settings.embed_model)

No cache_dir argument. Fastembed's fallback is the OS temp dir. Windows periodically purges Temp (Storage Sense, disk-cleanup runs, sometimes just reboots on some configs). It doesn't purge atomically — you can lose a subset of files in a snapshot directory while others survive. Which is exactly what the warning was telling me: the manifest says the snapshot should contain N files of certain sizes; on disk, one is missing. Hence NO_SUCHFILE on the specific model_optimized.onnx.

This had been a ticking bomb since I first added local embeddings. It'll re-download once, work for a while, then die again the next time the OS decides to clean up. Intermittent, invisible in dev because you retry and it heals itself, catastrophic in a worker that's supposed to run unattended.

The pattern that was already in the codebase

I'd solved this exact class of problem before — for Qdrant. When I searched for how other persistent data got anchored:

# src/pacos/shared/config.py
# CWD-relative path once caused the API to init an empty NESTED knowledge_repo.
PROJECT_ROOT = Path(__file__).resolve().parents[3]

That comment is a scar from a prior bug: two processes (API + worker) with different working directories were creating two Qdrant folders and silently disagreeing about state. The fix was to define PROJECT_ROOT once and resolve every data path against it. qdrant_path and knowledge_repo_path already lived under that rule.

Fastembed's model cache is the same shape of thing: regenerable, but expensive to redownload (~90 MB), shared between API and worker, must be stable across restarts. It just wasn't wearing the same seatbelt.

The fix

Three small changes.

1. Declare the path in config, alongside the other data paths:

# src/pacos/shared/config.py
embed_model: str = "BAAI/bge-small-en-v1.5"
# Anchored to PROJECT_ROOT so API and worker share one cache,
# and it survives OS temp-dir purges.
fastembed_cache_path: str = "./fastembed_cache"

2. Actually pass it to fastembed:

# src/pacos/knowledge/index/embeddings.py
from pathlib import Path
from pacos.shared.config import PROJECT_ROOT, settings

@lru_cache(maxsize=1)
def _model():
    from fastembed import TextEmbedding
    cache_dir = (PROJECT_ROOT / settings.fastembed_cache_path).resolve()
    cache_dir.mkdir(parents=True, exist_ok=True)
    return TextEmbedding(settings.embed_model, cache_dir=str(cache_dir))

3. Gitignore the download (it's regenerable, not source):

# .gitignore
qdrant_data/
fastembed_cache/
knowledge_repo/

Then I nuked the corrupted Temp snapshot and smoke-tested a cold init:

Fetching 5 files: 100%|##########| 5/5 [00:18<00:00,  3.77s/it]
embed OK, dim: 384

Clean re-download into D:\...\personal_career_growth\fastembed_cache, embedding returns the expected 384-dim vector, and the worker picks it up on restart.

The lesson worth taking

Any library that downloads model weights on first use is going to cache them somewhere. If you don't tell it where, it picks. In fastembed's case, and in a surprising number of ML libraries, that pick is tempfile.gettempdir() or a subfolder thereof. On Linux that's usually /tmp, which is tmpfs on many distros and blown away on reboot. On Windows it's inside AppData\Local\Temp, which the OS actively cleans. On macOS it's /var/folders/..., cleaned by periodic.

These caches are not really temporary from your app's perspective. They're expensive, shared across processes, and expected to survive restarts. They belong next to your other persistent-but-regenerable state — your Qdrant folder, your SQLite dev DB, your knowledge_repo/ — anchored to a project-relative path and gitignored.

A rule I'll be applying more aggressively from here on:

If two processes read from the same cache, or if reconstructing that cache costs more than a few seconds, it does not belong under TEMP. Point it at PROJECT_ROOT and let the filesystem outlive the OS's cleanup heuristics.

The symptom looked like corruption. The cause was a directory whose contract I hadn't read. One cache_dir= argument and a config entry, and the class of bug is gone.

---

Drafted by Claude Sonnet from my own Claude Code session transcript, then reviewed and edited before publishing.