DEV Community: yogeshchavan2008

When fastembed silently rotted my worker: anchor your caches, don't trust /tmp

yogeshchavan2008 — Wed, 15 Jul 2026 19:21:44 +0000

My worker just crashed on a research job, and the traceback pointed at a file that should have been there:

WARNING  fastembed.common.model_management:download_files_from_huggingface
         Local file sizes do not match the metadata.
run_research failed, NoSuchFile: [ONNXRuntimeError] : 3 : NO_SUCHFILE :
Load model from C:\Users\reach\AppData\Local\Temp\fastembed_cache\
models--qdrant--bge-small-en-v1.5-onnx-q\snapshots\<hash>\model_optimized.onnx
failed. File doesn't exist

The stack was ordinary — arq picking up a run_research job, calling into my answer_question, which tries to embed the query. What was weird is that this had worked yesterday. Same code, same model, same machine. And it wasn't a corrupt-download error either — the warning said file sizes didn't match metadata, and then one specific .onnx file was flat-out gone.

The tell: it's in `Temp`

Look at the path again: C:\Users\reach\AppData\Local\Temp\fastembed_cache\....

That's the Windows OS temp directory. Nothing in my app pointed fastembed there — I just never told it where to cache. Here's what my init looked like:

# src/pacos/knowledge/index/embeddings.py
from functools import lru_cache
from pacos.shared.config import settings

Vector = list[float]

@lru_cache(maxsize=1)
def _model():
    from fastembed import TextEmbedding
    return TextEmbedding(settings.embed_model)

No cache_dir argument. Fastembed's fallback is the OS temp dir. Windows periodically purges Temp (Storage Sense, disk-cleanup runs, sometimes just reboots on some configs). It doesn't purge atomically — you can lose a subset of files in a snapshot directory while others survive. Which is exactly what the warning was telling me: the manifest says the snapshot should contain N files of certain sizes; on disk, one is missing. Hence NO_SUCHFILE on the specific model_optimized.onnx.

This had been a ticking bomb since I first added local embeddings. It'll re-download once, work for a while, then die again the next time the OS decides to clean up. Intermittent, invisible in dev because you retry and it heals itself, catastrophic in a worker that's supposed to run unattended.

The pattern that was already in the codebase

I'd solved this exact class of problem before — for Qdrant. When I searched for how other persistent data got anchored:

# src/pacos/shared/config.py
# CWD-relative path once caused the API to init an empty NESTED knowledge_repo.
PROJECT_ROOT = Path(__file__).resolve().parents[3]

That comment is a scar from a prior bug: two processes (API + worker) with different working directories were creating two Qdrant folders and silently disagreeing about state. The fix was to define PROJECT_ROOT once and resolve every data path against it. qdrant_path and knowledge_repo_path already lived under that rule.

Fastembed's model cache is the same shape of thing: regenerable, but expensive to redownload (~90 MB), shared between API and worker, must be stable across restarts. It just wasn't wearing the same seatbelt.

The fix

Three small changes.

1. Declare the path in config, alongside the other data paths:

# src/pacos/shared/config.py
embed_model: str = "BAAI/bge-small-en-v1.5"
# Anchored to PROJECT_ROOT so API and worker share one cache,
# and it survives OS temp-dir purges.
fastembed_cache_path: str = "./fastembed_cache"

2. Actually pass it to fastembed:

# src/pacos/knowledge/index/embeddings.py
from pathlib import Path
from pacos.shared.config import PROJECT_ROOT, settings

@lru_cache(maxsize=1)
def _model():
    from fastembed import TextEmbedding
    cache_dir = (PROJECT_ROOT / settings.fastembed_cache_path).resolve()
    cache_dir.mkdir(parents=True, exist_ok=True)
    return TextEmbedding(settings.embed_model, cache_dir=str(cache_dir))

3. Gitignore the download (it's regenerable, not source):

# .gitignore
qdrant_data/
fastembed_cache/
knowledge_repo/

Then I nuked the corrupted Temp snapshot and smoke-tested a cold init:

Fetching 5 files: 100%|##########| 5/5 [00:18<00:00,  3.77s/it]
embed OK, dim: 384

Clean re-download into D:\...\personal_career_growth\fastembed_cache, embedding returns the expected 384-dim vector, and the worker picks it up on restart.

The lesson worth taking

Any library that downloads model weights on first use is going to cache them somewhere. If you don't tell it where, it picks. In fastembed's case, and in a surprising number of ML libraries, that pick is tempfile.gettempdir() or a subfolder thereof. On Linux that's usually /tmp, which is tmpfs on many distros and blown away on reboot. On Windows it's inside AppData\Local\Temp, which the OS actively cleans. On macOS it's /var/folders/..., cleaned by periodic.

These caches are not really temporary from your app's perspective. They're expensive, shared across processes, and expected to survive restarts. They belong next to your other persistent-but-regenerable state — your Qdrant folder, your SQLite dev DB, your knowledge_repo/ — anchored to a project-relative path and gitignored.

A rule I'll be applying more aggressively from here on:

If two processes read from the same cache, or if reconstructing that cache costs more than a few seconds, it does not belong under TEMP. Point it at PROJECT_ROOT and let the filesystem outlive the OS's cleanup heuristics.

The symptom looked like corruption. The cause was a directory whose contract I hadn't read. One cache_dir= argument and a config entry, and the class of bug is gone.

Drafted by Claude Sonnet from my own Claude Code session transcript, then reviewed and edited before publishing.

Sync to Async: Migrating FastAPI Endpoints to arq/Redis

yogeshchavan2008 — Tue, 14 Jul 2026 19:48:47 +0000

The problem: a PDF ingest that blocks for minutes

My ingest endpoint was synchronous. Upload a file, wait for OCR, wait for the LLM extraction pipeline, get a response. On a small note it took 3–5 seconds. On a dense PDF it could run for minutes. The original code even had a comment explaining the awkwardness:

@router.post("/ingest", response_model=list[IngestResult])
def ingest(file: UploadFile) -> list[IngestResult]:
    # Sync endpoint on purpose: the ingestion pipeline calls anyio.run()
    # internally (LLM extraction), which can't nest inside the request loop.

That comment is a smell. "Can't nest inside the request loop" means the work is too heavy for the request loop in the first place. Time to pull it out.

arq was already in pyproject.toml (arq>=0.26) and Redis was available locally. I just needed to wire it up.

Step 1: Probe the arq API before writing anything

Before touching production code I spent two minutes confirming the exact arq 0.28 surface I'd rely on:

from arq import create_pool, ArqRedis
from arq.connections import RedisSettings
from arq.jobs import Job, JobStatus, JobResult

print('JobStatus:', [s.value for s in JobStatus])
# JobStatus: ['deferred', 'queued', 'in_progress', 'complete', 'not_found']
print('from_dsn ok:', RedisSettings.from_dsn('redis://localhost:6379/0'))

Knowing the exact status strings matters because they flow through to the frontend polling loop and the OpenAPI schema.

Step 2: The worker module

I created src/pacos/platform/worker.py. The key pieces are the task function and WorkerSettings:

"""Arq background worker — runs long ingestion jobs off the request path.

Ingesting a PDF is OCR + LLM-heavy (seconds to minutes), which is too long to
block an HTTP request. The API enqueues a job here and returns a job id; the
client polls GET /api/knowledge/ingest/status/{job_id}.
"""

async def run_ingest(ctx: dict, filename: str, data: bytes) -> list[dict]:
    """Entry point called by arq. ctx is injected by the worker pool."""
    results = ingest_bytes(filename, data)   # the existing sync pipeline
    return [r.model_dump() for r in results]


class WorkerSettings:
    functions = [run_ingest]
    redis_settings = RedisSettings.from_dsn(settings.redis_url)
    max_jobs = 4
    job_timeout = 600  # 10 min ceiling for the worst PDFs

The task itself is thin: it just calls the existing ingest_bytes pipeline and returns serialisable dicts. All the complexity stays in the pipeline; the worker is just the carrier.

Step 3: Wiring the Redis pool into FastAPI's lifespan

The arq pool needs to live as long as the app and be shared across requests. FastAPI's lifespan context manager is the right place:

@asynccontextmanager
async def lifespan(_app: FastAPI):
    from pacos.shared.db import init_db
    init_db()

    # Arq pool — best-effort; if Redis is down we continue without it
    # (endpoints that need it will 503 explicitly).
    try:
        pool = await create_pool(RedisSettings.from_dsn(settings.redis_url))
        _app.state.arq_pool = pool
    except Exception:
        _app.state.arq_pool = None

    yield

    if _app.state.arq_pool:
        await _app.state.arq_pool.close()

Best-effort startup matters: during tests and offline dev I don't want a hard crash if Redis isn't reachable. Endpoints that actually enqueue jobs will 503; everything else keeps working.

Step 4: A clean dependency for the pool

Instead of reaching into request.app.state inside every route, I added one dependency to deps.py so routers stay clean:

from arq import ArqRedis
from fastapi import Depends, HTTPException, Request, status

def arq_pool(request: Request) -> ArqRedis:
    """The arq Redis pool set up at startup. 503 if Redis was unavailable."""
    pool = getattr(request.app.state, "arq_pool", None)
    if pool is None:
        raise HTTPException(
            status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
            detail="Job queue unavailable",
        )
    return pool

ArqPool = Annotated[ArqRedis, Depends(arq_pool)]

That Annotated alias is the same pattern I use for DbSession and CurrentUser. Routers just declare what they need; the DI container handles the wiring.

Step 5: The endpoint pair

The old single endpoint becomes two:

class IngestJob(BaseModel):
    job_id: str
    status: str  # always "queued" on return


@router.post("/ingest", response_model=IngestJob, status_code=202)
async def ingest(file: UploadFile, pool: ArqPool) -> IngestJob:
    """Accept a file, enqueue it, return a job handle immediately."""
    data = await file.read()
    job = await pool.enqueue_job("run_ingest", file.filename, data)
    return IngestJob(job_id=job.job_id, status="queued")


@router.get("/ingest/status/{job_id}")
async def ingest_status(job_id: str, pool: ArqPool) -> dict:
    """Poll for job progress. Frontend polls this until status == 'complete'."""
    job = Job(job_id, pool)
    status = await job.status()
    if status == JobStatus.not_found:
        raise HTTPException(status_code=404, detail="Job not found")
    if status == JobStatus.complete:
        info = await job.result_info()
        return {
            "status": "complete",
            "success": info.success,
            "result": info.result if info.success else None,
            "error": str(info.result) if not info.success else None,
        }
    return {"status": status.value}

The 202 Accepted status code is deliberate: the work hasn't finished, you got a receipt.

Step 6: The test suite problem I didn't anticipate

When I ran the existing API tests after wiring up the lifespan, they passed — but took 103 seconds instead of the usual 3. The lifespan was hitting Redis connection retries on every test run because the sandbox can't reach my local Redis.

Fix: patch out the pool in the test fixture:

# conftest.py / fixture
monkeypatch.setattr(app.state, "arq_pool", None)

With that, the suite dropped back to 3.45s and all 9 tests stayed green. The 503 behavior is correct: tests that exercise the ingest endpoint directly mock at a higher level.

Step 7: Frontend polling

On the client side I added two types and a polling helper to api.ts:

export interface IngestJob {
  job_id: string;
  status: string;
}

export type IngestPhase = "queued" | "in_progress" | "complete" | "error";

// In the api object:
ingest: (file: File) => upload<IngestJob>("/api/knowledge/ingest", form),
ingestStatus: (jobId: string) => get<IngestStatus>(
  `/api/knowledge/ingest/status/${jobId}`
),

The ingest page's doIngest handler became:

POST the file → get job_id back
Set phase to "queued"
setInterval polling ingestStatus(job_id) every 2s
On complete, clear the interval and refresh the concept list
On error, surface the message

The UI shows Queued… → Processing… → Done instead of a frozen spinner.

What the architecture looks like now

POST /ingest  ──► enqueue_job() ──► Redis queue
    ↓ 202 + job_id

GET /ingest/status/{id}  ◄── poll every 2s ◄── frontend
    ↓ {status: "in_progress"}
    ↓ {status: "complete", result: [...]}

arq worker process:
    dequeue ──► run_ingest() ──► LLM pipeline ──► write result to Redis

The API server never blocks. The worker is a separate process (arq pacos.platform.worker.WorkerSettings) that scales independently. The dependency injection pattern didn't change — routers just got a new ArqPool dependency alongside the existing DbSession.

The biggest lesson: don't wait until something is genuinely broken to pull heavy work off the request path. The blocking endpoint "worked fine" until it didn't. The arq integration turned out to be less than a day's work once I already had Redis available — most of the time went to the test fixture fix and the frontend polling loop, not the queue wiring itself.

Drafted by Claude Sonnet from my own Claude Code session transcript, then reviewed and edited before publishing.

Cut Claude API Costs 80% by Splitting Vision and Reasoning Tasks

yogeshchavan2008 — Thu, 09 Jul 2026 12:11:04 +0000

The Problem That Forced the Question

I was building an ingestion pipeline for a personal knowledge system—drop in a PDF, get structured OKF-format concept articles out the other side. Anthropic's vision models are genuinely good at reading image-only PDFs (scanned docs, Medium exports printed to PDF), so I wired them into my OCR layer.

Then I looked at the cost estimate for my sample: a 26-page image-only PDF. Running everything through claude-opus-4-8 would cost $1.00–$1.30 for a single document. Scale that to a few dozen docs and suddenly my personal knowledge base has a real operational cost.

The fix was obvious once I thought about it—but I hadn't thought about it.

The Insight: OCR Is Transcription, Not Reasoning

Vision OCR and concept extraction are fundamentally different tasks:

OCR: "What text is on this page?" Rote transcription. High-contrast text on a white background. No ambiguity. This does not need Opus.
Extraction: "Given this 15,000-character document about concurrency, identify the atomic concepts, their prerequisites, define their difficulty levels, and generate interview questions." This benefits from a stronger model.

Once I separated these mentally, the routing became obvious.

The Pricing Math

Current Claude API pricing (per 1M tokens):

Model	Input	Output
`claude-opus-4-8`	$5.00	$25.00
`claude-sonnet-4-6`	$3.00	$15.00
`claude-haiku-4-5`	$1.00	$5.00

Image tokens are roughly (width × height) / 750. My PDF rendered at 200 DPI produces Letter-size pages at ~1700×2200px—about 3.7MP—which hits near Opus's 4,800-token ceiling per page.

26 pages × ~4,800 image tokens per page = ~125,000 image tokens just for OCR.

Route	Est. cost	Notes
Opus for everything	~$1.10	wasteful
Haiku OCR + Opus extraction	~$0.20	OCR is cheap on Haiku
Haiku OCR + Sonnet extraction	~$0.18	sweet spot
Haiku for both	~$0.12	lower extraction quality

That's an 84% reduction by using Haiku for the vision calls and Sonnet for the one extraction call.

What This Looks Like in Code

In .env:

OCR_ENGINE=claude
CLAUDE_VISION_MODEL=claude-haiku-4-5
CLAUDE_MODEL=claude-sonnet-4-6

The Claude OCR engine sends each page image to the vision model:

# claude_engine.py
class ClaudeOCREngine:
    name = "claude"

    def ocr_page(self, png_bytes: bytes) -> str:
        settings = get_settings()
        client = anthropic.Anthropic(api_key=settings.anthropic_api_key)

        response = client.messages.create(
            model=settings.claude_vision_model,  # haiku-4-5
            max_tokens=4096,
            messages=[{
                "role": "user",
                "content": [
                    {
                        "type": "image",
                        "source": {
                            "type": "base64",
                            "media_type": "image/png",
                            "data": base64.b64encode(png_bytes).decode(),
                        },
                    },
                    {
                        "type": "text",
                        "text": "Extract all text from this page as clean markdown. "
                                "Ignore navigation bars, footers, share buttons, "
                                "and other page chrome."
                    }
                ],
            }]
        )
        return response.content[0].text

The extraction call goes to Sonnet with a much larger max_tokens budget:

# llm.py (extraction)
async def _extract_async(self, doc: ParsedDocument) -> ExtractionResult:
    result = await self.provider.complete(
        [Message(role="system", content=SYSTEM),
         Message(role="user", content=user_prompt)],
        temperature=0.1,
        max_tokens=24000,  # extraction needs headroom; Sonnet handles it
    )

The LLMProvider abstraction routes through settings.claude_model (Sonnet) for text completions and settings.claude_vision_model (Haiku) for image passes. Two env vars, completely separate call paths.

The Bug That Made This Non-Obvious

I hit a debugging session that illustrates why you need to actually measure this instead of guessing.

My first extraction run produced this error:

json.decoder.JSONDecodeError: Unterminated string at char 373

Same position on every run, even after bumping max_tokens from 8192 to 16000. Suspicious. I captured the raw output:

$py -c "
result = provider.complete(...)
open('extract_raw.txt','w').write(result)
"

The saved file was clean, well-formed JSON. So _coerce_json was mangling it. Traced through:

# THE BUG
def _coerce_json(text: str) -> dict:
    if text.startswith("```"):
        text = text.split("```", 2)[1]  # <-- catastrophic

The extraction document contained code samples. Sonnet wrapped its JSON in a json ` fence—standard behavior—but the article body also had code blocks with ` fences inside the JSON string values. My naive split("`", 2) shredded the output at the first interior fence, producing an empty string.

Fix: strip the outer fence line-by-line instead of splitting:

def _coerce_json(text: str) -> dict:
    lines = text.strip().splitlines()
    # Strip leading ```json fence
    if lines and lines[0].startswith("```"):
        lines = lines[1:]
    # Strip trailing ``` fence
    if lines and lines[-1].strip() == "```":
        lines = lines[:-1]
    raw = "\n".join(lines)
    start = raw.find("{")
    end = raw.rfind("}")
    return json.loads(raw[start:end + 1])

Also switched the Claude provider to streaming to avoid token truncation on large extractions—the original non-streaming call was silently hitting the max_tokens ceiling without error.

What the Pipeline Produced

After the fix, one Haiku OCR pass + one Sonnet extraction call over the 26-page PDF produced 8 atomic concept drafts:

concurrency/concurrency-definition (diff 2) prereqs=[] Qs=4 concurrency/parallelism-definition (diff 2) prereqs=['concurrency/concurrency-definition'] Qs=4 concurrency/async-io-event-loop (diff 3) prereqs=['concurrency/concurrency-definition'] Qs=4 concurrency/race-conditions-shared-state (diff 3) prereqs=['concurrency/parallelism-definition'] concurrency/amdahls-law (diff 4) concurrency/ruby-gvl (diff 4) prereqs=[4 ancestors] concurrency/ruby-ractors (diff 5) concurrency/concurrency-decision-framework (diff 3)

Each with a definition, examples, interview questions, prerequisite links, and provenance tied back to the source URL. Total cost: $0.18.

The Generalizable Pattern

This isn't just about OCR. The same logic applies anywhere you're mixing vision and reasoning:

Classification/routing ("is this a blog post, academic paper, or tutorial?"): Haiku
Structured data extraction from forms/tables: Haiku or Sonnet depending on complexity
Summarization: Sonnet usually enough
Deep synthesis across multiple sources: Sonnet or Opus
High-stakes generation (interview prep, code review): Opus

The mistake is reaching for the strongest model by default and never questioning it. OCR especially—it's an unusually clear case where you're paying Opus prices for a task that's mostly pixel-to-character mapping.

If you're building on the Claude API and haven't audited which model you're using for which step, that's probably the first place to look. The gap between Haiku and Opus is 5× on input tokens. Over any real volume, that compounds fast.

Drafted by Claude Sonnet from my own Claude Code session transcript, then reviewed and edited before publishing.

DEV Community: yogeshchavan2008

When fastembed silently rotted my worker: anchor your caches, don't trust /tmp

The tell: it's in Temp

The pattern that was already in the codebase

The fix

The lesson worth taking

Sync to Async: Migrating FastAPI Endpoints to arq/Redis

The problem: a PDF ingest that blocks for minutes

Step 1: Probe the arq API before writing anything

Step 2: The worker module

Step 3: Wiring the Redis pool into FastAPI's lifespan

Step 4: A clean dependency for the pool

Step 5: The endpoint pair

Step 6: The test suite problem I didn't anticipate

Step 7: Frontend polling

What the architecture looks like now

Cut Claude API Costs 80% by Splitting Vision and Reasoning Tasks

The Problem That Forced the Question

The Insight: OCR Is Transcription, Not Reasoning

The Pricing Math

What This Looks Like in Code

The Bug That Made This Non-Obvious

What the Pipeline Produced

The Generalizable Pattern

The tell: it's in `Temp`