I built a RAG engine that refuses to answer with garbage context (and it cost me way more debugging than expected)

Gabaoun — Thu, 06 Aug 2026 14:40:09 +0000

Most RAG demos on GitHub do the same thing: embed some chunks, cosine-similarity search, stuff the top-k into a prompt, done. I built one of those first too. Then I actually tried to use it on messy real-world docs and it fell apart in three specific ways. So I rebuilt it as an event-driven pipeline instead of a linear script, and that's what became Project Aether.

The three things that broke my first RAG attempt

1. Semantic cache that returned stale garbage. Caching LLM responses by exact query string is useless, nobody types the same question twice. I needed similarity-based caching (is this new query "close enough" to a cached one?), but that opens a nastier problem: what threshold counts as "close enough" without returning a wrong-but-plausible cached answer for a subtly different question? I ended up tuning this against a small eval set instead of guessing a cosine threshold and hoping.

2. Naive vector search missing exact-match terms. Pure dense embeddings are bad at exact keywords, part numbers, error codes, that kind of thing. Someone searches for a specific SKU and dense-only search returns five semantically-similar-but-wrong products. Fixed this with hybrid dense+sparse retrieval instead of dense-only.

3. Ingesting real documents means someone's PII ends up in your vector DB. This is the one nobody talks about in RAG tutorials. If you're chunking and embedding real support tickets or contracts, you're embedding whatever PII is in them too, permanently, in a vector store you may not fully control access to. So ingestion needed a PII-masking pass before anything gets chunked, not after.

What I ended up with

Project Aether is an event-driven RAG search engine:

FastAPI for the API layer, async end to end
LlamaIndex Workflows to model ingestion and query as actual event graphs instead of a call-this-then-that script, which made retries and partial failures way saner to reason about
Redis for semantic caching (this is the piece from problem #1 above)
Chroma Cloud for hybrid dense+sparse vector search (server-side Qwen/Splade embeddings), which is the fix for problem #2
RQ for background job processing on the ingestion side
Groq running Llama 3.3 70B for generation, mostly because I wanted fast inference without paying for GPU idle time

The ingestion pipeline masks PII before chunking, enriches chunks with LLM-generated metadata, and the query side does query transformation plus a relevance-judgment refinement loop before generation, with an optional cross-encoder reranking stage (BAAI/bge-reranker-v2-m3) that I keep off by default because it's too heavy for memory-constrained hosts.

The honest part

The live demo runs on Render's free tier, so it's query-only (ingestion runs as a local job, not exposed) and it WILL cold-start slow on first request. I'm not going to pretend otherwise, that's just what free hosting gets you. If you hit it and the first response takes a while, that's the container waking up, not the RAG pipeline being slow.

This is a solo project, no team, so if something looks half-finished it's because I built it around my own time constraints, not because I ran out of ideas.

If you want to poke at it

Repo's here: https://github.com/gabaoun/Project-Aether

I'd genuinely rather get a "this retrieval approach is wrong because X" comment than a star with no feedback, but if you find it useful a star helps other people find it too. Curious if anyone else has fought with semantic-cache threshold tuning, feels like the least-discussed hard problem in RAG right now.

DEV Community: Gabaoun

I built a RAG engine that refuses to answer with garbage context (and it cost me way more debugging than expected)

The three things that broke my first RAG attempt

What I ended up with

The honest part

If you want to poke at it