I built a RAG system to stop hallucinating. Then it started ghosting me.

samantha monis — Thu, 24 Sep 2026 09:15:57 +0000

A RAG system that flags its own hallucinations, compares documents side-by-side, and — five months later — taught me more about infrastructure rot than about retrieval.

Five months after I first shared the live demo of this project, I opened it again to write about it — and it was completely broken.

Turned out the backend had quietly gone to sleep on a free-tier host, and Groq had deprecated the exact model I was calling, all in the months I wasn't looking. Classic "it worked when I built it" story.

But before the bug, there's the actual reason this project exists.

Most RAG systems I'd used or read about had the same quiet flaw: they answer everything, confidently, whether or not the answer is actually true. They're people-pleasers by design — built to always produce a response, never to say "I'm not sure this is grounded in your documents." A retrieval system that can't tell you when it's making things up isn't actually trustworthy, it's just fluent.

So I built an Adaptive Retrieval System around one core idea: retrieval should know when it's guessing. On top of that, I added two things most basic RAG demos skip entirely — the ability to compare answers across multiple documents side-by-side, and a document map that visualizes embeddings and clusters, so you can actually see how your documents relate to each other instead of trusting a black box.

What's under the hood

Here's the thing about basic RAG: it looks solved. Chunk, embed, retrieve, generate — four steps, a weekend project, done. Except each of those four steps is quietly lying to you a little.

Retrieval isn't actually retrieving the right thing half the time — vector search finds what's semantically similar, not necessarily what's relevant. Your question, typed in casually, is rarely the ideal search query. And the model at the end will answer confidently no matter what — even when the honest answer is "I don't actually know, based on what you gave me."

I built this system by attacking each of those lies, one layer at a time.

The first layer is just making retrieval actually work. Hybrid search runs BM25 keyword matching alongside vector search and merges them with Reciprocal Rank Fusion — because pure vector search misses exact terms, and pure keyword search misses meaning. Then a CrossEncoder reranker takes a second, slower pass over whatever comes back, actually reading each chunk against the query instead of just comparing embedding distances. And before any of that happens, the LLM itself rewrites your question into a sharper search query, so "what about that?" becomes something a search index can actually use.

That gets you good retrieval. It doesn't get you trustworthy retrieval. So the second layer is about honesty. If two of your sources disagree with each other, the system tells you — it doesn't just quietly average them into a confident-sounding answer. Every response comes with a real confidence score, computed from actual retrieval quality, not a decorative progress bar. And this is the part I actually built the whole project around: CRAG, a self-check that runs after the answer is generated, comparing it back against the retrieved context to flag whether it looks grounded — or whether the model might be quietly making things up.

The third layer is for actually understanding what you're working with, not just querying it blindly. Document Map projects every chunk's embedding into 2D space with UMAP, clusters them, and labels the clusters with AI — so instead of trusting a black box, you can see the shape of your document. And Compare Docs runs the same query across multiple documents at once, laying answers side-by-side instead of forcing you to ask the same question five times in five tabs.

All of it sits on FastAPI and React, talks to Groq for inference, uses sentence-transformers for embeddings and reranking, and runs live across Hugging Face Spaces and Vercel.

When it all quietly broke

Five months is a long time for infrastructure you're not watching.

I opened the live demo to write this post, tried uploading a document, and got a flat "Failed" with no explanation. No stack trace, no error code — just failed.

No error code. No explanation. Just failed.

My first assumption was the obvious one: the backend, hosted on Render's free tier, had probably gone to sleep from inactivity. Free-tier services do that, and five months of silence is more than enough. I went to check — and the dashboard showed the service as "Deployed," green, running. Not asleep. Which meant my working theory was wrong, and I had nothing to replace it with yet.

So I went to the logs. Nothing. Zero requests, zero errors, for a full week — while I was actively trying to upload a file and getting a real "Failed" response in the browser. A backend that's up but has no record of the request that's supposedly hitting it is a strange kind of broken. It meant the request wasn't reaching this service at all.

That sent me into the browser's network inspector, and that's where the actual answer was hiding: the frontend wasn't calling my Render backend. It was calling a completely different URL — a Hugging Face Space I'd forgotten was even part of this deployment. Somewhere in the project's history, the backend had moved, and I'd been debugging the wrong service for the better part of an hour.

The real backend, hiding in plain sight in the Network tab.

The Hugging Face Space told a clearer story once I found it: Runtime error — Scheduling failure: unable to schedule. Not my code — HF's infrastructure simply couldn't find it a machine to run on. A restart fixed that part.

And then, right as uploads started working again, a new error showed up — cleaner, at least: model llama-3.3-70b-versatile does not exist or you do not have access to it. I'd been calling a model that, it turned out, Groq had deprecated months earlier and shut down entirely by mid-August. I swapped in a replacement, tested it, and hit the exact same error on a different model name — because that one had been deprecated too, in the same wave.

Fixed one problem. Met a new one immediately.

Two free-tier services, one silent migration between hosts, and two dead model names — all stacked on top of each other, all surfacing as the same unhelpful "Failed" message on the frontend. None of it was a bug in the traditional sense. It was just what happens when you build something, walk away, and the ground quietly shifts underneath it.

The fix, in the end, was small: point the model names at what Groq actually supports now. The debugging, going from "Failed" to knowing exactly what was wrong, took far longer than the fix itself — which is usually how it goes.

What I'd still fix

This isn't really "fixed" forever. Some model will get deprecated again someday, and I'll probably forget to check on this again too. First thing I'd actually add: a simple health check that pings the model endpoint and emails me if something breaks, instead of finding out five months later.

A few other things I know are missing: the confidence score doesn't explain why it's 62% and not 90%, CRAG flags bad answers but doesn't retry them, and there's no login system — right now it only really works for one person at a time.

It's not done. I don't think any project really is. But this is where it's at right now, and I'm okay sharing that.

Try it yourself

If you want to poke at it, the live app is here:
https://adaptive-rag-system-beta.vercel.app/
Upload something small first — a PDF or a few pages of notes work well to see the hybrid search and reranking actually do something visible. Try Document Map on a longer document too, that's the part I still find genuinely fun to look at, even having built it.

Code's on GitHub if you want to see how any of this is actually wired together: https://github.com/samanthajessica2004/adaptive-rag-system

If you try it and something breaks — tell me. Given everything above, it wouldn't even surprise me.

DEV Community: samantha monis

I built a RAG system to stop hallucinating. Then it started ghosting me.

What's under the hood

When it all quietly broke

What I'd still fix

Try it yourself