Anatomy of a Full RAG Application: Every Concept, One Self-Hosted Stack

"Chat with your documents" sounds simple. Then you build it, and you discover a good RAG system is really eight systems wearing a trench coat.

I recently finished myRAG — a fully self-hosted RAG stack: FastAPI backend, React frontend, and three storage engines (Qdrant, PostgreSQL, Neo4j), all orchestrated with Docker Compose. This post walks through every stage of the pipeline and the concept behind it, with real code from the project.

For a link to the codebase, scroll to bottom!

The architecture at a glance

Document ─► Docling ─► Chunker ─┬─► Dense embed (OpenRouter) ──┐
                                ├─► Sparse BM25 (fastembed) ───┼─► Qdrant (named vectors)
                                └─► LLM triple extraction ─────┼─► Neo4j (knowledge graph)
                                                               └─► PostgreSQL (metadata, doc_uuid)

Question ─► hybrid search (RRF) ─► rerank ─► + graph facts ─► + memory ─► token budget ─► LLM ─► SSE stream

Six containers: the app, the React/nginx frontend, docling-serve, Qdrant, Postgres, and Neo4j.

1. Parsing: retrieval quality is capped by parsing quality

Everything starts with converting PDFs/DOCX/HTML into text an LLM can use. I run Docling as a separate service that returns clean Markdown, preserving headings and tables.

Why Markdown? Because structure survives. A table flattened into a character soup is unsearchable no matter how good your embedding model is. Garbage in, garbage out — this stage silently determines your ceiling.

2. Chunking: the most underrated decision in RAG

Chunks are the retrieval unit. Too large and the embedding becomes a blurry average of many topics; too small and the chunk loses the context needed to be understood alone.

chunking:
  strategy: recursive
  chunk_size: 512
  chunk_overlap: 64

Recursive splitting respects paragraph and sentence boundaries, and the overlap is essential: without it, answers that straddle a chunk boundary fall into the gap and are never retrieved whole.

3. Index twice: dense for meaning, sparse for precision

Most tutorials embed once and call it done. I index every chunk two ways:

• Dense vectors (Qwen3-Embedding, 4096-dim, via OpenRouter) — semantic similarity. "Q3 revenue" matches "third-quarter earnings."
• Sparse BM25 vectors (computed locally with fastembed, no API cost) — lexical similarity. Exact part numbers, names, acronyms — the things embedding models fumble.

Qdrant stores both on the same point as named vectors, with BM25's IDF computed server-side:

self.client.create_collection(
    collection_name=...,
    vectors_config={"dense": VectorParams(size=4096, distance=Distance.COSINE)},
    sparse_vectors_config={"bm25": models.SparseVectorParams(modifier=models.Modifier.IDF)},
)

One subtlety: BM25 weights documents and queries differently, so ingestion uses embed() (term weighting) while queries use query_embed() (term presence).

4. Hybrid search: Reciprocal Rank Fusion

At query time, both searches run and Qdrant fuses them with RRF — which merges ranked lists instead of trying to normalize incomparable score scales:

result = self.client.query_points(
    collection_name=...,
    prefetch=[
        models.Prefetch(query=dense_vector, using="dense", limit=20),
        models.Prefetch(query=sparse_query, using="bm25", limit=20),
    ],
    query=models.FusionQuery(fusion=models.Fusion.RRF),
    limit=10,
)

RRF scores each document by Σ 1/(k + rank) across lists. A chunk ranked highly by either method surfaces; one ranked highly by both wins. No score calibration, no tuning — and it works disturbingly well.

5. Reranking: cheap recall, then expensive precision

Vector search compares a query against chunks that were embedded without knowing the question. A cross-encoder reranker (Cohere rerank, via OpenRouter) reads query and chunk together and produces a much sharper relevance score.

The pattern is a funnel: hybrid-retrieve 10 candidates cheaply, rerank down to the best 5. It's the same two-stage architecture search engines have used for decades — recall first, precision second.

6. The knowledge graph: RAG that connects the dots

Pure vector RAG struggles with relational questions — "Who reports to the person who founded X?" spans facts that live in different chunks.

So during ingestion, an LLM extracts (subject, relation, object) triples from every chunk into Neo4j:

You are a knowledge-graph extraction engine. From the text below, extract factual
relationships as a JSON array of triples... Only extract relationships explicitly
stated in the text. Return ONLY the JSON array, no prose.

At query time the flow is: extract entities from the question → match them against graph nodes via Neo4j's fulltext index → pull their 1-hop neighborhood → inject the triples into the prompt as structured facts:

Knowledge graph facts:
- Acme Corp --acquired--> Widget Inc
- Widget Inc --founded_by--> Jane Doe

Call it GraphRAG-lite: the vector store answers "what's relevant," the graph answers "how things relate." Every relationship is tagged with doc_uuid, so deleting a document prunes its facts (and any orphaned entities) cleanly.

7. Memory + token budgeting: context is a budget

Multi-turn chat needs history, but the context window is finite. Two mechanisms:

Rolling summarization — after N turns, older exchanges are compressed into a running summary by a small LLM. Long conversations cost a paragraph, not pages.

Token budgeting — before generation, the prompt is assembled against a hard cap with explicit priorities:

# Priority (always kept): system prompt, summary, graph facts, the question.
# Then newest history, then chunks best-first; lowest-ranked chunks drop first.
while history_msgs and fixed + est(history_msgs) > budget:
    history_msgs.pop(0)          # oldest history goes first

for chunk in chunks:             # rerank order: best first
    if used + est(chunk) > avail and len(kept) >= min_chunks:
        break
    kept.append(chunk)

The mental model: every token of history you keep is a token of evidence you can't include. Making that trade-off explicit — instead of letting the API truncate arbitrarily — noticeably improves long conversations.

8. Parallelism: same models, several times faster

Ingestion is embarrassingly parallel — embedding API calls, local BM25 encoding, and per-chunk graph extraction are all independent I/O. A single shared thread pool handles the fan-out:

# executor.py — one process-wide pool, order-preserving map
def map_parallel(fn, items):
    items = list(items)
    if len(items) <= 1:
        return [fn(i) for i in items]
    return list(get_pool().map(fn, items))

Three applications:

# 1. Dense embedding: sub-batches of 32 texts, requests in flight concurrently
batches = [texts[i:i+32] for i in range(0, len(texts), 32)]
vectors = [v for batch in map_parallel(self._embed_request, batches) for v in batch]

# 2. Sparse BM25 runs concurrently with the dense API round-trips
sparse_future = get_pool().submit(self.sparse_embedder.embed_batch, texts)
vectors = self.embedder.embed_batch(texts)
sparse_vectors = sparse_future.result()

# 3. Graph extraction: one LLM call per chunk, previously sequential — now fanned out
futures = [get_pool().submit(self.graph_extractor.extract, t) for t in texts]
for i, future in enumerate(as_completed(futures)):
    ...  # upsert triples + yield progress events as each completes

At query time, the graph fact lookup is submitted before retrieval starts, so it overlaps with retrieval + reranking instead of running after them. Graph extraction was the slowest ingestion phase by far — parallelizing it is the difference between 90 seconds and 15 for a mid-sized document. No async rewrite needed; threads are plenty for I/O-bound work.

9. The unglamorous parts that make it "production"

• Referential integrity. One doc_uuid ties each document across Postgres (primary key), Qdrant (point payload), and Neo4j (relationship property). Deletes cascade through all three stores.
• Streaming UX. Ingestion and chat are generator-based and stream progress over SSE — the UI shows parsing → chunking → embedding → storing → graph → done live. One gotcha: EventSource can't send headers, so the client streams via fetch + ReadableStream to pass the auth token.
• Auth. Every /api/* endpoint requires a bearer token, checked with secrets.compare_digest (constant-time). The React build bakes the key in at build time from an env var — no login screen; docker-compose sources frontend and backend from the same .env so they can't drift.
• Config as code. One config.yaml for every tunable, Pydantic Settings overlays secrets from .env, zero hardcoded values.

The takeaway

RAG isn't one technique. It's a pipeline of small, well-understood ideas:

Parse cleanly → chunk thoughtfully → index twice → fuse ranks → rerank deeply → add structure with a graph → budget your tokens → parallelize the waits.

None of these steps is hard alone. The engineering is in making them agree with each other — sharing one UUID, one config, one thread pool, and one honest token budget.

Access the full codebase here: https://github.com/sumannath/myRAG

Questions about any layer? The hybrid search and knowledge-graph stages delivered the biggest quality jumps for me, and I'm happy to go deeper on either in the comments.