Building a RAG System in Rust with Qdrant, Rig, and gRPC 🦀

With Qdrant, Rig, Tonic, ratatui — and a healthy obsession with what's actually happening underneath.

Why I Built This

A few weeks ago I came across Our First Production-Ready RAG Dev Journey in Pure Rust by the rust-dd team — and something clicked.

Reading it, I realized two things:

1. Rust is the right language for building AI systems — fast, safe, and built for the kind of infrastructure work AI actually needs.
2. I'd been wanting to build something like this myself for months and kept finding excuses not to start.

So I started. This post is what came out of it — and it's grown since the first version: it's now a proper Cargo workspace with a reusable engine, it indexes your docs automatically, it streams answers token-by-token, and it ships with an interactive terminal UI built in ratatui. 🦀

---

Most RAG tutorials hand you a framework, four function calls, and a working demo. You upload some documents, vectors get generated somewhere, an LLM answers your questions, and you walk away with a chatbot but no real understanding of what just happened.

This post takes the opposite approach. We'll build a small but complete RAG system in Rust using Qdrant for vector search, Rig as the AI framework, Tonic for a gRPC API, and ratatui for an interactive terminal UI — and explain why every piece exists. By the end you'll understand:

• 🧠 Why embeddings work and how semantic retrieval actually functions
• 🗄️ Why vector databases exist and what problem ANN indexing solves
• ⚡ Why async runtimes matter for retrieval pipelines
• 🦀 Why Rust is becoming genuinely interesting for AI infrastructure
• 🔌 How gRPC fits into modern AI service architectures
• 🖥️ How to stream LLM output into a live terminal UI without blocking

The full source is at github.com/Parikalp-Bhardwaj/qrag-rust. Clone it and read along.

---

🤔 Why RAG exists

LLMs are powerful, but they don't actually know anything about your data. They generate from patterns learned during training, which means:

• 🚫 Hallucinations when asked about specifics they never saw
• 📅 Outdated information — anything after the cutoff is invisible
• 🔒 No access to private knowledge — your docs, your codebase, your wiki
• 📏 Limited context windows — you can't paste everything in

A model trained in 2023 can't tell you what your internal API does or what changed in last week's release. That's not a model problem — it's a retrieval problem.

RAG fixes it by inserting a retrieval step before generation:

query
  → retrieve relevant context from your data
    → inject context into the prompt
      → generate a grounded response

This shifts the engineering focus. The LLM becomes one component among several, and the quality of the system depends on chunking strategy, embedding quality, retrieval accuracy, indexing, and latency. That's the territory most tutorials skip. We're going to live there.

---

🧩 The building blocks

Five pieces do most of the work in this project.

🔎 Qdrant — the vector database

In a normal database you search by exact values:

SELECT * FROM documents WHERE title = 'Rust';

That works for keywords. It doesn't work for meaning. Consider:

Query: "How does Rust prevent race conditions?"

Relevant doc: "Rust provides memory safety and fearless concurrency through ownership and borrowing."

Zero word overlap. A SQL LIKE won't find this. But a human reading both knows they're talking about the same thing.

Qdrant stores documents as vectors — numerical fingerprints of meaning. Two semantically similar texts produce two vectors that sit close together in high-dimensional space. Search becomes "find the vectors nearest to my query vector." A Qdrant point looks like:

Point {
  id: "doc-1",
  vector: [0.12, -0.44, 0.91, ...],   // 1536 dimensions
  payload: {
    "file_path": "rust-notes.md",
    "chunk_index": 3,
    "text": "Rust ownership prevents memory bugs..."
  }
}

Why a dedicated database? Naively, retrieval means comparing the query vector against every stored vector — O(n) per query. Fine for 100 documents, a disaster for 10 million. Qdrant uses Approximate Nearest Neighbor (ANN) indexing (HNSW under the hood) to find near matches without scanning everything. You trade a sliver of accuracy for a dramatic speedup. That's the entire reason vector databases exist as a category.

🦀 Rig — the AI application layer

Rig is a Rust framework for LLM apps. It handles the boring, provider-specific parts: embedding APIs, completion APIs, vector store glue, agent construction, and — importantly for this version — streaming completions. Without Rig we'd be hand-writing HTTP clients and SSE parsers for OpenRouter.

⚠️ Honest caveat: Rig reduces boilerplate, but it doesn't think for you. Chunking strategy, retrieval ranking, prompt construction — those are still your problem and they're what actually determine answer quality.

🚀 gRPC + Tonic — the service layer

gRPC is a high-performance RPC framework. Instead of POST /chat with JSON, you define services in Protocol Buffers and get strongly-typed clients and servers in any supported language. Tonic is the Rust implementation.

Why gRPC over REST? Two reasons:

1. Typed contracts. The .proto file is the single source of truth. Client and server can't drift apart.
2. Backend-to-backend fit. Real AI infrastructure looks like frontend → API gateway → RAG service → vector DB → LLM provider. gRPC is built for the internal hops.

🖥️ ratatui — the terminal UI

ratatui is a Rust library for building rich terminal user interfaces — panels, layouts, colored text, live updates. In this project it powers an interactive chat app that streams answers and shows retrieval sources side-by-side. It's what turns "a RAG backend" into "a thing you actually want to use."

⚡ Tokio — the async runtime

Every network call here is async — Qdrant queries, OpenRouter embeddings, LLM completions. Tokio lets thousands of them run concurrently on a small thread pool without us managing threads by hand. It's also what lets the TUI stream tokens in the background while staying responsive to your keystrokes. The quiet substrate underneath everything else.

---

🏗️ Architecture at a glance

The system has two phases that share one RagEngine.

Indexing (runs automatically on first startup, or on demand via Reindex):

./docs → load → chunk → embed (OpenRouter) → store (Qdrant)

Query (runs every time someone asks):

question → embed → search Qdrant → top-k chunks → prompt → LLM (streamed) → answer

Same embedding model on both sides — that's what makes the geometry work. You can't embed documents with one model and queries with another and expect distances to mean anything.

Two things changed from the first version and are worth calling out up front, because they shape everything else:

• The engine is now a library. All the RAG logic lives in a qrag-core crate that the server and the TUI depend on. One engine, many front ends.
• Indexing is automatic. On startup the app checks whether the collection has any vectors; if it's empty, it indexes ./docs for you. No more "why is it saying it can't find anything?" on first run.

---

📂 Project layout

The project is a Cargo workspace. The engine is a reusable library (qrag-core); the gRPC server and its simple chat client live in qrag-server; the interactive terminal UI is qrag-tui.

qrag-rust/
├── Cargo.toml                   # workspace root + shared dependency versions
├── docker-compose.yaml          # Qdrant container
├── .env.example                 # template for secrets
├── proto/
│   └── rag.proto                # gRPC service definition
├── docs/                        # your knowledge base lives here
│   ├── grpc.md
│   ├── rust.md
│   ├── tokio.md
│   └── Rust-for-Network-Programming-and-Automation.pdf
└── crates/
    ├── qrag-core/               # reusable RAG engine (library)
    │   └── src/
    │       ├── lib.rs           # re-exports the public API
    │       ├── config.rs        # env-var configuration
    │       ├── document_loader.rs  # read .md, .txt, .pdf
    │       ├── chunker.rs       # split into ~120-word chunks
    │       ├── qdrant_store.rs  # embeddings + vector storage
    │       ├── llm.rs           # prompt + streamed completion
    │       └── rag.rs           # orchestration (+ auto-index, streaming)
    ├── qrag-server/             # gRPC server + chat client
    │   ├── build.rs             # compiles .proto → Rust at build time
    │   └── src/
    │       ├── main.rs          # boots the gRPC server
    │       ├── grpc_service.rs  # tonic handlers
    │       └── bin/chat.rs      # simple terminal chat client
    └── qrag-tui/                # interactive ratatui UI (embeds qrag-core)
        └── src/
            ├── main.rs          # boots engine + opens the UI
            ├── app.rs           # state + async event loop
            └── ui.rs            # layout / rendering

Each file has one job, and now each crate has one job too. That's deliberate — RAG systems get complicated quickly, and clean seams are the only defense. Splitting the engine into a library is what made the TUI possible without copy-pasting logic.

Why a workspace?

In the first version, all the logic lived inside the server binary. That was fine until I wanted a second front end. The moment you want the same engine behind a gRPC server and a terminal UI (and maybe a desktop app later), you don't want two copies of the retrieval code — you want one library both of them call.

A Cargo workspace is the standard Rust answer. The root Cargo.toml defines the members and shares dependency versions:

[workspace]
resolver = "2"
members = [
    "crates/qrag-core",
    "crates/qrag-server",
     "crates/qrag-tui",
]

[workspace.package]
version = "0.1.0"
edition = "2024"

[workspace.lints.rust]
unsafe_code = "warn"

[workspace.lints.clippy]
all = { level = "warn", priority = -1 }
result_large_err = "allow"

[workspace.dependencies]
anyhow = "1.0.102"
tokio = { version = "1", features = ["full"] }
rig = "0.37.0"
rig-qdrant = "0.2"
qdrant-client = "1"
tonic = "0.12"
prost = "0.13"
tonic-build = "0.12"
dotenvy = "0.15"
uuid = { version = "1", features = ["v4"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
tracing = "0.1"
tracing-subscriber = "0.3"
ratatui = "0.29"
crossterm = { version = "0.28", features = ["event-stream"] }
futures = "0.3"

Each crate then opts in with foo.workspace = true, so versions stay in lockstep across the whole project. qrag-core is a pure library. qrag-server and qrag-tui both depend on it with qrag-core = { path = "../qrag-core" }.

---

🛠️ Prerequisites

Rust

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env
rustc --version   # needs a recent toolchain — the project uses edition 2024

System packages

A surprising number of Rust crates compile native C/C++ underneath. On Ubuntu:

sudo apt update
sudo apt install -y \
    build-essential pkg-config libssl-dev \
    clang cmake protobuf-compiler poppler-utils

Package	Why you need it
build-essential	GCC/G++ for native compilation
pkg-config	Locates system libraries
libssl-dev	OpenSSL headers — any HTTPS crate needs them
clang	LLVM toolchain (bindgen, ML runtime crates)
cmake	Used by several native ML libraries
protobuf-compiler	The protoc binary that turns .proto into Rust
poppler-utils	Provides pdftotext — we shell out to it for PDFs

On macOS: brew install protobuf poppler.

Verify protoc:

protoc --version

Docker + Qdrant

sudo apt install -y docker.io docker-compose-plugin
sudo systemctl enable --now docker
docker compose up -d   # uses the docker-compose.yaml in the repo

Port	What it serves
6333	REST API + dashboard at http://localhost:6333/dashboard
6334	gRPC API — what our Rust client talks to

OpenRouter key

We use OpenRouter so the same key serves both the embedding model and the completion model. Create a key, then set up your .env.

---

🔐 Setting up .env and Qdrant

Before you can run anything, two things need to be in place: a .env file with your API key, and a running Qdrant container.

1. Create the .env file

In the project root, create a file named .env:

# Required — your OpenRouter API key
OPENROUTER_API_KEY=sk-or-v1-paste-your-key-here

# Optional — defaults shown
SERVER_ADDR=127.0.0.1
PORT=50051
QDRANT_URL=http://127.0.0.1:6334
QDRANT_COLLECTION=question
MODEL=openai/gpt-4o-mini

Get a key at openrouter.ai/keys. Free tier is enough to test.

What each variable does:

Variable	Default	What it controls
OPENROUTER_API_KEY	(required)	Authenticates both embedding and LLM calls
SERVER_ADDR	127.0.0.1	Host the gRPC server binds to (use 0.0.0.0 for Docker)
PORT	50051	gRPC port your clients connect to
QDRANT_URL	http://127.0.0.1:6334	Where Qdrant's gRPC endpoint lives
QDRANT_COLLECTION	question	Name of the collection that stores your vectors
MODEL	openai/gpt-4o-mini	OpenRouter model used for answers

That last one is new: you can swap the LLM without touching code. Want to try a bigger model? Set MODEL=openai/gpt-4o (or any OpenRouter model) in .env and restart.

⚠️ Never commit .env to git. Add it to .gitignore:

echo ".env" >> .gitignore

If you've already committed it once, rotate the key — git history doesn't forget. Ship a .env.example with empty values instead so collaborators know what to set.

2. Start Qdrant with Docker

The repo includes a docker-compose.yaml:

services:
  qdrant:
    image: qdrant/qdrant:latest
    container_name: coderag-qdrant
    ports:
      - "6333:6333"   # REST API + dashboard
      - "6334:6334"   # gRPC API (this is what the Rust app uses)
    volumes:
      - qdrant_data:/qdrant/storage

volumes:
  qdrant_data:

Start it and verify:

docker compose up -d
docker ps                          # coderag-qdrant should be listed
curl http://localhost:6333/healthz # → healthz check passed

---

📖 Walking through the code

This is the part most tutorials skip. Let's go through the engine crate, then the two front ends.

qrag-core/src/config.rs — environment-driven config

use std::env;

#[derive(Debug, Clone)]
pub struct Config {
    pub addr: String,
    pub port: u16,
    pub qdrant_url: String,
    pub qdrant_collection: String,
    pub model: String,
}

impl Config {
    pub fn from_env() -> Self {
        Self {
            addr: env::var("SERVER_ADDR").unwrap_or_else(|_| "127.0.0.1".to_string()),
            port: env::var("PORT").unwrap_or_else(|_| "50051".to_string())
                .parse::<u16>().expect("PORT must be a valid number"),
            qdrant_url: env::var("QDRANT_URL")
                .unwrap_or_else(|_| "http://127.0.0.1:6334".to_string()),
            qdrant_collection: env::var("QDRANT_COLLECTION")
                .unwrap_or_else(|_| "question".to_string()),
            model: env::var("MODEL")
                .unwrap_or_else(|_| "openai/gpt-4o-mini".to_string()),
        }
    }

    pub fn server_addr(&self) -> String {
        format!("{}:{}", self.addr, self.port)
    }
}

Plain config struct populated from environment variables with sensible defaults. The new model field is what makes the LLM swappable — every default works out of the box for local dev, and production overrides come from .env.

qrag-core/src/document_loader.rs and chunker.rs

These are unchanged from the first version — the loader walks ./docs, reads .md/.txt/.pdf (shelling out to pdftotext for PDFs), and the chunker splits documents into ~120-word pieces, each with a UUID and its source path. Chunking is still the single most underrated decision in RAG: too large and retrieval gets vague, too small and meaning fragments across boundaries. 120 words is a solid baseline for technical docs.

qrag-core/src/qdrant_store.rs — retrieval + a new "is it empty?" check

The store still creates the collection, embeds and upserts chunks, and searches by question. The one addition this version needs is a way to ask Qdrant how many vectors it holds — that's what powers auto-indexing:

use qdrant_client::qdrant::CountPointsBuilder;

impl QdrantStore {
    /// How many points (indexed chunks) are stored in the collection.
    pub async fn count_points(&self) -> Result<u64> {
        let resp = self.client
            .count(CountPointsBuilder::new(&self.collection_name).exact(true))
            .await
            .context("Failed to count points in Qdrant collection")?;
        Ok(resp.result.map(|r| r.count).unwrap_or(0))
    }

    pub async fn is_empty(&self) -> Result<bool> {
        Ok(self.count_points().await? == 0)
    }
}

Everything else — ensure_collection (1536 dims, cosine distance), upsert_chunks, and search — works exactly as before. The 1536 still has to match text-embedding-3-small, and getting it wrong still fails every upsert. Some things never change.

qrag-core/src/llm.rs — streaming the answer

This is the biggest change in the whole project. In the first version the LLM call was one shot: send the prompt, wait, get the full string back. Now it streams — text arrives token-by-token, and we forward each delta to a callback as it lands.

use anyhow::{Context, Result};
use futures::StreamExt;
use rig::{
    agent::MultiTurnStreamItem,
    client::{CompletionClient, ProviderClient},
    providers::openrouter,
    streaming::{StreamedAssistantContent, StreamingPrompt},
};
use crate::{qdrant_store::RetrievedChunk, Config};

#[derive(Clone)]
pub struct LlmService {
    client: openrouter::Client,
}

impl LlmService {
    pub fn new() -> Result<Self> {
        let client = openrouter::Client::from_env()
            .context("Failed to create Rig OpenRouter client from OPENROUTER_API_KEY")?;
        Ok(Self { client })
    }

    /// Stream the answer token-by-token. `on_token` is called with each text
    /// delta as it arrives; the full answer is also accumulated and returned.
    pub async fn answer_question_streamed(
        &self,
        question: &str,
        chunks: &[RetrievedChunk],
        mut on_token: impl FnMut(&str),
    ) -> Result<String> {
        if chunks.is_empty() {
            let msg = format!("could not find relevant content for: {}", question);
            on_token(&msg);
            return Ok(msg);
        }

        let config = Config::from_env();
        let context = build_context(chunks);

        let prompt = format!(r#"
            You are a helpful Rust AI assistant.
            Answer the question using only the provided document context.

            Rules:
            - Be clear and concise.
            - If the context is not enough, say so.
            - Mention the source file when useful.
            - Do not invent facts outside the context.

            Context:
            {}

            Question:
            {}

            Answer:
            "#, context, question);

        let agent = self.client
            .agent(&config.model)   // model comes from the MODEL env var
            .preamble("You answer questions using retrieved chunks as grounded context.")
            .build();

        // stream_prompt yields assistant items; we forward the text deltas.
        let mut stream = agent.stream_prompt(prompt).await;
        let mut full = String::new();
        while let Some(item) = stream.next().await {
            let item = item.map_err(|e| anyhow::anyhow!("streaming failed: {e}"))?;
            if let MultiTurnStreamItem::StreamAssistantItem(
                StreamedAssistantContent::Text(t)
            ) = item {
                full.push_str(&t.text);
                on_token(&t.text);
            }
        }
        Ok(full)
    }
}

fn build_context(chunks: &[RetrievedChunk]) -> String {
    let mut context = String::new();
    for chunk in chunks {
        context.push_str(&format!(
            "\nSource: {}\nChunk: {}\nScore: {:.4}\nText: {}\n",
            chunk.file_path, chunk.chunk_index, chunk.score, chunk.text
        ));
    }
    context
}

The key move is agent.stream_prompt(prompt).await, which gives back a stream of assistant items instead of a single response. We pull StreamedAssistantContent::Text deltas out of it, append each to a running full string, and hand it to on_token so a UI can render it live. Callers that don't care about streaming (like the gRPC server) just pass a no-op closure |_| {} and use the returned full string.

The grounding rules in the prompt matter as much as ever. "Do not invent facts outside the context" is what keeps the model from filling gaps with plausible-sounding nonsense, and "If the context is not enough, say so" tells it that "I don't know" is a valid answer. And if retrieval returns nothing, we short-circuit before calling the model at all.

qrag-core/src/rag.rs — orchestration, auto-index, and streaming

RagEngine is still the conductor, but it grew two capabilities: streaming and auto-indexing.

#[derive(Debug)]
pub enum IndexStatus {
    AlreadyPopulated(u64),
    Indexed(usize),
}

impl RagEngine {
    // ... new(), initialize(), reindex_docs() unchanged ...

    /// Non-streaming answer (used by the gRPC server): discards the deltas.
    pub async fn ask_question(&self, question: String) -> Result<RagAnswer> {
        let chunk = self.qdrant_store.search(&question, 3).await?;
        let answer = self.llm
            .answer_question_streamed(&question, &chunk, |_| {})
            .await?;
        Ok(RagAnswer { answer, sources: chunk })
    }

    /// Retrieve sources without answering — lets a UI show them first.
    pub async fn retrieve(&self, question: &str) -> Result<Vec<RetrievedChunk>> {
        self.qdrant_store.search(question, 3).await
    }

    /// Stream the answer; on_token receives each delta.
    pub async fn answer_streamed(
        &self,
        question: &str,
        chunks: &[RetrievedChunk],
        on_token: impl FnMut(&str),
    ) -> Result<String> {
        self.llm.answer_question_streamed(question, chunks, on_token).await
    }

    /// Index ./docs automatically, but only if the collection is empty.
    pub async fn ensure_indexed(&self) -> Result<IndexStatus> {
        let existing = self.qdrant_store.count_points().await?;
        if existing > 0 {
            return Ok(IndexStatus::AlreadyPopulated(existing));
        }
        let indexed = self.reindex_docs().await?;
        Ok(IndexStatus::Indexed(indexed))
    }
}

ensure_indexed is the fix for the most common first-run confusion: you start the app, ask a question, and get "could not find relevant content" because you never ran /reindex. Now the app checks the collection on startup and indexes for you if it's empty — and skips the work (and the embedding bill) if it's already populated, so restarts stay fast.

The split into retrieve + answer_streamed is what lets the TUI show sources immediately and then stream the answer into place. ask_question keeps the old one-shot behavior for the gRPC server by passing a no-op token sink.

qrag-server/src/main.rs — auto-index on boot

The server now calls ensure_indexed() right after initialize() and logs what happened:

engine.initialize().await?;

match engine.ensure_indexed().await {
    Ok(IndexStatus::AlreadyPopulated(n)) =>
        info!("index already populated with {n} chunks — skipping auto-index"),
    Ok(IndexStatus::Indexed(n)) =>
        info!("auto-indexed {n} chunks from ./docs"),
    Err(e) =>
        warn!("auto-index failed ({e:#}); server will start empty — run /reindex once fixed"),
}

Everything else is the same linear boot: load .env, set up logging, build config, construct QdrantStore → LlmService → RagEngine, then start the tonic server.

The gRPC layer (grpc_service.rs, proto/rag.proto) is unchanged — AskQuestion and Reindex, with responses that carry the source chunks so callers can verify the grounding. Returning your sources is still the difference between a RAG system and a black box.

---

🖥️ The interactive TUI

This is the new front end, and the reason the engine became a library. qrag-tui embeds qrag-core directly — there's no server to run separately. One command boots the engine, auto-indexes, and opens a two-panel terminal app.

qrag-tui/src/main.rs — boot and hand off

#[tokio::main]
async fn main() -> Result<()> {
    dotenvy::dotenv().ok();

    let config = Config::from_env();
    let qdrant_store = QdrantStore::new(&config.qdrant_url, &config.qdrant_collection)?;
    let llm = LlmService::new()?;
    let engine = RagEngine::new("./docs", qdrant_store, llm);

    engine.initialize().await?;

    // Index before we take over the screen, so progress/errors are visible.
    println!("Preparing index…");
    match engine.ensure_indexed().await {
        Ok(IndexStatus::AlreadyPopulated(n)) => println!("Index ready ({n} chunks)."),
        Ok(IndexStatus::Indexed(n)) => println!("Auto-indexed {n} chunks from ./docs."),
        Err(e) => eprintln!("Auto-index failed ({e:#}); starting anyway."),
    }

    let mut terminal = ratatui::init();
    let result = App::new(engine).run(&mut terminal).await;
    ratatui::restore();
    result
}

ratatui::init() switches the terminal into raw mode and the alternate screen; ratatui::restore() puts it back when we're done. The whole app lives between those two calls.

qrag-tui/src/app.rs — the async event loop

The heart of the TUI is a tokio::select! loop that juggles three things at once: keyboard events, answer tokens coming back from a background query, and a timer tick that animates the "thinking" spinner.

pub async fn run(mut self, terminal: &mut DefaultTerminal) -> Result<()> {
    let (tx, mut rx) = mpsc::unbounded_channel::<StreamMsg>();
    let mut events = EventStream::new();
    let mut ticker = tokio::time::interval(Duration::from_millis(120));

    loop {
        terminal.draw(|frame| ui::render(frame, &self))?;
        if self.should_quit { break; }

        tokio::select! {
            maybe_event = events.next() => {
                if let Some(Ok(event)) = maybe_event {
                    self.handle_event(event, &tx);
                }
            }
            Some(msg) = rx.recv() => self.on_stream_msg(msg),
            _ = ticker.tick() => {
                if self.status == Status::Thinking {
                    self.spinner = self.spinner.wrapping_add(1);
                }
            }
        }
    }
    Ok(())
}

When you press Enter, the query runs in a spawned task so the UI never freezes. It posts messages back through a channel — first the sources, then a stream of tokens, then a "done" or "error":

enum StreamMsg { Sources(Vec<RetrievedChunk>), Token(String), Done, Error(String) }

fn submit(&mut self, tx: &mpsc::UnboundedSender<StreamMsg>) {
    let question = self.input.trim().to_string();
    if question.is_empty() || self.status == Status::Thinking { return; }

    self.messages.push(Message { role: Role::You, text: question.clone() });
    self.messages.push(Message { role: Role::Bot, text: String::new() }); // fills via tokens
    self.input.clear();
    self.status = Status::Thinking;

    let engine = self.engine.clone();
    let tx = tx.clone();
    tokio::spawn(async move {
        match engine.retrieve(&question).await {
            Ok(chunks) => {
                let _ = tx.send(StreamMsg::Sources(chunks.clone()));
                let sender = tx.clone();
                let result = engine.answer_streamed(&question, &chunks, |tok| {
                    let _ = sender.send(StreamMsg::Token(tok.to_string()));
                }).await;
                let _ = tx.send(match result {
                    Ok(_) => StreamMsg::Done,
                    Err(e) => StreamMsg::Error(format!("{e:#}")),
                });
            }
            Err(e) => { let _ = tx.send(StreamMsg::Error(format!("{e:#}"))); }
        }
    });
}

The trick that makes this clean is mpsc::unbounded_channel — its send is synchronous, so the on_token closure can fire it without being async. Each token that arrives gets appended to the last (bot) message, and the next terminal.draw shows it. That's the whole streaming effect.

qrag-tui/src/ui.rs — the layout

ui.rs draws three regions: a header (title + a ● ready / ⠹ thinking… 1.4s status), a body split into a Chat panel (left) and a Sources panel (right, showing each chunk's file name, similarity score, and a preview), and an input line at the bottom. Rounded borders, colored role prefixes (you › green, bot › cyan), and a spinner that animates while a query is in flight.

The result looks like this:

╭ qrag-rust  RAG over ./docs   ● ready ─────────────────────╮╭ Sources ────────────╮
│ you › what is tcp                                         ││ ...pdf   0.596       │
│ bot › TCP (Transmission Control Protocol) is a protocol  ││ used for time-sens…  │
│       within the TCP/IP suite that ensures reliable...   ││                      │
│                                                          ││ ...pdf   0.567       │
│ you › what is udp                                        ││ Transmission Contr…  │
│ bot › UDP, or User Datagram Protocol, is a communica...  ││                      │
╰──────────────────────────────────────────────────────────╯╰──────────────────────╯
╭ Ask (Enter to send) ──────────────────────────────────────────────────────────────╮
│ ›                                                                                   │
╰────────────────────────────────────────────────────────────────────────────────────╯

---

▶️ Running it end-to-end

Build everything

git clone https://github.com/Parikalp-Bhardwaj/qrag-rust
cd qrag-rust
cargo build

The first build is slow (tonic, qdrant-client, rig, and ratatui pull a lot of crates). Subsequent builds are quick.

The easy path: the TUI

Make sure Qdrant is up (docker compose up -d) and your .env has a key, then:

cargo run -p qrag-tui

On first run it prints Auto-indexed N chunks from ./docs. and opens the UI. Ask a question and watch the answer stream in, with sources and scores on the right. Keys: Enter send · ↑/↓ scroll · Esc / Ctrl-C quit. No /reindex, no separate server — it just works.

The service path: gRPC server + clients

# terminal 1 — server (auto-indexes on first run)
cargo run -p qrag-server --bin qrag-server

# terminal 2 — chat client
cargo run -p qrag-server --bin chat

Or query directly with grpcurl:

grpcurl -plaintext -d '{"question":"What is tokio?"}' \
  -import-path proto -proto rag.proto \
  127.0.0.1:50051 rag.RagService/AskQuestion

You'll get an answer plus the source chunks that produced it — for example:

{
  "answer": "Tokio is an asynchronous runtime for Rust that enables programs to run many async tasks concurrently... (Source: ./docs/tokio.md)",
  "sources": [
    { "filePath": "./docs/tokio.md", "preview": "# Tokio Runtime...", "score": 0.581 }
  ]
}

Re-indexing after adding documents

Auto-index only runs when the collection is empty, so after dropping new files into ./docs/, rebuild the index with the Reindex RPC:

grpcurl -plaintext -d '{}' \
  -import-path proto -proto rag.proto \
  127.0.0.1:50051 rag.RagService/Reindex

---

⚠️ Gotchas worth knowing

A handful of papercuts you'll save time on:

• Vector dimension mismatch. If you initialize the collection at the wrong dimension, every upsert fails. Drop it (docker compose down -v) and let it recreate. text-embedding-3-small is 1536-dim.
• Auto-index only runs when empty. After adding new docs, call Reindex — a restart alone won't pick them up if the collection already has data.
• PDFs need pdftotext. Without poppler-utils, PDF loading fails and startup auto-index logs a warning.
• streaming needs StreamExt in scope. If .next() on the stream doesn't resolve, you're missing use futures::StreamExt; and the futures dependency.
• Don't commit .env. If you already did, rotate the key — git history is forever.

---

🔭 Where to go from here

The system you just built is a solid, usable baseline. Things to try next, in rough order of payoff:

1. File-watcher. Auto-reindex when files in ./docs change, so you never call Reindex by hand.
2. Better chunking. Sentence-aware splits, sliding-window overlap, or semantic chunking. Probably the single biggest quality lever.
3. Reranking. After Qdrant returns top-20, use a cross-encoder to rerank down to top-3. Higher precision at the cost of latency.
4. Hybrid retrieval. Combine vector search with BM25 keyword search. Catches exact-match queries that pure vector search misses.
5. Per-tenant collections. One Qdrant collection per user or workspace.
6. A desktop app. The engine is already a library — a Tauri front end could sit next to the TUI with no engine changes.

If you want a structured course on RAG fundamentals in Rust — chunking strategies, retrieval evaluation, embedding selection — check out Foundations of RAG Systems with Rust on CodeSignal. Pair it with this post: course for the why, this build for the how.

---

💭 Closing thoughts

Modern AI engineering isn't really about models anymore. The models are a commodity you call over HTTP. The interesting work is everywhere else — splitting documents intelligently, indexing vectors at scale, routing requests with low latency, streaming results without blocking, keeping memory and concurrency under control.

That's systems engineering. And it's exactly the territory Rust is built for. Strong types, async without garbage collection, predictable performance, and a tooling ecosystem (tonic, tokio, rig, qdrant-client, ratatui) that's quietly catching up to anything Python has for AI infrastructure.

Splitting this into a workspace, adding streaming, and building a TUI on top of the same engine drove that home for me: once the core is a clean library, new front ends are cheap. That's the payoff of doing the systems work properly.

You don't need Rust to build RAG. You need Rust when RAG turns into a real system someone depends on.

Build the thing. Read the source. Break it. That's where the understanding comes from. 🦀

---

🙌 Thanks for reading

If you made it this far — seriously, thank you. 🙏

I'd love to hear what you're building, what didn't work for you, or what you'd do differently. Feel free to drop a comment, open an issue on the repo, or reach out on LinkedIn — always happy to talk Rust, AI, RAG, or anything in between.

---

🔗 Links

• 👤 Connect with me on LinkedIn
• 📦 Project repo: github.com/Parikalp-Bhardwaj/qrag-rust
• 📘 Qdrant docs
• 📘 Rig docs
• 📘 Tonic guide
• 📘 ratatui docs
• 📘 OpenRouter

Comments

[Harjot Rana]

Building RAG from internals (rather than importing a framework that hides it) is the best way to actually understand it - and Rust + Qdrant + gRPC is a serious, production-minded stack choice, not a toy. The thing people learn by building RAG by hand is that the magic isn't the vector search; it's everything around retrieval quality: chunking strategy, embedding choice, and re-ranking. A naive RAG retrieves "technically similar" chunks that are useless; a good one retrieves what actually answers the question. Most RAG disappointment is a retrieval-quality problem masquerading as a model problem.

The other lesson the internals teach: retrieval quality is also a cost lever - retrieve the RIGHT 3 chunks instead of 20 mediocre ones and you cut tokens AND improve the answer. Tight, relevant context beats more context, every time. That's the same scoped-context principle I lean on in Moonshift (a multi-agent pipeline that ships a prompt to a deployed SaaS) to keep builds ~$3 flat and high-quality. Excellent deep-dive, Rust is a bold and respectable choice for this. What moved your retrieval quality most - chunking, the embedding model, or adding a re-ranker? That ranking is the practical gold.

[Parikalp Bhardwaj]

Really appreciate this comment, the retrieval quality is a cost lever framing is sharper than how i put it in the post. I had top_k = 3 framed as a coverage/cost tradeoff, but you're right that the better way to see it is, tight relevant retrieval improves quality and lowers cost simultaneously.

Chunking moved the needle the most in my build. I landed on 120 words with split_whitespace() after feeling that bigger chunks got vague and smaller ones fragmented meaning across boundaries. The embedding model (text-embedding-3-small) hasn't felt like a ceiling yet on this corpus. Reranker is next on the list.

[Harjot Rana]

RAG quality lives in retrieval, not the model, hybrid (BM25 + vector) and a reranker move the needle far more than swapping the LLM. Building it in Rust with Qdrant is a nice call for the latency and control. The piece people skip is an eval harness, without it you can't tell whether a chunking or reranker change actually helped vs just felt better. I lean on this ordering in Moonshift, own the parts that decide quality. Have you added reranking on top of the vector search yet, cross-encoder or LLM-as-reranker?