vmodal_ai

Posted on Aug 11

Building a RAG Chatbot with Flutter and FastAPI

Retrieval-Augmented Generation (RAG) combines a language model with a searchable knowledge base.

Instead of asking an LLM to answer only from its training knowledge, a RAG system retrieves relevant documents and includes them as context.

A typical architecture is:

             ┌─────────────────┐
             │   Flutter App   │
             └────────┬────────┘
                      │
                    HTTPS
                      │
             ┌────────▼────────┐
             │    FastAPI      │
             └────────┬────────┘
                      │
             ┌────────▼────────┐
             │ Retriever / DB  │
             └────────┬────────┘
                      │
                Relevant chunks
                      │
             ┌────────▼────────┐
             │       LLM       │
             └────────┬────────┘
                      │
                    Answer
                      │
             ┌────────▼────────┐
             │   Flutter UI    │
             └─────────────────┘

What is RAG?

RAG usually contains two pipelines.

Indexing pipeline

Documents
   ↓
Text extraction
   ↓
Chunking
   ↓
Embeddings
   ↓
Vector database

Query pipeline

User question
   ↓
Query embedding
   ↓
Similarity search
   ↓
Relevant chunks
   ↓
Prompt + context
   ↓
LLM
   ↓
Answer

Step 1: Install FastAPI

pip install fastapi uvicorn pydantic

You will also need an embedding model and vector database.

Step 2: Create a FastAPI endpoint

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class ChatRequest(BaseModel):
    question: str

@app.post("/chat")
async def chat(request: ChatRequest):
    # 1. Create query embedding
    # 2. Search vector database
    # 3. Build context
    # 4. Call the LLM
    # 5. Return answer

    return {
        "answer": "Example response",
        "sources": []
    }

Step 3: Chunk documents

Large documents should generally be divided into smaller chunks.

Document
   ↓
Chunk 1
Chunk 2
Chunk 3
...
Chunk N

A chunk should be large enough to preserve meaning but small enough to retrieve precisely.

Experiment with:

chunk size
overlap
retrieval count
metadata filtering

Step 4: Generate embeddings

An embedding converts text into a vector:

"How do I reset my password?"
             ↓
[0.12, -0.31, 0.87, ...]

Semantically similar sentences should have vectors that are close in the embedding space.

A local model such as nomic-embed-text can be used through an appropriate local inference stack.

Step 5: Store vectors

A vector database stores:

ID
Embedding
Text chunk
Metadata

Metadata might contain:

{
  "document": "employee_handbook.pdf",
  "page": 14,
  "department": "HR"
}

Metadata enables filtered retrieval.

Step 6: Retrieve relevant chunks

For:

What is the company's annual leave policy?

retrieve the most relevant chunks.

Conceptually:

results = vector_store.search(
    query_embedding,
    top_k=5
)

The exact API depends on your vector database.

Step 7: Build the prompt

A RAG prompt might look like:

You are a helpful assistant.

Answer the user's question using only the supplied context.

Context:
{{retrieved_documents}}

Question:
{{user_question}}

If the answer is not present in the context,
say that you do not have enough information.

This reduces unsupported answers.

Step 8: Connect Flutter

Create a simple API client:

import 'dart:convert';
import 'package:http/http.dart' as http;

class RagApi {
  final String baseUrl;

  RagApi(this.baseUrl);

  Future<RagResponse> ask(String question) async {
    final response = await http.post(
      Uri.parse('$baseUrl/chat'),
      headers: {
        'Content-Type': 'application/json',
      },
      body: jsonEncode({
        'question': question,
      }),
    );

    if (response.statusCode != 200) {
      throw Exception('RAG request failed');
    }

    final json = jsonDecode(response.body);

    return RagResponse.fromJson(json);
  }
}

class RagResponse {
  final String answer;
  final List<dynamic> sources;

  RagResponse({
    required this.answer,
    required this.sources,
  });

  factory RagResponse.fromJson(Map<String, dynamic> json) {
    return RagResponse(
      answer: json['answer'] ?? '',
      sources: json['sources'] ?? [],
    );
  }
}

Display citations

Return source information:

{
  "answer": "Employees receive ...",
  "sources": [
    {
      "document": "employee_handbook.pdf",
      "page": 14
    }
  ]
}

Flutter can display:

Answer

Employees receive ...

Sources
• employee_handbook.pdf — page 14

This improves user trust and makes answers easier to verify.

Add conversation history

A chatbot normally needs conversation context:

User: What is the refund policy?
AI: The refund period is 30 days.

User: What about digital products?

The second question depends on the first.

A production system can maintain conversation history while still using retrieval for factual context.

RAG quality problems

A chatbot can fail even when the LLM is good.

Bad document extraction
        ↓
Bad chunks
        ↓
Bad embeddings
        ↓
Bad retrieval
        ↓
Wrong context
        ↓
Wrong answer

Evaluate retrieval independently from generation.

Improve retrieval

Useful techniques include:

metadata filtering
hybrid search
reranking
query rewriting
better chunking
multiple retrieval queries
domain-specific embeddings

Production architecture

Flutter
  |
Authentication
  |
FastAPI
  |
Conversation Service
  |
Retriever
  |
Vector DB
  |
Reranker
  |
Prompt Builder
  |
LLM
  |
Response + Sources

Security

The backend should enforce:

authentication
authorization
request limits
input validation
document access controls
logging
tenant isolation when required

If documents belong to different users or organizations, retrieval must enforce permissions before returning context to the LLM.

Conclusion

RAG is more than connecting a vector database to an LLM. Quality depends on the entire pipeline: document extraction, chunking, embeddings, retrieval, prompt construction, generation, and source presentation.

Flutter provides the user experience, while FastAPI provides a clean backend layer for retrieval and AI orchestration.