DEV Community

cheng zhang
cheng zhang

Posted on

How to Build Your Own RAG Knowledge Base Q&A System with DeepSeek

#ai #productivity #tutorial #beginners

Category: AI Application Engineering / RAG Knowledge Base Q&A

Target readers: backend developers, AI full-stack developers, enterprise digital teams, product managers, technical leads, and anyone who wants to turn company documents into an AI Q&A assistant

Test date: July 12, 2026

Bottom line: The recommended first version of a DeepSeek-based RAG system is: document parsing → text chunking → embeddings → vector retrieval → DeepSeek answer generation → source citations. Do not start with a complex agent. First build a minimal loop that answers accurately from your documents, refuses unsupported questions, and returns citations.

1. What Problem Does RAG Solve?

RAG means Retrieval-Augmented Generation.

A normal LLM has one obvious limitation:

It does not know your company policies, product manuals, support rules, project files, contract templates, implementation guides, or internal knowledge base.

If you directly ask a model, it may:

  • hallucinate answers;
  • return outdated information;
  • miss internal company knowledge;
  • fail to cite sources;
  • guess about policies, prices, contracts, and workflows.

RAG works like this:

User question
→ retrieve relevant materials from your knowledge base
→ provide those materials as context to the model
→ ask the model to answer only from that context
→ return answer + citations
Enter fullscreen mode Exit fullscreen mode

In one sentence:

RAG makes the model answer while looking at your documents, not from vague memory.

2. Why Use DeepSeek for RAG?

DeepSeek API has three practical advantages.

2.1 OpenAI-compatible integration

DeepSeek API documentation says the API is compatible with OpenAI / Anthropic API formats. With the OpenAI SDK, you mainly need to change base_url and api_key.

This is developer-friendly because you can reuse existing code, frameworks, and tooling.

2.2 Good cost profile for Q&A

RAG systems often involve repeated questions and context assembly. Model cost directly affects whether the system can run long term. DeepSeek’s model and pricing page lists the API Base URL as:

https://api.deepseek.com
Enter fullscreen mode Exit fullscreen mode

For knowledge-base Q&A, use cost-efficient models for normal questions and stronger reasoning models for complex cases.

2.3 Long context helps document Q&A

DeepSeek’s V4 Preview announcement says DeepSeek-V4-Pro and DeepSeek-V4-Flash support 1M context length and Thinking / Non-Thinking modes. Long context is useful for manuals, policies, contracts, and long documents.

But remember:

Long context does not eliminate the need for RAG.

You should not put every document into the prompt. It is expensive, slow, hard to cite, and less controllable. Retrieve first, generate second.

3. Standard RAG Architecture

A minimal RAG system usually contains eight modules:

Module Role Common tools
Document collection Upload PDFs, Word, Markdown, web pages, Excel upload service, crawler, enterprise docs
Document parsing Convert files into clean text pdfplumber, unstructured, docx, pandas
Text chunking Split long documents into chunks LangChain, LlamaIndex, custom splitter
Embedding Convert text into vectors BGE, Jina, Qwen Embedding, OpenAI-compatible services
Vector store Store and search vectors FAISS, Milvus, pgvector, Qdrant, Chroma
Retriever Find relevant chunks for a question top-k, hybrid search, reranking
Generator Use DeepSeek to answer from context DeepSeek Chat Completion API
Citation and evaluation Return sources, logs, feedback metadata, logs, evaluation sets

Workflow:

Upload documents
→ parse documents
→ clean text
→ split into chunks
→ embed chunks
→ write into vector store
→ user asks a question
→ embed the question
→ retrieve relevant chunks
→ build RAG prompt
→ DeepSeek generates answer
→ return answer, citations, and confidence notes
Enter fullscreen mode Exit fullscreen mode

4. Recommended Tech Stack for Version 1

Do not start with a complicated microservice system.

Minimal version

Module Recommendation
Backend Python + FastAPI
LLM DeepSeek API
Document format Markdown / TXT / PDF
Chunking custom splitter or LangChain TextSplitter
Embedding BGE / Jina / Qwen Embedding / OpenAI-compatible embedding service
Vector store Local FAISS
Frontend Streamlit / React / Next.js
Deployment Docker + cloud server

Enterprise upgrade

Module Recommendation
Backend FastAPI / Node.js / Java Spring Boot
Queue Redis Queue / Celery
Document storage MinIO / OSS / S3
Vector store Milvus / pgvector / Qdrant
Permission RBAC + department permissions + document permissions
Logs PostgreSQL + ClickHouse / Loki
Evaluation golden question set + human feedback + hit-rate metrics
Deployment Kubernetes / Docker Compose

Do not start with

  • complex agents;
  • multi-step tool calling;
  • automatic policy rewriting;
  • automatic approval flows;
  • legal, financial, medical, or contract Q&A without human review;
  • direct write access to production databases.

Version 1 goal:

When a user asks a question, the system finds evidence and answers from that evidence.

5. Test Tasks and Scoring

Test tasks

Task Content Goal
Task 1 Upload 20 product docs Test parsing and chunking
Task 2 Upload FAQ Test common Q&A accuracy
Task 3 Upload policy docs Test clause retrieval and citation
Task 4 Ask cross-document questions Test multi-chunk synthesis
Task 5 Ask unsupported questions Test refusal and anti-hallucination
Task 6 Ask procedural questions Test structured answers
Task 7 Ask mixed Chinese-English terms Test terminology handling
Task 8 Concurrent Q&A Test latency and cost

Scoring criteria

Total score: 100.

Dimension Weight What it measures
Integration difficulty 10 API and framework integration
Retrieval accuracy 20 Whether the correct chunks are found
Answer reliability 20 Whether answers are grounded and non-hallucinated
Source citation 15 File name, page, section, chunk source
Cost control 15 Model and retrieval cost
Response speed 10 User experience
Scalability 10 Upgrade path to enterprise system

Overall score

Dimension Score Notes
Integration difficulty 90/100 OpenAI-compatible and easy to adopt
Retrieval accuracy 84/100 Depends on chunking, embeddings, and reranking
Answer reliability 88/100 Stable with strict prompts
Source citation 86/100 Requires metadata design
Cost control 91/100 Suitable for medium/high-frequency Q&A
Response speed 85/100 Depends on model, top-k, and context size
Scalability 87/100 Can upgrade from FAISS to Milvus/pgvector

Overall score: 87 / 100

Verdict:

DeepSeek is a strong generation layer for RAG, but RAG quality is determined by the whole pipeline: documents, chunking, embeddings, retrieval, reranking, prompts, and evaluation.

6. Project Structure

Suggested structure:

deepseek-rag-demo/
├── app.py                  # FastAPI main app
├── config.py               # configuration
├── requirements.txt        # dependencies
├── .env                    # API key, never commit
├── data/
│   ├── raw/                # raw documents
│   └── processed/          # parsed text
├── vector_store/
│   └── faiss_index/        # FAISS index
├── rag/
│   ├── loader.py           # document loading
│   ├── splitter.py         # text chunking
│   ├── embeddings.py       # embedding calls
│   ├── retriever.py        # retrieval logic
│   ├── prompt.py           # prompt templates
│   └── generator.py        # DeepSeek generation
└── tests/
    └── eval_questions.json # test questions
Enter fullscreen mode Exit fullscreen mode

7. Install Dependencies

requirements.txt example:

fastapi==0.115.0
uvicorn==0.30.6
python-dotenv==1.0.1
openai==1.99.0
faiss-cpu==1.8.0
sentence-transformers==3.0.1
pydantic==2.8.2
numpy==1.26.4
pypdf==4.3.1
python-multipart==0.0.9
Enter fullscreen mode Exit fullscreen mode

Install:

pip install -r requirements.txt
Enter fullscreen mode Exit fullscreen mode

.env:

DEEPSEEK_API_KEY=your_deepseek_api_key
DEEPSEEK_BASE_URL=https://api.deepseek.com
DEEPSEEK_MODEL=deepseek-v4-flash
Enter fullscreen mode Exit fullscreen mode

If model names change, follow DeepSeek’s official model and pricing page.

8. Call DeepSeek API

Because DeepSeek is OpenAI-compatible, you can call it with the OpenAI SDK:

from openai import OpenAI
import os

client = OpenAI(
    api_key=os.getenv("DEEPSEEK_API_KEY"),
    base_url="https://api.deepseek.com"
)

def ask_deepseek(messages, model="deepseek-v4-flash"):
    response = client.chat.completions.create(
        model=model,
        messages=messages,
        temperature=0.2,
    )
    return response.choices[0].message.content
Enter fullscreen mode Exit fullscreen mode

RAG recommendations:

Parameter Advice
temperature 0–0.3 to reduce hallucination
top_p default unless randomness needs reduction
max_tokens limit by answer length
thinking mode use for complex reasoning, not every FAQ
model choice cheap model for normal Q&A, strong model for complex questions

9. Parse Documents

Start with TXT / Markdown / PDF.

from pathlib import Path
from pypdf import PdfReader


def load_txt(path: str) -> str:
    return Path(path).read_text(encoding="utf-8")


def load_pdf(path: str) -> str:
    reader = PdfReader(path)
    pages = []
    for i, page in enumerate(reader.pages):
        text = page.extract_text() or ""
        pages.append(f"\n[PAGE {i + 1}]\n{text}")
    return "\n".join(pages)


def load_document(path: str) -> str:
    if path.endswith(".pdf"):
        return load_pdf(path)
    if path.endswith(".txt") or path.endswith(".md"):
        return load_txt(path)
    raise ValueError("Unsupported file type")
Enter fullscreen mode Exit fullscreen mode

Enterprise parsing is harder:

  • scanned PDFs need OCR;
  • tables need structure preservation;
  • Word files need heading hierarchy;
  • Excel needs Sheet/row/column handling;
  • web pages need navigation and footer cleanup;
  • policy docs need section numbers.

10. Text Chunking Strategy

Chunking is critical.

If chunks are too large:

  • retrieval is less precise;
  • context cost is high;
  • answers contain irrelevant material.

If chunks are too small:

  • semantics are incomplete;
  • clause context is lost;
  • the model needs too many chunks.

Recommended chunk settings

Document type Chunk size Overlap
FAQ one Q&A per chunk 0
Product docs 500–800 Chinese characters 80–150
Policy clauses by chapter/clause keep parent title
Course notes 800–1200 Chinese characters 150–200
API docs by endpoint/method keep parameter docs

Simple splitter:

def split_text(text: str, chunk_size=800, overlap=120):
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        chunks.append(chunk)
        start = end - overlap
    return chunks
Enter fullscreen mode Exit fullscreen mode

Better method:

split by heading hierarchy
→ then by paragraphs
→ then by length fallback
Enter fullscreen mode Exit fullscreen mode

Store metadata for every chunk:

{
  "source": "support_policy.pdf",
  "page": 3,
  "section": "2.3 Return rules",
  "chunk_id": "policy_003_002"
}
Enter fullscreen mode Exit fullscreen mode

11. Embeddings and Vector Store

Embedding converts text into vectors. A vector store finds similar vectors.

LlamaIndex explains that RAG converts data and queries into embeddings, then a vector store finds data numerically similar to the query embedding. FAISS documentation describes Faiss as a library for efficient similarity search and clustering of dense vectors.

For version 1, use a local embedding model + FAISS:

from sentence_transformers import SentenceTransformer
import numpy as np
import faiss

embed_model = SentenceTransformer("BAAI/bge-small-zh-v1.5")


def embed_texts(texts):
    vectors = embed_model.encode(texts, normalize_embeddings=True)
    return np.array(vectors).astype("float32")


def build_faiss_index(chunks):
    vectors = embed_texts(chunks)
    dim = vectors.shape[1]
    index = faiss.IndexFlatIP(dim)
    index.add(vectors)
    return index, vectors
Enter fullscreen mode Exit fullscreen mode

Search:

def search(query, index, chunks, metadatas, top_k=5):
    q_vec = embed_texts([query])
    scores, ids = index.search(q_vec, top_k)
    results = []
    for score, idx in zip(scores[0], ids[0]):
        results.append({
            "score": float(score),
            "text": chunks[idx],
            "metadata": metadatas[idx]
        })
    return results
Enter fullscreen mode Exit fullscreen mode

Vector store choices

Stage Recommended
Local demo FAISS
Single-machine product Chroma / Qdrant
Enterprise system Milvus / pgvector / Qdrant
Strong relational permissions PostgreSQL + pgvector
Large-scale retrieval Milvus / Qdrant

12. Build the RAG Prompt

The prompt must constrain the model:

  • answer only from context;
  • say unknown when unsupported;
  • cite sources;
  • do not invent policies, prices, contracts, or workflows;
  • separate evidence from inference.

Template:

RAG_SYSTEM_PROMPT = """
You are an enterprise knowledge-base Q&A assistant.
You must answer strictly based on the provided [Reference Materials].

Rules:
1. If the answer is not in the reference materials, say: "The current knowledge base does not contain a clear answer."
2. Do not invent policies, prices, workflows, contract terms, contacts, or commitments.
3. Give the conclusion first, then the evidence.
4. If sources conflict, point out the conflict instead of deciding by yourself.
5. Cite each key conclusion with source markers such as [Source 1].
"""


def build_rag_messages(question, retrieved_docs):
    context_parts = []
    for i, doc in enumerate(retrieved_docs, start=1):
        meta = doc["metadata"]
        context_parts.append(
            f"[Source {i}] File: {meta.get('source')}, Page: {meta.get('page')}, Section: {meta.get('section')}\n{doc['text']}"
        )

    context = "\n\n".join(context_parts)

    user_prompt = f"""
[Reference Materials]
{context}

[User Question]
{question}

Please answer based on the reference materials.
"""

    return [
        {"role": "system", "content": RAG_SYSTEM_PROMPT},
        {"role": "user", "content": user_prompt}
    ]
Enter fullscreen mode Exit fullscreen mode

13. Full Q&A Chain 

def rag_answer(question, index, chunks, metadatas):
    retrieved = search(
        query=question,
        index=index,
        chunks=chunks,
        metadatas=metadatas,
        top_k=5
    )

    messages = build_rag_messages(question, retrieved)
    answer = ask_deepseek(messages)

    sources = [doc["metadata"] for doc in retrieved]

    return {
        "answer": answer,
        "sources": sources,
        "retrieved": retrieved
    }
Enter fullscreen mode Exit fullscreen mode

Example response:

{
  "answer": "According to the knowledge base, the standard support response time is within 24 hours. [Source 1] Hardware repair requires an SN code and fault photos first. [Source 2]",
  "sources": [
    {"source": "support_policy.pdf", "page": 3, "section": "2.1 Response time"},
    {"source": "hardware_repair.md", "page": 1, "section": "Required materials"}
  ]
}
Enter fullscreen mode Exit fullscreen mode

14. FastAPI Endpoint Example 

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI(title="DeepSeek RAG Knowledge Base")

class AskRequest(BaseModel):
    question: str

@app.post("/ask")
def ask(req: AskRequest):
    result = rag_answer(
        question=req.question,
        index=GLOBAL_INDEX,
        chunks=GLOBAL_CHUNKS,
        metadatas=GLOBAL_METADATAS
    )
    return result
Enter fullscreen mode Exit fullscreen mode

Run:

uvicorn app:app --host 0.0.0.0 --port 8000
Enter fullscreen mode Exit fullscreen mode

Request:

curl -X POST http://localhost:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"question":"What is the support response time?"}'
Enter fullscreen mode Exit fullscreen mode

15. Frontend Suggestions

Version 1 needs four areas:

Area Function
Left document panel upload, delete, update documents
Center chat window user question and AI answer
Right citation panel matched documents, pages, chunks
Bottom feedback bar useful / not useful / wrong answer

Enterprise version also needs:

  • login;
  • department permissions;
  • document permissions;
  • Q&A logs;
  • sensitive-word filtering;
  • human handoff;
  • question review;
  • knowledge update reminders.

16. How to Reduce Hallucination

The main issue is not whether the system can answer. It is whether it refuses when it should.

16.1 Prompt constraint

Use:

If the answer is not in the materials, say: "The current knowledge base does not contain a clear answer."
Enter fullscreen mode Exit fullscreen mode

16.2 Retrieval score threshold

If the best similarity score is too low, do not force an answer.

if retrieved[0]["score"] < 0.35:
    return {
        "answer": "The current knowledge base does not contain a clear answer. Please contact human support.",
        "sources": []
    }
Enter fullscreen mode Exit fullscreen mode

16.3 Require citations

Every key conclusion must cite a source.

16.4 Use low temperature

RAG Q&A usually does not need creativity.

temperature=0.2
Enter fullscreen mode Exit fullscreen mode

16.5 Block high-risk overreach

For contracts, prices, legal, financial, HR, and medical questions, add human fallback.

17. How to Improve Retrieval Accuracy

17.1 Optimize chunking

Split semantically, not just by character count.

17.2 Keep heading hierarchy

Example:

Product Manual > Account Management > Password Reset > Forgot Password
Enter fullscreen mode Exit fullscreen mode

17.3 Add metadata

Use:

  • file name;
  • page;
  • section;
  • document type;
  • effective date;
  • department;
  • permission level.

17.4 Use hybrid retrieval

Vector search handles semantic similarity. Keyword search handles exact terms.

Enterprise recommendation:

vector search + BM25 keyword search + reranking
Enter fullscreen mode Exit fullscreen mode

17.5 Build an evaluation set

Prepare 100–300 real questions:

Field Example
question How many days do customers have to request a return?
expected_source support_policy.pdf page 3
expected_answer within 7 days
category support

Run evaluation after every major update.

18. Cost Optimization

18.1 Embed documents only once

If documents do not change, do not re-embed them.

18.2 Cache frequent questions

Many FAQ questions repeat.

user question → normalize → check cache → run RAG only if cache misses
Enter fullscreen mode Exit fullscreen mode

18.3 Control top-k

More retrieved chunks are not always better.

Scenario Suggested top-k
FAQ 3
Product docs 5
Policy Q&A 5–8
Cross-document synthesis 8–12

18.4 Model routing

Question type Strategy
Simple FAQ fast low-cost model
Complex policy interpretation stronger model / thinking mode
unsupported question rules + retrieval threshold
summary mid-tier model

18.5 Limit context length

Only provide truly relevant chunks to the model.

19. Enterprise Considerations

19.1 Permissions

Different departments may access different documents.

Examples:

  • presales sees product materials;
  • support sees troubleshooting docs;
  • finance sees pricing policy;
  • normal employees cannot see contract templates.

Filter by permissions before retrieval.

19.2 Document versions

Old documents can pollute answers.

Add metadata:

{
  "version": "2026.07",
  "effective_date": "2026-07-01",
  "status": "active"
}
Enter fullscreen mode Exit fullscreen mode

19.3 Logs and audit

Log:

  • user question;
  • retrieved documents;
  • model answer;
  • model used;
  • token cost;
  • user feedback.

19.4 Human fallback

For low-confidence questions:

The current knowledge base does not contain a clear answer. Please contact human support.
Enter fullscreen mode Exit fullscreen mode

19.5 Sensitive information

Do not send these directly to external APIs:

  • ID numbers;
  • phone numbers;
  • full contracts;
  • customer privacy data;
  • internal credentials;
  • raw sensitive data.

20. Upgrade Path from Demo to Production

Stage 1: Local demo 

TXT / PDF → FAISS → DeepSeek → simple Q&A page
Enter fullscreen mode Exit fullscreen mode

Goal: validate the RAG loop.

Stage 2: Department knowledge base 

document management → permissions → vector store → citations → feedback
Enter fullscreen mode Exit fullscreen mode

Goal: let one department use it.

Stage 3: Enterprise knowledge base 

multi-department permissions → document versions → hybrid retrieval → evaluation set → audit logs
Enter fullscreen mode Exit fullscreen mode

Goal: stable internal Q&A.

Stage 4: Business agent 

RAG Q&A → tool calling → ticket system → CRM → ERP → human fallback
Enter fullscreen mode Exit fullscreen mode

Goal: move from answering questions to handling business tasks.

21. Common Mistakes

Mistake 1: Calling DeepSeek without retrieval

That is a chatbot, not RAG.

Mistake 2: Putting full documents into the prompt

It is expensive, slow, and hard to cite.

Mistake 3: Random chunking

Chunking determines much of RAG quality.

Mistake 4: No citations

Enterprise users need evidence, not just answers.

Mistake 5: No refusal mechanism

The system should say unknown when it does not know.

Mistake 6: No evaluation set

Without evaluation, you cannot tell whether updates improve or break the system.

Mistake 7: Ignoring permissions

Unauthorized answers create major risk.

22. Final Verdict

Building a RAG knowledge-base Q&A system with DeepSeek is not just calling a model API. The real system is a reliable pipeline:

documents → chunks → embeddings → retrieval → generation → citations → feedback → evaluation
Enter fullscreen mode Exit fullscreen mode

DeepSeek is suitable as the generation layer, especially for cost-sensitive Chinese Q&A, long-document Q&A, and enterprise knowledge-base scenarios.

But final quality depends on the whole engineering pipeline:

  • clean documents;
  • reasonable chunking;
  • matching embeddings;
  • accurate retrieval;
  • strict prompt constraints;
  • clear citations;
  • evaluation and feedback.

Final recommendation:

Do not start with a complex agent. First build a minimal RAG system with DeepSeek + FAISS + FastAPI that answers accurately from documents, returns citations, and refuses unsupported questions. Then upgrade to Milvus/pgvector, hybrid retrieval, permissions, logs, evaluation, and business-system integration.

23. SEO Information

SEO title: How to Build Your Own RAG Knowledge Base Q&A System with DeepSeek

SEO description: This guide explains how to build a DeepSeek-based RAG knowledge-base Q&A system, covering RAG architecture, document parsing, chunking, embeddings, FAISS vector search, DeepSeek API calls, prompt templates, FastAPI, hallucination control, cost optimization, permissions, and enterprise deployment.

Keywords: DeepSeek, DeepSeek API, RAG, knowledge base Q&A, vector database, FAISS, Embedding, FastAPI, LangChain, LlamaIndex, enterprise knowledge base, AI Q&A system, retrieval augmented generation

24. Data Sources and References

  1. DeepSeek API Docs: DeepSeek API is compatible with OpenAI / Anthropic API formats and can be used through SDK configuration changes.

    https://api-docs.deepseek.com/

  2. DeepSeek Models & Pricing: DeepSeek API Base URL, models, pricing, and Thinking / Non-Thinking mode information.

    https://api-docs.deepseek.com/quick_start/pricing

  3. DeepSeek Create Chat Completion: Chat Completion API, thinking, reasoning_effort, max_tokens, and related parameters.

    https://api-docs.deepseek.com/api/create-chat-completion

  4. DeepSeek V4 Preview Release: DeepSeek-V4-Pro / V4-Flash, 1M context length, and Thinking / Non-Thinking modes.

    https://api-docs.deepseek.com/news/news260424

  5. LlamaIndex RAG Introduction: embeddings, vector stores, and retrieval basics in RAG.

    https://developers.llamaindex.ai/python/framework/understanding/rag/

  6. LlamaIndex Vector Stores: vector stores contain embeddings of ingested document chunks and can be persisted.

    https://developers.llamaindex.ai/python/framework/module_guides/storing/vector_stores/

  7. FAISS documentation: Faiss is a library for efficient similarity search and clustering of dense vectors.

    https://faiss.ai/index.html

  8. LangChain GitHub: LangChain is a framework for building agents and LLM-powered applications.

    https://github.com/langchain-ai/langchain

Publish-ready Summary

This guide explains how to build a DeepSeek-based RAG knowledge-base Q&A system. It starts with the principle of RAG and explains why enterprise document questions should not be answered by the model alone. The article walks through document parsing, text chunking, embeddings, vector search, RAG prompt construction, DeepSeek API calls, FastAPI endpoints, citations, hallucination control, retrieval optimization, cost reduction, permissions, logging, and the upgrade path from local demo to enterprise deployment. The final recommendation is to first build a minimal RAG system with DeepSeek + FAISS + FastAPI, then upgrade to hybrid retrieval, reranking, permissions, evaluation, and business-system integration.

Originally published on Zyentor Picks.

Top comments (0)