Introduction: Why Automating Retrieval‑Augmented Generation (RAG) Matters
Building a knowledge‑base assistant often starts with a simple pipeline:
- Ingest raw documents (PDFs, markdown, web pages).
- Chunk the text and embed each piece.
- Store the embeddings in a vector DB.
- Retrieve the most relevant chunks at query time.
- Prompt a LLM with the retrieved context and the user question.
When you do this manually, each step becomes a source of friction:
- Inconsistent chunking leads to missed context.
- Embedding drift occurs when you change models but forget to re‑index.
- Vector‑DB hygiene (duplicate vectors, stale metadata) quickly degrades recall.
- Prompt engineering is repeated across projects, making it hard to share best practices.
The RAG Production Automation Kit (RAG‑PAK) solves these pain points by providing a ready‑to‑run, opinionated framework that wires together ingestion, embedding, storage, retrieval, and prompting with minimal code. The kit is built on open‑source components (LangChain, FAISS/PGVector, OpenAI/Claude APIs) but abstracts the boilerplate into reusable utilities.
Below is a step‑by‑step guide to get a knowledge‑base assistant up and running in under an hour.
Prerequisites
| Requirement | Recommended Version |
|---|---|
| Python | 3.9 – 3.12 |
| Virtual env |
venv or conda
|
| LLM API key | OpenAI (OPENAI_API_KEY) or Anthropic (ANTHROPIC_API_KEY) |
| Vector DB | FAISS (local) or PostgreSQL + PGVector (remote) |
| Git | To clone the kit repo |
Tip: The kit works out‑of‑the‑box with FAISS for quick prototyping. Switch to PGVector when you need persistence across restarts.
Step 1: Set Up the Project
# 1️⃣ Clone the repo
git clone https://github.com/3km-collab/rag-production-automation-kit.git
cd rag-production-automation-kit
# 2️⃣ Create a clean environment
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
# 3️⃣ Install dependencies
pip install -r requirements.txt
The requirements.txt pins the exact versions used by the kit, ensuring reproducibility.
Step 2: Prepare Your Knowledge Sources
Place every document you want the assistant to know inside the data/ folder. The kit supports:
-
Plain text (
.txt) -
Markdown (
.md) -
PDF (
.pdf) – parsed withpdfplumber -
HTML (
.html) – stripped of tags automatically
mkdir -p data
cp ~/my‑docs/*.pdf data/
cp ~/my‑docs/*.md data/
The ingestion script will walk the directory recursively, so you can organize files in subfolders if you wish.
Step 3: Run the Automated Ingestion Pipeline
python scripts/run_ingest.py \
--source-dir data \
--embedding-model openai:text-embedding-ada-002 \
--vector-store faiss \
--chunk-size 500 \
--overlap 50
What happens under the hood?
| Stage | Kit Component | Default Behaviour |
|---|---|---|
| Chunking | Chunker |
Splits on sentence boundaries, 500‑token windows with 50‑token overlap. |
| Embedding | Embedder |
Calls the selected OpenAI embedding endpoint; caches results in cache/. |
| Vector Store | VectorStoreFactory |
Instantiates a FAISS index (index.faiss) and persists it to store/. |
| Metadata | MetadataEnricher |
Adds source file path, page number (for PDFs), and a UUID to each vector. |
The script is idempotent: running it again will only embed new or changed files, thanks to a SHA‑256 checksum stored alongside each vector.
Step 4: Define a Retrieval‑Augmented Prompt Template
Create a file prompts/assistant.txt with the following content (feel free to tweak):
You are a helpful assistant with access to a corporate knowledge base.
Use ONLY the provided context to answer the user's question.
If the answer is not in the context, reply with "I don't have enough information."
Context:
{retrieved_chunks}
Question:
{question}
The kit will load this template and substitute {retrieved_chunks} with the top‑k results from the vector store.
Step 5: Spin Up the Query Service
python scripts/run_query.py \
--vector-store faiss \
--top-k 5 \
--prompt-file prompts/assistant.txt \
--llm openai:gpt-4o-mini
The script launches a tiny FastAPI server on http://localhost:8000. You can query it with curl or any HTTP client:
curl -X POST http://localhost:8000/query \
-H "Content-Type: application/json" \
-d '{"question": "What is our refund policy for enterprise customers?"}'
Response example
{
"answer": "Enterprise customers are eligible for a 30‑day refundable period after purchase, provided they have not exceeded usage limits.",
"sources": [
"data/policies/refund_policy.pdf (page 3)",
"data/terms/enterprise_terms.md (line 42‑48)"
]
}
The server automatically logs the raw LLM output, the retrieved chunks, and latency metrics to logs/, giving you observability without extra instrumentation.
Step 6: Add Continuous Updating (Optional)
For production use you often need to ingest new documents nightly. The kit ships a watchdog utility that monitors the data/ folder and triggers incremental re‑indexing:
python scripts/watchdog.py \
--source-dir data \
--interval 300 # check every 5 minutes
Combine this with a CI/CD step that restarts the FastAPI service after each successful re‑index, ensuring the assistant always serves the latest knowledge.
Step 7: Test and Iterate
-
Unit test the retrieval – Use
scripts/run_test_retrieval.pyto verify that a known query returns the expected chunk. -
Prompt tuning – Adjust
top-k, chunk size, or the prompt template based on observed answer quality. -
Scale the vector store – When you exceed a few hundred thousand vectors, switch to PGVector by changing
--vector-store pgvectorand providing a Postgres connection string.
Recap
| Goal | How RAG‑PAK Helps |
|---|---|
| Fast onboarding | One‑line commands for ingestion and serving. |
| Consistent chunking | Centralized Chunker with configurable size/overlap. |
| Embedding cache | Avoids duplicate API calls, saving cost. |
| Observability | Built‑in logging of retrieval, LLM response, and latency. |
| Extensibility | Swap vector DB, LLM, or prompt without touching core code. |
With these steps you have a production‑ready RAG pipeline that can be dropped into any AI startup or enterprise project.
Next Steps
- Explore the live demo to see the kit in action: https://3km-collab.github.io/rag-production-automation-kit-12/
- Grab the full package (including advanced monitoring hooks and Docker support) here: https://autonomousstudio.gumroad.com/l/xzxbdn
Happy building!
Autonomous Studio — Self-hosted AI Micro-APIs & Developer Tools.
Top comments (0)