DEV Community

Autonomous Studio
Autonomous Studio

Posted on

How to use RAG Production Automation Kit

Introduction: Why Automating Retrieval‑Augmented Generation (RAG) Matters

Building a knowledge‑base assistant often starts with a simple pipeline:

  1. Ingest raw documents (PDFs, markdown, web pages).
  2. Chunk the text and embed each piece.
  3. Store the embeddings in a vector DB.
  4. Retrieve the most relevant chunks at query time.
  5. Prompt a LLM with the retrieved context and the user question.

When you do this manually, each step becomes a source of friction:

  • Inconsistent chunking leads to missed context.
  • Embedding drift occurs when you change models but forget to re‑index.
  • Vector‑DB hygiene (duplicate vectors, stale metadata) quickly degrades recall.
  • Prompt engineering is repeated across projects, making it hard to share best practices.

The RAG Production Automation Kit (RAG‑PAK) solves these pain points by providing a ready‑to‑run, opinionated framework that wires together ingestion, embedding, storage, retrieval, and prompting with minimal code. The kit is built on open‑source components (LangChain, FAISS/PGVector, OpenAI/Claude APIs) but abstracts the boilerplate into reusable utilities.

Below is a step‑by‑step guide to get a knowledge‑base assistant up and running in under an hour.


Prerequisites

Requirement Recommended Version
Python 3.9 – 3.12
Virtual env venv or conda
LLM API key OpenAI (OPENAI_API_KEY) or Anthropic (ANTHROPIC_API_KEY)
Vector DB FAISS (local) or PostgreSQL + PGVector (remote)
Git To clone the kit repo

Tip: The kit works out‑of‑the‑box with FAISS for quick prototyping. Switch to PGVector when you need persistence across restarts.


Step 1: Set Up the Project

# 1️⃣ Clone the repo
git clone https://github.com/3km-collab/rag-production-automation-kit.git
cd rag-production-automation-kit

# 2️⃣ Create a clean environment
python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate

# 3️⃣ Install dependencies
pip install -r requirements.txt
Enter fullscreen mode Exit fullscreen mode

The requirements.txt pins the exact versions used by the kit, ensuring reproducibility.


Step 2: Prepare Your Knowledge Sources

Place every document you want the assistant to know inside the data/ folder. The kit supports:

  • Plain text (.txt)
  • Markdown (.md)
  • PDF (.pdf) – parsed with pdfplumber
  • HTML (.html) – stripped of tags automatically
mkdir -p data
cp ~/my‑docs/*.pdf data/
cp ~/my‑docs/*.md  data/
Enter fullscreen mode Exit fullscreen mode

The ingestion script will walk the directory recursively, so you can organize files in subfolders if you wish.


Step 3: Run the Automated Ingestion Pipeline

python scripts/run_ingest.py \
  --source-dir data \
  --embedding-model openai:text-embedding-ada-002 \
  --vector-store faiss \
  --chunk-size 500 \
  --overlap 50
Enter fullscreen mode Exit fullscreen mode

What happens under the hood?

Stage Kit Component Default Behaviour
Chunking Chunker Splits on sentence boundaries, 500‑token windows with 50‑token overlap.
Embedding Embedder Calls the selected OpenAI embedding endpoint; caches results in cache/.
Vector Store VectorStoreFactory Instantiates a FAISS index (index.faiss) and persists it to store/.
Metadata MetadataEnricher Adds source file path, page number (for PDFs), and a UUID to each vector.

The script is idempotent: running it again will only embed new or changed files, thanks to a SHA‑256 checksum stored alongside each vector.


Step 4: Define a Retrieval‑Augmented Prompt Template

Create a file prompts/assistant.txt with the following content (feel free to tweak):

You are a helpful assistant with access to a corporate knowledge base.
Use ONLY the provided context to answer the user's question.
If the answer is not in the context, reply with "I don't have enough information."

Context:
{retrieved_chunks}

Question:
{question}
Enter fullscreen mode Exit fullscreen mode

The kit will load this template and substitute {retrieved_chunks} with the top‑k results from the vector store.


Step 5: Spin Up the Query Service

python scripts/run_query.py \
  --vector-store faiss \
  --top-k 5 \
  --prompt-file prompts/assistant.txt \
  --llm openai:gpt-4o-mini
Enter fullscreen mode Exit fullscreen mode

The script launches a tiny FastAPI server on http://localhost:8000. You can query it with curl or any HTTP client:

curl -X POST http://localhost:8000/query \
  -H "Content-Type: application/json" \
  -d '{"question": "What is our refund policy for enterprise customers?"}'
Enter fullscreen mode Exit fullscreen mode

Response example

{
  "answer": "Enterprise customers are eligible for a 30‑day refundable period after purchase, provided they have not exceeded usage limits.",
  "sources": [
    "data/policies/refund_policy.pdf (page 3)",
    "data/terms/enterprise_terms.md (line 42‑48)"
  ]
}
Enter fullscreen mode Exit fullscreen mode

The server automatically logs the raw LLM output, the retrieved chunks, and latency metrics to logs/, giving you observability without extra instrumentation.


Step 6: Add Continuous Updating (Optional)

For production use you often need to ingest new documents nightly. The kit ships a watchdog utility that monitors the data/ folder and triggers incremental re‑indexing:

python scripts/watchdog.py \
  --source-dir data \
  --interval 300   # check every 5 minutes
Enter fullscreen mode Exit fullscreen mode

Combine this with a CI/CD step that restarts the FastAPI service after each successful re‑index, ensuring the assistant always serves the latest knowledge.


Step 7: Test and Iterate

  1. Unit test the retrieval – Use scripts/run_test_retrieval.py to verify that a known query returns the expected chunk.
  2. Prompt tuning – Adjust top-k, chunk size, or the prompt template based on observed answer quality.
  3. Scale the vector store – When you exceed a few hundred thousand vectors, switch to PGVector by changing --vector-store pgvector and providing a Postgres connection string.

Recap

Goal How RAG‑PAK Helps
Fast onboarding One‑line commands for ingestion and serving.
Consistent chunking Centralized Chunker with configurable size/overlap.
Embedding cache Avoids duplicate API calls, saving cost.
Observability Built‑in logging of retrieval, LLM response, and latency.
Extensibility Swap vector DB, LLM, or prompt without touching core code.

With these steps you have a production‑ready RAG pipeline that can be dropped into any AI startup or enterprise project.


Next Steps

Happy building!


Autonomous Studio — Self-hosted AI Micro-APIs & Developer Tools.

Get the full package · Live demo

Top comments (0)