DEV Community

syncore
syncore

Posted on

Build a Fast RAG Pipeline with Claude and Local Embeddings

#ai #claude #python #tutorial

3 min read · 588 words

Retrieval-Augmented Generation (RAG) is the quickest way to ground Claude in your private data without fine-tuning. But running a full RAG pipeline often means wrestling with heavy external vector databases or pricey embedding APIs.

In this article, we’ll build a lean, lightning-fast RAG system using Python, sentence-transformers for local embeddings, and claude-opus-5 for answer generation. No complex cloud infrastructure required—everything runs locally until we query the model.

Prerequisites

You'll need Python 3.10+ and the official Anthropic SDK, along with a few data science libraries for handling local vector search:

pip install anthropic sentence-transformers numpy
Enter fullscreen mode Exit fullscreen mode

Make sure your API key is set in your environment:

export ANTHROPIC_API_KEY="your-api-key-here"
Enter fullscreen mode Exit fullscreen mode

Step 1: Generate Local Embeddings

We'll start by taking a small knowledge base of text snippets, converting them into vector embeddings using a lightweight model from Hugging Face, and storing them in memory.

import numpy as np
from sentence_transformers import SentenceTransformer

# 1. Load a fast, lightweight local embedding model
embedding_model = SentenceTransformer("all-MiniLM-L6-v2")

# 2. Define our local knowledge base
documents = [
    "Project Apollo uses Python 3.11 and FastAPI for the backend services.",
    "The deployment pipeline for staging is triggered automatically via GitHub Actions.",
    "Database migrations are managed using Alembic. Never run raw DDL in production.",
    "To reset the local Redis cache, run 'redis-cli flushall' in your terminal."
]

# 3. Generate embeddings for our documents
doc_embeddings = embedding_model.encode(documents)
Enter fullscreen mode Exit fullscreen mode

Step 2: Retrieve Relevant Context and Query Claude

When a user asks a question, we embed the query, compute cosine similarities against our document embeddings, and pull the top matches. Then, we pass those snippets directly to claude-opus-5.

Remember: current Claude models rely purely on prompt steering (no temperature, top_p, or top_k), and use thinking={"type": "adaptive"} for extended reasoning.

import anthropic

client = anthropic.Anthropic()

def search_docs(query: str, top_k: int = 2):
    # Embed the incoming query
    query_vector = embedding_model.encode(query)

    # Compute cosine similarity
    similarities = np.dot(doc_embeddings, query_vector) / (
        np.linalg.norm(doc_embeddings, axis=1) * np.linalg.norm(query_vector)
    )

    # Get top_k indices sorted by highest similarity
    top_indices = np.argsort(similarities)[::-1][:top_k]
    return [documents[i] for i in top_indices]

def ask_rag(user_query: str):
    # Retrieve relevant context locally
    retrieved_chunks = search_docs(user_query)
    context = "\n---\n".join(retrieved_chunks)

    # Construct the grounded prompt
    prompt = f"""You are a helpful technical assistant. Answer the user's question using ONLY the provided context. If you don't know the answer based on the context, say so.

Context:
{context}

Question:
{user_query}"""

    # Call Claude using the current SDK and API rules
    response = client.messages.create(
        model="claude-opus-5",
        max_tokens=4000,
        thinking={"type": "adaptive"},
        output_config={"effort": "medium"},
        messages=[{"role": "user", "content": prompt}]
    )

    return response.content

# Test the RAG pipeline
answer = ask_rag("How do we handle database migrations?")
print(answer)
Enter fullscreen mode Exit fullscreen mode

Production Considerations

While this in-memory NumPy approach works great for prototypes and small codebases, scaling up involves a few best practices:

  1. Swap to a Vector DB: For thousands or millions of documents, replace the NumPy array with a persistent vector store like Chroma, FAISS, or pgvector.
  2. Chunking Strategy: Instead of whole sentences, split large Markdown or PDF files into 500-token chunks with 50-token overlaps to preserve context.
  3. Model Selection: Use claude-sonnet-5 if you're running high-volume automated Q&A pipelines where throughput is critical.

Conclusion

Building effective RAG doesn't require massive cloud stacks. By combining local embeddings with claude-opus-5, you get a secure, private, and extremely fast question-answering system that runs right inside your application workflow.

Have you built a RAG pipeline with Claude yet? Drop a comment below with your favorite embedding model or questions about the new API parameters!

Top comments (0)