Building Retrieval-Augmented Generation (RAG) Systems with LangChain and Pinecone

While LLMs are great, there are some limitations in using LLMs: LLMs can hallucinate, presenting factually incorrect information when they don't know the answers, and their knowledge gets frozen at the time of training. That's when Retrieval Augmented Generation (RAG) addresses both of these problems. It is the process of optimizing the output of the LLM.

This article walks through what RAG is, why it matters, and how to build a working RAG pipeline using two of the most popular tools in the space: LangChain, a framework for building LLM-powered applications, and Pinecone, a managed vector database designed for fast similarity search at scale.

A typical RAG pipeline has three core steps:

1. Retrieve: When a query is entered, the system searches an external data source (like a vector database) for the most relevant documents.
2. Augment: The system attaches those relevant retrieved documents to the original user prompt.
3. Generate: The LLM reads the appended context and formulates a highly accurate, grounded answer.

RAG is popular because it solves practical problems that pure fine-tuning or prompting can't easily solve:

• Freshness — You can update the knowledge base without retraining the model.
• Domain specificity — You can ground responses in your company's internal documents, product manuals, or proprietary data.
• Traceability — Because answers are based on retrieved documents, you can cite sources and reduce hallucination.
• Cost — Retrieval is far cheaper than fine-tuning a model every time your data changes.

Why LangChain and Pinecone?

LangChain drastically speeds up AI development. It is an open-source orchestration framework that provides pre-built components to connect Large Language Models (LLMs) to external data, manage memory, and create multi-step workflows. It abstracts away the complex boilerplate usually required to build production-ready AI applications.

Pinecone is a purpose-built vector database. Once your documents are converted into embeddings (numerical vectors that capture semantic meaning), Pinecone stores them and lets you perform fast, approximate nearest-neighbor search — finding the vectors most similar to a query vector, even across millions of records, with low latency.

Together, they form a clean, production-ready stack: LangChain handles orchestration and LLM calls, Pinecone handles storage and retrieval.

Building a RAG Pipeline Step by Step

1. Install Dependencies

pip install langchain langchain-openai langchain-pinecone pinecone-client

2. Set Up Pinecone

First, create a Pinecone account and get an API key. Then initialize an index — the container that holds your vectors.

from pinecone import Pinecone, ServerlessSpec

pc = Pinecone(api_key="YOUR_PINECONE_API_KEY")

index_name = "rag-demo"

# Create the index if it doesn't already exist
if index_name not in pc.list_indexes().names():
    pc.create_index(
        name=index_name,
        dimension=1536,  # must match your embedding model's output size
        metric="cosine",
        spec=ServerlessSpec(cloud="aws", region="us-east-1")
    )

index = pc.Index(index_name)

The dimension parameter must match the size of the embeddings you plan to store — for example, OpenAI's text-embedding-3-small outputs 1536-dimensional vectors.

3. Load and Split Your Documents

Raw documents are usually too long to embed as single chunks, so LangChain provides text splitters that break them into manageable, semantically coherent pieces.

from langchain_community.document_loaders import DirectoryLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter

loader = DirectoryLoader("./docs", glob="**/*.txt")
documents = loader.load()

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200
)
chunks = splitter.split_documents(documents)

The chunk_overlap ensures that context isn't awkwardly cut off between chunks, which helps preserve meaning across chunk boundaries.

4. Embed and Upload to Pinecone

LangChain's Pinecone integration handles both embedding generation and upsertion into the index in a single call.

from langchain_openai import OpenAIEmbeddings
from langchain_pinecone import PineconeVectorStore

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

vector_store = PineconeVectorStore.from_documents(
    documents=chunks,
    embedding=embeddings,
    index_name=index_name
)

Each chunk of text is converted into a vector and stored in Pinecone alongside metadata (like the source filename), which makes it possible to trace an answer back to its origin later.

5. Build the Retriever

Once your data is indexed, you can turn the vector store into a retriever that fetches the top-k most relevant chunks for any query.

retriever = vector_store.as_retriever(
    search_type="similarity",
    search_kwargs={"k": 4}
)

6. Wire Up the LLM and Prompt

Now combine retrieval with generation. LangChain's create_retrieval_chain (or the older RetrievalQA chain) makes this straightforward.

from langchain_openai import ChatOpenAI
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain.chains import create_retrieval_chain
from langchain_core.prompts import ChatPromptTemplate

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

prompt = ChatPromptTemplate.from_template("""
Answer the question based only on the following context.
If the answer isn't in the context, say you don't know.

Context:
{context}

Question: {input}
""")

document_chain = create_stuff_documents_chain(llm, prompt)
rag_chain = create_retrieval_chain(retriever, document_chain)

7. Query the System

response = rag_chain.invoke({"input": "What does our refund policy cover?"})
print(response["answer"])

Behind the scenes, LangChain has: embedded the query, sent it to Pinecone to retrieve the top matching chunks, inserted those chunks into the prompt template, and passed the whole thing to the LLM for a grounded answer.

Design Considerations for Production RAG

A working prototype is easy to build, but a few decisions matter a lot once you move toward production:

• Chunking strategy — Too small, and chunks lose context; too large, and retrieval becomes imprecise. Experiment with chunk size and overlap for your specific document types.
• Metadata filtering — Pinecone supports filtering by metadata (e.g., document type, date, department), which lets you narrow retrieval before similarity search runs, improving both speed and relevance.
• Hybrid search — Combining semantic (vector) search with traditional keyword search often improves retrieval quality, especially for queries containing exact terms, product codes, or names.
• Re-ranking — Adding a re-ranking step after initial retrieval (using a cross-encoder model) can significantly improve the quality of the final context passed to the LLM.
• Evaluation — Track retrieval precision/recall and answer faithfulness over time. Tools like RAGAS or custom LLM-based evaluators can help catch regressions as your document set evolves.
• Index maintenance — Set up a pipeline to re-embed and update Pinecone entries when source documents change, so the index doesn't go stale.

Conclusion

RAG has become one of the most practical patterns for making LLMs useful with real, current, and domain-specific information. LangChain and Pinecone form a natural pairing for this: LangChain handles the orchestration of loading, splitting, embedding, and prompting, while Pinecone provides fast, scalable vector search to ground responses in relevant retrieved context. The pipeline above is a solid starting point — from there, the real engineering work lies in tuning chunking, retrieval, and evaluation to fit your specific data and use case.