Building a RAG Pipeline from Scratch: Embeddings, Vector Store, and Generation

Large language models know a lot, but not your data. They don't know last week's policy update, your product catalog, or your internal docs, and when asked anyway they may confidently make something up. Retrieval-Augmented Generation (RAG) fixes this by finding the relevant facts first and giving them to the model with the question.

What RAG is and why it matters

RAG combines two steps: retrieve relevant passages from your own data, then generate an answer using those passages as context. It matters because it:

• Grounds answers in real sources, which reduces hallucinations.
• Stays current without retraining: update the index, and answers update too.
• Supports citations, so users can check where an answer came from.
• Respects access control, because you decide which documents each user can retrieve.

The pipeline

1. Document ingestion. Load PDFs, web pages, wiki pages, or database rows. Extract clean text and keep metadata such as title, URL, date, and permissions.
2. Chunking. Split text into passages of roughly 300 to 800 tokens with some overlap (around 10 to 20%). Split on headings and paragraphs where possible rather than at fixed character counts.

1. Embedding. Turn each chunk into a vector with an embedding model, such as text-embedding-3-small. Similar meanings produce nearby vectors.

1. Storing in a vector database. Save each chunk's text, vector, and metadata in a vector store like Azure AI Search, Pinecone, pgvector, or Qdrant.
2. Retrieval. Embed the user's question and find the nearest chunks. Hybrid search (vector plus keyword) usually beats vector search alone.

1. Prompt augmentation. Put the retrieved chunks into the prompt with clear instructions: answer only from the context, and cite sources.
2. Generation. The LLM writes the answer.

The retrieval-and-generation loop

Code example: Azure AI Search + Azure OpenAI

Install the packages:

pip install azure-search-documents openai

1. Create the index

import os
from azure.core.credentials import AzureKeyCredential
from azure.search.documents import SearchClient
from azure.search.documents.indexes import SearchIndexClient
from azure.search.documents.indexes.models import (
    SearchIndex, SimpleField, SearchableField, SearchField, SearchFieldDataType,
    VectorSearch, HnswAlgorithmConfiguration, VectorSearchProfile,
)

endpoint = os.environ["SEARCH_ENDPOINT"]
cred = AzureKeyCredential(os.environ["SEARCH_KEY"])
INDEX = "docs"

index = SearchIndex(
    name=INDEX,
    fields=[
        SimpleField(name="id", type=SearchFieldDataType.String, key=True),
        SearchableField(name="content", type=SearchFieldDataType.String),
        SimpleField(name="source", type=SearchFieldDataType.String),
        SearchField(name="vector", type=SearchFieldDataType.Collection(SearchFieldDataType.Single),
                    searchable=True, vector_search_dimensions=1536,
                    vector_search_profile_name="default"),
    ],
    vector_search=VectorSearch(
        algorithms=[HnswAlgorithmConfiguration(name="hnsw")],
        profiles=[VectorSearchProfile(name="default", algorithm_configuration_name="hnsw")],
    ),
)
SearchIndexClient(endpoint, cred).create_or_update_index(index)

2. Chunk, embed, and upload

from openai import AzureOpenAI

aoai = AzureOpenAI(
    azure_endpoint=os.environ["AOAI_ENDPOINT"],
    api_key=os.environ["AOAI_KEY"],
    api_version="2024-06-01",
)

def chunk(text, size=1500, overlap=200):
    step = size - overlap
    return [text[i:i + size] for i in range(0, len(text), step)]

def embed(texts):
    resp = aoai.embeddings.create(model="text-embedding-3-small", input=texts)
    return [d.embedding for d in resp.data]

search = SearchClient(endpoint, INDEX, cred)

def ingest(doc_id, text, source):
    pieces = chunk(text)
    vectors = embed(pieces)
    search.upload_documents([
        {"id": f"{doc_id}-{i}", "content": p, "source": source, "vector": v}
        for i, (p, v) in enumerate(zip(pieces, vectors))
    ])

3. Retrieve and generate

from azure.search.documents.models import VectorizedQuery

def answer(question, k=5):
    qvec = embed([question])[0]
    results = search.search(
        search_text=question,  # keyword half of hybrid search
        vector_queries=[VectorizedQuery(vector=qvec, k_nearest_neighbors=k, fields="vector")],
        top=k,
    )
    context = "\n\n".join(f"[{r['source']}] {r['content']}" for r in results)

    messages = [
        {"role": "system", "content":
            "Answer using only the context. Cite sources in [brackets]. "
            "If the context does not contain the answer, say you don't know."},
        {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
    ]
    resp = aoai.chat.completions.create(model="gpt-4o-mini", messages=messages, temperature=0)
    return resp.choices[0].message.content

print(answer("What is our refund policy for annual plans?"))

Using Pinecone instead only changes step 1 and the search call: index.upsert(vectors=[...]) to store, and index.query(vector=qvec, top_k=k, include_metadata=True) to retrieve.

Evaluation basics

Measure retrieval and generation separately.

• Retrieval: build a small test set of questions with the chunks that should be found. Track recall@k (did the right chunk appear in the top k?) and MRR (how high did it rank?).
• Generation:
  • Groundedness: is every claim supported by the retrieved context?
  • Answer relevance: does it actually answer the question?
  • Correctness: does it match a reference answer?
• Tooling: frameworks like Ragas or the Azure AI Foundry evaluators use an LLM as a judge for these metrics. Spot-check their scores by hand.
• Regression testing: run the test set every time you change chunking, embeddings, or prompts.

Common pitfalls

• Bad chunking. Chunks that are too small lose context, and chunks that are too big dilute relevance. Respect document structure.
• Vector search only. Exact terms like product codes and names need keyword search too, so use hybrid search plus a reranker.
• Mismatched embedding models. Queries and documents must use the same model. Re-embed everything when you switch.
• Stuffing too much context. More chunks isn't always better; irrelevant text confuses the model and costs tokens.
• No "I don't know" path. Without it, the model fills gaps with guesses.
• Ignoring permissions. Filter by user access at retrieval time, not after generation.
• Stale indexes. Schedule re-ingestion and delete chunks for removed documents.
• No evaluation. Without a test set, every change is a guess.

---

RAG isn't magic. It's a pipeline of ordinary, testable steps: get the text right, chunk it sensibly, retrieve with hybrid search, instruct the model clearly, and measure everything. Start with the small example above, add evaluation early, and improve one stage at a time.

Related reading

• From Requirements Doc to Working Code with Grok: Azure AI Search Integrated with a SQL Database: a hands-on walkthrough of setting up Azure AI Search on top of SQL Server data. Azure AI Search can serve as both the vector store and the retrieval layer in a RAG pipeline like the one in this article.

---

About the author

I write about Applied AI, MCP, Azure AI Search, and RAG. Follow me on Dev.to and Hashnode for more.