A Real RAG Pipeline on Azure: Internal Docs Q&A with Terraform (Model-Agnostic) 🔍

The same internal-docs Q&A problem every company has, solved on Azure AI Search and Azure OpenAI. Terraform provisions everything, model deployments are one-line swaps in tfvars, and a small evaluation catches regressions before they hit prod.

Same problem as every company: product docs, FAQs, and policy PDFs pile up in a shared drive, support agents can't find the right answer fast enough, and the same questions land in the same Slack channel every week. This post builds the fix - internal knowledge Q&A - on Azure AI Search paired with Azure OpenAI.

Azure's pattern connects three pieces in one call: Blob Storage holds your documents, Azure AI Search indexes them with vector and keyword search, and Azure OpenAI's chat completions API with a data source parameter handles retrieval, reranking, and generation together. Terraform provisions the infrastructure. Model deployments are variables, so upgrading either the embedding model or the generation model is a tfvars change, not a rewrite. 🎯

🏗️ The Architecture

Docs (PDF/HTML/TXT) → Blob Storage
        ↓
Azure AI Search (integrated vectorization + indexer)
        ↓
Vector + keyword index (hybrid search)
        ↓
Azure OpenAI "On Your Data" → grounded answer + citations

Azure AI Search's integrated vectorization means the indexer calls the embedding model for you during ingestion, and again at query time for the search text. You don't write embedding code.

🔧 Terraform: The Full Pipeline

Model Deployments - Model-Agnostic by Design

# variables.tf

variable "embedding_model" {
  description = "Embedding model deployment. Change this to upgrade embeddings."
  type = object({
    name       = string
    version    = string
    dimensions = number
  })
  default = {
    name       = "text-embedding-3-large"
    version    = "1"
    dimensions = 1024
  }
}

variable "generation_model" {
  description = "Chat model deployment. Change to upgrade generation quality."
  type = object({
    name    = string
    version = string
  })
  default = {
    name    = "gpt-4.1"
    version = "2025-04-14"
  }
}

Azure OpenAI Deployments

# openai.tf

resource "azurerm_cognitive_account" "openai" {
  name                = "${var.environment}-support-openai"
  resource_group_name = azurerm_resource_group.this.name
  location            = var.location
  kind                = "OpenAI"
  sku_name            = "S0"
}

resource "azurerm_cognitive_deployment" "embedding" {
  name                 = "embedding-${var.embedding_model.name}"
  cognitive_account_id = azurerm_cognitive_account.openai.id

  model {
    format  = "OpenAI"
    name    = var.embedding_model.name
    version = var.embedding_model.version
  }

  sku {
    name     = "Standard"
    capacity = var.embedding_tpm
  }
}

resource "azurerm_cognitive_deployment" "generation" {
  name                 = "chat-${var.generation_model.name}"
  cognitive_account_id = azurerm_cognitive_account.openai.id

  model {
    format  = "OpenAI"
    name    = var.generation_model.name
    version = var.generation_model.version
  }

  sku {
    name     = "GlobalStandard"
    capacity = var.generation_tpm
  }
}

Storage for Source Documents

# storage.tf

resource "azurerm_storage_account" "docs" {
  name                     = "${var.environment}supportdocs"
  resource_group_name     = azurerm_resource_group.this.name
  location                = var.location
  account_tier            = "Standard"
  account_replication_type = "LRS"
  min_tls_version         = "TLS1_2"
}

resource "azurerm_storage_container" "documents" {
  name                  = "documents"
  storage_account_id    = azurerm_storage_account.docs.id
  container_access_type = "private"
}

Azure AI Search Service

# search.tf

resource "azurerm_search_service" "this" {
  name                = "${var.environment}-support-search"
  resource_group_name = azurerm_resource_group.this.name
  location            = var.location
  sku                 = var.search_sku

  identity {
    type = "SystemAssigned"
  }
}

Config Bridge - Terraform to SDK

# config.tf

resource "local_file" "rag_config" {
  filename = "${path.module}/app/rag_config.json"
  content = jsonencode({
    search_endpoint      = "https://${azurerm_search_service.this.name}.search.windows.net"
    openai_endpoint      = azurerm_cognitive_account.openai.endpoint
    embedding_deployment = azurerm_cognitive_deployment.embedding.name
    generation_deployment = azurerm_cognitive_deployment.generation.name
    embedding_dimensions = var.embedding_model.dimensions
    index_name           = "support-docs-${var.embedding_model.name}-${var.embedding_model.dimensions}"
    storage_container    = azurerm_storage_container.documents.name
    storage_account      = azurerm_storage_account.docs.name
  })
}

Same trick as AWS and GCP. The index name is derived from the embedding model and its dimensions. Change the model, get a new index name automatically, without touching a single downstream resource.

🐍 Index, Skillset, and Indexer (SDK)

The azurerm provider doesn't manage index schemas, skillsets, or indexers directly - those are data-plane objects created via the SDK or REST API:

# app/setup_index.py
import json
from azure.search.documents.indexes import SearchIndexClient
from azure.search.documents.indexes.models import (
    SearchIndex, SearchField, SearchFieldDataType,
    VectorSearch, HnswAlgorithmConfiguration, VectorSearchProfile,
    AzureOpenAIVectorizer, AzureOpenAIVectorizerParameters,
)
from azure.identity import DefaultAzureCredential

with open("rag_config.json") as f:
    config = json.load(f)

credential = DefaultAzureCredential()
index_client = SearchIndexClient(config["search_endpoint"], credential)

vector_search = VectorSearch(
    algorithms=[HnswAlgorithmConfiguration(name="hnsw-config")],
    profiles=[VectorSearchProfile(
        name="vector-profile",
        algorithm_configuration_name="hnsw-config",
        vectorizer_name="openai-vectorizer",
    )],
    vectorizers=[AzureOpenAIVectorizer(
        vectorizer_name="openai-vectorizer",
        parameters=AzureOpenAIVectorizerParameters(
            resource_url=config["openai_endpoint"],
            deployment_name=config["embedding_deployment"],
            model_name=config["embedding_deployment"],
        ),
    )],
)

fields = [
    SearchField(name="id", type=SearchFieldDataType.String, key=True),
    SearchField(name="content", type=SearchFieldDataType.String, searchable=True),
    SearchField(
        name="contentVector",
        type=SearchFieldDataType.Collection(SearchFieldDataType.Single),
        searchable=True,
        vector_search_dimensions=config["embedding_dimensions"],
        vector_search_profile_name="vector-profile",
    ),
]

index = SearchIndex(name=config["index_name"], fields=fields, vector_search=vector_search)
index_client.create_or_update_index(index)
print(f"Index ready: {config['index_name']}")

Indexer setup (pulls from Blob Storage, chunks, and vectorizes automatically) follows the same pattern using SearchIndexerClient with a skillset that references the vectorizer. The index name carries the embedding model version, so this script is safe to re-run whenever you upgrade.

🐍 Query the Pipeline (Model-Agnostic Generation)

# app/query.py
from openai import AzureOpenAI
import json

with open("rag_config.json") as f:
    config = json.load(f)

client = AzureOpenAI(
    azure_endpoint=config["openai_endpoint"],
    azure_ad_token_provider=get_bearer_token_provider(),
    api_version="2025-01-01-preview",
)

def ask(question: str) -> dict:
    response = client.chat.completions.create(
        model=config["generation_deployment"],
        messages=[{"role": "user", "content": question}],
        extra_body={
            "data_sources": [{
                "type": "azure_search",
                "parameters": {
                    "endpoint": config["search_endpoint"],
                    "index_name": config["index_name"],
                    "query_type": "vector_semantic_hybrid",
                    "embedding_dependency": {
                        "type": "deployment_name",
                        "deployment_name": config["embedding_deployment"],
                    },
                },
            }]
        },
    )
    return response.choices[0].message.content

Swap generation_model.name from gpt-4.1 to gpt-4.1-mini or a future model in tfvars, terraform apply, done. No code changes. The data_sources parameter handles intent extraction, hybrid retrieval, semantic reranking, and generation in one call.

📐 Environment Configuration

# environments/dev.tfvars
environment      = "dev"
search_sku       = "basic"
embedding_model  = { name = "text-embedding-3-large", version = "1", dimensions = 1024 }
generation_model = { name = "gpt-4.1-mini", version = "2025-04-14" }
embedding_tpm    = 20
generation_tpm   = 10

# environments/prod.tfvars
environment      = "prod"
search_sku       = "standard"
embedding_model  = { name = "text-embedding-3-large", version = "1", dimensions = 1024 }
generation_model = { name = "gpt-4.1", version = "2025-04-14" }
embedding_tpm    = 50
generation_tpm   = 40

⚠️ The Embedding Dimension Honesty Check

Same reality as the other two clouds: a vector field's dimension count is fixed when the field is created. You cannot change vector_search_dimensions on an existing field, and you generally shouldn't mix embeddings from different models in one field even if the dimension count happens to match, since the vector spaces aren't comparable.

The index_name in the config already bakes in the embedding model and dimension count, so the upgrade pattern is the same one used for GCP:

1. Change embedding_model in tfvars
2. terraform apply - computes a new index_name, doesn't touch the running index
3. Re-run setup_index.py - creates the new index and indexer, re-vectorizes from the same Blob container
4. Run the evaluation below against both indexes
5. Point query.py at the new index once it wins, then delete the old one

🧪 Small Evaluation: Did the Upgrade Actually Help?

# evaluate.py
from app.query import ask

eval_set = [
    {"question": "What is our refund policy for annual plans?",
     "expected_keywords": ["30 days", "annual", "refund", "prorate"]},
    {"question": "How do I reset a customer's 2FA?",
     "expected_keywords": ["admin panel", "2FA", "reset", "support ticket"]},
    {"question": "What's the SLA for enterprise tier support?",
     "expected_keywords": ["1 hour", "enterprise", "SLA", "priority"]},
]

def run_eval() -> dict:
    hits, total_keyword_matches = 0, 0
    for case in eval_set:
        answer = ask(case["question"])
        if answer:
            hits += 1
        matched = sum(1 for kw in case["expected_keywords"] if kw.lower() in answer.lower())
        total_keyword_matches += matched

    return {
        "answer_rate": hits / len(eval_set),
        "avg_keyword_coverage": total_keyword_matches / len(eval_set),
    }

# Point rag_config.json at the old index/model, run, then repoint and run again
print(run_eval())

Point the config at the old index and model first, capture the scores, then repoint at the new one and compare. Ten to twenty real questions from actual support tickets is enough to catch a regression before it reaches production.

⚠️ Gotchas and Tips

Data-plane objects live outside Terraform state. Index schemas, skillsets, and indexers are managed by the SDK, not azurerm. Terraform owns the durable infrastructure; the SDK owns content and schema.

Semantic ranking needs Standard tier. The basic SKU works for dev but semantic reranking requires standard and an explicit semantic configuration - budget for this in prod tfvars.

Shared private links only work Azure-to-Azure. If your embedding model runs outside Azure OpenAI, the indexer connection has to go over the public internet.

Re-run the indexer after content changes, not just model changes. New documents in Blob Storage don't appear in the index until the indexer runs again, either on its schedule or triggered manually.

---

A model-agnostic RAG pipeline for real internal questions, with an honest process for upgrading embeddings and a built-in way to check that the upgrade actually helped. Terraform for the infrastructure, SDK for the index, evaluation before every cutover. 🔍

Found this helpful? Follow for more RAG deep-dives! 💬