A Real RAG Pipeline on AWS: Internal Docs Q&A with Terraform (Model-Agnostic) 🔍

Every company has the same problem - support agents and employees can't find answers buried in internal docs. Here's a simple, production-ready RAG pipeline on Bedrock Knowledge Base, built to swap embedding and generation models with a one-line tfvars change.

Almost every company hits this same wall: product docs, FAQs, and policy PDFs pile up in a shared drive, support agents can't find the right answer fast enough, and employees ping the same Slack channel with the same questions every week. This is the single most common RAG use case in the industry - internal knowledge Q&A.

This post builds that exact pipeline: documents in S3, a Bedrock Knowledge Base for retrieval, and an assistant that answers questions grounded in your docs with citations. The entire thing is Terraform, and every model reference is a variable - upgrading your embedding model or swapping your generation model is a tfvars change, not a rewrite. 🎯

🏗️ The Architecture

Docs (PDF/HTML/TXT) → S3 bucket
        ↓
Bedrock Knowledge Base (chunking + embedding)
        ↓
OpenSearch Serverless (vector storage)
        ↓
RetrieveAndGenerate API → grounded answer + citations

Bedrock Knowledge Bases handle chunking, embedding generation, and retrieval internally. You don't write ingestion code. You point it at S3 and it does the rest.

🔧 Terraform: The Full Pipeline

Variables - Model-Agnostic by Design

# variables.tf

variable "embedding_model" {
  description = "Embedding model ID. Change this to upgrade embeddings."
  type = object({
    id         = string
    dimensions = number
  })
  default = {
    id         = "amazon.titan-embed-text-v2:0"
    dimensions = 1024
  }
}

variable "generation_model_id" {
  description = "Foundation model for RetrieveAndGenerate. Change to upgrade."
  type        = string
  default     = "anthropic.claude-3-5-sonnet-20241022-v2:0"
}

variable "vector_index_name" {
  type    = string
  default = "support-docs-index"
}

This is the whole trick. Every downstream resource references var.embedding_model.id and var.generation_model_id. Bump the version in tfvars, run terraform apply, done.

S3 Bucket for Source Documents

# storage.tf

resource "aws_s3_bucket" "docs" {
  bucket = "${var.environment}-support-docs-${data.aws_caller_identity.current.account_id}"
}

resource "aws_s3_bucket_versioning" "docs" {
  bucket = aws_s3_bucket.docs.id
  versioning_configuration {
    status = "Enabled"
  }
}

OpenSearch Serverless (Vector Store)

# opensearch.tf

resource "aws_opensearchserverless_security_policy" "encryption" {
  name = "${var.environment}-kb-encryption"
  type = "encryption"
  policy = jsonencode({
    Rules = [{
      ResourceType = "collection"
      Resource     = ["collection/${var.environment}-support-kb"]
    }]
    AWSOwnedKey = true
  })
}

resource "aws_opensearchserverless_security_policy" "network" {
  name = "${var.environment}-kb-network"
  type = "network"
  policy = jsonencode([{
    Rules = [{
      ResourceType = "collection"
      Resource     = ["collection/${var.environment}-support-kb"]
    }]
    AllowFromPublic = true
  }])
}

resource "aws_opensearchserverless_collection" "kb" {
  name       = "${var.environment}-support-kb"
  type       = "VECTORSEARCH"
  depends_on = [aws_opensearchserverless_security_policy.encryption]
}

resource "aws_opensearchserverless_access_policy" "kb" {
  name = "${var.environment}-kb-access"
  type = "data"
  policy = jsonencode([{
    Rules = [
      {
        ResourceType = "collection"
        Resource     = ["collection/${var.environment}-support-kb"]
        Permission   = ["aoss:*"]
      },
      {
        ResourceType = "index"
        Resource     = ["index/${var.environment}-support-kb/*"]
        Permission   = ["aoss:*"]
      }
    ]
    Principal = [aws_iam_role.knowledge_base.arn]
  }])
}

IAM Role

# iam.tf

resource "aws_iam_role" "knowledge_base" {
  name = "${var.environment}-support-kb-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action    = "sts:AssumeRole"
      Effect    = "Allow"
      Principal = { Service = "bedrock.amazonaws.com" }
    }]
  })
}

resource "aws_iam_role_policy" "kb_permissions" {
  name = "kb-permissions"
  role = aws_iam_role.knowledge_base.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["s3:GetObject", "s3:ListBucket"]
        Resource = [aws_s3_bucket.docs.arn, "${aws_s3_bucket.docs.arn}/*"]
      },
      {
        Effect   = "Allow"
        Action   = ["bedrock:InvokeModel"]
        Resource = "arn:aws:bedrock:${var.region}::foundation-model/${var.embedding_model.id}"
      },
      {
        Effect   = "Allow"
        Action   = ["aoss:APIAccessAll"]
        Resource = aws_opensearchserverless_collection.kb.arn
      }
    ]
  })
}

Knowledge Base (Model-Agnostic Core)

# knowledge_base.tf

resource "aws_bedrockagent_knowledge_base" "support" {
  name     = "${var.environment}-support-docs"
  role_arn = aws_iam_role.knowledge_base.arn

  knowledge_base_configuration {
    type = "VECTOR"
    vector_knowledge_base_configuration {
      embedding_model_arn = "arn:aws:bedrock:${var.region}::foundation-model/${var.embedding_model.id}"

      embedding_model_configuration {
        bedrock_embedding_model_configuration {
          dimensions           = var.embedding_model.dimensions
          embedding_data_type  = "FLOAT32"
        }
      }
    }
  }

  storage_configuration {
    type = "OPENSEARCH_SERVERLESS"
    opensearch_serverless_configuration {
      collection_arn    = aws_opensearchserverless_collection.kb.arn
      vector_index_name = var.vector_index_name
      field_mapping {
        vector_field   = "bedrock-knowledge-base-default-vector"
        text_field     = "AMAZON_BEDROCK_TEXT_CHUNK"
        metadata_field = "AMAZON_BEDROCK_METADATA"
      }
    }
  }

  depends_on = [aws_opensearchserverless_access_policy.kb]
}

resource "aws_bedrockagent_data_source" "docs" {
  name              = "${var.environment}-support-docs-source"
  knowledge_base_id = aws_bedrockagent_knowledge_base.support.id

  data_source_configuration {
    type = "S3"
    s3_configuration {
      bucket_arn = aws_s3_bucket.docs.arn
    }
  }

  vector_ingestion_configuration {
    chunking_configuration {
      chunking_strategy = "FIXED_SIZE"
      fixed_size_chunking_configuration {
        max_tokens         = var.chunk_max_tokens
        overlap_percentage = var.chunk_overlap_percentage
      }
    }
  }
}

Notice embedding_model_arn is built entirely from var.embedding_model.id. Switching from Titan Embed v2 to Cohere Embed, or to a future Titan v3, means changing one variable. No resource restructuring.

📐 Environment Configuration

# environments/dev.tfvars
environment          = "dev"
embedding_model = {
  id         = "amazon.titan-embed-text-v2:0"
  dimensions = 1024
}
generation_model_id  = "anthropic.claude-3-5-haiku-20241022-v1:0"
chunk_max_tokens     = 300
chunk_overlap_percentage = 20

# environments/prod.tfvars
environment          = "prod"
embedding_model = {
  id         = "amazon.titan-embed-text-v2:0"
  dimensions = 1024
}
generation_model_id  = "anthropic.claude-3-5-sonnet-20241022-v2:0"
chunk_max_tokens     = 300
chunk_overlap_percentage = 20

To upgrade later: change embedding_model.id to "cohere.embed-english-v3" or a newer Titan version, run terraform apply. Bedrock re-embeds on the next data source sync. Same pattern for generation_model_id - swap Haiku for Sonnet, or Sonnet for a future model, with zero code changes.

🐍 Query the Pipeline

import boto3

client = boto3.client("bedrock-agent-runtime")

response = client.retrieve_and_generate(
    input={"text": "What is our refund policy for annual plans?"},
    retrieveAndGenerateConfiguration={
        "type": "KNOWLEDGE_BASE",
        "knowledgeBaseConfiguration": {
            "knowledgeBaseId": "KB_ID",
            "modelArn": "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-5-sonnet-20241022-v2:0",
        },
    },
)

print(response["output"]["text"])
for citation in response["citations"]:
    for ref in citation["retrievedReferences"]:
        print(f"Source: {ref['location']['s3Location']['uri']}")

🧪 Small Evaluation: Did the Upgrade Actually Help?

Before rolling a model upgrade to prod, run a quick regression check. This is intentionally lightweight - a fixed set of representative questions with known-good answers, scored on retrieval hit rate and answer relevance:

# evaluate.py
import boto3

client = boto3.client("bedrock-agent-runtime")

# Representative Q&A pairs pulled from real support tickets
eval_set = [
    {
        "question": "What is our refund policy for annual plans?",
        "expected_keywords": ["30 days", "annual", "refund", "prorate"],
    },
    {
        "question": "How do I reset a customer's 2FA?",
        "expected_keywords": ["admin panel", "2FA", "reset", "support ticket"],
    },
    {
        "question": "What's the SLA for enterprise tier support?",
        "expected_keywords": ["1 hour", "enterprise", "SLA", "priority"],
    },
]

def run_eval(kb_id: str, model_arn: str) -> dict:
    hits, total_keyword_matches = 0, 0
    for case in eval_set:
        response = client.retrieve_and_generate(
            input={"text": case["question"]},
            retrieveAndGenerateConfiguration={
                "type": "KNOWLEDGE_BASE",
                "knowledgeBaseConfiguration": {
                    "knowledgeBaseId": kb_id,
                    "modelArn": model_arn,
                },
            },
        )
        answer = response["output"]["text"].lower()
        matched = sum(1 for kw in case["expected_keywords"] if kw.lower() in answer)
        total_keyword_matches += matched
        if response["citations"] and response["citations"][0]["retrievedReferences"]:
            hits += 1

    return {
        "retrieval_hit_rate": hits / len(eval_set),
        "avg_keyword_coverage": total_keyword_matches / len(eval_set),
    }

# Compare old vs new model
old_results = run_eval("KB_ID", "arn:...claude-3-5-haiku-20241022-v1:0")
new_results = run_eval("KB_ID", "arn:...claude-3-5-sonnet-20241022-v2:0")

print(f"Old model: {old_results}")
print(f"New model: {new_results}")

Run this after every model swap. If retrieval_hit_rate drops, your embedding model change hurt retrieval quality - investigate before rolling to prod. If avg_keyword_coverage improves, the new generation model produces more complete answers. Ten to twenty real questions from actual support tickets is enough to catch regressions; you don't need a large eval framework for this kind of sanity check.

⚠️ Gotchas and Tips

Re-sync after model changes. Changing embedding_model requires re-ingesting your data source (start-ingestion-job) since old vectors used the previous model's dimensions.

Match dimensions to the model. Titan v2 outputs 1024 dimensions. Cohere Embed v3 outputs 1024 or 384 depending on config. Mismatched dimensions cause ingestion failures - check your model's docs before switching.

OpenSearch Serverless takes 5-10 minutes to provision. Budget for this in your first terraform apply.

Keep the eval set small and real. Pull actual questions from support tickets or Slack, not synthetic ones. Ten good questions beat a hundred made-up ones.

---

A model-agnostic RAG pipeline that answers real support questions, grounded in your docs, with a built-in way to check that your next model upgrade actually improves things. All in Terraform. 🔍

Found this helpful? Follow for the GCP and Azure versions coming next! 💬