A Real RAG Pipeline on GCP: Internal Docs Q&A with Terraform (Model-Agnostic) 🔍

The same internal-docs Q&A problem every company has, solved on Vertex AI RAG Engine. Terraform provisions the infrastructure, the SDK manages the corpus, and the design keeps your generation model swappable and your embedding upgrades painless.

Same problem as every company: product docs, FAQs, and policy PDFs pile up, support agents can't find the right answer fast enough, and the same questions land in the same Slack channel every week. This post builds the fix - internal knowledge Q&A - on Vertex AI RAG Engine, GCP's managed RAG service. RAG Engine handles chunking, embedding, and retrieval internally with a managed vector database.

Terraform provisions the infrastructure that changes rarely: APIs, the GCS bucket, IAM, and the RAG Engine's database tier. The Python SDK manages the corpus and file operations - the things that change often. The design keeps your generation model swappable with a one-line change, and gives you an honest, workable pattern for upgrading embedding models too. 🎯

🏗️ The Architecture

Docs (PDF/HTML/TXT) → GCS bucket
        ↓
RAG Engine Corpus (chunking + embedding)
        ↓
RagManagedDb (vector storage)
        ↓
retrieveContexts + generateContent → grounded answer

🔧 Terraform: Infrastructure Layer

APIs and Config

# rag/main.tf

resource "google_project_service" "required" {
  for_each = toset([
    "aiplatform.googleapis.com",
    "storage.googleapis.com",
  ])
  project = var.project_id
  service = each.value
}

resource "google_vertex_ai_rag_engine_config" "this" {
  region = var.region

  rag_managed_db_config {
    dynamic "basic" {
      for_each = var.rag_db_tier == "basic" ? [1] : []
      content {}
    }
    dynamic "scaled" {
      for_each = var.rag_db_tier == "scaled" ? [1] : []
      content {}
    }
  }

  depends_on = [google_project_service.required]
}

RAG Engine lets you scale your RagManagedDb instance using a choice of tiers: Basic, Scaled, and Unprovisioned. The tier is a project-level setting that impacts all RAG corpora using RagManagedDb. Use basic for dev, scaled for prod traffic.

GCS Bucket for Source Documents

# rag/storage.tf

resource "google_storage_bucket" "docs" {
  name                        = "${var.project_id}-${var.environment}-support-docs"
  location                    = var.region
  uniform_bucket_level_access = true

  versioning {
    enabled = true
  }
}

Service Account and IAM

# rag/iam.tf

resource "google_service_account" "rag_app" {
  account_id   = "${var.environment}-rag-app"
  display_name = "RAG Application Service Account"
  project      = var.project_id
}

resource "google_project_iam_member" "aiplatform_user" {
  project = var.project_id
  role    = "roles/aiplatform.user"
  member  = "serviceAccount:${google_service_account.rag_app.email}"
}

resource "google_storage_bucket_iam_member" "docs_reader" {
  bucket = google_storage_bucket.docs.name
  role   = "roles/storage.objectViewer"
  member = "serviceAccount:${google_service_account.rag_app.email}"
}

Model Config - Bridge to the SDK

# rag/config.tf

resource "local_file" "rag_config" {
  filename = "${path.module}/app/rag_config.json"
  content = jsonencode({
    project_id           = var.project_id
    region               = var.region
    gcs_bucket           = google_storage_bucket.docs.name
    embedding_model      = var.embedding_model
    generation_model_id  = var.generation_model_id
    corpus_display_name  = "${var.environment}-support-docs-${replace(var.embedding_model, "/[^a-zA-Z0-9]/", "-")}"
    chunk_size           = var.chunk_size
    chunk_overlap        = var.chunk_overlap
  })
}

This is the key design decision. The corpus display name is derived from the embedding model. Change the embedding model variable, and Terraform computes a new corpus name automatically - setting up the upgrade pattern covered below.

🐍 Corpus Creation and Ingestion (SDK)

# app/setup_corpus.py
import json
from vertexai import rag
import vertexai

with open("rag_config.json") as f:
    config = json.load(f)

vertexai.init(project=config["project_id"], location=config["region"])

embedding_model_config = rag.RagEmbeddingModelConfig(
    publisher_model=f"publishers/google/models/{config['embedding_model']}"
)

corpus = rag.create_corpus(
    display_name=config["corpus_display_name"],
    rag_embedding_model_config=embedding_model_config,
)

rag.import_files(
    corpus_name=corpus.name,
    paths=[f"gs://{config['gcs_bucket']}/"],
    transformation_config=rag.TransformationConfig(
        chunking_config=rag.ChunkingConfig(
            chunk_size=config["chunk_size"],
            chunk_overlap=config["chunk_overlap"],
        )
    ),
    max_embedding_requests_per_min=900,
)

print(f"Corpus ready: {corpus.name}")

🐍 Query the Pipeline (Model-Agnostic Generation)

# app/query.py
from vertexai import rag
from vertexai.generative_models import GenerativeModel, Tool

def ask(question: str, corpus_name: str, generation_model_id: str) -> str:
    rag_tool = Tool.from_retrieval(
        retrieval=rag.Retrieval(
            source=rag.VertexRagStore(
                rag_resources=[rag.RagResource(rag_corpus=corpus_name)],
                similarity_top_k=5,
            )
        )
    )
    model = GenerativeModel(generation_model_id, tools=[rag_tool])
    response = model.generate_content(question)
    return response.text

answer = ask(
    "What is our refund policy for annual plans?",
    corpus_name="projects/.../ragCorpora/...",
    generation_model_id="gemini-2.5-flash",
)

The generation model is fully swappable at query time. Swap gemini-2.5-flash for gemini-2.5-pro, or even a Model Garden model like Llama, by changing generation_model_id in tfvars. No corpus changes needed.

⚠️ The Embedding Model Honesty Check

Here's the part most tutorials skip: the embedding model is locked to the corpus at creation time. You cannot change it without recreating the corpus and re-importing all data. The association between your embedding model and the RAG corpus remains fixed for the lifetime of that corpus.

This is why the Terraform config above names the corpus after the embedding model. Upgrading embeddings means standing up a new corpus, not editing the old one. The pattern:

1. Change embedding_model in tfvars (e.g., text-embedding-004 → text-embedding-005)
2. terraform apply - this computes a new corpus_display_name, nothing destructive happens to the old corpus
3. Run setup_corpus.py again - creates a fresh corpus under the new name, re-imports from the same GCS bucket
4. Run the evaluation below against both corpora
5. If the new corpus wins, point your app config at the new corpus_name and delete the old corpus

This is genuinely simple once you know the pattern, and it means zero downtime during the swap since both corpora exist side by side until you cut over.

📐 Environment Configuration

# environments/dev.tfvars
environment          = "dev"
embedding_model      = "text-embedding-005"
generation_model_id  = "gemini-2.5-flash"
rag_db_tier          = "basic"
chunk_size           = 512
chunk_overlap        = 100

# environments/prod.tfvars
environment          = "prod"
embedding_model      = "text-embedding-005"
generation_model_id  = "gemini-2.5-pro"
rag_db_tier          = "scaled"
chunk_size           = 512
chunk_overlap        = 100

Chunking configuration is set per import operation, not per corpus, so you can re-import the same files with different chunking to test what works best, without even touching the embedding model.

🧪 Small Evaluation: Did the Upgrade Actually Help?

Same lightweight approach as any model regression check - a handful of real questions, scored on retrieval hit rate and answer keyword coverage:

# evaluate.py
from vertexai import rag
from app.query import ask

eval_set = [
    {"question": "What is our refund policy for annual plans?",
     "expected_keywords": ["30 days", "annual", "refund", "prorate"]},
    {"question": "How do I reset a customer's 2FA?",
     "expected_keywords": ["admin panel", "2FA", "reset", "support ticket"]},
    {"question": "What's the SLA for enterprise tier support?",
     "expected_keywords": ["1 hour", "enterprise", "SLA", "priority"]},
]

def run_eval(corpus_name: str, generation_model_id: str) -> dict:
    hits, total_keyword_matches = 0, 0
    for case in eval_set:
        contexts = rag.retrieval_query(
            rag_resources=[rag.RagResource(rag_corpus=corpus_name)],
            text=case["question"],
            rag_retrieval_config=rag.RagRetrievalConfig(top_k=5),
        )
        if contexts.contexts.contexts:
            hits += 1

        answer = ask(case["question"], corpus_name, generation_model_id).lower()
        matched = sum(1 for kw in case["expected_keywords"] if kw.lower() in answer)
        total_keyword_matches += matched

    return {
        "retrieval_hit_rate": hits / len(eval_set),
        "avg_keyword_coverage": total_keyword_matches / len(eval_set),
    }

old_results = run_eval("CORPUS_004_NAME", "gemini-2.5-flash")
new_results = run_eval("CORPUS_005_NAME", "gemini-2.5-flash")

print(f"text-embedding-004 corpus: {old_results}")
print(f"text-embedding-005 corpus: {new_results}")

Run this before cutting over. If the new embedding model's retrieval_hit_rate drops, don't delete the old corpus yet - investigate why. Ten to twenty real questions pulled from actual support tickets catch regressions well enough for this kind of sanity check.

⚠️ Gotchas and Tips

Corpus creation and import happen via SDK only. There's no native Terraform resource for the RAG corpus itself as of this writing. Terraform handles the durable infrastructure; the SDK handles corpus lifecycle.

Retrieval rate limit. Retrieval requests are capped at 600 RPM per region. Factor this into evaluation batch sizes and production query volume.

Delete old corpora deliberately, not automatically. Since corpora are created outside Terraform state, cleanup is a manual or scripted step, not something terraform destroy handles for you.

Watch for model deprecations. Google occasionally deprecates publisher embedding models. Since the corpus is locked to whatever model created it, plan a migration path before a deprecation date arrives, not after.

---

A model-agnostic RAG pipeline for real internal questions, with an honest process for upgrading embeddings and a built-in way to check that the upgrade actually helped. Terraform for the infrastructure, SDK for the corpus, evaluation before every cutover. 🔍

Found this helpful? Follow for the Azure version coming next! 💬