DEV Community: Glen Yu

Implementing a RAG system: Run

Glen Yu — Tue, 07 Apr 2026 17:00:00 +0000

In the "Crawl" and "Walk" phases, I introduced the basics of RAG and explored ways to optimize the pipeline to increase efficiency and accuracy. Armed with this knowledge, it's time to productionize our learnings.

Run

In the "Crawl" and "Walk" phases, we explored RAG fundamentals using local tools, proving how much document processing and re-ranking impact performance. While you could certainly scale those manual workflows into production, do you really wan to manage the infrastructure, data pipelines and scaling hurdles yourself?

Welcome to the "Run" phase. Here we leverage Google Cloud's Vertex AI RAG Engine - a fully managed solution that automates the entire pipeline so you can focus on building, not maintenance.

Vertex AI RAG Engine

Vertex AI RAG Engine is a low-code, fully managed solution for building AI applications on private data. It handles the ingestion, document processing, embedding, retrieval, ranking, and grounding to ensure that the response is highly accurate and relevant.

Optional document pre-processing with Docling

Though RAG Engine comes with its own parser options, I still opted to pre-process my documents using Docling first and upload them to a Google Cloud Storage bucket for ingestion:

if __name__ == "__main__":
    DATA_URL = "https://storage.googleapis.com/public-file-server/genai-downloads/bc_hr_policies.tgz"
    DATA_DIR = download_and_extract(DATA_URL)

    docling_docs = Path("./docling_docs")
    docling_docs.mkdir(parents=True, exist_ok=True)

    print("> Processing PDFs...")
    data_path = Path(DATA_DIR)
    pdf_files = list(data_path.glob("*.pdf"))

    for file in pdf_files:
        try:
            result = converter.convert(file)
            markdown_content = result.document.export_to_markdown()

            docling_doc_name = Path(file).stem + ".md"
            with open(f"docling_docs/{docling_doc_name}", "w", encoding="utf-8") as f:
                f.write(markdown_content)
        except Exception as e:
            print(f"Error on {file}: {e}")

    print("> Uploading Docling docs to GCS...")
    upload_folder_to_gcs("./docling_docs", GCS_BUCKET, GCS_BUCKET_PATH)

One benefit of doing it this way is I get more visibility and control over the document processing step and can validate the contents from the original PDFs with the Docling documents (Markdown). This allows me to use the Default parsing libraries option, which is also free compared to the LLM parser and Document AI layout parser options which have an additional cost and setup component to them.

I do lose out on the benefits of the hybrid chunking strategy that Docling would have provided (as seen in the "Walk" phase), because that is determined by the layout parser that I choose here. If I wasn't using Docling, I think the LLM parser would be the parser option that I'd be gravitating towards.

Vector database options

When it comes to vector database options, you'll see several choices in the RAG Engine menu (including regional "Preview" features). I chose the RagManaged Cloud Spanner (also referred to as the RagManagedDb) because it offers the fastest path from data to insights with the least amount of infrastructure management. While Spanner is typically an enterprise-grade database, the RAG Engine allows you to spin it up on the Basic tier. This allocates 100 processing units, which is 10% of a Spanner node, making it perfect for smaller datasets while still giving you the reliability of a managed service without the enterprise-grade cost.

IMPORTANT: Even on basic tier, this will still run you about $65 USD/month, so please remember to delete and clean up this RAG corpus once you're done experimenting with it.

For those prioritizing flexibility, the RAG Engine also supports third-party options like Pinecone and Weaviate. These are excellent choices if portability is a requirement, allowing you to maintain a consistent vector store even if you decide to shift parts of your RAG stack to a different cloud provider or platform later on.

Ranking & grounding included

Once the RAG corpus is created, you can perform some manual testing to validate. When you ask RAG Engine for search results, re-ranking and grounding is done automatically to ensure relevance and correctness:

Model Armor

In a production setting (especially if it's going to be public facing), you will want guardrails. I've written about Guardrails with Agent Development Kit in the past and is implemented through callbacks within ADK. It works the same way here and can be used to inspect text as it flows into and out of the LLM/agent. Key capabilities include:

Prompt injection & jailbreak detection: Attempts to trick the AI into ignoring instructions
Sensitive Data Protection: Natively integrates with Google's Data Loss Prevention (DLP) to scale for various types of sensitive information (PII)
Malicious URL detection
Responsible AI (RAI) filters: Hate speech, harassment, dangerous, and sexually explicit content

I configured my Model Armor policy template which I then invoked via the Python SDK library to determine whether the text was safe.

Updated example

You can find the code for "Run" phase → here

Querying the vector database (RAG Engine in this case) is a lot less involved as I don't have to write a many of the logic to pass the semantic search results to a re-ranker, because RAG Engine takes care of all of that for me!

I once again ran the same two benchmark questions as I did in the "Crawl" and "Walk" phases:

I threw in a couple of extra questions to make sure Model Armor wasn't sleeping on the job, but overall I liked the detail and accuracy of the answers I was provided.

NOTE: In my updated example, I only added a before_model_callback, meaning I'm only checking the input prompts and not the response. An after_model_callback should be implemented to ensure the generated response is also scanned and preventing the AI from accidentally leaking sensitive internal data it might have retrieved from the RAG corpus (I omitted the output check here simply because I know there's no sensitive data in this particular dataset).

Summary

The purpose of this "Crawl, Walk, Run" series was to take you on a journey from managing code to delivering value. In the earlier phases, we deconstructed the mechanics of RAG works to understand the roles that chunking, embedding, re-ranking play in the overall system. In this final phase, we see how Vertex AI RAG Engine and Model Armor streamline those manual components. By offloading infrastructure management and safety logic to Google Cloud's managed services, you can ensure your system is scalable, accurate, and secure from day one.

Next Steps

Currently in private preview is Vector Search 2.0 with RAG, but reading through its documentation and features, it looks pretty interesting, so once it becomes GA, I will definitely give it a try.
I'm also looking forward to all the new AI-related announcements that is sure to happen at Google Next!

Additional learning

Interested in finding out more about how to secure your agent by sanitizing input and output? Try out this Model Armor Codelab!
Vertex AI RAG Engine isn't your only option for a managed RAG, if you'd like to try a different option that uses Vector AI Search, might I suggest the Building Agents with Retrieval-Augmented Generation Codelab?

Happy learning!

Implementing a RAG system: Walk

Glen Yu — Tue, 31 Mar 2026 17:00:00 +0000

Now that we've established the basics in our "Crawl" phase, it's time to pick up the pace. In this guid, we'll move beyond the initial setup to focus on optimizing core architectural components for better performance and accuracy.

Walk

We ended the previous "Crawl" design with a functioning AI HR agent with a RAG system. The responses, however, could be better. I've introduced some new elements to the architecture to perform better document processing and chunking, as well as re-ranker model to sort the semantic retrieval results by relevance:

The ugly Docling

IBM's Docling is an open-source document processing tool and easily one of the most effective ones I've tested. It can convert various file formats (e.g., PDF, docx, HTML) into clean, structured formats like Markdown and JSON. By integrating AI models and OCR, it doesn't just extract text, but also preserve the original layout's integrity.
Through its hierarchical and hybrid chunking methods, Docling intelligently groups content by heading, merges smaller fragments for better context, and attaches rich metadata to streamline downstream searching and citations. Here's a Python function I use for chunking a PDF file:

def docling_chunk_pdf(file: str) -> tuple[list, list]:
    converter = DocumentConverter(
        format_options={
            InputFormat.PDF: PdfFormatOption(
                pipeline_options=pipeline_options,
            )
        }
    )

    result = converter.convert(file)
    doc = result.document
    chunker = HybridChunker()
    chunks = list(chunker.chunk(doc))
    chunk_texts = [c.text for c in chunks]

    return chunks, chunk_texts

I plan to take a deeper dive into Docling in a future article, so give me a follow so you won't miss it! 😄

Dot product vs cosine similarity

In the "Crawl" post, I talked briefly about cosine similarity and how it ignore magnitude and only focuses on the angle between two vectors. This is because normalization is baked into the cosine similarity formula.
Dot product is effectively cosine similarity but without the final normalization step, which is why its result is affected by the magnitude of the vectors. Since many modern embedding models output pre-normalized unit vectors, the extra normalization step in cosine similarity becomes a redundant calculation. By using dot product on these pre-normalized vectors, you can achieve identical results with higher computational efficiency.

NOTE #1: While switching to dot product can increase your raw retrieval throughput, the latency gains may feel negligible when considering the entire end-to-end RAG pipeline depending on your particular use case and scale.

NOTE #2: A friendly reminder that choosing dot product over cosine similarity has the hard requirement that your vectors be normalized beforehand, or the magnitude will skew your search results. It's also quite easy to update your search configuration to use one or the other. If you're ever in doubt, just run a quick test with both settings to verify that both methods return the exact same nearest neighbours (top semantic matches).

Re-ranking

Standard search is built for speed and not deep understanding, so it can sometimes miss nuances. Re-ranking takes a crucial "second look" at the standard retrieval results to see which one(s) actually address the user's query. While the Cosine distance represents how similar the query and document align in the vector space, a "close" match doesn't guarantee an answer. The re-ranker's job to is to bridge this gap by scrutinizing the top results to assign a true relevance_score and ensure the most helpful contexts rise to the top. Here's what that snippet of code looks like:

co = cohere.ClientV2()

response = co.rerank(
    model=RERANKING_MODEL,
    query=user_query,
    documents=documents_to_rerank,
    top_n=len(candidate_responses),
)

reranked_results = []
for res in response.results:
    original_data = candidate_responses[res.index]

    reranked_results.append({
        "content": original_data["content"],
        "source": original_data["source"],
        "heading": original_data["heading"],
        "page": original_data["page"],
        "search_distance": original_data["search_distance"],
        "relevance_score": res.relevance_score
    })

As part of the full code that performs the re-ranking, I assign a threshold for the relevance score. Scores lower than this threshold is deemed irrelevant.

Updated example

I'm shaking up the stack for the "Walk" phase! In addition to using a different document processor, I will also be using a different embedding model and vector database.
Since I wanted to try out Cohere's re-ranking model, I opted to lean into their full suite and use their embedding model as well. I made a deliberate choice here to set the embedding dimension to 384, which is a lower than the 768 I previous used in the "Crawl" example. I wanted to handicap the initial semantic search, and by doing so, we can more clearly see the re-ranker work its magic to fix the order or the results.
I switched out ChromaDB with LanceDB to showcase just how many robust, easy-to-use open-source local vector databases are available for use.

Querying the HR agent

While I kept the core agent configuration from the "Crawl" phase the same, the addition of the re-ranking step made a significant impact. I asked the same two benchmark questions and this time the results were more refined and accurate:

You can find the code for the "Walk" phase → here

Next steps

Now that we've manually optimized our retrieval and re-ranking, the next step is to scale. I will be migrating this architecture to Vertex AI's RAG Engine for a fully managed, high-performance RAG pipeline at an enterprise scale.

Additional learning

I used Cohere's embedding and re-ranking models in my example, but if you want to try out Vertex AI's re-ranking capabilities (and more), try out this Advanced RAG Techniques Codelab.

Implementing a RAG system: Crawl

Glen Yu — Tue, 24 Mar 2026 18:00:00 +0000

I'm starting a "Crawl, walk, run" series of posts on various topics and decided to start with Retrieval-Augmented Generation (RAG). Learn the basics and progress to a production-ready system!

Crawl

In this phase of your journey, we're going to learn about the core concepts of a Retrieval-Augmented Generation (RAG) system and then apply them in a simple example.
We're going to build a Human Resources (HR) agent that can help answer and navigate HR-related questions. Using the Government of British Columbia's HR Policy PDFs as our knowledge base, we will process, chunk, and embed the documents into a local vector database. This allows the agent to provided grounded answers and ensures that every response is rooted directly in the ingested BC government policies.

Why RAG?

RAG is a very common design pattern that turns a standard LLM into an informed AI agent. Standard models can be a "black box", but RAG gives your agent an "open-book test". It bypasses knowledge cutoffs by linking directly to your documents, providing factual grounding and citations. No fine-tuning is required, data can be updated quickly. RAG provides a real-time bridge between your LLM and your data.
This post will focus primarily on indexing and retrieval of the your data. Let's get started!

How do you eat an elephant? One bite at a time

It's not feasible to have to feed all the information into the AI every time you want to ask it a question. Instead, it is broken down into smaller, more manageable pieces called chunks, which the AI can process and retrieve efficiently.
We will use a "recursive character chunking" strategy which is a fast and smart and will try to split at natural boundaries like paragraphs, but can still cut off mid-sentence if the chunk is too big. An overlap is used to ensure that context isn't lost at the edges of a the cut if a split does occur.

Snippet of code used for splitting & chunking using LangChain:

from langchain_community.document_loaders import PyPDFLoader, DirectoryLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter

loader = DirectoryLoader(
    DATA_DIR,
    glob="./**/*.pdf",
    loader_cls=PyPDFLoader
)
docs = loader.load()

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=100
)
chunks = text_splitter.split_documents(docs)

Recursive character chunking is the successor to "fixed-sized chunking", which is just a fixed sliding window. Here, you always need the overlap because you never know how much of what sentence you're cutting off.

What's the vector representation of 'Life'?

The embedding process transforms text chunks into vectors, which are mathematical arrays of floating-point numbers that capture semantic meaning. However, higher dimensionality won't necessarily generate better results. For simple or straightforward documents, expanding the vector size often introduces latency and computational overhead without providing better search accuracy.

Two sides of the same coin: Indexing & retrieval

Indexing and retrieval are two parts of the same conversation, and you must use the same embedding model for both. This is important because every embedding model puts emphasis on words in a sentence differently. One might prioritize the subject, while another might prioritize the action, which would yield different results.

What happens when different embedding models try to embed “To be, or not to be…”:

Selection the appropriate embedding type is also very important. The documents that you embed and index are usually a long, structured documents where the focus is on the information it provides. This is in contrast to the user queries which are usually short, messy text, so the retrieval process focuses on the information it is looking for.

Finding a match

Once your user query is embedded, the RAG system performs a similarity search against the vector database to identify the most relevant answers. In most vector databases, this is calculated using cosine similarity. This metric focuses exclusively on the angle between vectors rather than their magnitude; it measures how closely the semantic "intent" (angle) of the query aligns with the document, regardless of the text's length or word frequency. This is important because it means the AI can recognize that a short question and a long technical manual can share the same intent even if their scale (magnitude) is completely different.

Putting it all together

Link to my GitHub repository → here

To handle HR questions, I'm building an agent using Google's Agent Development Kit (ADK) that connects directly to this RAG system:

from .tools import query_hr
hr_rag_tool = FunctionTool(func=query_hr)

hr_agent = LlmAgent(
    name="hr_agent",
    model="gemini-3.1-pro-preview",
    description="Specialist in company HR policies and procedures.",
    instruction=(
        "You are a professional HR assistant. Your goal is to answer questions "
        "using ONLY the information retrieved from the 'query_hr' tool. "
        "When calling the 'query_hr' tool, ensure all string arguments are properly formatted as standard JSON strings with double quotes.\n\n"
        "RULES:\n"
        "1. If the tool returns relevant information, summarize it clearly.\n"
        "2. You MUST cite your sources using the format: (Source: [Source Name], Page: [Page Number]).\n"
        "3. If the tool results do not contain the answer, state: 'I'm sorry, I couldn't find that in our HR documents.'\n"
        "4. Do not use outside knowledge or make up facts about company policy."
    ),
    tools=[query_hr],
)

By giving the agent clear instructions and the right tools to search our vector database, it should be able to pull precise answers for users in seconds:

It answers questions reliably, but if I'm being honest, I can't help but feel we're only scratching the surface of the "full" answers that we're looking for.

Next steps

We have a working prototype, but there's still plenty of room to grow. To transform this from a simple RAG system into a high performance engine, our next steps will focus on precision. We'll refine how we process and chunk documents and introduce a reranking layer to our search results to significantly boost the quality of the agent's responses.

Additional learning

If you haven't used Agent Development Kit yet, but would like to learn more, checkout this Codelab: "ADK Crash Course - From Beginner to Expert" (it comes with a link to claim some free GCP credits to get you through the course).