Building a generative AI application is easy; building one that is both blazingly fast and rigorously accurate is a completely different beast.
Recently, as part of Challenge 2 for the Google Cloud Gen AI Academy (APAC Edition), I was tasked with moving beyond simple prompting and diving deep into System Design Thinking. The scenario was straightforward but challenging: design an architecture utilizing an LLM, a user query, and a custom knowledge base that delivers responses that are both accurate and fast.
graph TD
%% Custom Styles
classDef userReq fill:#e1f5fe,stroke:#0288d1,stroke-width:2px,color:#000
classDef cache fill:#ffe0b2,stroke:#f57c00,stroke-width:2px,color:#000
classDef retrieval fill:#e8f5e9,stroke:#388e3c,stroke-width:2px,color:#000
classDef precision fill:#fff9c4,stroke:#fbc02d,stroke-width:2px,color:#000
classDef generation fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px,color:#000
%% Node Definitions
User((User Request)):::userReq
API[FastAPI Gateway]:::userReq
Cache{L1 Response Cache<br/>Redis}:::cache
CacheHit[Instant Cached Response<br/>Latency: ~50ms]:::cache
Embed[Embedding Model +<br/>Metadata Filter]:::retrieval
VectorDB[(Vertex AI Vector DB)]:::retrieval
Candidates[Top 20 Candidates]:::retrieval
Reranker{Cross-Encoder<br/>Re-ranker}:::precision
Context[Top 3 Gold Contexts]:::precision
Prompt[Constraint-Based<br/>Prompt Template]:::generation
LLM((Gemini Flash LLM)):::generation
Stream[SSE Streaming Delivery]:::generation
%% Flow Logic
User -->|Query: 'Policy on X?'| API
API -->|Check existing| Cache
%% Cache Branch
Cache -->|HIT| CacheHit
%% RAG Branch
Cache -->|MISS| Embed
Embed -->|Vector + Metadata| VectorDB
VectorDB -->|Fast Semantic Search| Candidates
%% Precision Branch
Candidates -->|Raw Chunks| Reranker
Reranker -->|Absolute Relevance Sort| Context
%% Generation Branch
Context --> Prompt
API -.->|Original Query| Prompt
Prompt -->|Context + Query| LLM
LLM -->|Token-by-Token Output| Stream
Stream -->|Cited Answer| User
Here is a breakdown of the architecture I designed to solve this exact problem, moving from a proof-of-concept to a robust, production-ready pipeline.
🏗️ The Core Architecture: Advanced RAG
To ground the LLM in reality and prevent hallucinations, a Retrieval-Augmented Generation (RAG) pipeline is non-negotiable. But a vanilla RAG setup isn't enough for high-stakes environments.
Here are the core components of my proposed system:
Vector Database: For fast semantic similarity searches.
Embedding Model: To convert text chunks into high-dimensional vectors.
LLM: Gemini Flash, specifically chosen for its ultra-low latency.
Re-ranker: A cross-encoder to sort retrieved contexts by absolute relevance.
Dual-Layer Caching: To intercept redundant queries before they hit the expensive LLM layer.
When bringing a system like this to life, I typically wrap the orchestration logic in a lightweight FastAPI backend. Containerizing the pipeline and deploying it to a serverless environment like Google Cloud Run ensures the API can scale down to zero to save costs, while instantly scaling up to handle traffic spikes without bottlenecking the response times.
🎯 Optimizing for Accuracy
You can't afford an AI assistant that guesses. To ensure the highest fidelity of information, the pipeline needs strict guardrails:
Metadata Pre-Filtering: Before performing a vector search, the system filters documents by metadata (e.g., date, category, access level). If a user asks about a "2026 policy," the vector search shouldn't even look at 2024 documents.
Cross-Encoder Re-ranking: Vector similarity isn't always semantic relevance. The Vector DB quickly grabs the top 20 candidate chunks, but a Cross-Encoder model meticulously re-ranks them, feeding only the absolute top 3 most relevant chunks to the LLM.
Strict Prompt Constraints: The prompt template acts as the final judge. It explicitly forces the model: "Answer using ONLY the provided context. If the answer is not present, reply with 'Data not available.' Always cite the source document."
⚡ Optimizing for Latency
Accuracy doesn't matter if the user has to wait 30 seconds for an answer. Speed is achieved through aggressive caching and smart delivery:
L1 Response Caching (Redis): If a user asks a common question (e.g., "What are the standard working hours?"), an in-memory cache instantly returns the pre-generated answer. Latency: ~50ms.
L2 Semantic Caching: What if the user asks, "Tell me the standard work hours?" instead? It's the same intent, different wording. By caching the query embeddings, we can measure semantic similarity to previous questions. If it's a match, we bypass the retrieval phase entirely.
Server-Sent Events (SSE) Streaming: Instead of waiting for the entire response to generate, the FastAPI backend streams the output token-by-token to the client. This reduces perceived latency to near zero, keeping the user engaged while the model works.
🔭 Future Scope: Where Do We Go From Here?
While this architecture solves the immediate need for speed and accuracy, system design is always evolving. For future iterations, I am exploring:
Dynamic Chunking Strategies: Moving away from fixed-size text chunks and using NLP-driven semantic chunking (splitting by logical headers or paragraphs) to maintain better context.
GraphRAG Integration: Combining traditional vector databases with Knowledge Graphs to map relationships between entities, drastically improving the system's ability to answer complex, multi-hop queries.
Agentic Routing: Implementing a lightweight semantic router at the API Gateway that decides whether a query needs the full RAG pipeline, a simple database lookup, or an API call to an external service.
Wrapping Up
Participating in this Hack2skill and Google Cloud challenge was an incredible exercise in balancing trade-offs. The biggest takeaway? The LLM is just the engine; the architecture is the vehicle. If you want to go fast and stay on track, you have to engineer the whole car.
How are you optimizing your Gen AI pipelines for production? Drop your thoughts in the comments! 👇
Top comments (0)