DEV Community: Yugandhar Surya

Building an Enterprise GenAI Platform on OCI — Part 1: Designing the Architecture

Yugandhar Surya — Fri, 17 Jul 2026 16:45:34 +0000

Every production system starts with a diagram, not code.

When I started this project, I thought I would spend most of my time choosing and configuring an LLM.

Instead, I spent days thinking about something else.

⚠️ New here? This is Part 1 of the series. If you'd like the backstory on why I built this platform, the mistakes I made, and the lessons from my first production RAG system, read Part 0 first:

Building an Enterprise GenAI Platform on OCI – Lessons from My First Production RAG System

Then come back here for the architecture deep dive.

The architecture.

The moment I stopped thinking about the model and started thinking about the system, everything changed. A production GenAI application needs far more than an LLM, it needs reliable data pipelines, efficient retrieval, scalable infrastructure, and secure deployment. So before writing any code, I asked myself: What exactly am I building? A chatbot, a RAG application, or an enterprise GenAI platform?

Thinking Beyond a Chatbot

Most RAG tutorials follow roughly the same flow.

Documents -> Embeddings -> Vector Database -> LLM -> Answer

For learning, that architecture works just fine. But production isn't just about retrieving documents and calling an LLM, it's about building a system that can ingest data continuously, scale reliably, and evolve over time. That's when I realized I wasn't designing a chatbot; I was designing a platform.

The Architecture

The architecture naturally separated into two independent pipelines.

1. Offline Pipeline

This pipeline prepares knowledge before any user sends a query.

Its responsibility is to transform raw information into something an LLM can retrieve efficiently.

The flow looks like this:

External Sources -> OCI Compute -> Object Storage -> Cleaning & Chunking -> Embedding Generation -> FAISS Index -> Artifacts

These jobs are compute intensive but don't need to run for every request.

Keeping them separate allows the knowledge base to evolve independently of the inference service.

2. Online Pipeline

Once the data is prepared, the online pipeline takes over.

User -> Oracle APEX -> FastAPI -> Query Embedding -> FAISS Retrieval -> Prompt Construction -> LLM -> Streaming Response

Unlike the offline pipeline, every millisecond matters here.

The goal isn't just to generate an answer.

The goal is to generate the right answer quickly.

That changes how every component is designed.

Why Split the Platform?

One question I kept asking myself was:

Why not put everything inside a single application?

Because different workloads have different requirements.

The ingestion pipeline may process thousands of documents in batches.

The inference service should respond within seconds.

Trying to solve both problems inside one application makes scaling harder and deployments riskier.

Separating the platform into independent services means each layer can evolve without affecting the others.

Designing Around Responsibilities

Once I stopped thinking about frameworks and started thinking about responsibilities, the architecture almost designed itself. Every component had one job: ingest data, process it, retrieve knowledge, generate responses, or serve the application. That's how the platform naturally evolved into six layers.

Layer	Responsibility
Data Ingestion	Collect raw knowledge
Storage	Persist datasets, embeddings, indexes, and artifacts
Processing	Clean, normalize, and chunk documents
Retrieval	Search relevant knowledge
Inference	Generate grounded responses
Application	Provide the enterprise user interface

Notice that none of these layers are tied to a specific technology and that's intentional. Good architecture should outlive the tools used to implement it. FastAPI can be replaced by another framework, FAISS by a managed vector database, or TinyLlama by a different LLM. The technologies may evolve, but the responsibilities of each layer remain the same.

Architecture Principles

While designing the platform, I kept returning to a small set of principles.

Modular – Every layer should have a single responsibility.
Cloud-native – Managed services where they make sense.
Scalable – Independent scaling for ingestion and inference.
Observable – Health, metrics, and logging built in.
Secure – Least-privilege access between services.
Cost aware – Optimize for learning without unnecessary infrastructure.

These principles influenced every decision that followed.

Why OCI?

One of the goals of this project was to explore how far a production-oriented GenAI platform could be built using Oracle Cloud Infrastructure.

OCI offered everything I needed for this stage of the project:

Compute for data collection
Object Storage for the data lake
Data Science for experimentation
Container Registry for deployment
IAM for secure access
Oracle APEX for the enterprise application layer

Rather than introducing every service at once, each will appear naturally as the project evolves throughout this series.

One Lesson I Learned Early

I started this project thinking the hardest part would be choosing the right LLM and refining prompts. Instead, I found myself spending far more time designing how data moved through the system. That was the moment I realized a great GenAI application isn't defined by its model, it's defined by its architecture.

What's Next?

Now that the blueprint is complete, it's time to start building.

In Part 2, we'll implement the first stage of the platform:

Collecting data with OCI Compute
Streaming it directly into Object Storage
Designing the data lake
Preparing thousands of documents for downstream processing

Because before we can build a great retrieval system, we first need great data.

Building an Enterprise GenAI Platform on OCI — Lessons from My First Production RAG System

Yugandhar Surya — Wed, 08 Jul 2026 13:28:51 +0000

I thought building a RAG application would be straightforward.

Scrape the data. Generate embeddings. Build a FAISS index. Connect an LLM. Expose an API.

That's what most tutorials make it look like.

Reality was very different.
Every optimization solved one problem and uncovered another.

Even after switching to a lightweight language model, every response still took several seconds.
That's when I realized the bottleneck wasn't the LLM. It was the architecture itself.

That realization completely changed the direction of this project. Instead of building just another chatbot, I decided to design a production-oriented GenAI platform on Oracle Cloud Infrastructure (OCI), focusing on data engineering, retrieval quality, inference optimization, deployment, observability, and LLMOps.

Over the past few weeks, I built this platform while exploring Oracle Cloud Infrastructure, documenting every architectural decision, optimization, and failure along the way.

Some Problems I Didn't Expect

What started as a simple RAG prototype quickly turned into an engineering exercise.

Along the way, I encountered challenges I hadn't anticipated:

Large Docker images and slow deployments
Infrastructure limitations on OCI Free Tier
Container architecture mismatches (ARM vs. AMD)
Retrieval and inference latency bottlenecks
IAM and networking complexities during deployment

Every challenge forced a new architectural decision and those decisions shaped this entire series.

Why Another RAG Series?

If you've searched for RAG tutorials, you've probably noticed a common pattern.

Most articles demonstrate how to connect an LLM to a vector database and answer questions from a PDF. While that's a great way to understand the basics, it leaves out many of the challenges you'll encounter when building a real-world system.

Along the way, I found myself asking questions like:

Where should the data actually live?
How do you keep it synchronized?
Why is retrieval slower than expected?
Should you use FAISS or a managed vector database?
How do you deploy and scale inference?
What happens when multiple users hit the API simultaneously?

Those were exactly the questions I started asking while building this project.

The Goal

I wanted to understand why production systems look so different from GitHub demos.

The objective wasn't simply to make an LLM answer questions, it was to understand the complete lifecycle of a production GenAI application.

That meant learning how to:

Build scalable data pipelines
Design an efficient retrieval layer
Optimize inference latency
Containerize and deploy services
Integrate with enterprise applications
Apply MLOps and LLMOps principles
Keep operational costs under control

Why Oracle Cloud Infrastructure(OCI)?

Most tutorials use AWS or Azure.

I intentionally chose Oracle Cloud Infrastructure (OCI) because I wanted to explore its AI ecosystem and understand how enterprise GenAI applications could be built using Oracle's cloud services.

The platform combines OCI Compute for ingestion, Object Storage as a data lake, Data Science for experimentation, Model Deployment for inference, and Oracle APEX as the application layer.

Choosing OCI also introduced a unique set of challenges around resource limits, deployment strategies, container architectures, and cloud-native design decisions many of which I'll cover throughout this series.

What We'll Build

By the end of this series, we'll have built an enterprise-oriented RAG platform capable of:

Automated data ingestion
Cloud-native data lake
Semantic search using embeddings
Hybrid Retrieval (FAISS + BM25)
Cross-Encoder Re-ranking
Prompt Engineering
Streaming responses
Guardrails
Dockerized deployment
OCI Model Deployment
Oracle APEX integration
CI/CD pipeline
LLMOps best practices

More importantly, we'll discuss why each architectural decision was made, along with the trade-offs involved.

Note:

This isn't another "build a chatbot in 15 minutes" tutorial. It's an engineering journal documenting the architectural decisions, trade-offs, failures, and lessons learned while building an enterprise GenAI platform on OCI.

If you're looking for a copy-and-paste RAG tutorial, this series probably isn't for you.

But if you're interested in understanding how enterprise GenAI systems are designed from data ingestion and retrieval to deployment, observability, and optimization. I hope you'll find this journey worthwhile.

Enterprise AI systems aren't built by choosing the largest model.
They're built through hundreds of engineering decisions.
This series is about those decisions.
See you in Part 1.

Series Roadmap

📚 This Series (10 Parts)

Architecture & Planning
Data Engineering Pipeline on OCI
Embeddings & Vector Search
Hybrid Retrieval and Re-ranking
FastAPI Inference Pipeline
Docker & OCI Model Deployment
Oracle APEX Integration
LLMOps & CI/CD
Performance Optimization
Lessons Learned & Production Considerations

Before we dive into the next article, I'd love to hear from you:

Have you built a RAG application beyond a simple proof of concept?
What was the biggest challenge you encountered—retrieval quality, latency, deployment, or something else?
Are you using OCI, AWS, Azure, or another cloud platform for GenAI workloads?
Is there a specific topic you'd like me to cover in more detail as this series progresses?

Feel free to share your experiences in the comments. I'm always interested in learning how others are approaching these problems.

If you'd like to follow my work or connect, you can find me here: