Every production system starts with a diagram, not code.
When I started this project, I thought I would spend most of my time choosing and configuring an LLM.
Instead, I spent days thinking about something else.
⚠️ New here? This is Part 1 of the series. If you'd like the backstory on why I built this platform, the mistakes I made, and the lessons from my first production RAG system, read Part 0 first:
Building an Enterprise GenAI Platform on OCI – Lessons from My First Production RAG System
Then come back here for the architecture deep dive.
The architecture.
The moment I stopped thinking about the model and started thinking about the system, everything changed. A production GenAI application needs far more than an LLM, it needs reliable data pipelines, efficient retrieval, scalable infrastructure, and secure deployment. So before writing any code, I asked myself: What exactly am I building? A chatbot, a RAG application, or an enterprise GenAI platform?
Thinking Beyond a Chatbot
Most RAG tutorials follow roughly the same flow.
Documents -> Embeddings -> Vector Database -> LLM -> Answer
For learning, that architecture works just fine. But production isn't just about retrieving documents and calling an LLM, it's about building a system that can ingest data continuously, scale reliably, and evolve over time. That's when I realized I wasn't designing a chatbot; I was designing a platform.
The Architecture
The architecture naturally separated into two independent pipelines.
1. Offline Pipeline
This pipeline prepares knowledge before any user sends a query.
Its responsibility is to transform raw information into something an LLM can retrieve efficiently.
The flow looks like this:
External Sources -> OCI Compute -> Object Storage -> Cleaning & Chunking -> Embedding Generation -> FAISS Index -> Artifacts
These jobs are compute intensive but don't need to run for every request.
Keeping them separate allows the knowledge base to evolve independently of the inference service.
2. Online Pipeline
Once the data is prepared, the online pipeline takes over.
User -> Oracle APEX -> FastAPI -> Query Embedding -> FAISS Retrieval -> Prompt Construction -> LLM -> Streaming Response
Unlike the offline pipeline, every millisecond matters here.
The goal isn't just to generate an answer.
The goal is to generate the right answer quickly.
That changes how every component is designed.
Why Split the Platform?
One question I kept asking myself was:
Why not put everything inside a single application?
Because different workloads have different requirements.
The ingestion pipeline may process thousands of documents in batches.
The inference service should respond within seconds.
Trying to solve both problems inside one application makes scaling harder and deployments riskier.
Separating the platform into independent services means each layer can evolve without affecting the others.
Designing Around Responsibilities
Once I stopped thinking about frameworks and started thinking about responsibilities, the architecture almost designed itself. Every component had one job: ingest data, process it, retrieve knowledge, generate responses, or serve the application. That's how the platform naturally evolved into six layers.
| Layer | Responsibility |
|---|---|
| Data Ingestion | Collect raw knowledge |
| Storage | Persist datasets, embeddings, indexes, and artifacts |
| Processing | Clean, normalize, and chunk documents |
| Retrieval | Search relevant knowledge |
| Inference | Generate grounded responses |
| Application | Provide the enterprise user interface |
Notice that none of these layers are tied to a specific technology and that's intentional. Good architecture should outlive the tools used to implement it. FastAPI can be replaced by another framework, FAISS by a managed vector database, or TinyLlama by a different LLM. The technologies may evolve, but the responsibilities of each layer remain the same.
Architecture Principles
While designing the platform, I kept returning to a small set of principles.
- Modular – Every layer should have a single responsibility.
- Cloud-native – Managed services where they make sense.
- Scalable – Independent scaling for ingestion and inference.
- Observable – Health, metrics, and logging built in.
- Secure – Least-privilege access between services.
- Cost aware – Optimize for learning without unnecessary infrastructure.
These principles influenced every decision that followed.
Why OCI?
One of the goals of this project was to explore how far a production-oriented GenAI platform could be built using Oracle Cloud Infrastructure.
OCI offered everything I needed for this stage of the project:
- Compute for data collection
- Object Storage for the data lake
- Data Science for experimentation
- Container Registry for deployment
- IAM for secure access
- Oracle APEX for the enterprise application layer
Rather than introducing every service at once, each will appear naturally as the project evolves throughout this series.
One Lesson I Learned Early
I started this project thinking the hardest part would be choosing the right LLM and refining prompts. Instead, I found myself spending far more time designing how data moved through the system. That was the moment I realized a great GenAI application isn't defined by its model, it's defined by its architecture.
What's Next?
Now that the blueprint is complete, it's time to start building.
In Part 2, we'll implement the first stage of the platform:
- Collecting data with OCI Compute
- Streaming it directly into Object Storage
- Designing the data lake
- Preparing thousands of documents for downstream processing
Because before we can build a great retrieval system, we first need great data.
Top comments (0)