DEV Community: Yugandhar Surya

Building an Enterprise GenAI Platform on OCI — Part 2: The Data Pipeline Nobody Talks About

Yugandhar Surya — Thu, 10 Sep 2026 19:54:00 +0000

The quality of a RAG system is decided long before the query reaches the LLM.

I learned that the hard way.

After designing the architecture in Part 1, my first instinct was to move straight to embeddings.

After all, embeddings are where RAG starts getting interesting, right?

Not quite.

Before generating a single vector, I had a much more basic problem to solve.

Before generating a single vector, I had a much more basic problem:

*How do you reliably move 70,000+ documents through a pipeline running on constrained infrastructure?
*
They needed to be collected, stored, cleaned, chunked, processed, and eventually transformed into something a retrieval system could understand.

My OCI Compute instance wasn't exactly a powerhouse either.

What initially looked like a scraping problem quickly became a data-engineering problem.

And that's where Part 2 begins.

The Pipeline Before the AI

There is a tendency when building GenAI applications to start here:

Document → Embedding → Vector DB → LLM

But... where did that document come from?

What happens when there are 70,000 of them?

What if the process crashes halfway?

What if you need to reprocess the data without scraping everything again?

What if your embedding strategy changes tomorrow?

Suddenly, this isn't an LLM problem.

It's a data lifecycle problem.

So I deliberately separated the pipeline into stages:

Each stage produces an artifact that the next stage can consume.

That decision became much more important later.

Stage 1: Getting the Data In

The first component was straightforward.

I needed technical knowledge for the DevOps assistant.

So I used an OCI Compute instance for scraping and data collection.

But I made an important decision early:

The compute instance should process data. It shouldn't become the permanent home of the data.

Why?
Because:

Compute is ephemeral.
Instances can be stopped.
Disks have limits.
Applications change.
Pipelines fail.

The knowledge base needed to survive independently of the machine that created it.

That made OCI Object Storage the natural persistence layer.

Why Not Just Save Everything Locally?

For a prototype, I could have done this:

scraper/ ├── data/ │ ├── file1.json │ ├── file2.json │ ├── file3.json │ └── ...

And initially, that feels simpler.

But now the data is tied to the Compute instance.

What happens when preprocessing moves to OCI Data Science?

I would need to:

transfer files
mount storage
expose an endpoint
synchronise directories

Or...

I could make both services communicate through durable object storage.

That is exactly what I did.

Now Compute and Data Science don't need to know anything about each other.

They only need to understand the storage contract.

That's decoupling.

And this was one of the first architectural decisions that made the platform considerably easier to evolve.

Streaming Data Instead of Hoarding It

There was another constraint.

The Compute instance, I was using had limited resources.

Loading thousands of documents into memory before uploading them would have been unnecessary and risky.

So instead of thinking:

I moved toward:

One object at a time.

This sounds like a small implementation detail.

It isn't.

It changes the memory profile of the ingestion pipeline.

Instead of memory consumption increasing with dataset size, the worker only needs enough memory for the data currently being processed.

Conceptually:

for document in documents:

    content = scrape(document)

    cleaned = basic_clean(content)

    upload_to_object_storage(cleaned)

    del content

The actual implementation evolved, but the principle stayed the same:

Move data through the pipeline instead of accumulating it inside the worker.

Object Storage Became the Data Backbone

Initially, I thought of Object Storage as:

"The place where I'll put my files."

That definition quickly became too simplistic.

It became the data backbone connecting the different stages of the platform.

I organised the bucket roughly like this:

mlops-llm-data/ ├── datasets/ ├── processed/ ├── features/ ├── models/ └── logs/

Each represented a different stage of the data lifecycle.

Prefix	Responsibility
`datasets/`	Raw ingested data
`processed/`	Cleaned and chunked documents
`features/`	Generated embedding artifacts
`models/`	Model and retrieval artifacts
`logs/`	Pipeline and operational logs
`docker-build/`	Docker build-related artifacts
`conda/`	Conda/environment-related artifacts

These aren't traditional filesystem directories. But architecturally, they create clear boundaries between stages.

Raw Data Should Stay Raw

This became one of the most important rules in the pipeline:

Never destroy the original dataset just because your processing logic changed.

Suppose I scraped 70,000 documents.

Then I cleaned them.

A week later, I change my cleaning logic.

If I overwrote the originals, I'd have to scrape everything again.

Instead: datasets/ -> remained the source of truth.

And: processed/ -> contained derived data.

That meant I could rebuild the downstream pipeline without repeating ingestion.

This is essentially an immutable raw-data pattern.

And it gave me something extremely valuable: reproducibility.

Then Came Chunking

Now the data existed. But an LLM retrieval pipeline doesn't necessarily want entire documents.
Imagine retrieving a 5,000-word article because the answer exists in three sentences somewhere in the middle.

That creates several problems:

More irrelevant context
Higher token consumption
Larger prompts
Increased inference latency
Greater chance of distracting the model

So documents needed to be divided into smaller semantic units.

Chunks.

Conceptually:
Document │ ├── Chunk 1 ├── Chunk 2 ├── Chunk 3 ├── Chunk 4 └── Chunk 5

Those chunks would later become the units used for embedding and retrieval.

But chunking introduces its own engineering question.

How big should a chunk be?

Chunk Size Is Not Just a Number

Make chunks too large and retrieval becomes noisy.
Make them too small and you destroy context.

Consider:

Docker containers package applications together with their dependencies, allowing them to run consistently across environments.

A sensible chunk preserves that idea.

But an aggressive split could produce:

`Chunk 1:
Docker containers package applications together

Chunk 2:
with their dependencies, allowing them

Chunk 3:
to run consistently across environments.`

Each chunk now carries less meaning on its own. That's where chunk overlap helps.

Instead of:

AAAA | BBBB | CCCC

we can create:
AAAA AABBBB BBBCCCC

Some information is intentionally repeated across neighbouring chunks.

That gives the retriever a better chance of preserving concepts that happen to cross chunk boundaries.

But overlap isn't free.

More overlap means:

more chunks -> more embeddings -> larger index -> more storage -> more processing
There is no universally perfect chunk size.

It depends on the documents, embedding model, retrieval strategy, and downstream context window.

This is a recurring theme in RAG:

Every retrieval optimisation has a cost somewhere else.

70,000 Chunks Changed the Problem

Eventually, preprocessing was producing roughly 70,000 chunks.

And that's when another lesson became obvious:

Code that works beautifully for: 100 documents doesn't necessarily behave beautifully for: 70,000 documents

The naive approach would be:

chunks = []

for file in all_files:
    chunks.append(load(file))

process(chunks)

basically says:

"Load everything first. Worry about memory later."

Not ideal.

Especially under constrained infrastructure. So the next architectural decision was obvious. Batch processing.

Batch Processing Instead of "Load Everything"

Rather than loading the entire dataset at once, I processed smaller groups:

70,000 objects
↓
┌───────────────┐
│ Batch 1 │
│ 200 objects │
└───────────────┘
↓
Process
↓
Release
↓
┌───────────────┐
│ Batch 2 │
│ 200 objects │
└───────────────┘
↓
Process
↓
Release
↓
...

The exact batch size is tunable.

The principle is what matters:

Bound the amount of data being processed at any given moment.

This gives you predictable memory usage and makes larger datasets manageable on relatively modest infrastructure.

Then a Small Bug Exposed a Bigger Problem

This was one of my favourite lessons from the entire pipeline.

The processed dataset contained roughly:

70,000 chunks
But the next stage reported:

Total feature files: 1000
Total vectors in index: 999
Training vectors shape: (999, 384)

Wait.
**
70,000 chunks in.

1,000 feature files out?**

Something was wrong.

And here's the interesting part:

It wasn't FAISS.

It wasn't the embedding model.

It wasn't the AI.

The data pipeline was incomplete.

The pipeline was unintentionally limiting the number of objects being listed from OCI Object Storage.

A listing operation wasn't traversing the complete collection.

The result?

No crash.

No exception.

Just incomplete data.

A pipeline can be technically successful and logically wrong.
That's a much scarier failure mode than a simple application crash.

Pagination: The Bug That Looked Like an AI Problem

Cloud APIs commonly paginate large responses.

Conceptually:

Request ↓ Objects 1–1000 ↓ Next Page Token ↓ Objects 1001–2000 ↓ Next Page Token ↓ ... ↓ All Objects

If you forget pagination, your code can still run perfectly.

No exception.

No crash.

No obvious error.

It simply processes an incomplete dataset.

After fixing pagination, the downstream stages could finally see the complete collection.

That changed how I thought about validation.

Counts Became a Data Quality Check

From that point onward, counts became a basic sanity check.

At every stage:

How many records entered?
How many succeeded?
How many failed?
How many were skipped?
How many artifacts were produced?

If ingestion produces:

70,000 documents ** but embedding generation sees: 1,000 chunks, something is clearly **wrong.

This is a simple form of data observability.

You don't need a huge monitoring platform to start.

Sometimes a few carefully placed counters can save hours of debugging.

Why OCI Data Science Entered the Architecture

As processing became heavier, running everything on Compute became increasingly uncomfortable.

I could have simply increased the Compute shape.

Instead, I asked:

Does this workload actually belong on the same machine?

The answer was no.

Scraping and preprocessing have different resource characteristics.

So heavier processing moved toward OCI Data Science notebook sessions.

The responsibilities became:

Compute collected the data.

Object Storage persisted it.

Data Science transformed it.

Again:

separation of concerns.

IAM Became Part of the AI Architecture

Once multiple OCI services started communicating, authentication became part of the design.

Instead of putting credentials inside configuration files, the notebook used OCI Resource Principals.

Conceptually:

OCI resource has an identity
+
IAM defines permissions

This changes the model from:

Application possesses credentials

to:

Cloud resource has an identity
+
IAM controls what it can access

That's a much better foundation for cloud-native workloads.

And it reinforces another principle: Least Privilege Access.

RAG may sound like an AI problem.

But once the application touches Object Storage, Data Science, Model Deployment, or Container Registry, identity becomes part of the AI architecture too.

What the Pipeline Looks Like Now

At this point, the data flow looked like this:

Notice what's missing.
The LLM.

And that's intentional.

Before generating a single response, we've already had to solve:

Data ingestion
Persistence
Decoupling
Raw-data management
Chunking
Batch processing
Pagination
Data validation
Resource isolation
IAM

That's the point.

The Bigger Lesson

Building RAG made me appreciate something that's easy to forget in the GenAI hype cycle:

RAG is as much a data-engineering problem as it is an AI problem.

The LLM only sees what the retrieval pipeline gives it.

The retrieval pipeline only searches what was indexed.

The index only contains what was embedded.

Embeddings only represent what was processed.

And processing can only operate on what ingestion successfully collected.

So the dependency chain is:

Data Quality -> Chunk Quality -> Embedding Quality -> Retrieval Quality -> Context Quality -> LLM Response Quality

A failure near the beginning propagates through everything downstream.

A bigger model won't fix missing data.

Prompt engineering won't fix a dataset accidentally truncated at 1,000 objects.

And a re-ranker can't rank documents that never entered the index.

Engineering Takeaways

This stage changed how I approached the rest of the project.

1. Treat Object Storage as an architectural boundary
It keeps ingestion, processing, and downstream workloads decoupled.
2. Keep raw data immutable
Processing strategies will change. Your original data shouldn't disappear with them.
3. Design for bounded memory
Batching becomes increasingly important as datasets grow.
4. Never assume an API returned everything
Pagination bugs can silently create incomplete ML datasets.
5. Validate every stage
Counts, failures, skips, and artifact counts are simple but powerful observability signals.
6. Separate workloads by responsibility
Scraping, preprocessing, embedding generation, and inference don't necessarily belong on the same infrastructure.
7. Treat cloud identity as part of application architecture
Resource Principals and IAM matter just as much as your Python code once services start communicating.

And Now We Can Finally Talk About Vectors

We started with raw technical documents.

We now have cleaned, chunked, reproducible data sitting in Object Storage.

Roughly 70,000 chunks are waiting.

But there's one problem.

FAISS can't search text.
It searches vectors.

So somehow this:

"How does Kubernetes service discovery work?"

needs to become something like:

[0.018, -0.042, 0.091, ..., 0.027]

And documents discussing Kubernetes networking need to end up close to that query in vector space.

That's where things get considerably more interesting.

Because in Part 3, we're going from:

Words → Numbers → Meaning → Search
We'll look at:

*Embeddings. Sentence Transformers. 384-dimensional vectors. Similarity search. FAISS. IVF indexes. Centroids. nlist. nprobe.
*
And why FAISS told me that 999 training vectors weren't enough.

That warning ended up teaching me more about vector search than simply getting the code to run ever could.

If you'd like to follow my work or connect, you can find me here:

Building an Enterprise GenAI Platform on OCI — Part 1: Designing the Architecture

Yugandhar Surya — Fri, 17 Jul 2026 16:45:34 +0000

Every production system starts with a diagram, not code.

When I started this project, I thought I would spend most of my time choosing and configuring an LLM.

Instead, I spent days thinking about something else.

⚠️ New here? This is Part 1 of the series. If you'd like the backstory on why I built this platform, the mistakes I made, and the lessons from my first production RAG system, read Part 0 first:

Building an Enterprise GenAI Platform on OCI – Lessons from My First Production RAG System

Then come back here for the architecture deep dive.

The architecture.

The moment I stopped thinking about the model and started thinking about the system, everything changed. A production GenAI application needs far more than an LLM, it needs reliable data pipelines, efficient retrieval, scalable infrastructure, and secure deployment. So before writing any code, I asked myself: What exactly am I building? A chatbot, a RAG application, or an enterprise GenAI platform?

Thinking Beyond a Chatbot

Most RAG tutorials follow roughly the same flow.

Documents -> Embeddings -> Vector Database -> LLM -> Answer

For learning, that architecture works just fine. But production isn't just about retrieving documents and calling an LLM, it's about building a system that can ingest data continuously, scale reliably, and evolve over time. That's when I realized I wasn't designing a chatbot; I was designing a platform.

The Architecture

The architecture naturally separated into two independent pipelines.

1. Offline Pipeline

This pipeline prepares knowledge before any user sends a query.

Its responsibility is to transform raw information into something an LLM can retrieve efficiently.

The flow looks like this:

External Sources -> OCI Compute -> Object Storage -> Cleaning & Chunking -> Embedding Generation -> FAISS Index -> Artifacts

These jobs are compute intensive but don't need to run for every request.

Keeping them separate allows the knowledge base to evolve independently of the inference service.

2. Online Pipeline

Once the data is prepared, the online pipeline takes over.

User -> Oracle APEX -> FastAPI -> Query Embedding -> FAISS Retrieval -> Prompt Construction -> LLM -> Streaming Response

Unlike the offline pipeline, every millisecond matters here.

The goal isn't just to generate an answer.

The goal is to generate the right answer quickly.

That changes how every component is designed.

Why Split the Platform?

One question I kept asking myself was:

Why not put everything inside a single application?

Because different workloads have different requirements.

The ingestion pipeline may process thousands of documents in batches.

The inference service should respond within seconds.

Trying to solve both problems inside one application makes scaling harder and deployments riskier.

Separating the platform into independent services means each layer can evolve without affecting the others.

Designing Around Responsibilities

Once I stopped thinking about frameworks and started thinking about responsibilities, the architecture almost designed itself. Every component had one job: ingest data, process it, retrieve knowledge, generate responses, or serve the application. That's how the platform naturally evolved into six layers.

Layer	Responsibility
Data Ingestion	Collect raw knowledge
Storage	Persist datasets, embeddings, indexes, and artifacts
Processing	Clean, normalize, and chunk documents
Retrieval	Search relevant knowledge
Inference	Generate grounded responses
Application	Provide the enterprise user interface

Notice that none of these layers are tied to a specific technology and that's intentional. Good architecture should outlive the tools used to implement it. FastAPI can be replaced by another framework, FAISS by a managed vector database, or TinyLlama by a different LLM. The technologies may evolve, but the responsibilities of each layer remain the same.

Architecture Principles

While designing the platform, I kept returning to a small set of principles.

Modular – Every layer should have a single responsibility.
Cloud-native – Managed services where they make sense.
Scalable – Independent scaling for ingestion and inference.
Observable – Health, metrics, and logging built in.
Secure – Least-privilege access between services.
Cost aware – Optimize for learning without unnecessary infrastructure.

These principles influenced every decision that followed.

Why OCI?

One of the goals of this project was to explore how far a production-oriented GenAI platform could be built using Oracle Cloud Infrastructure.

OCI offered everything I needed for this stage of the project:

Compute for data collection
Object Storage for the data lake
Data Science for experimentation
Container Registry for deployment
IAM for secure access
Oracle APEX for the enterprise application layer

Rather than introducing every service at once, each will appear naturally as the project evolves throughout this series.

One Lesson I Learned Early

I started this project thinking the hardest part would be choosing the right LLM and refining prompts. Instead, I found myself spending far more time designing how data moved through the system. That was the moment I realized a great GenAI application isn't defined by its model, it's defined by its architecture.

What's Next?

Now that the blueprint is complete, it's time to start building.

In Part 2, we'll implement the first stage of the platform:

Collecting data with OCI Compute
Streaming it directly into Object Storage
Designing the data lake
Preparing thousands of documents for downstream processing

Because before we can build a great retrieval system, we first need great data.

Building an Enterprise GenAI Platform on OCI — Lessons from My First Production RAG System

Yugandhar Surya — Wed, 08 Jul 2026 13:28:51 +0000

I thought building a RAG application would be straightforward.

Scrape the data. Generate embeddings. Build a FAISS index. Connect an LLM. Expose an API.

That's what most tutorials make it look like.

Reality was very different.
Every optimization solved one problem and uncovered another.

Even after switching to a lightweight language model, every response still took several seconds.
That's when I realized the bottleneck wasn't the LLM. It was the architecture itself.

That realization completely changed the direction of this project. Instead of building just another chatbot, I decided to design a production-oriented GenAI platform on Oracle Cloud Infrastructure (OCI), focusing on data engineering, retrieval quality, inference optimization, deployment, observability, and LLMOps.

Over the past few weeks, I built this platform while exploring Oracle Cloud Infrastructure, documenting every architectural decision, optimization, and failure along the way.

Some Problems I Didn't Expect

What started as a simple RAG prototype quickly turned into an engineering exercise.

Along the way, I encountered challenges I hadn't anticipated:

Large Docker images and slow deployments
Infrastructure limitations on OCI Free Tier
Container architecture mismatches (ARM vs. AMD)
Retrieval and inference latency bottlenecks
IAM and networking complexities during deployment

Every challenge forced a new architectural decision and those decisions shaped this entire series.

Why Another RAG Series?

If you've searched for RAG tutorials, you've probably noticed a common pattern.

Most articles demonstrate how to connect an LLM to a vector database and answer questions from a PDF. While that's a great way to understand the basics, it leaves out many of the challenges you'll encounter when building a real-world system.

Along the way, I found myself asking questions like:

Where should the data actually live?
How do you keep it synchronized?
Why is retrieval slower than expected?
Should you use FAISS or a managed vector database?
How do you deploy and scale inference?
What happens when multiple users hit the API simultaneously?

Those were exactly the questions I started asking while building this project.

The Goal

I wanted to understand why production systems look so different from GitHub demos.

The objective wasn't simply to make an LLM answer questions, it was to understand the complete lifecycle of a production GenAI application.

That meant learning how to:

Build scalable data pipelines
Design an efficient retrieval layer
Optimize inference latency
Containerize and deploy services
Integrate with enterprise applications
Apply MLOps and LLMOps principles
Keep operational costs under control

Why Oracle Cloud Infrastructure(OCI)?

Most tutorials use AWS or Azure.

I intentionally chose Oracle Cloud Infrastructure (OCI) because I wanted to explore its AI ecosystem and understand how enterprise GenAI applications could be built using Oracle's cloud services.

The platform combines OCI Compute for ingestion, Object Storage as a data lake, Data Science for experimentation, Model Deployment for inference, and Oracle APEX as the application layer.

Choosing OCI also introduced a unique set of challenges around resource limits, deployment strategies, container architectures, and cloud-native design decisions many of which I'll cover throughout this series.

What We'll Build

By the end of this series, we'll have built an enterprise-oriented RAG platform capable of:

Automated data ingestion
Cloud-native data lake
Semantic search using embeddings
Hybrid Retrieval (FAISS + BM25)
Cross-Encoder Re-ranking
Prompt Engineering
Streaming responses
Guardrails
Dockerized deployment
OCI Model Deployment
Oracle APEX integration
CI/CD pipeline
LLMOps best practices

More importantly, we'll discuss why each architectural decision was made, along with the trade-offs involved.

Note:

This isn't another "build a chatbot in 15 minutes" tutorial. It's an engineering journal documenting the architectural decisions, trade-offs, failures, and lessons learned while building an enterprise GenAI platform on OCI.

If you're looking for a copy-and-paste RAG tutorial, this series probably isn't for you.

But if you're interested in understanding how enterprise GenAI systems are designed from data ingestion and retrieval to deployment, observability, and optimization. I hope you'll find this journey worthwhile.

Enterprise AI systems aren't built by choosing the largest model.
They're built through hundreds of engineering decisions.
This series is about those decisions.
See you in Part 1.

Series Roadmap

📚 This Series (10 Parts)

Architecture & Planning
Data Engineering Pipeline on OCI
Embeddings & Vector Search
Hybrid Retrieval and Re-ranking
FastAPI Inference Pipeline
Docker & OCI Model Deployment
Oracle APEX Integration
LLMOps & CI/CD
Performance Optimization
Lessons Learned & Production Considerations

Before we dive into the next article, I'd love to hear from you:

Have you built a RAG application beyond a simple proof of concept?
What was the biggest challenge you encountered—retrieval quality, latency, deployment, or something else?
Are you using OCI, AWS, Azure, or another cloud platform for GenAI workloads?
Is there a specific topic you'd like me to cover in more detail as this series progresses?

Feel free to share your experiences in the comments. I'm always interested in learning how others are approaching these problems.

If you'd like to follow my work or connect, you can find me here: