I wanted to build a simple RAG system.
That sentence aged badly.
The original problem was actually pretty simple.
We were organizing a departmental event called Kairos, where participants had to study company information and create pitch decks.
The organizers had a lot of company reports.
The participants had to read them.
And the obvious solution was:
Print everything.
Except there was one tiny problem.
There were a lot of pages.
And there were a lot of participants.
Printing enough copies for everyone wasn't exactly friendly to the budget.
So I had a stupidly simple thought:
Why don't we just put all the documents into an AI system and let participants ask questions?
And that's how I accidentally created a much bigger problem for myself.
Problem #1: "Just build a RAG"
If you've built RAG systems before, you know how innocent that sentence sounds.
Upload PDFs → chunk them → embed them → retrieve relevant chunks → send them to an LLM.
Easy.
Until you actually try to make people use it.
I needed:
- PDF ingestion
- embeddings
- vector search
- retrieval
- reranking
- LLM generation
- citations
- a frontend
- a backend
- authentication
- rate limiting
- logging
- an admin dashboard
- and eventually, somehow, 100+ people using it at the same time
And I had one major constraint:
I didn't have a budget.
Not "small startup budget."
I mean ₹0 infrastructure budget.
Problem #2: The LLM was going to bankrupt my imaginary company
The next question was obvious:
Which LLM should I use?
Paid APIs?
No.
I didn't have money for that.
So I started looking at free API tiers.
I ended up using Gemini as the primary model and OpenRouter/Nemotron as a fallback.
Problem solved?
Absolutely not.
Free APIs have limits.
And when you have one user asking a question, rate limits are someone else's problem.
When you have 100+ users asking questions...
Congratulations. You have invented a distributed systems problem.
Problem #3: "What if we just use multiple API keys?"
This was probably my favourite hack.
Instead of relying on one API key, I created multiple API lanes.
Something roughly like:
Incoming Request
|
v
LLM Gateway
|
+-----------+-----------+
| |
Gemini lanes OpenRouter lanes
G01...G10 N01...N10
| |
+-----------+-----------+
|
v
Response
I ended up with 20 lanes:
- 10 Gemini lanes
- 10 OpenRouter/Nemotron fallback lanes
Each lane had its own quota and concurrency control. The backend could distribute requests across them instead of hammering a single API key until it died. And because multiple users could hit the system simultaneously, I used PostgreSQL-based locking to coordinate access.
Was this the most elegant infrastructure ever created? No. Was it considerably cheaper than paying for a giant API bill? Yes.
Problem #4: Now I needed a server
At this point I had an AI system. Now I needed somewhere to run it.
I looked at cloud hosting. Then I looked at the prices. Then I looked at my budget. Then I looked back at my laptop.
We reached an agreement. My laptop was now the cloud.
I ran the application locally and exposed it through ngrok
At some point I stopped calling it a laptop and started calling it production infrastructure. It made me feel better.
Problem #5: What if everyone asks questions at once?
The system wasn't being built for me. It wasn't: "Look, here's my RAG demo." People were actually going to use it. Potentially 100+ people. And I didn't want the first question on event day to be: "Why is the website down?"
So I added:
- Per-team question quotas
- Request logging & rate limiting
- API key routing & fallback handling
- Concurrency controls & event timers
- Monitoring/admin views
I limited each team to 10 questions. This wasn't only about saving API calls—it also made the event more interesting because we could later look at the prompts participants were asking and evaluate how they were using the system.
Problem #6: RAG started lying to me
This was where the project became more interesting technically. I didn't just want: "Here are some chunks that look relevant." I wanted the system to actually reason about what kind of information it needed to retrieve.
- Two-Stage Retrieval: The first stage interpreted the question and created a retrieval plan. The second stage actually searched the company knowledge base.
- Permission Boundaries: I kept the internal instruction knowledge separate from the company documents that participants were allowed to access.
- Page-Level Citations: Instead of "According to the document...", the system pointed participants toward the exact page containing the information.
Event Day
The system was running on my laptop. The laptop was connected to the internet. ngrok was exposing it. The APIs were running through free quotas. PostgreSQL was handling state. Qdrant was handling vector search. The RAG pipeline was running.
And then... people started using it. Questions came in. Answers came back. People were actually using the system to search through the company information.
And it worked.
No paid cloud infrastructure. No expensive LLM subscription. No giant GPU server. No fancy production cluster. Just a lot of engineering, questionable financial decisions, and one laptop that was working much harder than it deserved to.
What I Actually Learned
The biggest lesson wasn't about RAG. It was about constraints.
When you have unlimited resources, the solution is often:
- "Just use a bigger server."
- "Just use the paid API."
- "Just scale horizontally."
When you have ₹0, you can't say that. You have to ask: "Okay. What can I change instead?"
- Free API limits became an API-routing problem.
- No cloud budget became a deployment problem.
- Too many users became a concurrency problem.
- Large documents became a retrieval problem.
And eventually, all those little problems became an actual system. That's probably the part of the project I'm most proud of.
The Final Stack
| Layer | Technologies |
|---|---|
| Frontend | React, Vite, TypeScript, Tailwind, PDF.js |
| Backend | FastAPI, Uvicorn, Async REST APIs, SSE streaming |
| AI & Retrieval | Gemini, OpenRouter / Nemotron, SentenceTransformers (BAAI/bge-small-en-v1.5), Two-stage retrieval, Reranking, Adaptive response depth |
| Data & Storage | PostgreSQL, SQLAlchemy, Qdrant |
| Infrastructure | Docker, Docker Compose, ngrok (and my laptop) |
🔗 GitHub Repository
The complete project is open source:
https://github.com/Pawan-19012006/Kairos-RAG-System
One Thing I'd Do Differently
If I rebuilt this today, I'd spend much more time designing the evaluation and observability layer from the beginning.
Getting a RAG system to answer is relatively easy. Getting it to answer correctly, consistently, with the right evidence, under real usage constraints is where the interesting engineering begins.
And that's where this project stopped being "I built a chatbot" and became "I built a system that had to survive actual users."
That's a much more fun problem.
P.S. If you're building something similar and you're worried because you don't have a huge cloud budget: Don't let the lack of money decide whether the project is possible. Sometimes the most interesting engineering happens when you have no other option.
And sometimes... your laptop becomes the cloud. 😅
Top comments (0)