Most "scalable system design" articles describe theoretical systems handling millions of users that neither the writer nor the reader will ever actually build. This isn't that. This is what I actually learned architecting a multi-tenant AI chatbot builder — the kind of system where dozens of businesses share one backend, each with their own bots, knowledge bases, and messaging integrations — on a student budget, with free-tier infrastructure.
The system doesn't serve millions of requests a day. But the design questions you hit at 100 requests a day are the same ones you hit at 100,000 — you just get more room to be wrong and fix it. Here's what I learned about retrieval-augmented generation (RAG) and multi-tenancy before I actually had scale to worry about.
1. Design for tenants before you design for traffic
The chatbot builder needed to support many businesses (tenants), each with their own bots, knowledge bases, WhatsApp/Telegram connections, and usage limits — all on one shared backend.
The tempting shortcut is to bolt tenancy on later: one database, a userId field sprinkled into every collection, and hope nothing leaks. I went the other way and treated tenant isolation as a first-class part of the schema from day one — every document that mattered (bots, conversations, knowledge base chunks, subscription state) carried a tenant identifier that every query had to pass through, no exceptions.
The tradeoff: it's slower to build features early on, because you can't cut corners on "whose data is this." The payoff: when you add a new feature later, you're not doing an emergency audit of every collection to figure out where tenant data might be leaking into another tenant's view. Isolation-by-default is one of those decisions that's nearly free at the start and extremely expensive to retrofit.
2. Not every request needs the same amount of work
The chatbot's core feature is retrieval-augmented generation: a user asks a question, the system searches a knowledge base, and an LLM answers using the retrieved context. The naive version of this does one thing — embed the query, run a vector search, stuff results into a prompt — for every single request, regardless of how the question is phrased or what's actually being asked.
That's wasteful. A one-word follow-up question and a complex multi-part query don't need the same retrieval effort, but they'll cost the same if you treat them identically.
I ended up building the retrieval pipeline in tiers — cheaper, faster paths handle requests that don't need deep retrieval, and progressively heavier tiers only kick in when the query actually warrants it. In production this pipeline consistently retrieves relevant context with around 90% accuracy at roughly 780ms average latency — good enough to feel responsive in a chat interface, without paying full retrieval cost on every message.
The general lesson isn't "build a four-tier pipeline." It's: look for the cases where you're doing expensive work uniformly across requests that don't uniformly need it. That gap is usually where your easiest scalability wins live, and it's almost always cheaper than reaching for more infrastructure.
3. Your real bottleneck is probably someone else's API
The platform leans heavily on third-party APIs — Gemini for embeddings and generation, WhatsApp's Cloud API and Telegram's Bot API for messaging. These have their own rate limits, latency, and failure modes that you don't control, and at small scale it's easy to forget they're there until a burst of traffic hits one at once.
Multi-tenant WhatsApp integration made this concrete: each business brings its own WhatsApp number via Meta's Embedded Signup flow, but the app still has to be a good citizen of Meta's platform-wide rate limits across every tenant combined, not just per-tenant. That means rate limiting and backoff logic had to be designed at the application layer, not assumed away because "each tenant has their own number."
The lesson: when you sketch your architecture diagram, the boxes representing external APIs deserve as much design attention as the boxes you control. They will rate-limit you, they will have outages, and they will not always fail gracefully. Plan for it before you need to.
4. Cheap infrastructure, disciplined architecture
This system runs on free/low-tier hosting (Render), not a fleet of load-balanced servers — because that's the reality of building as a student. That constraint was useful. It forced decisions that scale regardless of the infrastructure underneath them: strict tenant isolation, tiered retrieval instead of brute-force retrieval, and rate-limit-aware integrations instead of naive ones.
The uncomfortable truth is that most "scalability" problems people reach for infrastructure to solve are actually architecture problems. Fixing the shape of the system first, on cheap infrastructure, means that if I do need to scale the infrastructure later, I'm scaling something already designed to be scaled — not retrofitting scalability onto something that was never built for it.
Takeaways
- Isolate tenants at the schema level from day one — retrofitting isolation later means auditing everything you've already built.
- Don't do uniform work for non-uniform requests — tiering retrieval effort to what a query actually needs is often a bigger win than adding infrastructure.
- Design for the external APIs you don't control — they're often the actual bottleneck, not your own code.
- Fix the architecture before reaching for infrastructure — cheap hosting with disciplined design scales further than you'd expect.
None of this required expensive infrastructure or massive traffic to learn — it just required paying attention to why a design choice mattered, not just whether it worked in the demo. That's the part that transfers, regardless of what you're building next.
I'm a final-year CS student building AI-integrated MERN stack applications. Part two of this series covers the async and real-time side of things, built for a very different kind of load — drop a comment if you've hit similar RAG or multi-tenancy tradeoffs.
Top comments (0)