Your Chatbot Demo Works. That Doesn't Mean It's Ready for 10,000 Users.
An architect's walkthrough of what has to exist around the LLM before you ship an AI assistant to real customers
The LLM call is the visible part. Almost everything that decides whether you survive launch week sits below it.
Picture this. An engineer at an e-commerce company — let's call it BlipKart — has built a chatbot. It answers questions, looks up products and even walks a customer through placing an order. The demo goes well. The CEO is impressed and asks for one thing: put it in front of 10,000 users by Monday.
It's a familiar moment, and it hides a trap. A demo proves that a model can produce a good answer when a friendly person asks a friendly question. Production asks a different question: can the system keep producing safe, correct, affordable answers when thousands of strangers use it at once, some of them trying to break it, while your model provider has a bad day?
Calling an LLM API is the easy part. Running a dependable product around it is the job. After fifteen years designing payments, cards and banking platforms — where "it worked in the demo" has never once been an acceptable sign-off — I find the same disciplines apply almost one-for-one to AI systems. Here is how I'd take BlipKart from demo to production.
1. Start with requirements you can measure
Before drawing a single box, separate two conversations that teams constantly blur.
High-level design is the map: which major components exist and how they talk. For BlipKart that's an API gateway, an intent router, a cache, a retrieval-augmented generation (RAG) pipeline, a model gateway and the commerce APIs.
Low-level design is the inside of each box: vector-store replicas and regions, chunk sizes, schemas, prompt templates, the classes that implement them. Mixing the two levels in one review is how a team spends an hour debating chunk size while nobody notices there is no authentication layer.
Then write the requirements down in two lists.
Functional — what the bot must do: answer FAQs, track orders, process refunds, hand certain conversations to a human.
Non-functional — how well it must do it, with numbers stakeholders sign off on:
- P95 latency under 3 seconds
- Monthly spend within ₹3 lakh
- Zero PII leaks
- 99.9% availability
If those numbers aren't written down, every later trade-off becomes an argument about opinions instead of a check against a target.
Do the capacity maths — and don't trust the average
10,000 users × 5 messages a day = 50,000 messages. Spread evenly across 86,400 seconds, that's roughly 0.58 requests per second. It sounds trivial. It's also misleading.
Same 50,000 messages, very different load. Even a gentle evening peak is more than double the daily average — and a sale campaign or push notification can be far worse.
Traffic clusters: lunch breaks, evenings, the minute a promotional push lands. Then add retries, multi-turn conversations and tool calls that each fan out to several model invocations. Size for the peak minute, not the day, and convert messages into tokens per month so you can talk to your provider about rate limits and reserved capacity before launch, not during an incident.
2. Protect customers at the boundary
A reference shape for the system. Everything inside the trust boundary runs before a single token is generated.
Authenticate before any account action
"Refund my last order" means nothing until the system knows whose last order. Identity has to be established with proper standards — OIDC, OAuth-based sign-in, SSO — using an identity provider such as Keycloak (or whatever your enterprise already runs).
One point I'd push harder than most write-ups do: authorisation must live in the tools, not in the prompt. The order-lookup tool should derive the customer ID from the verified token, never from text the model passes in. If the model can be talked into asking for "order 4471 for customer 9902", the only thing that should stop it is the API refusing — not the model's good manners.
Let the gateway absorb abuse
An API gateway is the right place to enforce per-user request limits and daily or monthly token budgets, before traffic reaches either your application or the LLM. Pair it with a web application firewall and load balancing. Cloudflare, NGINX, Kong, Apigee or your cloud provider's native gateway can all do this. Tune the limits to protect the provider and your bill without punishing a genuine customer who asks five follow-up questions.
Check what goes in — and what comes out
Prompt injection is not theoretical. There's a widely reported case of a car dealership's chatbot appearing to agree to sell a vehicle worth around $70,000 for $1 after a user instructed it to agree with everything. Whatever the legal outcome of such cases, the design lesson is clear: define, in advance, what the bot is never allowed to promise.
- Input guardrails flag disallowed or manipulative requests.
- Output guardrails catch PII, prices or commitments the bot has no authority to make before the response reaches the customer.
Legal and compliance should own the list of prohibited responses. Engineering owns enforcing it.
3. Route every request to the right place
Not every message deserves a large language model. A greeting, an order-status question, a refund request and a policy question are four different workloads.
An intent router sits in the middle of the design and picks the path:
- Greetings and thanks → a fixed response
- Order status, prices, refunds → commerce APIs and tools
- Policy and how-to questions → RAG
- Ambiguous sales or support questions → a classifier, and only then an LLM
Cache what's stable. Fetch what's live.
Semantic caching is a cost win for stable answers and a correctness bug for anything live or personal.
Semantic caching matches questions by meaning, so "how do I pay with UPI?" and "can I use UPI to pay?" can share one answer. That saves real money on repetitive FAQs.
It is also a sharp tool. Two rules I'd treat as non-negotiable:
- Never cache live facts or account actions. Prices come from the pricing API. Order status comes from the order service. A return request triggers a tool, not a remembered answer.
- Never let a cache key cross a customer boundary. A cached answer that contains anything personal is a PII leak waiting to happen.
Give every cached entry a TTL and an explicit invalidation trigger.
Keep RAG content honest
Policy PDFs and troubleshooting guides get chunked, embedded and indexed in a vector store — Qdrant, Chroma, Pinecone, pgvector and others. The failure mode isn't retrieval quality on day one; it's staleness on day ninety, when the return policy changed and the bot is still quoting the old one.
Wire change data capture into the content source. When a document changes, re-index only the affected chunks and evict the cache entries built on them. No nightly "re-index everything" job, and no stale answers lingering for a week.
Know when to hand over to a human
A customer disputing a charge or about to cancel a membership needs judgement and empathy, not a generated paragraph. Decide up front which intents the AI must never handle alone. And if the bot is allowed to offer something with commercial value — a retention discount, a goodwill credit — write explicit rules for when, how much and how often.
4. Control cost without lowering quality
Minimum sufficient intelligence
Climb only as high as the task needs. Every rung up costs more and behaves less predictably.
This is the principle I'd put on the wall: use the least complex mechanism that meets the quality bar.
- "Thanks!" needs a rule, not a model.
- "Where's my order?" needs an API call, not a model.
- Sorting a known set of request types is a job for a traditional ML classifier.
- A standard return-policy answer is fine on a small, cheap LLM.
- A split-shipment refund with a partial return and a voucher? That may genuinely need a larger reasoning model.
Rules are cheap, fast and predictable. Models are flexible. Spend flexibility only where you need it.
Route models by task, and prove it with evals
Model routing sends each request to the model best suited to it. The choice shouldn't be made on brand or parameter count — build an evaluation set from real BlipKart conversations and score candidates on accuracy, latency, cost and policy compliance. Re-run it every time you change a prompt or a model version.
Make spend visible before it becomes a surprise
If the monthly bill quietly drifts from ₹3 lakh to ₹6 lakh, you want to find out on day eight, not in the invoice. Set token budgets per user or account, track spend per model and per intent, and alert as you approach thresholds. Review regularly which intents actually justify the expensive model.
5. Build for spikes and outages
Queues absorb bursts; the model gateway hides provider failures behind a tested fallback.
Put a model gateway between you and your providers
An AI/model gateway keeps provider keys out of application code, centralises rate limits and token budgets, and can fail over to a second provider when the primary returns errors or throttles you. LiteLLM and Vercel AI Gateway are two examples.
The catch: a fallback you haven't tested is not a fallback. Prompts tuned for one model often behave differently on another — different formatting, different refusal behaviour, different tool-calling quirks. Run your evaluation suite against the fallback provider too, and load-test it at the traffic it would actually receive during a failover.
Queue, throttle, scale
When requests arrive faster than you can serve them, queues smooth the curve. Backpressure and throttling slow intake before you overwhelm the provider. Kafka or RabbitMQ can hand long-running work to background workers so the chat API doesn't block. Load balancing and autoscaling add and remove compute as traffic moves.
Streaming responses over server-sent events shows text as it's generated. It doesn't make the total answer faster — but it dramatically reduces the time a customer stares at a spinner, which is what they actually perceive.
Make releases and recovery boring
Infrastructure as code (Terraform) and CI/CD (GitHub Actions, GitLab, Jenkins) make deployments repeatable and reviewable. For AI systems I'd add one more gate: an automated evaluation run that blocks the deploy if quality drops. Every release needs a tested rollback path, and the team should rehearse recovery before a real outage forces them to improvise.
6. Measure health — and correctness
A green uptime dashboard doesn't mean the bot is right. A chatbot can be fast, available and confidently wrong.
Track the operational signals — P95 latency, token usage, spend, error rates — and the behavioural ones: did it call the right tool? Did the refund flow complete? Did the answer match current policy? Tools like Datadog, New Relic, LangSmith and MLflow each cover parts of this.
Set alerts on the targets you agreed in step one: latency above 3 seconds, spend crossing budget, guardrail trigger rates spiking. Route them by severity — email for trends, a page for an outage. Keep end-to-end traces so an engineer can see exactly which hop failed. And keep re-checking behaviour after every change to prompts, models, policies or integrations, because each one can shift answers in ways no unit test will catch.
The production-readiness checklist
Twelve questions to answer before opening access to real customers.
BlipKart's prototype is a good start. But production readiness was never about the model — it's about the system around it.
Before you open the doors to 10,000 users, confirm the requirements and the peak-traffic maths. Check authentication, abuse controls, guardrails, data freshness, human handoffs, cost limits, fallbacks, scaling, monitoring and recovery. Then run a load test at the expected peak and, just as importantly, prove the system fails safely — that when something breaks, the customer gets a polite handoff, not a wrong refund.
Every one of these choices is a trade-off between risk, quality, latency and budget. There's no universal stack. But there is a universal question worth asking before Monday:
If this goes wrong at 8 pm on launch day, what happens next — and who finds out first: you, or your customers?
I'm Raj Mohamad Sabibullah, a Senior Solution Architect working across payments, cards, banking platforms and applied AI. If you're taking an AI assistant from prototype to production, I'd like to hear what tripped you up — leave a response below or connect with me on LinkedIn.
Tags: Artificial Intelligence · System Design · LLM · Software Architecture · Generative AI
Top comments (0)