Part 1 of a series on building a production banking AI chatbot.
When our team kicked off the banking chatbot project, I genuinely thought my job was already solved.
I was the Node.js developer. My part sounded almost embarrassingly simple:
- Open a WebSocket when a user connects
- Send their message to the AI service
- Stream the response back
That's it. That's the whole job.
WebSockets are second nature to me — I've built enough real-time systems that I could probably wire one up half asleep. So I walked out of the kickoff meeting thinking, this one's going to be easy.
Biggest lie in software engineering.
User
│
▼
WebSocket
│
▼
LLM
│
▼
User
That was architecture v1. Four boxes. I was proud of it. It lasted less than a week.
Chapter 1 — The IPL Question
A few days in, during a sync with the AI team, I asked a question that was more instinct than insight.
Me: "What happens if someone asks the bot who won yesterday's IPL match?"
Nobody answered right away. Someone laughed, thinking I was joking. I wasn't.
AI Lead: "...Wait. What does happen?"
We opened the app and tried it, right there in the call. The bot took the question, searched a knowledge base full of FD rates and loan terms, found nothing close, and confidently answered anyway — something vaguely about "please contact your branch for match-related queries," stitched together from whatever fragments were closest in the vector search.
It was funny for about ten seconds. Then it wasn't.
Our chatbot had no concept of "this isn't mine to answer." And I already knew, from years of shipping things real users touch, that given half a chance, someone will ask the weird question. Not out of malice. Just because a chat box invites it.
So we needed a gatekeeper — something to sit in front of everything else and decide, is this even our problem?
That's intent detection. Not because it sounded good in a slide. Because a laugh on a call turned into a real gap the moment we tested it.
User
│
▼
Intent Detection
│
▼
Banking-related?
│
├── Yes → RAG → LLM → User
│
└── No → Reject / redirect
The AI team gave me one endpoint. Send a query, get back an intent and some metadata. My side was maybe ten lines. I was happy.
I didn't know it yet, but I'd just signed up for a much longer story.
Chapter 2 — "Can We Cache This?"
The intent layer worked. Demo went well. Client was happy. Everyone smiled.
Then someone actually measured it.
Every request — even something as boring as "what's today's FD rate" — now had to pass through a full LLM call just to figure out where the question should go, before it even reached the LLM that would answer it. That detour cost 200–300ms. Every single time.
We brought it up in the next review.
AI Lead: "Suraj, can we cache this?"
Me: "Yeah, sure. Redis."
Confident. Instant. The kind of answer you give when you've said "Redis" so many times in your career it comes out before you've actually thought about the question.
Five seconds later, it caught up with me.
Me: "...Actually — cache what, exactly?"
The room laughed. I laughed too, but I meant it. I genuinely didn't have an answer yet.
Chapter 3 — Back to the Whiteboard. Twice.
My first instinct was the instinct every backend dev has: store the query, store the response, key-value, done.
"What is today's FD rate?" → Cache → HIT ✅
Felt great — until the very next test.
"Can you tell me today's fixed deposit interest rate?" → Cache → MISS ❌
Same question. Same intent. Different string. Users don't type the same sentence twice in their life, and exact-match caching only knows characters, not meaning. Back to the whiteboard.
Attempt two felt smarter. Cache by intent, not by raw text. If the classifier already told us this was an FD_RATE question, why not key the cache off that instead?
Intent: FD_RATE
│
▼
Cache
Looked brilliant for about a day. Then someone tried two questions back to back:
"What is FD rate?" → FD_RATE
"What are FD benefits?" → FD_RATE
Same intent bucket. Completely different questions. Completely different answers. If I served the cached FD rate to someone asking about FD benefits, I'd have handed them a wrong answer with full confidence, which is arguably worse than no cache at all.
Back to the whiteboard. Again.
I was stuck for most of that week, and starting to genuinely doubt whether "cache the AI layer" was even a solvable problem with the tools I already knew.
Chapter 4 — The Tissue Paper
We were on a coffee break when our architect grabbed a napkin off the table. Didn't say anything at first. Just wrote two words on it and slid it across.
Semantic Cache
"Read about this tonight," he said, and went back to his coffee.
I nodded like I understood. I did not understand.
That night I opened the Redis documentation. Read the vector search page once. Closed the tab.
Opened it again.
Redis supports... vector search?
Since when?
I checked the changelog, half convinced I'd misread something. It hadn't just landed last week. It had been sitting there for a while. I'd just never needed it before, so I'd never gone looking.
Redis hadn't changed overnight. My understanding had.
Somewhere around midnight it actually clicked: instead of matching the exact text of a query, you convert it into an embedding — a vector that captures what it means — and compare that against previously cached embeddings. Close enough, and you serve the cached answer instead of paying for retrieval and generation all over again.
User Query
│
▼
Embedding
│
▼
Redis Vector Search
│
▼
Similarity ≥ 0.90 ?
│
├── YES → Return cached answer (fast path)
│
└── NO → Go to RAG (full path)
Now "what's today's FD rate" and "can you tell me today's fixed deposit interest rate" land close enough in vector space to hit the same entry, while "FD benefits" stays far enough away to miss it. Meaning, not characters.
We built it that week. I went home Friday thinking, for the first time on this project, I'm actually becoming a good backend engineer.
Chapter 5 — Monday Morning
Monday morning. Coffee in one hand, laptop in the other. I opened Grafana expecting to see a victory lap.
Cache Hit Ratio
8%
I stared at it for a solid ten seconds. Refreshed. Same number.
How is this even possible?
Turns out picking 0.90 as a similarity threshold because a Medium article used 0.90 is not the same as picking 0.90 because you tested it against your own traffic. Some genuinely similar questions were landing just under the line and missing. A few unrelated ones were sneaking just over it. We spent the better part of two days pulling real query logs and manually eyeballing where the line actually needed to sit.
Once it did, the ratio climbed for real. Costs dropped. Confidence, briefly, restored.
Then we measured end-to-end latency.
Barely moved.
We tried streaming next — sending tokens back as they were generated instead of waiting for the full response.
LLM
│
▼
token → token → token → token → ...
Users felt faster. The client loved it in the demo. But the actual time-to-completion, measured end to end, hadn't dropped by a single millisecond. Streaming fixed how the wait felt. It did nothing to the work itself.
Chapter 6 — Cutting Vegetables With a Sword
A senior engineer sat in on our next architecture review — brought in from outside the team to sanity-check the design. He looked at the diagram on the screen for a while. Then he smiled.
"You're cutting vegetables with a sword."
The room laughed, not entirely sure why yet.
He didn't explain it.
Chapter 7 — The Knife
He let it sit for a second, then pointed at the box on the diagram labeled Intent Detection — the same LLM call still costing us 200–300ms on every request.
"A sword looks impressive," he said. "A knife is the better tool for the job. That's not a job for a full model. That's a job for a classifier."
A small classifier isn't an LLM. It doesn't generate anything. Its entire job is to read a query and tell you which bucket it belongs to. Think of it like a receptionist — it doesn't solve your problem, it just points you to the right room.
User asks: "What is today's FD rate?"
Classifier returns:
{ "intent": "FD_RATE", "confidence": 0.99 }
We swapped our LLM-based intent step for a small transformer model trained on our own categories. Same accuracy, for what we actually needed. Latency dropped from ~300ms to 10–15ms. On the Node.js side, barely anything changed — my layer just calls a different, much smaller service now:
const intent = await intentService.predict(query);
if (intent === "FD_RATE") {
return getFDRate();
}
if (intent === "RESET_PIN") {
return resetPin();
}
We later learned bigger, more mature systems layer this even further — cheap rules first, small classifier second, and a full LLM only for the genuinely ambiguous cases that fall through both:
User Query
│
▼
Rule Engine
│
▼
Small Classifier
│
▼
Low confidence?
│
Yes ──────────► LLM Router (rare, expensive, but sometimes needed)
│
No
▼
Route Request
Fast and cheap for the 95% of requests that are obvious. The expensive model reserved for the 5% that genuinely need it.
Chapter 8 — The Stuff Nobody Puts in the Diagram
None of the above happened in isolation. While we were fixing intent and caching, a quieter list of problems was piling up in the background — the unglamorous kind that never make it into an architecture slide but absolutely decide whether a system survives production.
- WebSocket reconnects. Mobile networks drop. Users switch from Wi-Fi to data mid-conversation. Every reconnect had to resume the same session without the user noticing or repeating themselves.
- Redis going down. Once, briefly, in staging. Long enough to teach us that a semantic cache with no fallback isn't a performance layer anymore — it's a single point of failure.
- Duplicate messages. A flaky connection plus a retry on the client meant the same query occasionally hit our backend twice, milliseconds apart, and briefly, our bot answered its own question a second time.
- A queue we hadn't planned for. Once traffic wasn't perfectly smooth, some way to absorb bursts without dropping requests stopped being optional.
- Streaming bugs. Tokens arriving out of order under load, or a stream quietly dying mid-response while the UI kept showing a blinking cursor forever.
- Race conditions between the cache write and the cache read, when two users asked something close enough to collide on the same key at nearly the same instant.
- Logs and monitoring that told us something was slow, long before they told us what. Grafana became less of a dashboard and more of a diary of everything we'd gotten wrong that week.
None of these made it into the pitch deck. All of them made it into production.
What Nobody Planned
I looked at the architecture diagram one evening, much later into the project, and actually counted the boxes.
WebSocket. Authentication. Small Classifier. Semantic Cache. Retriever. RAG. LLM. Streaming. Queue. Monitoring. Analytics. Rate Limiter.
User
│
▼
Small Classifier
│
┌──────────────┼──────────────┐
│ │ │
▼ ▼ ▼
Semantic Cache Banking Out of Domain
│ │
Cache Hit? ▼
│ RAG Search
▼ │
Return Answer ▼
LLM Generation
And that's not even the whole diagram anymore — it's just the part that fits on a slide.
Nobody in that first kickoff meeting planned this. Nobody sat down on day one and designed twelve services. Every single box exists because one day, something broke, or someone laughed at a bad question, or a dashboard humbled us on a Monday morning, or a senior engineer compared our design to a man swinging a sword at a cucumber.
That's the part I didn't understand when this started. I thought I was going to build a chatbot. What I actually did, one bad assumption and one 2 a.m. Redis-docs binge at a time, was watch a system design itself in front of me — the way real systems always do, which is nothing like the clean diagram you draw on day one.
This was never really an AI story. AI was just the excuse. It's an engineering story that happened to have an LLM standing in the middle of it.
Next up: the part where I learned that parsing documents for RAG is a harder problem than RAG itself.
Top comments (0)