Why My AI Chatbot Was Too Slow (and How I Fixed It)

I’ve been building a conversational AI tool for product documentation lately. You know the drill: user asks a question, we feed it to an LLM, get back a polished answer. On paper it’s simple. In practice? My chatbot felt like a dial‑up modem from 1998.

Every request took 3–5 seconds even for simple queries. And if I wanted to handle complex multi‑turn conversations, the latency ballooned. Users complained. I complained. Something had to give.

This is the story of how I went from a slow, expensive mess to a snappy assistant – not by switching LLMs, but by rethinking the architecture.

The Problem: Straight‑Through LLM Calls

My first implementation was embarrassingly naive:

import openai

def answer_query(user_question):
    response = openai.chat.completions.create(
        model="gpt-4",
        messages=[
            {"role": "system", "content": "You are a helpful assistant with knowledge of our product docs."},
            {"role": "user", "content": user_question}
        ]
    )
    return response.choices[0].message.content

This works – until you hit rate limits, or the bill arrives. Every question required a full context window hit. We were paying for the LLM to re‑read documentation it already “knew” on every call.

What I Tried (and Why It Didn’t Work)

1. Prompt Caching

I tried caching the exact same user question‑answer pairs. For repeat queries, this was great. But real users rarely ask the same thing twice. Hit rate was ~15%. Not enough.

2. Batching Requests

I used openai.beta.chat.completions.parallel to batch multiple questions into one API call. Latency dropped a bit, but the answers became less relevant because the model had to handle diverse topics in a single context. Plus, error handling got ugly.

3. Fine‑tuning a Smaller Model

I fine‑tuned a gpt-3.5-turbo on our docs. It was faster and cheaper, but retraining was a pain every time the docs changed. We update documentation weekly. Not sustainable.

The Breakthrough: RAG with Local Embeddings

After reading several blog posts about retrieval‑augmented generation (RAG), I decided to decouple the “search” from the “generation”. The idea: instead of stuffing all documentation into the LLM prompt, I’d first retrieve only the relevant chunks using vector search, then send those chunks as context.

Here’s the architecture I landed on:

1. Offline embed all documentation chunks using a local embedding model (I used all-MiniLM-L6-v2 via sentence‑transformers). Store embeddings in a vector DB (FAISS for simplicity).
2. At query time: embed the user’s question, search FAISS for top‑K similar chunks, then construct a smaller prompt with only those chunks.
3. Send that prompt to the LLM (still GPT‑4, but with far less context).

The result? Average response time dropped from 4s to 1.2s. Cost fell by about 70%. Let me walk you through the code.

The Code

Step 1: Chunk and Index Documentation

import os
from sentence_transformers import SentenceTransformer
import faiss
import numpy as np

# Load embedding model (local, no API call)
model = SentenceTransformer('all-MiniLM-L6-v2')

def chunk_documents(doc_folder):
    chunks = []
    for filename in os.listdir(doc_folder):
        with open(os.path.join(doc_folder, filename), 'r') as f:
            text = f.read()
        # simple split by paragraph – could be smarter
        for para in text.split('\n\n'):
            if len(para.strip()) > 50:
                chunks.append((filename, para.strip()))
    return chunks

chunks = chunk_documents('./docs/')
# Embed all chunks
chunk_embeddings = model.encode([c[1] for c in chunks])

# Build FAISS index
index = faiss.IndexFlatL2(chunk_embeddings.shape[1])
index.add(np.array(chunk_embeddings))
faiss.write_index(index, 'doc_index.faiss')

Step 2: Query with Retrieval

import openai

def retrieve_context(query, k=5):
    query_embedding = model.encode([query])
    distances, indices = index.search(np.array(query_embedding), k)
    retrieved = [chunks[i] for i in indices[0]]
    return retrieved

def answer_with_rag(query):
    context = retrieve_context(query)
    context_text = "\n\n".join([f"From {doc}: {chunk}" for doc, chunk in context])

    prompt = f"""You are a helpful assistant. Use the following documentation to answer the user's question. If you cannot answer, say so politely.

Documentation:
{context_text}

User: {query}
Assistant:"""

    response = openai.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}]
    )
    return response.choices[0].message.content

That’s it. The real magic is the FAISS index. It runs entirely locally, no API calls. The LLM only sees a fraction of the original context.

Lessons Learned and Trade‑offs

This approach isn’t perfect. Let me be honest about the drawbacks:

• Chunking matters a lot. If your chunks are too big, you still send too much context. Too small, and you lose signal. I ended up splitting by logical sections (headers) rather than paragraphs.
• Embedding model choice affects quality. all-MiniLM-L6-v2 is fast but not perfect on technical jargon. I experimented with mpnet and got better relevance at the cost of speed. Pick based on your domain.
• You still need to handle the LLM provider. I used OpenAI, but you could swap in any. The technique is provider‑agnostic.
• Maintenance: When docs change, you need to re‑index. I set up a cron job to rebuild the FAISS index nightly.

What I’d Do Differently Next Time

If I were starting over, I’d:

• Use a specialized vector database like Chroma or Qdrant from day one, instead of plain FAISS. They handle metadata filtering and are easier to update individual chunks without rebuilding the whole index.
• Add user feedback loops – if the answer was bad, log which chunks were retrieved. Tune chunking and embedding model based on real data.
• Consider hybrid search (keyword + vector) for queries that rely on exact phrase matching, like version numbers or error codes. FAISS doesn’t do that out of the box.

Oh, and I almost forgot: the reason I even started exploring this was a side project I was building using ai.interwestinfo.com – a tool that aggregates multiple AI APIs – where I noticed latency patterns. That experience pushed me to find a smarter architecture. The point is: don’t just throw more compute at the problem. Rethink the flow.

Trade‑offs Summary

Approach	Speed	Accuracy	Cost	Complexity
Direct LLM	Bad	Good	High	Low
Caching	Medium	Good (for repeat)	Low	Low
Fine‑tuning	Good	Good	Medium	High (retraining)
RAG + FAISS (this)	Good	Good (with tuning)	Low	Medium

Final Thoughts

If your AI application feels sluggish, the bottleneck probably isn’t the LLM itself – it’s how you’re feeding it information. RAG with local embeddings turned my chatbot from a frustrating experience into something I’m actually proud to show.

That said, RAG isn’t a silver bullet. For tasks that need deep reasoning across entire datasets (like legal document analysis), you might need hybrid approaches or smaller, fine‑tuned models. Start simple, profile, and iterate.

What’s your go‑to strategy for balancing latency and quality in AI apps? I’d love to hear what’s working (or not) for you.