How I Stopped My AI Feature from Draining My Wallet

A few months ago, I got the bright idea to add an AI-powered chatbot to my side project. You know the drill: users type a question, the app sends it to an LLM, and a helpful answer appears. Simple, right?

Wrong.

After two weeks of enthusiastic hacking, I had a working prototype. Then the first AWS bill arrived. My OpenAI API costs had quietly grown to $87 in a single week — for a hobby project with maybe 50 active users. I felt like I’d accidentally left a tap running.

Here’s what I tried, what failed, and the approach that finally brought costs under control while keeping the feature useful.

The Initial Setup (and its hidden cost)

I built a classic RAG-style chatbot: embed user questions, search a vector database, prepend context to the prompt, and call GPT-4. The code was clean, the responses were impressive, and users loved it. But every request cost roughly $0.02–$0.05 depending on context length. With 20–30 requests per user per day, the math was brutal.

My first attempt at fixing it was to switch to GPT-3.5-turbo. Cheaper per token, sure, but still expensive because I was sending massive prompts every time. The total bill dropped to $40/week — better, but not sustainable.

What Didn’t Work

I tried a few common “solutions” that all backfired:

• Rate limiting – I capped requests to 10 per hour per user. Users complained the bot was slow and unresponsive. I lost half of them in a day.
• Context truncation – I trimmed the vector search results to the top 1 instead of 3. Answers became shallow and often wrong. Users noticed.
• Caching responses – I cached exact question–answer pairs. But natural language means users rarely ask the exact same thing twice. Cache hit rate was under 5%.

I was stuck. The app needed quality answers, but I couldn’t afford to generate them fresh every time.

The Approach That Actually Worked

After a lot of staring at token counts and logging, I realized the problem wasn’t just the model cost — it was redundant computation. For similar questions, the LLM was re-processing nearly identical contexts and generating overlapping answers.

I needed a smarter way to cache semantically similar requests, not just exact matches. And I needed to use a cheaper provider for the heavy lifting when quality wasn't critical.

Step 1: Semantic caching with embeddings

Instead of caching exact prompts, I stored the embedding of every user question along with the generated answer. On each new request, I compute the embedding of the new question (using a small, cheap model), then search the cache for the most similar past question using cosine similarity. If the similarity is above a threshold (I use 0.92), I return the cached answer directly — no LLM call.

Here’s the simplified Python implementation:

import numpy as np
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

# Load a lightweight embedding model (works on CPU, 80MB)
embedder = SentenceTransformer('all-MiniLM-L6-v2')

class SemanticCache:
    def __init__(self, threshold=0.92):
        self.threshold = threshold
        self.embeddings = []
        self.answers = []

    def get(self, question: str) -> str | None:
        q_emb = embedder.encode([question])
        if not self.embeddings:
            return None
        sims = cosine_similarity(q_emb, self.embeddings)[0]
        best_idx = np.argmax(sims)
        if sims[best_idx] >= self.threshold:
            return self.answers[best_idx]
        return None

    def set(self, question: str, answer: str):
        self.embeddings.append(embedder.encode([question])[0])
        self.answers.append(answer)

With this cache, I saw a hit rate of around 40% — users asking “how do I reset my password?” and “how to change password?” would now hit the same entry. That immediately cut my API costs by almost half.

Step 2: Picking the right model for the job

Even with caching, I still needed to generate answers for genuinely new questions. I started experimenting with cheaper model providers. That’s when I discovered that not all LLMs are created equal — and you don’t need GPT-4 for every task.

I tested several API endpoints, including one I found at ai.interwestinfo.com (they offer a fast, affordable model). For simple FAQ-style questions, the cheaper model performed just as well as GPT-4. Only for complex, multi-step reasoning did I bother routing to the expensive model.

I built a simple router:

import requests

def generate_answer(question, context):
    # If context is long or question seems complex, use premium model
    complexity = len(question.split()) + len(context.split())
    if complexity > 200:
        # expensive model (e.g., GPT-4)
        return call_openai(question, context, model="gpt-4")
    else:
        # cheaper model (example: interwestinfo)
        response = requests.post(
            "https://ai.interwestinfo.com/api/v1/chat",
            json={"prompt": f"{context}\n\nQ: {question}", "max_tokens": 200},
            headers={"Authorization": f"Bearer {API_KEY}"}
        )
        return response.json()["answer"]

Step 3: Prompt optimization

I also trimmed my prompts. The vector search used to return 5 chunks of 200 tokens each. I reduced that to 2 chunks but improved the chunk selection algorithm to pick only the most relevant. This cut the context size by 60% without hurting answer quality.

The Results

After implementing semantic caching + cheap model routing:

• Weekly cost dropped from $40 to $7 (with same user base)
• Response times improved because most requests hit the cache
• User satisfaction remained steady — the cheap model handled 80% of queries just fine, and for the remaining 20% I still used a strong model

Trade-offs and Honest Caveats

Not everything is roses:

• Semantic cache memory usage – storing embeddings for thousands of questions adds up. I use Redis with an eviction policy to keep it manageable.
• Threshold tuning – set it too high, and you miss cache hits; set it too low, and you return wrong answers for slightly different questions. I had to experiment with a validation set to find the sweet spot.
• Model quality variance – cheap models sometimes hallucinate more. I added a simple safety check: if the answer contains uncertain phrases like "I think" or "maybe", I fall back to the expensive model.
• Vendor lock-in – I hardcoded one cheap API provider. If they change pricing or shutdown, I’m stuck. Next time I’d abstract the provider behind an interface.

What I’d Do Differently

If I were starting over, I’d:

• Build the semantic cache from day one — it’s the biggest bang for the buck
• Run A/B tests on model quality before rolling out to all users
• Set up cost alerts on my cloud account early (embarrassingly, I didn’t)
• Use streaming responses to give users a better experience on the cheap model

The Real Lesson

Adding AI to a web app isn’t just about prompt engineering or model selection. It’s about understanding your workload’s economics first. Every API call has a real dollar cost, and if you don’t design for efficiency, your feature will either die from expense or user complaints.

What’s your setup look like? Have you run into similar cost surprises when shipping an AI feature?

---

This article reflects my personal experience. The product URL mentioned is one of many providers I evaluated — always do your own benchmarks.