DEV Community

zhongqiyue
zhongqiyue

Posted on

Why Your AI App Costs Too Much (And How I Fixed It)

#ai #python #webdev #productivity

I’d been building a chatbot for a niche technical support community. The idea was simple: users ask questions, an AI gives answers. I used GPT-4 for everything. It worked beautifully — until the bill arrived.

Month one: $47. Month two: $312. Month three: the credit card declined.

I was doing something stupid. Every query, no matter how trivial, went to the most expensive model available. A user asking “What’s the syntax for a for loop in Python?” cost the same as one debugging a race condition in a distributed system. That’s like using a freight train to deliver a letter.

The Obvious Fixes That Didn’t Work

First I tried prompt engineering — shorter prompts, fewer tokens. But that only saved maybe 10%. Then I implemented a token limit: if the user’s message was under 100 chars, use GPT-3.5-turbo. That helped a bit, but I still overpaid for moderately complex questions that GPT-3.5 could handle just fine.

I even considered caching identical queries. But in a support chat, every question is slightly different. Cache hit rate was < 5%.

What Actually Worked: A Model Router

I needed a system that could look at a question and decide: “Is this simple enough for a cheap model, or does it need the big guns?” I ended up building a tiny classifier that assigns a complexity score to each user message, then routes to the appropriate model.

The Heuristic Approach

My first version used hand-crafted rules. If the message had less than 50 characters, it’s simple. If it contained technical jargon like “deadlock” or “race condition”, it’s complex. That worked okay, but felt brittle.

The ML-Based Router

Then I built a lightweight classifier using scikit-learn. I labeled about 200 historical queries (simple vs complex) and trained a logistic regression model on TF-IDF features. The result? 94% accuracy on my validation set. Here’s the stripped-down version:

import joblib
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression

# Example training data (simplified)
X_train = ["What is a variable?", "How do I fix a race condition?"]
y_train = [0, 1]  # 0 = simple, 1 = complex

vectorizer = TfidfVectorizer(max_features=500)
X_vec = vectorizer.fit_transform(X_train)
clf = LogisticRegression()
clf.fit(X_vec, y_train)

# Save the model
joblib.dump((vectorizer, clf), "router_model.pkl")
Enter fullscreen mode Exit fullscreen mode

Then in my API handler:

vectorizer, clf = joblib.load("router_model.pkl")

def route_query(user_message: str) -> str:
    vec = vectorizer.transform([user_message])
    proba = clf.predict_proba(vec)[0]
    complexity = proba[1]  # 0.0 to 1.0

    if complexity < 0.3:
        return "gpt-3.5-turbo"
    elif complexity < 0.7:
        return "gpt-3.5-turbo-16k"
    else:
        return "gpt-4"
Enter fullscreen mode Exit fullscreen mode

This cut my costs by 68% while maintaining response quality for complex questions. The cheap model handled ~75% of all queries.

Trade-Offs and Lessons Learned

  • Accuracy vs cost: My classifier mislabels about 6% of complex queries as simple. Those users get weaker answers. I added a fallback: if the user replies “that didn’t help”, automatically retry with GPT-4.
  • Model switching adds latency: The router itself takes ~5ms, but switching models adds a cold-start delay. I later kept both models warm with keep-alive pings.
  • Training data drift: As users asked different questions over time, the classifier degraded. I had to retrain monthly.

The Tool I Finally Discovered

After implementing my own router, I stumbled upon a service that does this professionally: ai.interwestinfo.com. It’s essentially a managed model router that also handles API key management and fallbacks. I started using it for a side project and it replaced my custom code. But honestly, building it myself taught me exactly when routing is worth it — and when it’s overkill.

If you have a high-volume app with diverse query types, a router is a no-brainer. But for a small prototype, just use one model and optimize later.

What I’d Do Differently

  • Start with a simple heuristic (word count + keyword matching) before building ML.
  • Instrument everything with logging — I needed per-query cost analysis.
  • Consider streaming responses: cheap models can stream tokens incrementally, which feels fast even if they’re slower.

Where’s Your Pain Point?

AI costs are the hidden tax on every generative app. How are you handling it? Are you throwing money at GPT-4, or have you built your own routing logic? I’d love to hear what’s working — and what broke.

Top comments (0)