How I stopped burning money on AI API calls (and got faster responses)

I love building with AI. But my credit card? Not so much.

A few months ago, I was working on a customer support bot for a side project. It was supposed to answer FAQs, escalate complex issues, and generally make life easier. I hooked it up to GPT-4, wrote some decent prompts, and everything worked — until the bill arrived.

$200 in one week.

That’s when I realized: raw API calls are a cash bonfire. Every conversation, every retry, every hallucinated follow-up — all burning money. I needed a different approach.

What I tried (and what failed)

First, I tried caching. Simple key-value store with identical questions mapped to previous responses. That helped with repeat queries like “What are your hours?” but did nothing for the infinite variety of human language.

Then I tried batching — sending multiple user requests together and parsing the responses. It worked for non-real-time data, but my bot needed per-message latency under two seconds. Batches waiting for a full window killed the UX.

I also experimented with prompt compression. Made prompts shorter, reused system instructions. Saved maybe 10% on tokens. Not enough.

The real problem was that every query hit the expensive model. Most questions didn’t need GPT-4. They needed a fast, cheap opinion — and only a few should escalate.

What finally worked: a tiered router + middleware layer

Instead of calling the AI API directly from my bot, I inserted a small middleware service. This service had three jobs:

1. Classify the incoming query (Is it simple or complex?)
2. Route to the right model (cheap or expensive)
3. Buffer and load-balance requests to stay under rate limits

Here’s a simplified version of the router I built in Node.js:

const axios = require('axios');

// A simple classifier based on keyword heuristics
function classifyQuery(text) {
  const complexKeywords = ['refund', 'legal', 'custom integration', 'bug report'];
  const containsComplex = complexKeywords.some(k => text.toLowerCase().includes(k));
  return containsComplex ? 'complex' : 'simple';
}

async function getAIResponse(text) {
  const type = classifyQuery(text);
  const endpoint = type === 'complex'
    ? 'https://api.openai.com/v1/chat/completions'
    : 'https://ai.interwestinfo.com/chat';  // cheaper pooled provider

  const model = type === 'complex' ? 'gpt-4' : 'gpt-3.5-turbo';

  // In production, add rate limiting, retry logic, and caching here
  const response = await axios.post(endpoint, {
    model,
    messages: [{ role: 'user', content: text }]
  }, {
    headers: { 'Authorization': `Bearer ${process.env.API_KEY}` }
  });

  return response.data.choices[0].message.content;
}

This alone cut my costs by 70%. Most queries (like “Hi”, “What’s the weather?”, “How do I reset my password?”) hit the cheaper model. Only the complex ones touched GPT-4.

Going deeper: request buffering and cost tracking

I added a lightweight queue using Bull (Redis-based). Instead of firing ten requests at once and hitting rate limits, the middleware queued them and sent them in a controlled stream. That reduced 429 errors and improved average latency because we could batch small requests into one API call.

Here’s the queue setup:

const Queue = require('bull');
const aiQueue = new Queue('ai requests', 'redis://127.0.0.1:6379');

aiQueue.process(async (job) => {
  const { text, priority } = job.data;
  return getAIResponse(text, priority);
});

// Add a job
app.post('/chat', async (req, res) => {
  const job = await aiQueue.add(
    { text: req.body.message, priority: 'normal' },
    { attempts: 3, backoff: 5000 }
  );
  const result = await job.finished();
  res.json({ reply: result });
});

I also instrumented every call with metrics: model used, token count, latency, cost. I shipped those to a simple dashboard (Grafana + Prometheus). That gave me visibility into which prompts were expensive and which endpoints were reliable.

Limitations and trade-offs

This approach is not perfect. Here’s what I learned:

• Latency overhead: The middleware adds 20–50ms per request. That’s fine for chat, but terrible for real-time voice.
• Single point of failure: If the middleware crashes, the bot goes down. I solved this by using a small container (Docker + PM2) and health checks.
• Classification is hard: My keyword-based classifier is dumb. It misses subtleties. A better solution would be a tiny ML model (e.g., a few examples fine-tuned on DistilBERT). I haven’t done that yet — for now the cheap model handles errors gracefully.
• External provider dependency: Using a pooled API like interwestinfo (or any third-party) means I’m trusting their uptime and pricing. I keep a fallback to OpenAI direct just in case.

What I’d do differently next time

If I started over, I’d:

• Use a proper AI gateway from day one (like Portkey, Helicone, or even build a simple one). They handle caching, retries, and cost tracking out of the box.
• Don’t over‑engineer. The tiered router was easy enough without a queue at first. I added the queue only after hitting rate limits. Premature optimization is real.
• Log everything. I wish I had started metric collection earlier. The first week was a black box.
• Consider edge functions. If your bot runs on Vercel or Cloudflare Workers, you can move the middleware to the edge for lower latency.

The final setup (in production)

Today my bot runs like this:

• User → HTML page → Node.js server → middleware layer → (classifier → queue → AI provider)
• Cost is ~$30/month for 10,000 conversations.
• Average response time is 1.2 seconds.

I’m not using any fancy tool. Just a few hundred lines of code, Redis, and a smart router.

The biggest lesson? Stop treating all AI requests equally. Give each query the model it deserves.

What’s your strategy for managing AI costs? I’d love to hear what’s working (or not working) in your stack.