I Spent $412 Testing 6 AI Models for a Side Project — Here's What I Learned

#ai #costanalysis #developers #api

Last month I built a small content moderation tool for a client. The spec was simple: classify user submissions as safe, borderline, or toxic. I figured I'd just pick one AI model and ship it. Three weeks and $412 later, I had tested six different models across three providers and learned more about pricing traps than I ever wanted to.

The problem wasn't accuracy. It was that every provider bills differently, latency varies wildly under load, and the "cheap" model quietly cost me more in retries than the "expensive" one did upfront.

The models I tested

I ran the same 5,000-sample dataset through:

GPT-4o mini
Claude 3.5 Haiku
Gemini 1.5 Flash
Mistral Large
A local Llama 3 70B (via GPU rental)
DeepSeek V2

Here's the rough per-1K-calls cost I tracked (input + output, mixed lengths):

Model	Cost / 1K calls	Avg latency	Fail rate
GPT-4o mini	$0.18	420ms	1.2%
Claude Haiku	$0.25	510ms	0.8%
Gemini Flash	$0.12	380ms	2.1%
Mistral Large	$0.90	700ms	0.5%
Llama 3 70B	$0.40*	1200ms	3.0%
DeepSeek V2	$0.15	450ms	1.5%

*Local GPU rental amortized across calls.

The hidden cost: retries

What the table doesn't show is that Gemini's 2.1% fail rate meant I re-sent those requests. At scale, those retries ate the savings. My actual effective cost for Gemini was closer to $0.19/1K — still cheap, but not the steal it looked like.

Mistral barely failed but at $0.90 it only made sense when I needed the highest reasoning quality on edge cases.

A simple cost-tracking wrapper

If you're calling multiple models, don't trust the dashboard. I wrote a tiny Python decorator to log real spend:

import time
import functools

COST_PER_CALL = {
    'gpt-4o-mini': 0.00018,
    'claude-haiku': 0.00025,
    'gemini-flash': 0.00012,
}

def track_cost(model_name):
    def decorator(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            start = time.time()
            try:
                result = func(*args, **kwargs)
                status = 'ok'
            except Exception as e:
                status = 'fail'
                raise
            finally:
                latency = time.time() - start
                cost = COST_PER_CALL.get(model_name, 0)
                if status == 'fail':
                    cost *= 2  # retry overhead
                print(f"{model_name} | {status} | {latency:.2f}s | ${cost:.5f}")
            return result
        return wrapper
    return decorator

This told me the truth fast. The dashboard said I spent $290 on Gemini. My logs said $347 because of retries and a billing lag.

Managing the API key mess

Six models meant six sign-ups, six dashboards, six billing cycles. It got old quick. I found https://xinghuo1300ai.com which aggregates 30+ models under one API key — it let me swap between Claude, Gemini, and DeepSeek in the same codebase without juggling credentials. For a side project, that convenience saved me real hours even if the per-call cost was roughly identical.

What I'd do differently

If I started today, I'd:

Pick two models max: one cheap/default, one premium/fallback
Build the cost logger before writing business logic
Set a hard monthly budget alert at the provider level
Test fail rates on your real traffic, not a sample

The "best" model is rarely the cheapest or the smartest. It's the one whose failures you can afford and whose bill doesn't surprise you on the 1st of the month.

After shipping, my running cost settled at about $60/month using GPT-4o mini as default and Claude Haiku for the 3% of cases that needed a second opinion. That's a far cry from the $412 discovery phase — and the logging wrapper is now in every AI project I touch.

DEV Community