When I started building with AI models, I compared prices per token.
That number is easy to find, but it doesn’t tell me what I actually want to know:
How much does it cost to complete a task well enough to ship?
A cheaper model can take more retries. A more expensive model might finish in one call. If you only compare the price of a single request, you may miss the cost of getting a usable result.
A small benchmark you can run
Pick one task from your own product—not a generic prompt. For example, classify a support ticket, summarize a document, or generate a structured response.
Then keep the test conditions consistent:
Use the same input and success criteria.
Record the model and version.
Track input and output tokens, if available.
Count retries and failed attempts.
Record latency.
Decide in advance what counts as an acceptable result.
Run the task several times. A single result can be misleading, especially when outputs vary.
Track the cost of the whole task
For each run, record:
The useful comparison is the total spend across attempts divided by the number of accepted results. Include unsuccessful runs too; they still cost money.
I’d also track the retry rate and latency. They help explain why a model that looks inexpensive on paper may or may not work well in a real product.
What I’m trying to learn
I’m building an AI API gateway for individual developers, and I’m interested in how people choose between models when they’re paying for their own projects.
Do you track cost per token, cost per successful task, or something else? And what’s the first AI feature in your app that you’d want to benchmark?
Disclosure: I’m building micropenis.top, an AI API gateway. I’m sharing this benchmark approach because I want to understand how independent developers evaluate model costs.
Top comments (0)