Cost per successful AI task: a practical way to compare model routes

When teams compare LLM providers or gateway routes, the first spreadsheet is usually token price.

That is useful, but it is rarely enough.

A cheaper request that needs retries, produces unusable output, or breaks a workflow can cost more than a higher-priced request that finishes the task once.

A more practical metric is cost per successful task.

Disclosure: I work with ModelRouter, an independent third-party OpenAI-compatible AI API gateway. It is not an official model-provider service. The checklist below is provider-neutral, and the ModelRouter link at the end is included as one possible place to run this kind of controlled route evaluation.

The metric

Use this formula:

cost per successful task = total spend / number of tasks that meet the acceptance criteria

The important part is the acceptance criteria.

A successful task is not just a 200 response. It should mean the output was usable for the workflow.

Examples:

• a support reply that can be sent with light editing
• a JSON extraction that passes schema validation
• a code suggestion that passes tests
• a summary that includes the required fields
• a workflow step that completes without a manual rerun

Step 1: Pick one real workflow

Do not start with a vague benchmark.

Pick one workflow your team already understands:

• support triage
• lead enrichment
• invoice extraction
• n8n automation step
• Dify agent task
• Open WebUI internal assistant
• code review helper
• sales email classification

The workflow should have enough examples to show failures, but it should be small enough to review manually.

Step 2: Define success before running the test

Write down what counts as success.

For example:

Task: classify inbound support tickets
Success: correct category, priority, and summary in valid JSON
Failure: wrong category, invalid JSON, missing priority, hallucinated customer facts, timeout, or manual rerun required

If the success definition is fuzzy, the result will be fuzzy too.

Step 3: Run a small representative set

For a first pass, 20 to 50 tasks is often enough to avoid obvious mistakes.

For a team or client workflow, 50 to 200 tasks gives a better signal.

Track these fields:

route
model
input type
task id
status code
latency
retry count
raw cost
usable output: yes/no
failure reason
notes

The failure reason is where the learning happens.

Step 4: Include retries and manual fixes

If route A costs less per request but needs more retries, include that.

If route B produces valid JSON more consistently, include that.

If route C is fast but needs manual review every time, include that too.

A simple scoring table can look like this:

route | tasks | successful | spend | retries | cost per successful task
A     | 50    | 41         | $3.20 | 9       | $0.078
B     | 50    | 47         | $4.10 | 2       | $0.087
C     | 50    | 32         | $2.40 | 18      | $0.075

The cheapest per successful task may still not be the best route if latency, reliability, or review cost matters. This metric is a starting point, not a final verdict.

Step 5: Keep the first call boring

Before comparing routes, make sure one minimal request works.

A smoke test should answer:

• is the base URL correct?
• is the API key accepted?
• is the model route valid?
• does usage/spend show up where expected?
• can the client library send one chat completion?

Only after that should you evaluate real tasks.

Step 6: Decide stop, continue, or expand

At the end of a small run, choose one of three decisions:

1. stop, because the route does not improve the workflow
2. continue, because the route is promising but needs a larger sample
3. expand, because the route is clearly useful for this workflow

This prevents accidental production migration based on one good demo.

A minimal evaluation checklist

[ ] workflow selected
[ ] success criteria written
[ ] 20-50 representative tasks collected
[ ] first API call verified
[ ] route/model names recorded
[ ] spend tracked
[ ] retries counted
[ ] failures categorized
[ ] cost per successful task calculated
[ ] stop/continue/expand decision written

Optional controlled pilot

If you want to run this through an OpenAI-compatible gateway, the same pattern applies: start with one first call, then one workflow, then representative tasks.

ModelRouter is one independent third-party OpenAI-compatible gateway where this kind of route evaluation can be tested:

https://modelrouter.site/business-pilot/?utm_source=devto&utm_medium=tutorial&utm_campaign=cost_per_successful_task

Pricing and trust notes:

https://modelrouter.site/pricing-and-trust/?utm_source=devto&utm_medium=tutorial&utm_campaign=cost_per_successful_task

The safest evaluation is still small and boring: make the first call work, measure real tasks, and decide based on successful outcomes rather than token price alone.

Comments

[Raju Dandigam]

Cost per successful task is a much better metric than raw token price because it forces route decisions to include retries, unusable outputs, and downstream cleanup work. I’ve seen teams choose the cheapest request path on paper and then quietly pay the difference back in repair prompts and human review.

The useful part of your framing is that it turns model selection into a workflow KPI instead of a provider-pricing spreadsheet. That is usually the point where routing discussions get more honest, because the system has to optimize for completed work, not just cheap calls.

Curious what denominator has been most useful for you in practice: first-pass usable output, no-human-intervention completion, or a narrower task-specific acceptance test.

[Alex Shev]

Cost per successful task is a much better unit than raw token spend. The catch is defining success tightly enough that cheap failure does not look efficient. I would pair it with retry rate, human escalation rate, and the cost of fixing bad outputs after the model route “succeeded.”