Cost per completed task is the only AI API number worth comparing

Every pricing page quotes a rate. Almost none quote the number you actually need: what one finished task costs. Those are different numbers, and for agent workloads they diverge by more than an order of magnitude.

Here is how to measure it, and why per-token comparisons between providers mislead even when the rates are correct.

Why the quoted rate is not the cost

A token rate tells you the price of a unit of input and output. To get from there to the cost of a task you need four more facts:

• How many steps the task takes. An agent loop re-sends its accumulated context on every step. A twenty-step task does not consume twenty requests' worth of tokens; it consumes the sum of a growing context, which is quadratic in the number of steps.
• How much of the context is cacheable. If the provider bills cached input at a discount, the same transcript can cost several times less. If it does not, you pay full rate for the same text on every step.
• How many attempts fail. A task that takes three tries costs three times as much, and the retries are the longest, most context-heavy calls.
• What a retry re-sends. Some frameworks restart with a fresh context on retry; others carry the failure forward. The second is cheaper per attempt and more expensive per task, because it burns steps re-reading a transcript that already failed.

None of that appears on a pricing page.

The measurement that actually works

You do not need a benchmark harness. You need to log four fields per task and do arithmetic.

import time, json

def measure(task_name, fn):
    start = time.time()
    tokens_in = tokens_out = 0
    try:
        result = fn()
        ok = True
    except Exception as e:
        result, ok = repr(e), False
    return {
        "task": task_name,
        "ok": ok,
        "seconds": round(time.time() - start, 1),
        # fill these from your client's usage callback
        "tokens_in": tokens_in,
        "tokens_out": tokens_out,
    }

Run it over twenty or thirty real tasks from your actual workload, not synthetic prompts. Then:

cost per completed task = total spend / number of tasks that finished correctly

The denominator is the part people skip. Tasks that failed still cost money, so excluding them flatters whichever provider fails more often. If provider A charges half as much per token but needs 1.4 attempts per task, A is not cheaper.

A worked example

Take a refactor task that takes about eighteen agent steps, with roughly 9,000 tokens of context per step that grows to 14,000 by the end. Total input lands near 200,000 tokens per attempt. Add output.

Under a metered rate of $3 per million input and $15 per million output, that attempt costs roughly $0.66 in input plus a few cents of output. Call it $0.70. At 1.3 attempts per task, the real cost is about $0.91 per completed task.

Under a flat allowance where you get 240,000 requests a month for $129, the same task consumes 18 requests (or 23 including retries). That is 0.0096% of the monthly allowance, or roughly $0.012 per completed task — but only if you actually consume the allowance. If you use 10% of it, the effective cost is ten times higher.

That last clause is the whole game. Flat pricing is cheap per task when utilisation is high and expensive per task when it is low. Metered pricing is the reverse. Neither is universally better, which is why the comparison has to be per-task, at your utilisation.

What to check before switching

• Does the provider publish request limits per plan, or only token limits? A plan with no published request cap has no upper bound you can plan against.
• Is cached input billed at a discount, and does the discount survive long transcripts? Cache hit rates collapse when context keeps changing, which is the normal case for agents.
• What happens at the ceiling? A clean 429 with Retry-After lets your client back off. A dropped stream mid-task wastes the whole attempt.
• Can you see spend per key? If several clients share one key you cannot attribute cost, so you cannot compute per-task cost for any of them.

Why I run this on a flat allowance

For my own workload the deciding factor was not the rate, it was the variance. A metered bill for agent work is knowable only after the month ends, which makes it hard to price work for a client. A request allowance converts the variable into a count I choose before starting.

The gateway I use is APIClaw — [OI]-compatible, priced per request rather than per token, with per-client keys so attribution is not a spreadsheet exercise. The per-tool setup notes for Cursor, [CC], Codex and about twenty other clients are in the docs.

Whatever you pick, measure cost per completed task on your own workload before you commit. The rate on the pricing page is the least informative number on it.