Count accepted jobs, not successful API calls, when comparing AI workflows

An HTTP 200, valid JSON, and a useful application result are three different events. If a workflow retries twice and then produces one acceptable answer, it has one accepted job, not three successes.

Disclosure: I work on Tokatlas, a multi-model API platform. This post was generated by an AI agent at my direction. The example is synthetic and makes no claim about model quality, price or production savings.

Keep one row per attempt

Use a stable job_id across retries and a unique attempt_id for each call. Record acceptance using a rule fixed before comparing providers: for example, all required invoice fields must match the labeled source. Keep HTTP status, parse status and acceptance separate.

Do not infer a zero charge from a timeout, error response, missing usage field, or canceled client connection. Cost should come from a source whose meaning you understand, such as the provider’s billing record. If it is not available, keep it unknown.

This Python example summarizes an attempt log without hiding unknown costs:

from decimal import Decimal

def summarize(attempts):
    seen = set()
    jobs, accepted = set(), set()
    known_cost = Decimal("0")
    unknown_costs = 0
    for a in attempts:
        key = a["attempt_id"]
        if key in seen:
            raise ValueError("duplicate attempt_id")
        seen.add(key)
        jobs.add(a["job_id"])
        if a["accepted"] is True:
            accepted.add(a["job_id"])
        raw = a.get("billed_cost")
        if raw is None:
            unknown_costs += 1
        else:
            cost = Decimal(str(raw))
            if not cost.is_finite() or cost < 0:
                raise ValueError("invalid billed cost")
            known_cost += cost
    complete = unknown_costs == 0
    return {
        "jobs": len(jobs),
        "attempts": len(attempts),
        "accepted_jobs": len(accepted),
        "acceptance_rate": len(accepted) / len(jobs) if jobs else None,
        "known_cost_subtotal": str(known_cost),
        "attempts_with_unknown_cost": unknown_costs,
        "cost_per_accepted_job":
            str(known_cost / len(accepted)) if complete and accepted else None,
    }

rows = [
    {"job_id":"A", "attempt_id":"A-1", "accepted":False, "billed_cost":"0.01"},
    {"job_id":"A", "attempt_id":"A-2", "accepted":True,  "billed_cost":"0.02"},
    {"job_id":"B", "attempt_id":"B-1", "accepted":False, "billed_cost":None},
]
print(summarize(rows))

The numbers above are invented fixture values in a single assumed currency, not Tokatlas pricing. Their purpose is to show the accounting boundary: two jobs, three attempts, one accepted job, and one unknown cost. Cost per accepted job remains null because the total is incomplete.

Three useful checks before using the metric

1. One cohort: Compare the same input cases, acceptance rule and retry limit. Include failed attempts and jobs that never produced an accepted result.
2. One currency and scope: Do not mix currencies or per-call charges with a monthly invoice total. This snippet assumes the input log already follows that rule.
3. One completion boundary: Exclude still-running cohorts, or mark them explicitly as incomplete. A timeout may later acquire a billing record; reconcile it against the same attempt rather than adding a second attempt.

This is an API-cost metric, not full operational cost. Human review time, queueing, hosting and task latency need separate fields. A model that produces cheaper calls can still require more review; a more expensive call can still fail your acceptance rule.

If you are choosing an API for a concrete workflow, start with a small redacted or synthetic case set and a written acceptance rule. The question to bring to an API provider is not only “what is the token price?” but “what response and usage information will I have to reconcile each attempt?”

Which failure do you currently count incorrectly: a timeout, a parseable but wrong answer, or a retry that eventually succeeds?