Measured 2026-07-31
Cost per suite run is the cost of one run across all five categories, counting scored cases only. Tokens consumed by refusals, provider policy blocks, and no-output cases are not included. Calculated from observed token counts and published list rates for the route TAB uses. It does not include prompt-caching, batch, or negotiated discounts, which vary by provider and by workload. See the methodology note dated July 29, 2026 below for an earlier period when the Agentic Execution category was excluded from this figure.
Methodology note, July 29, 2026. From the admission of the Agentic Execution category through July 29, 2026, that category’s model cost and token usage were not captured. During that period the Cost per suite run shown for every model excludes the Agentic Execution contribution and is understated by that amount. The judge cost was zero throughout and is unaffected, and scores are unaffected: this concerns cost attribution only, not measurement results. The gap is not reconstructable, because per-call token usage for that category was never recorded, so there is no basis to restate the earlier figures without estimating, and TAB does not publish estimates as measurements. All prior rows remain exactly as recorded. From July 29, 2026 forward, Agentic Execution cost is captured on the same basis as the other categories.
The “vs. lowest” column shows each model’s cost per suite run as a multiple of the lowest cost among public models in this cycle. It compares cost within this cycle only, and is a cost comparison, not a value or quality comparison.
Each category is run at least three times per cycle under identical conditions. Consistency reports how many of a model’s five categories produced the same result across all runs this cycle. The scores shown are from the most recent run.
◆ category leader.
Some models decline to engage with certain test cases — most often security cases, where the model’s own safety classifier blocks the prompt before the model answers. That is a refusal, not a wrong answer. TAB records refusals as unscored so they neither inflate nor deflate the score: the count shown for each model (for example, “refused 3/12”) is exactly how many cases the model declined out of how many it was given, and the score reflects only the cases it engaged with.
tabverified.ai
Top comments (0)