I metered Kimi K3 against 4 other models from actual billing, not list prices

Russell Craft — Wed, 29 Jul 2026 13:02:34 +0000

Kimi K3 landed two days ago and the timeline is full of parameter counts. I wanted a different number: what does it actually cost to run a real task on it, compared to the alternatives?

So I ran the same coding task across five models and metered the cost. The result was not what I expected, and getting there surfaced a measurement problem worth writing up on its own.

The measurement problem (this matters more than the benchmark)

My first attempt derived prices from the gateway's pricing endpoint, which exposes a model_ratio and a completion_ratio per model. The usual convention is $/M = model_ratio * 2, so I used that.

Then I sanity-checked it against models whose real prices I already knew:

Model	My formula said	Reality
claude-sonnet-5	$75 / M	~$8 / M
deepseek-v4-flash	$0.44 / M	I had $2 / M on file

Both wrong, in opposite directions. The formula was junk.

So I threw it out and measured from ground truth instead: read the account's cumulative usage, make one call, read it again, take the difference.

usage() {
  curl -s https://<gateway>/dashboard/billing/usage \
    -H "Authorization: Bearer $KEY" | jq -r .total_usage
}

before=$(usage)
curl -s https://<gateway>/v1/chat/completions \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{"model":"Kimi-K3","messages":[...],"max_tokens":800}'
sleep 4
after=$(usage)

With known token counts on both sides you can solve for the real per-million rates. Two independent models resolved to the same coefficient to three significant figures, which is how you know it is not a coincidence:

$/M input  = model_ratio * 1.60
$/M output = model_ratio * completion_ratio * 1.60

Not * 2. And claude-sonnet-5 does not obey the formula at all: its model_ratio entry (37.5) is on some other scale, so it has to be measured directly.

Side effect: I found my own price table had deepseek marked up nearly 3x too high. If you display per-call costs to users, go verify your table against billing deltas. Mine was wrong and I had been showing it for days.

The benchmark

Same prompt to all five, one run each, non-streaming, no max_tokens cap:

Write a JavaScript function that dedupes an array preserving order, plus 3 edge-case assertions. Code only.

Model	Cost	Latency	Output
deepseek-v4-flash	$0.001404	26.5s	620 chars
claude-sonnet-5	$0.002253	9.3s	570 chars
GLM5.2	$0.005437	24.1s	479 chars
Kimi-K3-codex	$0.006650	9.4s	448 chars
Kimi-K3	$0.017102	35.8s	571 chars

Three things I did not expect

1. K3 is the most expensive option here, 12x the cheapest. Measured output rate lands around $12/M. On one call that is noise. On a batch job of a few thousand calls it is the difference between $2 and $25.

2. It is also the slowest. 35.8s versus claude-sonnet-5's 9.3s. K3 is a reasoning model, it spends tokens thinking before answering. The variance is large: a similar prompt came back in 11.9s on another run. The reasoning budget floats.

3. The codex variant is the actual bargain. Same list price as base K3, but on coding tasks it thinks far less: $0.0067 vs $0.0171, 9.4s vs 35.8s. Half the cost, roughly 4x faster. If you are reaching for K3 to write code, reach for Kimi-K3-codex instead.

One trap if you cap max_tokens

While probing I set max_tokens to 200 and got back a completely empty response, and still paid for 200 tokens. All of them went to reasoning_tokens; nothing was left for visible output.

At 600 the reasoning took 57 tokens and the answer came through fine. At 2048 it took 229.

This is not a K3 defect, it is how reasoning models behave when you starve them. But if your app applies a tight max_tokens ceiling across all models (mine did, at 2048), audit it. A user gets a blank reply and you still get billed.

Caveats

Single run per model. This is an order-of-magnitude read, not a statistically significant result. Latency in particular swings a lot between runs. The point is not the exact decimals, it is that the ranking was the opposite of what the launch-day discourse implied, and you only find that out by metering.

Reproduce it

All five models are running at https://ainetcafe.com/?utm_source=devto&utm_medium=article&utm_campaign=kimi-k3 — open-source AI apps hosted ready to use, no install and no API key needed to try. Every call shows its measured cost.

There is also an MCP server if you would rather have an agent run the comparison itself:

claude mcp add --transport http ai-netcafe https://ainetcafe.com/mcp

Then ask it to compare_models on whatever prompt you actually care about. That is the tool I used to produce the table above.

Full data and method: https://ainetcafe.com/lab/kimi-k3?utm_source=devto&utm_medium=article&utm_campaign=kimi-k3