Build a spend variance triage loop for AI API gateways

The first bad AI API invoice is usually not a math problem. It is an evidence problem.

A team expects one number before rollout. The gateway reports another number after traffic runs. The product owner asks which feature moved. Finance asks whether the rate changed. Support asks whether retries, fallback routes, or account groups affected usage. Engineering opens logs and finds a pile of request ids, token totals, and model names that almost answer the question.

Almost is not enough for Tier 1 and Tier 2 buyers. A serious AI gateway should be able to explain spend variance without reading raw prompts, exposing customer data, or guessing which rate card was current at the time.

AIWave is built for teams that want one OpenAI-compatible route to Chinese model families such as DeepSeek, GLM, Kimi, Qwen, ERNIE, MiniMAX, Doubao, StepFun, and MiMo. That single route is useful when teams want stable client code, one USD ledger, and multiple model choices. It also means the gateway has to keep route evidence close to every cost review.

This article gives a practical loop for triaging spend variance after real traffic runs.

Start with a variance record

Do not begin with a dashboard screenshot. Begin with a small variance record that names the expectation, the observation, and the evidence window.

Field	Example	Why it matters
window_start	2026-09-03T00:00:00Z	Defines the query boundary
window_end	2026-09-03T06:00:00Z	Avoids mixing partial periods
expected_usd	42.00	Captures the rollout assumption
observed_usd	57.80	Captures the review target
owner	platform-ai	Assigns a decision owner
pricing_source	https://aiwave.live/api/pricing	Shows where route fields came from
pricing_checked_at	2026-09-03T13:11:00Z	Prevents stale rows from becoming facts
pricing_version	a42d372ccf0b5dd13ecf71203521f9d2	Links the review to a catalog snapshot

That record is not the final answer. It is the case file. Every later query should either explain part of the variance or mark it as unknown.

Check the current pricing source

Before writing this article, I checked AIWave's public pricing endpoint on 2026-09-03 at 13:11 UTC. The endpoint returned success=true, pricing_version=a42d372ccf0b5dd13ecf71203521f9d2, 63 route records, auto_groups=["default"], and group ratios of default=3 and vip=1.

The same live response included these route fields:

Route	model_ratio	completion_ratio	cache_ratio	Enabled groups
deepseek-v4-flash	0.319	3	0.0318	default, vip, svip
deepseek-v4-pro	0.957	3	0.0333	default, vip, svip
glm-5.1	1.05	3.142857	0.32381	default, vip, svip
kimi-k3	2.25	5	0.2	default, vip, svip

Using AIWave's public VIP x1 rate-card conversion, those rows correspond to these per-1M token planning rates for the selected examples:

Route	Input	Cached input	Output
deepseek-v4-flash	$0.638	$0.0203	$1.914
deepseek-v4-pro	$1.914	$0.0637	$5.742
glm-5.1	$2.10	$0.6800	$6.60
kimi-k3	$4.50	$0.90	$22.50

These are dated planning inputs, not a promise that every account, workload, or future request will land on the same bill. Store the checked time, source URL, pricing version, effective group, route id, token buckets, and final usage beside the request.

Split variance into four buckets

Spend variance becomes easier to debug when every difference is assigned to one bucket first.

Bucket	Typical signal	First query
Volume variance	More calls or larger prompts than forecast	Count requests and input tokens by feature
Mix variance	More traffic moved to a different route	Group usage by resolved model
Shape variance	Output, cache, or tool usage changed	Compare input, cached input, and output buckets
Policy variance	Retries, fallbacks, or account groups changed	Group usage by policy version and effective group

Do not argue about the final invoice before this split exists. A model route may be correct while the output cap is too high. A pricing row may be stable while the product shipped a larger prompt. A fallback may be rare in request count but large in token share.

Build the request ledger

A useful ledger does not need raw prompts. It needs enough structured data to explain routing and billing.

create table gateway_request_ledger (
  request_id text primary key,
  account_id_hash text not null,
  feature text not null,
  environment text not null,
  requested_model text not null,
  resolved_model text not null,
  route_reason text not null,
  policy_version text not null,
  pricing_version text not null,
  effective_group text not null,
  source_checked_at text not null,
  input_tokens integer not null,
  cached_input_tokens integer not null default 0,
  output_tokens integer not null,
  tool_calls integer not null default 0,
  retry_count integer not null default 0,
  fallback_from text,
  fallback_reason text,
  status text not null,
  created_at text not null
);

The important fields are resolved_model, pricing_version, effective_group, token buckets, retries, and fallback metadata. If those are missing, the triage loop will drift into opinion.

Hash account identifiers before they enter analytical logs. Keep API keys, raw prompts, private documents, and customer-identifying text out of the ledger.

Recreate the expected spend

The first calculation should use the exact assumption that approved the rollout. If the rollout forecast assumed 80 percent deepseek-v4-flash and 20 percent deepseek-v4-pro, encode that assumption as data.

from dataclasses import dataclass
from decimal import Decimal


@dataclass(frozen=True)
class Rate:
    input_per_m: Decimal
    cached_input_per_m: Decimal
    output_per_m: Decimal


rates = {
    "deepseek-v4-flash": Rate(Decimal("0.638"), Decimal("0.0203"), Decimal("1.914")),
    "deepseek-v4-pro": Rate(Decimal("1.914"), Decimal("0.0637"), Decimal("5.742")),
    "glm-5.1": Rate(Decimal("2.10"), Decimal("0.6800"), Decimal("6.60")),
    "kimi-k3": Rate(Decimal("4.50"), Decimal("0.90"), Decimal("22.50")),
}


def estimate_usd(model: str, input_tokens: int, cached_input_tokens: int, output_tokens: int) -> Decimal:
    rate = rates[model]
    fresh_input = max(input_tokens - cached_input_tokens, 0)
    million = Decimal("1000000")
    return (
        Decimal(fresh_input) / million * rate.input_per_m
        + Decimal(cached_input_tokens) / million * rate.cached_input_per_m
        + Decimal(output_tokens) / million * rate.output_per_m
    )


print(estimate_usd("deepseek-v4-flash", 2_000_000, 1_500_000, 300_000))

This example is deliberately small. In production, load rates from a dated snapshot rather than hard-coding them in application code. The goal is to reproduce the forecast with the same source date before comparing it with observed usage.

Query by feature first

Most surprises begin at the feature boundary. A new assistant panel, document importer, code reviewer, or nightly job can change token shape without changing the model route.

select
  feature,
  count(*) as requests,
  sum(input_tokens) as input_tokens,
  sum(cached_input_tokens) as cached_input_tokens,
  sum(output_tokens) as output_tokens,
  sum(retry_count) as retries
from gateway_request_ledger
where created_at >= :window_start
  and created_at < :window_end
group by feature
order by output_tokens desc;

If one feature owns the delta, keep the review there. Check whether the prompt template grew, whether retrieved context expanded, whether output caps changed, or whether the job began running more often.

If no feature owns the delta, move to route mix.

Query by resolved model

The requested model field is useful for client behavior. The resolved model field is useful for cost explanation.

select
  resolved_model,
  pricing_version,
  effective_group,
  count(*) as requests,
  sum(input_tokens) as input_tokens,
  sum(cached_input_tokens) as cached_input_tokens,
  sum(output_tokens) as output_tokens
from gateway_request_ledger
where created_at >= :window_start
  and created_at < :window_end
group by resolved_model, pricing_version, effective_group
order by output_tokens desc;

This query catches route-mix drift. Maybe a planning workflow moved from deepseek-v4-flash to deepseek-v4-pro. Maybe a long-context batch started using kimi-k3. Maybe a GLM route became attractive for reasoning tasks but produced longer outputs than the forecast allowed.

The answer may be acceptable. The important part is that the gateway can show it.

Look for output drift

Output is often where a quiet variance hides. A prompt that asks for "a complete analysis" can produce a much larger completion than the forecast expected. A tool-heavy agent may also turn short input into long status reports.

Track these ratios:

Metric	Formula	Failure signal
Output share	output_tokens / (input_tokens + output_tokens)	The response is larger than planned
Cache share	cached_input_tokens / input_tokens	Repeated context is not being reused
Retry pressure	retry_count / requests	Transient failures are becoming spend
Fallback share	fallback_requests / requests	Route policy is doing hidden work

A variance loop should not stop at "model X costs more." It should ask whether the workload used model X as intended.

Check policy variance

Policy changes are easy to miss because they may be correct individually. A retry limit increases from one to three. A fallback rule starts allowing a stronger model for customer-visible work. An account moves from one group to another. The gateway still behaves as designed, but the invoice changes.

Use policy versions as first-class evidence:

select
  policy_version,
  route_reason,
  effective_group,
  count(*) as requests,
  sum(retry_count) as retries,
  count(fallback_from) as fallback_requests
from gateway_request_ledger
where created_at >= :window_start
  and created_at < :window_end
group by policy_version, route_reason, effective_group
order by requests desc;

If spend changed after a policy version changed, the review should include the rollout ticket. The fix may be a lower output cap, a narrower fallback rule, or a separate budget for high-risk jobs.

Produce a triage answer

The final answer should be short enough for finance and detailed enough for engineering.

Use this shape:

Section	What to write
Conclusion	One sentence explaining the main variance driver
Source window	Exact timestamps used in the query
Pricing evidence	Source URL, checked time, and pricing version
Model mix	Top routes by observed spend
Token shape	Input, cached input, output, retry, and fallback deltas
Policy changes	Any policy or account-group change in the window
Unknowns	Missing fields that weaken the conclusion
Action	The smallest change that prevents repeat confusion

Here is an example:

Observed spend exceeded forecast mainly because the support-summary feature
used 2.4x the planned output tokens after policy-2026-09-03 raised max_tokens
from 700 to 1800. Pricing source was https://aiwave.live/api/pricing checked at
2026-09-03T13:11:00Z with pricing_version a42d372ccf0b5dd13ecf71203521f9d2.
No raw prompt data was needed for this review.

That is the kind of answer a buyer can inspect. It does not claim perfect forecasting. It shows a dated, reproducible chain of evidence.

Source links to keep close

Keep these sources beside the variance review:

Source	Use
AIWave pricing	Public gateway pricing context
https://aiwave.live/api/pricing	Machine-readable pricing snapshot
AIWave models docs	Model catalog and route context
AIWave trust page	Public trust and operational posture
DeepSeek pricing	Provider bucket context for DeepSeek routes
QwenCloud pricing	Context, cache, tool, and failed-call billing behavior

Recheck these sources before copying numbers into a long-lived runbook. Model ids, cache ratios, output prices, and account policy can change after a successful rollout.

Final checklist

Before a spend review closes, make sure the gateway can answer these questions:

1. Which pricing source and version were used?
2. Which feature produced the variance?
3. Which exact model routes ran?
4. Which account group applied?
5. How much usage was fresh input, cached input, and output?
6. Did retries or fallbacks contribute?
7. Which policy version admitted the calls?
8. What data is missing from the ledger?
9. What small control prevents the same confusion next time?

The value of a gateway is not only that it can route requests. It should explain what happened after the route ran. A spend variance triage loop turns a surprising invoice into a dated engineering review, with enough evidence for platform, support, and finance to make the next decision without guessing.