DEV Community: Mattias chaw

Build Synthetic Shadow Traffic Gates for AI API Route Changes

Mattias chaw — Sat, 26 Sep 2026 13:55:38 +0000

AI API route changes are rarely simple config edits. A provider adds a new model, a gateway changes a cache ratio, a team wants to move a coding workload to a smaller route, or an incident pushes you toward a fallback. The tempting move is to switch a percentage of live traffic and watch the dashboard.

That is a weak gate.

Live canaries are useful, but they happen after real users are already involved. A synthetic shadow traffic gate gives you an earlier checkpoint. It replays carefully redacted request shapes against candidate routes, compares the evidence against the current production route, and only then decides whether a live canary deserves to exist.

I will use AIWave as the concrete source example because its public pricing and route sources can be checked without a private account. During this run on September 26, 2026, https://aiwave.live/api/v1/pricing returned 56 dated USD model rows, currency USD, unit per_1m_text_tokens, checked=2026-09-10, updated_at=2026-09-18, and pricing version 83f77abde81ee3a096a672ed959ccc096f5d37a45c177ae8e03229456b5415a5. The live route table at https://aiwave.live/api/pricing returned success=true, 73 route rows, pricing version a42d372ccf0b5dd13ecf71203521f9d2, auto_groups=["default"], and group ratios default=1 and vip=0.9.

The point is not to claim one route is universally better. The point is to make every route change explain itself before it touches production traffic.

What a Shadow Gate Should Decide

A route gate should answer five questions before promotion:

Does the candidate route accept the same request shapes?
Does it return the response contract your client expects?
Does the budget envelope still hold under dated pricing?
Does the route behave well enough under representative token sizes?
Can you roll back using a clear evidence record?

That sounds ordinary, but most teams skip one of these. They test one happy path, check one bill estimate, then promote. The failure shows up later as a client parse error, a surprise output pattern, or a support thread where nobody knows which route table version was active.

A synthetic gate is useful because it turns route change approval into a repeatable artifact.

Use Request Shapes, Not Private Payloads

Do not replay raw customer prompts into an experiment harness. You usually do not need them. You need the shape of the request:

model alias requested by the client
endpoint family
input token range
expected output range
streaming or non-streaming mode
tool and structured-output expectations
cacheable prefix length, if relevant
redacted error class, if the request came from an incident

For a gateway, the shape is the contract. The exact private content should stay out of the test artifact.

Here is a compact fixture format:

{
  "fixture_id": "route-shadow-001",
  "endpoint": "/v1/chat/completions",
  "client_model": "deepseek-v4-pro",
  "candidate_model": "deepseek-v4-flash",
  "mode": "streaming",
  "input_tokens_bucket": "8k_to_16k",
  "expected_output_tokens_bucket": "1k_to_2k",
  "requires_json_object": true,
  "requires_tool_calls": false,
  "cache_prefix_tokens_bucket": "4k_to_8k",
  "redaction_status": "synthetic_payload_only"
}

The fixture is boring on purpose. It carries enough detail to drive the route, budget, and response checks without carrying a secret, a prompt, or a customer identifier.

Pin the Pricing Inputs

The gate must not read a live price table in one step and use a different table in the next. Pin the pricing version and group multiplier before the run starts.

For AIWave, the public static catalog is a dated base-rate catalog. The dynamic route table is the live route source and includes the current group ratio map. That separation matters. A gate should record both:

{
  "pricing_catalog_url": "https://aiwave.live/api/v1/pricing",
  "pricing_catalog_version": "83f77abde81ee3a096a672ed959ccc096f5d37a45c177ae8e03229456b5415a5",
  "route_table_url": "https://aiwave.live/api/pricing",
  "route_table_version": "a42d372ccf0b5dd13ecf71203521f9d2",
  "catalog_checked": "2026-09-10",
  "catalog_updated_at": "2026-09-18",
  "group_ratios": {
    "default": 1,
    "vip": 0.9
  }
}

Do not infer an effective bill from a stale table. Do not mix account-group logic into a public base-rate row. Do not publish internal usage or customer data as proof. The gate only needs the route facts required to approve or reject the change.

Sample Without Creating a Second Incident

The safest sample is not a random slice of raw production traffic. Start with workload families that already have a known engineering owner: onboarding calls, coding-agent calls, retrieval-heavy calls, long-context analysis, or structured-output calls. Then turn each family into a small fixture set that covers the important shapes.

For each family, keep the sample narrow:

one short fixture for fast acceptance
one median fixture for normal route behavior
one large fixture near the approved token envelope
one fixture that exercises streaming or structured output
one incident-derived shape with only synthetic content

That gives the gate enough surface area to catch route mistakes without becoming a shadow data lake. If a workload needs stronger proof, add a fixture on purpose and record why it exists. Do not let the harness quietly grow into an uncontrolled copy of production.

Build the Replay Harness

The replay harness can be small. It loads fixtures, expands synthetic prompts to the right size bucket, runs the current route and candidate route, then records structured evidence.

import os
import time
from dataclasses import dataclass

from openai import OpenAI


@dataclass
class Fixture:
    fixture_id: str
    current_model: str
    candidate_model: str
    input_tokens_bucket: str
    expected_output_tokens_bucket: str
    requires_json_object: bool


client = OpenAI(
    api_key=os.environ.get("AIWAVE_API_KEY") or "YOUR_API_KEY_HERE",
    base_url="https://aiwave.live/v1",
)


def synthetic_prompt(fixture: Fixture) -> str:
    return (
        "Summarize a redacted engineering incident. "
        "Return only a JSON object with keys: summary, risk, rollback. "
        "Use synthetic details only. " * 80
    )


def run_route(model: str, fixture: Fixture) -> dict:
    started = time.time()
    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": synthetic_prompt(fixture)}],
        response_format={"type": "json_object"} if fixture.requires_json_object else None,
        temperature=0,
    )
    elapsed_ms = round((time.time() - started) * 1000)
    content = response.choices[0].message.content or ""
    return {
        "model": model,
        "elapsed_ms": elapsed_ms,
        "content_prefix": content[:120],
        "finish_reason": response.choices[0].finish_reason,
        "usage": response.usage.model_dump() if response.usage else None,
    }

This is not a load test. It is a contract test with enough token mass to catch route-shape mistakes. You can run load later, after the contract survives.

Compare Evidence, Not Vibes

The gate should produce a result that a reviewer can inspect without rerunning the experiment.

At minimum, compare:

request accepted or rejected
response parse contract
finish reason
token accounting fields
approximate latency bucket
budget envelope under pinned pricing
error class, if any
route and pricing versions

Avoid a single pass/fail flag without details. You want a small evidence table:

{
  "fixture_id": "route-shadow-001",
  "current_route": {
    "model": "deepseek-v4-pro",
    "status": "passed",
    "finish_reason": "stop",
    "parse_contract": "json_object_ok"
  },
  "candidate_route": {
    "model": "deepseek-v4-flash",
    "status": "needs_review",
    "finish_reason": "stop",
    "parse_contract": "json_object_ok",
    "review_reason": "budget_ok_but_output_style_changed"
  },
  "pricing_version": "83f77abde81ee3a096a672ed959ccc096f5d37a45c177ae8e03229456b5415a5",
  "route_table_version": "a42d372ccf0b5dd13ecf71203521f9d2"
}

The important detail is needs_review. A route can be technically valid and still not ready for promotion. Maybe it formats JSON correctly but changes tone, omits a field under long context, or uses a different finish reason pattern. The gate should give humans a place to say, "This is close, but not live yet."

Add Promotion Rules

Once the replay result exists, promotion rules become much easier to state:

Every fixture in the target workload family must pass request acceptance.
Structured-output fixtures must parse without repair logic.
Budget estimates must stay inside the approved envelope.
Unknown capability fields remain blockers unless the workload does not depend on them.
Any model listed in the catalog but not routable for the needed endpoint stays out of promotion.
The reviewer must see the pricing version, route table version, and rollback target.

That last item is not paperwork. It is what lets an operator reverse a change without hunting through chat logs.

Keep Unknowns Explicit

AI model catalogs often contain partial information. A public price row does not prove tool calling, JSON mode, vision support, or a Responses-compatible endpoint. A route table row does not prove your exact workload is safe.

Mark unknowns as unknowns.

{
  "capability_checks": {
    "json_object": "tested",
    "streaming": "tested",
    "tool_calls": "not_required",
    "vision": "unknown",
    "responses_api": "unknown"
  }
}

This looks conservative, but it saves time. It prevents a team from turning a model name into a capability claim.

What to Log After Promotion

If the shadow gate passes and you later run a live canary, keep the log narrow:

route change ID
route table version
pricing catalog version
candidate model
workload family
canary percentage
start and end time
rollback route
reviewer
result

Do not put prompts, reusable keys, customer identifiers, or precise private workload figures into the public article, the run record, or screenshots.

The Useful Failure Mode

The best outcome is not always promotion. Sometimes the useful outcome is a blocked route change with a clear reason:

a response contract changed
token accounting fields were missing
the budget envelope failed
endpoint support was unknown
a candidate route was catalog-only for this workload
rollback evidence was incomplete

That is a good day. You found the risk before users met it.

Closing Pattern

Here is the pattern I would use:

Convert recent production classes into redacted request shapes.
Pin the public pricing catalog and live route table versions.
Replay synthetic fixtures against current and candidate routes.
Compare acceptance, parse contract, budget, and route evidence.
Promote only when the evidence and rollback target are reviewable.

Shadow traffic gates do not replace production monitoring. They make production monitoring less dramatic. And for AI API gateways, less drama is a feature.

Build Route Decision Records for AI API Gateway Changes

Mattias chaw — Fri, 25 Sep 2026 13:08:37 +0000

Build Route Decision Records for AI API Gateway Changes

AI API route changes often look smaller than they are.

A pull request may change one model ID. A configuration update may switch a gateway row. A product owner may ask for longer output. A finance reviewer may approve a budget from last week's pricing table. Each change is understandable on its own, but the combined result can alter quality, latency, spend, privacy posture, and rollback behavior.

That is why route changes need a decision record.

An AI API route decision record is a compact release artifact. It says which route is being changed, which dated sources were checked, what behavior is expected, which unknowns remain, and how the team will reverse the change if production evidence disagrees.

This is not a compliance ceremony. It is a practical way for Tier 1 and Tier 2 engineering teams to avoid shipping a route change whose evidence is scattered across chats, spreadsheets, dashboards, and tribal memory.

I will use AIWave as the concrete example because its public pricing and route sources can be checked without a private account. During this run on September 25, 2026, https://aiwave.live/api/v1/pricing returned 56 dated USD model rows, currency USD, unit per_1m_text_tokens, checked=2026-09-10, updated_at=2026-09-18, and pricing version 83f77abde81ee3a096a672ed959ccc096f5d37a45c177ae8e03229456b5415a5. The live route table at https://aiwave.live/api/pricing returned success=true, 73 route rows, pricing version a42d372ccf0b5dd13ecf71203521f9d2, auto_groups=["default"], and group ratios default=1 and vip=0.9.

Those values are dated inputs for a decision record. They are not timeless promises.

Why route changes need a record

OpenAI-compatible APIs make client code portable. They do not make route decisions self-explanatory.

The same SDK call can hide several changes:

Change	What can move
Model ID	Quality, context behavior, tool support, output length
Provider route	Availability, latency, failure mode, receipt fields
Pricing source	Forecast basis, effective date, account group multiplier
Retry policy	User-visible delay, duplicate work, budget variance
Output cap	Product quality, completion tokens, timeout risk
Data class	What crosses the API boundary
Rollback path	Whether the previous route can be restored cleanly

A route decision record does not solve those issues automatically. It puts them in one place so the team can review the actual decision rather than reconstruct it after an incident.

A small decision record schema

Keep the schema boring. Boring survives release pressure.

{
  "decision_id": "rdr-support-summary-2026-09-25",
  "status": "proposed",
  "owner": "platform-ai",
  "checked_at": "2026-09-25T13:20:00Z",
  "client_shape": "openai_chat_completions",
  "change": {
    "from_model": "deepseek-v4-flash",
    "to_model": "glm-5",
    "reason": "evaluate longer reasoning for support summaries"
  },
  "sources": {
    "dated_pricing": {
      "url": "https://aiwave.live/api/v1/pricing",
      "checked": "2026-09-10",
      "updated_at": "2026-09-18",
      "pricing_version": "83f77abde81ee3a096a672ed959ccc096f5d37a45c177ae8e03229456b5415a5"
    },
    "live_routes": {
      "url": "https://aiwave.live/api/pricing",
      "pricing_version": "a42d372ccf0b5dd13ecf71203521f9d2",
      "route_rows": 73,
      "group_ratio": {
        "default": 1,
        "vip": 0.9
      }
    }
  },
  "release_limits": {
    "max_output_tokens": 600,
    "max_retry_attempts": 1,
    "timeout_ms": 45000
  },
  "receipt_contract": [
    "request_id",
    "model_id",
    "prompt_tokens",
    "completion_tokens",
    "retry_attempts",
    "group",
    "charged_amount"
  ],
  "rollback": {
    "previous_model": "deepseek-v4-flash",
    "trigger": "quality, receipt, or budget gate fails"
  }
}

There is no prompt text in that object. There is no reusable API key. There is no customer identifier. The record should be safe to attach to a release ticket while still being precise enough to audit.

Separate the decision from the estimate

A route decision record can include budget math, but it should not pretend to be the final bill.

from __future__ import annotations

from dataclasses import dataclass


@dataclass(frozen=True)
class RateRow:
    input_usd_per_1m: float
    cache_hit_usd_per_1m: float | None
    output_usd_per_1m: float


def usd_per_million(tokens: int, rate: float) -> float:
    return tokens / 1_000_000 * rate


def estimate_review_cost(
    input_tokens: int,
    cached_input_tokens: int,
    output_tokens: int,
    rate: RateRow,
    group_ratio: float,
) -> float:
    fresh_input = max(input_tokens - cached_input_tokens, 0)
    cache_rate = rate.cache_hit_usd_per_1m or rate.input_usd_per_1m
    subtotal = (
        usd_per_million(fresh_input, rate.input_usd_per_1m)
        + usd_per_million(cached_input_tokens, cache_rate)
        + usd_per_million(output_tokens, rate.output_usd_per_1m)
    )
    return round(subtotal * group_ratio, 6)

The estimate is useful because it is reproducible. It is still only a review calculation. Final usage can move because output length, retries, cache behavior, account group, provider route, and source rows can move.

The decision record should therefore store both the estimate and the conditions that would make the estimate invalid.

{
  "estimate_scope": {
    "requests": 1000,
    "input_tokens": 120000000,
    "cached_input_tokens": 0,
    "output_tokens": 1800000,
    "group": "default"
  },
  "estimate_unknowns": [
    "final completion length depends on production prompts",
    "cache behavior is not assumed unless receipts prove it",
    "final charge depends on selected account or key group"
  ]
}

That wording matters. It keeps the decision honest without slowing the team down.

Require source version checks

The record should fail closed when a source version moves unexpectedly.

import json
import urllib.request


def load_json(url: str) -> dict:
    with urllib.request.urlopen(url, timeout=20) as response:
        return json.loads(response.read().decode("utf-8"))


record = json.load(open("route-decision-record.json", encoding="utf-8"))
pricing = load_json(record["sources"]["dated_pricing"]["url"])
routes = load_json(record["sources"]["live_routes"]["url"])

if pricing["pricing_version"] != record["sources"]["dated_pricing"]["pricing_version"]:
    raise SystemExit("dated_pricing_version_moved")

if routes["pricing_version"] != record["sources"]["live_routes"]["pricing_version"]:
    raise SystemExit("live_route_version_moved")

For a deliberate re-approval, update the record and make the diff visible. For an accidental movement, stop and review.

This is especially important when a route has both a public dated rate card and a live route table. Those sources answer different questions:

The dated pricing JSON is a forecast source.
The live route table is current route availability and group metadata.

Treating them as one interchangeable table creates confusion later.

Make the acceptance matrix explicit

A useful decision record has a small acceptance matrix.

{
  "acceptance": [
    {
      "gate": "route_exists",
      "method": "live route table contains target model ID",
      "required": true
    },
    {
      "gate": "pricing_row_exists",
      "method": "dated pricing source contains forecast row",
      "required": true
    },
    {
      "gate": "quality_slice_passes",
      "method": "10 redacted fixtures pass reviewer rubric",
      "required": true
    },
    {
      "gate": "receipt_fields_present",
      "method": "request receipt includes model, tokens, group, charge, retry count",
      "required": true
    },
    {
      "gate": "budget_band",
      "method": "review cost remains inside approved variance band",
      "required": true
    }
  ]
}

The point is not to add paperwork. The point is to prevent vague approval.

"Looks good" is hard to audit.

"Ten redacted support-summary fixtures passed the rubric, the pricing version is pinned, and receipt fields include group and retry count" is much easier to defend.

Include rollback before rollout

Rollback should be part of the decision, not an afterthought.

{
  "rollback": {
    "previous_model": "deepseek-v4-flash",
    "previous_record_id": "rdr-support-summary-2026-09-18",
    "allowed_until": "2026-10-02",
    "triggers": [
      "quality_slice_regression",
      "missing_receipt_field",
      "timeout_rate_above_gate",
      "budget_variance_above_gate"
    ],
    "verification_after_rollback": [
      "live route resolves previous model",
      "fixture set passes previous rubric",
      "new receipts show previous model ID"
    ]
  }
}

This small block changes the release conversation. The team does not need to improvise when a route misbehaves. It already knows which signal triggers reversal and which evidence proves reversal worked.

Route decision records and public copy

Internal evidence should be stronger than public language.

Safe public copy says:

This route was reviewed against a dated pricing source and a live route table before release.

Risky public copy says:

This route will always reduce cost or improve quality.

The first statement is about process and evidence. The second predicts an outcome without knowing the future workload, prompts, output length, retries, cache status, account group, provider state, or pricing source.

Good route records help teams move faster because they reduce argument. Each record answers the same questions:

What changed?
Which sources were checked?
Which model IDs and group rules apply?
Which acceptance gates passed?
Which unknowns remain?
Which receipt fields will prove runtime behavior?
Which rollback trigger reverses the change?

If a team can answer those questions before production traffic arrives, the route change is no longer a leap of faith. It is a bounded decision with dated evidence.

Your First AIWave API Call in 5 Minutes

Mattias chaw — Wed, 23 Sep 2026 20:15:58 +0000

Your first AIWave API call in 5 minutes

Run one small request before building around an API route. This guide creates a key, calls deepseek-v4-flash, and checks the request record in Console.

AIWave uses the OpenAI-compatible base URL https://aiwave.live/v1. Existing clients usually need a new base URL and key. Copy the model ID exactly.

1. Create an API key

Open AIWave Console, go to API Keys, choose Create key, and copy it once. Keep it server side, never in a browser bundle, screenshot, ticket, or repository.

export AIWAVE_API_KEY="sk-your-api-key"
test -n "$AIWAVE_API_KEY" && echo "key is set"

Validation: confirms the variable exists without displaying the key.

On PowerShell:

$env:AIWAVE_API_KEY = "sk-your-api-key"
if ($env:AIWAVE_API_KEY) { "key is set" }

Validation: the PowerShell equivalent. Replace the placeholder locally.

2. Run the smallest useful request

The live quickstart uses deepseek-v4-flash and a short arithmetic prompt, making the response easy to check.

curl https://aiwave.live/v1/chat/completions \
  -H "Authorization: Bearer $AIWAVE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"deepseek-v4-flash","messages":[{"role":"user","content":"What is 2+2? Reply with just the number."}]}'

Validation: endpoint, path, headers, and model ID were copied from the live AIWave quickstart on 2026-09-23. Run it with your own key. Expect HTTP 200 and 4.

deepseek-v4-flash is a reasoning model, so the response may also contain a reasoning_content field. The answer you want is in choices[0].message.content.

3. Use the same route from Python

For an OpenAI Python client, change the endpoint configuration.

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AIWAVE_API_KEY"],
    base_url="https://aiwave.live/v1",
)

response = client.chat.completions.create(
    model="deepseek-v4-flash",
    messages=[
        {"role": "user", "content": "What is 2+2? Reply with just the number."}
    ],
)

print(response.choices[0].message.content)

Validation: matches the live quickstart. Run with openai installed and the variable set; expect 4.

4. Use the same route from Node.js

The OpenAI JavaScript client uses the same pattern.

npm install openai

Validation: installs the client only; it sends no request.

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AIWAVE_API_KEY,
  baseURL: "https://aiwave.live/v1",
});

const response = await client.chat.completions.create({
  model: "deepseek-v4-flash",
  messages: [
    { role: "user", content: "What is 2+2? Reply with just the number." },
  ],
});

console.log(response.choices[0].message.content);

Validation: matches the live quickstart. Run with Node.js and the variable set; expect 4.

Copy-ready smoke-test files

These files add status and error checks while keeping the request visible.

Bash

#!/usr/bin/env bash
set -Eeuo pipefail

: "${AIWAVE_API_KEY:?Set AIWAVE_API_KEY before running this file}"

endpoint="https://aiwave.live/v1/chat/completions"
payload='{"model":"deepseek-v4-flash","messages":[{"role":"user","content":"What is 2+2? Reply with just the number."}]}'

body_file="$(mktemp)"
trap 'rm -f "$body_file"' EXIT

status="$(curl --silent --show-error --output "$body_file" --write-out '%{http_code}' \
  "$endpoint" \
  -H "Authorization: Bearer $AIWAVE_API_KEY" \
  -H 'Content-Type: application/json' \
  --data "$payload")"

echo "HTTP $status"
if [ "$status" != "200" ]; then
  cat "$body_file"
  exit 1
fi

cat "$body_file"

Validation: checks status and body without printing the key. Run with a controlled key.

Python

import json
import os
import sys
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen

api_key = os.environ.get("AIWAVE_API_KEY")
if not api_key:
    raise SystemExit("Set AIWAVE_API_KEY before running this file")

request = Request(
    "https://aiwave.live/v1/chat/completions",
    data=json.dumps({
        "model": "deepseek-v4-flash",
        "messages": [{
            "role": "user",
            "content": "What is 2+2? Reply with just the number.",
        }],
    }).encode("utf-8"),
    headers={
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json",
    },
    method="POST",
)

try:
    with urlopen(request, timeout=30) as response:
        data = json.load(response)
        print(json.dumps(data, indent=2))
except HTTPError as error:
    print(f"HTTP {error.code}", file=sys.stderr)
    print(error.read().decode("utf-8", errors="replace"), file=sys.stderr)
    raise SystemExit(1)
except URLError as error:
    raise SystemExit(f"Network error: {error.reason}")

Validation: Python 3 standard HTTP client; compare status and content with curl.

Node.js

import OpenAI from "openai";

const apiKey = process.env.AIWAVE_API_KEY;
if (!apiKey) {
  throw new Error("Set AIWAVE_API_KEY before running this file");
}

const client = new OpenAI({
  apiKey,
  baseURL: "https://aiwave.live/v1",
});

try {
  const response = await client.chat.completions.create({
    model: "deepseek-v4-flash",
    messages: [{
      role: "user",
      content: "What is 2+2? Reply with just the number.",
    }],
  });

  const message = response.choices?.[0]?.message?.content;
  if (!message) throw new Error("The response did not contain assistant content");
  console.log(message);
} catch (error) {
  console.error("The first request failed. Check the HTTP error and request ID.");
  throw error;
}

Validation: install openai, run as an ES module, and expect 4; the key is not printed.

5. Check the result before scaling up

After success, inspect the request record in Console: model ID, usage, effective rate, group, and status. Check the dated pricing page before budgeting. Connectivity is not workload fit; compare models with the same measured task.

If the request fails

400: check JSON, model, messages, and content type.
401: check the key and Authorization header without printing the key.
402: check balance and billing state in Console.
404: copy the exact current model ID from Models. Do not invent an alias.
429: honor Retry-After when present and use bounded backoff.
5xx: save the request ID, retry once, and check Status.

See Request recovery and the SSE guide.

What to do next

Keep the base URL and model in configuration. Check Pricing before a larger workload, then use the same request shape in the agent guide.

Self-check

[x] Base URL and model ID copied from the live quickstart on 2026-09-23.
[x] Examples use sk-your-api-key only as a placeholder.
[x] Each code block has a validation note.
[x] No credentials, customer data, internal metrics, or balance amount.
[x] Internal links target live sign-in, docs, models, pricing, and status pages.
[ ] Run a live request with a controlled test key before publishing.
[ ] Run curl -I checks for every internal link immediately before publication.
[ ] Run the final banned-word scan and word-count/code-ratio check.

Build Streaming Backpressure Contracts for AI API Routes

Mattias chaw — Wed, 23 Sep 2026 13:10:19 +0000

Build Streaming Backpressure Contracts for AI API Routes

Streaming makes an AI API feel fast. It can also hide the moment where a system stops being controllable.

A non-streaming request has one obvious boundary: the response arrives or it does not. A streaming request has more moving parts. Tokens arrive over time. The client may pause. The browser tab may close. A worker may hit a timeout while the upstream model is still producing. A retry may start while the first stream is still unwinding. A budget estimate may assume the happy path even though the production route has to handle partial output.

That is why streaming routes need a backpressure contract.

The contract is not a benchmark. It does not claim that one provider streams better than another. It is a small pre-release artifact that says how a team expects a streaming route to behave when the client slows down, disconnects, retries, or reaches an output cap.

I will use AIWave as the concrete source example because its public pricing data can be checked without a private account. During this run on September 23, 2026, https://aiwave.live/api/v1/pricing returned 56 model rows, currency USD, unit per_1m_text_tokens, checked=2026-09-10, updated_at=2026-09-18, and pricing version 83f77abde81ee3a096a672ed959ccc096f5d37a45c177ae8e03229456b5415a5. The live route table at https://aiwave.live/api/pricing returned success=true, 73 route rows, pricing version a42d372ccf0b5dd13ecf71203521f9d2, and group ratios default=1 and vip=0.9.

Those values are dated inputs for an engineering check. They are not timeless pricing promises.

What backpressure means here

In a streaming AI route, backpressure is the gap between how quickly the upstream can produce chunks and how quickly the client can safely consume them.

That gap appears in ordinary production paths:

Surface	Example
Browser clients	The user switches tabs and the UI event loop slows down
Server-sent events	A proxy buffers chunks before the app receives them
Workers	The job runtime has a hard timeout while the stream is still active
Mobile networks	The connection stalls after partial output
Product policy	The route must stop at a cost or token boundary
Observability	The receipt must explain what was generated before cancellation

The contract should not try to model every network failure. It should define the behaviors your team will support and verify before the route ships.

Start with a route contract object

Keep the object synthetic. It should describe the stream shape, not a real prompt.

{
  "contract_id": "support-stream-route-2026-09-23",
  "checked_at": "2026-09-23T13:20:00Z",
  "client_shape": "openai_chat_completions_stream",
  "requested_model": "deepseek-v4-flash",
  "account_group": "default",
  "stream_policy": {
    "max_output_tokens": 1200,
    "client_read_timeout_ms": 45000,
    "idle_chunk_timeout_ms": 12000,
    "max_retry_attempts": 1,
    "retry_after_first_chunk": false,
    "cancel_upstream_on_disconnect": true
  },
  "receipt_contract": [
    "request_id",
    "model_id",
    "stream_started_at",
    "first_chunk_ms",
    "last_chunk_ms",
    "finish_reason",
    "prompt_tokens",
    "completion_tokens",
    "group",
    "charged_amount"
  ]
}

There is no reusable key in that object. There is no user content. There is no customer identifier. This makes it safe to attach to a pull request, incident rehearsal, or procurement review.

Resolve the model before streaming

Do not let the first streamed chunk be the first time the route is checked. Resolve the requested model against a dated pricing snapshot and the current route table.

from __future__ import annotations

import json
import urllib.request


def load_json(url: str) -> dict:
    with urllib.request.urlopen(url, timeout=20) as response:
        return json.loads(response.read().decode("utf-8"))


pricing = load_json("https://aiwave.live/api/v1/pricing")
routes = load_json("https://aiwave.live/api/pricing")


def pricing_row(model_id: str) -> dict | None:
    return next((row for row in pricing["models"] if row["id"] == model_id), None)


def live_route_names() -> set[str]:
    names: set[str] = set()
    for row in routes.get("data", []):
        value = row.get("model_name")
        if isinstance(value, str):
            names.add(value)
    return names


model = "deepseek-v4-flash"
if pricing_row(model) is None:
    raise SystemExit("missing_pricing_row")
if model not in live_route_names():
    raise SystemExit("missing_live_route")

This check is intentionally boring. It prevents a release where the streaming code path is correct but the route name is stale.

Define the stream state machine

A streaming route should have named states. Without states, retry and cancellation behavior becomes tribal knowledge.

from enum import Enum


class StreamState(str, Enum):
    CREATED = "created"
    UPSTREAM_CONNECTED = "upstream_connected"
    FIRST_CHUNK = "first_chunk"
    FLOWING = "flowing"
    CLIENT_SLOWED = "client_slowed"
    CLIENT_DISCONNECTED = "client_disconnected"
    CANCEL_SENT = "cancel_sent"
    COMPLETED = "completed"
    FAILED = "failed"

The important question is not whether your enum looks like this. The important question is whether everyone agrees on transitions:

Can a request retry after FIRST_CHUNK?
Does a client disconnect cancel the upstream request?
Does a proxy timeout count as failed, canceled, or completed partial output?
Which states create billable completion tokens?
Which receipt fields are required for each terminal state?

The contract should answer those questions before production traffic does.

Test slow readers

The easiest way to miss backpressure is to test only a fast local client. Add a fake reader that intentionally sleeps between chunks.

import asyncio
from collections.abc import AsyncIterator


async def slow_reader(chunks: AsyncIterator[bytes], delay_ms: int) -> int:
    received = 0
    async for chunk in chunks:
        received += len(chunk)
        await asyncio.sleep(delay_ms / 1000)
    return received

Your test double can feed synthetic chunks instead of calling a model.

async def synthetic_stream() -> AsyncIterator[bytes]:
    for index in range(12):
        yield f"data: chunk-{index}\\n\\n".encode("utf-8")
        await asyncio.sleep(0.05)


async def test_slow_reader_contract() -> None:
    received = await slow_reader(synthetic_stream(), delay_ms=250)
    assert received > 0

That test does not prove provider behavior. It proves your app can represent a slow stream without confusing it with a model error.

Set retry boundaries

Retries are where streaming gets expensive and confusing.

For a non-streaming request, a retry after a timeout may be acceptable if the operation is idempotent enough for your product. For a streaming route, a retry after the first chunk can duplicate partial work, double output, and create a receipt that is difficult to explain.

Use a small rule function:

def retry_allowed(state: str, retry_after_first_chunk: bool) -> bool:
    if state in {"created", "upstream_connected"}:
        return True
    if state in {"first_chunk", "flowing", "client_slowed"}:
        return retry_after_first_chunk
    return False


assert retry_allowed("created", False) is True
assert retry_allowed("first_chunk", False) is False
assert retry_allowed("first_chunk", True) is True
assert retry_allowed("completed", True) is False

The policy may vary by product. What matters is that it is explicit.

Estimate partial-output budget impact

Do not pretend a streaming budget estimate is a final bill. Treat it as a release review calculation.

def estimate_stream_review_cost(
    input_tokens: int,
    output_tokens: int,
    input_rate: float,
    output_rate: float,
    group_ratio: float,
) -> float:
    subtotal = (
        input_tokens / 1_000_000 * input_rate
        + output_tokens / 1_000_000 * output_rate
    )
    return round(subtotal * group_ratio, 6)

Now test stress cases:

scenarios = {
    "planned": 1200,
    "client_disconnect_after_25_percent": 300,
    "timeout_after_75_percent": 900,
    "one_retry_before_first_chunk": 1200,
}

The review should record which scenario drives the largest difference. If the route is safe only when every stream finishes cleanly, the contract is not ready.

Require receipt evidence

The most useful streaming contract is one that your logs can later explain.

REQUIRED_RECEIPT_FIELDS = {
    "request_id",
    "model_id",
    "stream_started_at",
    "first_chunk_ms",
    "last_chunk_ms",
    "finish_reason",
    "prompt_tokens",
    "completion_tokens",
    "group",
    "charged_amount",
}


def missing_receipt_fields(fields: list[str]) -> list[str]:
    return sorted(REQUIRED_RECEIPT_FIELDS - set(fields))

For cancellation-heavy products, add more:

CANCEL_FIELDS = {
    "client_disconnect_ms",
    "upstream_cancel_sent",
    "partial_output_recorded",
}

Do not fill missing fields with guesses. If the route cannot distinguish client_disconnected from upstream_timeout, write that unknown into the review result and decide whether the release can proceed.

A useful review result

The result should be small enough to paste into a release issue:

{
  "contract_id": "support-stream-route-2026-09-23",
  "verdict": "review_required",
  "requested_model": "deepseek-v4-flash",
  "pricing_source": {
    "checked": "2026-09-10",
    "updated_at": "2026-09-18",
    "pricing_version": "83f77abde81ee3a096a672ed959ccc096f5d37a45c177ae8e03229456b5415a5"
  },
  "route_source": {
    "pricing_version": "a42d372ccf0b5dd13ecf71203521f9d2",
    "group_ratio": {"default": 1, "vip": 0.9}
  },
  "checks": {
    "pricing_row": "present",
    "live_route": "present",
    "retry_after_first_chunk": "disabled",
    "cancel_on_disconnect": "required",
    "receipt_contract": "present"
  },
  "review_notes": [
    "partial output must be visible in receipts",
    "retry after first chunk requires separate product approval"
  ]
}

The verdict does not have to be pass or fail. review_required is often the honest result when the route is technically available but the cancellation, retry, or receipt behavior needs owner approval.

Keep public claims narrow

A backpressure contract is good engineering evidence. It is not a broad public claim about speed, uptime, or final cost.

Safe language:

The streaming route was reviewed against a dated pricing source, current route table, retry policy, and receipt contract before release.

Risky language:

Streaming is always faster and predictable for every workload.

The first sentence describes a check. The second makes a promise the route contract cannot prove.

For production AI API teams, that distinction matters. Streaming should improve user experience without weakening budget control, cancellation semantics, or incident evidence. A backpressure contract gives the team a shared way to test that before the first real stream starts flowing.

Build a Route Dry-Run Simulator for AI API Releases

Mattias chaw — Tue, 22 Sep 2026 13:09:58 +0000

Build a Route Dry-Run Simulator for AI API Releases

Most AI API release reviews happen after the route already answered traffic.

That is backwards.

By the time a route is live, the team may already be dealing with user-facing errors, unexpected output length, missing receipt fields, or a pricing source that changed since the original planning note. A Tier 1 or Tier 2 engineering team should be able to test the route contract before any private workload touches the gateway.

A route dry-run simulator is a small harness for that job. It does not call the model. It does not store prompts. It does not need a reusable production key. It checks whether a proposed AI API route can be explained with the evidence the team expects to rely on later.

The simulator answers five questions:

Does the requested model ID resolve to a current route?
Which dated pricing row will reviewers use?
Which account or key group multiplier applies?
What happens to the budget if retries or output tokens move?
Which redacted receipt fields must exist after production traffic runs?

I will use AIWave as the concrete example because its public pricing sources can be checked without a private account. During this run on September 22, 2026, https://aiwave.live/api/v1/pricing returned 56 model rows, currency USD, unit per_1m_text_tokens, checked=2026-09-10, updated_at=2026-09-18, and pricing version 83f77abde81ee3a096a672ed959ccc096f5d37a45c177ae8e03229456b5415a5. The live route table at https://aiwave.live/api/pricing returned success=true, 73 route rows, pricing version a42d372ccf0b5dd13ecf71203521f9d2, and group ratios default=1 and vip=0.9.

Those values are not marketing promises. They are dated inputs for a simulator.

Why dry-run before the first private request?

AI API releases often mix three different concerns:

Concern	Typical owner	Failure mode
Route selection	Platform engineering	Alias points to a different model than the product expected
Pricing evidence	Finance or procurement	Workbook uses a row that is no longer the checked source
Runtime receipts	Application engineering	Production logs cannot explain cache, retry, group, or output behavior

The dry-run simulator does not replace any of those owners. It gives them one shared preflight artifact.

That artifact is especially useful when the release uses an OpenAI-compatible interface. The client call may look stable, but the route behind it can still vary by provider, model alias, context mode, batch mode, cache behavior, tool fee, or account group.

The interface is compatible. The evidence still needs a contract.

A dry-run input object

Keep the simulator input synthetic. It should describe a workload shape, not a real user prompt.

{
  "dry_run_id": "support-triage-route-2026-09-22",
  "checked_at": "2026-09-22T13:20:00Z",
  "client_shape": "openai_chat_completions",
  "requested_model": "deepseek-v4-flash",
  "account_group": "default",
  "traffic_shape": {
    "requests": 1000,
    "input_tokens_per_request": 120000,
    "expected_cached_input_tokens_per_request": 0,
    "max_output_tokens_per_request": 900,
    "max_retry_attempts_per_request": 1
  },
  "receipt_contract": [
    "request_id",
    "model_id",
    "prompt_tokens",
    "completion_tokens",
    "cached_input_tokens",
    "retry_attempts",
    "group",
    "charged_amount"
  ]
}

There is no API key in that file. There is no request body. There is no customer identifier. That makes the object safe to attach to a release review while still being concrete enough for engineers to test.

Load dated pricing and live routes

The simulator needs two different sources:

a dated pricing snapshot for review math
a live route table for current route resolution

from __future__ import annotations

import json
import urllib.request


def load_json(url: str) -> dict:
    with urllib.request.urlopen(url, timeout=20) as response:
        return json.loads(response.read().decode("utf-8"))


pricing = load_json("https://aiwave.live/api/v1/pricing")
routes = load_json("https://aiwave.live/api/pricing")

assert pricing["currency"] == "USD"
assert pricing["unit"] == "per_1m_text_tokens"
assert routes["success"] is True

Do not hide these source versions. Print or store the checked, updated_at, and pricing_version values in the simulator result. A dry run without source versions is just another estimate that will be hard to audit later.

Resolve the model deliberately

A model ID can fail in two different ways:

it is absent from the pricing snapshot
it is absent from the live route table

Those are not the same failure. The first means the team cannot price the planned route from the dated source. The second means the route may not be callable in the current gateway table.

def find_pricing_row(pricing: dict, model_id: str) -> dict | None:
    return next((row for row in pricing["models"] if row["id"] == model_id), None)


def live_route_names(routes: dict) -> set[str]:
    names: set[str] = set()
    for row in routes.get("data", []):
        for key in ("id", "model", "model_name"):
            value = row.get(key)
            if isinstance(value, str):
                names.add(value)
    return names


requested_model = "deepseek-v4-flash"
pricing_row = find_pricing_row(pricing, requested_model)
route_exists = requested_model in live_route_names(routes)

if pricing_row is None:
    raise SystemExit("missing_pricing_row")
if not route_exists:
    raise SystemExit("missing_live_route")

In production, you may allow aliases, but they should resolve to a named canonical model before review. Otherwise a later incident will spend time asking which route actually shipped.

Simulate cost without pretending it is billing

The simulator should call its output an estimate or review cost, not an invoice.

def usd_per_million(tokens: int, rate: float) -> float:
    return tokens / 1_000_000 * rate


def estimate_run_cost(shape: dict, row: dict, group_ratio: float) -> float:
    requests = shape["requests"]
    input_tokens = requests * shape["input_tokens_per_request"]
    cached_tokens = requests * shape.get("expected_cached_input_tokens_per_request", 0)
    output_tokens = requests * shape["max_output_tokens_per_request"]

    fresh_tokens = max(input_tokens - cached_tokens, 0)
    input_rate = row["input_usd_per_1m_tokens"]
    cached_rate = row.get("cache_hit_input_usd_per_1m_tokens") or input_rate
    output_rate = row["output_usd_per_1m_tokens"]

    subtotal = (
        usd_per_million(fresh_tokens, input_rate)
        + usd_per_million(cached_tokens, cached_rate)
        + usd_per_million(output_tokens, output_rate)
    )
    return round(subtotal * group_ratio, 6)

The group ratio should come from the live route source or another controlled configuration, not from a comment in the release ticket.

group = "default"
group_ratio = routes["group_ratio"][group]
review_cost = estimate_run_cost(dry_run["traffic_shape"], pricing_row, group_ratio)

That number is useful because it is reproducible. It is still not the final bill. Real usage can change through output length, retries, cache status, tool calls, failed requests, upstream changes, and account configuration.

Add stress bands

A single estimate is too neat. Add small stress bands that represent common production movement.

def clone_shape(shape: dict, **updates: int) -> dict:
    next_shape = dict(shape)
    next_shape.update(updates)
    return next_shape


base = dry_run["traffic_shape"]
scenarios = {
    "planned": base,
    "long_outputs": clone_shape(base, max_output_tokens_per_request=1800),
    "one_retry": clone_shape(
        base,
        requests=base["requests"] * (1 + base["max_retry_attempts_per_request"]),
    ),
    "cache_observed": clone_shape(
        base,
        expected_cached_input_tokens_per_request=60000,
    ),
}

for name, scenario in scenarios.items():
    print(name, estimate_run_cost(scenario, pricing_row, group_ratio))

The point is not to scare people with a dramatic range. The point is to expose which assumption matters.

If long_outputs moves the review cost more than any other scenario, the release should focus on response length controls. If one_retry dominates, the team should inspect timeout and idempotency behavior. If cache_observed changes the result but the route cannot produce cache receipt fields, the release should not depend on cache savings yet.

Fail closed on missing receipt fields

The dry run should also check that the planned receipt contract is strong enough for later analysis.

REQUIRED_FIELDS = {
    "request_id",
    "model_id",
    "prompt_tokens",
    "completion_tokens",
    "retry_attempts",
    "group",
    "charged_amount",
}


def validate_receipt_contract(fields: list[str]) -> list[str]:
    present = set(fields)
    return sorted(REQUIRED_FIELDS - present)


missing = validate_receipt_contract(dry_run["receipt_contract"])
if missing:
    raise SystemExit({"missing_receipt_fields": missing})

Notice that cached_input_tokens is not in the minimum set above. Some routes may not support cache evidence. That is fine as long as the simulator records the unknown and the forecast does not rely on cache behavior.

For a cache-dependent release, promote that field to required:

if dry_run["traffic_shape"]["expected_cached_input_tokens_per_request"] > 0:
    REQUIRED_FIELDS.add("cached_input_tokens")

The simulator should preserve unknowns instead of filling them with guesses. Unknown is a valid release result. It is much better than a confident spreadsheet that cannot be reconciled later.

Produce a review result

The result should be small enough for a pull request, release note, or procurement answer.

{
  "dry_run_id": "support-triage-route-2026-09-22",
  "verdict": "review_required",
  "requested_model": "deepseek-v4-flash",
  "pricing_source": {
    "checked": "2026-09-10",
    "updated_at": "2026-09-18",
    "pricing_version": "83f77abde81ee3a096a672ed959ccc096f5d37a45c177ae8e03229456b5415a5"
  },
  "route_source": {
    "pricing_version": "a42d372ccf0b5dd13ecf71203521f9d2",
    "group_ratio": {"default": 1, "vip": 0.9}
  },
  "checks": {
    "pricing_row": "present",
    "live_route": "present",
    "receipt_contract": "present",
    "cache_evidence": "not_assumed"
  },
  "review_notes": [
    "output cap is the largest stress driver",
    "final billing depends on actual receipts and selected group"
  ]
}

That object gives each owner a clear next step. Platform engineering can review route resolution. Product engineering can review output caps. Finance can inspect the source versions. Security can check that private content stayed out of the artifact.

Keep public language narrower than internal proof

A dry-run simulator is good internal evidence. It is not a license to make broad claims.

Safe public copy says:

The route was dry-run against a dated pricing snapshot and live route table before release.

Risky public copy says:

The route will always cost less in production.

The first statement is about process and evidence. The second predicts a workload outcome without knowing the future request mix, cache behavior, retries, output length, group selection, or source changes.

For AI API gateways, the practical standard should be boring and strict:

Route IDs resolve before the release.
Pricing versions are recorded before the release.
Group rules are explicit before the release.
Stress bands are visible before the release.
Receipt fields are agreed before the release.
Private prompts and reusable credentials stay out of the artifact.

If the team can do that, the first production request is no longer the first time the route's cost and evidence contract has been tested.

Build Request-Level Cost Ledgers for AI API Routes

Mattias chaw — Mon, 21 Sep 2026 13:12:13 +0000

When an AI application uses one OpenAI-compatible client across several Chinese model routes, the hard part is rarely the first request. The hard part is explaining the bill after a rollout, especially when a route has cache-hit input, uncached input, output tokens, retries, tools, and dated pricing evidence.

That explanation should not live in a spreadsheet that someone updates when finance asks a question. It should be written at request time, next to the route decision that created the spend.

This article walks through a compact request-level cost ledger. It is meant for teams evaluating routes such as DeepSeek, GLM, Qwen, Kimi, MiniMax, Doubao, StepFun, and other Chinese model families through a single OpenAI-compatible API. The examples use public pricing evidence checked on September 21, 2026 against AIWave's live pricing endpoint, but the pattern works for any gateway or direct provider setup where prices and model catalogs can change over time.

The ledger row

A useful ledger row answers five questions:

Which model route handled the request?
What pricing version was used when the request ran?
How many input, cache-hit input, output, and tool-related tokens were billed?
Which retry or fallback path changed the final cost?
Which release or canary decision allowed the request to reach that route?

That leads to a schema like this:

type RouteCostLedgerRow = {
  request_id: string
  workflow_id: string
  route_id: string
  model: string
  provider_family: string
  pricing_checked_at: string
  pricing_source: string
  input_tokens: number
  cache_hit_input_tokens: number
  output_tokens: number
  tool_tokens: number
  retry_count: number
  fallback_from?: string
  estimated_usd: number
  final_usd?: number
  release_id: string
  policy_id: string
  decision_reason: string
}

There are two important details in that row.

First, pricing_checked_at and pricing_source are not decoration. They make the row auditable. If the route price changes next week, yesterday's decision should still point to the price evidence that existed when the request ran.

Second, estimated_usd and final_usd are separate. The estimate is what your router believed before or during dispatch. The final value is what your billing system confirms after the request settles. If they drift, that difference becomes an engineering signal rather than a support surprise.

Capture route evidence before the call

The route decision should be logged before the upstream call leaves your system. That record does not need to contain the prompt or response body. In fact, it should avoid them. A routing ledger is about operational evidence, not content retention.

function startLedgerRow(input: {
  requestId: string
  workflowId: string
  model: string
  providerFamily: string
  pricingVersion: string
  releaseId: string
  policyId: string
  reason: string
}): RouteCostLedgerRow {
  return {
    request_id: input.requestId,
    workflow_id: input.workflowId,
    route_id: `${input.providerFamily}:${input.model}`,
    model: input.model,
    provider_family: input.providerFamily,
    pricing_checked_at: input.pricingVersion,
    pricing_source: "https://aiwave.live/api/pricing",
    input_tokens: 0,
    cache_hit_input_tokens: 0,
    output_tokens: 0,
    tool_tokens: 0,
    retry_count: 0,
    estimated_usd: 0,
    release_id: input.releaseId,
    policy_id: input.policyId,
    decision_reason: input.reason,
  }
}

The decision_reason should be boring and specific:

default-route-for-chat
long-context-route
cache-sensitive-agent-run
canary-route-5-percent
fallback-after-timeout

Avoid vague labels such as smart-router or optimized. They sound nice in a deck, but they do not help an engineer explain why a request moved.

Estimate with dated prices

At run time, your router can hold a pricing snapshot in memory. The snapshot should be refreshed on a schedule and versioned by retrieval time. A practical estimator only needs a few fields:

type RoutePrice = {
  model: string
  inputUsdPerMillion: number
  cacheHitUsdPerMillion?: number
  outputUsdPerMillion: number
  pricingCheckedAt: string
}

function estimateCostUsd(row: RouteCostLedgerRow, price: RoutePrice) {
  const input = (row.input_tokens / 1_000_000) * price.inputUsdPerMillion
  const cacheHit =
    ((row.cache_hit_input_tokens || 0) / 1_000_000) *
    (price.cacheHitUsdPerMillion ?? price.inputUsdPerMillion)
  const output = (row.output_tokens / 1_000_000) * price.outputUsdPerMillion
  return Number((input + cacheHit + output).toFixed(8))
}

Notice the cache behavior. If a route publishes a cache-hit line, use it explicitly. If it does not, fall back to the normal input rate in the estimate and mark cache-specific savings as unknown. That is more honest than inventing a cache discount from the model name.

For AIWave, the live pricing endpoint checked during this run exposed current route rows for examples including deepseek-v4-pro, deepseek-v4-flash, glm-5, qwen3-max, MiniMax-M3, and doubao-seed-2-1-pro-260628. Treat that as a dated route-card observation, not a permanent catalog promise.

Separate retries from fallbacks

Retries and fallbacks have different meanings.

A retry says: the same route was attempted again because the previous attempt failed or timed out.

A fallback says: the router changed route after a condition was met.

Your ledger should keep those apart because they create different actions. Too many retries might point to timeout settings, regional connectivity, or request size. Too many fallbacks might point to an unhealthy primary route or a policy that is too aggressive.

function recordRetry(row: RouteCostLedgerRow) {
  row.retry_count += 1
}

function recordFallback(row: RouteCostLedgerRow, previousRoute: string) {
  row.fallback_from = previousRoute
  row.decision_reason = "fallback-after-timeout"
}

The distinction matters for cost review. If a request paid twice because of a retry, the fix may be idempotency and timeout tuning. If it paid more because of a fallback, the fix may be a narrower policy or a clearer route budget.

Put gates in front of route changes

Most teams review model quality before a route change. Fewer teams review cost variance with the same discipline.

Before moving a production workload to a different route, run a gate like this:

Replay a recent sample without storing request bodies in the ledger.
Compare estimated cost per workflow, not only per request.
Break out cache-hit input, uncached input, output, tool tokens, retries, and fallbacks.
Require a signed pricing snapshot date.
Ship the change as a canary with an automatic stop condition.

The stop condition should be written in operational language:

type CostGate = {
  releaseId: string
  maxWorkflowCostIncreasePct: number
  maxRetryRatePct: number
  maxFallbackRatePct: number
  sampleWindowMinutes: number
}

function shouldStopCanary(metrics: {
  workflowCostIncreasePct: number
  retryRatePct: number
  fallbackRatePct: number
}, gate: CostGate) {
  return (
    metrics.workflowCostIncreasePct > gate.maxWorkflowCostIncreasePct ||
    metrics.retryRatePct > gate.maxRetryRatePct ||
    metrics.fallbackRatePct > gate.maxFallbackRatePct
  )
}

This avoids a common failure mode: a new route looks fine in average request cost, but the whole workflow becomes more expensive because it produces longer outputs, triggers more tools, or causes a second pass in the agent loop.

Keep the ledger private, publish the method

A cost ledger will contain sensitive operational metadata. It may reveal release IDs, internal workflow names, route policy names, or workload shape. Keep the raw rows private.

What you can publish safely is the method:

pricing evidence is dated
request bodies are not stored in the ledger
estimates and settled values are separated
retries and fallbacks are separated
canaries include cost variance stop conditions

That gives developers and procurement reviewers a way to ask precise questions without exposing private traffic.

Reconcile by workflow, not by invoice line

The most useful review happens one level above a single request. A coding agent, RAG pipeline, data extraction job, or customer support assistant usually makes several calls to finish one unit of work. If you only inspect individual requests, you can miss the cost shape that actually matters.

Create a daily reconciliation job that groups ledger rows by workflow_id, release_id, and policy_id:

type WorkflowCostSummary = {
  workflow_id: string
  release_id: string
  policy_id: string
  request_count: number
  retry_count: number
  fallback_count: number
  estimated_usd: number
  final_usd: number
  variance_usd: number
}

The job should produce three views.

First, show release variance. If release-2026-09-21-a changes the median workflow cost, the owning team should see it before the next rollout expands.

Second, show route variance. A route can be technically healthy while still producing a longer answer shape, more tool calls, or more retries than the route it replaced.

Third, show policy variance. If a fallback policy saves latency but creates a higher settled cost for a workflow, that is a real trade-off. It may still be the right trade-off. The point is to make it visible.

This is also where you compare estimates with settled values:

function varianceUsd(summary: {
  estimated_usd: number
  final_usd: number
}) {
  return Number((summary.final_usd - summary.estimated_usd).toFixed(8))
}

If variance is consistently positive, do not patch it by changing a dashboard label. Check whether your pricing snapshot is stale, cache-hit accounting is missing, retries are being estimated once but billed multiple times, or output token counts are arriving after the estimate is written.

Decide what never enters the ledger

The ledger should be intentionally incomplete. Do not store prompts, responses, raw user identifiers, payment details, or reusable API keys in a cost ledger. You can usually answer the operational question with route ID, token counts, pricing version, release ID, and workflow ID.

For debugging, store a request hash or a short-lived trace pointer instead of the body. Keep the body in the system that already owns request observability and retention controls. The cost ledger should remain narrow enough that engineers, finance, and procurement can inspect it without turning it into a sensitive content database.

A simple rollout checklist

Before you send a multi-route AI workload into production, ask:

Can every request point to the route and pricing snapshot used at dispatch time?
Can you explain cache-hit input separately from uncached input?
Can you identify which release introduced a cost change?
Can you tell whether retries or fallbacks created the variance?
Can you stop a canary based on workflow-level cost, not only error rate?

If the answer is no, do not start with a new pricing argument. Start with evidence capture.

Pricing will keep changing. Model catalogs will keep changing. Provider capabilities will keep changing. A request-level ledger will not make those changes disappear, but it gives your team a stable way to reason about them.

That is the real operational win: every route decision becomes explainable after the fact.

Build Model Catalog Diff Tests for OpenAI-Compatible API Routes

Mattias chaw — Sun, 20 Sep 2026 13:08:12 +0000

Build Model Catalog Diff Tests for OpenAI-Compatible API Routes

Most OpenAI-compatible integrations start with a model ID pasted into a quickstart.

That is fine for the first request. It is fragile for the third team that copies the same example into a worker, a notebook, a support bot, and a pricing workbook. After a few releases, the question is no longer "does the API accept this string?" The question becomes:

Does every place that names a model still mean the same thing?

A model catalog diff test answers that question before the mismatch reaches production. It compares the catalog your docs and SDK examples expect with the live route table your gateway can serve. It also checks the pricing snapshot that finance and product teams use for planning.

This is not a benchmark. It does not claim that one model is universally better. It is a contract test for catalog shape, route availability, endpoint support, and dated pricing references.

I will use AIWave as the concrete example because its public pricing snapshot and live route table can be checked without a private account. As of September 20, 2026, https://aiwave.live/api/v1/pricing returned 56 model rows, currency USD, unit per_1m_text_tokens, checked=2026-09-10, updated_at=2026-09-18, and pricing version 83f77abde81ee3a096a672ed959ccc096f5d37a45c177ae8e03229456b5415a5. The live route table at https://aiwave.live/api/pricing returned success=true, 68 route rows, pricing version a42d372ccf0b5dd13ecf71203521f9d2, and group ratios default=1 and vip=0.9.

Those values are intentionally dated. Catalog tests should make source freshness visible instead of turning today's result into a timeless claim.

Why catalog drift hurts

Catalog drift is subtle because each individual system can still look healthy.

Your docs page may list a model ID. Your SDK example may use a similar alias. Your pricing workbook may have a dated row. Your gateway may serve a newer route table. Your product policy may allow only a subset of routes for a workload.

No single mismatch is dramatic. Together they create support tickets that are hard to explain:

Drift surface	Example failure
Docs versus route table	The quickstart names a model that no longer resolves
Pricing snapshot versus live route	The workbook plans against a row that is missing from runtime
SDK fixture versus endpoint support	The example calls an endpoint the route does not support
Product policy versus catalog	A route exists but should not be used for that workload
Group multiplier versus estimate	A key is billed under a different group assumption

The goal of a diff test is not to freeze the catalog. Catalogs move. The goal is to make movement visible, reviewable, and safe to ship.

Define the catalog contract

Start with the smallest contract your team can defend. A useful model row needs more than a name.

{
  "model_id": "deepseek-v4-pro",
  "provider": "DeepSeek",
  "source": "public_pricing_snapshot",
  "pricing_version": "83f77abde81ee3a096a672ed959ccc096f5d37a45c177ae8e03229456b5415a5",
  "effective_date": "2026-08-27",
  "unit": "per_1m_text_tokens",
  "required_endpoints": ["openai"],
  "allowed_groups": ["default", "vip"],
  "status": "expected"
}

That object is intentionally boring. It does not include prompts, customer data, or reusable credentials. It only says what the release believes to be true about a route.

For a production team, I would store this file near SDK examples and documentation fixtures. When a quickstart changes its default model, the contract changes in the same pull request. When a pricing snapshot refreshes, the version and effective date change in a visible review.

Pull the two public sources

The test needs two inputs:

a source-dated pricing snapshot, used for planning and docs
a live route table, used for runtime availability and endpoint support

Here is a small Python fetcher. It uses public endpoints and no API key.

from __future__ import annotations

import json
from dataclasses import dataclass
from urllib.request import urlopen


PRICING_SNAPSHOT_URL = "https://aiwave.live/api/v1/pricing"
LIVE_ROUTE_URL = "https://aiwave.live/api/pricing"


def fetch_json(url: str) -> dict:
    with urlopen(url, timeout=30) as response:
        return json.loads(response.read().decode("utf-8"))


@dataclass(frozen=True)
class Sources:
    pricing_snapshot: dict
    live_route_table: dict


def load_sources() -> Sources:
    return Sources(
        pricing_snapshot=fetch_json(PRICING_SNAPSHOT_URL),
        live_route_table=fetch_json(LIVE_ROUTE_URL),
    )

If your internal gateway requires authentication, still keep this pattern: read the API key from an environment variable and never put it in the fixture.

import os

api_key = os.environ.get("AIWAVE_API_KEY")
if not api_key:
    raise RuntimeError("AIWAVE_API_KEY is required for private probes")

The public catalog diff itself does not need that key. It should remain safe to run in CI and to attach to an engineering review.

Normalize before comparing

Do not diff raw JSON. Normalize the rows into the fields your release actually depends on.

def normalize_snapshot(snapshot: dict) -> dict[str, dict]:
    rows = snapshot.get("models", [])
    return {
        row["id"]: {
            "provider": row.get("provider"),
            "input_usd_per_1m_tokens": row.get("input_usd_per_1m_tokens"),
            "cache_hit_usd_per_1m_tokens": row.get("cache_hit_usd_per_1m_tokens"),
            "output_usd_per_1m_tokens": row.get("output_usd_per_1m_tokens"),
            "effective_date": row.get("effective_date"),
        }
        for row in rows
        if row.get("id")
    }


def normalize_live_routes(route_table: dict) -> dict[str, dict]:
    rows = route_table.get("data", [])
    return {
        row["model_name"]: {
            "enable_groups": sorted(row.get("enable_groups", [])),
            "supported_endpoint_types": sorted(row.get("supported_endpoint_types", [])),
            "model_ratio": row.get("model_ratio"),
            "completion_ratio": row.get("completion_ratio"),
            "cache_ratio": row.get("cache_ratio"),
        }
        for row in rows
        if row.get("model_name")
    }

This keeps the test focused. You do not want CI failing because the provider added an unrelated metadata field. You do want CI failing if the route your SDK example uses disappears or stops supporting the expected endpoint type.

Classify differences

Every difference should land in a clear bucket.

def diff_catalog(expected: dict[str, dict], live: dict[str, dict]) -> dict:
    expected_ids = set(expected)
    live_ids = set(live)

    missing_at_runtime = sorted(expected_ids - live_ids)
    new_at_runtime = sorted(live_ids - expected_ids)

    endpoint_mismatch = []
    group_mismatch = []

    for model_id in sorted(expected_ids & live_ids):
        contract = expected[model_id]
        route = live[model_id]

        required_endpoints = set(contract.get("required_endpoints", []))
        live_endpoints = set(route.get("supported_endpoint_types", []))
        if not required_endpoints.issubset(live_endpoints):
            endpoint_mismatch.append(model_id)

        allowed_groups = set(contract.get("allowed_groups", []))
        live_groups = set(route.get("enable_groups", []))
        if allowed_groups and not allowed_groups.issubset(live_groups):
            group_mismatch.append(model_id)

    return {
        "missing_at_runtime": missing_at_runtime,
        "new_at_runtime": new_at_runtime,
        "endpoint_mismatch": endpoint_mismatch,
        "group_mismatch": group_mismatch,
    }

This output is more useful than a boolean pass or fail. A missing runtime route is usually a blocker for examples and product policies. A new runtime route may be a review item. An endpoint mismatch is often a docs or SDK issue. A group mismatch belongs with account and key-management owners.

Decide what blocks a release

Not every diff should block the same way.

BLOCKING_FIELDS = {
    "missing_at_runtime",
    "endpoint_mismatch",
    "group_mismatch",
}


def release_verdict(diff: dict) -> str:
    blockers = {
        field: diff[field]
        for field in BLOCKING_FIELDS
        if diff.get(field)
    }
    if blockers:
        return "block_release"
    if diff.get("new_at_runtime"):
        return "review_new_routes"
    return "pass"

The rule is conservative on purpose. If a model appears in a quickstart, workbook, or route policy and the live route table cannot support the required shape, the release should stop. If the live table adds a model that the release does not use yet, that can be a review note instead of a deployment blocker.

Add freshness checks

Catalog diff tests should also warn when the source is stale or ambiguous.

def validate_source_metadata(snapshot: dict, route_table: dict) -> list[str]:
    warnings = []

    if snapshot.get("currency") != "USD":
        warnings.append("pricing_snapshot_currency_changed")
    if snapshot.get("unit") != "per_1m_text_tokens":
        warnings.append("pricing_snapshot_unit_changed")
    if not snapshot.get("pricing_version"):
        warnings.append("pricing_snapshot_missing_version")
    if route_table.get("success") is not True:
        warnings.append("live_route_table_not_successful")
    if not route_table.get("pricing_version"):
        warnings.append("live_route_table_missing_version")

    group_ratio = route_table.get("group_ratio", {})
    if group_ratio.get("default") != 1:
        warnings.append("default_group_ratio_changed")
    if group_ratio.get("vip") != 0.9:
        warnings.append("vip_group_ratio_changed")

    return warnings

On September 20, 2026, the public data I checked had default=1 and vip=0.9. That is a current-source observation, not a promise about future values. If those values change, the test should force a review of docs, pricing copy, and budget examples.

Keep runtime probes separate

A catalog diff test can tell you that a route is listed and shaped correctly. It cannot prove a full authenticated request path, quota state, streaming behavior, or latency.

Do not overload one test. Use separate probes for:

unauthenticated public catalog availability
authenticated smoke calls in a controlled account
request-level billing receipt checks
streaming behavior
timeout and retry behavior
provider incident response

That separation matters for trust. A public route table may be accurate while an account-level key is misconfigured. A smoke call may pass while pricing copy is stale. A model may be available while a specific endpoint mode is not.

Good engineering evidence keeps those claims apart.

What to publish in a release note

The internal diff can be detailed. The public note should be narrower.

Safe language:

The SDK examples were checked against a dated pricing snapshot and the current route table before release.

Risky language:

Every model is always available and priced the same way.

The first sentence describes a process and a source. The second sentence creates a support burden the catalog cannot honestly carry.

For Tier 1 and Tier 2 teams, this distinction is not cosmetic. It is how you keep docs, budget reviews, and production behavior aligned without exposing internal usage, customer data, or credentials.

A practical checklist

Before shipping an OpenAI-compatible API example or route policy, ask:

Does every documented model ID exist in the live route table?
Does the route support the endpoint shape the example uses?
Does the pricing snapshot have a version, source URL, unit, and effective date?
Are group multipliers explicit instead of implied?
Are newly available routes reviewed before they appear in examples?
Are removed or renamed routes handled as release blockers?
Are authenticated smoke tests separate from public catalog checks?

If the answer is yes, the model catalog becomes part of the release contract instead of a loose list of strings.

That is the point of the diff test: not to slow teams down, but to make sure the model ID in the code, the route in production, and the price row in the workbook are still talking about the same thing.

Build a Budget Variance Ledger for AI API Gateways

Mattias chaw — Sat, 19 Sep 2026 13:10:07 +0000

Build a Budget Variance Ledger for AI API Gateways

Most AI API budget reviews start too late.

The invoice arrives. Someone exports usage. Someone else finds the rate card that was current when the feature launched. A third person asks whether cache hits were counted, whether retries were included, and whether the model ID in the product code is the same one shown in the planning workbook.

By then the team is no longer debugging a single request. It is reconstructing a chain of assumptions.

A budget variance ledger gives that chain a compact shape. It compares what the release expected to happen with what the gateway receipts say actually happened, using the same dated pricing source and route version that the team approved before shipping.

This is not a replacement for billing. It is a review artifact for Tier 1 and Tier 2 engineering teams that need to explain model spend without exposing prompts, customer data, or reusable credentials.

I will use AIWave as the concrete example because its public pricing snapshot and live route table can be checked without a private account. As of September 19, 2026, https://aiwave.live/api/v1/pricing returned 56 model rows, currency USD, unit per_1m_text_tokens, checked=2026-09-10, updated_at=2026-09-18, and pricing version 83f77abde81ee3a096a672ed959ccc096f5d37a45c177ae8e03229456b5415a5. The live route table at https://aiwave.live/api/pricing returned 68 route rows, success=true, pricing version a42d372ccf0b5dd13ecf71203521f9d2, and group ratios default=1 and vip=0.9.

Those values are deliberately dated. A variance ledger should never pretend that a price row is timeless.

The problem: expected spend is not actual spend

Planning math usually begins with a friendly formula:

estimated_cost =
  input_tokens * input_rate
+ cached_input_tokens * cache_hit_rate
+ output_tokens * output_rate

That formula is useful, but it hides the parts that change during production:

Source of variance	What usually went missing
Model route changed	The release used a different model ID than the workbook
Output grew	The product allowed longer answers than the estimate assumed
Retries multiplied	Timeout or 429 handling repeated the same user action
Cache did not hold	The forecast used a cache-hit row without receipt proof
Group multiplier differed	The API key or account group changed the applied rate
Pricing version moved	The checked source was refreshed after release approval

The ledger does not need to accuse anyone. It just needs to make these differences visible.

A minimal ledger row

A row should compare one workload slice, not the whole business. Start with a feature, customer tier, route policy, or scheduled job.

{
  "ledger_id": "support-summary-2026-09-19",
  "checked_at": "2026-09-19T13:20:00Z",
  "feature": "support_summary",
  "client_shape": "openai_chat_completions",
  "model_id": "deepseek-flash",
  "pricing_snapshot": {
    "url": "https://aiwave.live/api/v1/pricing",
    "checked": "2026-09-10",
    "updated_at": "2026-09-18",
    "pricing_version": "83f77abde81ee3a096a672ed959ccc096f5d37a45c177ae8e03229456b5415a5"
  },
  "route_table": {
    "url": "https://aiwave.live/api/pricing",
    "pricing_version": "a42d372ccf0b5dd13ecf71203521f9d2",
    "group_ratio": {
      "default": 1,
      "vip": 0.9
    }
  },
  "expected": {
    "requests": 1000,
    "input_tokens": 180000000,
    "cached_input_tokens": 0,
    "output_tokens": 22000000,
    "max_retries": 1
  },
  "actual": {
    "requests": 1000,
    "input_tokens": 176400000,
    "cached_input_tokens": 24800000,
    "output_tokens": 31800000,
    "retry_attempts": 86
  },
  "variance_review": {
    "output_tokens": "above_plan",
    "cache_evidence": "observed_in_receipts",
    "retry_policy": "within_limit",
    "verdict": "review_output_cap_before_next_release"
  }
}

There is no prompt text in that object. There is no API key. There is no user identifier. The point is to let engineering, finance, and procurement inspect the spend mechanics without turning the review file into a private data dump.

Calculate variance in code

The ledger can stay plain JSON, while a small check computes the variance bands.

from __future__ import annotations

from dataclasses import dataclass


@dataclass(frozen=True)
class Rate:
    input: float
    cached_input: float | None
    output: float


def usd_per_million(tokens: int, rate: float) -> float:
    return tokens / 1_000_000 * rate


def estimate(row: dict, rate: Rate, group_ratio: float = 1.0) -> float:
    expected = row["expected"]
    cached_tokens = expected.get("cached_input_tokens", 0)
    fresh_tokens = max(expected["input_tokens"] - cached_tokens, 0)
    cached_rate = rate.cached_input if rate.cached_input is not None else rate.input

    subtotal = (
        usd_per_million(fresh_tokens, rate.input)
        + usd_per_million(cached_tokens, cached_rate)
        + usd_per_million(expected["output_tokens"], rate.output)
    )
    return subtotal * group_ratio


def actual_cost(row: dict, rate: Rate, group_ratio: float = 1.0) -> float:
    actual = row["actual"]
    cached_tokens = actual.get("cached_input_tokens", 0)
    fresh_tokens = max(actual["input_tokens"] - cached_tokens, 0)
    cached_rate = rate.cached_input if rate.cached_input is not None else rate.input

    subtotal = (
        usd_per_million(fresh_tokens, rate.input)
        + usd_per_million(cached_tokens, cached_rate)
        + usd_per_million(actual["output_tokens"], rate.output)
    )
    return subtotal * group_ratio

For a production gate, do not hard-code the rate values. Pull them from the dated snapshot, match the exact model ID, and fail closed if the pricing version or model row changed.

def classify_variance(planned: float, observed: float) -> str:
    if planned <= 0:
        return "missing_plan"
    ratio = (observed - planned) / planned
    if ratio > 0.20:
        return "review_required"
    if ratio > 0.05:
        return "watch"
    if ratio < -0.20:
        return "under_plan_explain"
    return "within_band"

The exact thresholds are a business decision. The important part is that they are explicit before the run, not invented after the invoice.

Split variance by cause

A single percentage is not enough. A useful ledger names the likely cause.

{
  "variance_causes": [
    {
      "field": "output_tokens",
      "expected": 22000000,
      "actual": 31800000,
      "action": "tighten_answer_length_or_streaming_stop_rule"
    },
    {
      "field": "retry_attempts",
      "expected_max": 100,
      "actual": 86,
      "action": "no_change"
    },
    {
      "field": "pricing_version",
      "expected": "83f77abde81ee3a096a672ed959ccc096f5d37a45c177ae8e03229456b5415a5",
      "actual": "83f77abde81ee3a096a672ed959ccc096f5d37a45c177ae8e03229456b5415a5",
      "action": "no_change"
    }
  ]
}

That structure makes review calmer. If output growth caused the variance, the answer is not to debate the entire provider strategy. It is to review the product behavior that let output grow.

If pricing version drift caused the variance, the answer is not to blame retries. It is to update the source snapshot and re-approve the route.

Treat cache as a receipt field

Cache-hit pricing is easy to misuse because it looks precise in a table. A variance ledger should separate three ideas:

the listed cache-hit rate
the forecast assumption
the runtime receipt evidence

If your gateway cannot prove cache status per request, the ledger should not classify reduced spend as cache success. It should say the cause is unknown.

That may feel strict, but it protects the next release. Teams should not scale a workload because a spreadsheet predicted cache behavior. They should scale it because receipts show the behavior actually happened.

Add a replay fixture

The most useful variance reviews are reproducible. Keep a tiny replay fixture next to the ledger so the team can rerun the same calculation when the route, snapshot, or output cap changes.

The fixture should be synthetic. Do not replay customer prompts. Instead, store a narrow set of redacted receipt shapes:

[
  {
    "case": "normal_path",
    "model_id": "deepseek-flash",
    "prompt_tokens": 180000,
    "completion_tokens": 22000,
    "cached_input_tokens": 0,
    "retry_attempts": 0,
    "group": "default"
  },
  {
    "case": "long_answer_path",
    "model_id": "deepseek-flash",
    "prompt_tokens": 176000,
    "completion_tokens": 42000,
    "cached_input_tokens": 18000,
    "retry_attempts": 1,
    "group": "default"
  }
]

Run the fixture against the same pricing snapshot that the ledger names. If the code can only pass with today's live table, the review is not reproducible. If it can pass with the pinned snapshot and also show the diff against the live table, the reviewer can separate two questions:

Did the release behave differently from the approved plan?
Did the source table move after the approval date?

That distinction matters. A product bug and a pricing-source update need different owners.

Route alerts to the right owner

Once the ledger classifies variance, route the alert by cause.

Cause	First owner
Output tokens above plan	Product or prompt owner
Retry attempts above plan	Platform reliability owner
Model ID mismatch	Release engineering owner
Cache evidence missing	Gateway instrumentation owner
Pricing version drift	Pricing-source owner
Group multiplier mismatch	Account or key-management owner

This keeps the review from becoming a vague "AI costs are high" meeting. The right owner gets a concrete field, a dated source, and a next action.

For example, output growth might lead to a stricter response contract, streaming stop rule, or per-feature max token cap. Retry growth might lead to better timeout budgets or idempotency checks. Pricing drift might simply require re-approving the route with a new source date.

The ledger should make those paths visible without claiming that any one route is universally better.

Keep the ledger narrow enough to publish internally

A good variance ledger is boring in the best way:

It names the feature or route.
It names the pricing snapshot and route version.
It stores expected and actual token counts.
It classifies variance using pre-agreed bands.
It preserves unknowns instead of filling them with guesses.
It avoids prompts, responses, customer details, and reusable credentials.

That makes it safe to attach to a release review, an incident recap, or a procurement answer.

Public copy should stay narrower than the ledger. Safe public language might say:

The route was reviewed against a dated pricing snapshot and request-level receipts.

Risky language would claim a universal cost outcome. The ledger can support operational confidence, but it cannot predict every workload shape, retry pattern, cache behavior, account group, or future price row.

The practical question for a gateway team is simple:

When spend changes, can we explain whether the cause was model choice, output length, retry behavior, cache evidence, group multiplier, or pricing version drift?

If the answer is yes, the budget review stops being archaeology.

Build Versioned Cost Manifests for AI API Releases

Mattias chaw — Fri, 18 Sep 2026 13:07:55 +0000

Build Versioned Cost Manifests for AI API Releases

AI API cost reviews often fail in a quiet way.

The application team can point to the model ID. The platform team can point to the gateway route. Finance can point to a workbook. Security can point to a redaction rule. Procurement can point to a dated rate card. None of those artifacts are wrong, but they are often disconnected.

That disconnect matters when a Tier 1 or Tier 2 team moves from a prototype into a release train. The route that passed a demo last week may still answer requests today, while the pricing snapshot, cache assumption, route group, or receipt contract has changed.

A versioned cost manifest gives the release one small object to review.

It is not a billing system. It is not a marketing page. It is not a private log dump. It is a release artifact that says:

This AI API route was reviewed against these dated sources, with these assumptions, and these receipt fields must exist after the workload runs.

I will use AIWave as the concrete example because it exposes a public pricing snapshot at https://aiwave.live/api/v1/pricing and a live route table at https://aiwave.live/api/pricing. The pattern works for any OpenAI-compatible gateway or internal model platform.

As of September 18, 2026, the public AIWave pricing snapshot returned 56 model rows, checked=2026-09-10, updated_at=2026-09-18, currency USD, unit per_1m_text_tokens, and pricing version 83f77abde81ee3a096a672ed959ccc096f5d37a45c177ae8e03229456b5415a5. The live route table returned 68 route rows, success=true, pricing version a42d372ccf0b5dd13ecf71203521f9d2, and group ratios default=1 and vip=0.9.

Those facts are not evergreen copy. Put the dates and versions into the manifest.

Why a manifest beats a spreadsheet note

Spreadsheets are useful for exploration. They are weak as release evidence.

A workbook can estimate expected spend, but it often loses the exact source version that fed the calculation. A release note can mention the model ID, but not the cache row or route group. A support ticket can hold a request receipt, but not the policy that made the route acceptable.

A manifest ties those pieces together without exposing private data.

At minimum, it should record:

Field	Why it exists
`manifest_id`	Stable handle for the release review
`checked_at`	When the evidence was collected
`model_id`	Exact ID used by the client or route
`client_shape`	API family, such as OpenAI-compatible chat completions
`pricing_snapshot`	URL, checked date, updated date, and version
`route_table`	URL, route count, group ratios, and version
`rate_basis`	Input, cache-hit input, and output rates used for planning
`cache_assumption`	Whether the release assumes cache hit behavior
`output_cap`	Maximum planned output for this path
`retry_policy`	Retry count and timeout budget
`receipt_contract`	Redacted fields expected after each run
`unknowns`	Missing evidence that reviewers must not infer

The unknowns field is not optional. It is where the manifest prevents optimism from becoming architecture.

A small manifest schema

Start with a boring JSON object.

{
  "manifest_id": "support-agent-route-2026-09-18",
  "checked_at": "2026-09-18T13:20:00Z",
  "client_shape": "openai_chat_completions",
  "model_id": "deepseek-flash",
  "pricing_snapshot": {
    "url": "https://aiwave.live/api/v1/pricing",
    "checked": "2026-09-10",
    "updated_at": "2026-09-18",
    "pricing_version": "83f77abde81ee3a096a672ed959ccc096f5d37a45c177ae8e03229456b5415a5"
  },
  "route_table": {
    "url": "https://aiwave.live/api/pricing",
    "pricing_version": "a42d372ccf0b5dd13ecf71203521f9d2",
    "group_ratio": {
      "default": 1,
      "vip": 0.9
    }
  },
  "rate_basis": {
    "currency": "USD",
    "unit": "per_1m_text_tokens",
    "input": 0.7,
    "cache_hit_input": 0.0233,
    "output": 2.1,
    "effective_date": "2026-09-10"
  },
  "release_limits": {
    "max_output_tokens": 900,
    "max_retries": 1,
    "timeout_ms": 45000
  },
  "receipt_contract": [
    "request_id",
    "model_id",
    "prompt_tokens",
    "completion_tokens",
    "cache_status",
    "charged_amount"
  ],
  "unknowns": [
    "final invoice depends on selected account or key group",
    "cache behavior must be confirmed from request receipts"
  ]
}

This is intentionally not a prompt transcript. It does not contain an API key, request body, user identifier, customer name, or response text. A reviewer can understand the release economics without seeing private workload data.

Diff the manifest, not just the code

When the route changes, review the manifest diff beside the pull request.

For example, a code diff from deepseek-flash to glm-5.1 is easy to miss if it is hidden behind an environment variable. A manifest diff makes the change obvious:

- "model_id": "deepseek-flash"
+ "model_id": "glm-5.1"

- "input": 0.7
- "cache_hit_input": 0.0233
- "output": 2.1
+ "input": 2.1
+ "cache_hit_input": 0.680001
+ "output": 6.5999997

- "effective_date": "2026-09-10"
+ "effective_date": "2026-08-27"

That diff does not say one route is better. It says the release has a materially different cost profile and should be reviewed as such.

This is the right place to be precise. Avoid vague notes like "pricing changed" or "model upgraded". Say which row changed, which version was checked, and which assumption must be revalidated.

Treat cache as evidence, not hope

Many teams include cache-hit rows in forecasts because the number is attractive. The manifest should make a stricter distinction:

rate_basis.cache_hit_input is the listed planning rate.
cache_assumption is the workload belief.
receipt_contract.cache_status is the runtime proof.

If the release cannot produce a receipt field that distinguishes cache behavior, do not let the forecast depend on it. The manifest can still list the cache-hit row, but the verdict should say that cache savings are not release evidence yet.

Example:

{
  "cache_assumption": {
    "used_in_forecast": false,
    "reason": "receipt validation has not yet proved cache status for this path"
  },
  "review_verdict": "ship_without_cache_savings_claim"
}

That sentence is dull, which is exactly the point. It keeps a launch from borrowing certainty from a pricing table.

Add a pre-release gate

A simple gate can run in CI or as a release checklist:

from __future__ import annotations

import json
import urllib.request


def load_json(url: str) -> dict:
    with urllib.request.urlopen(url, timeout=20) as response:
        return json.loads(response.read().decode("utf-8"))


manifest = json.load(open("cost-manifest.json", encoding="utf-8"))
snapshot = load_json(manifest["pricing_snapshot"]["url"])
routes = load_json(manifest["route_table"]["url"])

assert snapshot["pricing_version"] == manifest["pricing_snapshot"]["pricing_version"]
assert snapshot["updated_at"] == manifest["pricing_snapshot"]["updated_at"]
assert routes["pricing_version"] == manifest["route_table"]["pricing_version"]

row = next(
    item for item in snapshot["models"]
    if item["id"] == manifest["model_id"]
)

assert row["input_usd_per_1m_tokens"] == manifest["rate_basis"]["input"]
assert row["output_usd_per_1m_tokens"] == manifest["rate_basis"]["output"]
assert "charged_amount" in manifest["receipt_contract"]
assert "API_KEY" not in json.dumps(manifest)

In production, replace the assert statements with structured failures and an audit file. The gate should fail closed when a version moves, a row disappears, or the receipt contract no longer matches the route.

Keep the public claim narrower than the private proof

A release manifest can support internal decisions, but it should not automatically become public copy.

Safe public language:

This route was planned against a dated public pricing snapshot and a live route table checked on September 18, 2026.

Risky public language:

This route will always reduce cost.

The first statement names evidence. The second predicts an outcome without knowing workload shape, cache behavior, retries, output length, account group, and final receipt.

For AI API gateways, trust usually comes from narrower claims with better evidence. A versioned cost manifest is one way to keep the release honest: it lets engineering move quickly while giving finance, security, and procurement something concrete to review.

The result is not a heavier process. It is a smaller surface for argument.

Before the route ships, everyone can answer the same question:

Which dated pricing row, route version, group rule, cache assumption, and receipt contract did we approve?

Build Route Evidence Capsules for AI API Gateways

Mattias chaw — Thu, 17 Sep 2026 13:08:14 +0000

Build Route Evidence Capsules for AI API Gateways

A model route can look stable from the client side while the evidence around it is already drifting.

The HTTP call still works. The OpenAI-compatible request shape still works. The model alias still resolves. The application team sees no broken test. Meanwhile, the dated price row may have changed, a cache field may no longer match the forecast, a policy exception may have moved from temporary to permanent, or the buyer may be looking at a stale spreadsheet that does not name the source date.

That gap is where Tier 1 and Tier 2 teams lose time. Engineering can explain the request. Finance can explain the budget. Security can explain which inputs were redacted. Procurement can explain why a route was approved. But those explanations often live in different places.

A route evidence capsule puts them in one small artifact.

The capsule is not a dashboard. It is not a public usage claim. It is not a private log export. It is a compact review object that travels with a route change, workbook, SDK example, incident review, or deployment gate.

It answers one question:

If this AI API route ships today, can a reviewer see exactly which model row, policy checks, pricing source, and redacted receipt shape supported the decision?

I will use AIWave as the concrete example because it exposes both a live route table and a dated public pricing snapshot. The same pattern applies to any OpenAI-compatible gateway that gives engineers one client surface while still requiring accountable route choices behind the scenes.

What belongs in the capsule

A useful capsule is intentionally small. It should be easy to attach to a pull request, release note, procurement packet, support escalation, or internal runbook.

At minimum, include these fields:

Field	Purpose
`capsule_id`	Stable identifier for the route decision
`checked_at`	When the evidence was collected
`route_name`	The exact model or gateway route used by the client
`client_shape`	The API family expected by the application
`pricing_sources`	Live and dated sources that were checked
`price_row`	The public base row used for planning
`group_note`	Reminder that account or key group can alter final billing
`policy_checks`	Data, region, retry, timeout, and tool-use boundaries
`receipt_shape`	Redacted fields expected after a request completes
`unknowns`	Items that must not be inferred
`verdict`	`pass`, `review`, or `block`

The most important field is often unknowns. Teams get into trouble when they let a missing denominator, missing cache field, or missing route receipt turn into a confident statement. The capsule should make that uncomfortable. If the evidence is absent, say so.

Source check from September 17, 2026

During this run on September 17, 2026, I checked AIWave's public pricing sources before writing the article.

The live route table at https://aiwave.live/api/pricing returned HTTP 200, success=true, 68 route rows, auto_groups=["default"], and pricing version a42d372ccf0b5dd13ecf71203521f9d2.

The dated public snapshot at https://aiwave.live/api/v1/pricing returned HTTP 200, checked=2026-09-10, updated_at=2026-09-10, currency USD, unit per_1m_text_tokens, source /api/pricing, source page https://aiwave.live/pricing, pricing version 8c7a0c0b30661ccbc13d142cb54d1e4ae445fe774b2c6fa501080db97c7a3e56, and 64 public model rows.

Those two sources should not be collapsed into one vague statement like "current pricing". The live route table is the operational source. The dated snapshot is the public evidence artifact. A capsule can use both, but it should record which one provided which fact.

Example rows from the dated public snapshot:

Model ID	Provider family	Input / 1M text tokens	Cache-hit input / 1M	Output / 1M text tokens	Effective date
`deepseek-v4-flash`	DeepSeek	`$0.638`	`$0.0202884`	`$1.914`	2026-08-27
`deepseek-v4-pro`	DeepSeek	`$1.914`	`$0.0637362`	`$5.742`	2026-08-27
`glm-4.5`	GLM	`$0.6975`	`$0.1800003375`	`$2.1699999225`	2026-08-27
`minimax-m2.5`	MiniMax	`$0.46866`	`$0.046866`	`$1.87464`	2026-08-27
`kimi-k2.7-code`	Kimi	`$1.89`	`$0.28500066`	`$5.99999967`	2026-08-27
`qwen3.8-27b`	Qwen	`$0.6695133382`	not listed	`$2.6780533528`	2026-08-27

These rows are public base rates, not final invoices. Final cost still depends on the account or key group, selected route, cache behavior, retry count, output length, and the private request receipt.

That is exactly why the capsule should carry both planning evidence and receipt expectations.

A capsule schema

Start with a schema that is boring on purpose.

{
  "capsule_id": "route_capsule_deepseek_v4_flash_2026_09_17",
  "checked_at": "2026-09-17T13:05:00Z",
  "route_name": "deepseek-v4-flash",
  "client_shape": "openai.chat.completions",
  "pricing_sources": {
    "live_route_table": "https://aiwave.live/api/pricing",
    "dated_snapshot": "https://aiwave.live/api/v1/pricing"
  },
  "price_row": {
    "currency": "USD",
    "unit": "per_1m_text_tokens",
    "input": 0.638,
    "cache_hit_input": 0.0202884,
    "output": 1.914,
    "effective_date": "2026-08-27"
  },
  "group_note": "Public row is a base rate. The final request receipt controls applied billing.",
  "policy_checks": {
    "prompt_logged": false,
    "api_key_logged": false,
    "retry_cap": 1,
    "timeout_ms": 60000,
    "tool_budget_required": true
  },
  "receipt_shape": {
    "request_id": "present",
    "model": "present",
    "input_tokens": "numeric",
    "output_tokens": "numeric",
    "cache_hit_tokens": "numeric_or_null",
    "charged_amount": "private_receipt_only"
  },
  "unknowns": [
    "Future route changes after checked_at",
    "Final output length before request completion",
    "Private account group unless authenticated receipt is inspected"
  ],
  "verdict": "review"
}

Notice that charged_amount is not copied into a public artifact. The capsule records the expected receipt shape, not customer-specific billing data. That distinction keeps the artifact useful without dragging private operations into places they do not belong.

Build it in Python

Here is a small capsule builder. It fetches the dated public snapshot, selects a route, and writes a review object. It uses an environment variable for the API key pattern in the example, but this specific public pricing call does not require a secret.

import datetime as dt
import json
import os
import urllib.request

SNAPSHOT_URL = "https://aiwave.live/api/v1/pricing"
LIVE_URL = "https://aiwave.live/api/pricing"


def fetch_json(url: str) -> dict:
    request = urllib.request.Request(
        url,
        headers={"User-Agent": "route-evidence-capsule/1.0"},
    )
    with urllib.request.urlopen(request, timeout=20) as response:
        return json.loads(response.read().decode("utf-8"))


def find_model(snapshot: dict, model_id: str) -> dict:
    for row in snapshot.get("models", []):
        if row.get("id") == model_id:
            return row
    raise ValueError(f"model row not found: {model_id}")


def build_capsule(model_id: str) -> dict:
    snapshot = fetch_json(SNAPSHOT_URL)
    row = find_model(snapshot, model_id)

    missing = []
    if row.get("cache_hit_usd_per_1m_tokens") is None:
        missing.append("cache_hit_input is not listed for this public row")

    verdict = "pass" if not missing else "review"

    return {
        "capsule_id": f"route_capsule_{model_id}_{dt.date.today().isoformat()}",
        "checked_at": dt.datetime.now(dt.UTC).isoformat(),
        "route_name": model_id,
        "client_shape": "openai.chat.completions",
        "pricing_sources": {
            "live_route_table": LIVE_URL,
            "dated_snapshot": SNAPSHOT_URL,
            "snapshot_checked": snapshot.get("checked"),
            "snapshot_version": snapshot.get("pricing_version"),
        },
        "price_row": {
            "provider": row.get("provider"),
            "currency": snapshot.get("currency"),
            "unit": snapshot.get("unit"),
            "input": row.get("input_usd_per_1m_tokens"),
            "cache_hit_input": row.get("cache_hit_usd_per_1m_tokens"),
            "output": row.get("output_usd_per_1m_tokens"),
            "effective_date": row.get("effective_date"),
        },
        "policy_checks": {
            "api_key_source": "os.environ.get('AIWAVE_API_KEY')",
            "api_key_logged": False,
            "prompt_logged": False,
            "timeout_ms": 60000,
            "retry_cap": 1,
        },
        "unknowns": missing,
        "verdict": verdict,
    }


if __name__ == "__main__":
    # Pattern only. Do not print or persist the actual key.
    api_key = os.environ.get("AIWAVE_API_KEY")
    capsule = build_capsule("deepseek-v4-flash")
    print(json.dumps(capsule, indent=2))

In production, I would add a second fetch against the live route table and compare route availability, endpoint support, and pricing version. I would also reject a capsule if the model row exists in only one source unless the release owner explicitly accepts that mismatch.

Review gates

The capsule becomes useful when it blocks vague approvals.

Add these gates before a route ships:

Gate	Pass condition	Review condition	Block condition
Source freshness	Pricing source checked in the current run	Dated snapshot is older than planned	Source cannot be fetched
Row identity	Exact model ID exists	Alias maps through an undocumented route	Only a similar model is found
Unit clarity	Currency and unit are present	Unit exists but examples mix units	Unit is missing
Cache semantics	Cache row is present or explicitly absent	Cache field differs across sources	Forecast assumes cache when no row exists
Receipt safety	Receipt shape is redacted	A reviewer asks for private fields	Secret, prompt, or customer data appears
Billing group	Group note is explicit	Group is known only after auth	Public text claims final billing without receipt
Retry behavior	Retry cap is named	Retry policy differs by route	Forecast ignores retries

This is a better workflow than a long meeting where everyone says "pricing looks current" without naming the source.

The gate should produce one of three verdicts:

pass   - evidence supports the route decision
review - route can proceed only with named unknowns accepted
block  - source, row, unit, policy, or receipt safety failed

For many model changes, review is the honest result. A missing cache row is not always a deployment blocker. A changed version field is not always a bug. But both should be visible before the route becomes a default in code, a spreadsheet, or a public example.

Where to store capsules

Do not bury route evidence in chat history. Store capsules near the thing they approve.

For an SDK, put them under evidence/routes/ and link them from the pull request. For a procurement workbook, attach the capsule JSON next to the workbook export. For a production route change, store the capsule with the release artifact. For an incident, create a post-incident capsule that records which assumptions were wrong.

Keep the artifact small enough that a reviewer reads it.

A practical naming scheme:

evidence/routes/
  2026-09-17-deepseek-v4-flash-chat-completions.json
  2026-09-17-glm-4-5-cache-policy.json
  2026-09-17-qwen3-8-27b-no-cache-row-review.json

The name should include the date, model ID, and reason for review. That makes later searches useful without opening every file.

What not to put in the capsule

The capsule is an evidence boundary. Keep it clean.

Do not include API keys. Do not include prompts. Do not include raw responses. Do not include customer identifiers. Do not include private revenue, registration, or request volume. Do not use the capsule as marketing proof. Do not claim a route is faster, more reliable, or better unless the exact benchmark and date are in scope.

The capsule should help a serious reviewer say:

Which route was chosen.
Which pricing sources were checked.
Which date and unit apply.
Which policy constraints were accepted.
Which unknowns remain.
Which private facts are intentionally excluded.

That last point matters. A good artifact is not only clear about what it knows. It is clear about what it refuses to expose.

A release checklist

Before a route evidence capsule is accepted, run this checklist:

[ ] Exact route name appears in the live route table or approved source
[ ] Public snapshot row is present, dated, and uses USD per 1M text tokens
[ ] Cache-hit field is either recorded or explicitly absent
[ ] Account or key group note is present
[ ] Retry cap and timeout are named
[ ] Receipt shape is redacted
[ ] No API key, prompt body, raw customer data, or private usage number appears
[ ] Verdict is pass, review, or block
[ ] Reviewer can reproduce the public source checks

This checklist is deliberately plain. Fancy controls fail when people cannot run them during a real release. A capsule should be easy to regenerate and easy to challenge.

Why this helps Tier 1 and Tier 2 teams

Tier 1 and Tier 2 buyers rarely reject AI API work because an engineer cannot write a request. They slow down because every serious route change crosses multiple concerns: cost, source date, region, retention posture, retry behavior, support path, and receipt auditability.

Route evidence capsules reduce translation work between those concerns.

Engineering gets a stable artifact for code review. Finance gets dated units and source URLs. Security gets a redaction boundary. Procurement gets a small trail that explains why one route was accepted without relying on vague claims. Support gets a way to inspect drift without asking for secrets.

The real benefit is cultural: a team stops treating model choice as a hidden preference and starts treating it as a reviewable decision.

When the next route changes, the question is no longer "Which model did we like?" It becomes "Which evidence capsule supports this route today?"

That is a much better question to ship with.

Build Tool-Fee Budget Gates for AI API Routes

Mattias chaw — Wed, 16 Sep 2026 13:08:25 +0000

Tool-enabled AI routes change the budget problem.

A plain chat-completion estimate usually starts with input tokens, output tokens, model id, and retry policy. That is already enough to make finance and engineering compare notes. Once the route can call web search, fetch documents, process images, create media, or run long tool chains, a token-only estimate becomes too thin.

The request may still look OpenAI-compatible from the client. The operational evidence behind it is wider:

which model route was admitted;
which tools were allowed;
whether the tool result was billed separately or folded into model usage;
which pricing source was checked;
which account group or route group applied;
which release approved the tool policy;
and which runtime receipt proves the actual usage.

For Tier 1 and Tier 2 teams, the useful question is not "can the model call tools?" It is "can the platform prove which tool-enabled route was allowed to spend, why it was allowed, and what evidence will be available after the run?"

This article gives a compact tool-fee budget gate for AI API routes. I use AIWave as the concrete example because it exposes dated public pricing JSON and OpenAI-compatible route metadata, but the pattern applies to any multi-model gateway or internal model platform.

For this run, I rechecked AIWave's public pricing endpoints on September 16, 2026. The static customer-facing endpoint at https://aiwave.live/api/v1/pricing returned 64 model rows, checked: 2026-09-10, updated_at: 2026-09-10, and pricing version 8c7a0c0b30661ccbc13d142cb54d1e4ae445fe774b2c6fa501080db97c7a3e56. The dynamic route table at https://aiwave.live/api/pricing returned 64 rows, pricing version a42d372ccf0b5dd13ecf71203521f9d2, route groups default, vip, and svip, and supported endpoint type openai. Treat those as dated source facts, not evergreen commercial copy.

Separate route price from tool policy

The first mistake is putting the model price and tool policy into the same unlabeled field.

Model pricing answers one question: what is the dated base rate for text tokens on a route?

Tool policy answers a different question: what extra capabilities may this workload invoke, under what limits, and with what receipt fields?

Keep them separate in code and in review packets. A route may be acceptable for normal chat but blocked for search-enabled work. Another route may be acceptable for search only when the application stores a source list, a query count, a redacted tool receipt, and an output cap.

A useful policy record can be small:

{
  "policy_id": "support-search-summary-v3",
  "model": "glm-5",
  "endpoint_shape": "openai_chat_completions",
  "pricing_source": "https://aiwave.live/api/v1/pricing",
  "pricing_checked": "2026-09-10",
  "pricing_view": "public_base_rate",
  "allowed_tools": ["web_search"],
  "max_tool_calls": 2,
  "max_output_tokens": 900,
  "requires_receipt": true,
  "owner": "platform-ai"
}

Notice what is absent: no prompt, no response body, no reusable key, no customer name, and no private account detail. The record is a release contract, not a debug transcript.

Admit the run before the first call

The budget gate should run before the first model call, not after the invoice review.

It should combine four inputs:

Input	Example field	Why it matters
Route	`model`, `endpoint_shape`	Prevents unsupported request shapes from shipping
Pricing	`pricing_checked`, `pricing_version`	Keeps estimates tied to dated source evidence
Tools	`allowed_tools`, `max_tool_calls`	Stops hidden search or media work from expanding silently
Workload	`input_tokens_estimate`, `output_cap`	Creates a bounded envelope before execution

Here is a minimal Python shape:

from dataclasses import dataclass
from decimal import Decimal


@dataclass(frozen=True)
class RoutePrice:
    model: str
    input_per_1m: Decimal
    output_per_1m: Decimal
    cache_hit_per_1m: Decimal | None
    pricing_checked: str
    pricing_version: str


@dataclass(frozen=True)
class ToolPolicy:
    policy_id: str
    allowed_tools: tuple[str, ...]
    max_tool_calls: int
    max_output_tokens: int
    receipt_required: bool = True


@dataclass(frozen=True)
class RunEstimate:
    input_tokens: int
    cached_input_tokens: int
    output_tokens_cap: int
    requested_tools: tuple[str, ...]


def estimate_text_usd(price: RoutePrice, run: RunEstimate) -> Decimal:
    fresh_input = max(run.input_tokens - run.cached_input_tokens, 0)
    total = Decimal(fresh_input) * price.input_per_1m / Decimal(1_000_000)
    if price.cache_hit_per_1m is not None:
        total += Decimal(run.cached_input_tokens) * price.cache_hit_per_1m / Decimal(1_000_000)
    total += Decimal(run.output_tokens_cap) * price.output_per_1m / Decimal(1_000_000)
    return total


def admit(policy: ToolPolicy, run: RunEstimate) -> list[str]:
    problems: list[str] = []
    unknown_tools = sorted(set(run.requested_tools) - set(policy.allowed_tools))
    if unknown_tools:
        problems.append(f"tools_not_allowed:{','.join(unknown_tools)}")
    if len(run.requested_tools) > policy.max_tool_calls:
        problems.append("too_many_tool_calls")
    if run.output_tokens_cap > policy.max_output_tokens:
        problems.append("output_cap_exceeds_policy")
    return problems

The important part is not the exact function. The important part is that admission produces a decision before the route spends. A rejected run should fail with a policy reason. An admitted run should carry the policy id, pricing version, and estimate id into the runtime receipt.

Add a separate tool-fee envelope

Some tool usage has a simple unit, such as number of searches. Some has a route-specific charge. Some is bundled by the platform. Some depends on upstream behavior that your application cannot safely infer from a model name.

Do not hide that uncertainty in a token estimate.

Add a separate envelope:

{
  "tool_fee_envelope": {
    "billing_view": "policy_enforced_limit",
    "web_search": {
      "allowed": true,
      "max_calls": 2,
      "pricing_source": "route_policy",
      "runtime_receipt_field": "tool_calls.web_search"
    },
    "media_input": {
      "allowed": false,
      "reason": "not_approved_for_this_workload"
    }
  }
}

That envelope makes the review honest. If the team has not approved media processing for a workflow, the runtime should not get to enable it just because the model can accept media. If search is allowed, the receipt should record how many searches happened, which policy admitted them, and whether the result was used.

Record the runtime receipt

After execution, the platform should export a receipt that can be reviewed without raw prompts or private user data.

A compact receipt can include:

request_id_hash
policy_id
model
pricing_source
pricing_checked
pricing_version
account_group_label
input_tokens
cached_input_tokens
output_tokens
tool_calls
tool_policy_result
retry_count
final_status

The receipt should not include the API key, prompt text, response body, payment identifier, IP address, or customer name. If a support engineer needs to debug content quality, use a separate redacted fixture process. The budget receipt only proves route, price, tool, and usage facts.

This separation helps during incident review. A run may fail because a tool was blocked by policy, because the output cap was too small, because a route was unavailable, or because a request exceeded a token budget. Those outcomes should produce different support messages and different engineering actions.

Run a canary when the policy changes

The last gate is a canary.

Run it whenever a route policy changes, a model is added to a tool-enabled workflow, a pricing snapshot changes, or an SDK example starts using a new tool.

The canary should check:

The model exists in the approved pricing snapshot.
The route supports the endpoint shape the SDK will call.
The policy lists every requested tool.
The output cap and tool-call cap are bounded.
The receipt schema can store the expected tool evidence.
The public example uses an obvious placeholder such as YOUR_API_KEY_HERE, never a realistic key shape.

This is a small test, but it catches the quiet failures: stale model names, unlabeled pricing dates, examples that imply unsupported tools, and routes that accidentally allow a tool without a receipt.

What to ship

A tool-fee budget gate does not need to be a large platform. Start with four artifacts:

a dated pricing snapshot;
a route policy file;
an admission function;
and a redacted runtime receipt.

That is enough to make tool-enabled AI routes reviewable. Engineering can explain what shipped. Finance can see which dated pricing source was used. Support can tell whether a tool was allowed or blocked. Security can verify that the shared evidence does not contain prompts, reusable credentials, or customer-identifying data.

The model route is still useful. The tools are still useful. The difference is that each tool-enabled run now leaves evidence before and after it spends.

That is the contract production buyers actually need.

Capture Official Price Page Drift Before AI API Workbooks Ship

Mattias chaw — Tue, 15 Sep 2026 13:42:18 +0000

AI API cost workbooks drift faster than most teams expect.

The workbook starts as a useful bridge between engineering, finance, and procurement. It has a model column, input token assumptions, output token assumptions, cache-hit assumptions, a route owner, and a projected monthly cost. Then one upstream page changes. A gateway normalizes a new route. A billing group multiplier is clarified. An old model alias stays in one sheet tab because nobody touched that tab during the last release.

Nothing breaks at compile time.

The broken part appears later, usually in a budget review or incident review, when someone asks why the shipped SDK example, the workbook estimate, and the runtime ledger all disagree.

For Tier 1 and Tier 2 teams, the answer cannot be "the spreadsheet was probably close enough." The answer should be a receipt:

which pricing source was checked;
when it was checked;
which exact model rows entered the workbook;
whether the workbook shows base rates or effective account rates;
who approved a meaningful change;
and which release consumed that approved snapshot.

This article shows a compact drift-capture pattern for AI API pricing workbooks. The examples use AIWave's public pricing endpoints because they are date-stamped and OpenAI-compatible gateway oriented, but the pattern applies to any multi-model API gateway or internal platform.

For this article, I rechecked the public AIWave pricing endpoints on September 15, 2026. The static endpoint at https://aiwave.live/api/v1/pricing returned 64 model rows, checked: 2026-09-10, updated_at: 2026-09-10, and a pricing version. The dynamic endpoint at https://aiwave.live/api/pricing returned 64 route rows and current group ratios of default: 1 and vip: 0.9. Treat those as source facts with dates, not evergreen copy.

The workbook should not be the source of record

A workbook is a decision surface. It is not the source of record.

That distinction matters. A workbook often has scenario columns, finance assumptions, annualized projections, currency preferences, and team-specific workload estimates. Those fields are useful, but they are not the same thing as the published pricing row or the runtime route table.

Keep three layers separate:

Source facts: the public pricing endpoint, official provider page, route table, or approved internal export.
Workbook assumptions: workload size, cache-hit assumption, retry policy, billing view, currency conversion date, and owner.
Runtime receipts: the actual model, account group, token counts, cache behavior, retry count, and final billed result.

When teams collapse those layers into one sheet, every discussion becomes ambiguous. Did the model price change, or did the scenario change? Did the default rate change, or did someone apply a VIP key multiplier? Did the runtime call hit cache, or did the workbook assume cache?

The drift-capture job should update source facts. It should not silently rewrite business assumptions.

Capture a snapshot before the workbook opens

Start with a small snapshot command. It fetches the pricing source, records the source dates, and stores a content fingerprint. It does not need API credentials if the pricing source is public.

import hashlib
import json
import urllib.request
from datetime import datetime, timezone
from pathlib import Path


PRICING_URL = "https://aiwave.live/api/v1/pricing"


def fetch_json(url: str) -> tuple[dict, bytes]:
    request = urllib.request.Request(
        url,
        headers={"User-Agent": "pricing-drift-capture/1.0"},
    )
    with urllib.request.urlopen(request, timeout=20) as response:
        raw = response.read()
    return json.loads(raw.decode("utf-8")), raw


def model_rows(payload: dict) -> list[dict]:
    return payload.get("models") or payload.get("data") or []


payload, raw = fetch_json(PRICING_URL)
snapshot = {
    "captured_at": datetime.now(timezone.utc).isoformat(),
    "pricing_source": PRICING_URL,
    "pricing_checked": payload.get("checked"),
    "pricing_updated_at": payload.get("updated_at"),
    "pricing_version": payload.get("pricing_version"),
    "content_sha256": hashlib.sha256(raw).hexdigest(),
    "row_count": len(model_rows(payload)),
    "models": model_rows(payload),
}

Path("pricing-snapshot.json").write_text(
    json.dumps(snapshot, indent=2, sort_keys=True),
    encoding="utf-8",
)

That file is boring in the best possible way. It gives reviewers one artifact to diff. It also makes the workbook reproducible. If someone asks what the workbook believed on release day, the answer is in a JSON file, not in a tab that may have been edited later.

Normalize only the fields the workbook needs

The next step is not to copy every provider-specific field into a spreadsheet. Normalize only what the workbook needs for its estimates and reviews.

A compact normalized row can look like this:

{
  "model": "deepseek-v4-pro",
  "provider": "DeepSeek",
  "unit": "per_1m_text_tokens",
  "input_usd_per_1m_tokens": 1.914,
  "cache_hit_usd_per_1m_tokens": 0.0637362,
  "output_usd_per_1m_tokens": 5.742,
  "effective_date": "2026-08-27",
  "pricing_checked": "2026-09-10",
  "billing_view": "base_default_rate"
}

The important field is billing_view.

If the workbook shows base default rates, say that. If it shows effective account rates, say that instead and include the multiplier source. A discount, negotiated group, or account-specific setting should never be mixed into a base pricing table without a label.

AIWave's current public base price view is default 1. A VIP key may use 0.9 where the application and account configuration allow it. A workbook can show both, but it should not let a reader confuse one for the other.

Compare against the last approved snapshot

A nightly job can fetch pricing every day. It should not automatically push every change into finance workbooks.

Instead, compare the new snapshot against the last approved snapshot and classify the diff.

from dataclasses import dataclass


PRICE_FIELDS = (
    "input_usd_per_1m_tokens",
    "cache_hit_usd_per_1m_tokens",
    "output_usd_per_1m_tokens",
)


@dataclass
class Drift:
    code: str
    model: str
    field: str
    before: object
    after: object


def index_by_model(rows: list[dict]) -> dict[str, dict]:
    return {row["model"]: row for row in rows if row.get("model")}


def compare_snapshots(before_rows: list[dict], after_rows: list[dict]) -> list[Drift]:
    before = index_by_model(before_rows)
    after = index_by_model(after_rows)
    drifts: list[Drift] = []

    for model in sorted(set(before) | set(after)):
        if model not in before:
            drifts.append(Drift("model_added", model, "", None, after[model]))
            continue
        if model not in after:
            drifts.append(Drift("model_removed", model, "", before[model], None))
            continue

        for field in PRICE_FIELDS:
            if before[model].get(field) != after[model].get(field):
                drifts.append(
                    Drift(
                        "price_field_changed",
                        model,
                        field,
                        before[model].get(field),
                        after[model].get(field),
                    )
                )

        if before[model].get("effective_date") != after[model].get("effective_date"):
            drifts.append(
                Drift(
                    "effective_date_changed",
                    model,
                    "effective_date",
                    before[model].get("effective_date"),
                    after[model].get("effective_date"),
                )
            )

    return drifts

Do not treat every diff as a release blocker. A new model row may simply be catalog growth. A changed effective date may require human review. A missing route used by an SDK example should block the release until a fallback or deprecation note is approved.

The useful categories are simple:

Informational: new row, metadata clarification, provider display-name change.
Review required: price field changed, effective date changed, billing multiplier changed.
Blocking: workbook references a removed model, unsupported endpoint, missing billing view, or stale source date.

That classification lets finance and engineering work from the same evidence without making every nightly change an emergency.

Write back with an approval receipt

The workbook update should be a deliberate action. It can be automated, but it should still leave a receipt.

{
  "receipt_type": "pricing_workbook_update",
  "approved_at": "2026-09-15T02:18:00Z",
  "pricing_source": "https://aiwave.live/api/v1/pricing",
  "pricing_checked": "2026-09-10",
  "pricing_version": "8c7a0c0b30661ccbc13d142cb54d1e4ae445fe774b2c6fa501080db97c7a3e56",
  "billing_view": "base_default_rate",
  "row_count": 64,
  "changed_models": ["deepseek-flash"],
  "reviewer": "platform-owner",
  "workbook": "ai-api-cost-model.xlsx"
}

That receipt is safe to store in a repo or CI artifact if it contains no credentials, request bodies, customer identifiers, or private account state. It should say what changed and why the workbook was allowed to consume it.

If your workbook includes effective account rates, add the multiplier source:

{
  "billing_view": "effective_account_rate",
  "group_ratio_source": "https://aiwave.live/api/pricing",
  "group_ratio_checked_at": "2026-09-15T02:18:00Z",
  "group_ratio": {
    "default": 1,
    "vip": 0.9
  }
}

Keep the base-rate receipt and the effective-rate receipt separate. That separation saves hours during procurement review.

Add a workbook contract test

Once the workbook consumes a snapshot, add a small contract test around the export.

The test should fail when the workbook:

references a model that is missing from the approved snapshot;
uses a price row without effective_date;
omits pricing_source;
omits billing_view;
mixes base and effective account rates in one unlabeled column;
or ships with a snapshot older than your review window.

Here is a minimal CSV-oriented version:

import csv
import json
from pathlib import Path


REQUIRED_COLUMNS = {
    "model",
    "pricing_source",
    "pricing_checked",
    "effective_date",
    "billing_view",
    "input_usd_per_1m_tokens",
    "output_usd_per_1m_tokens",
}


def load_snapshot_models(path: str) -> set[str]:
    snapshot = json.loads(Path(path).read_text(encoding="utf-8"))
    return {
        row["model"]
        for row in snapshot["models"]
        if row.get("model")
    }


def check_workbook_export(csv_path: str, snapshot_path: str) -> list[str]:
    approved_models = load_snapshot_models(snapshot_path)
    failures: list[str] = []

    with open(csv_path, newline="", encoding="utf-8") as handle:
        reader = csv.DictReader(handle)
        missing = REQUIRED_COLUMNS - set(reader.fieldnames or [])
        if missing:
            failures.append(f"missing columns: {sorted(missing)}")
            return failures

        billing_views = set()
        for row_number, row in enumerate(reader, start=2):
            model = row["model"]
            billing_views.add(row["billing_view"])

            if model not in approved_models:
                failures.append(f"row {row_number}: unknown model {model}")
            if not row["effective_date"]:
                failures.append(f"row {row_number}: missing effective date")
            if not row["pricing_checked"]:
                failures.append(f"row {row_number}: missing pricing checked date")

        if len(billing_views) > 1:
            failures.append("mixed billing views require separate labeled sections")

    return failures

That test is intentionally narrow. It does not calculate your whole forecast. It only checks whether the workbook can defend its source facts.

Decide what should block a release

Not every workbook issue deserves the same response.

Use a release rule that engineers and finance can both understand:

An AI API cost workbook may ship only when:
1. every model row comes from an approved dated snapshot,
2. the pricing source and pricing version are recorded,
3. base rates and effective account rates are labeled separately,
4. source drift above the review threshold has an approval receipt,
5. and the runtime SDK examples point to the same billing view.

That last point is easy to miss. A workbook can be correct while the README is stale. A README can be correct while the demo notebook still has an old route name. External docs, SDK examples, notebooks, and procurement sheets should all point to the same pricing source and billing view.

The release gate does not need private business data. It does not need real API keys. It does not need customer examples. It needs dated public facts, explicit assumptions, and a receipt trail.

Keep source drift separate from runtime variance

Source drift and runtime variance are different failures.

Source drift means the workbook used stale or unlabeled pricing facts. Runtime variance means the actual run had different token counts, cache behavior, retry behavior, model resolution, or account group than the estimate assumed.

Do not solve both with one giant spreadsheet. Capture pricing drift before the workbook updates. Capture runtime variance in request-level receipts after calls execute.

Then join the two only during review:

The workbook explains the expected route and budget.
The pricing snapshot explains the source facts.
The runtime receipt explains what actually happened.
The variance review explains whether the policy, implementation, or estimate should change.

That workflow is slower than pasting a new row into a sheet. It is also much easier to trust.

A practical rollout path

If your team already has a pricing workbook, do not rebuild it first.

Start with a snapshot file and one contract test. Add pricing_source, pricing_checked, effective_date, and billing_view to the rows that feed production SDK examples. Run the test in warning mode for one week.

Next, add a nightly drift job. Let it open a review item when the public source changes, but do not let it rewrite scenarios by itself. Humans should decide whether a change affects active estimates, docs, demos, or only future catalog exploration.

Finally, require an approval receipt before any release that changes model price assumptions or billing-group assumptions.

The goal is not to make workbooks fancy. The goal is to make them answerable.

When a buyer asks why a number changed, you should be able to point to a source URL, a date, a version, a diff, and a receipt. That is the difference between a spreadsheet and an operational contract.