DEV Community: Mattias chaw

Build a first-success telemetry loop for AI API onboarding

Mattias chaw — Wed, 09 Sep 2026 13:12:43 +0000

Build a first-success telemetry loop for AI API onboarding

Most AI API onboarding funnels overcount setup and undercount success.

They know when a developer creates an account. They often know when a developer creates a key. They may know when someone clicks a run button inside a dashboard. But many serious evaluators do the important step somewhere else: they copy a curl command, paste it into a terminal, move the base URL into a Python script, wire the key into a CI secret store, or call the API from an existing service.

If your telemetry only watches the dashboard button, you can end up staring at a false zero. The event endpoint may be healthy. The product may have real usage. The logs may contain requests. The funnel still says nobody tried.

That is a measurement problem before it is a growth problem.

This article lays out a first-success telemetry loop for OpenAI-compatible AI API onboarding. The examples use AIWave as the concrete gateway context because AIWave exposes dated public pricing JSON, a status page, model docs, and an OpenAI-compatible base URL at https://aiwave.live/v1. The same pattern applies to any gateway that routes work across several upstream providers.

The objective is narrow: record whether a new developer reached a first real API attempt and a first successful response, without storing prompts, responses, reusable credentials, IP addresses, or private customer identifiers in the activation log.

Define success before optimizing conversion

A new user funnel is usually drawn like this:

Stage	Common event	Why it is incomplete
Account created	signup	Does not prove intent to integrate
Credential created	key_created	Does not prove the key was used
Dashboard sample clicked	run_clicked	Misses terminal and SDK usage
First API attempt	first_attempt	Must include copied examples and existing clients
First successful response	first_success	Should be tied to an actual request path
First bounded failure	first_error_type	Needs classification without leaking data

For an API product, the real activation event is not a page view. It is the first successful request against the documented base URL.

For an OpenAI-compatible gateway, that success should include at least:

the route family or model string
the documented base URL version
the client mode, such as curl, Python, Node.js, dashboard, or backend
the final status class
the source date for pricing or model metadata
a redacted receipt identifier when available

It should not include the prompt, completion, full credential, user email, billing amount, or customer name. The activation log is a control surface, not a shadow copy of production traffic.

Start with public evidence

Before writing any onboarding claim into docs or examples, check the public contract that a developer will actually use.

For AIWave, a read-only check on Sep 9, 2026 returned:

Public asset	Result
`https://aiwave.live/api/pricing`	HTTP 200, `success=true`, 63 route records
`https://aiwave.live/api/v1/pricing`	HTTP 200, 63 model rows, 9 providers
Currency and unit	USD, per 1M text tokens
Pricing version	`a42d372ccf0b5dd13ecf71203521f9d2`
Snapshot date in public v1 JSON	`2026-08-27`

That does not mean every evaluator will pay the same amount. Account group, route policy, cache-hit behavior, output length, and actual request receipts still matter. But the public contract is strong enough to build examples around dated pricing evidence instead of copy that drifts quietly.

The onboarding loop should store the pricing source date near the first-success event. When a user later asks why a forecast differed from a bill, support can separate three questions:

Which public rate card did the onboarding sample reference?
Which model and route did the first request use?
Which usage fields did the final receipt record?

That separation matters when an API supports cache-aware pricing, long context, multiple providers, and model aliases.

Instrument every client path

The most common bug in API onboarding analytics is path bias.

The dashboard path is easy to instrument. A button click can emit first_attempt, and a successful response handler can emit first_success. But a developer who is serious about production may skip that path after key creation. They may copy the example and run it locally, or move directly into an SDK.

The telemetry loop needs two layers:

Layer	Source	Purpose
Product events	Dashboard and docs UI	Capture intended onboarding actions
Request-derived reconciliation	Redacted API request logs	Capture real attempts outside the UI

The second layer should be read-only against the operational database or log stream. It should append bounded events into a separate onboarding event store. It should not modify user groups, payment state, token groups, quota, route configuration, or the gateway core.

That division keeps the measurement fix small. You are not changing billing behavior. You are not changing access control. You are only repairing visibility into whether the first request happened.

Use a sidecar reconciler

A sidecar reconciler can be simple.

It reads recent request metadata, finds users or credentials that have a key-created event but no attempt event, and appends one missing attempt event when a qualifying API log exists. If a qualifying request completed successfully, it appends one success event. If the first visible request failed, it appends a bounded error class.

The reconciler should be idempotent. Running it every five minutes should not duplicate events.

Here is a stripped-down schema:

create table onboarding_events (
  event_id text primary key,
  user_ref text not null,
  event_type text not null,
  event_day text not null,
  source text not null,
  client_family text,
  route_family text,
  status_class text,
  error_type text,
  pricing_version text,
  pricing_checked_at text,
  created_at text not null
);

The user_ref should be a stable internal reference or salted hash that your analytics system can join safely. Do not export it into public reports. If you do not need user-level joins outside the secure environment, aggregate as early as possible.

The error_type should be bounded. Good examples are auth, quota, rate_limit, timeout, provider, validation, and unknown. Bad examples are raw exception bodies, upstream response payloads, or anything that can contain a prompt.

Keep the event writer boring

The writer should be designed so a reviewer can answer three questions quickly:

Can it leak reusable credentials?
Can it leak request content?
Can it change production behavior?

If the answer to any of those is unclear, the telemetry system is too ambitious.

A minimal Python-shaped example looks like this:

from dataclasses import dataclass
from datetime import datetime, timezone
from hashlib import sha256

PRICING_VERSION = "a42d372ccf0b5dd13ecf71203521f9d2"
PRICING_CHECKED_AT = "2026-09-09"

@dataclass(frozen=True)
class ApiLogSummary:
    user_id: str
    request_id: str
    model: str
    status_code: int
    client_family: str

def pseudonymous_ref(user_id: str, salt: str) -> str:
    return sha256(f"{salt}:{user_id}".encode("utf-8")).hexdigest()[:32]

def status_class(code: int) -> str:
    if 200 <= code < 300:
        return "success"
    if code in (401, 403):
        return "auth"
    if code == 402:
        return "quota"
    if code == 429:
        return "rate_limit"
    if code >= 500:
        return "provider"
    return "validation"

def first_success_event(row: ApiLogSummary, salt: str) -> dict:
    kind = status_class(row.status_code)
    return {
        "event_id": f"first:{row.request_id}:{kind}",
        "user_ref": pseudonymous_ref(row.user_id, salt),
        "event_type": "first_success" if kind == "success" else "first_error_type",
        "event_day": datetime.now(timezone.utc).date().isoformat(),
        "source": "request_log_reconciler",
        "client_family": row.client_family,
        "route_family": row.model,
        "status_class": kind,
        "error_type": None if kind == "success" else kind,
        "pricing_version": PRICING_VERSION,
        "pricing_checked_at": PRICING_CHECKED_AT,
        "created_at": datetime.now(timezone.utc).isoformat(),
    }

This is intentionally plain. No prompt. No completion. No credential. No email. No billing claim. No private request body.

The production implementation can be more careful about batching, durable cursors, transaction isolation, and retry behavior. The security shape should stay the same.

Deduplicate by event meaning

An onboarding reconciler should not append an event every time it sees another request. The first success is a milestone. Once the milestone exists, later requests belong in product analytics, billing, reliability, or customer health systems.

Useful deduplication keys are:

Event	Suggested natural key
`first_attempt`	user reference plus first qualifying request
`first_success`	user reference plus first successful request
`first_error_type`	user reference plus first bounded failure class

Do not deduplicate only by timestamp. A replay, delayed log write, or cron restart can shift timestamps and create duplicate milestones.

Also avoid a model where first_error_type blocks first_success. A user can fail on auth, fix the key, and succeed five minutes later. The funnel should show both: first known failure class, then first success.

Report aggregates, not private detail

The daily report should be useful to the operator without becoming a privacy risk.

A good report has rows like:

Day	Registrations	Credentials created	First attempts	First successes	Auth errors	Quota errors	Rate limits
2026-09-09	n	n	n	n	n	n	n

Keep the report at the cohort level unless there is a specific support case with consent and access controls. A public blog post should not publish early cohort counts, paid-customer counts, revenue, request volume, or customer workload size. Those figures can be valid internally and still be wrong for public proof.

For public material, it is enough to explain the mechanism:

dashboard events are useful but incomplete
request-derived reconciliation covers copied examples and SDK traffic
event logs exclude prompts, responses, credentials, and personal contact data
first-success milestones are deduplicated
pricing evidence is dated
failures use bounded classes

That is the lesson a Tier 1 or Tier 2 engineering team can reuse.

Make base URL verification part of onboarding

The fastest way to damage activation data is to give the user a sample that cannot run.

Every daily onboarding check should validate:

docs page returns HTTP 200
pricing JSON returns HTTP 200
status page returns HTTP 200
the documented base URL resolves
an unauthenticated call fails with the expected auth class
an authenticated disposable test can complete a small request
curl, Python, and Node.js examples use the same base URL

For AIWave's public examples, the OpenAI-compatible base URL is:

https://aiwave.live/v1

A redacted curl sample should look like this:

curl https://aiwave.live/v1/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY_HERE" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-v4-flash",
    "messages": [
      {"role": "user", "content": "Return a three-step migration checklist."}
    ],
    "max_tokens": 300
  }'

The placeholder is deliberate. Public docs should never include a reusable credential, even a short-lived one. The dashboard can provide secure key copy controls after authentication. The public article should show shape, not secrets.

Treat email as a later-stage control

It is tempting to email every user who created an account and did not call the API. That can be a mistake.

First, fix measurement. If terminal and SDK attempts were invisible, the "never called" segment may be contaminated. Second, separate historical cohorts from current onboarding. Old registrations from a different offer, a different credit policy, or a different product narrative may not tell you much about today's activation.

A safer activation email policy starts with constraints:

only include accounts that are old enough to have tried
recheck success before every send
suppress unsubscribes, bounces, complaints, and support-sensitive accounts
send docs and troubleshooting, not a credential
cap daily volume during the first run
keep a holdout group if the sample is large enough

The email should point the user back to the authenticated dashboard or docs. It should not send an API key. It should not include private usage detail. It should not assume the lack of a dashboard event means the developer had no interest.

Procurement questions

Teams buying an AI API gateway should ask onboarding questions during evaluation, not after rollout.

Ask:

What exactly counts as first success?
Does the funnel include copied curl and SDK calls?
Are prompts and completions excluded from activation events?
Are API keys excluded from logs, reports, and emails?
Are failure classes bounded?
Is the pricing source date stored with examples?
Can the platform distinguish dashboard usage from backend usage?
Are account group and route policy preserved for receipt review?
Is a first-success report available without exposing personal data?

These questions are practical. They tell you whether the gateway can help a developer get to the first working request and whether the operator can improve onboarding without guessing.

Final checklist

A first-success telemetry loop is ready when:

key creation is not treated as activation
dashboard events and request-derived events are reconciled
the reconciler is read-only against operational data
event writes are append-only and idempotent
prompts, responses, credentials, IP addresses, and personal contact data are excluded
error categories are bounded
public examples use the verified base URL
pricing evidence has a source date and version
daily reports are aggregate
email activation waits until measurement is trustworthy

For AIWave-style gateways, this loop is especially important because the product is not a single model button. Developers are evaluating route stability, model aliases, pricing evidence, cache behavior, and receipt quality at the same time.

If the funnel cannot see the first real request, the team may optimize the wrong thing. Measure first success directly, keep the event small, and let the next product decision start from evidence instead of a dashboard-only guess.

Build a public evidence bundle for AI API gateways

Mattias chaw — Tue, 08 Sep 2026 13:15:23 +0000

Build a public evidence bundle for AI API gateways

An AI API gateway is easy to describe and hard to trust.

The client code may only change a base URL and a model name. The operating surface behind that line is larger: provider availability, model aliases, cache behavior, account multipliers, failed request semantics, billing records, status checks, and documentation that search engines and AI assistants can read. When any of those surfaces is undocumented, the gateway turns into a black box.

For Tier 1 and Tier 2 engineering teams, the question is rarely "can this endpoint return a chat completion?" The more useful question is: "Can we explain what route was available, what rate card applied, what the gateway claimed publicly, and what our own run recorded at the time we made the decision?"

That is the job of a public evidence bundle.

This article describes a small bundle an AI gateway can publish and a small verifier a buyer or platform team can run before moving a workload. The pattern is not a substitute for security review, procurement, latency testing, or application-specific load tests. It is a way to make public claims inspectable instead of asking readers to trust a landing page.

I work on AIWave, an OpenAI-compatible route for supported Chinese model families. Before writing this article on 2026-09-08, I checked AIWave's public pricing endpoints. The live https://aiwave.live/api/pricing endpoint returned HTTP 200, success=true, 63 route records, and pricing version a42d372ccf0b5dd13ecf71203521f9d2. The public dated JSON at https://aiwave.live/api/v1/pricing returned HTTP 200, 63 model rows, 9 provider families, currency USD, unit per_1m_text_tokens, and checked date 2026-08-27.

Those are dated facts, not a promise that every future route, account, or invoice will look identical. That distinction is the point.

What belongs in the bundle

A public evidence bundle should answer four questions.

First, what can be called? This belongs in a model catalog or route catalog. It should include stable model IDs, provider family, supported endpoint shape, and any route-specific notes the user needs before sending traffic. If a route name is an alias, the catalog should say so or expose the resolved target in request records.

Second, what does it cost at the published point in time? This belongs in a dated pricing JSON file and a readable pricing page. The JSON should include effective date, currency, unit, input rate, cached-input rate when applicable, output rate, and source metadata. A team should be able to archive one row beside a workload estimate.

Third, is the public service currently reachable? This belongs in a status endpoint or status page. Early status data should be humble. A current probe is useful. A broad uptime claim requires a measured window, clear probe locations, and rules for partial incidents.

Fourth, how will a completed request be explained? This belongs in private account records, not in the public bundle. The public side can publish the schema and examples with synthetic values. Real records must avoid prompt text, response text, credentials, direct personal identifiers, and customer names.

The useful bundle is boring:

{
  "generated_at": "2026-09-08T13:20:00Z",
  "gateway": "example-gateway",
  "pricing": {
    "url": "https://example.com/api/v1/pricing",
    "checked": "2026-08-27",
    "version": "a42d372ccf0b5dd13ecf71203521f9d2",
    "currency": "USD",
    "unit": "per_1m_text_tokens"
  },
  "status": {
    "url": "https://example.com/status",
    "scope": "current public probe"
  },
  "catalog": {
    "url": "https://example.com/models",
    "id_policy": "pin exact model IDs in production"
  },
  "receipt_schema": {
    "visibility": "private account record",
    "public_example": "synthetic only"
  }
}

It should not contain a real reusable credential, customer account ID, prompt, output, payment identifier, or internal traffic metric.

Why this matters before the first production run

Most API evaluations start with a successful request. That is necessary, but it is not enough.

The first successful request only proves that one payload reached one route at one moment. It does not prove that the pricing row was the row your estimate used. It does not prove that a model alias will keep resolving the same way next month. It does not prove that cached input was counted separately. It does not prove that a 402, 429, timeout, retry, or provider fallback will leave enough evidence for a reviewer.

A public bundle narrows the gap between demo success and operational trust.

For a gateway, it also disciplines marketing language. If the public site says "dated pricing," there should be a dated source that machines can fetch. If it says "OpenAI-compatible," docs should show the endpoint shape and known boundaries. If it says "status," the page should clarify whether it is showing a current probe or a historical availability window. If it says "request-level billing evidence," the docs should explain what fields a user can inspect after a request.

The bundle does not need to reveal scale. In many early products, public scale claims create more risk than confidence. Capability evidence is cleaner: endpoint shape, provider family, rate source, status surface, receipt schema, and operational limits.

The pricing file

Treat pricing as source data, not prose.

A readable pricing page is useful for humans. A JSON file is useful for tests, procurement notes, calculators, and AI assistants that need exact fields. It should be possible to fetch the pricing file, select the model row, and store the relevant values beside an estimate.

Here is a minimal shape:

{
  "checked": "2026-08-27",
  "updated_at": "2026-08-27",
  "currency": "USD",
  "unit": "per_1m_text_tokens",
  "pricing_version": "a42d372ccf0b5dd13ecf71203521f9d2",
  "models": [
    {
      "id": "deepseek-v4-pro",
      "provider": "DeepSeek",
      "input_usd_per_1m_tokens": 1.914,
      "cache_hit_usd_per_1m_tokens": 0.0637362,
      "output_usd_per_1m_tokens": 5.742,
      "effective_date": "2026-08-27"
    }
  ]
}

The exact names are less important than the invariants:

The date is explicit.
The unit is explicit.
Cached input is separate from fresh input when the route exposes it.
Output is separate from input.
The model identifier is the value a client can actually send.
A version or hash lets a verifier detect a changed source.

Do not bury multipliers in prose. If account group, plan, region, time window, or route policy can change the applied rate, expose that fact in a structured field or in the private receipt that applies to the account. A public base rate and a private applied rate can both be true; the mistake is pretending they are the same field.

The status file

Status evidence should be scoped.

A gateway that has only started collecting public status history should say that. A current probe tells a developer whether the public surface is reachable right now. A measured uptime statement requires more: probe frequency, geographic scope, excluded maintenance, partial outage rules, and the time window.

A minimal public status payload can be plain:

{
  "checked_at": "2026-09-08T13:20:00Z",
  "status": "ok",
  "scope": "public edge probe",
  "components": [
    {
      "name": "pricing-json",
      "path": "/api/v1/pricing",
      "http_status": 200
    },
    {
      "name": "docs",
      "path": "/docs",
      "http_status": 200
    }
  ],
  "history": {
    "state": "collecting data"
  }
}

This is not exciting copy. It is useful copy. A buyer can archive it. A support engineer can compare it with an incident report. A crawler can discover it without rendering a JavaScript dashboard.

The receipt schema

The public bundle should publish a receipt schema, not real customer receipts.

A private receipt record should help a user reconcile a request after it completes. It should not expose the prompt. It should not expose the response. It should not leak a reusable credential. It should not turn a support export into a customer dossier.

A synthetic example can show the contract:

{
  "request_id": "req_demo_20260908_001",
  "created_at": "2026-09-08T13:20:00Z",
  "requested_model": "deepseek-v4-pro",
  "resolved_model": "deepseek-v4-pro",
  "endpoint_shape": "openai.chat.completions",
  "pricing_source": "https://example.com/api/v1/pricing",
  "pricing_version": "a42d372ccf0b5dd13ecf71203521f9d2",
  "pricing_date": "2026-08-27",
  "account_rate_scope": "private account setting",
  "usage": {
    "fresh_input_tokens": 120000,
    "cached_input_tokens": 80000,
    "output_tokens": 6000
  },
  "charge": {
    "currency": "USD",
    "unit": "request",
    "estimated": false
  },
  "privacy": {
    "prompt_text_included": false,
    "response_text_included": false,
    "credential_included": false
  }
}

The receipt should be built for disputes and regressions. If the same workflow suddenly costs more, the team can inspect whether fresh input increased, cached input fell, the output grew, the model changed, the pricing version changed, or retries multiplied. Without those fields, the only available answer is a vague one.

A verifier script

A buyer does not need access to a gateway's private database to verify public evidence. A small script can fetch public sources, check required fields, and produce a local admission report.

This Python script uses only public URLs and stores no credential:

import datetime as dt
import json
import urllib.request


PRICING_URL = "https://example.com/api/v1/pricing"
STATUS_URL = "https://example.com/status"
MODEL_ID = "deepseek-v4-pro"


def fetch_json(url: str) -> dict:
    req = urllib.request.Request(url, headers={"User-Agent": "evidence-check/1.0"})
    with urllib.request.urlopen(req, timeout=20) as response:
        if response.status != 200:
            raise RuntimeError(f"{url} returned HTTP {response.status}")
        return json.loads(response.read().decode("utf-8"))


def require_fields(obj: dict, fields: list[str], label: str) -> None:
    missing = [field for field in fields if field not in obj]
    if missing:
        raise RuntimeError(f"{label} missing fields: {', '.join(missing)}")


pricing = fetch_json(PRICING_URL)
require_fields(pricing, ["checked", "currency", "unit", "models"], "pricing")

rows = [row for row in pricing["models"] if row.get("id") == MODEL_ID]
if not rows:
    raise RuntimeError(f"model row not found: {MODEL_ID}")

row = rows[0]
require_fields(
    row,
    ["provider", "input_usd_per_1m_tokens", "output_usd_per_1m_tokens"],
    "model row",
)

status = fetch_json(STATUS_URL)
require_fields(status, ["checked_at", "status"], "status")

report = {
    "checked_at": dt.datetime.now(dt.UTC).isoformat(),
    "pricing_url": PRICING_URL,
    "status_url": STATUS_URL,
    "model_id": MODEL_ID,
    "pricing_checked": pricing["checked"],
    "pricing_currency": pricing["currency"],
    "pricing_unit": pricing["unit"],
    "provider": row["provider"],
    "status": status["status"],
}

print(json.dumps(report, indent=2))

In a production evaluation, write that report to your repository or procurement folder. Keep it beside the workload estimate, the test payload class, the selected model ID, and the rollback condition. Re-run it before a high-volume job, a route migration, or a contract review.

Admission checks for your own gateway

If you operate the gateway, add a CI job that validates the bundle before each public release.

The checks do not need to be complex:

Pricing JSON returns HTTP 200.
Pricing JSON parses as JSON.
Every model row has an ID, provider, currency context, effective date, input rate, and output rate.
Cached-input fields are either numeric or explicitly null.
The pricing page links to the JSON source.
The docs link to the model catalog and pricing page.
The status page distinguishes current probes from historical availability.
The public examples contain no reusable credentials.
The public pages contain no withdrawn scale or revenue claims.
The RSS feed or update log exposes recent changes for readers that do not visit the site manually.

That last point matters more than it looks. A pricing file without an update trail can still be inspected, but it is hard to notice change. A feed, changelog, or llms.txt style index gives readers and AI assistants a stable path into the current public evidence.

What not to publish

Do not publish raw logs as proof.

A raw log can contain fields that were never meant for public distribution: prompt fragments, user identifiers, payment metadata, network details, internal route names, or provider error bodies. Even when a log is technically harmless, it can teach outsiders how your system is operated.

Publish schemas, synthetic examples, aggregate capability facts, and current public endpoints. Keep account-specific evidence inside authenticated views. If you need to support a customer dispute, export the smallest private record that answers the dispute and redact everything else.

Also avoid stale certainty. A dated rate card is useful because it admits it has a date. A status probe is useful because it admits its scope. A model catalog is useful because it can be rechecked. The bundle should make change visible, not pretend change will stop.

A useful review packet

When a team evaluates a gateway, the review packet can be short:

Gateway:
Date checked:
Workload owner:
Selected model ID:
Pricing source URL:
Pricing checked date:
Pricing version:
Input / cached-input / output fields present:
Status source URL:
Status scope:
Docs URL:
Model catalog URL:
Receipt fields available after completion:
Rollback condition:
Unknowns before production:

The packet should be written before the workload moves. After the workload runs, attach the private receipt or billing export with sensitive fields removed. If the estimate and completed request do not match, compare fields rather than arguing from memory.

Applying the pattern to AIWave

For AIWave, the public pieces a developer should check are the model catalog, the pricing page, the pricing JSON, the docs, the status page, the feed, and the current site index for AI assistants. The private piece is the account-side request record after a call completes.

The relationship between those pieces is the important part:

The catalog tells you the route name to send.
The pricing JSON gives you a dated source for estimates.
The docs show the OpenAI-compatible request shape.
The status page gives the current public probe scope.
The request record lets you reconcile the completed run privately.

No single page proves production readiness. Together, they reduce ambiguity before the first larger run.

If you are building a gateway, this is the standard worth aiming for: a developer should be able to fetch your public evidence, pin the exact sources used in an estimate, run a representative request, and explain the completed charge without revealing private workload content.

That is less glamorous than a benchmark chart. It is also more useful when an engineering lead, finance reviewer, or support engineer asks what actually happened.

Build a cache-hit regression harness for long-context AI gateways

Mattias chaw — Mon, 07 Sep 2026 13:13:07 +0000

Build a cache-hit regression harness for long-context AI gateways

Long-context AI workloads often look stable until a small routing or prompt change breaks the cache shape.

The application still works. The model still answers. The request log still shows familiar model names. But the bill attached to the same workflow can move because repeated input that used to land in a cached bucket is now charged as fresh input. A support engineer may see no obvious error. A finance reviewer may see a spend jump. A platform engineer may only see that one prompt template, one system prefix, or one model route changed last week.

That is why cache behavior should be tested as a production contract.

This article describes a cache-hit regression harness for teams running long-context workloads through OpenAI-compatible AI gateways. The pattern is useful when a product uses large reusable context, model routing, prompt templates, retrieval packs, or provider fallback. It is written for Tier 1 and Tier 2 engineering teams that need cost evidence without exposing prompts, private documents, API keys, payment identifiers, or customer names.

AIWave is built around one OpenAI-compatible route to Chinese model families such as DeepSeek, GLM, Kimi, Qwen, ERNIE, MiniMAX, Doubao, StepFun, and MiMo. The same client can call different model routes while the platform keeps dated pricing and request-level billing evidence visible. That makes cache regression testing valuable: the integration can stay stable while the economic behavior behind a long-context run still changes.

The bug is not always a failure

Most API test suites are built around functional output.

They check that a request returns 200. They check JSON shape. They check that a response contains a useful answer. They may check latency and retry count. Those tests are necessary, but they do not catch a common long-context regression: the workload keeps succeeding while its cached-input ratio collapses.

This can happen after ordinary engineering changes:

A system prompt gains a new timestamp at the top.
A retrieval pack changes document order.
A route switches from one model variant to another.
A fallback path omits the stable prefix.
A middleware layer injects request-specific metadata into the reusable context.
A streaming adapter retries with a rebuilt message array.
A prompt compression step rewrites the part that used to stay constant.

None of those changes has to be wrong. Some are useful. The problem is that the team may not notice the cost behavior changed until later.

A cache-hit regression harness treats the cached-input ratio as a testable property. It does not need raw prompt text. It needs deterministic fingerprints, token bucket counts, model route identifiers, pricing source metadata, and acceptance thresholds.

Pin the pricing evidence

Before writing this article, I checked AIWave's public pricing source on 2026-09-07 at 13:10 UTC. The live endpoint https://aiwave.live/api/pricing reconciled against the public pricing page with 63 route records, 63 matched HTML rows, pricing version a42d372ccf0b5dd13ecf71203521f9d2, and group ratios of default=3 and vip=1.

The same check showed coverage across 9 provider families:

Provider family	Route records
DeepSeek	3
GLM	7
Kimi	12
ERNIE	2
MiniMax	8
Qwen	23
Doubao	4
StepFun	3
MiMo	1

These are dated planning facts. They are not a guarantee about future pricing, every account, every upstream route, or every invoice. That is the point of pinning pricing evidence in the harness. A regression result should say which rate source, checked time, pricing version, account group, and resolved model were used when the test ran.

Define the cache contract

Start with a small contract for each long-context workflow.

The contract should define what is expected to stay stable, what is allowed to change, and which token buckets matter. It should be reviewed by the team that owns the workflow, not only by the platform team.

{
  "contract_id": "support_summary_cache_2026_09",
  "workflow": "support_summary",
  "owner": "platform-ai",
  "client_surface": "openai-compatible-chat",
  "requested_model": "deepseek-v4-pro",
  "resolved_model_family": "DeepSeek",
  "pricing_source": "https://aiwave.live/api/pricing",
  "pricing_version": "a42d372ccf0b5dd13ecf71203521f9d2",
  "effective_group": "vip",
  "stable_segments": [
    "system_policy",
    "tool_schema",
    "knowledge_pack"
  ],
  "variable_segments": [
    "user_question",
    "conversation_tail"
  ],
  "minimum_cached_input_ratio": 0.70,
  "maximum_output_tokens": 1800,
  "maximum_retry_count": 1
}

The exact threshold depends on the workload. A small chat request may have no meaningful reusable context. A coding assistant with a large repository summary may expect most of its prefix to remain stable across a short evaluation run. A retrieval workflow may expect stability only inside a fixed evaluation fixture.

Do not set a universal threshold. Set a threshold per workflow and record why it is reasonable.

Fingerprint segments, not secrets

The harness should never export raw prompts or private source documents to a public report. It should fingerprint segments.

A segment fingerprint is enough to answer the operational question: did the reusable part stay the same between the baseline and the candidate run?

import hashlib
import json


def stable_json(value):
    return json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":"))


def fingerprint_segment(segment):
    encoded = stable_json(segment).encode("utf-8")
    return hashlib.sha256(encoded).hexdigest()


def build_segment_manifest(messages, tool_schema, knowledge_pack_id):
    return {
        "system_policy": fingerprint_segment(messages[0]),
        "tool_schema": fingerprint_segment(tool_schema),
        "knowledge_pack": fingerprint_segment({"id": knowledge_pack_id}),
        "message_count": len(messages)
    }

The fingerprint should be deterministic. That means sorting JSON keys, stripping fields that are intentionally request-specific, and keeping timestamps out of stable segments unless the timestamp is part of the contract.

When a regression happens, the manifest helps separate two cases:

The cache ratio fell because a stable segment changed.
The cache ratio fell even though the stable segment fingerprints match.

The first case is usually a prompt or packaging change. The second case may point to route behavior, provider behavior, retry behavior, cache TTL, or an adapter bug.

Capture token buckets per run

A useful harness captures token buckets at the same boundary as the workload.

For long-context work, a single workflow may include planning, retrieval, drafting, validation, and summarization. Testing one request in isolation can hide the real regression. The run-level record should preserve per-step detail and aggregate totals.

type CacheRegressionRun = {
  run_id: string
  contract_id: string
  checked_at: string
  requested_model: string
  resolved_model: string
  provider_family: string
  pricing_version: string
  effective_group: string
  segment_fingerprints: Record<string, string>
  steps: Array<{
    step_name: string
    input_tokens: number
    cached_input_tokens: number
    output_tokens: number
    retry_count: number
    fallback_used: boolean
  }>
  totals: {
    input_tokens: number
    cached_input_tokens: number
    output_tokens: number
    retry_count: number
  }
  verdict: "pass" | "review" | "fail"
}

The fields are intentionally operational. They let a reviewer answer the questions that matter:

Did the model route change?
Did the account or token group change?
Did the reusable prompt material change?
Did retries or fallbacks change the token shape?
Did cached input fall below the accepted threshold?

This record can live in CI artifacts, an internal dashboard, or a billing evidence store. It should not require prompt access to be useful.

Compare baseline and candidate

The harness needs a stable baseline. That baseline should be refreshed deliberately, not silently overwritten on every run.

Use a candidate run to compare against the approved baseline:

def cached_ratio(totals):
    input_tokens = totals["input_tokens"]
    cached = totals["cached_input_tokens"]
    if input_tokens <= 0:
        return 0.0
    return cached / input_tokens


def compare_cache_contract(contract, baseline, candidate):
    issues = []

    if candidate["pricing_version"] != contract["pricing_version"]:
        issues.append({
            "severity": "review",
            "code": "pricing_version_changed",
            "message": "The candidate used a different dated pricing source."
        })

    if candidate["resolved_model"] != baseline["resolved_model"]:
        issues.append({
            "severity": "review",
            "code": "resolved_model_changed",
            "message": "The candidate resolved to a different model route."
        })

    stable_segments = contract["stable_segments"]
    for name in stable_segments:
        if candidate["segment_fingerprints"].get(name) != baseline["segment_fingerprints"].get(name):
            issues.append({
                "severity": "fail",
                "code": "stable_segment_changed",
                "segment": name,
                "message": "A segment marked stable changed between runs."
            })

    ratio = cached_ratio(candidate["totals"])
    if ratio < contract["minimum_cached_input_ratio"]:
        issues.append({
            "severity": "fail",
            "code": "cached_input_ratio_below_contract",
            "observed": round(ratio, 4),
            "minimum": contract["minimum_cached_input_ratio"]
        })

    if candidate["totals"]["retry_count"] > contract["maximum_retry_count"]:
        issues.append({
            "severity": "review",
            "code": "retry_count_above_contract",
            "observed": candidate["totals"]["retry_count"]
        })

    return issues

The result should be boring. A pass means the workflow kept its route, stable segments, pricing source, retry envelope, and cached-input shape within contract. A review means something changed and a human should decide whether to bless a new baseline. A fail means the candidate should not be promoted until the owner explains the change.

Add a CI gate without blocking every experiment

The harness should not make experimentation painful. It should protect production promotion.

A practical setup has three modes:

Mode	When it runs	What it can do
`observe`	Local development and exploratory branches	Write evidence, never block
`review`	Pull requests that touch prompts, routing, or adapters	Comment with diffs and ask for owner review
`enforce`	Release branches and production route changes	Block when hard thresholds fail

This keeps the gate proportional. A developer can experiment with prompt structure without fighting CI on every edit. The release path still requires evidence before a long-context workload changes its cost shape.

The trigger list should include more than prompt files:

Prompt templates
Retrieval packaging
Tool schemas
Model route configuration
Fallback policy
Streaming adapters
Retry middleware
Pricing source readers
Ledger writers
Token accounting code

Cache regressions often enter through infrastructure files, not just prompt files.

Make the report useful to finance and support

The report should not be a wall of traces. It should answer a short set of questions.

{
  "contract_id": "support_summary_cache_2026_09",
  "candidate_run_id": "ci_9482",
  "verdict": "fail",
  "checked_at": "2026-09-07T13:10:10Z",
  "pricing_version": "a42d372ccf0b5dd13ecf71203521f9d2",
  "resolved_model": "deepseek-v4-pro",
  "cached_input_ratio": 0.42,
  "minimum_cached_input_ratio": 0.70,
  "changed_segments": [
    "system_policy"
  ],
  "retry_count": 0,
  "fallback_used": false,
  "next_action": "Review prompt package change before promotion."
}

A support engineer can see that the request did not fail, but the reusable prefix changed. A finance reviewer can see that the issue is token-bucket shape, not only volume. A platform engineer can see which contract needs a new baseline or a fix.

Keep the report shareable inside the company. That means no raw prompts, no private documents, no real API keys, no customer names, and no payment identifiers.

Decide when to bless a new baseline

A regression is not always a bug. Sometimes the team intentionally changes the reusable context to improve answer quality, add safety policy, or support a new tool. In that case, the right action is to bless a new baseline.

Require a short baseline change record:

{
  "baseline_change_id": "cache_baseline_2026_09_07_support_summary",
  "contract_id": "support_summary_cache_2026_09",
  "reason": "Added structured refusal policy to the stable system segment.",
  "expected_effect": "Cached-input ratio may reset during rollout, then recover on repeated calls.",
  "approved_by": "platform-ai-owner",
  "pricing_version": "a42d372ccf0b5dd13ecf71203521f9d2",
  "rollback_signal": "Cached-input ratio remains below contract after the rollout window."
}

The important part is not the exact JSON. The important part is that the baseline change is explicit. Silent baseline replacement turns the harness into a dashboard that always agrees with the last run.

Common mistakes

The first mistake is testing only one happy-path request. Long-context costs often emerge across a run, not in a single call.

The second mistake is hashing the whole prompt as one blob. That makes every small variable field look like a total cache break. Segment the material into stable and variable regions.

The third mistake is ignoring the resolved model. If a request asks for one model alias but resolves to another model route, the cache and pricing behavior may change even when the client code did not.

The fourth mistake is using a pricing table without a checked time and version. A cost regression report without dated pricing evidence is hard to review later.

The fifth mistake is exposing too much evidence. The harness should prove operational behavior, not export secrets or customer material.

What to build first

Start with one long-context workflow that already has repeated context. Add segment fingerprints, run-level token bucket capture, and a single cached-input threshold. Store the pricing source URL, checked time, pricing version, effective group, requested model, and resolved model beside the run.

Run the harness in observe mode for a week. Do not block releases immediately. Use that period to learn the normal range of cached-input ratios, retry counts, and route behavior. Then choose which threshold belongs in review mode and which threshold belongs in enforce mode.

For an OpenAI-compatible gateway, this is the practical goal: a route can change, a prompt can evolve, and a pricing source can be refreshed, but the team can still explain whether the long-context economics stayed inside contract.

Cache hits are not only a provider feature. For production teams, they are an evidence boundary. Treat them like one.

Build an entitlement drift monitor for AI API gateways

Mattias chaw — Sun, 06 Sep 2026 13:11:00 +0000

Build an entitlement drift monitor for AI API gateways

Multi-provider AI gateways usually start with a simple promise: one client, one key, many model routes. That promise is useful, but it creates a quiet operational problem. Account state, token state, model-route state, and pricing state can drift away from each other.

The drift is rarely dramatic at first. A customer completes a payment, but one legacy token still uses the previous route group. A rate card is refreshed, but one worker keeps an older snapshot in memory. A model alias is retired, but a background job still sends traffic to it. A request ledger records the model name and token counts, but not the effective entitlement that decided the route.

Each issue looks small in isolation. Together they produce the kind of support ticket that is hard to answer: "Why did this request behave differently from the same request yesterday?"

The fix is not to trust every subsystem more. The fix is to make entitlement state observable, versioned, and easy to reconcile.

This article describes a practical entitlement drift monitor for AI API gateways. It is written for teams running production workloads across multiple model providers, especially where billing groups, model availability, and route policy can change independently.

What an entitlement means

In a gateway, an entitlement is the permission and billing context that makes a request valid.

It can include:

Account status
Billing group
Token group
Allowed model families
Allowed model slugs
Request-level route policy
Pricing snapshot version
Ledger policy version

That list is intentionally broader than "paid or not paid." Production access is not a boolean. A team may have access to one model family but not another. A service token may be allowed to call batch routes but not interactive routes. A billing group may use a different multiplier from a default trial group. A region or compliance setting may affect which upstream route is acceptable.

If those fields are spread across several tables, caches, workers, and dashboards, drift becomes likely. The gateway still works, but nobody can explain it under pressure.

The drift classes that matter

An entitlement monitor should look for a small set of high-value drift classes.

Account-to-token drift happens when the account has one billing or access group, but one or more tokens attached to that account carry a different group. This is common after upgrades, migrations, manual support changes, or old token imports.

Token-to-route drift happens when a token is allowed to use a route group that no longer matches the current catalog. The request may still succeed if a fallback exists, but the ledger becomes harder to interpret.

Route-to-pricing drift happens when the route table and the pricing table disagree. The model may exist in one place and be absent in another, or the provider label may differ. This is dangerous because cost forecasts are built from the pricing side while traffic flows through the route side.

Worker-to-snapshot drift happens when a long-running process keeps an older in-memory copy after a catalog or pricing change. The API may return the new rate card while the worker still decides from yesterday's map.

Ledger-to-decision drift happens when the request ledger records usage but omits the entitlement that made the request possible. That makes later diagnosis dependent on reconstructing old state from mutable tables.

The monitor does not need to solve all of these in the first version. It should detect them, label their severity, and produce a repair queue that humans and automation can trust.

The minimum schema

Start with a normalized snapshot, not with a dashboard. A dashboard can only be useful if the underlying record is boring and complete.

type EntitlementSnapshot = {
  checked_at: string
  account_id_hash: string
  account_group: string
  token_id_hash: string
  token_group: string | null
  token_status: "active" | "disabled" | "expired"
  allowed_model_families: string[]
  allowed_route_group: string
  pricing_version: string
  catalog_version: string
  ledger_policy_version: string
}

Hash internal identifiers before they leave the secure environment. The monitor does not need raw customer identifiers to catch drift. It needs stable correlation keys, group labels, timestamps, and versions.

For request-level evidence, add a smaller receipt:

type EntitlementReceipt = {
  request_id: string
  timestamp: string
  model: string
  account_group: string
  token_group: string
  route_group: string
  pricing_version: string
  catalog_version: string
  input_tokens: number
  cached_input_tokens: number
  output_tokens: number
  retry_count: number
  status_code: number
}

Do not log prompt bodies in this monitor. Entitlement drift is about control-plane state, not content inspection.

Reconcile before repair

The first job should be read-only. It should compare current account, token, catalog, route, and pricing state without changing production.

The reconciliation output can be compact:

{
  "checked_at": "2026-09-06T13:07:52Z",
  "pricing_version": "a42d372ccf0b5dd13ecf71203521f9d2",
  "accounts_checked": "computed_at_runtime",
  "tokens_checked": "computed_at_runtime",
  "issues": [
    {
      "type": "account_token_group_mismatch",
      "severity": "repairable",
      "account_id_hash": "acct_7fd9",
      "token_id_hash": "tok_91aa",
      "account_group": "vip",
      "token_group": "default"
    }
  ]
}

The count fields above are intentionally placeholders. Replace them with your own query results, and keep the run date visible.

The important rule is sequencing: reconcile before repair. A repair script that cannot explain what it would change is a liability. A read-only reconcile report gives support, finance, and engineering the same artifact before any write occurs.

Use a deterministic issue taxonomy

Avoid generic "bad state" alerts. Give each drift class a stable type and a narrow owner.

Recommended first-pass types:

type EntitlementIssueType =
  | "account_token_group_mismatch"
  | "active_token_missing_group"
  | "disabled_token_has_route_group"
  | "route_missing_pricing_row"
  | "pricing_missing_route"
  | "worker_snapshot_stale"
  | "ledger_missing_entitlement_fields"
  | "unknown_model_family"

Each issue type should carry:

Severity
Detection query
Suggested owner
Safe repair path
Rollback note
Suppression rule

Suppression rules are important. Without them, the monitor becomes background noise. A disabled token with a stale group may be a cleanup task, while an active production token in the wrong group is a customer-facing risk.

Keep pricing and catalog versions visible

Pricing drift is especially sensitive in AI gateways because cost depends on model name, input tokens, cached input tokens, output tokens, retry behavior, and group policy.

Before writing this article, I checked AIWave's live pricing API against the generated pricing page. The check passed with 63 matched model rows across 9 provider families, no duplicate model names, no missing providers, group ratios of default=3 and vip=1, and pricing version a42d372ccf0b5dd13ecf71203521f9d2.

That kind of statement is useful because it is dated and reproducible. It does not say that a price will remain unchanged. It says which version was checked, what was compared, and what would fail the gate.

For your own gateway, store these fields beside every request:

{
  "model": "deepseek-v4-pro",
  "pricing_version": "ratecard_2026_09_06_1307z",
  "catalog_version": "catalog_2026_09_06_1307z",
  "account_group": "vip",
  "token_group": "vip",
  "route_group": "vip"
}

When the rate card changes, old receipts remain explainable. When a worker is stale, new receipts expose the mismatch.

Design the monitor as a gate

The monitor should run in three places.

First, run it before a route catalog deploy. If a model exists in the route map but not the pricing table, fail the deploy. If a provider label is unknown, fail the deploy. If the effective billing group cannot be computed for a request fixture, fail the deploy.

Second, run it on a schedule against production state. This catches manual fixes, delayed token upgrades, and cache problems that do not appear in CI.

Third, run it as part of support diagnostics. When a customer reports a billing or access issue, the support artifact should include the entitlement snapshot and request receipt for the affected time window.

The same detector can serve all three contexts if it is read-only by default.

A simple detector

Here is a simplified detector shape:

type Row = {
  accountIdHash: string
  accountGroup: string
  tokenIdHash: string
  tokenGroup: string | null
  tokenStatus: "active" | "disabled" | "expired"
}

function detectAccountTokenDrift(rows: Row[]) {
  return rows.flatMap((row) => {
    if (row.tokenStatus !== "active") return []

    if (!row.tokenGroup) {
      return [{
        type: "active_token_missing_group",
        severity: "repairable",
        accountIdHash: row.accountIdHash,
        tokenIdHash: row.tokenIdHash,
        expected: row.accountGroup,
        actual: null
      }]
    }

    if (row.accountGroup !== row.tokenGroup) {
      return [{
        type: "account_token_group_mismatch",
        severity: "repairable",
        accountIdHash: row.accountIdHash,
        tokenIdHash: row.tokenIdHash,
        expected: row.accountGroup,
        actual: row.tokenGroup
      }]
    }

    return []
  })
}

In production, the expected value may be more complex than accountGroup. You may have service-token overrides, enterprise contracts, per-project route policies, or temporary migration states. Put that logic in one resolver and test it with fixtures.

Separate detection from mutation

The repair path should be a second step. A good pattern is:

Read-only scan
Human-readable report
Machine-readable issue file
Dry-run repair plan
Approved repair
Post-repair scan

The repair plan should be explicit:

{
  "repair_id": "entitlement_repair_2026_09_06_01",
  "precheck_report": "entitlement_scan_2026_09_06_1307z.json",
  "actions": [
    {
      "type": "set_token_group",
      "token_id_hash": "tok_91aa",
      "from": "default",
      "to": "vip",
      "reason": "account_token_group_mismatch"
    }
  ],
  "rollback": "restore token group values from precheck snapshot"
}

Do not let the repair script rediscover the world independently. Feed it the reviewed issue file. That keeps the detection boundary and the mutation boundary clear.

Put stale workers on the same board

Many AI gateway bugs look like database bugs but are actually worker-snapshot bugs. A deployment updated the rate card, but a queue worker still has the old catalog. A cache refresh failed in one region. A background evaluator still uses an old model alias.

Expose a worker heartbeat:

{
  "worker": "gateway-dispatch-3",
  "seen_at": "2026-09-06T13:08:12Z",
  "catalog_version": "catalog_2026_09_06_1307z",
  "pricing_version": "ratecard_2026_09_06_1307z",
  "ledger_policy_version": "ledger_v4"
}

Then compare it to the canonical versions. If a worker is stale for more than one refresh interval, alert the on-call owner. If a stale worker also processes requests, escalate severity.

This is more useful than a generic "worker alive" check. A worker can be alive and still make old routing decisions.

Make the support answer boring

The target support answer should be concise:

"For request req_abc, the account group, token group, and route group were all vip. The request used pricing version ratecard_2026_09_06_1307z and catalog version catalog_2026_09_06_1307z. The ledger recorded input, cached input, output, retry count, and status. No entitlement drift was detected for the token in the surrounding scan."

Or:

"The account was upgraded at 10:03 UTC, but token tok_91aa still carried the previous route group until the 10:08 repair run. Requests before 10:08 used the previous entitlement; requests after 10:08 used the upgraded entitlement. The repair scan closed the mismatch."

Neither answer needs drama. It needs evidence.

Watch the words you use publicly

If your gateway serves developers in Tier 1 and Tier 2 markets, avoid claims that sound attractive but cannot survive procurement review.

Do not lead with broad price-superiority claims. Do not publish customer scale or workload details unless you have explicit approval and a clean privacy review. Do not imply an SLA unless you have the contract and measurement system to support it.

Use claims that can be checked:

One OpenAI-compatible endpoint
One key across supported routes
Dated rate card
Per-request ledger
Explicit provider families
Stated retention boundary
Visible support path

An entitlement monitor supports that posture because it makes access and billing behavior inspectable.

Operational checklist

For a first implementation, I would build the monitor in this order:

Export account-token entitlement snapshots with hashed identifiers.
Export the current route catalog and pricing snapshot with versions.
Add entitlement fields to request receipts.
Detect account-token mismatches for active tokens.
Detect route-pricing mismatches.
Add worker heartbeats with catalog and pricing versions.
Produce a dry-run repair plan.
Require post-repair scans before closing an issue.

Keep the first version narrow. The best monitor is the one that catches the painful drift classes every day without becoming another dashboard people ignore.

Final thought

Multi-provider AI gateways are control-plane products as much as model-access products. The customer does not only need a request to complete. They need the route, bill, entitlement, and support answer to line up afterward.

An entitlement drift monitor gives you that alignment. It turns "why did this happen?" into a dated snapshot, a request receipt, and a small set of repairable states.

That is the kind of infrastructure work that makes a gateway credible when workloads, model catalogs, and pricing tables keep changing.

Build a route change control gate for AI API gateways

Mattias chaw — Sat, 05 Sep 2026 13:11:31 +0000

Build a route change control gate for AI API gateways

Changing an AI model route is easy when the client speaks an OpenAI-compatible API. Changing it responsibly is harder.

A platform team may want to move one workflow from a general chat model to a stronger reasoning model. A product team may want to add a fallback route for customer-visible failures. Finance may ask why a workload that looked stable last week now burns through a different mix of input, cached input, and output tokens. Support may need to explain which route served a specific run without exposing prompts or customer identifiers.

The route switch itself may be one line of configuration. The control plane around that switch should be more deliberate.

AIWave is built around one OpenAI-compatible route to Chinese model families such as DeepSeek, GLM, Kimi, Qwen, ERNIE, MiniMAX, Doubao, StepFun, and MiMo. That kind of gateway is useful because a team can keep client code stable while choosing different model routes behind the same interface. For Tier 1 and Tier 2 teams, the useful question is not only "can the route change?" It is "can we prove what changed, why it changed, and what evidence should be reviewed before traffic moves?"

This article gives a practical route change control gate for multi-model AI API gateways.

Treat route changes as releases

A route change is a production release, even when no application code changes.

The model behind a route can affect latency, output length, tool behavior, retry pressure, cache-hit shape, and cost. The fallback policy can affect which users see degraded output versus a stronger model. The account group or pricing snapshot can affect the invoice attached to the same token buckets.

Put those changes through a small release gate:

Gate	Question	Evidence
Intent	Why is this route changing?	Owner, feature, user impact
Pricing	Which rate snapshot applies?	Source URL, checked time, version
Behavior	Does the workflow still pass?	Fixtures, sample runs, validation result
Budget	What usage shape is expected?	Token buckets, output caps, retry limits
Privacy	What evidence can be shared?	No raw prompts, keys, payment ids, or customer ids
Rollback	How does traffic return?	Prior route, config diff, stop trigger

The point is not ceremony. The point is to make a route change reviewable by someone who was not in the Slack thread where the switch was discussed.

Pin the pricing evidence first

Before writing this article, I checked AIWave's public pricing source on 2026-09-05 at 13:07 UTC. The live endpoint https://aiwave.live/api/pricing reconciled against the public pricing page with 63 route records, pricing version a42d372ccf0b5dd13ecf71203521f9d2, and group ratios of default=3 and vip=1.

The same reconciliation showed coverage across 9 provider families:

Provider family	Route records
DeepSeek	3
GLM	7
Kimi	12
ERNIE	2
MiniMax	8
Qwen	23
Doubao	4
StepFun	3
MiMo	1

These are dated planning facts. They are not a guarantee about future pricing, every account, every upstream route, or every production invoice. That is exactly why a route gate should store the source URL, checked time, pricing version, effective account group, and resolved model beside the change.

Define the route change record

Start with a small record that can be reviewed before the change reaches production.

{
  "change_id": "route_change_2026_09_05_support_summary",
  "change_type": "model_route_update",
  "created_at": "2026-09-05T13:07:34Z",
  "owner": "platform-ai",
  "feature": "support_summary",
  "environment": "production",
  "current_route": {
    "requested_model": "deepseek-v4-flash",
    "resolved_model": "deepseek-v4-flash",
    "policy_version": "support-summary-routing-2026-08-31"
  },
  "proposed_route": {
    "requested_model": "deepseek-v4-pro",
    "resolved_model": "deepseek-v4-pro",
    "policy_version": "support-summary-routing-2026-09-05"
  },
  "pricing": {
    "source_url": "https://aiwave.live/api/pricing",
    "checked_at": "2026-09-05T13:07:34Z",
    "pricing_version": "a42d372ccf0b5dd13ecf71203521f9d2",
    "effective_group": "vip"
  },
  "approval": {
    "status": "pending",
    "required_reviewers": ["platform", "finance", "support"]
  }
}

This record should not include raw prompts, completions, API keys, customer emails, payment identifiers, or private files. It should include the operational facts needed to review the change.

Separate requested and resolved model

Always record both requested_model and resolved_model.

The requested model tells you what the client or policy asked for. The resolved model tells you what the gateway actually used after aliases, fallback policy, availability rules, or routing configuration were applied.

That distinction matters during route changes:

Scenario	Requested model	Resolved model	Review question
Direct route	`deepseek-v4-flash`	`deepseek-v4-flash`	Did the direct route still pass fixtures?
Alias update	`support-fast`	`deepseek-v4-flash`	Did the alias target change?
Reasoning upgrade	`support-review`	`deepseek-v4-pro`	Did output length and spend change?
Fallback event	`deepseek-v4-flash`	`glm-5.1`	Why did fallback trigger?

If a gateway only stores the requested model, the team may believe the route stayed stable while the resolved model changed underneath.

Add an admission check

The route gate should fail closed when required evidence is missing.

from dataclasses import dataclass
from typing import Iterable


@dataclass(frozen=True)
class RouteChange:
    change_id: str
    current_resolved_model: str
    proposed_resolved_model: str
    pricing_source_url: str
    pricing_checked_at: str
    pricing_version: str
    effective_group: str
    fixture_result: str
    rollback_route: str
    privacy_scan_passed: bool
    owner: str


def route_change_errors(change: RouteChange, allowed_models: Iterable[str]) -> list[str]:
    errors: list[str] = []
    allowed = set(allowed_models)

    if change.proposed_resolved_model not in allowed:
        errors.append("proposed_resolved_model is not in the reviewed model catalog")
    if not change.pricing_source_url:
        errors.append("missing pricing source URL")
    if not change.pricing_checked_at:
        errors.append("missing pricing checked timestamp")
    if not change.pricing_version:
        errors.append("missing pricing version")
    if change.effective_group not in {"default", "vip", "svip"}:
        errors.append("unknown effective group")
    if change.fixture_result != "passed":
        errors.append("route fixtures did not pass")
    if not change.rollback_route:
        errors.append("missing rollback route")
    if not change.privacy_scan_passed:
        errors.append("privacy scan did not pass")
    if not change.owner:
        errors.append("missing change owner")

    return errors

The allowed model list should come from the current reviewed catalog, not from a hard-coded comment in application code. The pricing source should come from a live or stored snapshot that the reviewer can open later.

Test behavior with fixed fixtures

Route changes should be tested with fixed fixtures before traffic moves.

Use fixtures that represent the workflow, not generic one-line prompts. A support summary route should test long ticket context, missing customer metadata, structured output, and a human-review label. A coding agent route should test patch generation, tool output, retry behavior, and a failing test result. A document extraction route should test long input, repeated context, and schema validation.

For each fixture, store:

Field	Example
`fixture_id`	`support-summary-long-ticket-v3`
`input_shape`	`long_context_with_repeated_policy`
`expected_contract`	`json_schema_support_summary_v2`
`current_route_result`	`passed`
`proposed_route_result`	`passed`
`known_difference`	`longer rationale field`
`review_status`	`accepted`

Do not store private prompt text in the route gate. Store fixture ids, hashes, output contracts, and review results. If an internal team needs the fixture body, keep it in a separate test repository with access controls.

Budget against token shape

Route changes are rarely explained by input tokens alone.

A stronger route may produce longer output. A route with better instruction following may reduce retries. A long-context route may improve cache reuse if the prompt prefix is stable. A fallback route may only serve a small number of requests but still dominate output spend.

Make the route gate compare token shape:

select
  resolved_model,
  count(*) as requests,
  sum(input_tokens) as input_tokens,
  sum(cached_input_tokens) as cached_input_tokens,
  sum(output_tokens) as output_tokens,
  sum(retry_count) as retries,
  count(fallback_from) as fallback_events
from gateway_request_ledger
where feature = :feature
  and created_at >= :baseline_start
  and created_at < :baseline_end
group by resolved_model
order by output_tokens desc;

The review should produce a simple expected shape for the proposed route:

Metric	Baseline	Proposed guardrail
Output tokens per request	900	1,200 max before review
Cached input share	60%	50% minimum before review
Retries per 100 requests	4	6 max before review
Fallback events per 100 requests	1	3 max before review

These are example guardrails, not universal thresholds. The important part is that the gate names the metrics before the rollout.

Keep rollback boring

Rollback should be a first-class field in the route change record.

{
  "rollback": {
    "route": "deepseek-v4-flash",
    "policy_version": "support-summary-routing-2026-08-31",
    "trigger": "error_rate_above_guardrail_or_budget_shape_exceeded",
    "owner": "platform-ai",
    "requires_code_deploy": false
  }
}

The best rollback is a small config change that returns traffic to the previous route and preserves the audit trail. Avoid deleting the failed change record. Mark it as rolled back, link the reason, and keep the evidence. Future reviews need to know that the route was tried and why it stopped.

Publish a review note

Every approved route change should produce a short review note for internal readers.

Use this shape:

Route change route_change_2026_09_05_support_summary moves support_summary
from deepseek-v4-flash to deepseek-v4-pro under policy support-summary-routing-2026-09-05.
Pricing source: https://aiwave.live/api/pricing.
Checked at: 2026-09-05T13:07:34Z.
Pricing version: a42d372ccf0b5dd13ecf71203521f9d2.
Effective group reviewed: vip.
Fixtures passed: 6/6.
Guardrails: output/request <= 1,200, cached input share >= 50%, retries/100 <= 6.
Rollback route: deepseek-v4-flash.
Privacy scan: passed; no raw prompt, completion text, API key, customer id, or payment id included.

The note should be readable by support, finance, and engineering. If a later invoice or support ticket questions the route, this note gives the team a starting point.

Link route changes to run receipts

After the route change is approved, every workload should carry the active policy version and pricing version into its run receipt or request ledger.

That closes the loop:

Before rollout	During traffic	After review
Route change record	Request ledger rows	Spend or support review
Fixture results	Resolved model and token buckets	Variance explanation
Pricing snapshot	Effective group and pricing version	Buyer-facing evidence packet
Rollback trigger	Error, retry, fallback counts	Decision to keep, adjust, or roll back

A route gate without runtime evidence becomes paperwork. Runtime evidence without a route gate becomes archaeology. Tie them together with stable ids.

Avoid common failure modes

These are the route-change mistakes that show up later as billing or support confusion:

Failure mode	What happens	Prevent it with
Missing pricing version	Reviewers cannot tell which rate snapshot applied	Required pricing fields
Alias-only logging	Actual model route is hidden	Store requested and resolved model
Prompt-heavy evidence	Support packet cannot be shared	Field-minimized receipts
No baseline window	Budget comparison is opinion	Baseline query and dated window
No rollback route	Bad route lingers while people debate	Pre-approved rollback field
No fixture contract	HTTP 200 is mistaken for workflow success	Schema and outcome checks

These controls are small. They are much easier to add before the first route incident than after a buyer asks for an explanation.

Final checklist

Before approving a route change, make sure the gateway can answer these questions:

Which feature or workload is changing?
Which requested and resolved model routes are involved?
Which pricing source, checked time, and pricing version were reviewed?
Which effective account group was used for planning?
Which fixtures passed and what contracts did they test?
What token-shape guardrails were set?
What fallback and retry behavior is allowed?
What privacy scan prevents sensitive material from entering the review?
What is the rollback route and trigger?
How will runtime receipts link back to this route change?

A multi-model gateway is valuable because teams can change routes without rewriting clients. For serious production work, that flexibility needs a control gate. Pin the pricing evidence, record requested and resolved models, test real workflow fixtures, set token-shape guardrails, and keep rollback simple. Then a route change becomes an explainable release instead of a mystery hiding behind one compatible API endpoint.

Source links to keep close

Keep these sources beside route-change reviews:

Source	Use
AIWave pricing	Public gateway pricing context
`https://aiwave.live/api/pricing`	Machine-readable pricing snapshot
AIWave models docs	Model catalog and route context
AIWave trust page	Public trust and operational posture
AIWave Chat Completions docs	OpenAI-compatible request shape

Recheck these sources before turning a route gate into a durable runbook. Route records, account groups, provider behavior, and pricing versions can change after an integration already works.

Build a run receipt schema for multi-provider AI workloads

Mattias chaw — Fri, 04 Sep 2026 13:16:17 +0000

Build a run receipt schema for multi-provider AI workloads

An AI workload needs more than a request log when it becomes part of a real product.

A request log can tell you that a call happened. A trace can show the shape of an agent run. A billing table can show token counts and debits. Those pieces are useful, but they often live in different systems. When a production workflow fails, spends more than expected, or needs to be reviewed by support, finance, or security, the team needs one portable record that explains what ran and what can be verified.

That record is a run receipt.

I use the term run receipt for a compact, exportable evidence packet attached to one AI workload. It does not need raw prompts, private documents, API keys, payment identifiers, or customer names. It needs the operational facts that let another engineer reconstruct the route, pricing source, token shape, failures, retries, and missing evidence.

AIWave is built around one OpenAI-compatible route to Chinese model families such as DeepSeek, GLM, Kimi, Qwen, ERNIE, MiniMAX, Doubao, StepFun, and MiMo. A unified route is useful because client code can stay stable while model choices change. That same convenience creates an evidence requirement: every serious workload should leave behind enough structured facts to explain the route later.

This article gives a practical schema and implementation pattern for a run receipt.

Start with the receipt boundary

A run receipt should be scoped to a workload, not to a single HTTP request.

One workload may include planning, retrieval, model calls, tool calls, retries, validation, and a final response. If each request is reviewed alone, the team may miss the real failure. A model call may be correct while the tool output is stale. A retry may look harmless until ten steps multiply it. A fallback may be rare in count but important in spend.

Use a receipt boundary that matches the question a human will ask.

Boundary	Good fit	Weak fit
`request`	Debugging one API response	Explaining a multi-step agent run
`step`	Comparing model choices inside an agent	Explaining total cost
`run`	Reviewing one workload end to end	High-volume aggregate monitoring
`window`	Weekly finance review	Debugging one failure

For most production agent and batch workflows, run is the right default. It is small enough to inspect and large enough to connect routing, cost, and outcome evidence.

Check the current pricing source

Before writing this article, I checked AIWave's public pricing endpoint on 2026-09-04 at 13:13 UTC. The endpoint returned success=true, 63 route records, pricing_version=a42d372ccf0b5dd13ecf71203521f9d2, auto_groups=["default"], and group ratios of default=3 and vip=1.

The same live response included these route fields:

Route	`model_ratio`	`completion_ratio`	`cache_ratio`	Enabled groups
`deepseek-v4-flash`	`0.319`	`3`	`0.0318`	`default`, `vip`, `svip`
`deepseek-v4-pro`	`0.957`	`3`	`0.0333`	`default`, `vip`, `svip`
`glm-5.1`	`1.05`	`3.142857`	`0.32381`	`default`, `vip`, `svip`
`kimi-k3`	`2.25`	`5`	`0.2`	`default`, `vip`, `svip`
`qwen3-max`	`0.7810988945590882`	`4`	absent	`default`, `vip`, `svip`

Using AIWave's public VIP x1 rate-card conversion, selected examples correspond to these per-1M token planning rates:

Route	Input	Cached input	Output
`deepseek-v4-flash`	`$0.638`	`$0.0203`	`$1.914`
`deepseek-v4-pro`	`$1.914`	`$0.0637`	`$5.742`
`glm-5.1`	`$2.10`	`$0.6800`	`$6.60`
`kimi-k3`	`$4.50`	`$0.90`	`$22.50`

These are dated planning inputs, not a promise that every account, request, future route, or provider-side table will produce the same invoice. A receipt should store the source URL, checked time, pricing version, effective group, resolved model, and token buckets beside the run.

Define the minimum receipt

The minimum receipt should answer eight questions:

What workload ran?
Who owns the review without exposing the customer?
Which route and model actually served each step?
Which pricing version and account group applied?
How many input, cached input, and output tokens were recorded?
Did retries, fallbacks, or tool calls change the result?
What outcome was observed?
Which fields are missing or unverifiable?

Here is a compact JSON shape:

{
  "receipt_version": "run-receipt-v0.1",
  "receipt_id": "rr_2026_09_04_example_001",
  "run_id": "support_summary_2026_09_04_042",
  "created_at": "2026-09-04T13:13:27Z",
  "environment": "production",
  "account_ref": "acct_hash_7f2b",
  "feature": "support_summary",
  "pricing": {
    "source_url": "https://aiwave.live/api/pricing",
    "checked_at": "2026-09-04T13:13:27Z",
    "pricing_version": "a42d372ccf0b5dd13ecf71203521f9d2",
    "effective_group": "vip"
  },
  "summary": {
    "status": "completed",
    "outcome_label": "passed_schema_validation",
    "total_steps": 3,
    "total_retries": 1,
    "fallbacks": 0
  },
  "totals": {
    "input_tokens": 2200000,
    "cached_input_tokens": 1600000,
    "output_tokens": 340000
  },
  "privacy": {
    "contains_raw_prompt": false,
    "contains_api_key": false,
    "redaction_policy": "field-minimized-v1"
  },
  "unknowns": ["provider_internal_queue_time"]
}

This object is not a substitute for the provider invoice. It is the workload's portable explanation. It tells another reviewer which source to check, which route was used, which fields were intentionally absent, and which fields could not be verified.

Store steps as append-only events

Receipts become unreliable when they are overwritten. Store step events append-only, then render the receipt from those events.

create table ai_run_events (
  event_id text primary key,
  receipt_id text not null,
  run_id text not null,
  step_id text not null,
  sequence integer not null,
  event_type text not null,
  created_at text not null,
  requested_model text,
  resolved_model text,
  provider text,
  pricing_version text,
  effective_group text,
  input_tokens integer,
  cached_input_tokens integer,
  output_tokens integer,
  latency_ms integer,
  retry_count integer not null default 0,
  fallback_from text,
  fallback_reason text,
  status text not null,
  error_code text,
  evidence_hash text,
  redaction_policy text not null
);

If a correction is needed, add a new event that explains the correction. Do not edit the original event in place. That gives support and engineering a review trail without pretending the first record never existed.

This design also keeps the receipt flexible. A simple chat call may emit one model event. A coding agent may emit planning, retrieval, model, tool, validation, retry, and finalization events. The rendered receipt can summarize both without changing the core model.

Keep model evidence separate from business outcome

Do not let the model response become the only success signal.

A model call may return HTTP 200 and still fail the workflow. A structured output may parse but violate a business rule. A code generation run may finish but fail tests. A customer support summary may be fluent but omit the ticket status.

Put route evidence and outcome evidence in different fields:

Field	Example	Meaning
`request_status`	`success`	The model request completed
`validation_status`	`schema_passed`	The response matched a technical contract
`outcome_label`	`needs_human_review`	The workload result still requires review
`test_result`	`unit_tests_failed`	A downstream check rejected the output
`decision_status`	`not_approved`	A human or policy gate has not approved execution

This separation matters for agent systems. A route can work while the run should stop. A receipt should make that distinction visible.

Calculate cost from buckets, not guesses

The receipt should store the buckets needed for cost explanation. At minimum, keep fresh input, cached input, output, retry count, fallback count, pricing version, and effective account group.

from dataclasses import dataclass
from decimal import Decimal


@dataclass(frozen=True)
class Rate:
    input_per_m: Decimal
    cached_input_per_m: Decimal
    output_per_m: Decimal


rates = {
    "deepseek-v4-flash": Rate(Decimal("0.638"), Decimal("0.0203"), Decimal("1.914")),
    "deepseek-v4-pro": Rate(Decimal("1.914"), Decimal("0.0637"), Decimal("5.742")),
    "glm-5.1": Rate(Decimal("2.10"), Decimal("0.6800"), Decimal("6.60")),
    "kimi-k3": Rate(Decimal("4.50"), Decimal("0.90"), Decimal("22.50")),
}


def estimate_step_usd(model: str, input_tokens: int, cached_input_tokens: int, output_tokens: int) -> Decimal:
    rate = rates[model]
    fresh_input = max(input_tokens - cached_input_tokens, 0)
    million = Decimal("1000000")
    return (
        Decimal(fresh_input) / million * rate.input_per_m
        + Decimal(cached_input_tokens) / million * rate.cached_input_per_m
        + Decimal(output_tokens) / million * rate.output_per_m
    )


print(estimate_step_usd("deepseek-v4-flash", 2_000_000, 1_500_000, 300_000))

In production, load rates from a dated snapshot rather than hard-coding them. The example is here to show the shape of the calculation, not to replace the gateway's billing source.

Add privacy rules to the schema

A receipt should be useful because it avoids sensitive material by default.

Do not include raw prompts, completions, API keys, customer emails, phone numbers, payment identifiers, or private documents in the core receipt. If a support case needs a redacted excerpt, attach it as a separate evidence object with its own retention class and approval status.

Use explicit privacy fields:

{
  "privacy": {
    "contains_raw_prompt": false,
    "contains_completion_text": false,
    "contains_customer_identifier": false,
    "contains_payment_identifier": false,
    "redaction_policy": "field-minimized-v1",
    "retention_class": "support-30d"
  }
}

This keeps the receipt shareable across engineering, finance, and support. It also forces the team to admit when a field is missing instead of filling the gap with customer content.

Export in boring formats

A receipt that only works inside one dashboard is not portable evidence.

Start with JSON for machines, CSV for finance review, and HTML for humans. PDF can come later if buyers ask for it, but the first version should keep the original structured data intact.

Format	Primary user	Requirement
JSON	Engineering	Exact fields, stable schema version
CSV	Finance and operations	One row per step or cost bucket
HTML	Support and customer review	Readable summary with source links

Every export should include the schema version, creation time, pricing source, pricing version, and unknown fields. If a value is derived, label it. If a value is not available, write unknown instead of inventing a number.

Render a human summary

The human summary should be short and inspectable.

Use this shape:

Run support_summary_2026_09_04_042 completed with 3 steps and 1 retry.
The run used deepseek-v4-flash for extraction and deepseek-v4-pro for final review.
Pricing source was https://aiwave.live/api/pricing, checked at 2026-09-04T13:13:27Z,
with pricing_version a42d372ccf0b5dd13ecf71203521f9d2 and effective_group vip.
No raw prompt, completion text, API key, or payment identifier is included.
Unknown: provider_internal_queue_time.

That is enough for a first review. A finance teammate can see the pricing source. A support teammate can see the missing field. An engineer can open the JSON for details.

Validate before publishing or sharing

Treat run receipts like release artifacts. Validate them before they leave the system.

FORBIDDEN_KEYS = {
    "api_key",
    "authorization",
    "password",
    "secret",
    "payment_id",
    "customer_email",
}


def validate_receipt(receipt: dict) -> list[str]:
    errors = []
    text = str(receipt).lower()

    for key in FORBIDDEN_KEYS:
        if key in text:
            errors.append(f"forbidden field or value contains {key}")

    pricing = receipt.get("pricing", {})
    if not pricing.get("source_url"):
        errors.append("missing pricing.source_url")
    if not pricing.get("checked_at"):
        errors.append("missing pricing.checked_at")
    if not pricing.get("pricing_version"):
        errors.append("missing pricing.pricing_version")

    privacy = receipt.get("privacy", {})
    if privacy.get("contains_api_key") is not False:
        errors.append("privacy.contains_api_key must be false")
    if privacy.get("contains_raw_prompt") is not False:
        errors.append("privacy.contains_raw_prompt must be false")

    return errors

The validator should be stricter than the dashboard. If a receipt cannot prove its source, version, and privacy posture, it should stay internal until fixed.

Use receipts as a product boundary

A run receipt should not become an all-purpose observability platform on day one. Keep the first version narrow:

Include now	Defer until needed
Run id, steps, model route, tokens, status	Full trace viewer
Pricing source and version	Real-time contract negotiation
Retry and fallback counts	Automated policy changes
Privacy and unknown fields	Long retention by default
JSON, CSV, HTML export	Complex role management

This boundary is important for small teams. The receipt proves what a workload can explain today. It does not need to solve every monitoring, audit, and governance problem at once.

Final checklist

Before you rely on a run receipt, make sure it answers these questions:

Which workload does this receipt describe?
Which exact route served each model step?
Which pricing source and version were checked?
Which effective account group applied?
Which usage buckets were recorded?
Which retries, fallbacks, and errors happened?
Which business or validation outcome was observed?
Which sensitive fields are intentionally excluded?
Which fields are still unknown?
Which export format lets another team review the evidence?

The value of a multi-provider gateway is not only that it can route traffic. It should also help teams explain what happened after a workload ran. A run receipt turns that explanation into a small, dated, portable record that developers, support, and finance can inspect without exposing the data that should stay private.

Source links to keep close

Keep these sources beside receipt reviews:

Source	Use
AIWave pricing	Public gateway pricing context
`https://aiwave.live/api/pricing`	Machine-readable pricing snapshot
AIWave models docs	Model catalog and route context
AIWave trust page	Public trust and operational posture
AIWave Chat Completions docs	OpenAI-compatible request shape

Recheck these sources before copying numbers into a long-lived runbook. Pricing versions, route fields, model ids, and account policy can change after a successful integration.

Build a spend variance triage loop for AI API gateways

Mattias chaw — Thu, 03 Sep 2026 13:13:25 +0000

Build a spend variance triage loop for AI API gateways

The first bad AI API invoice is usually not a math problem. It is an evidence problem.

A team expects one number before rollout. The gateway reports another number after traffic runs. The product owner asks which feature moved. Finance asks whether the rate changed. Support asks whether retries, fallback routes, or account groups affected usage. Engineering opens logs and finds a pile of request ids, token totals, and model names that almost answer the question.

Almost is not enough for Tier 1 and Tier 2 buyers. A serious AI gateway should be able to explain spend variance without reading raw prompts, exposing customer data, or guessing which rate card was current at the time.

AIWave is built for teams that want one OpenAI-compatible route to Chinese model families such as DeepSeek, GLM, Kimi, Qwen, ERNIE, MiniMAX, Doubao, StepFun, and MiMo. That single route is useful when teams want stable client code, one USD ledger, and multiple model choices. It also means the gateway has to keep route evidence close to every cost review.

This article gives a practical loop for triaging spend variance after real traffic runs.

Start with a variance record

Do not begin with a dashboard screenshot. Begin with a small variance record that names the expectation, the observation, and the evidence window.

Field	Example	Why it matters
`window_start`	`2026-09-03T00:00:00Z`	Defines the query boundary
`window_end`	`2026-09-03T06:00:00Z`	Avoids mixing partial periods
`expected_usd`	`42.00`	Captures the rollout assumption
`observed_usd`	`57.80`	Captures the review target
`owner`	`platform-ai`	Assigns a decision owner
`pricing_source`	`https://aiwave.live/api/pricing`	Shows where route fields came from
`pricing_checked_at`	`2026-09-03T13:11:00Z`	Prevents stale rows from becoming facts
`pricing_version`	`a42d372ccf0b5dd13ecf71203521f9d2`	Links the review to a catalog snapshot

That record is not the final answer. It is the case file. Every later query should either explain part of the variance or mark it as unknown.

Check the current pricing source

Before writing this article, I checked AIWave's public pricing endpoint on 2026-09-03 at 13:11 UTC. The endpoint returned success=true, pricing_version=a42d372ccf0b5dd13ecf71203521f9d2, 63 route records, auto_groups=["default"], and group ratios of default=3 and vip=1.

The same live response included these route fields:

Route	`model_ratio`	`completion_ratio`	`cache_ratio`	Enabled groups
`deepseek-v4-flash`	`0.319`	`3`	`0.0318`	`default`, `vip`, `svip`
`deepseek-v4-pro`	`0.957`	`3`	`0.0333`	`default`, `vip`, `svip`
`glm-5.1`	`1.05`	`3.142857`	`0.32381`	`default`, `vip`, `svip`
`kimi-k3`	`2.25`	`5`	`0.2`	`default`, `vip`, `svip`

Using AIWave's public VIP x1 rate-card conversion, those rows correspond to these per-1M token planning rates for the selected examples:

Route	Input	Cached input	Output
`deepseek-v4-flash`	`$0.638`	`$0.0203`	`$1.914`
`deepseek-v4-pro`	`$1.914`	`$0.0637`	`$5.742`
`glm-5.1`	`$2.10`	`$0.6800`	`$6.60`
`kimi-k3`	`$4.50`	`$0.90`	`$22.50`

These are dated planning inputs, not a promise that every account, workload, or future request will land on the same bill. Store the checked time, source URL, pricing version, effective group, route id, token buckets, and final usage beside the request.

Split variance into four buckets

Spend variance becomes easier to debug when every difference is assigned to one bucket first.

Bucket	Typical signal	First query
Volume variance	More calls or larger prompts than forecast	Count requests and input tokens by feature
Mix variance	More traffic moved to a different route	Group usage by resolved model
Shape variance	Output, cache, or tool usage changed	Compare input, cached input, and output buckets
Policy variance	Retries, fallbacks, or account groups changed	Group usage by policy version and effective group

Do not argue about the final invoice before this split exists. A model route may be correct while the output cap is too high. A pricing row may be stable while the product shipped a larger prompt. A fallback may be rare in request count but large in token share.

Build the request ledger

A useful ledger does not need raw prompts. It needs enough structured data to explain routing and billing.

create table gateway_request_ledger (
  request_id text primary key,
  account_id_hash text not null,
  feature text not null,
  environment text not null,
  requested_model text not null,
  resolved_model text not null,
  route_reason text not null,
  policy_version text not null,
  pricing_version text not null,
  effective_group text not null,
  source_checked_at text not null,
  input_tokens integer not null,
  cached_input_tokens integer not null default 0,
  output_tokens integer not null,
  tool_calls integer not null default 0,
  retry_count integer not null default 0,
  fallback_from text,
  fallback_reason text,
  status text not null,
  created_at text not null
);

The important fields are resolved_model, pricing_version, effective_group, token buckets, retries, and fallback metadata. If those are missing, the triage loop will drift into opinion.

Hash account identifiers before they enter analytical logs. Keep API keys, raw prompts, private documents, and customer-identifying text out of the ledger.

Recreate the expected spend

The first calculation should use the exact assumption that approved the rollout. If the rollout forecast assumed 80 percent deepseek-v4-flash and 20 percent deepseek-v4-pro, encode that assumption as data.

from dataclasses import dataclass
from decimal import Decimal


@dataclass(frozen=True)
class Rate:
    input_per_m: Decimal
    cached_input_per_m: Decimal
    output_per_m: Decimal


rates = {
    "deepseek-v4-flash": Rate(Decimal("0.638"), Decimal("0.0203"), Decimal("1.914")),
    "deepseek-v4-pro": Rate(Decimal("1.914"), Decimal("0.0637"), Decimal("5.742")),
    "glm-5.1": Rate(Decimal("2.10"), Decimal("0.6800"), Decimal("6.60")),
    "kimi-k3": Rate(Decimal("4.50"), Decimal("0.90"), Decimal("22.50")),
}


def estimate_usd(model: str, input_tokens: int, cached_input_tokens: int, output_tokens: int) -> Decimal:
    rate = rates[model]
    fresh_input = max(input_tokens - cached_input_tokens, 0)
    million = Decimal("1000000")
    return (
        Decimal(fresh_input) / million * rate.input_per_m
        + Decimal(cached_input_tokens) / million * rate.cached_input_per_m
        + Decimal(output_tokens) / million * rate.output_per_m
    )


print(estimate_usd("deepseek-v4-flash", 2_000_000, 1_500_000, 300_000))

This example is deliberately small. In production, load rates from a dated snapshot rather than hard-coding them in application code. The goal is to reproduce the forecast with the same source date before comparing it with observed usage.

Query by feature first

Most surprises begin at the feature boundary. A new assistant panel, document importer, code reviewer, or nightly job can change token shape without changing the model route.

select
  feature,
  count(*) as requests,
  sum(input_tokens) as input_tokens,
  sum(cached_input_tokens) as cached_input_tokens,
  sum(output_tokens) as output_tokens,
  sum(retry_count) as retries
from gateway_request_ledger
where created_at >= :window_start
  and created_at < :window_end
group by feature
order by output_tokens desc;

If one feature owns the delta, keep the review there. Check whether the prompt template grew, whether retrieved context expanded, whether output caps changed, or whether the job began running more often.

If no feature owns the delta, move to route mix.

Query by resolved model

The requested model field is useful for client behavior. The resolved model field is useful for cost explanation.

select
  resolved_model,
  pricing_version,
  effective_group,
  count(*) as requests,
  sum(input_tokens) as input_tokens,
  sum(cached_input_tokens) as cached_input_tokens,
  sum(output_tokens) as output_tokens
from gateway_request_ledger
where created_at >= :window_start
  and created_at < :window_end
group by resolved_model, pricing_version, effective_group
order by output_tokens desc;

This query catches route-mix drift. Maybe a planning workflow moved from deepseek-v4-flash to deepseek-v4-pro. Maybe a long-context batch started using kimi-k3. Maybe a GLM route became attractive for reasoning tasks but produced longer outputs than the forecast allowed.

The answer may be acceptable. The important part is that the gateway can show it.

Look for output drift

Output is often where a quiet variance hides. A prompt that asks for "a complete analysis" can produce a much larger completion than the forecast expected. A tool-heavy agent may also turn short input into long status reports.

Track these ratios:

Metric	Formula	Failure signal
Output share	`output_tokens / (input_tokens + output_tokens)`	The response is larger than planned
Cache share	`cached_input_tokens / input_tokens`	Repeated context is not being reused
Retry pressure	`retry_count / requests`	Transient failures are becoming spend
Fallback share	`fallback_requests / requests`	Route policy is doing hidden work

A variance loop should not stop at "model X costs more." It should ask whether the workload used model X as intended.

Check policy variance

Policy changes are easy to miss because they may be correct individually. A retry limit increases from one to three. A fallback rule starts allowing a stronger model for customer-visible work. An account moves from one group to another. The gateway still behaves as designed, but the invoice changes.

Use policy versions as first-class evidence:

select
  policy_version,
  route_reason,
  effective_group,
  count(*) as requests,
  sum(retry_count) as retries,
  count(fallback_from) as fallback_requests
from gateway_request_ledger
where created_at >= :window_start
  and created_at < :window_end
group by policy_version, route_reason, effective_group
order by requests desc;

If spend changed after a policy version changed, the review should include the rollout ticket. The fix may be a lower output cap, a narrower fallback rule, or a separate budget for high-risk jobs.

Produce a triage answer

The final answer should be short enough for finance and detailed enough for engineering.

Use this shape:

Section	What to write
Conclusion	One sentence explaining the main variance driver
Source window	Exact timestamps used in the query
Pricing evidence	Source URL, checked time, and pricing version
Model mix	Top routes by observed spend
Token shape	Input, cached input, output, retry, and fallback deltas
Policy changes	Any policy or account-group change in the window
Unknowns	Missing fields that weaken the conclusion
Action	The smallest change that prevents repeat confusion

Here is an example:

Observed spend exceeded forecast mainly because the support-summary feature
used 2.4x the planned output tokens after policy-2026-09-03 raised max_tokens
from 700 to 1800. Pricing source was https://aiwave.live/api/pricing checked at
2026-09-03T13:11:00Z with pricing_version a42d372ccf0b5dd13ecf71203521f9d2.
No raw prompt data was needed for this review.

That is the kind of answer a buyer can inspect. It does not claim perfect forecasting. It shows a dated, reproducible chain of evidence.

Source links to keep close

Keep these sources beside the variance review:

Source	Use
AIWave pricing	Public gateway pricing context
`https://aiwave.live/api/pricing`	Machine-readable pricing snapshot
AIWave models docs	Model catalog and route context
AIWave trust page	Public trust and operational posture
DeepSeek pricing	Provider bucket context for DeepSeek routes
QwenCloud pricing	Context, cache, tool, and failed-call billing behavior

Recheck these sources before copying numbers into a long-lived runbook. Model ids, cache ratios, output prices, and account policy can change after a successful rollout.

Final checklist

Before a spend review closes, make sure the gateway can answer these questions:

Which pricing source and version were used?
Which feature produced the variance?
Which exact model routes ran?
Which account group applied?
How much usage was fresh input, cached input, and output?
Did retries or fallbacks contribute?
Which policy version admitted the calls?
What data is missing from the ledger?
What small control prevents the same confusion next time?

The value of a gateway is not only that it can route requests. It should explain what happened after the route ran. A spend variance triage loop turns a surprising invoice into a dated engineering review, with enough evidence for platform, support, and finance to make the next decision without guessing.

Build a model alias sunset plan for AI API gateways

Mattias chaw — Wed, 02 Sep 2026 13:15:53 +0000

Build a model alias sunset plan for AI API gateways

Model aliases are convenient until they become part of your production contract.

An alias such as latest, pro, fast, or default looks harmless in a demo. It lets the SDK call a route without forcing the application to know every provider model id. That is useful during early exploration. It becomes risky when a production workflow, an invoice review, or a support investigation has to answer one plain question: which model actually ran?

OpenAI-compatible gateways need a clean way to retire aliases. A good sunset plan keeps old integrations working for a short period, gives new requests an exact model id, records the pricing version that admitted the call, and leaves enough evidence for engineering, support, and procurement to review the change later.

AIWave is built for teams that want one OpenAI-compatible route to Chinese model families such as DeepSeek, GLM, Kimi, Qwen, ERNIE, MiniMAX, Doubao, StepFun, and MiMo. That routing surface is useful when the client should stay stable while the route policy changes. It also means alias changes need a written migration path.

This article gives one pattern for sunsetting model aliases without hiding behavior from the teams that depend on the gateway.

Start with the failure mode

The dangerous alias change is silent.

A request used deepseek-fast yesterday. Today the same alias resolves to another route, or to a new provider variant, or to a model with a different cache bucket. The application still returns text, so the incident does not look like an outage. The problem shows up later when cost, latency, answer shape, or tool behavior no longer matches the test evidence.

For Tier 1 and Tier 2 teams, that is a bad procurement story. A platform owner cannot ask finance to trust a gateway that changes effective routes without a dated record. A support engineer cannot debug a failed agent run if the logs only show the alias. A security reviewer cannot approve a migration if the provider boundary is described by a nickname.

The alias may stay in the request. The resolved model must never be vague.

Check the current pricing source

Before writing this article, I checked AIWave's public pricing endpoint on 2026-09-02 at 13:11 UTC. The endpoint returned success=true, pricing_version=a42d372ccf0b5dd13ecf71203521f9d2, 63 route records, and auto_groups=["default"]. The response also exposed group ratios of default=3 and vip=1.

The same live response included these route fields:

Route	`model_ratio`	`completion_ratio`	`cache_ratio`	Enabled groups
`deepseek-v4-flash`	`0.319`	`3`	`0.0318`	`default`, `vip`, `svip`
`deepseek-v4-pro`	`0.957`	`3`	`0.0333`	`default`, `vip`, `svip`
`glm-5.1`	`1.05`	`3.142857`	`0.32381`	`default`, `vip`, `svip`
`kimi-k3`	`2.25`	`5`	`0.2`	`default`, `vip`, `svip`

Those are gateway catalog fields, not a claim that every request will produce the same bill. Store the source URL, checked time, pricing version, effective group, requested alias, resolved route, and token buckets beside each migration test.

If an alias sunset plan cannot say which pricing version applied, the plan is not ready.

Define three names for every request

Alias migrations go wrong when one field does too much work.

Use three separate names:

The alias names in this section are design examples, not current public AIWave model ids. Use exact catalog ids such as deepseek-v4-flash, deepseek-v4-pro, glm-5.1, and kimi-k3 when you configure production clients.

Field	Example	Purpose
`requested_model`	`deepseek-fast`	What the client sent
`resolved_model`	`deepseek-v4-flash`	What the gateway executed
`policy_version`	`alias-policy-2026-09-02`	Which alias map made the decision

This separation protects old clients while giving reviewers a real route name. It also lets you phase out the alias without losing the historical request shape.

{
  "request_id": "req_01J_alias_example",
  "requested_model": "deepseek-fast",
  "resolved_model": "deepseek-v4-flash",
  "policy_version": "alias-policy-2026-09-02",
  "pricing_version": "a42d372ccf0b5dd13ecf71203521f9d2",
  "effective_group": "default",
  "status": "success"
}

Do not store raw prompts or secrets in this record. The alias evidence needs route, policy, pricing, token, and status fields. It does not need user content.

Inventory aliases before you change them

The first step is a read-only inventory. Count aliases by customer segment, SDK version, endpoint, and job type. A web chat request and a nightly agent run should not share the same migration risk score.

Use a query like this against your request ledger:

select
  requested_model,
  resolved_model,
  count(*) as calls,
  count(distinct account_id_hash) as accounts,
  min(created_at) as first_seen,
  max(created_at) as last_seen
from gateway_request_log
where created_at >= datetime('now', '-30 days')
group by requested_model, resolved_model
order by calls desc;

The result tells you which aliases are still live and which exact routes they produce. If one alias already maps to multiple resolved models, stop and document why. The answer may be A/B testing, fallback policy, account groups, or a bug. Do not announce the sunset until you can explain that spread.

Put aliases into states

A sunset plan needs a state machine. Dates alone are not enough because some teams will miss a date and some clients will lag.

Use states that the gateway can enforce:

State	Gateway behavior	User-facing behavior
`active`	Alias resolves normally	No warning
`deprecated`	Alias resolves, warning attached	Response header or dashboard warning
`pinned`	Alias resolves only to its old route	No silent upgrade
`blocked_for_new_keys`	Existing keys work, new keys must use exact ids	Console rejects new alias use
`removed`	Alias returns a clear error	Error includes replacement ids

The pinned state matters. It gives existing production traffic stability while new integrations move to exact ids. Without that state, teams often choose between a risky silent change and a hard break.

Make warnings machine-readable

A warning in a blog post or changelog is not enough. Put the warning where SDKs, dashboards, and CI checks can read it.

For OpenAI-compatible responses, the body shape may be constrained by the client library. Headers are a practical place for migration hints:

X-Gateway-Requested-Model: deepseek-fast
X-Gateway-Resolved-Model: deepseek-v4-flash
X-Gateway-Alias-State: deprecated
X-Gateway-Alias-Replacement: deepseek-v4-flash
X-Gateway-Policy-Version: alias-policy-2026-09-02
X-Gateway-Pricing-Version: a42d372ccf0b5dd13ecf71203521f9d2

If your gateway does not expose custom headers, write the same fields into an account-visible usage ledger. The important part is that the migration evidence is attached to real requests, not only to release notes.

Add a client-side guard

Platform teams can catch alias use before requests reach production. Keep the check boring and explicit.

from dataclasses import dataclass


@dataclass(frozen=True)
class ModelPolicy:
    allowed_exact_models: set[str]
    deprecated_aliases: dict[str, str]
    allow_aliases_in_dev: bool = True


def validate_model_for_environment(
    *,
    requested_model: str,
    environment: str,
    policy: ModelPolicy,
) -> str:
    if requested_model in policy.allowed_exact_models:
        return requested_model

    replacement = policy.deprecated_aliases.get(requested_model)
    if replacement and environment in {"local", "staging"} and policy.allow_aliases_in_dev:
        return replacement

    if replacement:
        raise ValueError(
            f"model alias {requested_model!r} is deprecated; "
            f"use exact model id {replacement!r}"
        )

    raise ValueError(f"model {requested_model!r} is not allowed by policy")


policy = ModelPolicy(
    allowed_exact_models={"deepseek-v4-flash", "deepseek-v4-pro", "glm-5.1", "kimi-k3"},
    deprecated_aliases={"deepseek-fast": "deepseek-v4-flash"},
)

model = validate_model_for_environment(
    requested_model="deepseek-fast",
    environment="production",
    policy=policy,
)

This guard does not need to know the provider's full catalog. It only needs to protect the application's supported set.

Tie the migration to pricing evidence

Aliases are not only a routing problem. They are also a cost explanation problem.

When the alias policy changes, capture a small evidence packet:

Evidence	Why it belongs
Alias policy version	Explains the resolution rule
Pricing source URL	Lets reviewers recheck the catalog
Pricing checked time	Prevents stale screenshots from becoming facts
Pricing version	Links the request to the gateway snapshot
Effective account group	Explains group-specific calculation behavior
Token buckets	Separates input, cached input, and output
Replacement model ids	Gives developers exact strings to use

A request may still be billed by units, credits, quota, or invoice rows, depending on the gateway. The public article should not invent a blended public price from internal ratios. Keep the evidence in the same unit the gateway reports.

Test old and new clients together

Run the alias migration as a compatibility test, not as a copy edit.

Build a fixture that sends the same redacted prompt through both the old alias and the replacement id in a staging gateway that still accepts the alias. Record resolved model, pricing version, usage fields, response status, and stop reason.

from openai import OpenAI

client = OpenAI(
    base_url="https://gateway.example/v1",
    api_key="YOUR_API_KEY_HERE",
)


def probe(model: str) -> dict:
    response = client.chat.completions.create(
        model=model,
        messages=[
            {
                "role": "user",
                "content": "Return a JSON checklist for a redacted API migration test.",
            }
        ],
        temperature=0,
        max_tokens=500,
    )

    return {
        "requested_model": model,
        "response_model": response.model,
        "usage": response.usage.model_dump() if response.usage else None,
    }


for model in ["deepseek-fast", "deepseek-v4-flash"]:
    print(probe(model))

Use synthetic or redacted prompts for this test. Do not use customer transcripts to prove that an alias resolves.

Publish a migration window

Give developers a narrow, dated path:

Date	State	Action
Day 0	`deprecated`	Warnings begin in ledger and response metadata
Day 14	`blocked_for_new_keys`	New keys must use exact model ids
Day 30	`pinned`	Existing alias stays on its old resolved route
Day 60	`removed`	Alias returns a clear replacement error

The exact dates should match your support load and customer contracts. The structure matters more than the length. New traffic moves first. Existing traffic gets a stable route while owners update code.

Write the removal error for developers

A removal error should tell the developer what to do next. It should not force them to open a ticket just to find the replacement id.

{
  "error": {
    "type": "model_alias_removed",
    "message": "The model alias 'deepseek-fast' has been removed. Use 'deepseek-v4-flash' or another exact model id from the catalog.",
    "requested_model": "deepseek-fast",
    "replacement_models": ["deepseek-v4-flash"],
    "policy_version": "alias-policy-2026-09-02",
    "docs_url": "https://aiwave.live/docs/models"
  }
}

This is more useful than a generic unknown-model error. It tells the developer the alias used to exist and names the supported path forward.

What procurement should see

Procurement does not need every request body. It needs proof that the route did not change behind the team's back.

Give them a short migration report:

Field	Example
Alias retired	`deepseek-fast`
Replacement	`deepseek-v4-flash`
First warning date	`2026-09-02`
New-key block date	`2026-09-16`
Removal target	`2026-11-01`
Pricing source	`https://aiwave.live/api/pricing`
Pricing checked at	`2026-09-02T13:11:09Z`
Pricing version	`a42d372ccf0b5dd13ecf71203521f9d2`

That report turns the migration from a vague platform change into an auditable event.

Operational checks

Run these checks daily during the migration window:

Check	Failure signal
Alias used by new keys	The console or API allowed new deprecated usage
Missing resolved model	The ledger cannot prove what ran
Missing policy version	Support cannot reproduce the alias map
Missing pricing version	Finance cannot tie usage to a dated catalog
Multiple replacements without policy	The alias is doing hidden routing work
Production alias count not falling	Owners may not have seen the warning
Secret pattern in migration notes	A draft or log contains key-like material

The last row belongs in every content and support workflow. Migration examples can show placeholders, but they should never publish real keys, customer ids, private prompts, or internal payment details.

Source links to keep close

Keep these sources beside the migration ticket:

Source	Use
AIWave pricing	Public gateway pricing context
`https://aiwave.live/api/pricing`	Machine-readable pricing snapshot
AIWave models docs	Exact model ids and catalog context
AIWave trust page	Public trust and operational context
DeepSeek pricing	Provider bucket context for DeepSeek routes
QwenCloud pricing	Context, cache, and tool billing behavior

Recheck these sources before changing a rollout date. Model ids, cache fields, provider capabilities, and account groups can change after a successful test.

Final checklist

Before retiring a model alias, make sure the gateway can answer these questions:

Which clients still send the alias?
Which exact model does the alias resolve to?
Which policy version made that decision?
Which pricing version admitted the request?
Which account group applied?
Which replacement id should new integrations use?
When do warnings, new-key blocks, pinned routing, and removal begin?
What evidence can support and procurement inspect without seeing secrets?

Aliases make exploration easier. Exact resolved routes make production explainable. A sunset plan lets you keep both for long enough to migrate without turning yesterday's shortcut into tomorrow's mystery bill.

When an AIWave trial should stop before the first request

Mattias chaw — Tue, 01 Sep 2026 13:16:42 +0000

When an AIWave trial should stop before the first request

The easiest API trial to start is often the one you should delay.

A developer can point the OpenAI SDK at a compatible endpoint in a few minutes. That does not mean the workload is ready for a gateway trial. A serious trial needs a data boundary, an owner, a budget ceiling, a route policy, and a dated pricing source before the first request leaves the application.

AIWave is built for teams that want one OpenAI-compatible route to Chinese model families such as DeepSeek, GLM, Kimi, Qwen, ERNIE, MiniMAX, Doubao, StepFun, and MiMo. That is useful when the team wants to keep the client stable while comparing model behavior, billing shape, and route fit. It is not a shortcut around procurement, security review, or workload design.

This article gives a pre-trial fit screen for Tier 1 and Tier 2 engineering teams. The goal is blunt: decide when an AIWave trial should stop before it begins.

Start with rejection criteria

Most API evaluations begin with a success checklist. The better first artifact is a stop checklist.

Write down the conditions that make the trial invalid:

Stop condition	Why it matters	Better next step
A signed enterprise SLA is required before any request	Public docs and public route evidence are not a negotiated contract	Use the contracted vendor path first
The prompt contains regulated sensitive data	Route compatibility does not answer legal or residency questions	Run security and legal review with redacted examples
The workload needs a direct-provider-only feature	A gateway may not expose every provider console, file, or admin feature	Test that feature in the provider account
The agent has no output cap or retry rule	One vague task can become many model calls	Add max tokens, retry ceilings, and stop reasons
There is no trial budget	Usage evidence requires real, bounded requests	Create a small funded test envelope
Finance cannot accept the payment or invoice path	A passing demo can still fail procurement	Resolve billing workflow before engineering work

This table should live beside the trial ticket. If any row is true, the right move is not a clever prompt. The right move is to stop and fix the trial shape.

Check the current pricing source

Before writing this article, I checked AIWave's public pricing endpoint on 2026-09-01 at 13:11 UTC. The endpoint returned success=true, pricing_version=a42d372ccf0b5dd13ecf71203521f9d2, 63 route records, and auto_groups=["default"].

The same live response exposed these route fields:

Route	`model_ratio`	`completion_ratio`	`cache_ratio`	Enabled groups
`deepseek-v4-flash`	`0.319`	`3`	`0.0318`	`default`, `vip`, `svip`
`deepseek-v4-pro`	`0.957`	`3`	`0.0333`	`default`, `vip`, `svip`
`glm-5.1`	`1.05`	`3.142857`	`0.32381`	`default`, `vip`, `svip`
`kimi-k3`	`2.25`	`5`	`0.2`	`default`, `vip`, `svip`

Those are gateway catalog fields, not a promise that every workload will produce the same bill. The useful habit is to store the source URL, checked time, pricing version, model ID, effective account group, token buckets, and route decision for the trial.

If your trial plan cannot say which pricing facts it will use, stop before the first request.

Data policy comes before SDK setup

An OpenAI-compatible endpoint makes migration familiar:

from openai import OpenAI

client = OpenAI(
    base_url="https://aiwave.live/v1",
    api_key="YOUR_API_KEY_HERE",
)

That snippet proves the client shape. It does not prove the data policy.

Before the trial, classify the prompt data:

Data class	Trial rule
Public docs, synthetic prompts, redacted tickets	Good first test material
Customer content with identifiers removed	Review the redaction method first
Payment, health, legal, employment, or account secrets	Do not send until policy is approved
Production transcripts	Use only after retention, access, and support boundaries are written
API keys, passwords, private tokens	Never put them in prompts, logs, drafts, or examples

The trial should use enough realistic structure to test the route without carrying real customer identity or secrets. If the team cannot make a redacted acceptance set, it probably cannot explain the trial later.

SLA needs can invalidate the trial

There is a difference between evaluating a public API route and approving a production dependency.

Some teams require a signed SLA, negotiated support terms, vendor security paperwork, or purchase order flow before any production data moves. If that is your process, do not treat a gateway trial as a way around it. Use public docs, pricing pages, and a redacted technical probe for research, then wait for the commercial path that your organization needs.

This is especially true for internal platforms that serve many product teams. A single engineer can make the SDK work. A platform owner has to explain who owns incidents, how outages are reported, which account group applies, how invoices are reviewed, and where support evidence lives.

If those answers are mandatory before integration, stop the trial at procurement review.

Direct-provider features may be the real requirement

A gateway is strongest when the job is multi-model access through one compatible client, route evidence, and a readable billing trail. It is weaker when the job depends on a provider-specific console feature that is not exposed through the compatible route.

Examples include:

Requirement	Why direct access may fit better
Provider-native fine tuning	The training workflow may live outside chat completions
Provider-specific files or agents	The object model may not map cleanly to the compatible API
Admin console controls	Procurement may need tenant-level settings in the provider console
Vendor evaluation suite	The benchmark tool may require a direct account
Contracted regional deployment	The route decision may be tied to a negotiated provider agreement

None of these make the gateway bad. They define the workload. If the trial's main question is a direct-provider feature, test that feature directly.

Unbounded agents should fail the fit screen

Agent trials are where budgets drift fastest.

A basic chat call has one prompt and one answer. An agent can plan, retry, search, summarize, call tools, compress history, choose another model, and write a final answer. Without controls, a trial result may be impossible to reproduce or explain.

Set these limits before the first agent run:

Control	Minimum value to record
`max_tokens`	Route-specific output ceiling
`retry_policy`	Attempt count and allowed failure classes
`tool_policy`	Which tools can run, and how many times
`allowed_models`	Exact model IDs admitted for the trial
`pricing_version`	Version used at admission time
`source_checked_at`	Timestamp for pricing and provider docs
`budget_usd_limit`	Spend envelope for the test
`stop_reason`	Why the trial stopped or continued

The control does not need to be elaborate. It needs to be visible.

A small fit-screen function

Put the fit screen in code so the same rule runs in CI, staging, and local tests.

from dataclasses import dataclass


@dataclass(frozen=True)
class TrialRequest:
    signed_sla_required_today: bool
    regulated_sensitive_data: bool
    direct_provider_only_feature: bool
    trial_budget_usd: float
    max_tokens: int | None
    allowed_models: list[str]
    pricing_version: str | None
    source_checked_at: str | None


def screen_aiwave_trial(trial: TrialRequest) -> list[str]:
    reasons: list[str] = []

    if trial.signed_sla_required_today:
        reasons.append("signed SLA is required before technical routing")
    if trial.regulated_sensitive_data:
        reasons.append("regulated sensitive data is not approved for the trial")
    if trial.direct_provider_only_feature:
        reasons.append("the workload depends on a provider-only feature")
    if trial.trial_budget_usd <= 0:
        reasons.append("trial budget is missing")
    if trial.max_tokens is None or trial.max_tokens <= 0:
        reasons.append("output cap is missing")
    if not trial.allowed_models:
        reasons.append("allowed model list is empty")
    if not trial.pricing_version or not trial.source_checked_at:
        reasons.append("dated pricing source is missing")

    return reasons


trial = TrialRequest(
    signed_sla_required_today=False,
    regulated_sensitive_data=False,
    direct_provider_only_feature=False,
    trial_budget_usd=10,
    max_tokens=700,
    allowed_models=["deepseek-v4-flash", "glm-5.1"],
    pricing_version="a42d372ccf0b5dd13ecf71203521f9d2",
    source_checked_at="2026-09-01T13:11:22Z",
)

blocked = screen_aiwave_trial(trial)
if blocked:
    raise SystemExit("Do not start trial: " + "; ".join(blocked))

This function is intentionally plain. It keeps the decision out of the prompt and inside the system that owns the budget.

What a valid first request looks like

A good first request is narrow, redacted, and easy to audit.

response = client.chat.completions.create(
    model="deepseek-v4-flash",
    messages=[
        {
            "role": "user",
            "content": (
                "Return a JSON checklist for evaluating a redacted "
                "Chinese AI API routing trial. Do not include customer data."
            ),
        }
    ],
    temperature=0,
    max_tokens=700,
)

print({
    "model": response.model,
    "usage": response.usage.model_dump() if response.usage else None,
    "pricing_version": trial.pricing_version,
})

The output should not be judged only by whether it looks good. Record model, usage fields, stop reason, pricing version, checked time, and reviewer decision. If a fallback route runs, record the fallback reason and the second model too.

Procurement should get evidence, not a slogan

Procurement does not need every prompt. It does need evidence that the team knows what it tested.

Give procurement a short packet:

Evidence	Why it belongs
Pricing source URL and checked time	Shows the row was not copied from memory
Pricing version	Lets support and finance reproduce the catalog context
Allowed model IDs	Blocks silent route swaps
Effective account group	Explains account-specific billing behavior
Redacted prompt set	Shows the test did not require sensitive data
Token buckets	Separates input, cached input, and output where available
Retry and tool policy	Explains extra calls
Stop conditions	Shows when the team would reject rollout

This packet is more useful than a broad statement that the gateway is easy to use. Easy setup is not the same as a controlled trial.

Source links to keep close

Keep the sources beside the trial code:

Source	Use
AIWave pricing	Public gateway pricing context
`https://aiwave.live/api/pricing`	Machine-readable pricing snapshot
AIWave models docs	Route and catalog context
AIWave trust page	Public trust and operational context
DeepSeek pricing	Provider pricing and context fields
QwenCloud pricing	Context, tool, and failed-call billing behavior

Recheck these sources before changing rollout policy. A pricing row, model name, cache behavior, or provider feature can change after a successful test.

Final fit checklist

Start the AIWave trial only when the team can answer these questions:

What data is allowed in the prompt?
Which model IDs can run?
Which pricing version admitted the trial?
What output and retry limits apply?
Which tool calls are allowed?
What budget stops the test?
Which evidence will procurement and support review?
Which condition would block rollout?

If those answers are missing, the trial is not ready. Stop before the first request, fix the operating boundary, then run a small redacted test with dated pricing evidence.

That is the difference between "the SDK worked on my laptop" and a trial your finance, security, and platform teams can actually inspect.

Pin a Pricing Snapshot Before an AI Agent Run

Mattias chaw — Mon, 31 Aug 2026 13:14:05 +0000

Pin a Pricing Snapshot Before an AI Agent Run

Long-running AI agents do not fail like normal API calls.

A chat completion either returns or it does not. An agent run may plan, search, call tools, retry a failing route, summarize intermediate state, and then ask another model to write the final answer. By the time the user sees a result, the original request may have turned into ten model calls across two or three routes.

That is useful when the work is complex. It is also a billing problem if the system does not know which prices and limits were in force when the run started.

The fix is to treat pricing as an admission input, not a page someone checks after the invoice looks strange. Before an agent run begins, the gateway should pin a pricing snapshot, calculate a budget envelope, record the selected billing group, and attach those facts to every model call that belongs to the run.

This article shows one practical pattern for doing that in an OpenAI-compatible gateway.

The problem with late pricing lookup

Most teams start with a simple model wrapper:

response = client.chat.completions.create(
    model="deepseek-v4-pro",
    messages=messages,
)

That wrapper is enough for a demo. It is not enough for a production agent.

An agent run has several cost states:

State	What can change
Planning	The model may read a large reusable prefix and decide which tools to call.
Tool use	Search, retrieval, code execution, or browser steps can add context.
Retry	A failed call may be retried on the same model or a fallback route.
Compression	The agent may summarize history before the next step.
Final answer	A stronger route may write the final response.

If pricing is fetched only when the invoice is rendered, the system can accidentally explain yesterday's run with today's catalog. That becomes more likely when providers expose separate buckets for uncached input, cached input, output, context creation, tool use, and group-specific multipliers.

The request needs a durable answer to a plain question: which pricing facts admitted this run?

Use a dated gateway snapshot

Before writing this article, I checked AIWave's public pricing endpoint on 2026-08-31 at 21:11 Asia/Shanghai time. The endpoint returned success=true, pricing_version=a42d372ccf0b5dd13ecf71203521f9d2, 63 model records, auto_groups=["default"], and group ratios of default=3 and vip=1.

The same snapshot exposed route fields for models such as:

Route	`model_ratio`	`completion_ratio`	`cache_ratio`	Enabled groups
`deepseek-v4-flash`	`0.319`	`3`	`0.0318`	`default`, `vip`, `svip`
`deepseek-v4-pro`	`0.957`	`3`	`0.0333`	`default`, `vip`, `svip`
`glm-5.1`	`1.05`	`3.142857`	`0.32381`	`default`, `vip`, `svip`
`kimi-k3`	`2.25`	`5`	`0.2`	`default`, `vip`, `svip`

Those fields are useful because they are machine-readable. A gateway can use them before routing, and support can use the same version later when reviewing a run.

This does not mean every article, dashboard, or invoice should invent a new public price table. For public claims, keep a dated source link beside the number. For internal admission control, store the snapshot version and the fields used by the policy.

Admission control should happen before planning

Agent systems often run a planner first, then decide what work to do. That planner still costs money. It can also load a large prompt prefix.

Admission control should happen before the planner call:

Fetch or load a fresh enough pricing snapshot.
Resolve the user's effective billing group.
Estimate the starting context size.
Choose allowed routes for the run.
Calculate a budget envelope.
Create an agent_run record with the snapshot version.
Reject, downscope, or ask for confirmation if the run cannot fit.

The point is not to predict every token exactly. The point is to define the cost boundary before the agent starts making decisions.

A compact run record

Use one run record and many request records. The run record pins policy. The request records describe what happened.

create table agent_run (
  run_id text primary key,
  account_id_hash text not null,
  effective_group text not null,
  pricing_version text not null,
  pricing_source text not null,
  pricing_checked_at text not null,
  budget_usd_limit real not null,
  budget_token_limit integer not null,
  status text not null,
  started_at text not null,
  finished_at text
);

create table agent_model_call (
  call_id text primary key,
  run_id text not null references agent_run(run_id),
  parent_call_id text,
  attempt integer not null,
  purpose text not null,
  requested_model text not null,
  resolved_model text not null,
  input_tokens integer,
  cache_hit_tokens integer,
  output_tokens integer,
  status text not null,
  provider_error_code text,
  created_at text not null
);

The purpose field should use boring values: plan, tool_summary, retrieval_answer, retry, compression, or final_answer. Avoid a blended "agent cost" row. It hides the behavior you will need to debug.

Fetch the snapshot once

The gateway does not need to call pricing on every model call inside a run. It needs a snapshot that is current under your policy.

import datetime as dt
import json
import urllib.request

PRICING_URL = "https://aiwave.live/api/pricing"


def utc_now() -> str:
    return dt.datetime.now(dt.timezone.utc).isoformat()


def fetch_pricing_snapshot() -> dict:
    req = urllib.request.Request(
        PRICING_URL,
        headers={"User-Agent": "agent-admission-example"},
    )
    with urllib.request.urlopen(req, timeout=20) as response:
        payload = json.loads(response.read().decode("utf-8"))

    if payload.get("success") is not True:
        raise RuntimeError("pricing endpoint did not return success=true")

    rows = payload.get("data") or []
    return {
        "source": PRICING_URL,
        "checked_at": utc_now(),
        "pricing_version": payload.get("pricing_version"),
        "group_ratio": payload.get("group_ratio") or {},
        "rows_by_model": {row["model_name"]: row for row in rows},
    }

You can cache this result for a short window. The important part is that the run stores the version it used, not just the model name.

Estimate with token buckets

Agent budgets should keep input, cached input, and output separate. A single blended estimate is usually where surprises enter the system.

def estimate_call_units(
    *,
    model_row: dict,
    group_ratio: float,
    uncached_input_tokens: int,
    cached_input_tokens: int,
    output_tokens: int,
) -> float:
    model_ratio = float(model_row["model_ratio"])
    completion_ratio = float(model_row["completion_ratio"])
    cache_ratio = float(model_row.get("cache_ratio") or 0)

    return (
        uncached_input_tokens * model_ratio * group_ratio
        + cached_input_tokens * cache_ratio * group_ratio
        + output_tokens * completion_ratio * group_ratio
    )

This example returns relative billable units because gateway catalogs often express model ratios before invoice conversion. Your production version can convert those units into USD, credits, quota, or invoice rows. Keep the source unit explicit in the field name.

Admit the run

The admission function should be small enough to test.

import hashlib
import os
import uuid


def digest_account(account_id: str) -> str:
    salt = os.environ["AUDIT_HASH_SALT"]
    return hashlib.sha256((salt + account_id).encode("utf-8")).hexdigest()[:20]


def admit_agent_run(
    *,
    account_id: str,
    effective_group: str,
    requested_models: list[str],
    estimated_uncached_tokens: int,
    estimated_cached_tokens: int,
    estimated_output_tokens: int,
    budget_token_limit: int,
    budget_usd_limit: float,
    snapshot: dict,
) -> dict:
    group_ratio = float(snapshot["group_ratio"][effective_group])
    rows_by_model = snapshot["rows_by_model"]

    missing = [model for model in requested_models if model not in rows_by_model]
    if missing:
        raise ValueError(f"unknown model route: {missing[0]}")

    estimates = {}
    for model in requested_models:
        estimates[model] = estimate_call_units(
            model_row=rows_by_model[model],
            group_ratio=group_ratio,
            uncached_input_tokens=estimated_uncached_tokens,
            cached_input_tokens=estimated_cached_tokens,
            output_tokens=estimated_output_tokens,
        )

    return {
        "run_id": str(uuid.uuid4()),
        "account_id_hash": digest_account(account_id),
        "effective_group": effective_group,
        "pricing_version": snapshot["pricing_version"],
        "pricing_source": snapshot["source"],
        "pricing_checked_at": snapshot["checked_at"],
        "requested_models": requested_models,
        "estimated_billable_units_by_model": estimates,
        "budget_token_limit": budget_token_limit,
        "budget_usd_limit": budget_usd_limit,
        "status": "admitted",
    }

The example uses a hash for the account id and keeps credentials out of the record. Do the same for token ids, workspace ids, and support links.

Route every call through the run

Once the run exists, every model call should inherit its pricing policy.

from openai import OpenAI

client = OpenAI(
    base_url="https://aiwave.live/v1",
    api_key=os.environ["AIWAVE_API_KEY"],
)


def create_model_call(run: dict, *, purpose: str, model: str, messages: list[dict]):
    if model not in run["requested_models"]:
        raise ValueError(f"model {model} was not admitted for this run")

    response = client.chat.completions.create(
        model=model,
        messages=messages,
        temperature=0,
        max_tokens=800,
    )

    usage = response.usage
    return {
        "run_id": run["run_id"],
        "purpose": purpose,
        "resolved_model": model,
        "pricing_version": run["pricing_version"],
        "input_tokens": getattr(usage, "prompt_tokens", None),
        "output_tokens": getattr(usage, "completion_tokens", None),
        "status": "success",
    }

The wrapper blocks a quiet route change. If the planner wants to use a model that was not admitted, it has to ask the admission layer for a new run policy or fail closed.

Retries need a separate rule

Retries should not consume an unlimited share of the budget just because the agent framework is trying to be helpful.

Define retry rules against the run budget:

Retry case	Policy
Provider timeout before output	Retry once on the same route if the remaining budget allows it.
Provider error after partial output	Record the partial call and require an explicit continuation policy.
Context overflow	Do not retry blindly. Compress or reject with a clear reason.
Route unavailable	Use only fallback routes admitted at run start.
Tool failure	Do not spend model tokens until the tool error is classified.

Store retries as model calls with parent_call_id and attempt. That keeps retry spend visible.

select
  run_id,
  resolved_model,
  count(*) as calls,
  sum(case when attempt > 1 then 1 else 0 end) as retry_calls,
  sum(coalesce(input_tokens, 0)) as input_tokens,
  sum(coalesce(output_tokens, 0)) as output_tokens
from agent_model_call
where created_at >= datetime('now', '-1 day')
group by run_id, resolved_model;

This query gives engineering and finance the same view of the run. If retry calls are high, the issue may be timeout policy, SDK behavior, or provider stability, not the rate card.

What the user should see

Most users do not need the full event log. They do need enough evidence to trust the result.

A usage page for an agent run should show:

Field	Why it helps
Run status	Completed, interrupted, failed, or stopped by budget
Models used	Shows actual routes, not only the requested family
Pricing snapshot	Gives a dated source for the calculation
Token buckets	Separates input, cached input, and output where available
Retry count	Explains extra calls without hiding them
Budget result	Shows whether the run stopped inside the approved envelope

This is especially important for teams evaluating a multi-model gateway from a Tier 1 or Tier 2 market. They may be fine with a flexible route. They are usually not fine with an invoice that cannot explain which route, group, and token bucket produced the charge.

What not to store

An admission record is not a transcript archive.

Do not store raw API keys, full prompts, full completions, payment details, or customer-identifying data in the billing trail. Store ids, hashes, route names, pricing versions, token counts, statuses, and source timestamps. If support needs prompt-level inspection for a specific case, put that behind a separate access-controlled workflow with a short retention window.

This boundary matters during procurement. A stronger answer is not "we keep everything." A stronger answer is: the gateway keeps enough billing evidence to explain the run while keeping content and credentials out of the accounting log.

Operational checks

Run these checks daily:

Check	Failure signal
Missing pricing version	A run cannot be recalculated later.
Stale snapshot	Admission used a catalog older than policy allows.
Unknown group	The effective group was not present in the snapshot.
Unadmitted route	A call used a model outside the run policy.
Retry overspend	Retry calls consumed more than the allowed budget share.
Missing usage	A successful call returned no usage fields.
Secret pattern	A key-like string entered logs or article drafts.

The last check belongs in content pipelines too. A technical article can show environment-variable usage, but it should never publish a real key or a customer-specific trace.

Source links to keep with the runbook

Keep dated source links close to the admission code:

Source	Use
AIWave pricing	Public gateway pricing and route context
`https://aiwave.live/api/pricing`	Machine-readable snapshot for admission
AIWave model docs	Route names and model catalog context
DeepSeek pricing	Provider bucket calibration for DeepSeek routes
QwenCloud pricing	Context and tool billing behavior

Recheck sources before changing policy. Do not assume a provider's rate card, cache bucket, or tool billing rule stayed the same because yesterday's tests passed.

Implementation checklist

Before your agent planner sends its first model call, the gateway should know:

Which pricing snapshot admitted the run.
Which effective billing group applies.
Which model routes are allowed.
Which budget envelope the run must stay inside.
How retries consume that envelope.
Where token buckets will be recorded.
Which evidence can be shown to support without exposing secrets.

This is a small amount of structure, but it changes the failure mode. Instead of explaining an agent bill after the fact, the gateway can say what it allowed, what actually ran, and which dated pricing source tied those two facts together.

For high-volume agent workloads, that is the difference between a flexible gateway and an auditable one.

Build a Streaming Usage Ledger for OpenAI-Compatible AI Gateways

Mattias chaw — Sun, 30 Aug 2026 13:10:31 +0000

Build a Streaming Usage Ledger for OpenAI-Compatible AI Gateways

Streaming chat completions make AI products feel fast. They also make billing and reliability harder to explain.

In a normal request, the client sends a prompt, the server waits, and one final JSON response carries the generated text plus usage metadata. A ledger can record one row after the response finishes.

Streaming changes the shape of the problem. The user may see 200 tokens before the model times out. A mobile client may disconnect after receiving a useful answer. A retry may start while the first request is still flushing chunks. Some upstream providers send final usage data in the last event; others require you to calculate or reconcile it later. If your gateway supports DeepSeek, GLM, Kimi, Qwen, MiniMax, and other OpenAI-compatible routes, the accounting problem becomes a distributed systems problem.

This article shows a practical pattern: a streaming usage ledger that records what was requested, what was emitted, what was acknowledged by the client, what was billed by the provider, and what should be shown to the user.

The goal is not to make streaming complicated. The goal is to make partial answers auditable.

Why streaming needs its own ledger

Most teams start with one usage table:

Field	Meaning
request_id	Application request identifier
user_id	Internal user or workspace
model	Selected model
input_tokens	Prompt tokens
output_tokens	Final completion tokens
cost_usd	Calculated request cost
status	success or failure

That table is useful for non-streaming calls, but it loses important facts when the response is streamed.

For example, imagine a coding agent that asks a model to rewrite a file. The model streams 900 output tokens. The browser receives 760 tokens, then the user closes the tab. The upstream provider continues for another second and reports 940 output tokens in the final usage frame. Your application retries the same task on a faster model because the client never received the final done event.

If you only store one final row, you cannot answer these questions:

Question	Why it matters
Did the user receive useful output?	Support and refund reviews need evidence.
Did the provider finish after the client disconnected?	Gateway margins depend on upstream work, not only browser delivery.
Was the retry necessary?	Agent frameworks often retry aggressively after stream interruptions.
Which model was actually charged?	Routing can switch from planning to execution models.
Which pricing version was used?	Rate cards and group multipliers change over time.

A streaming ledger keeps these facts separate instead of forcing them into one status field.

Pricing facts must be versioned

Before designing the ledger, capture the pricing snapshot used for the calculation.

For this article, I checked AIWave live pricing on 2026-08-30. The pricing API returned success=true, pricing_version=a42d372ccf0b5dd13ecf71203521f9d2, 63 model records, auto_groups=["default"], and group ratios of default=3 and vip=1. Target OpenAI-compatible rows included deepseek-v4-flash, deepseek-v4-pro, kimi-k3, glm-5, glm-5.1, Qwen 3.5/3.6/3.7 routes, and MiniMax M2/M3 routes.

That is the level of specificity a ledger needs. It does not need to quote every public provider rate in the hot path. It does need to store enough metadata to reproduce the calculation later:

Ledger field	Example
pricing_version	`a42d372ccf0b5dd13ecf71203521f9d2`
billing_group	`default` or `vip`
group_ratio	`3` or `1`
model_ratio	Model row ratio at request time
completion_ratio	Output multiplier at request time
cache_ratio	Cached-input multiplier when exposed
source_checked_at	`2026-08-30T21:00:00+08:00`

Without those fields, a support engineer reviewing a request next week may accidentally recalculate it with today's rates instead of the rates that were active when the stream ran.

Event model

Use one request row and many event rows.

The request row is stable:

create table ai_request (
  request_id text primary key,
  user_id text not null,
  route text not null,
  requested_model text not null,
  resolved_model text not null,
  billing_group text not null,
  pricing_version text not null,
  status text not null,
  started_at text not null,
  finished_at text
);

The event rows describe what happened:

create table stream_event (
  event_id text primary key,
  request_id text not null references ai_request(request_id),
  sequence integer not null,
  event_type text not null,
  output_chars integer not null default 0,
  output_tokens_estimate integer not null default 0,
  provider_input_tokens integer,
  provider_output_tokens integer,
  error_code text,
  created_at text not null
);

Keep event_type boring and explicit:

Event type	Meaning
`request_started`	Gateway accepted the request.
`provider_connected`	Upstream connection opened.
`chunk_received`	Gateway received streamed content.
`chunk_delivered`	Gateway wrote content to the client socket.
`client_disconnected`	Client closed the connection before done.
`provider_usage`	Provider reported final token usage.
`retry_started`	Application or gateway began a retry.
`request_finished`	Gateway closed the request lifecycle.

This structure avoids a common mistake: treating streamed text as the same thing as billable usage. Text delivery is a user-experience event. Provider usage is a billing event. They often line up, but they are not the same fact.

Minimal Python implementation

Here is a small SQLite-backed ledger. It is intentionally plain so you can drop it into a FastAPI, Flask, or background-worker codebase.

import os
import sqlite3
import time
import uuid
from contextlib import contextmanager

DB_PATH = os.environ.get("LEDGER_DB_PATH", "stream-ledger.sqlite3")


@contextmanager
def db():
    con = sqlite3.connect(DB_PATH)
    con.row_factory = sqlite3.Row
    try:
        yield con
        con.commit()
    finally:
        con.close()


def now_iso():
    return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())


def init_schema():
    with db() as con:
        con.executescript("""
        create table if not exists ai_request (
          request_id text primary key,
          user_id text not null,
          route text not null,
          requested_model text not null,
          resolved_model text not null,
          billing_group text not null,
          pricing_version text not null,
          status text not null,
          started_at text not null,
          finished_at text
        );

        create table if not exists stream_event (
          event_id text primary key,
          request_id text not null references ai_request(request_id),
          sequence integer not null,
          event_type text not null,
          output_chars integer not null default 0,
          output_tokens_estimate integer not null default 0,
          provider_input_tokens integer,
          provider_output_tokens integer,
          error_code text,
          created_at text not null
        );
        """)


def start_request(user_id, route, requested_model, resolved_model, group, pricing_version):
    request_id = str(uuid.uuid4())
    with db() as con:
        con.execute(
            """
            insert into ai_request
            (request_id, user_id, route, requested_model, resolved_model,
             billing_group, pricing_version, status, started_at)
            values (?, ?, ?, ?, ?, ?, ?, ?, ?)
            """,
            (
                request_id,
                user_id,
                route,
                requested_model,
                resolved_model,
                group,
                pricing_version,
                "streaming",
                now_iso(),
            ),
        )
    add_event(request_id, 0, "request_started")
    return request_id


def add_event(request_id, sequence, event_type, **fields):
    with db() as con:
        con.execute(
            """
            insert into stream_event
            (event_id, request_id, sequence, event_type, output_chars,
             output_tokens_estimate, provider_input_tokens, provider_output_tokens,
             error_code, created_at)
            values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
            """,
            (
                str(uuid.uuid4()),
                request_id,
                sequence,
                event_type,
                fields.get("output_chars", 0),
                fields.get("output_tokens_estimate", 0),
                fields.get("provider_input_tokens"),
                fields.get("provider_output_tokens"),
                fields.get("error_code"),
                now_iso(),
            ),
        )


def finish_request(request_id, status):
    with db() as con:
        con.execute(
            "update ai_request set status = ?, finished_at = ? where request_id = ?",
            (status, now_iso(), request_id),
        )

Capturing a stream

The gateway should log both provider chunks and client delivery. That distinction matters when a socket write fails.

def estimate_tokens(text):
    # Replace with your tokenizer for production. This estimate is for interim
    # progress only; final provider usage should override it when available.
    return max(1, len(text) // 4)


def stream_to_client(request_id, provider_events, client_writer):
    delivered_chars = 0

    for sequence, event in enumerate(provider_events, start=1):
        if event["type"] == "content":
            text = event["text"]
            add_event(
                request_id,
                sequence,
                "chunk_received",
                output_chars=len(text),
                output_tokens_estimate=estimate_tokens(text),
            )

            try:
                client_writer.write(text)
                delivered_chars += len(text)
                add_event(
                    request_id,
                    sequence,
                    "chunk_delivered",
                    output_chars=len(text),
                    output_tokens_estimate=estimate_tokens(text),
                )
            except BrokenPipeError:
                add_event(request_id, sequence, "client_disconnected")
                finish_request(request_id, "client_disconnected")
                return

        if event["type"] == "usage":
            add_event(
                request_id,
                sequence,
                "provider_usage",
                provider_input_tokens=event["input_tokens"],
                provider_output_tokens=event["output_tokens"],
            )

    finish_request(request_id, "completed")

For a real gateway, also record the upstream request id when the provider exposes one. It gives your finance, support, and engineering teams a shared join key.

Reconciliation query

At the end of the request, build a reconciled view. This query compares delivered output with provider-reported usage.

select
  r.request_id,
  r.user_id,
  r.resolved_model,
  r.billing_group,
  r.pricing_version,
  r.status,
  sum(case when e.event_type = 'chunk_delivered' then e.output_chars else 0 end)
    as delivered_chars,
  sum(case when e.event_type = 'chunk_delivered' then e.output_tokens_estimate else 0 end)
    as delivered_tokens_estimate,
  max(e.provider_input_tokens) as provider_input_tokens,
  max(e.provider_output_tokens) as provider_output_tokens,
  sum(case when e.event_type = 'client_disconnected' then 1 else 0 end)
    as client_disconnects
from ai_request r
join stream_event e on e.request_id = r.request_id
where r.started_at >= datetime('now', '-1 day')
group by r.request_id;

The dashboard should display both numbers:

Metric	Source	Use
Delivered tokens estimate	Gateway chunk events	User-visible progress and UX debugging
Provider output tokens	Final provider usage	Billing and margin calculation
Client disconnect count	Gateway socket events	Retry and support analysis
Pricing version	Pricing snapshot	Audit and recalculation

Do not hide the mismatch. The mismatch is the evidence.

Retry rules

Retries are where streaming ledgers become valuable. A retry after zero delivered tokens is different from a retry after 900 delivered tokens.

Use policy bands:

Delivered output	Retry policy
0 tokens	Safe to retry automatically within the user budget.
Small prefix only	Retry with explicit `previous_attempt_id`.
Useful partial answer	Ask the product layer whether to continue, summarize, or stop.
Final answer delivered but done event missing	Do not blindly retry; reconcile first.

The retry row should reference the previous attempt:

create table request_retry (
  retry_id text primary key,
  previous_request_id text not null,
  next_request_id text not null,
  reason text not null,
  created_at text not null
);

This lets you answer a critical question: "How much of our monthly model spend comes from retries after partial streamed responses?"

If the answer is high, you may not have a pricing problem. You may have an SDK timeout problem, a proxy buffering problem, or an agent framework that interprets missing done events too aggressively.

What to show users

A user-facing usage page should not expose every internal event, but it should show enough detail to build trust.

For each streamed request, show:

User-facing field	Example
Model	`deepseek-v4-pro`
Status	Completed, interrupted, or retried
Input tokens	Provider-reported count when available
Output tokens	Provider-reported count when available
Delivered output	Present only for interrupted streams
Billing group	Default, VIP, or contract group
Rate snapshot	Pricing version and date

For AIWave-style gateways, this is especially important because teams may switch between Chinese AI model families behind one OpenAI-compatible client. A single USD invoice is much easier to trust when each row can explain the route, model, group, and pricing snapshot used at the time.

Operational checks

Add these alerts before traffic grows:

Alert	Why
Provider usage missing on completed streams	Billing will fall back to estimates.
High disconnect rate by region or client version	SDK or network issue.
Retry spend above threshold	Agent loops may be wasting budget.
Pricing version missing	Request cannot be audited later.
Delivered tokens much lower than provider tokens	Client is not receiving what upstream produced.

None of these alerts require customer-identifying data in the log. Store internal ids, request ids, and aggregated counters. Keep API keys out of events entirely.

Data retention boundaries

The ledger should be useful without becoming a transcript archive.

For most teams, the event table should store counts, states, ids, and route metadata. It should not store full prompts, full streamed text, secrets, payment details, or customer contact information. If support needs to inspect an individual incident, link the ledger row to a short-lived support case with explicit access controls instead of putting sensitive content into the accounting database.

That boundary also helps with procurement. A Tier 1 buyer may ask how a gateway handles data retention, billing evidence, and incident review. The stronger answer is not "we store everything." The stronger answer is: "we store the minimum evidence needed to explain usage, we separate billing events from content, and we can delete or aggregate old event rows while preserving invoice totals."

A simple retention policy works well:

Data class	Suggested treatment
Request and usage totals	Keep for finance and tax records.
Stream event metadata	Keep for a shorter operational window.
Provider request ids	Keep while support disputes are possible.
Prompt or completion text	Do not store in the ledger.

This keeps the ledger focused on accounting, not surveillance.

Implementation checklist

If you are adding streaming to an OpenAI-compatible gateway, ship the ledger with the stream, not after the first billing dispute.

Create a request row before opening the upstream stream.
Record the resolved model, billing group, and pricing version at request start.
Log provider chunk receipt separately from client socket delivery.
Treat final provider usage as the billing source when it is available.
Keep interim token estimates labeled as estimates.
Link retries to previous attempts.
Make interrupted streams visible in support tooling.
Alert when completed streams do not contain provider usage.

Streaming is a product feature, but partial streaming is an accounting state. Once you model it that way, retries become explainable, support reviews become faster, and users can see why a request was billed the way it was.

That is the difference between "the model streamed something" and "the gateway can prove what happened."

Build a Prompt Prefix Registry for Long-Context AI Gateways

Mattias chaw — Sat, 29 Aug 2026 13:14:51 +0000

Build a Prompt Prefix Registry for Long-Context AI Gateways

Long-context AI agents do not become predictable just because the model accepts a million tokens.

A coding agent may reuse the same system prompt, repository map, style guide, security policy, tool schema bundle, and task rubric across hundreds of runs. A support agent may reuse the same product manual, escalation policy, and customer-safe answer format. A procurement workflow may reuse the same contract clauses and compliance checklist.

Those repeated blocks are valuable. They are also easy to lose.

If every request assembles the prefix in a slightly different order, inserts timestamps into stable text, changes whitespace, swaps tool schema order, or moves route metadata into the prompt, the gateway cannot reason clearly about cache-hit input. Engineering expects reuse. Finance sees fresh input. Customer support has no evidence for what changed.

A prompt prefix registry fixes the boring part of the problem. It records reusable prompt blocks as named, versioned artifacts. The router can then attach a stable prefix fingerprint to each request, keep cache assumptions out of the prompt body, and reconcile expected cache behavior against actual usage.

This article uses AIWave examples because AIWave exposes OpenAI-compatible routes across Chinese model providers. The pattern is not AIWave-specific. It applies to any gateway routing long-context workloads across DeepSeek, Qwen, GLM, Kimi, ERNIE, MiniMAX, Doubao, StepFun, MiMo, or another provider family.

The Problem Is Prefix Drift

Most teams start with a simple prompt builder:

prompt = system_prompt + "\n\n" + repo_summary + "\n\n" + user_task

That is fine for a prototype. It is not enough for production.

The reusable portion of a request needs its own lifecycle. A stable prefix should answer these questions before the request reaches the model:

Question	Why it matters
Which reusable blocks were included?	Explains what the agent knew before the task
Which exact versions were used?	Makes regression review possible
Was the block eligible for cache planning?	Separates stable context from task-specific context
Which route saw the request?	Cache behavior can differ by provider and model
Which pricing snapshot approved the route?	Keeps billing review tied to a dated source
Did actual usage match the cache assumption?	Finds prompt churn and route switching

Without those fields, a cache miss is just an invoice surprise. With them, it becomes a debuggable event.

Use Dated Pricing Evidence

Before writing this article, I checked AIWave's live pricing endpoint on August 29, 2026. It returned success=true, 63 model records, auto_groups=["default"], and pricing_version=a42d372ccf0b5dd13ecf71203521f9d2. The DeepSeek V4 rows exposed OpenAI-compatible route metadata for deepseek-v4-flash and deepseek-v4-pro, including model_ratio, completion_ratio, cache_ratio, create_cache_ratio, and enabled groups of default, vip, and svip.

That live endpoint is useful gateway evidence. It is not a reason to invent a public USD table. Public price claims need dated source links.

DeepSeek's official pricing page checked on August 29, 2026 lists deepseek-v4-flash and deepseek-v4-pro with cache-hit input, cache-miss input, and output rows. The visible USD table separates peak and off-peak prices. QwenCloud's pricing docs describe pay-as-you-go billing, context-tiered request billing, Batch API discounts, context caching, thinking-token billing, built-in tool fees, and bill-query views. Z.AI's pricing page separates input, cached input, cached input storage, output, and tool-related rows. Kimi's public API pricing page describes long-context billing, context caching, web search, reasoning effort, JSON mode, and OpenAI-compatible API usage, but the fetched page body did not expose every K3 price row, so do not quote K3 amounts without checking the billing console or a current official table.

The important point is structural: modern AI pricing has buckets. A prefix registry helps keep the cache bucket honest.

Registry Shape

The registry does not need to be complex. Start with a table or JSON document that defines reusable blocks.

{
  "prefix_id": "coding-agent-base",
  "version": "2026-08-29.1",
  "owner": "agent-platform",
  "purpose": "Shared instructions for repository analysis and patch generation",
  "cache_eligible": true,
  "content_sha256": "example_prefix_hash",
  "token_estimate": 18400,
  "allowed_routes": [
    "deepseek-v4-flash",
    "deepseek-v4-pro",
    "qwen3.5-27b",
    "glm-5.1"
  ],
  "source_files": [
    "system/coding-agent.md",
    "policy/security-review.md",
    "tools/repo-map-schema.json"
  ],
  "created_at": "2026-08-29T13:05:00Z"
}

Keep the registry separate from the prompt text. The prompt text can live in source control, object storage, or your application database. The registry is the auditable manifest that says which stable blocks were intended to be reused.

For a small gateway, a JSON file committed with the router may be enough. For a larger platform, use a database table with immutable versions and a review workflow.

Request Manifest

Each request should carry a manifest that joins the reusable prefix to the task-specific payload.

{
  "request_id": "req_20260829_001",
  "tenant_id_hash": "tenant_hash_example",
  "model": "deepseek-v4-pro",
  "prefixes": [
    {
      "prefix_id": "coding-agent-base",
      "version": "2026-08-29.1",
      "content_sha256": "example_prefix_hash",
      "cache_eligible": true,
      "token_estimate": 18400
    },
    {
      "prefix_id": "repo-map-aiwave-sdk",
      "version": "2026-08-29.3",
      "content_sha256": "example_repo_hash",
      "cache_eligible": true,
      "token_estimate": 61200
    }
  ],
  "task_tokens_estimate": 2200,
  "max_output_tokens": 1200,
  "pricing_version": "a42d372ccf0b5dd13ecf71203521f9d2",
  "pricing_checked_at": "2026-08-29T13:20:00Z"
}

This manifest gives you an audit trail without logging prompts, API keys, or customer content. You can answer which stable context was expected, which exact route handled the call, and which pricing snapshot approved the dispatch.

Build The Prefix Loader

The loader should produce deterministic text. That means stable ordering, stable separators, and no runtime-only values inside cache-planned blocks.

import hashlib
import json
from dataclasses import dataclass
from pathlib import Path


@dataclass(frozen=True)
class PrefixBlock:
    prefix_id: str
    version: str
    path: Path
    cache_eligible: bool


def normalize_text(text: str) -> str:
    lines = [line.rstrip() for line in text.splitlines()]
    return "\n".join(lines).strip() + "\n"


def digest(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()


def load_prefixes(blocks: list[PrefixBlock]) -> tuple[str, list[dict]]:
    parts: list[str] = []
    manifest: list[dict] = []

    for block in sorted(blocks, key=lambda item: (item.prefix_id, item.version)):
        text = normalize_text(block.path.read_text(encoding="utf-8"))
        parts.append(f"<prefix id=\"{block.prefix_id}\" version=\"{block.version}\">\n{text}</prefix>")
        manifest.append({
            "prefix_id": block.prefix_id,
            "version": block.version,
            "content_sha256": digest(text),
            "cache_eligible": block.cache_eligible,
            "bytes": len(text.encode("utf-8")),
        })

    return "\n\n".join(parts), manifest


prefix_text, prefix_manifest = load_prefixes([
    PrefixBlock("coding-agent-base", "2026-08-29.1", Path("system/coding-agent.md"), True),
    PrefixBlock("repo-map-aiwave-sdk", "2026-08-29.3", Path("context/repo-map.md"), True),
])

print(json.dumps(prefix_manifest, indent=2))

The XML-like wrapper is not required. Use whatever format your model routes handle well. The key is determinism. If the same prefix version produces a different hash tomorrow, the registry should fail the build.

Keep Runtime State Out Of Cache-Planned Text

Some values change on every request. They should not be placed inside the stable prefix:

Runtime value	Better location
Request ID	audit manifest
Current timestamp	request metadata
User task	task payload
Effective billing group	gateway event
Pricing version	route manifest
Retry attempt	dispatch event
Trace ID	observability span

This separation feels strict, but it prevents accidental cache churn. A timestamp inside the stable prefix turns every request into a different prefix. A pricing version inside the prompt can do the same. The model does not need to read billing metadata unless the task is explicitly about billing.

Route With Prefix Awareness

The router should select a model after it sees the prefix manifest and task size. It should not route only on the user's requested model.

from dataclasses import dataclass


@dataclass(frozen=True)
class Route:
    model: str
    max_context_tokens: int
    supports_cache_planning: bool
    freshness_minutes: int


def choose_route(
    routes: list[Route],
    *,
    requested_model: str,
    prefix_tokens: int,
    task_tokens: int,
    output_cap: int,
) -> Route:
    required = prefix_tokens + task_tokens + output_cap
    candidates = [
        route for route in routes
        if route.model == requested_model and route.max_context_tokens >= required
    ]
    if not candidates:
        raise ValueError(f"no route can fit {required} tokens for {requested_model}")
    return candidates[0]


route = choose_route(
    [
        Route("deepseek-v4-flash", 1_000_000, True, 30),
        Route("deepseek-v4-pro", 1_000_000, True, 30),
        Route("glm-5.1", 128_000, True, 30),
    ],
    requested_model="deepseek-v4-pro",
    prefix_tokens=79_600,
    task_tokens=2_200,
    output_cap=1_200,
)

This example is intentionally small. A real router may also use latency class, allowed provider families, regional policy, tool support, retry policy, and account group. The prefix registry gives that router one missing input: the size and identity of reusable context.

Call The OpenAI-Compatible Endpoint

Once the route is chosen, the request can use a normal OpenAI-compatible client.

import os
from openai import OpenAI


client = OpenAI(
    base_url="https://aiwave.live/v1",
    api_key=os.environ["AIWAVE_API_KEY"],
)


def run_agent(prefix_text: str, task: str, model: str):
    return client.chat.completions.create(
        model=model,
        temperature=0,
        max_tokens=1200,
        messages=[
            {"role": "system", "content": prefix_text},
            {"role": "user", "content": task},
        ],
    )

The example keeps credentials in environment variables. It does not print the key. It does not log the prompt. In production, attach the request manifest to your tracing and billing events, not to the model-visible prompt unless the model truly needs it.

Reconcile Expected And Actual Usage

The registry earns its keep after the response returns.

def build_reconciliation_event(response, request_manifest: dict) -> dict:
    usage = getattr(response, "usage", None)
    prompt_tokens = getattr(usage, "prompt_tokens", None)
    completion_tokens = getattr(usage, "completion_tokens", None)

    expected_prefix_tokens = sum(
        item["token_estimate"]
        for item in request_manifest["prefixes"]
        if item["cache_eligible"]
    )

    return {
        "request_id": request_manifest["request_id"],
        "model": request_manifest["model"],
        "pricing_version": request_manifest["pricing_version"],
        "prefix_fingerprints": [
            f"{item['prefix_id']}@{item['version']}:{item['content_sha256'][:12]}"
            for item in request_manifest["prefixes"]
        ],
        "expected_cache_eligible_tokens": expected_prefix_tokens,
        "actual_prompt_tokens": prompt_tokens,
        "actual_completion_tokens": completion_tokens,
        "cache_reconciliation_status": "needs_provider_usage_join"
    }

If your provider or gateway returns cached-token usage directly, join it here. If it does not, keep the event anyway. The status makes the evidence gap visible. That is better than silently treating all prompt tokens as either cached or uncached.

Failure Modes To Alert On

The daily check should be small and specific.

Check	Failure signal
Prefix hash drift	Same `prefix_id` and version now produce a different hash
Prefix order drift	The same block set serializes in a different order
Cache expectation drift	Cache-eligible token estimate changes without a new version
Route mismatch	A prefix approved for one route is sent to another
Pricing drift	Request uses a stale `pricing_version` beyond policy
Usage gap	Cache-capable route has no cached-token evidence
Prompt logging risk	Raw prefix or user task appears in billing logs

These checks are cheap to run. They also create the evidence a Tier 1/2 buyer expects during procurement: not a broad promise, but a reproducible control.

Rollout Plan

Start with one high-volume workflow. Pick a coding agent, support triage agent, or document analysis agent that reuses large context. Do not start with every route.

First, put the stable blocks under registry control and compute fingerprints in shadow mode. Keep dispatch behavior unchanged. Compare prefix hashes for a week. If hashes drift daily, fix the builder before talking about cache optimization.

Second, attach request manifests to traces. Store prefix IDs, versions, hashes, model route, pricing version, and token estimates. Do not store raw customer prompts.

Third, block only obvious mistakes: missing prefix version, stale pricing snapshot, raw key pattern in prompt logs, or a route that cannot fit the planned context. These checks protect reliability without requiring perfect cache accounting.

Fourth, reconcile usage. Join actual prompt, output, and cached-token fields where available. Segment by workflow, model, account group, and prefix version. If a new prefix version increases fresh input materially, treat it like a production change.

Finally, use the registry in customer-facing support. When a buyer asks why a long-context run cost what it did, you should be able to answer with route, prefix versions, token buckets, pricing source, and request outcome. Do not ask support to reverse-engineer it from one blended usage number.

Source Links To Keep Beside The Registry

Keep source links in metadata, not inside the reusable prompt body.

For gateway routes, keep dated checks for AIWave pricing, AIWave model docs, and the live pricing endpoint at https://aiwave.live/api/pricing.

For provider calibration, keep dated checks for DeepSeek pricing, QwenCloud pricing, Z.AI pricing, and Kimi API pricing. Recheck them before changing a route policy.

Final Checklist

Before you call a long-context route in production, the gateway should know:

Question	Pass condition
Is the reusable context versioned?	Every stable block has an ID, version, and hash
Is the prompt deterministic?	Same versions produce the same serialized prefix
Is runtime state separate?	Request IDs, timestamps, pricing versions, and trace IDs stay out of the stable prefix
Is the route compatible?	Context size and allowed model routes are checked before dispatch
Is pricing dated?	The request stores the pricing snapshot used for admission
Is cache evidence visible?	Expected and actual cache behavior can be reconciled
Are secrets excluded?	Keys, prompts, and customer content are not written to billing logs

A long context window gives an agent room to work. A prompt prefix registry gives the gateway a way to prove what stayed stable, what changed, and which request should have received cache treatment.

That proof is what turns a clever prompt builder into production infrastructure.