When an AI feature breaks in production, the first question is usually too broad: "What happened?" The better question is smaller and much more useful: "Can we replay the shape of the failed request without copying the user's prompt, API key, response body, or account details into a debug channel?"
That is the job of a redacted request fixture.
For teams in the United States, the United Kingdom, Germany, Singapore, Japan, Canada, Australia, and other Tier 1 or Tier 2 markets, AI gateway incidents often involve more than one model route. A single OpenAI-compatible client may send traffic to DeepSeek, GLM, Kimi, Qwen, MiniMax, ERNIE, Doubao, StepFun, or MiMo through one base URL. The SDK call looks familiar, but the operational evidence behind it is wider: model ID, account group, dated pricing row, cache-hit treatment, timeout class, retry policy, and gateway error code.
If that evidence only exists inside raw logs, every incident review becomes risky. Engineers need the request shape. Support needs enough detail to explain the next step. Finance may need the pricing snapshot. Security needs proof that no secret material was copied into a ticket. A redacted fixture gives each group the narrow slice it needs.
Disclosure: this article is from AIWave. I use AIWave as the running example because it exposes OpenAI-compatible routes, public dated pricing, and request-level billing concepts. The fixture pattern is general enough to use with any gateway or internal model router.
What a fixture should prove
A fixture is not a transcript. It is not a log dump. It is a compact object that proves the operational facts needed to replay or reason about an incident.
For an AI API gateway, the fixture should answer six questions:
| Question | Fixture field |
|---|---|
| Which client path was used? | SDK name, base URL, endpoint family |
| Which route was requested? | Model ID, route policy, account group label |
| What was the request shape? | Token estimate, message count, tool count, stream flag |
| What happened? | Status, bounded error code, retry class, request ID |
| Which price context applied? | Pricing snapshot ID, checked date, effective date |
| What is safe to replay? | Sanitized request skeleton, not raw prompt content |
That last row is the hard boundary. A fixture can preserve whether the request had a system message, whether tools were present, whether streaming was enabled, and whether the input was roughly 8k or 180k tokens. It should not preserve the user's actual prompt, uploaded documents, API key, email address, account balance, or customer identifier.
The point is not to hide everything. It is to make the safe evidence useful enough that nobody is tempted to paste raw logs into Slack.
Start with an allowlist
Redaction should be allowlist-based. Do not start with a raw payload and try to remove every dangerous field. That creates a long tail of missed secrets: nested tool arguments, webhook URLs, bearer tokens, personal names, email addresses, file names, database IDs, and vendor-specific metadata.
Instead, define the small set of fields that are allowed to survive.
from __future__ import annotations
from dataclasses import dataclass, asdict
from datetime import datetime, timezone
from typing import Any
@dataclass
class SafeFixture:
captured_at: str
endpoint: str
base_url_host: str
sdk: str
model: str
stream: bool
message_count: int
tool_count: int
estimated_input_tokens: int | None
requested_output_limit: int | None
status: int | None
error_code: str | None
retry_class: str | None
request_id: str | None
pricing_snapshot_id: str | None
pricing_checked_date: str | None
model_price_effective_date: str | None
def build_safe_fixture(
*,
payload: dict[str, Any],
response_meta: dict[str, Any],
pricing_meta: dict[str, Any],
) -> dict[str, Any]:
messages = payload.get("messages") or []
tools = payload.get("tools") or []
fixture = SafeFixture(
captured_at=datetime.now(timezone.utc).isoformat(),
endpoint="/v1/chat/completions",
base_url_host="aiwave.live",
sdk="openai-python",
model=str(payload.get("model", "unknown")),
stream=bool(payload.get("stream", False)),
message_count=len(messages),
tool_count=len(tools),
estimated_input_tokens=response_meta.get("estimated_input_tokens"),
requested_output_limit=payload.get("max_tokens"),
status=response_meta.get("status"),
error_code=response_meta.get("error_code"),
retry_class=response_meta.get("retry_class"),
request_id=response_meta.get("request_id"),
pricing_snapshot_id=pricing_meta.get("pricing_version"),
pricing_checked_date=pricing_meta.get("checked"),
model_price_effective_date=pricing_meta.get("effective_date"),
)
return asdict(fixture)
Notice what the fixture does not include. There is no Authorization header. There is no prompt body. There is no response text. There is no customer email. The fixture preserves the shape of the call and the gateway decision context, not the sensitive content.
That makes it much easier to attach a fixture to an issue, a CI replay, or an incident report.
Separate shape from content
A good fixture should keep the skeleton of the request while dropping the content.
For chat completions, preserve the role sequence and approximate size of each message. Replace the actual text with a short descriptor.
def summarize_messages(messages: list[dict[str, Any]]) -> list[dict[str, Any]]:
safe_messages = []
for message in messages:
content = message.get("content")
if isinstance(content, str):
content_kind = "text"
content_chars = len(content)
elif isinstance(content, list):
content_kind = "parts"
content_chars = sum(len(str(part)) for part in content)
else:
content_kind = "unknown"
content_chars = None
safe_messages.append(
{
"role": message.get("role", "unknown"),
"content_kind": content_kind,
"content_chars": content_chars,
}
)
return safe_messages
This is enough to answer useful debugging questions:
- Did the app send a system message?
- Did the user message become unexpectedly large?
- Did a tool result return into the next model call?
- Did streaming change the failure surface?
- Did a retry resend a larger payload than the first attempt?
It is not enough to reconstruct the user's private task. That is the correct trade.
Tool calls need the same treatment. Keep the tool names if they are generic product identifiers. Drop arguments by default. If a tool name itself can reveal a customer or internal system, map it to a stable alias such as tool_1, tool_2, and store the alias map only in a private environment.
Attach dated pricing context
Pricing belongs in the fixture because many AI incidents are really budget, quota, or routing incidents.
I fetched AIWave's public pricing JSON for this run on 2026-09-12. The response reported checked: 2026-09-10, updated_at: 2026-09-10, currency USD, unit per_1m_text_tokens, and pricing version 8c7a0c0b30661ccbc13d142cb54d1e4ae445fe774b2c6fa501080db97c7a3e56. The public endpoint also states that rates are dated base rates and that the effective account group controls the applied multiplier.
Selected rows from that live response included:
| Model ID | Provider | Input per 1M | Cache-hit per 1M | Output per 1M | Effective date |
|---|---|---|---|---|---|
deepseek-v4-pro |
DeepSeek | 1.914 USD | 0.0637362 USD | 5.742 USD | 2026-08-27 |
deepseek-v4-flash |
DeepSeek | 0.638 USD | 0.0202884 USD | 1.914 USD | 2026-08-27 |
glm-5.1 |
GLM | 2.1 USD | 0.680001 USD | 6.5999997 USD | 2026-08-27 |
kimi-k3 |
Kimi | 4.5 USD | 0.9 USD | 22.5 USD | 2026-08-27 |
moonshot-v1-128k |
Kimi | 1.8 USD | not listed | 4.5 USD | 2026-08-27 |
qwen3-max |
Qwen | 1.5621977891181764 USD | not listed | 6.248791156472706 USD | 2026-08-27 |
The fixture should not try to become a full invoice. It should pin the pricing version and the row that was relevant when the request ran. That is enough to distinguish a stale estimate from a current route, and it helps a buyer understand why the same SDK call can have different cost behavior across account groups or model families.
Give every failure a replay class
A redacted fixture becomes more useful when it carries a replay class.
Do not replay everything. Some failures are safe to replay in a test environment. Others should only be inspected.
def replay_class(status: int | None, error_code: str | None) -> str:
if status == 401:
return "do_not_replay_fix_key_source"
if status == 402:
return "do_not_replay_account_state"
if status == 403 and error_code == "insufficient_user_quota":
return "do_not_replay_account_state"
if status == 429:
return "replay_with_small_fixture_and_backoff"
if status in {408, 500, 502, 503, 504}:
return "replay_if_idempotent"
if status is None:
return "inspect_client_side_failure"
return "inspect_before_replay"
The distinction matters. A 429 can often be reproduced with a tiny fixture that checks backoff behavior. A quota failure usually should not be replayed automatically because the account state is the point. A 401 should send the developer back to key loading and base URL configuration, not into repeated calls.
For gateway clients, I like to store both retry_class and replay_class. Retry class is what the live application should do. Replay class is what engineering can safely do after the fact.
Make the fixture runnable in CI
The safest fixture is still useful in tests. You can turn it into a contract test without calling a real model.
def assert_fixture_contract(fixture: dict[str, Any]) -> None:
forbidden_keys = {
"authorization",
"api_key",
"prompt",
"response_body",
"email",
"customer_id",
"messages",
}
lowered = {key.lower() for key in fixture.keys()}
leaked = forbidden_keys & lowered
if leaked:
raise AssertionError(f"unsafe fixture keys: {sorted(leaked)}")
required = {
"captured_at",
"endpoint",
"model",
"status",
"retry_class",
"pricing_snapshot_id",
"pricing_checked_date",
}
missing = required - set(fixture.keys())
if missing:
raise AssertionError(f"missing fixture keys: {sorted(missing)}")
Then add representative fixture files to your SDK tests:
- A valid request that times out upstream.
- A request rejected for quota.
- A request rejected for route policy.
- A 429 with retry timing.
- A streaming request interrupted before final usage arrives.
- A tool request where arguments are dropped.
These tests will not prove model quality. They prove that the client keeps secrets out of fixtures and still preserves enough evidence for an engineer to route the incident.
That is a practical boundary. Most teams do not need a perfect simulator for every provider. They need proof that the next incident will produce a safe, small, structured record instead of a pile of raw logs.
Use source links, not copied assumptions
The fixture should link to public evidence where possible:
- AIWave pricing JSON:
https://aiwave.live/api/v1/pricing - AIWave pricing page:
https://aiwave.live/pricing - AIWave docs:
https://aiwave.live/docs/quickstart - AIWave status page:
https://aiwave.live/status
Do not copy private account details into the fixture. Do not publish internal success rates or customer activity. Do not turn one failed request into a broad reliability claim. The fixture is a narrow engineering artifact, not a marketing statistic.
For teams evaluating Chinese model routes from Tier 1 or Tier 2 markets, that narrowness is a strength. A redacted fixture lets a developer compare OpenAI-compatible request behavior, pricing provenance, and failure handling without handing sensitive content to every debugging surface.
The next time an AI gateway incident happens, the goal should not be a heroic log hunt. It should be a safe fixture, a clear replay class, and a short path from symptom to fix.
Top comments (0)