Build Model Catalog Diff Tests for OpenAI-Compatible API Routes

Most OpenAI-compatible integrations start with a model ID pasted into a quickstart.

That is fine for the first request. It is fragile for the third team that copies the same example into a worker, a notebook, a support bot, and a pricing workbook. After a few releases, the question is no longer "does the API accept this string?" The question becomes:

Does every place that names a model still mean the same thing?

A model catalog diff test answers that question before the mismatch reaches production. It compares the catalog your docs and SDK examples expect with the live route table your gateway can serve. It also checks the pricing snapshot that finance and product teams use for planning.

This is not a benchmark. It does not claim that one model is universally better. It is a contract test for catalog shape, route availability, endpoint support, and dated pricing references.

I will use AIWave as the concrete example because its public pricing snapshot and live route table can be checked without a private account. As of September 20, 2026, https://aiwave.live/api/v1/pricing returned 56 model rows, currency USD, unit per_1m_text_tokens, checked=2026-09-10, updated_at=2026-09-18, and pricing version 83f77abde81ee3a096a672ed959ccc096f5d37a45c177ae8e03229456b5415a5. The live route table at https://aiwave.live/api/pricing returned success=true, 68 route rows, pricing version a42d372ccf0b5dd13ecf71203521f9d2, and group ratios default=1 and vip=0.9.

Those values are intentionally dated. Catalog tests should make source freshness visible instead of turning today's result into a timeless claim.

Why catalog drift hurts

Catalog drift is subtle because each individual system can still look healthy.

Your docs page may list a model ID. Your SDK example may use a similar alias. Your pricing workbook may have a dated row. Your gateway may serve a newer route table. Your product policy may allow only a subset of routes for a workload.

No single mismatch is dramatic. Together they create support tickets that are hard to explain:

Drift surface	Example failure
Docs versus route table	The quickstart names a model that no longer resolves
Pricing snapshot versus live route	The workbook plans against a row that is missing from runtime
SDK fixture versus endpoint support	The example calls an endpoint the route does not support
Product policy versus catalog	A route exists but should not be used for that workload
Group multiplier versus estimate	A key is billed under a different group assumption

The goal of a diff test is not to freeze the catalog. Catalogs move. The goal is to make movement visible, reviewable, and safe to ship.

Define the catalog contract

Start with the smallest contract your team can defend. A useful model row needs more than a name.

{
  "model_id": "deepseek-v4-pro",
  "provider": "DeepSeek",
  "source": "public_pricing_snapshot",
  "pricing_version": "83f77abde81ee3a096a672ed959ccc096f5d37a45c177ae8e03229456b5415a5",
  "effective_date": "2026-08-27",
  "unit": "per_1m_text_tokens",
  "required_endpoints": ["openai"],
  "allowed_groups": ["default", "vip"],
  "status": "expected"
}

That object is intentionally boring. It does not include prompts, customer data, or reusable credentials. It only says what the release believes to be true about a route.

For a production team, I would store this file near SDK examples and documentation fixtures. When a quickstart changes its default model, the contract changes in the same pull request. When a pricing snapshot refreshes, the version and effective date change in a visible review.

Pull the two public sources

The test needs two inputs:

• a source-dated pricing snapshot, used for planning and docs
• a live route table, used for runtime availability and endpoint support

Here is a small Python fetcher. It uses public endpoints and no API key.

from __future__ import annotations

import json
from dataclasses import dataclass
from urllib.request import urlopen


PRICING_SNAPSHOT_URL = "https://aiwave.live/api/v1/pricing"
LIVE_ROUTE_URL = "https://aiwave.live/api/pricing"


def fetch_json(url: str) -> dict:
    with urlopen(url, timeout=30) as response:
        return json.loads(response.read().decode("utf-8"))


@dataclass(frozen=True)
class Sources:
    pricing_snapshot: dict
    live_route_table: dict


def load_sources() -> Sources:
    return Sources(
        pricing_snapshot=fetch_json(PRICING_SNAPSHOT_URL),
        live_route_table=fetch_json(LIVE_ROUTE_URL),
    )

If your internal gateway requires authentication, still keep this pattern: read the API key from an environment variable and never put it in the fixture.

import os

api_key = os.environ.get("AIWAVE_API_KEY")
if not api_key:
    raise RuntimeError("AIWAVE_API_KEY is required for private probes")

The public catalog diff itself does not need that key. It should remain safe to run in CI and to attach to an engineering review.

Normalize before comparing

Do not diff raw JSON. Normalize the rows into the fields your release actually depends on.

def normalize_snapshot(snapshot: dict) -> dict[str, dict]:
    rows = snapshot.get("models", [])
    return {
        row["id"]: {
            "provider": row.get("provider"),
            "input_usd_per_1m_tokens": row.get("input_usd_per_1m_tokens"),
            "cache_hit_usd_per_1m_tokens": row.get("cache_hit_usd_per_1m_tokens"),
            "output_usd_per_1m_tokens": row.get("output_usd_per_1m_tokens"),
            "effective_date": row.get("effective_date"),
        }
        for row in rows
        if row.get("id")
    }


def normalize_live_routes(route_table: dict) -> dict[str, dict]:
    rows = route_table.get("data", [])
    return {
        row["model_name"]: {
            "enable_groups": sorted(row.get("enable_groups", [])),
            "supported_endpoint_types": sorted(row.get("supported_endpoint_types", [])),
            "model_ratio": row.get("model_ratio"),
            "completion_ratio": row.get("completion_ratio"),
            "cache_ratio": row.get("cache_ratio"),
        }
        for row in rows
        if row.get("model_name")
    }

This keeps the test focused. You do not want CI failing because the provider added an unrelated metadata field. You do want CI failing if the route your SDK example uses disappears or stops supporting the expected endpoint type.

Classify differences

Every difference should land in a clear bucket.

def diff_catalog(expected: dict[str, dict], live: dict[str, dict]) -> dict:
    expected_ids = set(expected)
    live_ids = set(live)

    missing_at_runtime = sorted(expected_ids - live_ids)
    new_at_runtime = sorted(live_ids - expected_ids)

    endpoint_mismatch = []
    group_mismatch = []

    for model_id in sorted(expected_ids & live_ids):
        contract = expected[model_id]
        route = live[model_id]

        required_endpoints = set(contract.get("required_endpoints", []))
        live_endpoints = set(route.get("supported_endpoint_types", []))
        if not required_endpoints.issubset(live_endpoints):
            endpoint_mismatch.append(model_id)

        allowed_groups = set(contract.get("allowed_groups", []))
        live_groups = set(route.get("enable_groups", []))
        if allowed_groups and not allowed_groups.issubset(live_groups):
            group_mismatch.append(model_id)

    return {
        "missing_at_runtime": missing_at_runtime,
        "new_at_runtime": new_at_runtime,
        "endpoint_mismatch": endpoint_mismatch,
        "group_mismatch": group_mismatch,
    }

This output is more useful than a boolean pass or fail. A missing runtime route is usually a blocker for examples and product policies. A new runtime route may be a review item. An endpoint mismatch is often a docs or SDK issue. A group mismatch belongs with account and key-management owners.

Decide what blocks a release

Not every diff should block the same way.

BLOCKING_FIELDS = {
    "missing_at_runtime",
    "endpoint_mismatch",
    "group_mismatch",
}


def release_verdict(diff: dict) -> str:
    blockers = {
        field: diff[field]
        for field in BLOCKING_FIELDS
        if diff.get(field)
    }
    if blockers:
        return "block_release"
    if diff.get("new_at_runtime"):
        return "review_new_routes"
    return "pass"

The rule is conservative on purpose. If a model appears in a quickstart, workbook, or route policy and the live route table cannot support the required shape, the release should stop. If the live table adds a model that the release does not use yet, that can be a review note instead of a deployment blocker.

Add freshness checks

Catalog diff tests should also warn when the source is stale or ambiguous.

def validate_source_metadata(snapshot: dict, route_table: dict) -> list[str]:
    warnings = []

    if snapshot.get("currency") != "USD":
        warnings.append("pricing_snapshot_currency_changed")
    if snapshot.get("unit") != "per_1m_text_tokens":
        warnings.append("pricing_snapshot_unit_changed")
    if not snapshot.get("pricing_version"):
        warnings.append("pricing_snapshot_missing_version")
    if route_table.get("success") is not True:
        warnings.append("live_route_table_not_successful")
    if not route_table.get("pricing_version"):
        warnings.append("live_route_table_missing_version")

    group_ratio = route_table.get("group_ratio", {})
    if group_ratio.get("default") != 1:
        warnings.append("default_group_ratio_changed")
    if group_ratio.get("vip") != 0.9:
        warnings.append("vip_group_ratio_changed")

    return warnings

On September 20, 2026, the public data I checked had default=1 and vip=0.9. That is a current-source observation, not a promise about future values. If those values change, the test should force a review of docs, pricing copy, and budget examples.

Keep runtime probes separate

A catalog diff test can tell you that a route is listed and shaped correctly. It cannot prove a full authenticated request path, quota state, streaming behavior, or latency.

Do not overload one test. Use separate probes for:

• unauthenticated public catalog availability
• authenticated smoke calls in a controlled account
• request-level billing receipt checks
• streaming behavior
• timeout and retry behavior
• provider incident response

That separation matters for trust. A public route table may be accurate while an account-level key is misconfigured. A smoke call may pass while pricing copy is stale. A model may be available while a specific endpoint mode is not.

Good engineering evidence keeps those claims apart.

What to publish in a release note

The internal diff can be detailed. The public note should be narrower.

Safe language:

The SDK examples were checked against a dated pricing snapshot and the current route table before release.

Risky language:

Every model is always available and priced the same way.

The first sentence describes a process and a source. The second sentence creates a support burden the catalog cannot honestly carry.

For Tier 1 and Tier 2 teams, this distinction is not cosmetic. It is how you keep docs, budget reviews, and production behavior aligned without exposing internal usage, customer data, or credentials.

A practical checklist

Before shipping an OpenAI-compatible API example or route policy, ask:

• Does every documented model ID exist in the live route table?
• Does the route support the endpoint shape the example uses?
• Does the pricing snapshot have a version, source URL, unit, and effective date?
• Are group multipliers explicit instead of implied?
• Are newly available routes reviewed before they appear in examples?
• Are removed or renamed routes handled as release blockers?
• Are authenticated smoke tests separate from public catalog checks?

If the answer is yes, the model catalog becomes part of the release contract instead of a loose list of strings.

That is the point of the diff test: not to slow teams down, but to make sure the model ID in the code, the route in production, and the price row in the workbook are still talking about the same thing.