Build a Nightly Model Evaluation Harness for AI Gateways

Multi-model AI gateways age quickly. A route that looked solid last Monday can become risky after a provider changes a model alias, adjusts a rate card, shifts a context window, adds a tool fee, or returns a different error envelope. The problem is not only cost. It is the combination of quality, latency, cache behavior, billing shape, compliance metadata, and operational rollback.

This is especially visible when a team routes workloads across Chinese model families such as DeepSeek, Qwen, GLM, Kimi, and ERNIE through an OpenAI-compatible API. The providers do not expose identical pricing fields. Some separate cached input from uncached input. Some apply request-level context tiers. Some bill thinking tokens as output. Some charge built-in tools per call. Some publish all-day prices, while others require a schedule-aware estimate.

The right answer is not a spreadsheet that someone updates by hand once a month. The better pattern is a nightly evaluation harness that records evidence before traffic moves. It should run a stable prompt pack, calculate source-dated cost estimates, compare quality assertions, measure latency, verify error handling, and produce a route decision that can be reviewed by engineering and finance.

Below is a compact implementation pattern you can adapt for a production AI gateway. The examples use environment variables and placeholder model IDs only. Replace the model list with the providers and routes your own account is authorized to call.

What the Harness Should Catch

A useful evaluation harness is not a leaderboard. It is a release gate for routing policy. It should answer a smaller set of questions:

Check	Why it matters	Fail condition
Price source age	Rate cards move faster than application releases	Any active route uses a stale source date
Cache accounting	Repeated prompts can bill very differently from new prompts	Cache-hit and cache-miss tokens are merged
Output control	Long answers often dominate completed-task cost	Output cap missing for high-volume routes
Quality regression	A model can pass syntax but fail task intent	Golden prompt score drops below threshold
Latency distribution	Median alone hides bad tail behavior	p95 exceeds the route service-level target
Error envelope	Retry logic depends on stable status and body shape	429 or 5xx is not classified
Rollback readiness	A route switch should be reversible in minutes	No previous passing route exists

This table is deliberately boring. That is the point. You want a repeatable decision record, not a dramatic benchmark post. The harness should make it difficult to push traffic to a route whose pricing evidence is stale or whose behavior is no longer compatible with the client contract.

Use Source-Dated Pricing, Not Memory

As of August 26, 2026, the public AIWave pricing material for DeepSeek V4 budgeting lists all-day AIWave rates for V4 Flash at $0.638 input, $1.914 output, and $0.0203 cache-hit input per one million tokens. It lists V4 Pro at $1.914 input, $5.742 output, and $0.0638 cache-hit input per one million tokens. The same page separates those all-day gateway rates from DeepSeek official peak/off-peak examples and tells readers to recheck current provider pages before procurement.

DeepSeek's official model pricing documentation also separates cache-hit input, cache-miss input, and output pricing for V4 Flash and V4 Pro, and it warns that prices may change. QwenCloud's pricing documentation describes pay-as-you-go billing, context-tiered text pricing, Batch API discounts, context caching, thinking-token billing, and built-in tool fees. Z.AI's pricing page lists GLM text-model prices with input, cached-input, storage, and output columns, plus a separate web-search tool fee.

The operational lesson is simple: store the source URL, rate date, billing unit, token bucket, route, and provider in the same record as the evaluation result. Do not let a route policy say "DeepSeek is approved" without also saying which rate card was checked and when.

Here is a small pricing schema that avoids hiding important differences:

from dataclasses import dataclass
from datetime import date
from decimal import Decimal

@dataclass(frozen=True)
class PriceCard:
    provider: str
    route: str
    model: str
    source_url: str
    checked_on: date
    input_per_m: Decimal
    output_per_m: Decimal
    cache_hit_input_per_m: Decimal | None = None
    tool_call_per_1k: Decimal | None = None
    notes: str = ""

    def estimate(
        self,
        uncached_input_tokens: int,
        output_tokens: int,
        cache_hit_input_tokens: int = 0,
        tool_calls: int = 0,
    ) -> Decimal:
        million = Decimal("1000000")
        total = (Decimal(uncached_input_tokens) / million) * self.input_per_m
        total += (Decimal(output_tokens) / million) * self.output_per_m
        if self.cache_hit_input_per_m is not None:
            total += (Decimal(cache_hit_input_tokens) / million) * self.cache_hit_input_per_m
        elif cache_hit_input_tokens:
            total += (Decimal(cache_hit_input_tokens) / million) * self.input_per_m
        if self.tool_call_per_1k is not None:
            total += (Decimal(tool_calls) / Decimal("1000")) * self.tool_call_per_1k
        return total.quantize(Decimal("0.000001"))

Notice what this does not do. It does not assume every provider has the same cache definition. It does not assume a tool call is always included in token pricing. It does not flatten all input into one number when the provider exposes separate cache buckets.

Build a Stable Prompt Pack

The prompt pack should represent real production work, but it should be small enough to run every night. For a developer-facing gateway, I usually want four task families:

Family	Example prompt	Scoring signal
Classification	Route a support ticket into a strict JSON category	Valid JSON and exact label match
Code edit	Patch a small function while preserving tests	Diff applies and test assertion passes
Retrieval answer	Answer using only supplied context	Cites provided record IDs
Refusal boundary	Reject a disallowed secret-handling request	No credential-shaped output

Keep each task deterministic. Store the expected answer shape. Add a few adversarial cases that used to fail. The goal is not to prove a model is generally smart. The goal is to know whether this route still satisfies your own gateway contract.

GOLDEN_TASKS = [
    {
        "id": "json-support-router-001",
        "messages": [
            {"role": "system", "content": "Return strict JSON only."},
            {"role": "user", "content": "User says billing export is missing VAT fields. Classify as billing, docs, reliability, or abuse."},
        ],
        "expect": {"category": "billing"},
        "max_output_tokens": 120,
    },
    {
        "id": "retrieval-boundary-001",
        "messages": [
            {"role": "system", "content": "Answer only from the provided records."},
            {"role": "user", "content": "Records: [A] EU logs stay in configured region. [B] Invoices show token buckets. Question: what evidence supports audit review?"},
        ],
        "expect_contains": ["EU logs", "token buckets"],
        "max_output_tokens": 180,
    },
]

Use production-like prompts, but remove secrets, customer identifiers, private keys, payment details, and anything that should not leave your environment. If your gateway supports zero data retention or regional routing, the prompt pack should also verify that policy metadata is attached before the request is sent.

Run Every Candidate Through One Client

An OpenAI-compatible gateway lets you test many routes through one SDK shape. That is helpful because the harness can isolate provider behavior from client code. The route can be a model ID, a provider alias, or a gateway policy name.

import os
import time
from openai import OpenAI

client = OpenAI(
    api_key=os.environ.get("AIWAVE_API_KEY", "YOUR_API_KEY_HERE"),
    base_url=os.environ.get("AIWAVE_BASE_URL", "https://api.aiwave.live/v1"),
)

def run_task(model: str, task: dict) -> dict:
    started = time.perf_counter()
    try:
        response = client.chat.completions.create(
            model=model,
            messages=task["messages"],
            max_tokens=task["max_output_tokens"],
            temperature=0,
        )
        elapsed_ms = round((time.perf_counter() - started) * 1000)
        usage = getattr(response, "usage", None)
        text = response.choices[0].message.content or ""
        return {
            "ok": True,
            "model": model,
            "task_id": task["id"],
            "elapsed_ms": elapsed_ms,
            "text": text,
            "usage": usage.model_dump() if usage else {},
        }
    except Exception as exc:
        elapsed_ms = round((time.perf_counter() - started) * 1000)
        return {
            "ok": False,
            "model": model,
            "task_id": task["id"],
            "elapsed_ms": elapsed_ms,
            "error_type": type(exc).__name__,
            "error": str(exc)[:500],
        }

In production, add request IDs, region, provider route, retry count, and cache telemetry from your gateway. If your provider returns separate cached-token fields, preserve them. If it does not, record that absence explicitly. Missing evidence is useful evidence.

Score the Output Before You Score the Cost

Cost estimates are meaningless when the answer fails. Score each task first, then compute cost for passing and failed calls separately. Failed calls still matter because retries create real spend and user-visible latency, even when a provider does not bill a particular failure class.

import json

def score(task: dict, result: dict) -> dict:
    if not result["ok"]:
        return {"passed": False, "reason": "request_error"}

    text = result["text"]
    if "expect" in task:
        try:
            parsed = json.loads(text)
        except json.JSONDecodeError:
            return {"passed": False, "reason": "invalid_json"}
        for key, value in task["expect"].items():
            if parsed.get(key) != value:
                return {"passed": False, "reason": f"mismatch_{key}"}
        return {"passed": True, "reason": "exact_json_match"}

    for needle in task.get("expect_contains", []):
        if needle.lower() not in text.lower():
            return {"passed": False, "reason": f"missing_{needle}"}
    return {"passed": True, "reason": "contains_required_evidence"}

A nightly run should produce route-level status such as pass, watch, or block. A route with strong quality but stale pricing evidence should not be promoted. A route with current pricing but failing JSON output should not be promoted either.

Produce a Route Decision Record

The final artifact should be a machine-readable JSON line and a short human summary. Keep it small enough that a developer can review it in a pull request.

from datetime import datetime, timezone

def decision_record(route: str, price_card: PriceCard, results: list[dict]) -> dict:
    passed = sum(1 for r in results if r["score"]["passed"])
    total = len(results)
    latencies = sorted(r["elapsed_ms"] for r in results)
    p95 = latencies[int(0.95 * (len(latencies) - 1))] if latencies else None
    price_age_days = (datetime.now(timezone.utc).date() - price_card.checked_on).days

    status = "pass"
    reasons = []
    if passed < total:
        status = "block"
        reasons.append("golden_task_failure")
    if price_age_days > 3:
        status = "block"
        reasons.append("stale_price_source")
    if p95 is not None and p95 > 8000:
        status = "watch" if status == "pass" else status
        reasons.append("latency_p95_watch")

    return {
        "checked_at": datetime.now(timezone.utc).isoformat(),
        "route": route,
        "status": status,
        "reasons": reasons,
        "tasks_passed": passed,
        "tasks_total": total,
        "latency_p95_ms": p95,
        "price_source": price_card.source_url,
        "price_checked_on": price_card.checked_on.isoformat(),
    }

Route decisions should be append-only. When someone asks why traffic moved from one model to another, you want a record that includes the prompt pack version, model ID, price source, latency distribution, failure classes, and rollback route. This is also useful for finance reviews because it ties spend estimates to the same evidence engineering used.

Add Promotion Rules

The harness becomes valuable when it controls promotion. A simple policy is enough:

Status	Meaning	Automation behavior
pass	Quality passed, pricing evidence current, latency inside target	Eligible for canary
watch	Quality passed, but latency or cost moved materially	Keep current route, alert owner
block	Quality failed, source stale, or errors unclassified	Do not promote

For canaries, start with a narrow slice of traffic. Store the previous passing route and make rollback a config change, not a code deploy. If the route handles regulated or enterprise workloads, require policy metadata too: data-retention mode, region, provider, model owner, and billing source.

Why This Works Better Than Static Comparisons

Static comparisons are useful for discovery, but they get stale. A nightly evaluation harness turns model selection into a controlled engineering loop. It catches pricing drift without turning price into the only criterion. It makes cache behavior visible. It keeps output caps and tool costs attached to the route. It gives support and finance the same record engineering uses.

For Tier 1 and Tier 2 teams adopting Chinese AI models, this discipline matters. You may be comparing direct provider APIs, OpenAI-compatible aggregators, and internal gateways at the same time. A route that is attractive for scheduled batch extraction may be wrong for interactive reasoning. A long-context model may be excellent for code review but too variable for high-volume support triage. A provider may expose a strong model but a billing structure that needs more telemetry before promotion.

The harness does not remove judgment. It gives judgment better inputs. Run the same prompts every night, preserve source-dated pricing, keep the evidence append-only, and promote only routes that pass the contract your product actually needs.

Sources to recheck before production use: AIWave predictable pricing, DeepSeek API pricing, QwenCloud pricing, and Z.AI pricing.