Build a cache-hit regression harness for long-context AI gateways

Long-context AI workloads often look stable until a small routing or prompt change breaks the cache shape.

The application still works. The model still answers. The request log still shows familiar model names. But the bill attached to the same workflow can move because repeated input that used to land in a cached bucket is now charged as fresh input. A support engineer may see no obvious error. A finance reviewer may see a spend jump. A platform engineer may only see that one prompt template, one system prefix, or one model route changed last week.

That is why cache behavior should be tested as a production contract.

This article describes a cache-hit regression harness for teams running long-context workloads through OpenAI-compatible AI gateways. The pattern is useful when a product uses large reusable context, model routing, prompt templates, retrieval packs, or provider fallback. It is written for Tier 1 and Tier 2 engineering teams that need cost evidence without exposing prompts, private documents, API keys, payment identifiers, or customer names.

AIWave is built around one OpenAI-compatible route to Chinese model families such as DeepSeek, GLM, Kimi, Qwen, ERNIE, MiniMAX, Doubao, StepFun, and MiMo. The same client can call different model routes while the platform keeps dated pricing and request-level billing evidence visible. That makes cache regression testing valuable: the integration can stay stable while the economic behavior behind a long-context run still changes.

The bug is not always a failure

Most API test suites are built around functional output.

They check that a request returns 200. They check JSON shape. They check that a response contains a useful answer. They may check latency and retry count. Those tests are necessary, but they do not catch a common long-context regression: the workload keeps succeeding while its cached-input ratio collapses.

This can happen after ordinary engineering changes:

• A system prompt gains a new timestamp at the top.
• A retrieval pack changes document order.
• A route switches from one model variant to another.
• A fallback path omits the stable prefix.
• A middleware layer injects request-specific metadata into the reusable context.
• A streaming adapter retries with a rebuilt message array.
• A prompt compression step rewrites the part that used to stay constant.

None of those changes has to be wrong. Some are useful. The problem is that the team may not notice the cost behavior changed until later.

A cache-hit regression harness treats the cached-input ratio as a testable property. It does not need raw prompt text. It needs deterministic fingerprints, token bucket counts, model route identifiers, pricing source metadata, and acceptance thresholds.

Pin the pricing evidence

Before writing this article, I checked AIWave's public pricing source on 2026-09-07 at 13:10 UTC. The live endpoint https://aiwave.live/api/pricing reconciled against the public pricing page with 63 route records, 63 matched HTML rows, pricing version a42d372ccf0b5dd13ecf71203521f9d2, and group ratios of default=3 and vip=1.

The same check showed coverage across 9 provider families:

Provider family	Route records
DeepSeek	3
GLM	7
Kimi	12
ERNIE	2
MiniMax	8
Qwen	23
Doubao	4
StepFun	3
MiMo	1

These are dated planning facts. They are not a guarantee about future pricing, every account, every upstream route, or every invoice. That is the point of pinning pricing evidence in the harness. A regression result should say which rate source, checked time, pricing version, account group, and resolved model were used when the test ran.

Define the cache contract

Start with a small contract for each long-context workflow.

The contract should define what is expected to stay stable, what is allowed to change, and which token buckets matter. It should be reviewed by the team that owns the workflow, not only by the platform team.

{
  "contract_id": "support_summary_cache_2026_09",
  "workflow": "support_summary",
  "owner": "platform-ai",
  "client_surface": "openai-compatible-chat",
  "requested_model": "deepseek-v4-pro",
  "resolved_model_family": "DeepSeek",
  "pricing_source": "https://aiwave.live/api/pricing",
  "pricing_version": "a42d372ccf0b5dd13ecf71203521f9d2",
  "effective_group": "vip",
  "stable_segments": [
    "system_policy",
    "tool_schema",
    "knowledge_pack"
  ],
  "variable_segments": [
    "user_question",
    "conversation_tail"
  ],
  "minimum_cached_input_ratio": 0.70,
  "maximum_output_tokens": 1800,
  "maximum_retry_count": 1
}

The exact threshold depends on the workload. A small chat request may have no meaningful reusable context. A coding assistant with a large repository summary may expect most of its prefix to remain stable across a short evaluation run. A retrieval workflow may expect stability only inside a fixed evaluation fixture.

Do not set a universal threshold. Set a threshold per workflow and record why it is reasonable.

Fingerprint segments, not secrets

The harness should never export raw prompts or private source documents to a public report. It should fingerprint segments.

A segment fingerprint is enough to answer the operational question: did the reusable part stay the same between the baseline and the candidate run?

import hashlib
import json


def stable_json(value):
    return json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":"))


def fingerprint_segment(segment):
    encoded = stable_json(segment).encode("utf-8")
    return hashlib.sha256(encoded).hexdigest()


def build_segment_manifest(messages, tool_schema, knowledge_pack_id):
    return {
        "system_policy": fingerprint_segment(messages[0]),
        "tool_schema": fingerprint_segment(tool_schema),
        "knowledge_pack": fingerprint_segment({"id": knowledge_pack_id}),
        "message_count": len(messages)
    }

The fingerprint should be deterministic. That means sorting JSON keys, stripping fields that are intentionally request-specific, and keeping timestamps out of stable segments unless the timestamp is part of the contract.

When a regression happens, the manifest helps separate two cases:

• The cache ratio fell because a stable segment changed.
• The cache ratio fell even though the stable segment fingerprints match.

The first case is usually a prompt or packaging change. The second case may point to route behavior, provider behavior, retry behavior, cache TTL, or an adapter bug.

Capture token buckets per run

A useful harness captures token buckets at the same boundary as the workload.

For long-context work, a single workflow may include planning, retrieval, drafting, validation, and summarization. Testing one request in isolation can hide the real regression. The run-level record should preserve per-step detail and aggregate totals.

type CacheRegressionRun = {
  run_id: string
  contract_id: string
  checked_at: string
  requested_model: string
  resolved_model: string
  provider_family: string
  pricing_version: string
  effective_group: string
  segment_fingerprints: Record<string, string>
  steps: Array<{
    step_name: string
    input_tokens: number
    cached_input_tokens: number
    output_tokens: number
    retry_count: number
    fallback_used: boolean
  }>
  totals: {
    input_tokens: number
    cached_input_tokens: number
    output_tokens: number
    retry_count: number
  }
  verdict: "pass" | "review" | "fail"
}

The fields are intentionally operational. They let a reviewer answer the questions that matter:

• Did the model route change?
• Did the account or token group change?
• Did the reusable prompt material change?
• Did retries or fallbacks change the token shape?
• Did cached input fall below the accepted threshold?

This record can live in CI artifacts, an internal dashboard, or a billing evidence store. It should not require prompt access to be useful.

Compare baseline and candidate

The harness needs a stable baseline. That baseline should be refreshed deliberately, not silently overwritten on every run.

Use a candidate run to compare against the approved baseline:

def cached_ratio(totals):
    input_tokens = totals["input_tokens"]
    cached = totals["cached_input_tokens"]
    if input_tokens <= 0:
        return 0.0
    return cached / input_tokens


def compare_cache_contract(contract, baseline, candidate):
    issues = []

    if candidate["pricing_version"] != contract["pricing_version"]:
        issues.append({
            "severity": "review",
            "code": "pricing_version_changed",
            "message": "The candidate used a different dated pricing source."
        })

    if candidate["resolved_model"] != baseline["resolved_model"]:
        issues.append({
            "severity": "review",
            "code": "resolved_model_changed",
            "message": "The candidate resolved to a different model route."
        })

    stable_segments = contract["stable_segments"]
    for name in stable_segments:
        if candidate["segment_fingerprints"].get(name) != baseline["segment_fingerprints"].get(name):
            issues.append({
                "severity": "fail",
                "code": "stable_segment_changed",
                "segment": name,
                "message": "A segment marked stable changed between runs."
            })

    ratio = cached_ratio(candidate["totals"])
    if ratio < contract["minimum_cached_input_ratio"]:
        issues.append({
            "severity": "fail",
            "code": "cached_input_ratio_below_contract",
            "observed": round(ratio, 4),
            "minimum": contract["minimum_cached_input_ratio"]
        })

    if candidate["totals"]["retry_count"] > contract["maximum_retry_count"]:
        issues.append({
            "severity": "review",
            "code": "retry_count_above_contract",
            "observed": candidate["totals"]["retry_count"]
        })

    return issues

The result should be boring. A pass means the workflow kept its route, stable segments, pricing source, retry envelope, and cached-input shape within contract. A review means something changed and a human should decide whether to bless a new baseline. A fail means the candidate should not be promoted until the owner explains the change.

Add a CI gate without blocking every experiment

The harness should not make experimentation painful. It should protect production promotion.

A practical setup has three modes:

Mode	When it runs	What it can do
observe	Local development and exploratory branches	Write evidence, never block
review	Pull requests that touch prompts, routing, or adapters	Comment with diffs and ask for owner review
enforce	Release branches and production route changes	Block when hard thresholds fail

This keeps the gate proportional. A developer can experiment with prompt structure without fighting CI on every edit. The release path still requires evidence before a long-context workload changes its cost shape.

The trigger list should include more than prompt files:

• Prompt templates
• Retrieval packaging
• Tool schemas
• Model route configuration
• Fallback policy
• Streaming adapters
• Retry middleware
• Pricing source readers
• Ledger writers
• Token accounting code

Cache regressions often enter through infrastructure files, not just prompt files.

Make the report useful to finance and support

The report should not be a wall of traces. It should answer a short set of questions.

{
  "contract_id": "support_summary_cache_2026_09",
  "candidate_run_id": "ci_9482",
  "verdict": "fail",
  "checked_at": "2026-09-07T13:10:10Z",
  "pricing_version": "a42d372ccf0b5dd13ecf71203521f9d2",
  "resolved_model": "deepseek-v4-pro",
  "cached_input_ratio": 0.42,
  "minimum_cached_input_ratio": 0.70,
  "changed_segments": [
    "system_policy"
  ],
  "retry_count": 0,
  "fallback_used": false,
  "next_action": "Review prompt package change before promotion."
}

A support engineer can see that the request did not fail, but the reusable prefix changed. A finance reviewer can see that the issue is token-bucket shape, not only volume. A platform engineer can see which contract needs a new baseline or a fix.

Keep the report shareable inside the company. That means no raw prompts, no private documents, no real API keys, no customer names, and no payment identifiers.

Decide when to bless a new baseline

A regression is not always a bug. Sometimes the team intentionally changes the reusable context to improve answer quality, add safety policy, or support a new tool. In that case, the right action is to bless a new baseline.

Require a short baseline change record:

{
  "baseline_change_id": "cache_baseline_2026_09_07_support_summary",
  "contract_id": "support_summary_cache_2026_09",
  "reason": "Added structured refusal policy to the stable system segment.",
  "expected_effect": "Cached-input ratio may reset during rollout, then recover on repeated calls.",
  "approved_by": "platform-ai-owner",
  "pricing_version": "a42d372ccf0b5dd13ecf71203521f9d2",
  "rollback_signal": "Cached-input ratio remains below contract after the rollout window."
}

The important part is not the exact JSON. The important part is that the baseline change is explicit. Silent baseline replacement turns the harness into a dashboard that always agrees with the last run.

Common mistakes

The first mistake is testing only one happy-path request. Long-context costs often emerge across a run, not in a single call.

The second mistake is hashing the whole prompt as one blob. That makes every small variable field look like a total cache break. Segment the material into stable and variable regions.

The third mistake is ignoring the resolved model. If a request asks for one model alias but resolves to another model route, the cache and pricing behavior may change even when the client code did not.

The fourth mistake is using a pricing table without a checked time and version. A cost regression report without dated pricing evidence is hard to review later.

The fifth mistake is exposing too much evidence. The harness should prove operational behavior, not export secrets or customer material.

What to build first

Start with one long-context workflow that already has repeated context. Add segment fingerprints, run-level token bucket capture, and a single cached-input threshold. Store the pricing source URL, checked time, pricing version, effective group, requested model, and resolved model beside the run.

Run the harness in observe mode for a week. Do not block releases immediately. Use that period to learn the normal range of cached-input ratios, retry counts, and route behavior. Then choose which threshold belongs in review mode and which threshold belongs in enforce mode.

For an OpenAI-compatible gateway, this is the practical goal: a route can change, a prompt can evolve, and a pricing source can be refreshed, but the team can still explain whether the long-context economics stayed inside contract.

Cache hits are not only a provider feature. For production teams, they are an evidence boundary. Treat them like one.