That 200 OK From Your LLM Gateway Probably Means Nothing

Every AI gateway on the market — LiteLLM, Portkey, OpenRouter, Olla — checks the same things: HTTP status code, response time, token usage. If the backup provider returns HTTP 200 with valid JSON, the gateway declares success.

But HTTP 200 only tells you the request completed. It says nothing about whether the response is correct.

The Silent Failure Pattern

In production monitoring across multi-provider setups, a consistent pattern emerges during failover events:

1. Primary provider returns 5xx or times out
2. Gateway routes to backup provider
3. Backup returns HTTP 200 with complete, well-formed JSON
4. Gateway declares success
5. The response contains subtly wrong content — hallucinated entities, missing fields, contradictory reasoning
6. The consuming agent or application continues processing bad data

The gateway logs show "failover successful." Monitoring shows no errors. But the output is wrong.

Why Existing Gateways Can't Catch This

All major LLM gateways operate at the transport level:

# Every gateway does this:
def handle_failover(request, providers):
    for provider in providers:
        try:
            response = provider.complete(request)
            if response.status_code == 200:
                return response  # "Success!"
        except Exception as e:
            log(f"Provider failed: {e}")
            continue  # Try next

Transport-level checks validate:

• ✅ HTTP status is 2xx
• ✅ Response is valid JSON
• ✅ Response time within threshold
• ✅ Token usage within limits

What they don't validate:

• ❌ Does the response contain all required fields?
• ❌ Are field types and values correct?
• ❌ Does the output contradict itself?
• ❌ Are there hallucinated entities?
• ❌ Does the response actually answer the original request?

What Response Validation Looks Like in Practice

Instead of accepting any 200 OK, add a contract validation step after failover:

from dataclasses import dataclass
from typing import List, Optional
import json

@dataclass
class ResponseContract:
    """Define what a valid response looks like."""
    required_fields: List[str]
    forbidden_patterns: List[str]
    max_tokens: int
    require_json: bool = True
    field_constraints: dict = None


def validate_response(response: dict, contract: ResponseContract) -> dict:
    """Validate response against contract. Returns validation result."""
    issues = []

    # 1. Structural checks (~45µs P50)
    for field in contract.required_fields:
        if field not in response:
            issues.append(f"Missing required field: {field}")

    # 2. Field type validation
    if contract.field_constraints:
        for field, expected_type in contract.field_constraints.items():
            if field in response:
                if not isinstance(response[field], expected_type):
                    issues.append(f"Field {field}: expected {expected_type.__name__}, got {type(response[field]).__name__}")

    # 3. Content pattern checks
    if isinstance(response.get("content", ""), str):
        content = response["content"]
        for pattern in contract.forbidden_patterns:
            if pattern.lower() in content.lower():
                issues.append(f"Forbidden pattern found: {pattern}")

    return {
        "valid": len(issues) == 0,
        "issues": issues,
        "issue_count": len(issues),
    }

# Usage example
def validated_failover(request, providers, contract):
    """Failover with response validation."""
    for provider in providers:
        try:
            response = provider.complete(request)
            result = validate_response(response, contract)
            if result["valid"]:
                return response
            else:
                log(f"Provider {provider.name}: contract validation failed - {result['issues']}")
                # Option: retry with next provider, or surface degradation
        except Exception as e:
            log(f"Provider {provider.name} error: {e}")
    raise AllProvidersFailed("All providers failed or produced invalid responses")

This pattern adds 45µs P50 overhead (diagnostic engine microbenchmark, 70,000 fault injections across 7 failure types) — negligible compared to the 700-900ms of a typical LLM API call.

Three Failure Categories That Slip Through HTTP 200

Based on the arXiv:2606.14589 taxonomy from a production LLM agent runtime:

1. Schema-Valid but Wrong

The response is structurally perfect — all fields present, correct types, valid JSON. The content is just wrong.

Example: You ask for pricing of GPT-4o. The backup model returns valid JSON with a plausible price that happens to be outdated or from a different model.

Detection: Field-level constraints and cross-field validation (e.g., "model_name + price must match known pricing table").

2. Contradictory Chain Outputs

In multi-step agent workflows, each individual response looks fine — but the combination produces contradictions.

Example: Step 1 says "user is in California." Step 3 says "applying NY state tax." Each response is independently valid.

Detection: Stateful validation across the conversation context, checking for logical consistency between steps.

3. Hallucinated Grounding

The response is coherent, well-structured, and cites sources — but the citations don't exist or don't support the claim.

Example: An analysis that cites specific research papers, but the papers don't contain the claimed findings.

Detection: Structured predicates that verify assertions against known reference data.

Where to Add Validation

The validation layer belongs in the proxy, not the application:

┌─────────────┐     ┌──────────────┐     ┌─────────────┐
│  Application │────▶│    Gateway   │────▶│  Provider 1 │
│  (Agent/App)  │     │  + Validation│     ├─────────────┤
└─────────────┘     │     Layer    │     │  Provider 2 │
                    │              │     ├─────────────┤
                    │  After every  │     │  Provider 3 │
                    │  response:    │     └─────────────┘
                    │  1. Validate  │
                    │  2. If fail → │
                    │     retry or  │
                    │     flag      │
                    └──────────────┘

Benefits of proxy-level validation:

• Zero application changes — legacy apps get validation for free
• Unified policy — one contract definition, enforced everywhere
• Failover-aware — validation is most critical exactly when failover happens
• Measurable — track validation pass/fail rates by provider

The Benchmark That Matters

When evaluating a gateway, add one more row to your comparison spreadsheet:

Capability	Any current gateway	Should be standard
Provider routing	✅	✅
Failover	✅	✅
Circuit breakers	✅	✅
Rate limiting	✅	✅
Cost tracking	✅	✅
Response validation	❌	✅ Required
Semantic correctness	❌	✅ Required

The microsecond-level overhead (45µs P50, 102µs P99) makes this a no-brainer addition to the proxy layer.

Try It Yourself

The validation approach shown above is simplified for illustration. A production-grade implementation — with configurable contracts, multi-provider support, and MCP integration — is what we're building at Correctover.

But the pattern itself is framework-agnostic. You can add response validation to any gateway today with < 100 lines of Python.

---

References:

• arXiv:2606.14589 — "When Errors Become Narratives: A Longitudinal Taxonomy of Silent Failures in a Production LLM Agent Runtime"
• Microbenchmark: 70,000 fault injections, 7 fault types, diagnostic P50=45µs, P99=102µs
• Correctover.com