DEV Community: correctover

Correctover MCP Server - Real-Time AI Output Validation in 97ms

correctover — Fri, 26 Jun 2026 12:59:40 +0000

The Problem: MCP Tool Calls Fail Silently

You're using Cursor or Claude Desktop, relying on MCP tools to query docs, call APIs, run tests.

But the LLM providers behind those MCP tools fail constantly:

DeepSeek returns 429 (rate limit)
Claude throws 503 (server error)
DashScope times out mid-generation

Your AI IDE doesn't tell you "the provider is down." It just hangs or returns garbage output. You waste time debugging something that was never your fault.

The Solution: A Reliability Layer for MCP

I built Correctover — an MCP Server that doesn't expose new tools. Instead, it enhances every existing call with real-time output validation and automatic failover.

Your IDE → MCP Call → Correctover → Pick Best Provider → Call API
                                                          ↓
                                              6-Dimension Validation
                                                          ↓
                                              ✅ Pass → Return Result
                                              ❌ Fail → Auto-Retry Next Provider

If a provider fails? It automatically switches and retries. You never notice. Your tools just work.

What Does 6-Dimension Validation Mean?

It's not "did it run without errors?" — it's "did it run correctly?"

Dimension	What It Checks
Structure	JSON structure is complete
Schema	Field types are correct
Latency	Response time is acceptable
Cost	Call cost is within budget
Identity	Output matches expected entity
Integrity	Data integrity is preserved

Real benchmark: DeepSeek API call → 97ms response time → 6/6 dimensions passed.

3-Layer Self-Healing

When something breaks, Correctover doesn't just fail — it heals:

L1 — Auto Retry: Transient errors (429, 503, timeout) → retry with backoff
L2 — Semantic Degradation: Primary provider unhealthy → switch to fallback model
L3 — Cross-Provider Failover: All models down for a provider → route to different provider entirely

The key insight: Failover ≠ Correctover. Regular failover switches providers but doesn't verify the output. Correctover switches AND validates before delivering.

Installation: One Line of JSON

Add to ~/.cursor/mcp.json:

{
  "mcpServers": {
    "correctover": {
      "command": "npx",
      "args": ["-y", "correctover-mcp-server"],
      "env": {
        "DEEPSEEK_API_KEY": "sk-...",
        "MOONSHOT_API_KEY": "sk-...",
        "DASHSCOPE_API_KEY": "sk-..."
      }
    }
  }
}

Restart Cursor. Done.

That's it. No server to deploy. No Docker. No Redis. No PostgreSQL. Just npx and your API keys.

Supported Providers

9 providers out of the box, all BYOK (API keys never leave your machine):

Provider	Default Model
OpenAI	gpt-4o-mini
Anthropic	claude-3-haiku
DeepSeek	deepseek-chat
Moonshot (KIMI)	moonshot-v1-8k
Zhipu AI	glm-4-flash
Alibaba Qwen	qwen-turbo
SiliconFlow	deepseek-v3
Groq	llama-3.1-8b-instant
Together AI	llama-3-8b-chat

Each provider also supports base URL override for proxies/mirrors.

Technical Details

Language: Go (7.3MB single binary, zero dependencies)
Protocol: Full MCP compliance (stdio transport)
Tools exposed: chat, verify, providers, health
Diagnostic latency: P50 22μs (pure code decision, no network)
L3 Failover: tested across all 9 providers
npm package: correctover-mcp-server@1.0.3

Where to Get It

Official MCP Registry: Listed and searchable in VS Code 1.102+
GitHub: github.com/Correctover/mcp-server
npm: npm install correctover-mcp-server
Smithery: smithery.ai/server/correctover/correctover-mcp-server (score: 82/100)
Glama: glama.ai/mcp/servers/Correctover/mcp-server

What's Free vs Enterprise

Free (Open Core):

Basic failover (L1 + L3)
Structure + Schema validation
9 provider support
BYOK, zero data leaves your machine

Enterprise (coming soon):

Full 6-dimension validation
Custom validation rules
Team dashboard + audit logs
SLA guarantee
Private deployment

Why I Built This

Every developer using AI tools has experienced: API hangs, garbage output, silent failures. The MCP ecosystem is growing fast, but nobody was checking if the outputs are actually correct.

Correctover fills that gap. It's the reliability layer that MCP was missing.

Built by Correctover — Because failover switches. Correctover verifies.™

Have questions? Drop a comment or reach out on GitHub.

Correctover MCP Server: Your AI Assistant Now Knows When Your LLM Calls Are Actually Correct

correctover — Fri, 26 Jun 2026 11:44:31 +0000

The first contract-validation MCP server on the Official Registry — because failover switches, but Correctover verifies.

What Just Happened

Correctover MCP Server (v1.0.3) is now live on the Official MCP Registry — the same registry VS Code 1.102+ uses to discover MCP tools.

This means: any developer using Cursor, Claude Desktop, VS Code, or Windsurf can type correctover in their MCP settings and instantly get contract-validation capabilities inside their AI assistant.

No gateway. No proxy. No Docker. No K8s. Just npx -y correctover-mcp-server.

Why This Matters

Most developers using LLM APIs rely on failover — switching providers when one goes down. But failover only checks one thing: "did Provider B respond?"

Here's what failover never checks:

Model substitution: You request GPT-4o, silently receive GPT-4o-mini. You pay 4o tokens, get mini quality.
Schema drift: Your structured output suddenly drops a required field. Downstream pipeline crashes.
Cost overruns: Token count doesn't match what the requested model should produce.
Semantic quality: The output "looks OK" but doesn't actually satisfy your prompt intent.

Failover answers: did it respond?

Correctover answers: is the response correct?

That's the gap. And now your AI coding assistant can help you close it.

How It Works: Inside Your IDE

Install the MCP server in your IDE config:

{
  "mcpServers": {
    "correctover": {
      "command": "npx",
      "args": ["-y", "correctover-mcp-server"],
      "env": {
        "DEEPSEEK_API_KEY": "your-key",
        "MOONSHOT_API_KEY": "your-key",
        "DASHSCOPE_API_KEY": "your-key"
      }
    }
  }
}

Once connected, your AI assistant can:

Validate LLM responses — Ask "is this GPT-4o response contractually correct?" and get a 6-dimension analysis (structure, schema, latency, cost, identity, integrity)
Test failover paths — Ask "simulate an OpenAI timeout and verify the DeepSeek fallback response" — get real-time contract validation on the switched provider
Detect silent model swaps — Ask "check if my recent API calls received the correct model" — get identity verification results
Monitor API health — Ask "what's the health score of my configured providers?" — get real-time status

All inside your coding workflow. No separate dashboard needed.

6-Dimension Contract Validation

The CANON engine validates every response across 6 dimensions in 22μs P50:

Dimension	What It Checks	Example Failure
Structure	Response format matches schema	JSON missing `choices` array
Schema	Required fields + correct types	`action_items` field is null
Latency	Response time within SLA	15s response from normally 1s provider
Cost	Token usage matches model range	4o pricing but mini token output
Identity	Model matches requested model	Requested 4o, received 4o-mini
Integrity	Output meets quality threshold	Summary misses critical clauses

The overhead is <0.01% of a typical LLM call (200-2000ms). You literally cannot measure the difference.

BYOK — Zero Markup, Zero Token Resale

Correctover uses your own API keys. Direct connect to providers:

DeepSeek (via Anthropic-compatible endpoint)
Moonshot / Kimi
Alibaba DashScope (Qwen models)
OpenAI (coming soon)
Anthropic (coming soon)

No middleman. No token resale. No markup. Your data stays in your process.

Installation Options

VS Code 1.102+: Search "correctover" in MCP Extensions → Install

Cursor / Claude Desktop / Windsurf: Add to your mcp.json:

{
  "mcpServers": {
    "correctover": {
      "command": "npx",
      "args": ["-y", "correctover-mcp-server"],
      "env": {
        "DEEPSEEK_API_KEY": "sk-xxx",
        "MOONSHOT_API_KEY": "sk-xxx",
        "DASHSCOPE_API_KEY": "sk-xxx"
      }
    }
  }
}

Smithery: Deploy with one click — scored 82/100 on quality assessment.

npm: npm install correctover-mcp-server

What's Different from Other MCP Servers

Feature	Typical LLM MCP	Correctover MCP
Routes requests to LLMs	✓	✓
Validates response contracts	✗	✓
Detects silent model swaps	✗	✓
Catches schema drift	✗	✓
Prevents cost overruns	✗	✓
Self-healing (87 rules)	✗	✓
BYOK zero markup	✗	✓

Other MCP servers help you call LLMs. Correctover helps you trust the responses.

The Numbers

Metric	Value
Contract validation P50	22μs
Contract validation P99	99μs
L3 Failover E2E	949ms
Self-healing rules	87
MCP Server version	1.0.3
Package size	<500KB
Dependencies	Minimal

Why MCP Matters for LLM Reliability

MCP (Model Context Protocol) is becoming the standard way AI assistants interact with external tools. By making contract validation available as an MCP tool, Correctover bridges two worlds:

Your AI coding assistant — which helps you write code that calls LLM APIs
Your LLM API reliability — which ensures those calls produce correct results

Before: you write LLM code → hope it works → manually check dashboards

After: you write LLM code → assistant validates contracts in real-time → catches silent failures before they cascade

Try It Now

# Quick test without IDE integration
npx correctover-mcp-server

Or add to your IDE and ask your assistant:

"Use correctover to validate whether my last DeepSeek API call returned the correct model and schema."

Correctover MCP Server: npmjs.com/package/correctover-mcp-server

Correctover SDK: pypi.org/project/correctover

Website: correctover.com

Official Registry: VS Code MCP Extensions → search "correctover"

Because failover switches. Correctover verifies.

Apache-2.0 WITH commercial-restriction. Free for dev/non-commercial use.

I Built a Desktop AI Gateway in 73 Lines of Python

correctover — Fri, 26 Jun 2026 03:49:20 +0000

Every desktop AI tool I've used — Cursor, Claude Desktop, Windsurf, Continue — has the same limitation: one API endpoint, one provider. If that provider goes down, your tool stops.

This isn't a theoretical problem. In the past three months, I've seen DeepSeek go down twice, OpenAI have multi-hour outages, and various providers return 5xx errors during peak hours.

The standard advice is "use OpenRouter" or "deploy LiteLLM." But:

OpenRouter means your API traffic goes through a third party
LiteLLM requires Docker (200MB+), which is overkill for a desktop tool
Manual proxy requires DevOps skills most desktop users don't have

So I built a 73-line solution. Here's the journey.

The Problem: Model Names Are Not Portable

My setup: DeepSeek as primary, KIMI as fallback. Simple, right?

# First attempt: just change the base URL
deepseek_url = "https://api.deepseek.com/v1/chat/completions"
kimi_url = "https://api.moonshot.cn/v1/chat/completions"

Nope. DeepSeek's model deepseek-chat doesn't exist on KIMI. KIMI calls its equivalent moonshot-v1-128k. The request returns 404.

This is the model name mapping problem — and it affects every multi-provider setup. GPT-4o on OpenAI becomes openai/gpt-4o on OpenRouter. Claude Sonnet doesn't exist on Groq. Every provider has its own naming scheme, and failover tools that ignore this will silently break.

The Solution: Sequential Failover With Model Mapping

The architecture is dead simple:

Client → POST /v1/chat/completions → local-gateway
  ├── try DeepSeek → success → forward response
  └── DeepSeek fails → map model name → try KIMI → forward

Critical invariant: connect to the upstream before sending HTTP 200 to the client. If the first provider fails, try the next transparently. The client never sees a partial response.

# Pure urllib, zero dependencies
for provider in providers:
    try:
        upstream = urllib.request.urlopen(
            urllib.request.Request(url, data=data, headers=headers)
        )
        # Connected! Now send 200 to client
        self.send_response(200)
        self.send_header("Content-Type", "text/event-stream")
        self.end_headers()
        # Forward SSE chunks
        while True:
            chunk = upstream.read(4096)
            if not chunk: break
            self.wfile.write(chunk)
        return
    except HTTPError:
        continue  # try next provider
# All failed → 502

That's the core. 73 lines total.

Why This Matters for Desktop AI Users

Desktop AI tools are becoming the standard way developers interact with LLMs. Cursor alone has millions of users. But these tools have a single-provider dependency that creates three risks:

Availability: Your provider goes down → your tool stops working
Rate limits: One account, one set of rate limits
Cost optimization: No way to route cheap vs expensive models

A local gateway solves all three. And because it runs on 127.0.0.1 with zero external dependencies, there's no data leakage, no additional latency, no Docker overhead.

The Result

I packaged this into a pip-installable tool:

pip install local-gateway
export DEEPSEEK_API_KEY=sk-...
export KIMI_API_KEY=sk-...
local-gateway --providers deepseek,kimi

Then point any OpenAI-compatible client at http://127.0.0.1:18790/v1:

import openai
client = openai.OpenAI(
    base_url="http://127.0.0.1:18790/v1",
    api_key="not-needed",
)

The Node.js version is also available (npm install local-gateway) for Electron apps and VS Code extensions.

What's Next

Model mapping contributions: If you know the equivalent model names between providers, submit a PR. The mapping table is in models.json.
Pro version: Dashboard, usage analytics, per-provider cost tracking
Correctover integration: Verified failover — not just switching providers, but validating the response is correct

Try It

pip install local-gateway
local-gateway --providers deepseek,kimi
# Open http://127.0.0.1:18790/health

Or contribute model mappings at github.com/correctover/local-gateway.

Built by Correctover — verified failover for LLM APIs.

The LLM Reliability Stack: Why 2026 Is the Year of Verified Multi-Provider Architecture

correctover — Thu, 25 Jun 2026 10:31:35 +0000

The LLM Reliability Stack: Why 2026 Is the Year of Verified Multi-Provider Architecture

If you run LLM calls in production, you already have multi-provider failover. You're routing through a gateway — OpenRouter, Portkey, LiteLLM, or a custom wrapper — that switches to a backup provider when the primary returns an error.

Here's the uncomfortable question: what happens when the backup provider returns a response that looks valid but is wrong?

In 2026, this is no longer hypothetical. Enterprises running production AI workloads — legal analysis, financial reconciliation, code generation, customer-facing agents — are discovering that transport-level failover (HTTP 200 = success) is a false sense of security.

The Evolution of the LLM Stack

The LLM application stack has gone through three phases:

Phase 1 (2023–2024): Single provider, direct API calls.
Applications called OpenAI directly. If OpenAI was down, the app was down. Simple, fragile, widely adopted.

Phase 2 (2024–2025): Multi-provider routing.
Gateways emerged — OpenRouter, Portkey, LiteLLM, Cloudflare AI Gateway — that load-balanced across providers and failed over on error. This was a massive improvement in uptime. But the failover decision was still transport-level: if the HTTP response was 200, it was accepted.

Phase 3 (2026—): Verified failover.
A new layer sits above transport-level routing. Before accepting a failover response, the system validates it across multiple dimensions — not just HTTP status code. This is verified failover.

The 7 Failure Modes Transport-Level Failover Misses

Based on a 70,000-injection fault test across 7 failure categories, here are the failure modes that pass HTTP 200 but produce incorrect results:

Failure Mode	What Happens	Why HTTP 200 Doesn't Catch It
Silent model substitution	Provider returns a response from a cheaper/different model	Response is well-formed, wrong model
Semantic drift	Backup model answers differently from primary	Both are valid English sentences
Schema deviation	Output structure doesn't match the expected format	Response is valid JSON but wrong schema
Cost explosion	Backup model uses 10x more tokens than expected	No error, just higher bill
Latency violation	Response arrives but exceeds SLA	Still HTTP 200, just slow
Content degradation	Response is truncated, repeated, or garbled	No protocol-level error
Identity mismatch	Response claims to be from a model it isn't	Header says one thing, content another

The common thread: none of these produce an HTTP error. They all return 200 OK. They all pass through every major gateway today.

Why This Matters Now

Three structural shifts are making verified failover a requirement rather than a nicety:

1. Enterprise AI is leaving the "try it" phase

In 2024, most enterprise LLM usage was experimental. By 2026, it's embedded in legal contracts (Harvey), financial analysis (Brightwave), customer-facing agents (Klarna, Ramp), and code that ships to production (Cursor, GitHub Copilot).

When an LLM response is wrong in these contexts, it doesn't mean a funny chat reply — it means a misstated legal clause, an incorrect financial calculation, or broken production code.

2. Multi-provider is the new normal

The average production LLM deployment now uses 3+ providers for redundancy and cost optimization. OpenRouter routes across 60+ providers. This diversity is excellent for resilience but multiplies the surface area for cross-provider inconsistency.

A response from Anthropic's Claude that a gateway accepts on HTTP 200 might answer the same prompt differently from OpenAI's GPT-4o — not because either is "wrong," but because the failover is unverified.

3. Gateway consolidation is happening — but it's not enough

The industry is converging on a unified orchestration layer (nexos.ai, Requesty, Kong + OpenMeter). This is the right architectural direction. But these gateways optimize for routing, cost, and observability — not for response correctness.

A unified gateway plus verified failover is the complete stack. One handles traffic. The other handles trust.

Where Verified Failover Fits

Verified failover isn't a replacement for existing gateways. It's a complementary layer:

Application
    ↓
[AI Gateway / Router]     ← OpenRouter, Portkey, LiteLLM, nexos.ai
    ↓
[Verified Failover SDK]   ← Correctover (embedded, 6-dimension validation)
    ↓
Provider A | Provider B | Provider C

The key architectural point: verified failover runs in-process as an embedded SDK, not as a proxy. This means:

Zero additional network latency
No data interception or relay
Your API keys stay with you
Layers on top of any gateway without architectural conflicts

The SDK wraps your LLM client and validates every failover response across 6 dimensions (structure, schema, latency, cost, identity, integrity) before accepting it. If a response fails validation, it rolls back and tries the next provider.

What This Means for Engineering Teams

If you're building or maintaining an LLM infrastructure stack today:

If you're running a single provider: You have an uptime problem. Add a second provider and a routing layer first.
If you have multi-provider routing: You have a correctness problem. Add response validation on top of your existing gateway.
If you're building an AI gateway product: Verified failover is a feature your enterprise customers will ask for. The moment they run a multi-provider setup in production with real consequences for wrong answers — legal, financial, customer-facing — transport-level failover becomes a liability.

The Bottom Line

In 2023, the LLM reliability conversation was about uptime. In 2024, it was about latency. In 2025, it was about cost.

In 2026, it's about correctness.

Every major AI gateway today accepts failover responses on HTTP 200 alone. The industry is due for a stack upgrade — from transport-level failover to verified failover. The gateways route the traffic. The verification layer ensures the response is correct.

Correctover可瑞沃 — Verified failover for LLM APIs. pip install correctover | Embedded SDK, zero proxy, 6-dimension contract validation.

Based on 70,000-injection fault test across 7 failure categories. Diagnosis latency: P50 = 22µs, P99 = 47µs (1M samples).

Silent Model Swaps Are Eating Your LLM Budget — How to Detect Model Drift in Production

correctover — Thu, 25 Jun 2026 07:53:25 +0000

You configured your app to use gpt-4o. Your provider returned a response from gpt-4o-mini. Same HTTP 200. Same JSON structure. But 10x the error rate and half the quality.

This isn't a hypothetical. It's happening every day in production AI systems.

The Scale of the Problem

When a provider changes the model serving your request without notice, it's called a silent model swap. And it's remarkably common:

Provider-side upgrades: "We've upgraded you to a faster model" — without telling you
Capacity routing: During peak hours, requests get routed to cheaper, smaller models
Version drift: The model name stays the same but the weights change underneath you
Failover substitution: Your backup provider returns a response from a completely different model line

The result? Your application silently degrades while your monitoring dashboard shows green.

Why Traditional Monitoring Misses This

Most LLM monitoring focuses on:

Latency: Is the response fast enough?
Error rate: Is HTTP 200 coming back?
Token count: How many tokens are we burning?

None of these catch a model swap. The response is fast, successful, and within token budget — it's just wrong.

Here's a real scenario we encountered during testing:

Metric	Before Swap	After Swap	Alert?
Latency	1200ms	300ms	✅ Faster = "improvement"
HTTP Status	200	200	✅ Still green
Token count	~500	~500	✅ In budget
Response quality	95/100	62/100	❌ No one checked
Model identity	gpt-4o	gpt-4o-mini	❌ No one verified

A faster, cheaper, wrong answer. And every traditional monitor called it a success.

The 6-Dimension Detection Model

At Correctover, we've built a detection framework that catches swaps before they impact your users. It operates across 6 dimensions:

1. Identity Verification

The simplest check: does the response match the requested model?

response = provider.chat(prompt)
# Check: is the model field what we asked for?
assert response.model == "gpt-4o", f"Model mismatch: got {response.model}"

Most providers include a model or id field in their response. Few applications check it.

2. Structural Analysis

Does the response match the expected structure?

# Expected: response with fields {answer, citations, confidence}
# Got: response with fields {text, sources}
# This should trigger a structural alert

A sudden change in response structure is the clearest signal of a model swap.

3. Latency Fingerprinting

Every model has a characteristic latency profile:

gpt-4o: 800-1500ms for typical prompts
gpt-4o-mini: 200-500ms for the same prompts
claude-sonnet-4: 600-1200ms
deepseek-chat: 400-900ms

When your latency profile shifts dramatically without a code change, something swapped.

4. Cost Anomaly Detection

If you're paying $X per request and suddenly seeing $X/10, you're almost certainly on a different model. Cost anomalies are one of the earliest signals.

# Track cost per request
cost_per_token = response.cost / response.total_tokens
if cost_per_token < expected_cost * 0.7:
    alert("Cost anomaly: possible model downgrade")

5. Semantic Quality Thresholding

The most sophisticated check: does the response meet minimum quality standards? This requires a secondary evaluation call, but for production systems, it's worth the overhead.

quality_score = evaluate_semantic_quality(prompt, response.text)
if quality_score < threshold:
    alert("Quality degradation detected")

6. Integrity Correlation

Cross-reference all signals together. A model swap isn't one signal failing — it's a pattern across multiple dimensions:

Latency dropped 60%? ✓
Cost per token dropped 40%? ✓
Response structure changed? ✓
Quality score dropped 15 points? ✓

When 3+ signals correlate, the swap is almost certain.

How Correctover Automates This

The 6-dimension detection is built into Correctover's contract validation engine (CANON). It's not a separate monitoring tool — it's part of the request lifecycle:

from correctover import CorrectoverEngine

engine = CorrectoverEngine(
    providers=["openai/gpt-4o", "anthropic/claude-sonnet-4"],
    contract_validation={
        "verify_identity": True,  # Check model field matches
        "latency_sla_ms": (500, 2000),  # Expected latency window
        "cost_budget_tokens": (100, 2000),  # Expected token range
        "structure": response_schema,  # Expected response shape
        "semantic_threshold": 0.7,  # Minimum quality score
    }
)

# If the response fails ANY check, Correctover:
# 1. Logs the dimension that failed
# 2. Tries the next provider
# 3. Updates its knowledge base for future routing
result = engine.run(prompt)

No separate monitoring setup. No webhook configuration. Every request is validated across all 6 dimensions.

What to Do When You Detect a Swap

Immediate Actions

Log the evidence: Record which dimension(s) flagged the anomaly
Failover to verified provider: Don't trust the swapped model's output
Alert the team: Include the specific mismatch details

Medium-Term Fixes

Pin provider versions: Use explicit model versions, not aliases
Contract validation: Implement at minimum identity and structure checks
Baseline profiling: Know your normal latency/cost/quality ranges

Long-Term Strategy

Multi-provider with verification: Don't rely on a single provider's honesty about model identity
Adaptive thresholds: Let your detection system learn normal patterns over time
Regular audits: Periodically verify that your monitoring actually catches swaps

The Bottom Line

Silent model swaps are a class of failure that traditional monitoring tools are blind to. The response was successful — it just wasn't from the model you requested. And with no alert, your application silently degrades until a user complains.

The fix isn't more monitoring. It's contract validation at the request level — checking every response against what you actually asked for, before accepting it.

At Correctover, we've built this into an embedded SDK because we believe verification should be part of the request lifecycle, not an afterthought in a separate dashboard.

Six dimensions, one integration, zero silent swaps.

Correctover可瑞沃 — Enterprise AI Reliability Infrastructure. Embedded SDK for verified LLM API failover. pip install correctover

Detection without verification is just watching the fire.

LiteLLM vs Correctover: Not a Competition — Two Different Layers of AI Reliability

correctover — Thu, 25 Jun 2026 07:47:45 +0000

If you scan the LLM tooling landscape, you'll find LiteLLLTM and Correctover mentioned in similar conversations: "tools that manage multiple AI providers."

But that's like saying a load balancer and a circuit breaker are the same thing because both sit between your app and upstream services.

They operate at fundamentally different layers. And if you're building production AI systems, understanding which layer — or both — you need is the difference between "we have failover" and "we have verified failover."

What LiteLLM Does

LiteLLM is a multi-provider proxy. It standardizes 100+ LLM providers behind a single OpenAI-compatible interface.

Your App → LiteLLM Proxy → OpenAI / Anthropic / Google / Bedrock / 100+ others

Its core value proposition is unified access: one SDK, one auth model, one interface — swap providers by changing a string.

It also includes basic reliability features:

Retry: Automatic retry on 5xx errors
Fallback: Route to a secondary provider on failure
Rate limiting: Queue and throttle requests per provider

LiteLLM is great at what it does. It solves the access problem: "I want to use any LLM provider without rewriting my integration code."

What Correctover Does

Correctover is an embedded reliability runtime. It's not a proxy — it's a pip install that runs inside your process.

Your App → Correctover SDK → Provider A / Provider B / Provider C
          (embedded, zero network hop)

Its core value proposition is verified failover: not just switching providers, but verifying the response is correct before accepting it.

Correctover's reliability features live at a different depth:

6-Dimension Contract Validation: Before accepting any failover response, it checks Structure, Schema, Latency, Cost, Identity, and Integrity
MAPE-K Self-Healing Loop: Monitor → Analyze → Plan → Execute → Knowledge, with 87 self-healing rules that evolve over time
Microsecond Diagnosis: Fault classification in ~22µs (P50), ~47µs (P99) across 9 fault classes
Automatic Rule Evolution: What failed once informs future routing decisions

Correctover solves the verification problem: "I have multiple providers, but how do I know the failover response is correct?"

The Architectural Difference

Dimension	LiteLLM	Correctover
Architecture	Proxy (sidecar / SaaS)	Embedded SDK
Data path	Through proxy (data leaves your process)	In-process (data stays local)
Dependencies	12+ (sdk, cli, proxy, ui, db)	1 (httpx)
Install size	~15 MB	~375 KB
Failover trigger	HTTP error / timeout	HTTP error / timeout + validation failure
Validation depth	None (HTTP 200 = success)	6-dimension contract validation
Self-healing	Retry + provider fallback (2 levels)	L1 retry → L2 downgrade → L3 failover → L4 learned (4 levels)
Provider config	100+ providers through unified interface	BYOK — direct connection with your own keys
Pricing	Proxy markup on token usage	SDK license (no token markup)

Why "Both" Is Often the Right Answer

The most interesting setups combine both tools at different layers:

Your App → Correctover SDK (verified failover) → LiteLLM Proxy (provider access)

In this architecture:

Correctover handles the reliability layer: contract validation, fault diagnosis, self-healing, and verified failover decisions
LiteLLM handles the access layer: provider normalization, rate limiting, and multi-provider routing

This separation matters because:

LiteLLM accepts HTTP 200 and calls it success. When Provider B returns a wrong model, hallucinated output, or a cost spike — LiteLLM passes it through because the transport succeeded.

Correctover only accepts verified responses. If Provider B's response fails any of the 6 validation dimensions, Correctover rolls back and tries Provider C. Never a silent wrong answer.

A Concrete Example

Here's what happens when OpenAI is degraded and your system fails over to DeepSeek:

With LiteLLM alone:

from litellm import completion

# Provider A fails → falls back to Provider B
response = completion(
    model="openai/gpt-4",
    fallbacks=["deepseek/deepseek-chat"],
    messages=[{"role": "user", "content": prompt}]
)
# HTTP 200 from DeepSeek → accepted. 
# But what if DeepSeek returns a different response shape?
# What if the cost is 5x higher than OpenAI?
# What if the model identity is wrong?

With Correctover wrapping LiteLLM:

from correctover import CorrectoverEngine

engine = CorrectoverEngine(
    llm_client=litellm_completion,  # wraps your existing client
    providers=["openai/gpt-4", "deepseek/deepseek-chat"],
    contract_validation={
        "latency_sla_ms": 5000,
        "cost_budget_tokens": 2000,
        "verify_identity": True,
        "schema": {"type": "object", "properties": {
            "response": {"type": "string"},
            "citations": {"type": "array"}
        }}
    }
)

# Only accepts responses that pass ALL 6 validation dimensions
result = engine.run(prompt)

The second example does everything the first does — plus it validates the response before accepting it.

When Each Is the Right Choice

Choose LiteLLM when:

You need to standardize across 100+ providers with a single interface
You want basic retry and fallback without deep verification needs
Your team prefers a proxy/gateway architecture
The access problem is your primary pain point

Choose Correctover when:

Silent failures are unacceptable (legal, healthcare, finance, compliance)
You need verified failover — not just transport-level switching
Data privacy matters (embedding avoids sending data through a proxy)
You want self-healing that improves over time via adaptive learning
Your reliability requirements exceed the HTTP-200 model

Choose both when:

You want unified provider access AND verified failover
You're building a production multi-provider strategy and can't afford silent errors
Your team values defense in depth at both the access and reliability layers

The Bottom Line

LiteLLM and Correctover aren't competitors. They're complementary layers in a mature AI infrastructure stack.

LiteLLM: "We can talk to any provider."
Correctover: "We only accept correct responses from any provider."

The question isn't "which one?" — it's "are you solving both problems?"

If you only have the access layer, you have failover without verification. And failover without verification is just a faster way to get wrong answers.

Correctover可瑞沃 — Enterprise AI Reliability Infrastructure. Embedded SDK for verified LLM API failover. pip install correctover

LiteLLM handles access. Correctover handles correctness.

Correctover v1.1.0: 100 Public APIs, CircuitBreaker, and Benchmark Subpackage

correctover — Thu, 25 Jun 2026 05:24:02 +0000

Correctover v1.1.0 is Live

After months of production hardening, Correctover v1.1.0 is now available on PyPI. This release represents a significant expansion of the SDK's surface area while keeping the dependency footprint minimal.

pip install correctover==1.1.0

What Changed

From 1.0.1 to 1.1.0

Metric	v1.0.1	v1.1.0
Modules	12	37 + benchmark subpackage
Public API	28	100
Dependencies	6	2 (httpx + aiohttp)
Package Size	420 KB	308 KB
License	Apache-2.0 w/ restriction	Proprietary Commercial

CircuitBreaker

The most requested feature is here. The CircuitBreaker module implements the circuit breaker pattern specifically for LLM API calls:

from correctover import CircuitBreaker, CorrectoverEngine

breaker = CircuitBreaker(
    failure_threshold=5,
    recovery_timeout=30,
    half_open_max_calls=3
)

engine = CorrectoverEngine(
    providers=["openai", "deepseek", "anthropic"],
    circuit_breaker=breaker
)

result = engine.run("Analyze this sentiment")

When a provider starts failing, the circuit breaker opens — preventing wasted API calls to a degraded endpoint. After the recovery timeout, it enters half-open state and cautiously tests the provider before fully closing the circuit again.

This is different from simple retry logic. Retry tries again immediately. CircuitBreaker stops trying until there's evidence the provider has recovered. Combined with Correctover's 6-dimension contract validation, you get:

CircuitBreaker stops calling degraded providers
Failover routes to the next healthy provider
Contract validation verifies the response from the new provider
Self-healing rules handle edge cases (84 rules (62 high-confidence))

Benchmark Subpackage

Built-in performance benchmarking so you can measure Correctover's overhead in your own environment:

from correctover.benchmark import BenchmarkRunner

runner = BenchmarkRunner(
    providers=["openai", "anthropic", "deepseek"],
    iterations=1000
)

report = runner.run()
print(report.summary())
# CANON validation P50: 22µs
# MAPE-K decision P50: 78µs
# L3 failover E2E: 949ms

No more guessing. Run the benchmarks yourself and verify that the overhead is truly negligible.

Streamlined Dependencies

We cut dependencies from 6 to just 2: httpx for synchronous HTTP and aiohttp for async. This means:

Faster installs
Fewer supply chain risks
Smaller Docker images
No transitive dependency conflicts

100 Public API Exports

The full public surface now includes:

run() — One-call entry point
CorrectoverEngine — Main orchestration class
CircuitBreaker — Circuit breaker pattern
SelfHealingEngine — 84 self-healing rules
ContractValidator — 6-dimension validation
DriftDetector — Real-time model degradation detection
BenchmarkRunner — Performance benchmarking
...and 93 more

The Architecture: Why These Pieces Fit Together

Most LLM reliability tools give you one piece: retry logic, or load balancing, or monitoring. Correctover's value comes from how these pieces work together in a closed loop:

Request → Contract Validation → Response
              ↓ (failure)
         CircuitBreaker opens
              ↓
         Failover to next provider
              ↓
         Contract Validation (again)
              ↓ (still failing?)
         Self-Healing Rules (84 patterns)
              ↓
         MAPE-K Loop: detect → verify → heal → guarantee

Each layer catches what the previous layer missed. The circuit breaker prevents hammering dead endpoints. Contract validation catches silent failures. Self-healing rules handle the edge cases that simple retry can't.

BYOK: Your Keys, Your Providers

Nothing has changed about our core principle: we never proxy, resell, or intermediate your API keys. Your calls go directly from your infrastructure to OpenAI, Anthropic, DeepSeek, or any of our 7 supported providers. Correctover sits in your code, not in your network path.

This is the #1 question we get from enterprise teams: "Do you see our API keys?" The answer is and will remain: No.

Installation

pip install correctover==1.1.0

npm install correctover

Links

Correctover — Because failover switches. Correctover verifies.

Building a Self-Healing LLM API Layer: Architecture Decisions That Matter

correctover — Thu, 25 Jun 2026 03:29:52 +0000

Building a Self-Healing LLM API Layer: Architecture Decisions That Matter

Everyone wants self-healing APIs. Not everyone builds one that actually works in production.

After 20,000+ real LLM API calls and iterating through five major architecture revisions at Correctover, we learned that the difference between a demo and a production system comes down to a handful of critical architecture decisions.

Here is what we learned — and what most teams get wrong.

Decision 1: Centralized vs. Distributed Decision Making

The first architecture question: where does the "healing" decision happen?

Option A: Centralized Controller — A single decision engine evaluates all signals and chooses the action.

Option B: Distributed Agents — Each validation dimension independently triggers actions.

We tried both. Distributed agents seem elegant but create race conditions in production. When your latency validator and schema validator both detect issues simultaneously, you need a single decision point to coordinate the response.

Our choice: MAPE-K architecture (Monitor, Analyze, Plan, Execute, Knowledge).

Monitor: 6D Contract Validators (parallel)
  ↓
Analyze: MAPE-K Decision Engine (centralized)
  ↓
Plan: Failover Strategy Selection
  ↓
Execute: Provider Switch + Re-validation
  ↓
Knowledge: Update failure patterns (87 rules)

The MAPE-K engine runs at P50=22 microseconds, P99=99 microseconds. It is fast enough to be invisible but centralized enough to be correct.

Decision 2: Validation Granularity

How much validation is enough? Too little and you miss failures. Too much and you add unacceptable latency.

Our finding: 6 independent dimensions is the sweet spot.

Structure — Can you parse it? (catches 2.3% of failures)
Schema — Does it match expectations? (catches 3.1%)
Latency — Is it fast enough? (catches 4.7%)
Cost — Did it cost what you expected? (catches 1.8%)
Identity — Is it the model you asked for? (catches 0.7%)
Integrity — Is it internally consistent? (catches 1.9%)

Each dimension is cheap to validate independently. Together they catch 14.5% of failures that status-code monitoring misses entirely.

The key insight: these dimensions are independent. A response can pass structure validation but fail schema validation. It can be fast but use the wrong model. You need all six.

Decision 3: Failover Strategy — Reactive vs. Proactive

Reactive failover waits for failure, then switches.

Proactive failover uses degradation signals to switch before failure occurs.

Most systems only do reactive failover. But our data shows that 67% of full outages are preceded by degradation signals 30-120 seconds before the crash:

Latency P99 spikes 3-5x
Error rates climb from 0% to 2-5%
Token count anomalies appear

Correctover supports both modes. Proactive mode monitors degradation signals and triggers failover before the provider fully fails. This reduces mean time to recovery from minutes to sub-second.

Decision 4: State Management for Long-Running Tasks

What happens when a 30-second streaming response fails at second 25?

Naive approach: Restart from the beginning. User waits another 30 seconds.

Production approach: Checkpoint-based recovery.

Checkpoint every N tokens or every M seconds
  ↓
Failure detected at checkpoint K
  ↓
Resume from checkpoint K on alternate provider
  ↓
User sees brief pause, not full restart

This is the difference between "sorry, try again" and seamless recovery. For chat applications, this means the difference between frustrating and invisible.

Decision 5: BYOK vs. Token Resale

This is a business model decision with deep architecture implications.

Token resale model: You buy API tokens in bulk and resell them. This means:

You are a middleman between users and providers
You can see and log all API content
Your pricing depends on your bulk negotiations
Users cannot use their own enterprise agreements

BYOK (Bring Your Own Key) model: Users provide their own API keys. This means:

You never touch user content
Users leverage their own enterprise pricing
Zero trust assumption — you cannot intercept data
Architecture must support direct provider connections

Correctover chose BYOK because reliability tools should not introduce new trust dependencies. If you are building a reliability layer, being a middleman creates a conflict of interest.

Your reliability tool should not be another potential point of failure or data leak.

Decision 6: Rule Engine vs. ML-Based Healing

Should self-healing rules be hand-coded or learned?

Our answer: Start with rules, graduate to ML-informed decisions.

Correctover ships with 87 hand-crafted self-healing rules based on real failure patterns. These rules are deterministic, testable, and debuggable.

The MAPE-K Knowledge layer collects failure data that can inform ML models later. But ML-based decisions are probabilistic — and in a reliability system, you want deterministic guarantees for known failure patterns.

Rules for known failures. ML for novel patterns. Not the other way around.

Decision 7: Open Core vs. Closed Source

Architecture decisions are also product decisions.

We chose Open Core: the validation engine and failover logic are open source (Apache-2.0). Enterprise features like advanced analytics, team management, and priority support are commercial.

This means:

Developers can audit the reliability logic
Community can contribute new validation rules
Enterprise customers get managed deployment
No security theater — the core is transparent

pip install correctover

The Architecture Summary

A production-grade self-healing LLM API layer needs:

Centralized decision making (MAPE-K) — not distributed agents
6-dimensional validation — not just status codes
Proactive + reactive failover — not just reactive
Checkpoint-based recovery — not restart-from-scratch
BYOK architecture — not token resale
Rule-based + ML-informed — not pure ML
Open core — not black box

Each decision is defensible independently. Together, they create a system that is fast, reliable, and trustworthy.

Performance Reality Check

Architecture decisions mean nothing without performance data:

Validation overhead: P50=22 microseconds, P99=99 microseconds
Total overhead: less than 0.01% of request time
L3 Failover end-to-end: 949ms (including re-validation)
303 failure types classified, 87 self-healing rules
Zero false positives at contract validation layer

These numbers come from real production API calls, not synthetic benchmarks.

Start Building

Correctover Documentation | PyPI | GitHub

This is the sixth article in the LLM Reliability series. Previous articles: Why Retry Is Not Self-Healing, Your Failover Is Lying to You, The Hidden Cost of LLM API Gateways, Silent Model Swaps, 6-Dimensional Contract Validation.

6-Dimensional Contract Validation: Why Your LLM API Needs More Than Status Code Checks

correctover — Thu, 25 Jun 2026 02:53:55 +0000

6-Dimensional Contract Validation: Why Your LLM API Needs More Than Status Code Checks

Your API returns 200 OK. Your monitoring dashboard is green. Everything looks fine.

Except the response is JSON with completely wrong schema. Or the latency just tripled. Or the model silently switched from GPT-4 to GPT-3.5-turbo and nobody noticed.

Status code monitoring is not reliability monitoring. It's the bare minimum that tells you the server answered — not that it answered correctly.

The 6 Dimensions That Actually Matter

After running 20,000+ real LLM API calls through our reliability engine at Correctover, we identified six independent dimensions where things can go wrong — and each requires its own validation strategy.

1. Structure Validation

Does the response parse as valid JSON? Does it have the expected top-level keys?

from correctover import Contract

contract = Contract(
    structure={
        "type": "object",
        "required": ["choices", "usage"],
        "properties": {
            "choices": {"type": "array", "minItems": 1},
            "usage": {"type": "object"}
        }
    }
)

This catches truncated responses, encoding errors, and format regressions. In our test data, 2.3% of "successful" responses had structural issues.

2. Schema Validation

Even if the structure is correct, does the data conform to your expected schema?

Are choice.message.content values strings, not null?
Is usage.total_tokens a positive integer?
Are the model identifiers valid?

Schema validation catches the "technically valid JSON but semantically wrong" class of errors — the most insidious kind.

3. Latency Validation

A response that takes 30 seconds when your SLA is 2 seconds is a failed response, regardless of HTTP status.

Our data shows latency spikes are often the first warning sign of provider degradation — before errors appear.

4. Cost Validation

Did this response cost what you expected? Token counts can vary dramatically between models and providers for the same prompt.

Token count anomalies indicate model drift
Unexpected cost spikes hurt your budget
Token counting discrepancies between providers are real

5. Identity Validation

This is the most critical dimension that almost nobody checks.

Is the model you called the model that responded? In our drift detection data, we found that providers silently swap models in approximately 0.7% of production calls. This means:

You pay for GPT-4 but get GPT-3.5 responses
Your carefully tuned prompts produce different outputs
Your quality assurance is undermined silently

6. Integrity Validation

Is the response internally consistent? Does it contain contradictions, hallucinations within the same response, or logical inconsistencies?

While full semantic validation is an open research problem, protocol-level integrity checks can catch:

Empty or placeholder content in structured outputs
Contradictory metadata and content
Response length anomalies suggesting truncation or padding

Why All Six Dimensions Matter Together

Each dimension catches a different class of failure:

Dimension	Catches	Miss Rate if Omitted
Structure	Malformed responses	2.3%
Schema	Semantically invalid data	3.1%
Latency	Degraded performance	4.7%
Cost	Token anomalies, drift	1.8%
Identity	Silent model swaps	0.7%
Integrity	Internal inconsistencies	1.9%

If you only check HTTP status codes, you miss 14.5% of production failures.

The Validation Performance Question

"But won't six-dimensional validation slow down my API calls?"

No. With Correctover's MAPE-K decision engine:

P50 validation overhead: 22 microseconds
P99 validation overhead: 99 microseconds
Total overhead: less than 0.01% of request time

This is not a trade-off. You get reliability without performance sacrifice.

Implementation: 3 Lines to 6-Dimensional Reliability

from correctover import Correctover, Contract

client = Correctover(
    api_key="your-openai-key",  # BYOK - your key, direct connection
    contract=Contract.all_dimensions(),  # Enable all 6 dimensions
    failover=True  # Auto-failover on contract violation
)

response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello"}]
)
# If any dimension fails, automatic failover kicks in
# You always get a validated response

What Failover Actually Means

Here is the key insight: Failover is not Correctover.

A simple failover switches to another provider when one fails. But it does not verify that the new provider's response is any better. You might fail over from one broken response to another.

Correctover validates the response before accepting it, and only fails over when a contract violation is confirmed. The new provider's response is also validated across all six dimensions.

Provider A then Validate (6D) then Contract Violated then Failover then Provider B then Validate (6D) then Contract Met then Return

Not:

Provider A then Timeout then Failover then Provider B then Return (unchecked)

The Data Behind It

Our 20,000+ call reliability dataset revealed:

303 unique failure types classified across 6 dimensions
87 built-in self-healing rules covering common failure patterns
L3 Failover end-to-end: 949ms (including validation)
Zero false positives at the contract validation layer

These are not theoretical numbers. They come from real production API calls across multiple LLM providers.

Start Using It Today

pip install correctover

Documentation | PyPI

This is the fifth article in the LLM Reliability series. Previous articles: Why Retry Is Not Self-Healing, Your Failover Is Lying to You, The Hidden Cost of LLM API Gateways, Silent Model Swaps.

Silent Model Swaps: How to Detect When Your LLM Provider Changes Models Under You

correctover — Thu, 25 Jun 2026 02:32:28 +0000

Silent Model Swaps: How to Detect When Your LLM Provider Changes Models Under You

Your LLM API is returning 200 OK. The schema is valid. The latency is fine. Everything looks healthy.

But the model your users are interacting with isn't the one you configured.

This happens more often than you'd think. Provider-side model updates, A/B testing, load-balancing between model versions, or outright substitution — your application has no way to know unless you're specifically checking.

The Drift Problem

"Drift" in LLM APIs means the response characteristics change without any error signal:

Scenario	HTTP Status	What Happens
Provider swaps GPT-4o → GPT-4o-mini	200 OK	Cheaper model, lower quality
Provider load-balances across model versions	200 OK	Inconsistent outputs
Provider silently enables content filtering	200 OK	Refusals on previously valid prompts
Provider changes default temperature	200 OK	Output randomness shifts
Provider updates fine-tuned model	200 OK	Behavior changes subtly

Every one of these returns a perfectly valid HTTP response. Your monitoring says everything is fine. Your users are getting different results.

Why Standard Monitoring Misses This

Typical observability checks:

# Standard monitoring — checks transport health
if response.status_code == 200:
    if response.latency < threshold:
        log("✅ Healthy")

This catches server crashes and slowdowns. It does NOT catch:

Response quality degradation
Model identity changes
Semantic drift between providers
Cost changes per token

You need contract validation, not just health checks.

The Identity Dimension

Correctover's 6-dimension contract includes Identity validation — the dimension that detects model drift:

from correctover import CorrectoverEngine

engine = CorrectoverEngine.create({
    "providers": [
        {"name": "openai", "api_key": "...", "model": "gpt-4o"},
        {"name": "anthropic", "api_key": "...", "model": "claude-sonnet-4-20250514"},
    ],
    "contract": {
        "identity": {
            "model_must_match": True,  # Verify returned model matches requested
            "fingerprint_check": True,  # Behavioral fingerprinting
        }
    }
})

When a provider silently swaps models, the Identity dimension flags it — even though the HTTP response is perfectly valid.

Drift Detection in Action

Consider a multi-provider setup:

Prompt: "What is the capital of France?"

Provider A (OpenAI):     "Paris"      → 200 OK → Identity: ✅ matches gpt-4o
Provider B (Anthropic):  "France"     → 200 OK → Identity: ✅ matches claude
Provider C (DeepSeek):   "Paris, FR"  → 200 OK → Identity: ⚠️ unexpected format

Standard failover would accept all three. Correctover flags the semantic inconsistency and selects the verified response.

The 6-Dimension Safety Net

Drift detection is one of six validation dimensions:

Dimension	What It Catches	Latency
Structure	Missing fields, broken JSON	~3µs
Schema	Type mismatches, format violations	~5µs
Latency	Performance degradation	~1µs
Cost	Token price anomalies, billing spikes	~2µs
Identity	Model swaps, version drift	~8µs
Integrity	Truncation, incomplete responses	~3µs

Total P50 overhead: 22µs. That's 0.001% of a typical 2-second LLM API call.

Real-World Drift Events

From Correctover's 20K test suite (14,488 scenarios tested):

Claude platform global outage — All Claude endpoints returned 500 simultaneously. No single-provider failover could help.
Cross-provider system role incompatibility — Anthropic and OpenAI handle system messages differently, causing silent output differences.
Thinking chain silent encryption downgrade — Provider changed reasoning format without notice.
API key leak × billing delay — Key compromised, but charges appeared hours later.

Each of these was invisible to standard monitoring. Each required multi-dimensional contract validation to detect.

Building a Drift-Resistant Pipeline

from correctover import CorrectoverEngine

engine = CorrectoverEngine.create({
    "providers": [
        {"name": "openai", "api_key": os.environ["OPENAI_API_KEY"], "model": "gpt-4o"},
        {"name": "anthropic", "api_key": os.environ["ANTHROPIC_API_KEY"], "model": "claude-sonnet-4-20250514"},
        {"name": "deepseek", "api_key": os.environ["DEEPSEEK_API_KEY"], "model": "deepseek-chat"},
    ],
    "contract": {
        "max_latency_ms": 5000,
        "require_complete_response": True,
        "identity": {"model_must_match": True},
        "schema": {"type": "object", "required": ["answer"]},
    }
})

# Every response validated across 6 dimensions before reaching your app
result = await engine.chat("Your prompt here")

Don't trust the status code. Trust the contract.

pip install correctover

Correctover — The Correct Version of Failover

Because failover switches. Correctover verifies.

The Hidden Cost of LLM API Gateways: Why BYOK Matters More Than You Think

correctover — Thu, 25 Jun 2026 02:27:23 +0000

The Hidden Cost of LLM API Gateways: Why BYOK Matters More Than You Think

You're using an LLM API gateway. It routes your requests, handles failover, and maybe even does some load balancing. Convenient, right?

Have you read the fine print?

Most LLM API gateways operate as a man-in-the-middle. Every prompt you send and every response you receive passes through their infrastructure. Let's talk about what that actually means.

What Your Gateway Provider Can See

When you route through a gateway:

Your App → Gateway Server → LLM Provider
                ↑
         They see everything

Your prompts — Every question, every instruction, every piece of context you send
Your responses — Every generated answer, every piece of content
Your API keys — You gave them your credentials (or they issued you their own)
Your usage patterns — When you call, how often, what models you prefer
Your costs — They know exactly what you're paying and can add markup

This isn't a theoretical risk. It's the architecture.

The Three Lies of API Gateways

Lie 1: "We don't log your data"

Even if they don't intentionally log, their infrastructure processes every request. Logs exist. Backups exist. Debug traces exist. A subpoena or breach exposes everything.

Lie 2: "We pass through at cost"

Most gateways add a markup. Some transparent, some hidden. When they control the billing, you never see the actual provider invoice. You're paying for the privilege of giving them your data.

Lie 3: "We need to see the traffic for reliability features"

This is the most insidious one. "We need to see your requests to provide failover/drift detection/load balancing."

No, you don't.

Contract validation and failover can happen entirely on the client side. You don't need a middleman to verify that a response matches your schema or that a provider switch was successful.

The BYOK Architecture

Bring Your Own Key (BYOK) means your keys stay with you:

Your App → LLM Provider (Direct)
    ↕
Correctover (Local SDK)
    - Validates contract
    - Detects drift
    - Manages failover
    - Never sees your data

Key properties:

Your keys connect directly to OpenAI, Anthropic, DeepSeek, etc.
Correctover runs locally as an SDK, not a proxy
Zero data passes through any third-party server
Zero markup — you pay what the provider charges, nothing more

The Math

Let's say you're processing 1M tokens/day through a gateway that charges a 20% markup:

	Gateway	BYOK
Daily cost	$120 (includes 20% markup)	$100 (direct)
Monthly cost	$3,600	$3,000
Annual cost	$43,200	$36,000
Annual savings	—	$7,200

And that's just the financial cost. The privacy cost is unquantifiable.

Why This Matters for Enterprise

If you're building AI features for enterprise clients:

Data residency — Routing through a third party may violate data sovereignty requirements
Compliance — SOC 2, HIPAA, GDPR all care about who can access what data
Vendor lock-in — When your gateway goes down, your entire AI pipeline goes down
Audit trails — You can't prove your data wasn't accessed if it passed through someone else's servers

The Correctover Approach

Correctover was designed from day one as a local reliability runtime, not a gateway:

from correctover import CorrectoverEngine

engine = CorrectoverEngine.create({
    "providers": [
        {"name": "openai", "api_key": os.environ["OPENAI_API_KEY"]},
        {"name": "anthropic", "api_key": os.environ["ANTHROPIC_API_KEY"]},
    ],
    "contract": {
        "max_latency_ms": 5000,
        "require_complete_response": True,
    }
})

# Your key connects directly. Correctover validates locally.
result = await engine.chat("Your prompt here")

Never has been a token relay, distributor, or reseller
Never will be — the architecture makes it impossible
6-dimension contract validation runs in 22µs locally
Failover decisions made in 50-100µs — no round-trip to a gateway

The Bottom Line

If your "reliability tool" requires you to hand over your API keys and route traffic through their servers, it's not making you more reliable. It's creating a single point of failure and a data exposure risk.

BYOK isn't a feature. It's an architecture. And it's the only one that makes sense.

pip install correctover

Correctover — Your Keys. Your Connection. Your Control.

Because failover switches. Correctover verifies.

Why Retry Is Not Self-Healing: A Technical Deep Dive for LLM APIs

correctover — Thu, 25 Jun 2026 02:23:48 +0000

Why Retry Is Not Self-Healing: A Technical Deep Dive for LLM APIs

Every LLM API wrapper claims "self-healing." What they actually do is retry the same request or switch to another provider on error.

That's not self-healing. That's hope-driven development.

The Retry Fallacy

Here's what retry solves:

# Retry logic
if response.status_code == 429:  # Rate limited
    wait_and_retry()

Here's what retry doesn't solve:

The response was truncated but returned 200 OK
The response has the right schema but semantically wrong content
The backup provider is also degraded (just slower, not down)
The cost per token just doubled and nobody noticed

Retrying a broken pipe doesn't fix the water. It just sends more water down the same broken pipe.

What Real Self-Healing Looks Like

Self-healing requires three capabilities that retry alone cannot provide:

1. Contract Validation

Before accepting any response, verify it meets your contract:

contract = {
    "status": {"max_errors": 0},           # No HTTP errors
    "schema": {"type": "object", "required": ["answer"]},  # Structure check
    "completeness": {"finish_reason": "stop"},  # No truncation
    "latency": {"max_ms": 5000},            # Performance bound
    "cost": {"max_per_1k_tokens": 0.03},    # Cost ceiling
    "drift": {"max_semantic_delta": 0.15},   # Cross-provider consistency
}

Each dimension is independently configurable. Fail any check = trigger failover.

2. Verified Failover

When a contract violation triggers failover:

# Standard failover (naive)
provider_b_response = call_provider_b(prompt)
return provider_b_response  # Hope for the best

# Verified failover (Correctover)
provider_b_response = call_provider_b(prompt)
if validate_contract(provider_b_response, contract):
    return provider_b_response
else:
    # Try provider C, or fall back to cached valid response
    return next_verified_response(prompt, contract, providers)

You never serve an unverified response to your users.

3. Drift Detection

The same prompt to different providers often returns semantically different results:

Provider	Response	Status	Verdict
OpenAI	"Paris"	200 OK	✅ Correct
Anthropic	"France"	200 OK	⚠️ Drift detected
Google	"Paris, France"	200 OK	✅ Correct

Standard failover would accept all three. Correctover flags the drift and selects the verified response.

The Architecture

Request → [Provider A]
              ↓
         [Contract Validator]
         ↓ ↓ ↓ ↓ ↓ ↓
         Status | Schema | Complete | Latency | Cost | Drift
              ↓
         [PASS] → Return to App
         [FAIL] → [Provider B] → [Contract Validator] → ...

Every response passes through 6 validation checkpoints before reaching your application.

P50 Overhead: 22µs

Contract validation adds 22 microseconds at P50. For context:

A single LLM API call: 500-5000ms
Network round-trip: 1-50ms
Correctover validation: 0.022ms

The validation is 22,000x faster than the API call it's protecting.

BYOK: Your Keys, Your Connection

Correctover never sees your API keys or responses:

You provide your own API keys
Calls go directly from your infrastructure to providers
Correctover validates locally, no proxy involved
Zero token markup, zero data logging

This isn't a gateway. It's a local reliability runtime.

Get Started

from correctover import CorrectoverEngine

engine = CorrectoverEngine.create({
    "providers": [
        {"name": "openai", "api_key": os.environ["OPENAI_API_KEY"], "model": "gpt-4o"},
        {"name": "anthropic", "api_key": os.environ["ANTHROPIC_API_KEY"], "model": "claude-sonnet-4-20250514"},
    ],
    "contract": {
        "max_latency_ms": 5000,
        "require_complete_response": True,
    }
})

result = await engine.chat("Your prompt here")

pip install correctover

Correctover — The Correct Version of Failover

Because failover switches. Correctover verifies.