We built the Cryptographic Compliance Standard (CCS) — a verification protocol for LLM output integrity in production agent systems. Then we tested 8 major LLM providers against it.
The results are worse than expected.
The Test
20 standardized verification cases across 8 providers. Each case exercises a production failure mode: HTTP errors, timeout cascades, model substitution, arithmetic corruption, hallucinated citations.
The results:
| Provider | Pass Rate | Primary Failure |
|---|---|---|
| Microsoft Phi-3.5-MoE | 0% | HTTP 404 |
| Microsoft Phi-4-Multimodal | 0% | HTTP 400 |
| OpenAI GPT-OSS-120B | 17% | Timeout + arithmetic errors |
| Meta Llama-3.1-70B | 80% | Hallucinated citations |
| Databricks DBRX | 0% | HTTP 404 |
| IBM Granite-34B | 0% | HTTP 404 |
| Google Gemma-3-12B | 0% | HTTP 404 |
62.5% of models are completely non-functional. The remaining models exhibit silent output corruption: arithmetic errors (2+3=6), hallucinated citations, and structural defects.
Why This Matters
Policy engines decide WHO can act in an agent system. Nobody verifies WHAT the model actually outputs.
An LLM that says 2+3=6 in a financial pipeline isn't "creative" — it's silently corrupting data. An agent framework that marks an HTTP 404 as "success" because it switched providers isn't recovering — it's failing blind.
Agent frameworks are building production systems on this. The industry's approach to reliability is "Best Practice Guides" and retry libraries. Nobody checks whether the output is actually correct before it hits production tools.
CCS v1.0: The Minimum Viable Compliance Standard
CCS defines 5 verification dimensions for production agents:
- Schema Validation — Is the response format-compliant?
- Cryptographic Provenance — Can the output be attributed and verified?
- Hallucination Detection — Does the output contain fabricated claims?
- Drift Monitoring — Is the model behaving consistently over time?
- Cost/Token Auditing — Are production budgets being respected?
Access the Data
- Full Audit Report: https://correctover.github.io/disclosures/20260707-llm-verification-failures.html
- CCS Specification: https://correctover.github.io
- 20K Verification Dataset (DOI): https://doi.org/10.5281/zenodo.21234580
- PyPI Package: https://pypi.org/project/correctover/
This is an open standard, not a product pitch. If you're running agents in production, you need output verification.
Correctover Research Group | Q3 Industry Reliability Benchmark | 2026-07-08
Top comments (0)