You added a second LLM provider so a bad output from one could not take down your product. That is a sensible instinct, and in a lot of codebases it does not work. The two models fail on the same inputs, for the same reasons, at the same time.
The observation behind this piece is that competing models now give strikingly similar answers even on charged, subjective questions about people. That convergence is not limited to controversial topics. It is a structural property of how modern models are trained.
It directly undermines any architecture that treats different providers as independent failure domains.
By the end you will be able to measure how correlated your models actually are, add a test that fails when redundancy becomes an illusion, and choose diversification strategies that hold up.
The redundancy assumption is untested
A fallback or ensemble only helps if the second model succeeds where the first fails. The math behind ensembling assumes errors are roughly independent. Get a handful of models to disagree on the easy stuff and their votes cancel out individual mistakes.
Nobody checks that assumption. Teams wire up a primary model, add a secondary provider behind a retry policy, run the happy path through both, and ship. The coverage you believe you have is coverage you have never measured.
What you often find when you do measure it is that both models refuse the same prompts, hallucinate the same citations, and misread the same ambiguous table. Swapping providers feels like a meaningful change and behaves like a cosmetic one.
Models are trained to converge
This is not a coincidence, and it is not a temporary alignment gap. Several well-documented forces push models toward the same behavior.
- Shared data lineage. Everyone trains on overlapping crawls of the open web, so the factual priors are largely shared.
- The same post-training recipe. Reinforcement learning from human feedback (RLHF), where humans rate model outputs and the model learns to match those ratings, rewards a specific polite, hedging, helpful-but-cautious voice. Every major lab applies some version of it.
- Distillation between labs. Models are routinely trained on outputs from other models, which copies behavior directly rather than rediscovering it.
- Benchmark optimization. When every team tunes against the same public evals, they optimize for the same narrow definition of correct.
The result is that two models from two companies are not two independent draws. They are two fine-tunes of a similar prior. Where that prior is wrong, both are wrong, and your retry just costs you twice as much.
Measure the failure overlap
Before changing anything, find out how much of your failure set is shared. Run your existing eval set against each model, record which prompts each one fails, and compute the overlap.
def failure_overlap(results_a, results_b):
"""results: {prompt_id: passed_bool} from your eval harness"""
failed_a = {k for k, v in results_a.items() if not v}
failed_b = {k for k, v in results_b.items() if not v}
shared = failed_a & failed_b
union = failed_a | failed_b
jaccard = len(shared) / len(union) if union else 0.0
return {
"failed_a": len(failed_a),
"failed_b": len(failed_b),
"shared": len(shared),
"jaccard": round(jaccard, 3),
}
The Jaccard score is the number that matters. A value near zero means the models fail on disjoint inputs and your fallback is earning its keep. A value near one means the second model adds latency and cost while covering almost none of the first model's mistakes.
Read the shared failures first. Those prompts are your real single point of failure, and they are usually the interesting ones: ambiguous instructions, edge-case formatting, and anything where the safe answer is also the wrong answer.
Add a diversity test to CI
Eval results drift as providers retrain their models behind the same API endpoint. A one-off measurement goes stale, so make the check continuous.
EDGE_CASES = [
# prompts both providers currently handle differently
]
def test_providers_stay_diverse(run, judge, threshold=0.9):
a = run(model="primary", prompts=EDGE_CASES)
b = run(model="secondary", prompts=EDGE_CASES)
agree = sum(1 for x, y in zip(a, b) if judge(x, y) == "same")
rate = agree / len(EDGE_CASES)
assert rate < threshold, (
f"providers agree on {rate:.0%} of edge cases; "
"fallback no longer provides independent coverage"
)
The judge can be an exact match, a schema comparison, or a third model grading whether two outputs are substantively the same. Exact match is the honest default. A model judge will often call two answers equivalent when a human would not.
This test belongs in the same suite as your accuracy evals. When a provider silently ships a new version and your two "independent" models start agreeing on everything, the test tells you instead of your incident channel telling you.
Choose diversification that actually diversifies
Once you know the overlap, pick mitigations by what kind of failure you are trying to survive.
| Approach | What it buys | Tradeoff |
|---|---|---|
| Second provider | Different weights and post-training | Shared data lineage, correlated refusals and errors |
| Adversarial system prompts | Different framing of the same task | Cheap but shallow; the same model still does the reasoning |
| Temperature and sampling variation | Different outputs per call | Only surface variety; does not fix systematic errors |
| Self-consistency voting within one model | Catches sampling noise | Useless when the model is confidently wrong |
| Grounding in your own retrieval | Answers come from your corpus | Needs a corpus; does not cover judgment calls |
| Deterministic rules and schema validation | Predictable, fully testable behavior | Only covers failures you anticipated |
The pattern that holds up is mixing model-based and non-model-based checks. A regex, a JSON schema validator, or a lookup against your database fails independently of any LLM, because it does not share the training data.
Layer one of those over your model calls and you get real coverage for the failure classes you can name, which is usually most of them.
Agreement is not accuracy
One trap worth naming: a high agreement rate between models feels like validation. Two models giving the same answer is comforting in a way one model giving an answer is not. That comfort is unearned. Correlated models agreeing is one opinion stated twice.
The flip side matters too. A diversity test that fails when models agree too much can also fail for the wrong reason, like one provider changing its output format. Distinguish formatting drift from behavioral drift before you act on a failing test.
Key takeaways
- Multi-provider redundancy assumes independent errors, and modern LLMs are trained toward convergence, so that assumption is usually false.
- Compute the Jaccard overlap of your two models' failure sets. A score near one means your fallback is decorative.
- Put a diversity assertion in CI, because providers retrain silently and correlation creeps in over time.
- Prefer a mix of model and non-model checks. Deterministic validators fail independently of any LLM.
- Never read model agreement as accuracy. Two correlated models agreeing is one opinion counted twice.
Source
Every AI model has the same "controversial" take on humans
The source documents the symptom: competing models give near-identical answers on subjective questions. This article takes that observation into production code, adding a way to measure failure correlation between providers, a CI test for drift in that correlation, and a comparison of diversification strategies.
Support this work
These write-ups are researched and published with no paywall, sponsor, or tracking. If one saved you an afternoon, a small tip keeps them coming.
USDT, USDC or USDD · TRC-20 (Tron)
TFTNsfyomKrnUutRjBTGVULp19ByW29KbY
Top comments (0)