DEV Community: Jason Lau

Your Churn Model Works. Your Churn Intervention Doesn't.

Jason Lau — Sun, 01 Nov 2026 14:33:09 +0000

The quarterly retention review had a good slide. The churn model's top decile had been sent a 20% discount offer, and only 18% of those customers left, against the 31% the model had predicted for them. Thirteen points, at a few thousand customers, came to a satisfying number in saved revenue.

Then someone asked the question that ends most of these meetings: "Compared to what?"

The 31% was the model's forecast for those customers if nothing changed. But the model was trained on last year's customers, the offer went out in a quarter with a price change and a new competitor, and nobody had held back a group of top-decile customers who got no offer. So nobody could say whether the discount saved those customers, whether they would have stayed anyway, or whether it pushed some of them out the door. The model was fine. The program had no way to show it had done anything.

This example is invented, but the design is the standard one: score everyone for churn risk, treat the riskiest, report the result against the model's own forecast. It has a gap in it that a better model can't close.

Two questions that sound like one

A churn model answers: if we do nothing, how likely is this customer to leave? That's a prediction about one possible future.

A retention program needs a different answer: how much does contacting this customer change whether they leave? That's a comparison between two futures, one where you act and one where you don't, and for any single customer you only ever see one of them. It's a causal question, and it has its own name in marketing analytics: uplift.

The two answers don't have to line up. A customer can be at high risk and impossible to keep, because they've already signed with a competitor. A customer can be at low risk and very easy to influence. And some customers are at low risk until you contact them.

Eva Ascarza (2018) tested this directly. Combining two field experiments, at a wireless provider and a membership organisation, with machine learning, she showed that "customers identified as having the highest risk of churning are not necessarily the best targets for proactive churn programs." Her conclusion is blunt: retention programs "are sometimes futile not because firms offer the wrong incentives but because they do not apply the right targeting rules." Targeting on sensitivity to the offer, regardless of risk, was "significantly more effective than the standard practice of targeting customers with the highest risk of churning."

Some customers leave because you called

The uncomfortable part is that a retention contact can cause churn. A call about a contract renewal reminds a customer that they're out of contract. A discount tells a customer who hadn't been thinking about price that they may be paying too much.

Radcliffe and Surry's review of real-world uplift modelling (2011) includes a case from a mobile operator: "A mobile churn reduction initiative actually increased churn from 9% to 10%." With an uplift model, the operator found a 30% subsegment where the campaign helped. Targeting only that group "reduced overall churn from 9% to under 8%, while reducing spend by 70%." The same paper says this kind of backfire is common in retention: it is "not uncommon, particularly in the area of retention, for uplift modelling to deliver more value by identifying populations where negative effects are prevalent" than by ranking the rest. They list when it tends to happen: interventions that are intrusive, and customers who "are already unhappy with the supplier."

Look at that list again with a churn model in mind. Unhappy customers are exactly who a churn model puts at the top.

What a randomised holdout shows that the model can't

Public retention experiments with a random control group are rare, but there's a well-known randomised marketing test with the same structure. Kevin Hillstrom's MineThatData e-mail dataset has 64,000 customers. A third "were randomly chosen to receive an e-mail campaign featuring Womens merchandise," a third got a men's e-mail and a third "were randomly chosen to not receive an e-mail campaign." Visits were tracked for two weeks.

I treated it as a retention problem: a customer who doesn't visit has lapsed, the women's e-mail is the retention action, and the no-e-mail group is the holdout. Then I built the two kinds of model on half the data and compared them on the other half, ten times with different random splits:

A risk model, trained only on holdout customers, that predicts the chance of lapsing if nobody e-mails them. That's what a churn model trained on history estimates.
A simple uplift model: one model for the e-mailed customers, one for the holdout, and the difference between their predicted visit rates.

Because the e-mail was assigned at random, the effect inside any group of test customers can be measured directly: their visit rate if e-mailed, minus their visit rate if not. No model is involved in that number.

First, the risk model works. It ranks lapse risk better than chance (AUC 0.63 on holdout customers it hadn't seen), and the gap between its deciles is real: 97.7% of the top decile lapsed without an e-mail, against 81.4% of the bottom decile. Here is what the e-mail did in each decile, averaged over the ten splits:

Risk decile	Lapsed, no e-mail	Lapsed, e-mailed	Extra visits per 1,000 e-mailed
1 (highest risk)	97.7%	92.7%	49
2	93.9%	89.2%	47
3	92.5%	87.8%	46
4	91.0%	86.8%	43
5	90.0%	85.3%	47
6	88.2%	84.3%	39
7	88.1%	83.4%	47
8	86.6%	82.4%	42
9	84.8%	79.9%	48
10 (lowest risk)	81.4%	76.5%	49

The risk column falls steadily. The effect column doesn't move. Across the whole test half the e-mail added 45 visits per 1,000 customers, and every decile sits close to that. The model knows who is likely to lapse. It knows nothing about who the e-mail reaches.

That shows up as soon as you have a budget and can only contact some customers. Say the budget covers 10%. Each model scores every test customer and the top 10% are the ones it would e-mail. Because the real e-mail went out at random, about half of that group got it and half didn't. The gap in visit rate between those two halves is what the e-mail did for exactly the customers that model chooses.

Here is that gap, as extra visits per 1,000 customers e-mailed:

Budget (share of customers e-mailed)	Chosen at random	Chosen by highest churn risk	Chosen by highest uplift
10%	45	49 (range 37–57)	73 (range 54–93)
20%	45	48 (range 33–59)	70 (range 55–82)
30%	45	47 (range 41–53)	67 (range 59–77)
50%	45	46 (range 41–51)	61 (range 56–66)

Mean over ten random splits, with the lowest and highest split in brackets. "Chosen at random" is the effect across the whole test half, so it's the same at every budget.

Targeting by risk was about as good as picking customers at random. Targeting by uplift got 40–50% more out of every e-mail, and it beat risk targeting in all ten splits at 20% and 30%. The ranges are wide because each test half holds only about 2,000 customers per arm in a 20% slice, which is a point we'll come back to.

Two cautions. This is an e-mail campaign measured on visits, not a retention offer measured on cancellations, and this data shows no group that the e-mail drove away. The mobile operator above shows that group does exist in retention. The data is a free download and the whole comparison is under 100 lines of pandas and scikit-learn, with the raw results for all ten splits, so you can swap in a better model, or the men's e-mail as the action, and check.

The model you need can only be built from an experiment

Notice what made the uplift model possible: a group of comparable customers who randomly got no e-mail. Without that, the uplift model has nothing to learn the difference from, and the evaluation has nothing to measure it against.

Radcliffe and Surry put this first on their checklist for when uplift modelling is worth doing: "if no valid control group exists in the historical data, uplift modelling cannot normally be employed. (But one can be created for the next iteration of the campaign.)"

It's tempting to skip the experiment and fit an uplift model to past campaigns, where account managers chose who to call. A 2026 study suggests that's a poor bet. Pacella, Papadia and Giliberti compared uplift models with conventional baselines on the public Telco Customer Churn dataset in two setups: "a causally controlled semisynthetic scenario and an observational proxy scenario," where who got treated was confounded with who was likely to churn. Uplift models won in the controlled setup. In the second, "traditional baselines remain economically superior." They also found that "the highest-yielding causal policy frequently suffers from severe targeting instability": which customers it picked changed a lot between runs. That's the same instability visible in the ranges above.

The evidence that uplift modelling pays off at all comes from settings with randomised data. Devriendt, Berrevoets and Verbeke (2021) start from the observation that "the literature, however, does not provide conclusive empirical evidence that uplift modeling outperforms predictive modeling," then compare the two head to head on a financial-services retention campaign and find that "uplift models outperform predictive models and lead to improved profitability of retention campaigns." Companies running these programs at scale build on experiments too. An AI Council 2026 talk by Intuit's Avik Basu opens with "accurate predictions do not imply better decisions" and describes Intuit using uplift modelling "to enhance targeting and experimentation."

So the missing piece in most churn programs isn't a better model. It's an experiment.

The one question to ask about your retention program

When the last retention campaign went out, was a random group of eligible customers held back and given nothing?

If the answer is no, nobody can say what the campaign did, and the slide comparing actual churn to predicted churn doesn't change that. The prediction is a forecast for a world where nothing happened, made by a model trained on a different period. It isn't a measurement.

If the answer is yes, you can already do the analysis above with no new model: split the campaign by the churn model's deciles and compare treated with held-out customers in each one. A table like this is enough:

import pandas as pd

def effect_by_decile(df):
    """df: one row per eligible customer, with churn_score, held_out (bool), churned (0/1)."""
    df = df.assign(decile=pd.qcut(df.churn_score.rank(method="first", ascending=False), 10,
                                  labels=range(1, 11)))
    rates = df.pivot_table(index="decile", columns="held_out", values="churned",
                           aggfunc=["mean", "size"], observed=True)
    rates.columns = ["churn_contacted", "churn_held_out", "n_contacted", "n_held_out"]
    rates["points_saved"] = 100 * (rates.churn_held_out - rates.churn_contacted)
    return rates

A negative points_saved in the top deciles is the sign to look for. That's where your offer may be losing customers it was meant to keep. With a few hundred held-out customers per decile, a single decile can swing by several points on noise alone, so look for a pattern across neighbouring deciles, or bootstrap the table, before acting on one number.

Setting up the holdout for next quarter

A few things decide whether the holdout is useful:

Hold out across every decile, not just the bottom ones. If you only hold back low-risk customers, you can't measure the effect where you spend the most.
Randomise, and don't let anyone override it. An account manager who "just calls this one customer" from the holdout has turned it back into an observational comparison.
Size it for the effect you expect, which is usually small. Detecting a drop from 10% to 9% churn with a standard two-sided test at 5% significance and 80% power takes about 13,500 customers in each group. Radcliffe and Surry's rule of thumb for modelling uplift, rather than just measuring it, is stricter: the overall uplift times the size of each group should be at least 500. A one-point uplift means at least 50,000 customers per group. If you're well below that, measure the average effect first and target later.
Count the cost of the holdout honestly. It's the customers you might have saved by contacting them. If the campaign has never been measured, that cost buys you the only number that says whether the campaign works.

None of this replaces the churn model. Risk is still useful for sizing the problem, for forecasting revenue and for deciding when to act. What it can't do is pick who to act on. That's the same distinction SophiArch's Applied Machine Learning course draws in its interpretability lesson, where SHAP values on a churn model come with the warning that they "are not causal explanations. They describe the model's reasoning, not the world." A feature that pushes a customer's churn score up isn't a lever you can pull, and a high score isn't a reason to call.

What this doesn't show

The experiment above is one public dataset with a cheap action (an e-mail) and a soft outcome (a visit). The uplift model is the simplest kind, and better ones exist. The evidence that retention offers drive some customers away comes from published case studies and field experiments, not from my numbers. And uplift models are noisy: the ranges above, and the instability Pacella and colleagues report, are a reason to measure a targeting rule on a fresh holdout before trusting it, not a reason to skip the model.

What it does show is the gap. A model can rank risk well and still be no better than random at choosing who to contact. The only way to find out which situation you're in is to hold some customers back.

References

Ascarza, E. (2018). Retention Futility: Targeting High-Risk Customers Might Be Ineffective. Journal of Marketing Research, 55(1), 80–98. Two field experiments showing that the highest-risk customers are not necessarily the best targets, and that targeting by sensitivity to the intervention reduces churn more.
Radcliffe, N. J., & Surry, P. D. (2011). Real-World Uplift Modelling with Significance-Based Uplift Trees. Stochastic Solutions white paper / Portrait Technical Report TR-2011-1. Source of the mobile churn case (9% to 10%, then under 8%), negative effects in retention, and the control-group and volume checklist.
Pacella, M., Papadia, G., & Giliberti, V. (2026). Stability-Aware Uplift Policy Selection for Customer Retention: From Predictive Scores to Actionable Segments. Applied Sciences, 16(10). Uplift models win under controlled treatment assignment, baselines win on confounded data, and the best-value policy is often unstable.
Devriendt, F., Berrevoets, J., & Verbeke, W. (2021). Why you should stop predicting customer churn and start using uplift models. Information Sciences, 548, 497–515. Head-to-head comparison of churn prediction and uplift models on a financial-services retention campaign.
Hillstrom, K. (2008). The MineThatData E-Mail Analytics and Data Mining Challenge. The randomised 64,000-customer e-mail test used for the experiment in this article.
Basu, A. (2026). From Prediction to Uplift: Causal Modeling for Better Decisions. AI Council 2026. Practitioner talk on uplift modelling at Intuit.
SophiArch. Applied Machine Learning. Course covering evaluation with statistical rigour, imbalanced data, calibration, SHAP interpretability and what it can't tell you, and production pipelines.

Your Vector Database Thinks "Not Covered" Means "Covered"

Jason Lau — Wed, 21 Oct 2026 13:03:05 +0000

TL;DR: An embedding's similarity score measures how much two texts share in topic and wording, and a negation is a one-word edit. I wrote 40 invented policy facts and tested six open embedding models. On every model, "Which plan doesn't cover X?" scored closer to "Which plan covers X?" than a plain rewording of the same question did, for 77.5–100% of the items. At a common semantic-cache cut-off (similarity 0.90, RedisVL's default), one model would serve the cached answer to 97.5% of the opposite questions and to 7.5% of the genuine rewordings. In retrieval, models found the "covered" chunk almost every time and the "not covered" chunk between 27.5% and 92.5% of the time. Adding BM25 through rank fusion made five of the six worse. Dropping English stop words, which removes "not", took BM25 to a coin flip. Re-ranking the top ten with a small cross-encoder took every model to 97.5% or better on the negated questions. Run the same test on your own model before you trust a similarity threshold.

A benefits assistant had a semantic cache in front of it, so repeated questions skipped the LLM. On Monday someone asked "Which plan covers dental implants?" and got the right answer: the Gold plan. On Tuesday someone else asked "Which plan doesn't cover dental implants?"

The cache embedded the new question, found Monday's at similarity 0.97, and returned "The Gold plan." That's the opposite of the right answer, delivered in a fraction of a second with no LLM call to blame. Someone asking "What plan pays for dental implants?", which means exactly the same as Monday's question, would have scored 0.85 and missed the cache.

The assistant is invented. The two numbers aren't: they're what all-MiniLM-L6-v2, the most downloaded sentence-similarity model on Hugging Face (about 200 million downloads a month in October 2026), gives for those three questions. The rest of this article is what happened when I ran the same comparison across six models, three tests and 40 facts.

What the score is actually measuring

Embedding models are trained to put texts that go together (a question and its answer, a title and its abstract, two captions of one picture) close together. Texts that go together share a topic and, usually, a lot of words. So the model learns a geometry where shared topic and shared wording count for most of the distance, and nothing in training forces a single "not" to move a sentence to the other side of the space.

From the vector's point of view, "Which plan doesn't cover dental implants?" is the original question with one word added. "What plan pays for dental implants?" swaps two words and adds a third. The negation is the smaller edit, so it lands closer.

This isn't new. NevIR (Weller, Lawrie and Van Durme, EACL 2024) built document pairs that differ only by negation and found most retrieval models, including state-of-the-art ones, "performing the same or worse than a random ranking", with cross-encoders doing best. Cao (ECAI 2025) tested general-purpose text embeddings and reported "a significant lack of negation awareness in these models". A June 2026 preprint on agent memory (Yadav, arXiv 2606.26511) reported that cosine similarity separated contradicted facts from duplicates with an AUROC of 0.59, close to chance, and gave the same reason: contradictions came out more similar to the original than duplicates did "because a value-flip is a smaller edit than a genuine rephrasing of the same fact." That result is small (98 labelled pairs, one embedding model), so I wanted to see how far it generalised. An August 2026 audit of similarity thresholds in agent frameworks (Frias, arXiv 2608.10216) is the closest to what follows: a production drift guard "caught 0 of 56 meaning-breaking mutations", including "withhold the study drug" → "administer the study drug", which "came in at cosine 0.9608".

Six models, 40 facts, three tests

I wrote 40 short policy facts across twelve domains: dental and health plans, home, travel and car insurance, refunds, software tiers and licences, staff benefits, airline fares, warranties, bank accounts, gym memberships and drug formularies. Each fact has three forms:

the fact: "The Gold plan covers dental implants."
a rewording with the same meaning: "Dental implants are paid for under the Gold plan."
a contradiction: "The Gold plan does not cover dental implants."

The contradictions use several kinds of negation on purpose: "does not", "isn't", "excludes", "no", "never", "unpaid", "non-refundable". Each fact also has three questions: one asking which entity covers it, a rewording of that question, and the opposite question ("Which plan doesn't cover dental implants?"). Most opposite questions carry a negation word, but a few flip the meaning without one: "Which tier limits data exports?", "Who gets unpaid parental leave?", and "Which account charges foreign transaction fees?", where it's the original that says "no".

The six models are all open and run locally: all-MiniLM-L6-v2, bge-base-en-v1.5, e5-base-v2, gte-base, mxbai-embed-large-v1 and Qwen3-Embedding-0.6B, each with the query and passage prefixes its model card asks for. I didn't test hosted embedding APIs, which is the main reason to run the check below on whatever you use.

Test 1: questions, the semantic-cache case

For each of the 40 items: how similar is the original question to its rewording, and to its opposite?

Model	Mean similarity, rewording	Mean similarity, opposite	Opposite closer than rewording	Served at 0.90: rewordings	Served at 0.90: opposites
all-MiniLM-L6-v2	0.74	0.96	100%	7.5%	97.5%
bge-base-en-v1.5	0.84	0.93	90%	15%	87.5%
e5-base-v2	0.92	0.97	92.5%	70%	100%
gte-base	0.92	0.97	90%	72.5%	100%
mxbai-embed-large-v1	0.83	0.89	77.5%	12.5%	37.5%
Qwen3-Embedding-0.6B	0.81	0.93	90%	5%	85%

The last two columns use 0.90, which is what RedisVL's SemanticCache defaults to (a cosine distance of 0.1). That's RedisVL's threshold applied to these six models, not RedisVL's own default embedding model, which I didn't test. On every model, a cache at that setting answers the opposite question more often than it answers the same question asked differently.

You can't tune your way out of it. Set the threshold low enough to catch 95% of genuine rewordings on each model, and it catches all 40 opposites, on all six. As a separation score, similarity managed an AUROC between 0.01 and 0.25, where 0.5 is chance. A number that low means the score is reliably backwards: higher similarity was evidence that the question meant the opposite.

Test 2: statements, the deduplication case

The same comparison between the fact, its rewording and its contradiction. This is what a deduplication step or a memory store faces when it decides whether a new fact is "the same" as one it already has.

Model	Contradiction closer than rewording	AUROC, rewording vs. contradiction
all-MiniLM-L6-v2	52.5%	0.46
bge-base-en-v1.5	2.5%	0.94
e5-base-v2	47.5%	0.57
gte-base	15%	0.81
mxbai-embed-large-v1	0%	0.98
Qwen3-Embedding-0.6B	55%	0.43

This one is mixed, and it's worth saying so. Three models separate statements well, and mxbai separates them almost perfectly. Three don't: MiniLM, e5 and Qwen3 sit near chance, so "X is not covered" is as likely as a rewording to look like a duplicate of "X is covered". The models that cope with statements still failed the question test. Part of that is my data: the question rewordings change more words than the statement rewordings do, and on every model the rewordings score lower for questions. That's also how people actually ask questions, so I don't think it makes the question result less real. The safe conclusion isn't "embeddings can't do negation". It's that it depends on the model and the text shape, and you can't know which case you're in without measuring.

Test 3: retrieval, and what fixes it

The cache and deduplication cases compare two texts directly. Retrieval is subtler, because the query and the chunk aren't phrased alike. So I built an index where every item contributes two chunks about different entities: "The Gold plan covers dental implants" and "The Silver plan does not cover dental implants". Then I asked both questions. "Which plan covers dental implants?" should rank Gold's chunk above Silver's, and "Which plan doesn't cover dental implants?" should do the reverse. All 80 chunks sit in the same index, so the other 39 items act as distractors.

Every dense model answered the positive questions well (90–100%). The negated questions are where they split:

Model	Dense alone	Dense + BM25 (rank fusion)	Dense top 10, re-ranked: ms-marco-MiniLM-L-6-v2	Dense top 10, re-ranked: bge-reranker-base
all-MiniLM-L6-v2	27.5%	32.5%	97.5%	100%
bge-base-en-v1.5	85%	70%	97.5%	100%
e5-base-v2	92.5%	77.5%	97.5%	100%
gte-base	70%	57.5%	97.5%	100%
mxbai-embed-large-v1	77.5%	65%	97.5%	100%
Qwen3-Embedding-0.6B	80%	72.5%	97.5%	100%

Share of the 40 opposite questions ("doesn't", "excludes", "no", "unpaid", "limits" and so on) where the right chunk ranked above its opposite. A coin flip scores 50%.

The MiniLM row is worth a second look. Counting an item as right only when both its questions came out right, the way NevIR scores it, MiniLM got 17.5%. Guessing at random would get 25%.

The fix everyone recommends made it worse

The standard advice for negation is hybrid retrieval: the "not" is a keyword, so add BM25 and fuse the rankings. I believed that going in, and it's what the outline for this article said.

In this test, BM25 on its own got 77.5% of the negated questions right, which beats two of the dense models and ties a third. Fusing it with them using reciprocal rank fusion (Cormack, Clarke and Büttcher, 2009, k = 60) lowered the score for five of the six. Only MiniLM improved, from 27.5% to 32.5%. The reason is in the wording. The keyword arm only sees the negation when the question and the chunk use the same word. In 25 of the 40 items they did ("excludes" and "excludes", "does not" and "does not"), and BM25 got all 25 right. In the other 15 they didn't ("doesn't" against "not covered", "won't refund" against "non-refundable", "no API access" against "unavailable"), and BM25 got 6 right, which is below a coin flip. There it rewards whichever chunk shares more terms with the question, and that's often the wrong one. Rank fusion gives that vote the same weight as the dense model's.

There's a second trap underneath, and it's a configuration setting. Elasticsearch's english analyzer and its stop token filter both default to the _english_ stop list, which is Lucene's ENGLISH_STOP_WORDS_SET. That list contains "no" and "not". So the keyword that's supposed to rescue you is deleted before it reaches the index. My BM25 is a simple Python one (rank_bm25, lowercase word tokens, no stemming), not Elasticsearch itself, but I gave it the same stop list. With that list applied, it scored exactly 50% on the negated questions, a coin flip, and every hybrid score dropped further. (The standard analyzer keeps stop words by default. Check which one your text fields actually use.)

What worked was the slow, boring stage. A cross-encoder reads the question and the chunk together, so "doesn't" can change its answer in a way that two separately computed vectors can't. Re-ranking each dense model's top ten with ms-marco-MiniLM-L-6-v2, a 22-million-parameter model, took every row to 97.5% on the negated questions. bge-reranker-base took them all to 100%. The dense model you started with stopped mattering here, because the right chunk and its opposite were both in every top ten. The ordering inside it was the problem. That's an easy case: an 80-chunk index where the two rivals share most of their words. In a large index full of near-identical clauses, check that both make it into the candidates you re-rank. This matches NevIR's ranking, and it's why re-ranking gets a whole lesson of its own in SophiArch's Retrieval & RAG Systems course, two lessons after the one on hybrid retrieval: the two stages fix different failures.

The re-rankers aren't perfect, and both made a couple of mistakes in the other direction. bge-reranker-base put "The standard warranty excludes accidental drops" above "The extended warranty covers accidental drops" for "Which warranty covers accidental drops?". ms-marco-MiniLM missed "Which account offers no overdraft?" and "Who refunds return shipping costs?". e5 also missed the overdraft question, and the opposite of the shipping one ("Who doesn't refund return shipping costs?"). That's 2 misses in 80 questions for each re-ranker, against 3 for the best dense model (e5) and 33 for MiniLM. One caution from Frias's audit: swapping in an NLI model as the gate "did no better" there. A re-ranker trained for search relevance and an NLI classifier aren't the same tool, and short policy questions are an easy case. So re-run the triples with the re-ranker in the loop before you rely on it.

Try it on your model before you set a threshold

This takes a few minutes and needs no labelled data. Write 20 question triples from your own domain: a question, a rewording, and its opposite. Then run:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("your-embedding-model")  # or call your hosted embedding API

triples = [
    ("Which plan covers dental implants?",
     "What plan pays for dental implants?",
     "Which plan doesn't cover dental implants?"),
    # ... 19 more from your own domain
]

backwards = 0
for question, reworded, opposite in triples:
    q, r, o = model.encode([question, reworded, opposite], normalize_embeddings=True)
    sim_r, sim_o = float(q @ r), float(q @ o)
    backwards += sim_o > sim_r
    print(f"{sim_r:.2f}  {sim_o:.2f}  {question}")

print(f"opposite closer than rewording: {backwards}/{len(triples)}")

If the opposite wins most rows, and on every model I tried it did, then no similarity threshold on that model can tell a repeat question from its opposite. Use the same triples for statements if you deduplicate or merge facts.

Where each fix goes

Semantic caches. Don't serve a hit on similarity alone. The cheapest guard is a polarity check: if one question contains a negation word or a word like "exclude" or "without" and the other doesn't, treat it as a miss. A stronger one is to score the (new question, cached question) pair with a cross-encoder before serving. Either way, test the cache with opposite questions, not just rewordings, since rewordings are the only thing most cache evaluations try.
Retrieval. Re-rank the top k with a cross-encoder. In this test, putting it after any of the six dense models beat every combination without one. If you also run BM25, check the analyzer's stop list for "no" and "not", and measure whether fusion helps on your queries rather than assuming it does.
Deduplication and memory. Don't merge or drop a fact because it's close to one you already have. A contradiction is often closer than a duplicate. Compare the claims themselves, with a cross-encoder or an LLM check, before deciding they're the same.

What this test doesn't show

The facts are invented, short and in English, and there are 40 of them. Real policy text is longer, and a longer chunk dilutes the negation further, so I'd expect statement-level separation to get worse, not better. The retrieval index has 80 chunks, so both rivals always reached the top ten. In a large index, recall into the re-ranker becomes a separate thing to measure. The BM25 numbers come from a simple Python implementation without stemming, not from an Elasticsearch analyzer, so treat them as a direction, not a figure for your cluster. I only tested open models; hosted embedding APIs may do better or worse, which is the point of running the triples yourself. And the rewordings change more words than the negations do. That's deliberate, because it's the comparison a cache faces in production (people reword freely, and a negation is one word), but it means the results describe how real questions differ, not a controlled single-word edit.

None of it means vector search is broken. It means a similarity score answers "are these about the same thing?" and gets used as if it answered "do these mean the same thing?". For dental implants, those are opposite answers.

References

Weller, O., Lawrie, D., & Van Durme, B. (2024). NevIR: Negation in Neural Information Retrieval. arXiv:2305.07614 (EACL 2024). Negation-pair benchmark; most retrievers perform at or below random, cross-encoders best.
Cao, H. (2025). Semantic Adapter for Universal Text Embeddings: Diagnosing and Mitigating Negation Blindness to Enhance Universality. arXiv:2504.00584 (ECAI 2025). Negation awareness in general-purpose text embeddings.
Yadav, N. (2026). Temporal Validity in Retrieval Memory: Eliminating Stale-Fact Errors for AI Agents over Evolving Knowledge. arXiv:2606.26511 (v1). Cosine similarity separating contradicted facts from duplicates at AUROC 0.59; the "smaller edit" explanation.
Frias, S. E. (2026). Similarity Gates Approve Reversals: A Validity Audit of Embedding-Cosine Thresholds in Agent Systems. arXiv:2608.10216 (v1). Cosine-threshold gates in agent frameworks (deduplication, semantic caches, drift guards) approving meaning reversals; the NLI drop-in result.
Cormack, G. V., Clarke, C. L. A., & Büttcher, S. (2009). Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods. SIGIR 2009. The rank-fusion method used for the hybrid rows (k = 60).
Redis. RedisVL cache API: SemanticCache. Default distance_threshold=0.1 in cosine distance units.
Elastic. Language analyzers, Stop token filter and Standard analyzer. The english analyzer's _english_ stop list and the standard analyzer's _none_ default.
Apache Lucene. EnglishAnalyzer.java. ENGLISH_STOP_WORDS_SET, which includes "no" and "not".
Model cards for the six embedding models and two re-rankers tested: all-MiniLM-L6-v2, bge-base-en-v1.5, e5-base-v2, gte-base, mxbai-embed-large-v1, Qwen3-Embedding-0.6B, ms-marco-MiniLM-L-6-v2, bge-reranker-base.
SophiArch. Retrieval & RAG Systems. Course covering chunking, hybrid retrieval with rank fusion, query rewriting, cross-encoder re-ranking, retrieval metrics and RAG evaluation.

Most of Your LLM Calls Should Return a Label and a Probability, Not a Sentence

Jason Lau — Wed, 14 Oct 2026 15:44:49 +0000

TL;DR: Routing, triage, moderation and approval prompts pick an answer from a list you wrote in advance. That makes them classifiers, but they usually skip what makes a classifier safe to run: label definitions the model actually follows, a labelled test set, a probability that means something, and a threshold with an escalation path. Schema-constrained output guarantees a valid label, not a right one. In a September 2026 study, typed decision models were given the same definitions with the label names swapped between them. With yes/no names, one model changed 76.9% of its decisions, against 6.5% with 0/1 names, and its ranking went from 93.8% AUC to a below-chance 23.2%. The type-error rate was 0% the whole time. The same test runs on your own prompt in an afternoon and needs no labels. Then check whether the model's confidence ranks its own errors, set the threshold from what a mistake costs, and route everything below it to a person.

The ticket router had been in production for five months when the billing team asked why they kept getting password resets.

It was a single LLM call. The prompt listed six queues (billing, account access, technical, sales, urgent and general), each with a one-line description, and asked for JSON with one field: queue. The response was validated against a schema, so the queue name was always one of the six. The integration had never thrown an error. Latency was fine. The dashboard showed six queues filling up at plausible rates.

When someone finally labelled 500 recent tickets by hand, 40 of them were in the wrong queue. That's 8%, and at 20,000 tickets a month it's about 1,600 tickets a month landing with a team that has to read them, work out they aren't theirs and pass them on. Nothing had flagged it, because nothing in the system could. Every output was a valid queue name.

This example is invented, but the design is common. The fix isn't a better prompt. It's recognising what the call actually is.

It was a classifier the whole time

A call with a fixed set of possible answers, known before any input arrives, is a classification problem. Ticket routing, content moderation, "does this transaction need review", "is this lead qualified", "which of these five intents is the user expressing". These are classifiers, and classification has a mature discipline that most LLM pipelines skip:

A label set with written definitions, where the definition decides the answer, not the everyday meaning of the label's name.
A labelled evaluation set from your own traffic, big enough to measure the error rate you care about.
A score that ranks the model's errors, ideally calibrated so that "0.9" means right about nine times in ten.
A threshold and an escalation path: above the threshold the decision goes through, below it a person decides.

The router had a version of the first item and nothing else. Its prompt returned a label and no score, so there was nothing to set a threshold on and nowhere to send the unsure cases.

Structured output made this harder to see, not easier. Constraining the response to a JSON schema removes the one error you would definitely have noticed, a malformed answer, and leaves every other kind in place. In September, TypeSafe AI took the idea one step further with Jev, a model that doesn't generate text at all. It takes a question, an input and a declared set of options, and returns a probability distribution over those options, so its answer can only ever be one of them. That's a real improvement over parsing free text. It is also a guarantee about the format of the answer, and a study published the same month shows how little that guarantee covers.

A valid label is not the same as a right one

Type-Safe Is Not Error-Free (Sun, Xu, Shi and Yang, Fudan University and USTC, September 2026) tested Jev and two open-weight models built the same way: Laya, built on ModernBERT, and Open-Jev, built on DeBERTa-v3. In all three, each option is a short name plus a definition of what the option means. For example, the name "yes" might carry the definition "A human should inspect this run", and "no" the definition "No human attention is warranted."

The experiment was simple. The authors kept the question, the input, the names and the definition texts exactly the same, and only swapped which definition each name was attached to. The new pairing was written out in the input. A model that reads the definitions picks the same definition in both versions, so it returns the other name. A model that reads the names keeps returning the same name, which now means the opposite. The test set was 1,200 binary decisions across four workflows: invoice reconciliation, agent-trace triage, security-alert classification and customer-service escalation.

Here is what happened to Laya:

Option names	Decisions that flipped	AUC, original pairing	AUC, swapped pairing
yes / no	76.9%	93.8%	23.2%
false / true	49.7%	96.2%	57.7%
rejected / accepted	45.8%	90.8%	63.6%
0 / 1 (neutral)	6.5%	94.2%	92.4%
A / B (neutral)	6.0%	94.2%	94.1%

Selected rows from Table 1 of the paper.

With yes/no names, three decisions in four flipped. The AUC of 23.2% is the more important number. It's below the 50% you'd get by guessing, which means the model's scores still separated the two classes, just in the wrong direction. As the authors point out, AUC doesn't depend on the decision threshold, so "this error cannot be corrected by adjusting the threshold."

Jev itself was less extreme but showed the same pattern. With yes/no names it flipped 32.5% of decisions, 30.4 points more than with 0/1, and its AUC fell from 81.5% to 58.1%. Because Jev isn't deterministic, the authors also ran the original version twice to see how much it changes on its own. At most 1.33% of decisions changed between those runs, so the yes/no flip rate was 24 times that floor.

Across every condition in the paper, the type-error rate was 0%. Every answer was a valid option. The authors' conclusion is the one to remember: "Type safety guarantees where a decision can land, but not what the options mean to the model."

Two results point at the fix. Replacing the names with random five-character strings put all three models back at the neutral-name flip rate, and their accuracy before the swap was within 3.3 points of what they scored with neutral names. The meaning was carried by the definitions, and the model followed them. Polarity mattered more than familiarity, too: "absent/present", "negative/positive" and "rejected/accepted", pairs Laya had never produced during training, still raised its flip rate by 41 points over neutral names.

Why this matters when your names and definitions agree

Nobody deliberately attaches "no" to the definition of yes, so it's fair to ask what this has to do with a production prompt. The swap is a measurement tool, not a production scenario. It's the cleanest way to find out whether a model reads your definitions or your label names. On ordinary inputs the two agree, so accuracy can't tell you which one the model is following.

It matters wherever your definition is narrower than the word. The router's "urgent" queue was defined as "a production system is down or customers cannot log in." A ticket that says "I urgently need a copy of last month's invoice" is a billing ticket by that definition and an urgent one by the dictionary. If the model weights the name over the definition, that's where it goes. The same applies to "spam" defined to exclude legitimate cold sales email, "fraud" defined as a specific set of chargeback patterns, or "qualified" defined by your sales team's criteria rather than the word's general sense.

None of this is new for language models in general. Wei et al. (2023) showed that small language models "ignore flipped labels presented in-context and thus rely primarily on semantic priors from pretraining," that overriding those priors is "an emergent ability of model scale," and that instruction tuning "strengthens both the use of semantic priors and the capacity to learn input-label mappings, but more of the former." Liusie, Manakul and Gales (2023) found that in zero-shot prompt classifiers "the precise choice of the prompt template and label words can largely influence performance, with semantically equivalent settings often showing notable performance difference." What the 2026 paper adds is a clean way to measure the effect, run on the newest kind of model built to avoid it.

Run the swap on your own prompt

The useful part is that the test needs no labels. It only needs your prompt, a sample of real inputs and two extra calls per input. Here's a minimal version for a prompt that lists options as name: definition. classify stands for your existing call, constrained to return one of the allowed names:

def build_prompt(text, options):
    """options: list of (name, definition) pairs, shown in this order."""
    listed = "\n".join(f"- {name}: {definition}" for name, definition in options)
    return (
        "Pick the option whose definition fits the ticket. "
        "Reply with the option name only.\n\n"
        f"Options:\n{listed}\n\nTicket:\n{text}"
    )

def picked_definition(text, options, classify):
    name = classify(build_prompt(text, options), allowed=[n for n, _ in options])
    return dict(options)[name]  # compare what was chosen by meaning, not by name

def flip_rate(texts, names, definitions, classify, shift=1):
    """Share of inputs where moving each name to another definition changes the decision."""
    original = list(zip(names, definitions))
    moved = list(zip(names[shift:] + names[:shift], definitions))  # shift=0 measures run-to-run noise
    flips = sum(
        picked_definition(t, original, classify) != picked_definition(t, moved, classify)
        for t in texts
    )
    return flips / len(texts)

Run it three ways on a few hundred real inputs:

Your real names, shifted (shift=1). This is the number you're after.
Neutral names, shifted: replace your names with A, B, C and so on and run it again. This is the control, since moving neutral names around should change nothing.
Your real names, not shifted (shift=0): the same prompt twice. This is the floor, because a model sampled at a non-zero temperature changes some answers between identical runs.

If the first number is close to the other two, the model is reading your definitions. If it's far above them, your definitions are partly decoration, and the label names are making some of the decisions. In that case the fix the paper suggests is cheap: use neutral IDs as option names, put all the meaning in the definitions, and map the ID back to your queue name in code. Then re-run your accuracy check, because the change isn't automatically free. On questions with three to sixteen options, even switching Laya to neutral letters changed about half its decisions, though that model was only 56.4% accurate on those questions to begin with. Measure your own prompt rather than assume.

A caveat on scope: the paper tested typed decision models, not chat models answering through a JSON schema. Whether your model behaves like Laya, like Jev or like neither is exactly what these three numbers tell you.

A probability you can set a threshold on

The second thing the router lacked was a score. There are two common ways to get one from an LLM, and only one of them is worth much.

Asking the model for its confidence. Adding a "confidence" field to the schema is easy and mostly uninformative. In a large evaluation of this approach, Xiong et al. found that LLMs "tend to be highly overconfident when verbalizing their confidence," and that the values "predominantly fall within the 80% to 100% range and are typically in multiples of 5, similar to how humans talk about confidence." In the router example, the model reported 0.9 or higher on 488 of the 500 labelled tickets. A threshold has nothing to separate.

Reading the probability of the label itself. If your provider exposes token log-probabilities for the chosen output (not every API or model does), use the probability the model assigned to the label it picked. Single-token, distinct option names make this straightforward. It's a real score, but don't assume it's calibrated. The GPT-4 technical report showed that "the pre-trained model is highly calibrated," but "after the post-training process, the calibration is reduced," and its calibration plot is captioned "the post-training hurts calibration significantly." Typed decision models like Jev return a distribution by design, which is one of their real advantages. The same checks still apply.

What you need to know is whether the score ranks the model's mistakes. Group your labelled set by score and count how often each group is right. Here is what that table might look like for the router (illustrative numbers):

Probability of the chosen queue	Tickets	Correct	Share correct
0.95 and above	330	323	97.9%
0.80 to 0.95	90	81	90.0%
0.60 to 0.80	50	36	72.0%
below 0.60	30	20	66.7%
All	500	460	92.0%

The errors aren't spread evenly. A third of the tickets (the 170 below 0.95) contain 33 of the 40 mistakes. That's what makes a threshold possible.

The table also shows the scores aren't perfectly calibrated: the tickets below 0.60 are right two times in three. If you want the number itself to be trustworthy, fit a calibration correction on a held-out slice of your labels: temperature scaling, which Guo et al. (2017) found "surprisingly effective" for neural networks, or Platt scaling or isotonic regression on the raw scores. Reliability diagrams and those correction methods are the subject of the calibration lesson in SophiArch's Applied Machine Learning course, which works through them on gradient-boosted and random-forest models, whose raw probabilities need the same kind of correction.

Pick the threshold from what a mistake costs

A threshold isn't a quality setting. It's a trade between two costs, and you should write both down. Suppose a misrouted ticket costs about 20 minutes of someone's time (the wrong team reads it, works out it isn't theirs, passes it on, and the customer waits longer), and a person sorting a ticket by hand takes about 1 minute and gets it right. Per 500 tickets:

Auto-route if probability is at least	Auto-routed	Errors that get through	Sent to a person	Cost in minutes
no threshold	500	40	0	40 × 20 = 800
0.60	470	30	30	600 + 30 = 630
0.80	420	16	80	320 + 80 = 400
0.95	330	7	170	140 + 170 = 310

At these costs, sending everything below 0.95 to a person cuts the cost of routing by about 60% (from 800 minutes per 500 tickets to 310). If sorting a ticket by hand took three minutes instead of one, the 0.80 threshold would win: 320 + 240 = 560 minutes against 140 + 510 = 650. The right threshold comes from your costs, not from the model.

The tickets sent to a person do something else too. Every one of them comes back with a correct label, from exactly the cases the model was least sure about. After a few months that's a better training set than you could have designed.

Climbing the ladder

Once a decision has labels, a score and a threshold, you can compare different ways of making it on equal terms. A reasonable order to try them in:

The model you already use, with neutral option IDs, output constrained to the label set, and the probability of the chosen label as the score. Stop here if the accuracy, the swap test and the score table are all acceptable at your volume.
The same model with a calibration correction fitted on held-out labels, if you need the probability to mean what it says, for example because a downstream rule reads it as a risk level.
A small classifier trained on your labels, now that the escalation queue is producing them. Even a logistic regression on TF-IDF features is worth trying as a baseline, because it's deterministic, its probabilities can be calibrated with standard tools, and it costs almost nothing to run. Whether a fine-tuned or distilled small model is worth the step up is a separate cost question.
A purpose-built decision model such as Jev. These return a distribution by design and can be fast. Run the swap test on them too, because that's the kind of model the paper studied.

The exit test is the same at every rung: accuracy, the score table and the swap flip rate, all on your own labelled traffic. Not the vendor's benchmark, and not the model's own description of how confident it is.

Five questions for any LLM call that returns a label

Is the answer always one of a fixed set? Then it's a classifier, whatever the call looks like.
Do you have a few hundred real inputs labelled by a person, and do you know the error rate on them?
When you move the names to other definitions, how many decisions change compared with neutral names and with a plain re-run?
Does the score put the mistakes at the bottom of the table, and is anything reading it as a probability?
Where do the low-scoring cases go, and who labels them?

The router team answered all five in two weeks. The billing queue stopped getting password resets, and the "urgent" queue got a lot quieter.

References

Sun, Y., Xu, J., Shi, J., & Yang, Z. (2026). Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to It. arXiv:2609.26758. The name–definition swap on Jev, Laya and Open-Jev; source of the flip rates, AUC figures, test-retest floor, random-string control, multi-option results and the neutral-name mitigation.
TypeSafe AI. Introducing System One models and Jev. Launch post for the typed decision model studied above.
Wei, J., Wei, J., Tay, Y., et al. (2023). Larger language models do in-context learning differently. arXiv:2303.03846. Flipped-label and semantically unrelated label experiments; semantic priors and instruction tuning.
Liusie, A., Manakul, P., & Gales, M. (2023). Mitigating Word Bias in Zero-shot Prompt-based Classifiers. arXiv:2309.04992. How the choice of label words shifts zero-shot classifier performance, and a reweighting fix.
Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., & Hooi, B. (2024). Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. arXiv:2306.13063 (ICLR 2024). Overconfidence and the 80–100% clustering of verbalized confidence.
OpenAI (2023). GPT-4 Technical Report. arXiv:2303.08774. Calibration of the pre-trained versus post-trained model on MMLU (Figure 8).
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. arXiv:1706.04599 (ICML 2017). Miscalibration in modern networks and temperature scaling.
SophiArch. Applied Machine Learning. Course covering evaluation with statistical rigour, imbalanced data, cross-validation and leakage, calibration with reliability diagrams and Platt/isotonic scaling, and production pipelines.

Most Multi-Agent Systems Should Be One Agent With Better Tools

Jason Lau — Wed, 07 Oct 2026 03:08:13 +0000

TL;DR: Most published multi-agent wins compare a team of agents against a single agent given far less compute. Studies that match the budget find the advantage mostly disappears: automatically generated multi-agent systems underperformed a simple self-consistency baseline while costing up to 10x more, and on a sequential planning benchmark every multi-agent variant tested scored 39–70% worse than a single agent. In a worked five-agent refund pipeline, 64% of the tokens go on duplicated context and handoff notes, and a fact dropped in one handoff produces a wrong refund. Merging the agents into one fixes the lost fact but not the bill. Redesigning the tools does both, cutting the cost to under half. The test for whether you need a second agent: can you say what it would be evaluated on by itself? Parallel research, context isolation, separate permissions and verification pass that test. Splitting work by job title (planner, coder, tester, reviewer) usually doesn't. Before adding an agent, give your single agent the same token budget and compare them on your own labelled cases.

"Add another agent" has become the default answer to an agent that isn't good enough. The planner misses things, so a reviewer agent is added. The reviewer lacks context, so a researcher agent is added to feed it. Six months later there's an org chart of five agents, each with its own prompt, and nobody can say which one is responsible when a customer gets the wrong answer.

The 2026 name for everything around the model is the harness: tools, permissions, loops, memory, verification. Multi-agent design is a harness decision, and usually the most expensive one available. This article makes the case that most multi-agent systems are one agent whose tools are badly designed. It also gives a test for the cases where that isn't true: if you can't say what each agent would be evaluated on by itself, you have one agent.

A support pipeline with five agents

Here's an invented but typical example: a B2B software company handling refund requests. The team built it the way the architecture diagrams suggest:

A router reads the ticket and decides what kind of request it is.
An account agent looks up the customer and summarises their account.
A billing agent pulls invoices and payments and summarises billing history.
A policy agent reads the refund policy and decides what the customer is owed.
A writer drafts the reply.

Each agent passes a short written note to the next. It demos well. Every agent has a clear job title, and the diagram has five neat boxes.

Then a ticket comes in from a customer cancelling an annual plan. They paid $4,800 in March. In June they downgraded and received a $1,200 credit. The billing agent's note reads: "Annual plan, $4,800, renewed 3 March, no open disputes." All of that is true. The June credit is in the raw ledger the billing agent read, but it isn't in the note, because the note had a length limit and the agent picked what looked most relevant. The policy agent never sees the ledger, only the note. It calculates a pro-rated refund on the full $4,800. The writer promises it to the customer.

No agent made a mistake given what it was shown. The failure is in the handoff, and debugging it means opening five traces and working out which summary dropped which fact. Anthropic calls this the "telephone game": when agents are split by problem type, they end up passing "information back and forth with each handoff degrading fidelity."

Dat Tran and Douwe Kiela put a formal argument behind the same intuition, using the Data Processing Inequality from information theory. Every agent downstream of a handoff works from a processed version of the original context, and processing can lose information but never add it. A single agent with the whole context in front of it doesn't pay that cost. Their argument also says when multi-agent designs can come out ahead. One case is when a single agent's use of its own context breaks down: "once effective single-agent context utilization deteriorates sufficiently, a well-designed MAS may recover task-relevant information more reliably than a degraded single pass." The other is when the multi-agent system simply gets to spend more compute. That second condition turns out to explain a lot of the published results.

The comparisons that flattered multi-agent systems

Most of the evidence for multi-agent designs compares a multi-agent system against a single agent that was given much less to work with. That's the confound each of the three studies below goes after.

Match the compute and the advantage mostly disappears. Tran and Kiela's starting point is that reported multi-agent gains are often confounded by extra test-time computation. They compared single-agent and multi-agent setups on multi-hop reasoning, across three model families, with the thinking-token budget held equal, and found single agents matched or beat the multi-agent systems once computation was normalised. A typical row from their results: on four-hop MuSiQue questions with DeepSeek-R1-Distill-Llama-70B at a 2,000-token budget, the single agent scored 0.418, a sequential multi-agent pipeline 0.327, and no multi-agent design in that row scored above 0.352. Their error analysis found the sequential pipeline "over-explores and drifts," and their conclusion is that "many reported MAS gains are better explained by compute and context effects than by inherent architectural superiority."

Automatically generated multi-agent systems lost to a simple baseline at a fraction of the cost. The Illusion of Multi-Agent Advantage (Jwalapuram et al., Salesforce Research and collaborators, June 2026) tested six frameworks that design multi-agent systems automatically against one single-agent baseline: chain-of-thought with self-consistency, which just samples the same model five times and takes a majority vote. The authors point out that earlier comparisons "rarely control for inference budgets such as number of LLM calls, total cost, retries, or sampled paths." When they controlled for them, the result was blunt: "automatic MAS consistently underperform CoT-SC despite being up to 10x more expensive."

The paper then took the generated systems apart to see what the extra agents were doing. Mostly, nothing useful:

In DyLAN, a framework built on the premise that agent diversity drives performance, agents "reach immediate, unanimous consensus in ∼70% of GPT-4o cases and >90% of GPT-5 cases." Agents that agree instantly are one agent's answer, paid for several times over.
Replacing the task-specific expert roles with generic "assistant" agents scored slightly better (54.4% vs. 53.4%).
In a framework that searches for the best workflow, 7 of the 14 final workflows it found were a single prompt run three times and aggregated, which is self-consistency under a more complicated name.

The authors' name for agents that cost full price and change nothing is "expensive witnesses."

Sequential and tool-heavy work gets worse, not better. Google Research and MIT's Towards a Science of Scaling Agent Systems (Kim et al.) is the largest study of the three: 260 configurations across six agentic benchmarks, five architectures and three model families, with tools, prompts and compute standardised so that only the architecture varied. Three of its findings map directly onto production systems:

On a planning benchmark built from sequential constraints, "all multi-agent variants universally degrade performance," by 39% to 70%. A refund decision is a sequential task. Each step depends on what the previous one found.
Under fixed budgets, "tool-heavy tasks suffer disproportionately from multi-agent inefficiency," because splitting the budget across agents leaves each one less room to use its tools.
Once a single agent already scored above roughly 45%, adding agents gave negative returns.

The same paper also shows where multi-agent designs do win. Centralised coordination improved performance by 80.8% on a decomposable financial-reasoning benchmark. So "never use multiple agents" is the wrong conclusion. The right one is narrower: the benefit is real for work that splits into genuinely independent pieces, and negative for most other work.

Even Anthropic's widely cited result needs reading carefully. Its multi-agent research system "outperformed single-agent Claude Opus 4 by 90.2% on our internal research eval," on the kind of breadth-first research that parallelises well. The same post reports that "token usage by itself explains 80% of the variance" in performance on the BrowseComp benchmark, and that "multi-agent systems use about 15× more tokens than chats." That 15× is measured against a chat interaction, not against a single agent. Much of the improvement came from spending more tokens, which a single agent can also do. The post is clear about where it doesn't apply: "most coding tasks involve fewer truly parallelizable tasks than research."

Where the tokens go

Back to the support pipeline. Here's the token accounting for one ticket, using made-up but plausible sizes: a 1,200-token ticket, an 800-token system prompt per agent, 1,500 tokens from each account and billing lookup, and 2,000 tokens of retrieved refund policy.

Agent	System prompt	Ticket	Notes read	Tool results	Input	Output
Router	800	1,200	—	—	2,000	250 (note)
Account	800	1,200	250	1,500	3,750	400 (note)
Billing	800	1,200	650	1,500	4,150	400 (note)
Policy	800	1,200	1,050	2,000	5,050	400 (note)
Writer	800	1,200	1,450	—	3,450	350 (reply)
Total	4,000	6,000	3,400	5,000	18,400	1,800

That's 20,200 tokens per ticket. Now split it by what the tokens were for. The work only one agent had to do is reading the ticket once, one system prompt, the 5,000 tokens of tool results and the 350-token reply: 7,350 tokens, or 36%. The other 64% is the cost of there being five agents. The ticket is read four extra times (4,800), there are four extra system prompts (3,200), and handoff notes are written (1,450) and then re-read by every later agent (3,400). Anthropic's own guidance describes the same breakdown: "In our testing, multi-agent implementations typically use 3-10x more tokens than single-agent approaches for equivalent tasks," from "duplicating context across agents, coordination messages between agents, and summarizing results for handoffs."

Now the part that surprises people. Collapse the five agents into one agent that calls the same three tools, and the saving almost disappears. An agent loop re-reads its growing context on every turn, so four turns (three tool calls and a reply) with a 1,200-token system prompt come to about 19,700 input tokens and 650 output: 20,350 in total. It's no cheaper. It does fix the refund, because the one agent read the raw ledger and saw the June credit. But the token bill is the same.

The saving comes from the tools. The original lookups return raw API output, full invoice objects and the whole policy document, because the tools were written to return whatever the backend returned. Rewrite them around what the agent actually needs to decide:

get_billing_context(customer_id) returns one pre-joined record of about 600 tokens: plan, amount paid, and every credit, refund and plan change with its date.
check_refund_eligibility(customer_id, reason) runs the refund policy as code and returns about 150 tokens: eligible or not, the maximum amount, which rule applied, and why.

The loop becomes three turns (look up, check eligibility, reply): about 8,850 input and 550 output tokens, or 9,400 in total. That's under half the five-agent pipeline, and it gets the June credit right, because the credit is a field in the tool's output and not something a summary might drop. Prompt caching would lower the repeated system-prompt cost further. Your ratios will differ with your sizes, but the pattern holds. Removing agents fixes the lost context. Better tools reduce the cost.

This accounting is only possible if you can see the assembled prompt and the per-step token count for every call, which most orchestration frameworks hide by default. The orchestration lesson in SophiArch's AI Applications with LLMs course starts from that problem: work out what the framework conceals before deciding whether to use one.

Notice what happened to the policy agent. It became a function. That's common. Look closely at a multi-agent diagram and you'll often find a business rule that has been given a prompt and a job title. A rule written as code can be unit-tested, never paraphrases the policy, and costs nothing per call.

The test: what would each agent be evaluated on by itself?

Better tools are now standard advice. Anthropic's guide puts it plainly: "a well-designed single agent with appropriate tools can accomplish far more than many developers expect," and notes that "we've seen teams invest months building elaborate multi-agent architectures only to discover that improved prompting on a single agent achieved equivalent results." The advice exists, and teams still reach for a second agent, so something more concrete is needed at the moment of that decision.

Here's the test. For every agent in the design, write down the evaluation you would run on that agent by itself, with inputs, a pass condition, and a labelled set to run it on, without running the rest of the system. If you can write it, that agent may be worth separating. If the only way to judge the agent is to look at the final output of the whole system, it isn't a separate agent. It's a step inside one.

Apply it to the support pipeline:

Agent	Independent evaluation?	Verdict
Router	Yes: routing accuracy against labelled tickets. But it has a fixed label set, no tools and no loop.	A classification call, not an agent
Account agent	The lookup can be tested (right customer, right records). The summary can't: "complete enough" depends on what the next agent needs.	A tool
Billing agent	Same as above. The note that dropped the June credit had no pass condition it could have failed.	A tool
Policy agent	Yes: the correct outcome for each policy case is known in advance.	A function, tested with unit tests
Writer	Only by judging the final reply, which is the whole system's output.	The agent

Five agents turn into one agent, two tools, a function and a classifier. Nothing was lost. Every part that could be tested separately is now easier to test than before, because a function with fixed cases is easier to test than a prompt.

The test also explains the published results. The successful system in The Illusion of Multi-Agent Advantage was the one the authors designed by hand for a task built to reward decomposition: a finance problem with independent per-investor calculations over a large table of prices. Their "Expert-MAS" used an extractor agent that retrieves specific data and a calculator agent that reasons over isolated snippets, both of which can be evaluated on their own. Coordination was done by a deterministic Python orchestrator, and the final comparison was computed in code. With GPT-5 it scored 96.5%, against 57.0% for the self-consistency baseline, "with cost comparable to CoT-SC." The multi-agent system that worked was mostly a program, with the model called only for the pieces that could each be checked.

When a second agent passes the test

There are real cases, and they're narrow enough to name. Each one comes with an evaluation that doesn't need the rest of the system:

Genuinely parallel work. Five independent research questions, five subagents, each judged on whether it found the facts for its own question. This is the case Anthropic's research system and the 80.8% financial-reasoning result both come from.
Context protection. A subagent reads 300 pages of logs and returns 20 lines. Evaluation: does the summary contain the line that explains the incident? The main agent's context stays clean. This is also the condition in which Tran and Kiela's argument says a multi-agent design can come out ahead: when one agent's context would otherwise be too long or too noisy to use well.
Different permissions. A subagent that can write to a production system, and nothing else, evaluated on whether the end state is correct and it took only allowed actions. The separation exists because of what it's allowed to do, not because of its job title.
Verification. Anthropic's guide calls the verification subagent "one multi-agent pattern that consistently works well across domains." It has the cleanest independent evaluation of all: seed known defects and measure how many it catches. Kim et al.'s error-amplification finding points the same way. Independent agents with no checkpoint amplified errors 17.2×, while centralised coordination with a validation step held it to 4.4×.

What's missing from that list is splitting by job title. Planner, implementer, tester and reviewer is the most common multi-agent design and the one the evidence argues against most directly. Anthropic's guide says "dividing by type of work (one agent writes features, another writes tests, a third reviews code) creates constant coordination overhead," and in their own experiment with those roles "the subagents spent more tokens on coordination than on actual work." A recent preprint, At Equal Inference Cost, Multi-Agent Structure Does Not Beat a Single Frozen Agent (Dylan et al.), tested the planner–executor–critic split directly, holding the number of model calls fixed and optimising each role's prompt. All of the gain came from the executor. The planner and critic prompts "froze empty," and the full team (0.769) was not statistically separable from the single agent (0.754) while using 1.8× the calls. It's a small study, with one 7B model and two benchmarks, but it points the same way. Berkeley's study of multi-agent failures (Cemri et al.) found failure rates of 41% to 86.7% across seven open-source multi-agent frameworks, and grouped 14 failure modes under three headings: system design, misalignment between agents, and task verification. The middle one, misalignment between agents, is a failure a single agent can't have at all.

Run the budget-matched comparison this week

If you already run a multi-agent system, there's a cheap way to find out whether it's earning its cost. It needs one thing you may already have: a set of labelled cases with a checkable pass condition.

def budget_matched(cases, run_multi, run_single):
    """Compare a multi-agent system against one agent given the same token budget."""
    rows = []
    for case in cases:
        multi = run_multi(case.input)                 # -> answer, tokens_used
        single = run_single(case.input,               # same model, all the tools,
                            token_budget=multi.tokens_used)  # spend up to the same budget
        rows.append({
            "multi_pass": case.passes(multi.answer),
            "single_pass": case.passes(single.answer),
            "multi_tokens": multi.tokens_used,
            "single_tokens": single.tokens_used,
        })
    return rows

How run_single spends the budget is up to you: more reasoning tokens, or several samples with a majority vote, which is exactly the baseline the Illusion paper used. The one rule is that it gets the same budget as the multi-agent run, not a fraction of it. Compare the tokens each side actually used, not the budget you asked for. Tran and Kiela found that with Gemini 2.5, "the visible thought text produced by SAS tends to plateau well below the requested budget, while MAS surfaces more visible thought content under the same requested budget B, due to multiple calls." A single agent that stops early hasn't had a fair comparison. Then read the results in this order:

If the single agent matches or beats the multi-agent system at the same budget, the extra agents are spending tokens, not adding capability.
If the multi-agent system wins, look at which cases it wins. If they're the cases that split into independent parts, you've found a real use for a second agent. Keep it, and give that agent its own evaluation.
Either way, look at the single agent's tool calls on the cases it failed. Tools that return raw API output, and business rules the model has to re-derive from a document every time, are where the next improvement is.

The one-line version

A second agent is worth adding when you can evaluate it without the rest of the system. When you can't, what you actually need is a better tool, a function or a cleaner context, and all three are cheaper, faster and easier to test than another agent.

References

Jwalapuram, P., Lin, H., Li, C., Jiao, F., Wang, S., Ming, Y., Ke, Z., Qin, C., Carenini, G., & Joty, S. (2026). The Illusion of Multi-Agent Advantage. arXiv:2606.13003. Six automatic multi-agent frameworks against budget-controlled chain-of-thought with self-consistency; source of the "up to 10x more expensive" result, the consensus and role-ablation findings, and the Expert-MAS result.
Tran, D., & Kiela, D. (2026). Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets. arXiv:2604.02460. Equal-budget comparison, the Data Processing Inequality argument, the MuSiQue results and the budget-tracking caveat.
Kim, Y., et al. (2025, revised 2026). Towards a Science of Scaling Agent Systems. arXiv:2512.08296v3. 260 controlled configurations across six agentic benchmarks, five architectures and three model families; source of the sequential-planning degradation, tool-coordination trade-off, capability threshold, Finance Agent and error-amplification figures.
Dylan, D., Brennan, A., Murphy, C., O'Sullivan, N., Kelly, C., & Walsh, S. (2026). At Equal Inference Cost, Multi-Agent Structure Does Not Beat a Single Frozen Agent. arXiv:2609.04217. Planner–executor–critic team against a single agent at a fixed number of model calls.
Cemri, M., Pan, M. Z., Yang, S., et al. (2025). Why Do Multi-Agent LLM Systems Fail? arXiv:2503.13657. 1,642 annotated traces from seven frameworks; the MAST taxonomy of 14 failure modes.
Hadfield, J., Zhang, B., Lien, K., Scholz, F., Fox, J., & Ford, D. (2025, June 13). How we built our multi-agent research system. Anthropic Engineering. Source of the 90.2%, 15×-versus-chat and 80%-of-variance figures.
Phillips, C. (2026, January 23). Building multi-agent systems: When and how to use them. Claude blog. Source of the 3–10× single-agent comparison, the "telephone game" and type-of-work decomposition warnings, and the verification-subagent pattern.
SophiArch. AI Applications with LLMs. Course covering the probabilistic contract, context window architecture, output validation layers, evaluation frameworks, orchestration patterns (including what frameworks hide), and cost and latency engineering for LLM systems.

Your First Model Should Be Embarrassing

Jason Lau — Wed, 30 Sep 2026 08:29:33 +0000

TL;DR: Before anything clever, fit a model that guesses the majority class and a plain logistic regression. Together they take seconds, and they're the only way to tell what the expensive model actually bought you. On six public datasets, guessing alone was 95.3% accurate on hypothyroid. Default boosted trees beat logistic regression by 2 to 15 AUC points. A 200-fit hyperparameter search added more than half a point on only one dataset, and across four random splits its gain on the other five averaged between −0.3 and +0.2 points. Skip the cheap models and the complex one takes credit for all of it.

Ask an AI coding assistant to "build a model to predict churn" from a CSV and you'll often get a notebook that looks finished: preprocessing, a gradient-boosted model, a hyperparameter search, a classification report, maybe a feature-importance chart. Before you read any of the numbers, search the notebook for one thing: a model that ignores the features entirely.

If there isn't one, none of the numbers mean anything yet. "95% accuracy" is excellent if guessing gets you 60%, and embarrassing if guessing gets you 95%. A 0.93 AUC from a tuned ensemble is a big win if a plain logistic regression gets 0.80, and the ensemble is mostly overhead if the logistic regression gets 0.92. The assistant didn't do anything wrong. It answered the question you asked, and "build a model" doesn't ask for a floor to measure it against.

So the first model you fit should be one you'd be embarrassed to ship. It isn't there to be deployed. It's there to set the score everything else has to beat.

Four models, six datasets

To see how much the floor changes the story, here are four models of increasing effort on six public binary-classification datasets from the Penn Machine Learning Benchmark (PMLB) collection:

Guess the majority class. scikit-learn ships this as DummyClassifier, and its documentation is blunt about the job: it "makes predictions that ignore the input features" and "serves as a simple baseline to compare against other more complex classifiers."
Logistic regression, with one-hot encoded categories and scaled numeric columns. Nothing tuned.
Gradient-boosted trees on default settings (scikit-learn's HistGradientBoostingClassifier, a histogram-based implementation inspired by LightGBM). On the two datasets with more than 10,000 rows, the defaults also switch on early stopping, which is a small amount of built-in tuning.
The same boosted trees after a hyperparameter search: 40 random configurations over learning rate, tree size, leaf size, regularisation and number of rounds, each scored with 5-fold cross-validation. That's 200 model fits plus a final refit of the winner, and it's the step that makes a notebook look thorough.

Each dataset gets one stratified 80/20 train/test split. The search only sees the training data. Scores are on the held-out 20%. Categorical columns are one-hot encoded for the logistic regression and passed to the trees as native categories. The churn data's phone-number column, which is a unique ID per customer, is dropped.

Dataset (rows)	Guess the majority	Logistic regression	Boosted trees, defaults	Boosted trees, tuned
Census income (48,842)	76.1% · 0.500	85.6% · 0.909	87.7% · 0.931	87.6% · 0.931
Telecom churn (5,000)	85.9% · 0.500	86.6% · 0.797	95.5% · 0.929	95.3% · 0.923
Spam email (4,601)	60.6% · 0.500	93.2% · 0.970	96.0% · 0.990	96.0% · 0.990
Gamma telescope (19,020)	64.8% · 0.500	79.3% · 0.847	89.0% · 0.942	89.3% · 0.946
Hypothyroid (3,163)	95.3% · 0.500	96.2% · 0.917	98.3% · 0.989	98.0% · 0.983
Phoneme (5,404)	70.7% · 0.500	74.0% · 0.803	90.1% · 0.957	92.0% · 0.967

Each cell shows test accuracy, then ROC AUC. The majority guess scores 0.500 AUC by definition, because it gives every row the same score. Bold is the best AUC in the row, after rounding. The default trees fit in under a second on every dataset. The search took between 43 and 152 seconds per dataset on a 4-core machine. Run with scikit-learn 1.9.1.

Each rung of that ladder answers a different question, and the rung below it supplies the number you need to answer it.

What the bottom rung tells you

Look at the accuracy column for hypothyroid. A model that has never looked at a patient, and says "no" every time, is 95.3% accurate. That's what 4.8% prevalence does to accuracy. The logistic regression's 96.2% sounds strong on its own, and it's less than one point above doing nothing. Telecom churn is the same: 85.9% for predicting that nobody leaves, 86.6% for the logistic regression. Report "our churn model is 87% accurate" in a meeting and it sounds like a result. Next to the majority-class guess, it's close to nothing.

This is the cheapest check in applied ML, and it catches the most embarrassing failures before anyone else does. It also shows that accuracy is the wrong headline metric for imbalanced problems like these two. That's a larger topic, but you only notice the problem once you have the floor.

What the middle rung tells you

The jump from logistic regression to default boosted trees is where the real decisions are, and the table shows two very different situations.

On telecom churn, phoneme and the gamma telescope data, the trees add 9 to 15 points of AUC (a point here is 0.01). Hypothyroid is less settled: 7 points on this split, but anywhere from 2 to 8 across the four splits described below. The gap comes from interactions and non-linear thresholds that a plain linear model can't represent. Some of it could be won back with feature engineering, such as binning the number of customer-service calls in the churn data, so treat this logistic regression as a floor, not the best simple model possible. Even so, on those datasets the trees are easily worth the loss of interpretability. On census income and spam, the gain is about 2 points. Whether 2 points is worth giving up coefficients you can read out to a risk committee, a regulator or a customer is a business decision. It might well be. But you can only make that decision if you have the logistic regression number. Start with the ensemble and the question never comes up.

What the top rung tells you

The tuned search is the most expensive step by two orders of magnitude, and on this run it was the least informative. It improved AUC by more than half a point on one dataset of six (phoneme, +1.0). On census income and spam it changed nothing. On churn and hypothyroid the tuned model scored slightly worse on the test set than the defaults. That's most likely the search fitting noise in the cross-validation folds, which is easy to do when you pick the best of 40 configurations scored on a few thousand rows.

This is not a claim that tuning never matters. Phoneme gained a real point, and on other data it can matter a lot. McElfresh et al., comparing 19 algorithms across 176 datasets, found that "for a surprisingly high number of datasets, either the performance difference between GBDTs and NNs is negligible, or light hyperparameter tuning on a GBDT is more important than choosing between NNs and GBDTs." Note what that compares: light tuning against switching model families, not tuning against good defaults. On these six datasets, the defaults were already close to what a 200-fit search found. The only way to know which situation your dataset is in is to have the untuned number to compare against. Without it, the search's best score looks like something the search earned.

A caveat that works in the argument's favour: the test sets here range from 633 rows to about 9,800. A half-point AUC difference on a thousand held-out rows is within the variation you'd get from a different random split. Re-running the whole comparison on three more random splits didn't change the picture. Phoneme gained about a point from tuning every time. On the other five datasets the tuning gain averaged between −0.3 and +0.2 points across the four splits. On four of them it landed on both sides of zero from one split to the next. Gamma telescope gained every time, but only 0.1 to 0.4 points. The honest reading of the tuned column is "indistinguishable from defaults" on most rows, which is exactly the thing you can't see without a default to compare.

The research literature keeps getting embarrassed the same way

None of this is new. Whole subfields have found out, sometimes years late, that their sophisticated methods hadn't been compared against a properly built simple one.

In recommender systems, Ferrari Dacrema, Cremonesi and Jannach tried to reproduce 18 neural recommendation algorithms from top conferences in 2019. They could reproduce 7 with reasonable effort, and 6 of those 7 could often be beaten by simple nearest-neighbour or graph-based heuristics. The paper won best paper at RecSys that year. The problem hasn't gone away. A May 2026 study (Han et al.) tested an "intentionally simple graph heuristic" against modern generative recommenders. It starts from only the last one or two items a user interacted with and uses no training at all. It "matches or outperforms many modern baselines, with relative NDCG@10 improvements of 38.10% and 44.18% over the best competing baseline" on two widely used Amazon review datasets, and stays competitive on 10 of the 14 datasets tested.

In time-series forecasting, Zeng et al. introduced what they called "a set of embarrassingly simple one-layer linear models" as a sanity check against the Transformer architectures that had dominated long-horizon forecasting research. Their result: "LTSF-Linear surprisingly outperforms existing sophisticated Transformer-based LTSF models in all cases, and often by a large margin." Later Transformer designs such as PatchTST did beat those linear models, which is the point: once a simple baseline existed, the bar went up.

In clinical prediction, a systematic review by Christodoulou et al. (Journal of Clinical Epidemiology, 2019) pooled 282 comparisons between logistic regression and machine-learning models from 71 studies. Among the 145 comparisons at low risk of bias, the difference in discrimination between the two was 0.00 on the logit(AUC) scale (95% CI −0.18 to 0.18). A machine-learning advantage only appeared in the 137 comparisons at high risk of bias. Their conclusion: "We found no evidence of superior performance of ML over LR."

And on ordinary tabular data, Grinsztajn, Oyallon and Varoquaux benchmarked deep learning against tree ensembles on 45 datasets and found that "tree-based models remain state-of-the-art on medium-sized data (∼10K samples) even without accounting for their superior speed", which is why trees, not neural networks, are rung three above.

The common thread isn't "simple models are better". Sometimes they're not, as the churn row shows. It's that the complicated method was never properly compared against the simple one. In a research paper that costs credibility. In a company it costs a production system that's harder to explain, slower to retrain and more fragile than it needed to be, delivering an improvement nobody measured.

Reproducing the bottom three rungs

Here are the first three rungs for the census-income dataset. It needs pip install pmlb scikit-learn and runs in a few seconds. It reproduces the table's numbers on scikit-learn 1.9.1; other versions can differ in the third decimal.

from pmlb import fetch_data
from sklearn.compose import make_column_transformer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

df = fetch_data("adult")
y = (df.pop("target") == 0).astype(int)  # 1 = income above $50K
cats = ["workclass", "education", "marital-status", "occupation",
        "relationship", "race", "sex", "native-country"]
nums = [c for c in df.columns if c not in cats]
X_tr, X_te, y_tr, y_te = train_test_split(
    df, y, test_size=0.2, stratify=y, random_state=0)

ladder = {
    "guess the majority": DummyClassifier(strategy="most_frequent"),
    "logistic regression": make_pipeline(
        make_column_transformer(
            (OneHotEncoder(handle_unknown="ignore"), cats),
            (StandardScaler(), nums)),
        LogisticRegression(max_iter=2000)),
    "boosted trees, defaults": HistGradientBoostingClassifier(
        categorical_features=[c in cats for c in df.columns], random_state=0),
}

for name, model in ladder.items():
    model.fit(X_tr, y_tr)
    auc = roc_auc_score(y_te, model.predict_proba(X_te)[:, 1])
    print(f"{name:24} accuracy {model.score(X_te, y_te):.3f}   AUC {auc:.3f}")

Swap in your own data and you have the three numbers every later model has to be reported against. If you're working with an assistant, ask for the floor explicitly, because "build a model" won't reliably produce it. Something like: "Before any other model, fit a majority-class DummyClassifier and a plain logistic regression on the same split, and report every later model as an improvement over both." That's a one-sentence addition to the prompt, and it changes what the rest of the notebook can prove.

The list of impressive first moves keeps growing. After gradient boosting came tabular deep learning, and now pretrained tabular foundation models, which TabArena, a continuously maintained benchmark, reports "excel on smaller datasets." Each one is worth trying. Each one also makes it easier to skip the floor, because the first result already looks like a finished product.

Two lines in every model report

The fix is a reporting habit, not a technique. Every model summary, whether it's a notebook, a slide or a pull request description, should include two numbers next to the headline score: the improvement over the majority-class guess, and the improvement over the simplest reasonable model. If the first number is small, you don't have a model yet. If the second is small, you have a decision to make about whether the complexity is worth keeping, and now you have what you need to make it.

It's the same judgement the tuning lesson in SophiArch's Introduction to Machine Learning course is built around: it starts from untouched defaults and gives a whole section to deciding when to stop. The table above is that decision with real numbers attached.

The embarrassing model isn't the one you ship. It's the one that tells you whether the model you ship was worth building.

References

Olson, R. S., La Cava, W., Orzechowski, P., Urbanowicz, R. J., & Moore, J. H. (2017). PMLB: a large benchmark suite for machine learning evaluation and comparison. BioData Mining, 10, 36. Source of the six datasets; the data is at github.com/EpistasisLab/pmlb.
scikit-learn. DummyClassifier. The majority-class baseline used as rung one; quoted wording checked against the docstring in scikit-learn 1.9.1.
McElfresh, D., Khandagale, S., Valverde, J., Prasad C, V., Feuer, B., Hegde, C., Ramakrishnan, G., Goldblum, M., & White, C. (2023). When Do Neural Nets Outperform Boosted Trees on Tabular Data? NeurIPS 2023 Datasets and Benchmarks Track. 19 algorithms, 176 datasets; source of the light-tuning finding.
Ferrari Dacrema, M., Cremonesi, P., & Jannach, D. (2019). Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches. RecSys '19. 7 of 18 algorithms reproducible, 6 of those often beaten by simple heuristics.
Han, H., Ma, L., Wang, H., Li, B., Zha, D., et al. (2026). An Embarrassingly Simple Graph Heuristic Reveals Shortcut-Solvable Benchmarks for Sequential Recommendation. arXiv:2605.07125. Simple graph heuristic competitive on 10 of 14 datasets.
Zeng, A., Chen, M., Zhang, L., & Xu, Q. (2023). Are Transformers Effective for Time Series Forecasting? AAAI 2023. Source of the one-layer linear (LTSF-Linear) result.
Nie, Y., Nguyen, N. H., Sinthong, P., & Kalagnanam, J. (2023). A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. ICLR 2023. PatchTST, a Transformer design that beat the LTSF-Linear baselines.
Christodoulou, E., Ma, J., Collins, G. S., Steyerberg, E. W., Verbakel, J. Y., & Van Calster, B. (2019). A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. Journal of Clinical Epidemiology, 110, 12–22.
Grinsztajn, L., Oyallon, E., & Varoquaux, G. (2022). Why do tree-based models still outperform deep learning on tabular data? NeurIPS 2022 Datasets and Benchmarks Track. 45-dataset benchmark of trees against deep learning.
Erickson, N., Purucker, L., Tschalzev, A., Holzmüller, D., et al. (2025). TabArena: A Living Benchmark for Machine Learning on Tabular Data. arXiv:2506.16791. Continuously maintained tabular benchmark; source of the foundation-model finding.
SophiArch. Introduction to Machine Learning. Course covering supervised learning fundamentals, model evaluation metrics, and a hyperparameter tuning workflow that starts from defaults and includes explicit stopping criteria.

Your Agent Has Observability. It Doesn't Have Evals.

Jason Lau — Thu, 24 Sep 2026 05:43:11 +0000

TLDR: In LangChain's survey of 1,340 practitioners, 89% had implemented observability for their agents and 94% of teams with agents in production had it. Offline evaluations: 52.4%. Online evaluations: 37.3%. Fewer than a third of respondents run both. That gap is not a tooling gap — it's a category error. A trace records what the agent did. Nothing in a trace records whether it was right, because a confidently wrong tool-call chain emits exactly the same telemetry as a correct one: same token counts, same latencies, same finish_reason, same green spans end to end. The OpenTelemetry GenAI conventions make this literal — there is no attribute for correctness until you run an evaluator and write one in.

Two traces

Here is the shape of the problem, in the only form that makes it obvious. A support agent handles refund requests. Two runs, side by side, as your observability platform renders them.

// Run A
{ "trace_id": "a41f…", "gen_ai.operation.name": "agent_run",
  "gen_ai.request.model": "claude-sonnet-4-6", "duration_ms": 3180,
  "gen_ai.usage.input_tokens": 2847, "gen_ai.usage.output_tokens": 193,
  "gen_ai.response.finish_reasons": ["stop"],
  "spans": [
    { "name": "lookup_customer",  "status": "OK", "duration_ms": 112 },
    { "name": "list_orders",      "status": "OK", "duration_ms": 340 },
    { "name": "issue_refund",     "status": "OK", "duration_ms": 908 }
  ], "error_count": 0 }

// Run B
{ "trace_id": "b73c…", "gen_ai.operation.name": "agent_run",
  "gen_ai.request.model": "claude-sonnet-4-6", "duration_ms": 3204,
  "gen_ai.usage.input_tokens": 2851, "gen_ai.usage.output_tokens": 188,
  "gen_ai.response.finish_reasons": ["stop"],
  "spans": [
    { "name": "lookup_customer",  "status": "OK", "duration_ms": 118 },
    { "name": "list_orders",      "status": "OK", "duration_ms": 336 },
    { "name": "issue_refund",     "status": "OK", "duration_ms": 913 }
  ], "error_count": 0 }

Every field that your monitoring dashboard aggregates is, for practical purposes, identical. Both runs are three spans, all OK, no errors, ~3.2 seconds, ~2,850 input tokens, a clean stop finish. On a latency percentile chart they are the same point twice. On a token-spend chart they are the same point twice. On an error-rate chart they contribute equally to a rate of zero.

In Run B the customer had two orders — a $40 one they were complaining about and a $2,300 one they weren't. The agent called issue_refund on the second.

Nothing in that trace is wrong. issue_refund really was called. It really did return OK, because refunding $2,300 is an entirely successful refund. The span is green because the API call succeeded, and the API call succeeding is the only thing a span status has ever meant.

The gap has a number attached to it

This is not a hypothetical asymmetry, and it isn't rare. LangChain's State of Agent Engineering report, published 23 May 2026 and drawn from 1,340 responses collected between 18 November and 2 December 2025, puts numbers on both halves:

	Adoption
Agents in production	57% (67% at 10,000+ employees)
Observability implemented	89% overall, 94% among production teams
Full per-step tracing	62% overall, 71.5% among production teams
Offline evaluations on a test set	52.4%
Online evaluations in production	37.3% (44.8% among production teams)
Both offline and online	fewer than a third

Observability is close to universal; systematic evaluation is roughly a coin flip. And the same survey names quality — accuracy, relevance, consistency, tone — as the top barrier to getting agents into production, cited by about a third of respondents. Those two findings sit awkwardly together. Quality is the thing most likely to block a deployment, and it is the thing least likely to be measured. Roughly nine teams in ten can reconstruct what their agent did last Tuesday. About five in ten can tell you whether it was any good.

The instinct to instrument first is a reasonable one, and the tooling market rewarded it — tracing is a solved, buyable, drop-in problem. Evaluation is not buyable in the same way, because it requires you to state what correct means for your domain, and nobody can do that for you. Observability got adopted at the speed of a vendor integration. Evaluation adoption moves at the speed of the hardest product conversation on your team.

Why the dashboard structurally cannot tell you

It's worth being precise about why the traces above are indistinguishable, because the reason isn't an oversight anyone will patch.

The OpenTelemetry GenAI semantic conventions define the standard attribute registry for instrumenting LLM and agent workloads — the fields your platform is almost certainly collecting. The core ones:

gen_ai.request.model, gen_ai.provider.name — which model answered
gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, plus cache-read, cache-creation and reasoning token counts — how much it cost
gen_ai.response.finish_reasons — why it stopped
gen_ai.operation.name — chat, tool_call, agent_run
gen_ai.input.messages, gen_ai.output.messages — what went in and came out

Read that list looking for the field that goes red when the agent refunds the wrong order. It isn't there, and it can't be, because every one of those attributes is a property of the call rather than of the answer. Cost, latency, model version and stop reason are all knowable from inside the request/response cycle. Correctness is only knowable with reference to something outside it — a ground truth, a business rule, a resulting state you can inspect.

The conventions do include an evaluation namespace — gen_ai.evaluation.name, gen_ai.evaluation.score.value, gen_ai.evaluation.score.label, gen_ai.evaluation.explanation — and score.label accepts values like "correct". But that namespace is a slot, not a sensor. Those attributes are empty until something you wrote computes a judgement and populates them. The standard has a correctness-shaped hole in it and expects you to bring your own filling. Most teams ship the ninety per cent that auto-instruments and never fill the hole.

This is the practical consequence: your alerting is wired to signals that do not move when quality moves. Latency won't regress when the agent starts refunding the wrong order — it might even improve. Token counts won't move. Error rate stays at zero, because from the runtime's point of view nothing failed. You will find out from a customer.

Every field a trace carries is a property of the call. Correctness is a property of the outcome, and the outcome has to be checked against something the request doesn't contain.

What measuring the outcome actually looks like

The fix has a well-established precedent, and it predates the current agent wave: verify the resulting state, not the transcript.

τ-bench, the tool-agent-user benchmark from Sierra, is built on exactly this move. Its authors describe the method plainly: they "employ an efficient and faithful evaluation process that compares the database state at the end of a conversation with the annotated goal state." Not the wording of the reply. Not whether the right tool name appeared. The database, afterwards, against what the database should have contained.

Applied to Run B above, that check is unglamorous and about four lines long:

def verify_refund(conversation, db_before, db_after):
    expected = conversation.annotated_goal      # refund order #4471, $40.00
    actual   = diff(db_before, db_after)        # refunded order #4473, $2,300.00
    assert actual.refunded_order_ids == expected.refunded_order_ids
    assert actual.total_refunded_cents == expected.total_refunded_cents

That assertion fails on Run B and passes on Run A. No telemetry field distinguished them; a four-line state diff does it instantly. The same pattern generalises to whatever your agent actually touches — if it writes code, run the test suite; if it files tickets, assert on the ticket's fields; if it produces a document, check the document against the source.

τ-bench also contributes a metric worth stealing outright. Alongside the usual pass@k ("at least one of k attempts succeeded"), the paper proposes pass^k — all k attempts succeeded — "to evaluate the reliability of agent behavior over multiple trials." This is the honest metric for anything customer-facing, because your customers are not running your agent eight times and keeping the best result; they're getting one run each. The arithmetic is brutal and worth internalising: an agent that succeeds 90% of the time, if failures are roughly independent, is fully reliable across eight consecutive runs only 0.9⁸ ≈ 43% of the time.

The benchmark's own headline finding, from the abstract, is the part most teams have not absorbed: "even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail)." That was measured on a purpose-built benchmark with annotated goal states. Your production agent has neither, and the assumption that it is doing better is currently unfalsifiable — which is precisely the problem.

"We'll just add an LLM judge"

This is the standard next move, and it's the right direction, but it is not free and it is not a shortcut past defining correctness. A judge is another model whose agreement with your actual standard is an empirical question — one that has now been measured at scale, and the results argue for treating judges as instruments that need calibration rather than as oracles.

The largest systematic study of the approach to date — 21 judges from nine providers across MT-Bench, JudgeBench and RewardBench, 118 runs and roughly 541,000 individual judgments — opens by naming the methodological problem directly: "judge validation in practice relies on exact-match agreement, a metric that does not correct for chance and systematically overstates discriminative ability." Two findings in particular should change how you read a judge's score:

Chance correction matters enormously. The paper reports that "kappa deflation between exact match and Cohen's κ is universal (33–41 pp on MT-Bench)." A judge advertising 80% raw agreement with human labels may be delivering far less genuine discriminative power than that number implies, because a meaningful share of those agreements are what you'd get by guessing.
Consistency is not accuracy. The authors document a "consistency–bias paradox," finding that "high test–retest reliability (>0.95) coexists with severe position bias (>0.10)" in production judges. A judge that returns the same verdict every time looks trustworthy on a stability check and can still be reliably wrong in a direction that tracks the order you presented the options in.

None of this makes LLM-as-judge unusable — it remains the only practical option for open-ended generation where no reference string exists. It does mean a judge is a component that itself requires a labelled validation set, a chance-corrected agreement statistic, and a position-swap check before you let it gate a release. Where you can verify state instead, verify state: an assertion on a database row has no position bias.

Turning the traces you already have into the evals you don't

The useful thing about being in the 89% is that you are already sitting on the raw material. Traces are unlabelled evaluation data. The work is labelling and replaying them.

Pull fifty real traces from last month, weighted toward the unusual: long tool chains, retries, sessions a human took over, anything a customer followed up on. Not fifty happy paths — happy paths inflate every number you're about to compute.
For each one, write down what should have happened, in a form a program can check. An order id. A row count. A set of fields. An exit code. If you cannot express it as an assertion, that's the product conversation surfacing, and it is better to have it now than during an incident.
Replay them on every change — prompt edits, model version bumps, tool-schema changes, dependency upgrades — and gate the release on the result. This is the step that converts the set from a document into a regression gate. It is also, mechanically, unit testing; the only novel part is that the assertions run against end state rather than return values.
Report pass^k, not just the mean. Run each case several times and count only the cases that pass every time. The number will be worse than your average, and it is the one that corresponds to what a single user experiences.
Feed failures back. Every production failure becomes case fifty-one. The set gets stronger exactly where the system is weakest, which is the whole point of curating it by hand.

Steps 2 and 3 are where teams stall, and the reason is rarely technical — it's that "what should have happened" turns out to be genuinely contested between engineering, support and finance. Working through that with a versioned golden set and a regression gate on every change is a chunk of what SophiArch's AI Applications with LLMs course spends its validation module on, alongside the observability side that most teams have already built. The two lessons sit deliberately close together, because shipping one without the other is the failure this whole article is about.

The one-line version

Observability answers "what happened?" Evaluation answers "should it have?" The first question is solved, buyable and nearly universally adopted. The second is the one your customers are actually asking, and roughly half of teams running agents in production still have no systematic way to answer it.

If you have 89% of the stack, the remaining work isn't another dashboard. It's fifty labelled traces and an assertion that fails.

References

LangChain. (2026, May 23). State of Agent Engineering — survey of 1,340 practitioners, fieldwork 18 Nov–2 Dec 2025; source of the observability (89%), offline eval (52.4%), online eval (37.3%) and production (57%) figures. Accessed 2026-09-23.
Yao, S., Shinn, N., Razavi, P., & Narasimhan, K. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045 — source of the database-state evaluation method, the pass^k metric, and the pass^8 <25% retail result. Accessed 2026-09-23.
Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias. (2026). arXiv:2606.19544 — 21 judges, nine providers, ~541,000 judgments; source of the kappa-deflation and consistency–bias findings. Accessed 2026-09-23.
OpenTelemetry. Gen AI Attributes registry — the standard gen_ai.* span attributes, including the gen_ai.evaluation.* namespace referenced above. Accessed 2026-09-23.
SophiArch. AI Applications with LLMs — course covering the probabilistic contract, context window architecture, output validation layers, evaluation frameworks and golden sets, orchestration patterns, and LLM observability, including the regression-gate workflow described here.

Text-to-SQL Accuracy Isn't a Model Problem. It's a Schema Problem.

Jason Lau — Wed, 16 Sep 2026 08:10:15 +0000

TLDR: dbt Labs ran the same eleven questions four different ways and found that modelling the schema — with no semantic layer involved at all — moved text-to-SQL accuracy from 64.5% to 90.0%. Adding a semantic layer on top of that modelled schema moved it a further 8 points. Snowflake, using a semantic model across four BIRD databases, measured a 21-point average lift. MotherDuck pointed three frontier models at small, clean schemas with nothing but the DDL and reported 95% — though those same runs score 58–64% under BIRD's strict scoring. Read side by side, these look contradictory. They aren't. None of them is measuring the model. They're measuring how much translation work somebody already did to the schema, and how forgivingly the answers were graded.

Ask an AI assistant to write a query against your warehouse and it will, almost every time, produce something that runs. That's the trap. A query that executes and returns a number looks exactly like a query that answers the question — right up until someone downstream builds a forecast on it.

Two queries, one question, two answers

Here's the shape this takes on an ordinary warehouse, away from the benchmark leaderboards. A SaaS company wants last month's revenue. An AI assistant, asked two reasonable questions, produces two queries.

-- Query A: "what was our revenue last month?"
SELECT SUM(amount_due) AS revenue
FROM invoices
WHERE status = 'paid'
  AND billing_period = '2026-08';

-- Query B: "break down last month's revenue by product"
SELECT p.name AS product,
       SUM(i.amount_due) AS revenue
FROM invoices i
JOIN invoice_line_items ili ON ili.invoice_id = i.invoice_id
JOIN products p ON p.product_id = ili.product_id
WHERE i.status = 'paid'
  AND i.billing_period = '2026-08'
GROUP BY p.name;

-- Query A result:
revenue = 1,148,000

-- Query B result:
product            revenue
-----------------  ---------
Platform (base)    1,664,000
Extra seats          988,000
Usage overage        705,000
-----------------  ---------
total              3,357,000

Both queries ran without error. Both came from reasonable, good-faith prompts. But Query B sums i.amount_due — the invoice total — once for every line item attached to that invoice. A customer with a base plan, an extra seat and a usage overage contributes their entire invoice total three times, once under each product. "Platform (base)" doesn't show platform revenue; it shows the full value of every invoice that happened to contain a platform line.

The fix is one column: sum ili.line_amount, the line's own amount, not the invoice's. Nobody wrote a bug. The join changed what one row means, and SUM kept adding as if it hadn't. If Query B's breakdown reaches a board deck before anyone reconciles it against Query A, that takes an awkward meeting to walk back.

This is not a model failure. Point a frontier model at this schema and it will make the same mistake confidently and articulately, because nothing in the schema says that invoice_line_items has a different grain than invoices, or that amount_due is only additive at the invoice level. That knowledge lives in someone's head, or it lives in a semantic layer. If it lives in neither, which query you get is a coin flip.

The spread that looks like disagreement

Three of the most-cited text-to-SQL results around, read together, look like they're arguing. Read separately, each is internally consistent — which is the tell that they're measuring different things.

dbt Labs' April 2026 benchmark tested four configurations over an insurance dataset, eleven questions, twenty runs per model. The one people quote is text-to-SQL versus semantic layer on a modelled project: Claude Sonnet 4.6 at 90.0% against 98.2%, GPT-5.3-Codex at 84.1% against 100.0%. The more interesting comparison is the one underneath it. Against the original normalised tables, plain text-to-SQL managed 64.5% across all eleven questions; the same method against a modelled project hit 90.0%. That 25.5-point improvement came from modelling alone, with no semantic layer anywhere in the picture — and dbt built it by prompting an LLM to write "as few dbt models as possible," which produced just three. Their summary: "Adding even minimal modeling on top of raw tables improved results across the board."
Snowflake's semantic-model evaluation (March 2025, using Claude 3.5 Sonnet) is often cited as "57% to 78% on BIRD." It's really an average across four BIRD databases, and the spread inside it matters: debit_card_specializing went 52% → 83%, california_schools 63% → 80%, thrombosis_prediction 45% → 70%, toxicology 69% → 79%. Averaged, an "approximately 20% increase in accuracy" — 21 points — from adding a semantic model, not from a newer model.
MotherDuck ran the opposite experiment, pointing Claude Opus 4.5, GPT-5.2 and Gemini 3 Flash at 500 BIRD questions with, in their words, "No semantic layer. No query history. No special context. Just the schema." The reported result is 95%. That figure sits at the top of a four-tier evaluation ladder, and the bottom rung is a lot less flattering: 64.0% (train) and 58.2% (test) under strict BIRD execution matching, rising through correction of known benchmark errors and tolerance for formatting differences to 94.9%/94.4% once an LLM judges whether the answer is defensible. Their argument is two-part — that BIRD's strict scoring is itself misleading, because "to score above 62% under strict rules, you have to start reproducing the benchmark's mistakes," and that "Good data modeling is the semantic layer."

So there are two variables moving here, not one, and honest reading requires separating them. Scoring strictness explains most of the distance between MotherDuck's 95% and everyone else's numbers; you cannot line that figure up against dbt's or Snowflake's and treat the gap as real capability. Schema modelling explains the rest, and it's the variable you control. dbt moved text-to-SQL a long way with three models and no semantic layer. Snowflake moved it 21 points with a semantic model over databases they didn't restructure. MotherDuck got high marks with no layer at all — over BIRD databases that, as they note, average seven tables. None of these is a claim about which model writes better SQL.

There's a ceiling worth keeping in view too. BIRD's own paper reports that its human baseline — data engineers and database students — reached "the human result of 92.96%" execution accuracy. People who write SQL for a living, working on databases of seven tables, get roughly one question in fourteen wrong. The bar was never "flawless." It's "at least as reliable as the analyst who used to own this."

What "the schema needs modelling" actually means

"Modelling" sounds like an abstraction exercise. In practice, on a schema like the one above, it's answering three concrete questions before anything touches a model:

What is the grain of each table, in one sentence? invoices is one row per invoice. invoice_line_items is one row per line on an invoice. Join them and your result set's grain becomes "one row per line item" — so any SUM of an invoice-level column is now double-, triple- or n-counting, depending on how many lines the average invoice carries. That single fact is what would have caught Query B.

Which columns carry business meaning that isn't in their name? status = 'paid' looks unambiguous. Does it need to exclude refunds? Is a partially refunded invoice still "paid" for revenue purposes? These are policy decisions someone made once, verbally, in a meeting nobody minuted — and no assistant can recover them from the schema alone.

Is there more than one path to the same number? Revenue might come from invoices.amount_due, from summing invoice_line_items.line_amount, or from an mrr_snapshots table a finance job populates nightly. Three legitimate paths, three numbers that won't quite agree, and none of them "wrong" — they answer slightly different questions that all get asked as "what's our revenue."

"Just model your warehouse" is easy advice to give

The obvious objection to all of this: dbt's benchmark fixed its schema with three new models over an eleven-question insurance dataset. You have four hundred tables, three teams who each define revenue differently, and a migration budget of zero. "Model the schema properly" is not a thing you can do this quarter, and anyone who has tried knows that the modelling is the easy half — getting three departments to agree which definition wins is the hard one.

That constraint is exactly why semantic layers exist. They're the retrofit path: a place to encode grain, join paths and metric definitions without restructuring the warehouse underneath. Snowflake's 21-point average came this way, over databases nobody rebuilt. Treating "clean DDL" and "semantic layer" as interchangeable is true for benchmarks and false for anyone with legacy tables and a roadmap.

There's also a second argument for the retrofit that the accuracy columns actively hide. In dbt's unmodelled configuration, the semantic layer scored 0.0% on the subset of questions requiring too many joins to resolve — not because it answered them wrongly, but because it declined to answer at all. As dbt puts it: "the Semantic Layer tells you it can't answer. It never returns invalid data. Text-to-SQL will cheerfully give you a wrong number." Plain text-to-SQL scored 70–100% on that same subset, which looks like a win until you remember that nothing in those results distinguishes a correct answer from a confident one. A coverage gap you can see beats a coverage gap that quietly returns 3,357,000. That asymmetry appears in no accuracy column anywhere, and for a number headed to a board deck it may matter more than the points do.

What both routes share is scope. Neither dbt's three models nor a semantic layer requires modelling the whole warehouse — only the slice that answers the questions people actually ask. That's the practical move: take the handful of questions your team asks weekly, model just the tables those touch, and leave the other three hundred and eighty alone until someone asks them something.

The benchmarks didn't find a model problem. They found a translation-work problem, and measured what happens when someone does the translation once instead of leaving it to a fresh guess on every query.

The checklist that transfers to your warehouse

None of the studies above tells you which side of the gap your own schema sits on. Run this instead:

Take your five most-asked business questions and write down, for each table involved, its grain in one sentence. If you can't do that quickly, an assistant generating SQL against those tables can't either.
Find every column whose name promises more precision than its values deliver — status, type, category especially. If status needs a footnote to interpret correctly, that footnote has to live somewhere the assistant can read: a view, a semantic layer, a column comment. Not a Slack thread from 2024.
Ask the same business question two ways and see whether you get two numbers. That's what happened above. If it happens on your warehouse, you have a modelling gap rather than an AI-competence gap, and no model upgrade closes it.

The numbers make the same point from three directions: 90% accuracy sounds like a solved problem until you notice the missing 10% is where double-counted joins and silently redefined metrics live, invisible precisely because the query still runs. Building the judgment to catch that — reading a join and knowing what it did to the grain, reconciling a number against a second path before trusting it — is what SophiArch's SQL for Data Analysis course is built around, with a dedicated lesson on joins and double-counting early on and a closing module on auditing SQL that an AI assistant wrote.

References

Ganz, J., & Perigaud, B. (2026, April 7). Semantic Layer vs. Text-to-SQL: 2026 Benchmark Update. dbt Developer Blog. Accessed 2026-09-16.
Zayats, T. (2025, March 31). Agentic Semantic Model Improvement: Elevating Text-to-SQL Performance. Snowflake Engineering Blog. Accessed 2026-09-16.
MotherDuck. Your Data Model Is the Semantic Layer. MotherDuck Blog (undated). Accessed 2026-09-16.
Li, J., Hui, B., Qu, G., et al. (2023). Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs. arXiv:2305.03111 — source of the 92.96% human-performance baseline on BIRD-SQL. Accessed 2026-09-16.
SophiArch. SQL for Data Analysis — course covering filtering, aggregation, joins, CTEs, window functions and cohort analysis, closing with a module on auditing AI-generated SQL for the silent failure modes described here: double-counting joins, NULL traps and misplaced filters.

Long Context Didn't Kill RAG. It Killed Lazy RAG

Jason Lau — Wed, 09 Sep 2026 06:41:30 +0000

Short version: In a controlled comparison published in June 2026, long-context prompting beat semantic RAG on correctness — 73.1% against 65.4% — and cost 26 times as much per query to do it. That single row is the whole argument: long context is genuinely better at the thing people claim it's better at, and the reasons to retrieve anyway are cost, corpus size, freshness, and permissions, not accuracy. What long context actually made obsolete is the naive pipeline — fixed chunks, one embedding model, top-k by cosine, no re-ranking — which was never competitive with a big window and is barely competitive with a well-built retriever either.

Somebody says it in a planning meeting roughly once a quarter now: models take a million tokens, so why are we still maintaining a vector database? It is a fair question, asked in bad faith about half the time, and the usual response is a defensive listicle titled "RAG isn't dead." Those posts are mostly right and almost never useful, because they argue the conclusion instead of giving you a way to reach it for your own corpus.

So let's do it the other way round. Concede the strongest version of the opposing case first, then work out what's left.

The strongest case for "just put it in the window"

The most useful comparison I've found isn't a vendor benchmark, it's a preprint that ran both architectures over the same document-grounded task and reported the trade-off as a frontier rather than a winner. Hamilton et al.'s "The Token Tax of Epistemic Accuracy" (June 2026) found that "Long-context prompting achieved the highest correctness (73.1% vs. 65.4% for semantic RAG), but at 26 times the per-query token cost."

Read that without flinching. Long context won on correctness by nearly eight points. Every architectural argument that starts "but retrieval is more accurate" is arguing against a measurement, and if you have been repeating that line, it needs retiring. Stuffing the documents in works, and it works better than a mid-quality retriever, for the obvious reason: a retriever that never surfaces the right passage has capped the system's accuracy at zero for that query, whereas a big window at least has the passage in the room.

The honest framing is that retrieval buys you everything except peak accuracy on a corpus that fits.

What it doesn't buy you, and where the "26x" goes

Start with the arithmetic, because it's the part people wave at rather than compute. Take a 200,000-token prompt — a modest corpus by 2026 standards — at an illustrative $3 per million input tokens. That's $0.60 per query before the model has emitted a single output token. At 50,000 queries a month, $30,000. Answer the same question from six retrieved chunks, call it 4,000 tokens, and you're at $0.012 a query, or $600 a month. Same model, same questions, fiftyfold difference, and none of it shows up in the demo because the demo ran forty queries.

There's a real counter-argument here that RAG advocates usually miss: prompt caching. If the 200,000-token prefix is identical across queries, most providers will serve the repeated portion at a steep discount, and the gap narrows dramatically. But notice the condition. Caching pays off when the corpus is small enough to fit, stable enough not to churn, and shared across users — which is precisely the regime where nobody needed retrieval in the first place. The moment the prefix differs per user, per tenant, or per hour, the discount evaporates and you are back to paying full freight on every token you didn't need.

Then there's the accuracy claim itself, which is conditional in a way the headline number hides. Three independent lines of evidence say a long window is not a uniform one:

Chroma's Context Rot study (Hong, Troynikov and Huber, July 2025) evaluated 18 models and found that "model performance varies significantly as input length changes, even on simple tasks," with degradation worsening when distractors are present and when the question shares little wording with the passage that answers it.
NoLiMa (Modarressi et al., ICML 2025) tested 13 models that all claim 128K-plus context and reported that "at 32K, for instance, 11 models drop below 50% of their strong short-length baselines" — with GPT-4o falling from a 99.3% baseline to 69.7%.
Leng et al. found across 20 models that "while retrieving more documents can improve performance, only a handful of the most recent state of the art LLMs can maintain consistent accuracy at long context above 64k tokens."

Put those next to the Token Tax result and the picture resolves: long context wins when the relevant material fits comfortably and the query wears its keywords on its sleeve. It degrades exactly where enterprise questions live — paraphrased queries, near-duplicate distractors, corpora far past the window.

Three questions that settle it for your corpus

The reason the general debate never converges is that it's the wrong altitude. Run your own numbers instead. In order:

1. Does the material relevant to a single query fit in the window with room to spare — every time? Not the average query, the worst one. If your corpus is a 40-page product spec, retrieval is architecture you don't need. If it's 80,000 support tickets and the answer depends on which three of them are similar, no window is large enough and the question is settled.

2. What does it cost at your query volume, after caching? Do the multiplication above with your real rate and your real monthly volume. Then ask whether the prefix is genuinely shared across queries or varies per user. Shared and stable, caching rescues long context; per-user or fast-changing, it doesn't.

3. Does the corpus change, and does who's asking change what they may see? This is the axis that decides more real systems than the other two combined, and it barely appears in the online argument. A window has no concept of freshness — you rebuild the prompt or you serve stale content. It has no concept of permission either: everything in the prompt is visible to whoever's asking. An index has both. Documents are re-embedded when they change; filters are applied per user at query time. If your answer to either half is yes, you need a retrieval layer regardless of what the matrix below says about accuracy alone.

Accuracy alone points at long context. Freshness and permissions move three of the four quadrants to retrieval.

"Better retrieval" isn't a vibe, it's a short and specific list

If questions 1–3 land you on retrieval, the Token Tax result becomes a warning rather than a footnote: a mediocre retriever loses to a big window by eight points of correctness. So the pipeline that "everyone knows" — split at 512 tokens, embed, take the top five by cosine, done — isn't a starting point to defend. It's the thing long context genuinely made obsolete.

What replaces it is four changes with measurable effects, each of which gets a lesson in the Retrieval & RAG Systems course I teach, because they are the dials that actually move recall:

Hybrid retrieval. Combine lexical (BM25) and dense scoring, fused by reciprocal rank. A 2026 benchmark of ten retrieval strategies over 23,088 financial queries found the best configuration was a two-stage hybrid-plus-reranking pipeline — and, more pointedly, that "BM25 outperforms state-of-the-art dense retrieval on financial documents." Exact identifiers, product codes and figures are where embeddings are weakest and keyword matching is strongest.
Cross-encoder re-ranking. Retrieve broadly, then score query-document pairs jointly. Rosa et al. found cross-encoders beat comparable bi-encoders by over four points on BEIR, with the gap widening on domains the model wasn't trained on — and, awkwardly for the default stack, that bi-encoder first-stage retrieval "provides no gains in comparison to a simpler retriever such as BM25 on out-of-domain tasks."
Chunking as a retrieval decision. Chunk size is not a formatting preference; it decides what a single embedding is asked to represent. Splitting mid-argument is how you get a passage that scores well and answers nothing.
Measurement before cleverness. The same benchmark (Akarsu et al.) found that "query expansion methods (HyDE, multi-query) and adaptive retrieval provide limited benefit for precise numerical queries." Sophistication isn't automatically an improvement. Recall@k on your own labelled queries is what tells you which of these earned its latency.

Two retrieval signals fused by rank, then jointly scored against the query — not a single embedding search returning whatever ranks fifth by cosine.

That last one is the actual discipline. Every technique above is a hypothesis about your corpus, and the corpus gets a vote.

So: not "is RAG dead." Does the material fit, what does it cost at volume, and does it change or vary by reader. Three questions, answerable in an afternoon with your own numbers, and they will land you somewhere more defensible than either side of the argument you were being asked to join.

References

Hamilton, A., Singh, R., Wise, M., Yousif, I., Carvalho, A., Shan, Z., Mayyas, M., Cavuoto, L. A., & Megahed, F. M. (2026). The Token Tax of Epistemic Accuracy: Comparing RAG and Long-Context Architectures for Document-Grounded Generative AI Applications. arXiv:2606.20898.
Hong, K., Troynikov, A., & Huber, J. (2025). Context Rot: How Increasing Input Tokens Impacts LLM Performance. Chroma Research.
Modarressi, A., Deilamsalehy, H., Dernoncourt, F., Bui, T., Rossi, R. A., Yoon, S., & Schütze, H. (2025). NoLiMa: Long-Context Evaluation Beyond Literal Matching. ICML 2025.
Leng, Q., Portes, J., Havens, S., Zaharia, M., & Carbin, M. (2024). Long Context RAG Performance of Large Language Models. arXiv:2411.03538.
Akarsu, M., Karaman, R. K., & Mierbach, C. (2026). From BM25 to Corrective RAG: Benchmarking Retrieval Strategies for Text-and-Table Documents. arXiv:2604.01733.
Rosa, G., Bonifacio, L., Jeronymo, V., Abonizio, H., Fadaee, M., Lotufo, R., & Nogueira, R. (2022). In Defense of Cross-Encoders for Zero-Shot Retrieval. arXiv:2212.06121.
SophiArch. Retrieval & RAG Systems — course covering chunking strategy, hybrid BM25-plus-dense retrieval with reciprocal rank fusion, cross-encoder re-ranking, retrieval metrics, index freshness and permission-aware retrieval.

Prompt Engineering Won't Survive — the Skill Underneath It Will

Jason Lau — Wed, 02 Sep 2026 09:49:37 +0000

For about two years, "prompt engineer" was the job title recruiters attached to six-figure listings and career-changers scrambled to add to a resume. Then the interest mostly stopped converting into jobs. The Wall Street Journal reported that Indeed searches for the role spiked to 144 per million U.S. searches in April 2023 and have since flatlined at 20-30 per million — and, per Indeed VP of AI Hannah Calhoon, that interest was never matched by actual employer job postings in the first place: "Maybe they talked about the value of prompt engineers, but they weren't then actually hiring for that." A title that rose and mostly stalled out inside two model generations. Microsoft's Jared Spataro put the reason bluntly: "You don't have to have the perfect prompt."

That's a fair obituary for the "prompt engineer" job title. It's a bad obituary for the underlying work, and the mistake is treating both as the same thing.

What actually died was a set of tricks tuned to specific models

Open any 2023-era prompt-engineering article and it's mostly mantras: "think step by step," phrase it as a role assignment, wrap the instruction in triple quotes, promise the model a tip. These worked because they guided a specific model around a specific weakness — and weaknesses don't persist across model generations.

Chain-of-thought is the clearest case, because it's the most-cited prompting technique there is and it's now been re-measured against current models. A 2025 technical report from Wharton's Generative AI Labs ran chain-of-thought prompting against eight models spanning both non-reasoning and reasoning types. The results don't point one direction. Gemini 2.0 Flash gained 13.5% and Sonnet 3.5 gained 11.7% — real improvements. GPT-4o-mini gained 4.4%, a difference the authors couldn't call statistically significant. Gemini Pro 1.5's rate of getting every part of a problem fully correct dropped 17.2 percentage points with chain-of-thought turned on. Among the reasoning models — the ones already trained to produce their own extended reasoning — o3-mini and o4-mini picked up a marginal 2.9-3.1%, while Gemini Flash 2.5 lost 3.3%. All of this arrived with a real tax: 35-600% longer response times for non-reasoning models, 20-80% longer for reasoning ones. The paper's own framing is careful rather than dramatic — "its effectiveness depends significantly on model type and specific use case" — but read against the trend, that's a technique whose payoff is shrinking and inverting as the models it was compensating for get better at reasoning on their own.

A 2025 arXiv paper on what it calls the "Prompting Inversion" makes the same point sharper by comparing one technique across two model generations directly. The authors built "Sculpting," a heavily constrained prompting method, and pitted it against plain chain-of-thought. On GPT-4o, Sculpting won clearly — 97% accuracy against chain-of-thought's 93%. On GPT-5, the ranking flips: Sculpting drops to 94.00% while plain chain-of-thought reaches 96.36%. Their explanation is that the same constraints acting as guardrails for a mid-tier model become "handcuffs" on a more capable one — the scaffolding that stopped GPT-4o from wandering off just gets in GPT-5's way. A technique doesn't get more true as models improve; it can flip sign entirely.

The instability was already visible in how fragile the tricks were

None of this should be surprising if you look at how sensitive prompt outputs already were to changes that had nothing to do with meaning. Sclar, Choi, Tsvetkov, and Suhr measured how much purely cosmetic prompt formatting — whitespace, separator characters, capitalization — could move accuracy on the same task with the same content, and found swings of up to 76 accuracy points on LLaMA-2-13B. Critically, they found this sensitivity "remains even when increasing model size, the number of few-shot examples, or performing instruction tuning" — it isn't a small-model problem that scale quietly fixes.

That finding reframes what a lot of "prompt engineering" tips actually were. If reformatting a prompt with no change in meaning can swing accuracy by 76 points, then a technique's measured 10-point gain on some benchmark was never a stable property of the technique — it was one sample from a distribution of formatting accidents that happened to land favorably on that model, at that snapshot in time. Chasing that kind of gain by memorizing the exact phrasing that worked last week is chasing noise, and noise doesn't transfer to the next model release. You have to separate the parts of a prompt that are doing real work (the schema, the goal, the constraints) from the parts that are stylistic residue — which is exactly the split SophiArch's AI-Assisted Data Science course spends its second lesson on, precisely because the residue is what breaks first when your model evolves.

What was actually being taught the whole time

Strip out the mantras and what's left in a good prompt is closer to a specification than a sacred phrase. A 2024 peer-reviewed framework in Frontiers in Education defines prompt engineering as "the skill of communicating the problem, its context, and the constraints imposed on the desirable solution to an LLM to solve it correctly as fast as possible" — and its authors explicitly push back on the idea that this reduces to generic communication skill, arguing it's closer to task decomposition: knowing what you actually want closely enough to state it, before you ever open a chat window.

That's the part that doesn't expire when a new model ships, because it isn't a fact about models — it's a fact about the gap between what's in your head and what's on the page. Consider the difference between two requests to deduplicate a customer table:

"Clean up the duplicate customers in this table."

versus:

"Two rows are the same customer if email matches exactly (case-insensitive) OR both phone and last_name match. When rows conflict, keep the one with the most recent last_order_date; if that's tied, keep the one with more non-null fields. Do not merge rows that share only a last_name — that's a coincidence, not a duplicate. Return a mapping from removed customer_id to the customer_id it was merged into, so the removal is auditable."

Neither of these is a prompting trick. No chain-of-thought cue, no role-play framing, no promised tip. The second one just doesn't leave the model — or a junior analyst, for that matter — a single ambiguous decision to make on its own: what counts as a match, which record wins a conflict, what NOT to merge, and how the result gets checked. A model that's gotten better at reasoning will make smarter guesses at filling those gaps than last year's model did. It will still be guessing, and on a dataset that decides refund eligibility or fraud flags, "the model guessed well" is not the standard anyone should be signing off on.

That's the actual content of "prompt engineering" once you take the model-specific mantras out of it, and it's also why the skill was never new — it's requirements-writing and problem specification, wearing a 2023 label because a chat interface made the gap between vague and precise suddenly visible to everyone, all day, on every request. The interface is what's new. The gap it exposed is not, and closing it is a durable skill for exactly the reason the mantras weren't: it doesn't depend on which model happens to be running underneath.

References

Meincke, L., Mollick, E. R., Mollick, L., & Shapiro, D. (2025). Prompting Science Report 2: The Decreasing Value of Chain of Thought in Prompting. SSRN / Wharton Generative AI Labs.
You Don't Need Prompt Engineering Anymore: The Prompting Inversion (2025). arXiv.
Sclar, M., Choi, Y., Tsvetkov, Y., & Suhr, A. (2023). Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design. arXiv.
Bousquette, I. (2025). The Hottest AI Job of 2023 Is Already Obsolete. The Wall Street Journal.
Prompt Engineering as a New 21st Century Skill (2024). Frontiers in Education.
SophiArch. AI-Assisted Data Science — second lesson module on separating durable prompt structure (schema, goal, constraints) from model-specific tricks.

Your RAG Pipeline Doesn't Have an Accuracy Problem - It Has an Evaluation Problem

Jason Lau — Wed, 26 Aug 2026 03:37:34 +0000

A team builds a retrieval-augmented chatbot over the company's internal policy documents. In the demo, someone asks "how many days of parental leave do we get?" and the bot answers correctly, citing the right PDF. Someone asks about expense limits - correct again. Ten questions, ten good answers, applause, ship it.

Three months later an employee asks whether their contractor status qualifies for the health stipend, gets a confident "yes" assembled from a policy that was superseded last year, and files the claim. Nobody on the team can say when the pipeline started producing answers like that, because nothing was ever measuring whether it did.

That is the real state of most RAG systems in production. The pipeline is not unusually inaccurate - every retrieval system misses sometimes. What's missing is the apparatus that would notice. The demo was treated as the evaluation, and a demo is the one test a RAG system essentially cannot fail: the questions were chosen by the people who built the index, phrased the way the documents phrase things, asked about content everyone knew was in there.

A wrong answer and a right answer look identical

Traditional software fails loudly. A broken API call throws an exception; a bad deploy serves 500 errors; a failing test case turns red. A RAG regression does none of this. Swap the embedding model, change the chunk size, re-index after a document update - the system keeps returning fluent, well-formatted, confidently cited answers. Whether they are grounded answers is invisible in every signal you get: no exception, no latency spike, no schema violation.

This isn't an edge case of RAG engineering; it is a recurring finding in the engineering literature on it. A CAIN 2024 experience report across three RAG case studies (research, education, and biomedical domains) catalogued seven recurring failure points - missing content, missed top-ranked documents, answers retrieved but lost in consolidation, answers present in context but not extracted, wrong format, wrong specificity, incomplete answers - and its two headline takeaways are blunt: "validation of a RAG system is only feasible during operation, and the robustness of a RAG system evolves rather than designed in at the start."

Read that first clause again. You cannot fully validate this class of system before shipping it. Which means the eval harness is not a nice-to-have you add after launch - it is validation-during-operation, systematised: built from the questions real users actually asked, and run continuously as the system and its corpus change. It is the only mechanism by which you ever learn whether the thing works.

"RAG fixes hallucinations" is marketing, and there's a measurement to prove it

The reason teams skip evaluation is an ingrained assumption: retrieval grounds the model, so the hallucination problem is handled. The strongest counterexample comes from a domain with real money behind getting this right. Legal research vendors marketed their RAG products as "eliminating" or "avoid[ing]" hallucinations, even guaranteeing "hallucination-free" citations. When Stanford's RegLab ran the first preregistered empirical evaluation of these tools, the flagship products from LexisNexis and Thomson Reuters "each hallucinate between 17% and 33% of the time."

These are professional-grade systems built by teams with enormous resources over curated, authoritative corpora - the best case for RAG. Retrieval reduced hallucination relative to a bare model, but between one in six and one in three answers still contained fabrication. The gap between "we added retrieval" and "we measured what retrieval actually delivers on our queries" is exactly the gap the vendors' marketing fell into. If they can fall into it, your internal chatbot certainly can.

"Is it accurate?" is two questions wearing one trenchcoat

Suppose you accept the premise and ask: fine, how accurate is my pipeline? The question is underspecified, and that underspecification is why demo-vibes evaluation persists - there was never an agreed definition of correct to check against.

A RAG answer can fail in two independent places, and they need different fixes:

Retrieval failed. The chunks handed to the model didn't contain the answer - it wasn't indexed, didn't rank into the top-k, or got cut during context assembly. No prompt engineering will fix this; you need to change chunking, embeddings, or ranking.
Generation failed. The answer was in the retrieved context and the model ignored it, contradicted it, or embellished beyond it. Re-ranking harder won't fix this; you need to change the prompt contract, the model, or add output validation.

An aggregate "accuracy" number collapses these together and leaves you optimising blind. This decomposition is precisely what the RAGAS evaluation framework formalised into separately measurable quantities: context relevance scores the retrieval step (did the right evidence show up, without drowning in irrelevant material?), while faithfulness and answer relevance score the generation step (is every claim in the answer supported by that evidence, and does it address the question?). The Ragas library has since split the retrieval side further into context precision and context recall. The individual metric implementations have known rough edges - but the decomposition is the part that matters, because it turns "the bot was wrong" into a bug report that identifies a component.

What an actual evaluation harness looks like

None of this requires a research team. A minimum viable harness is three artefacts and a habit.

1. A golden set: your domain's definition of correct, written down. Collect 50-100 real questions - from support logs, pilot users, and the subject-matter expert who knows where the knowledge is buried. For each, record the expected answer and which document(s) it must come from:

{
  "id": "policy_031",
  "question": "Does the health stipend apply to contractors?",
  "expected_answer": "No - eligibility requires full-time employment status.",
  "must_cite": ["benefits-eligibility-2026.pdf"],
  "trap": "superseded 2024 policy still in corpus says yes"
}

That trap field is where the value lives. Easy questions inflate your score; the golden set earns its keep on superseded documents, questions whose answer spans two chunks, questions the corpus cannot answer (the correct behaviour is "I don't know" - the first failure point in the CAIN taxonomy is systems that fabricate rather than decline), and negation cases where the retrieved text says the opposite of what the surface phrasing suggests.

2. Scoring that preserves the retrieval/generation split. Score "did the must-cite document appear in the retrieved set?" separately from "is the answer faithful to what was retrieved?" For the generation side at scale you'll likely use an LLM judge - which is workable but not free of pathology: the MT-Bench study of LLM-as-a-judge found strong judges agree with human raters over 80% of the time, while also documenting systematic verbosity bias (longer answers score better regardless of quality) and signs of self-enhancement bias (judges may favour their own model's outputs). Use a judge, but spot-check it against human labels before trusting it as your regression signal.

3. A regression gate. The harness runs on every change - new embedding model, new chunking strategy, re-index, prompt edit, model version bump - and a score drop blocks the change, the same way a failing test suite blocks a merge. This is the step that converts evaluation from a one-time report into an engineering control. Without the gate, your golden set is a benchmark you ran once in a notebook; with it, "did we just get worse?" has an answer before users provide it.

The demo answered the questions you chose. The golden set answers the ones your users will actually ask - including the ones designed to make your pipeline lie. Build the second thing before you trust the first.

References

Barnett, S., Kurniawan, S., Thudumu, S., Brannelly, Z., & Abdelrazek, M. (2024). Seven Failure Points When Engineering a Retrieval Augmented Generation System. CAIN '24.
Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. (2023). RAGAS: Automated Evaluation of Retrieval Augmented Generation. arXiv.
Magesh, V. et al. Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. Stanford RegLab.
Zheng, L. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS.
Ragas documentation. List of available metrics.

If you want to go deeper than a blog post can, evaluation is where SophiArch's Building AI Applications with LLMs course plants its flag: a full module on golden sets, LLM-as-judge trade-offs, and regression gates, sitting inside a validation architecture that runs from prompt contracts through output checking to production observability.

Clustering Isn't Discovery - It's a Hypothesis You Still Have to Test

Jason Lau — Wed, 19 Aug 2026 09:56:42 +0000

An analyst runs KMeans(n_clusters=4) on dataset of customer purchase history, it gets back four groups, and he/she writes up the report as "from the data, it revealed four customer segments!" Management simply loves this - after long periods struggling to grasp at their customer base, they now have data-backed insights on real-names associated with each segment (Loyal Regulars, Bargain Hunters, At-Risk, New & Exploring), and pretty soon all the strategy meetings will silently assume those four groups are real and nobody even dares to question.

Unfortunately, those groups are not real and clustering does not "discover" hidden nuggets of insights in the data. Instead clustering imposes a partitioning based on what the analyst chooses as a similarity rule. If the analyst had chosen another reasonable rule, clustering can also fit the same dataset in a different way. Let's be clear - the four groups are a hypothesis about how to group the customers. To determine if the grouping is accurate, as in we can use this grouping to predict customer behavior, now that is a separate question that the clustering method can never answer by itself.

Same data, different "discovery"

To illustrate, try running KMeans clustering on an RFM table (recency, frequency, monetary value per customer) and only change the random seed:

from sklearn.cluster import KMeans

km_a = KMeans(n_clusters=4, n_init=1, random_state=7).fit(X_scaled)
km_b = KMeans(n_clusters=4, n_init=1, random_state=42).fit(X_scaled)

print(km_a.inertia_, km_b.inertia_)
# 8421.3   8103.9 - different local optima, neither one "wrong"

inertia: measure of how tightly grouped the data points are within the clusters

Notice that you get different inertia values. KMeans does not find the globally best grouping - it cannot. Instead the algorithm finds a local optimum based on randomly chosen initial centroids and two runs on the same data can converge to genuinely different groupings. In fact scikit-learn's user guide states "Given enough time, K-means will always converge, however this may be to a local minimum. This is highly dependent on the initialization of the centroids". As mitigation, the default behavior is to run the algorithm several times with different centroid seeds (n_init) and retain the run that scored the lowest on inertia - it does not guarantee a single answer.

What if you change to use a different method - say hierarchical clustering with average linkage on the same RFM dataset? In hierarchical clustering, you do not need to specify the number of clusters and hence, the group counts that appear "natural" in your dataset also shifts. This is because hierarchical clustering with average linkage and KMeans are optimizing on different objectives ( nested merge vs compact spherical clusters )

This dissonance between run-to-run, method-to-method is well studied under clustering stability - a collection of research whose overview by von Luxburg (2010) shows how sensitive clustering results are to the seed, sample, algorithm and why stability by itself is not sufficient for meaningful clustering.

Aren't there any methods to decide?

One might say: "We are very data-driven and we use the elbow method to pick k."

There are various metrics or methods to measure how well your clustering algorithm groups the dataset:

elbow (inertia vs k)
average silhouette method (silhouette score vs k)
gap statistic

Side-by-side comparisons of various methods on same dataset often leads to contradictory recommendations on cluster count. There is no one-metric and final decision requires human judgement based on your domain knowledge.

At times, you might have another situation:

from sklearn.metrics import silhouette_score

for k in range(2, 7):
    labels = KMeans(n_clusters=k, n_init=10, random_state=0).fit_predict(X_scaled)
    print(k, silhouette_score(X_scaled, labels))

# 2  0.41   <- highest score
# 3  0.33
# 4  0.29   <- the k the business narrative was built around
# 5  0.24
# 6  0.22

Here, the silhouette score shows k=2 is highest but you get push back from Marketing - "We need 4 separate personas to target!". Nothing wrong with that request, 2 clusters will usually lead to "Loyal" vs "Non-Loyal" segments and it is meaningless for marketing planning. 4 clusters do help in the business narrative to target specific market segments. If this is the case, a disclaimer is in order that k=4 was due to business narrative (instead of "insights from dataset" as authority)

Cluster on demographics may not mean different customer behavior

Your clustering tells you have four groups of customers and they look different based on the features you cluster on - recency, frequency, spend. Note that their behaviors that businesses care about - who leaves for a competitor, who upgrades to higher tier, who responds to coupons - may look different across those same groups. This discrepancy is what marketing teams struggle with using demographic and behavioral personas - the behavioral patterns often cut across various demographic personas.

The fix is not to have a better clustering metric but to validate externally based on your business objectives and goals (churn in the next quarter, response rate to campaign, etc) and check if they differ between clusters more than within the clusters.

What is the correct approach to clustering?

The above does not mean clustering is a useless technique to avoid. Instead what it means is that clustering is only a starting hypothesis for further investigations.

Before presenting the clustering results:

re-run with different seeds and subset of data
- this is to verify the stability of the cluster
check with multiple metrics and methods (elbow, silhouette, gap statistics)
- highlight in report when they disagree
validate externally
- select a business outcome
- confirm it varies across clusters more than within cluster
treat the chosen number of clusters as modeling decision

A clustering that complies with all four does not mean it is "true" but it does earn the right to be acted upon.

References

Kleinberg, J. (2002). An Impossibility Theorem for Clustering. NeurIPS.
von Luxburg, U. (2010). Clustering Stability: An Overview. arXiv.
scikit-learn. Clustering - K-means, User Guide - local-minimum convergence and multiple-initialization behavior.
scikit-learn. sklearn.cluster.KMeans API reference - n_init and random_state parameter definitions.
Datanovia. Optimal Number of Clusters: Elbow, Silhouette, Gap Statistic.
Ahmed, M. et al. (2022). K-Means Clustering Approach for Intelligent Customer Segmentation Using Customer Purchase Behavior Data. Sustainability, 14(12).
MarTech. Why behavioral segmentation beats personas for real personalization.

SophiArch's Unsupervised Learning & Clustering course covers k-means, hierarchical clustering, and validation together for this reason - an algorithm that returns groups is the easy half; judging whether those groups mean anything to the business is the half that determines whether the analysis survives contact with a skeptical stakeholder.

You Don't Need a Math PhD for Data Science — You Need to Stop Skipping the Boring Step

Jason Lau — Wed, 12 Aug 2026 03:08:07 +0000

Ask someone switching into data science what worries them and it's almost always the same thing: do I know enough math? So they spend weeks on gradient descent derivations and eigenvalues before they've ever wrangled a messy CSV. Then the job starts, a real dataset lands on their desk, and it turns out the model was never the hard part.

The hard part is everything upstream of it. Framing a vague question precisely enough to answer it. Checking whether the data can actually support that answer. Noticing the assumption that breaks silently instead of throwing an error. None of that needs research-level math — most of it doesn't need statistics at all, just the discipline to check before trusting. It needs the willingness to slow down exactly when it's tempting to jump to the model, because the model is the part that feels like progress.

Three moments from a fairly ordinary analytics project show what that looks like in practice

The question shows up vague, and STAYS vague
A stakeholder asks "are our customers happy?" — which is a request for a feeling, not something a query can return. Happy compared to what: last quarter, a competitor, what they expected at signup? Which customers — everyone, or just the ones active in the last 90 days? Measured by what — a survey score, a churn rate, or support-ticket volume, which could reflect a product bug as easily as unhappiness? The junior move is to pick a metric and start coding. The useful move is rewriting the question until two different analysts would agree on exactly what to compute — same population, same comparison point, same metric — even if their pulls differ by a rounding error. Nobody drills that rewrite explicitly; it doesn't sit next to "regression" and "clustering" in a syllabus, but it decides everything that follows it.
Somewhere in the pipeline, rows disappear and NOBODY notices
Say the next step is joining an orders table to a customers table:

merged = orders.merge(customers, on="customer_id")
print(len(orders), len(merged))   # 48,201  →  44,987

Three thousand orders just vanished — every one with a customer_id that didn't find a match on the other side. They might be test accounts. They might be guest checkouts. They might be exactly the segment the analysis was supposed to describe, quietly deleted by an inner join before anyone got to look at them. merge() doesn't warn either way; a silently biased dataset and a clean one produce the exact same "no errors" console output. The only defense is checking row counts before and after every join and being able to explain any drop.

Data type problems hide the same way. A column named signup_date sounds unambiguous right up until .dtype returns object instead of datetime64, and a sample of five values turns up '2024-01-15', '01/15/2024', '2024-15-01', and 'Jan 15 2024' sitting in the same column — three years of manual entry, a spreadsheet import, and an API migration, never reconciled. 01/15/2024 alone is ambiguous between January 15th and an invalid 15th month, depending on which locale wrote it. Parse that carelessly and some rows land on the wrong date, quietly skewing any monthly bucket or tenure calculation downstream. Catching it costs about five minutes: check .dtype, then eyeball a random sample of raw values before trusting what the column name implies. Skipping it costs a wrong number that someone else finds later.

A join doesn't just drop rows — it can multiply them, and a plausible-looking total hides it

Say the next step is tagging orders with any promo code they used, to break out revenue by month:

revenue_by_month = orders.merge(
  promotions,on="order_id"
).groupby("month")["revenue"].sum()
# reported total: $1,284,000 — actual total: $1,061,000

If even a handful of orders matched two promotion rows instead of one — a coupon logged twice, a duplicate entry from a batch re-run — the merge doesn't collapse them, it repeats that order's revenue row once per match. The total comes out $223,000 too high, and nothing about the output looks wrong: no error, no null, just a bigger number than reality. The fix costs nothing statistical — it's checking that the row count and the total after a join still match a number you trust from somewhere else, before either one gets reported.

None of these three moments required anything past intro-level statistics — most required none at all. Each one involved noticing something easy to miss and choosing to stop and check it rather than push on to the modeling step, which is the part that actually feels like work. That's closer to the real shape of an entry-level data science job than most course outlines suggest — the algorithms are documented, implemented, and one import away in scikit-learn; the dropped rows, the ambiguous date format, and the inflated total are not, and they're what a hiring manager is actually screening for whether or not the posting says so.

For anyone building toward a first role, that argues for a different practice list than "learn another algorithm":

Rewrite a vague business question into one with an unambiguous metric and comparison, before writing any code
Check row counts before and after every join or filter, and be able to explain any change
Inspect column dtypes and a sample of raw values before trusting what a column's name implies
After any join, check that both the row count and a key total still match a number you trust from elsewhere
Practice stating a finding's limitations in the same breath as the finding itself

None of it is glamorous, and none of it shows up as a line on a certificate. It's also the difference between an analysis that survives a second question from a skeptical stakeholder and one that quietly comes apart the first time someone re-runs it on next month's data.

SophiArch's Intro to Data Science course is built around exactly this — not another pass through the algorithm list, but deliberate practice framing questions, auditing data, and catching this class of mistake before it reaches someone else's desk.