DEV Community: Jason Lau

Your Agent Has Observability. It Doesn't Have Evals.

Jason Lau — Thu, 24 Sep 2026 05:43:11 +0000

TLDR: In LangChain's survey of 1,340 practitioners, 89% had implemented observability for their agents and 94% of teams with agents in production had it. Offline evaluations: 52.4%. Online evaluations: 37.3%. Fewer than a third of respondents run both. That gap is not a tooling gap — it's a category error. A trace records what the agent did. Nothing in a trace records whether it was right, because a confidently wrong tool-call chain emits exactly the same telemetry as a correct one: same token counts, same latencies, same finish_reason, same green spans end to end. The OpenTelemetry GenAI conventions make this literal — there is no attribute for correctness until you run an evaluator and write one in.

Two traces

Here is the shape of the problem, in the only form that makes it obvious. A support agent handles refund requests. Two runs, side by side, as your observability platform renders them.

// Run A
{ "trace_id": "a41f…", "gen_ai.operation.name": "agent_run",
  "gen_ai.request.model": "claude-sonnet-4-6", "duration_ms": 3180,
  "gen_ai.usage.input_tokens": 2847, "gen_ai.usage.output_tokens": 193,
  "gen_ai.response.finish_reasons": ["stop"],
  "spans": [
    { "name": "lookup_customer",  "status": "OK", "duration_ms": 112 },
    { "name": "list_orders",      "status": "OK", "duration_ms": 340 },
    { "name": "issue_refund",     "status": "OK", "duration_ms": 908 }
  ], "error_count": 0 }

// Run B
{ "trace_id": "b73c…", "gen_ai.operation.name": "agent_run",
  "gen_ai.request.model": "claude-sonnet-4-6", "duration_ms": 3204,
  "gen_ai.usage.input_tokens": 2851, "gen_ai.usage.output_tokens": 188,
  "gen_ai.response.finish_reasons": ["stop"],
  "spans": [
    { "name": "lookup_customer",  "status": "OK", "duration_ms": 118 },
    { "name": "list_orders",      "status": "OK", "duration_ms": 336 },
    { "name": "issue_refund",     "status": "OK", "duration_ms": 913 }
  ], "error_count": 0 }

Every field that your monitoring dashboard aggregates is, for practical purposes, identical. Both runs are three spans, all OK, no errors, ~3.2 seconds, ~2,850 input tokens, a clean stop finish. On a latency percentile chart they are the same point twice. On a token-spend chart they are the same point twice. On an error-rate chart they contribute equally to a rate of zero.

In Run B the customer had two orders — a $40 one they were complaining about and a $2,300 one they weren't. The agent called issue_refund on the second.

Nothing in that trace is wrong. issue_refund really was called. It really did return OK, because refunding $2,300 is an entirely successful refund. The span is green because the API call succeeded, and the API call succeeding is the only thing a span status has ever meant.

The gap has a number attached to it

This is not a hypothetical asymmetry, and it isn't rare. LangChain's State of Agent Engineering report, published 23 May 2026 and drawn from 1,340 responses collected between 18 November and 2 December 2025, puts numbers on both halves:

	Adoption
Agents in production	57% (67% at 10,000+ employees)
Observability implemented	89% overall, 94% among production teams
Full per-step tracing	62% overall, 71.5% among production teams
Offline evaluations on a test set	52.4%
Online evaluations in production	37.3% (44.8% among production teams)
Both offline and online	fewer than a third

Observability is close to universal; systematic evaluation is roughly a coin flip. And the same survey names quality — accuracy, relevance, consistency, tone — as the top barrier to getting agents into production, cited by about a third of respondents. Those two findings sit awkwardly together. Quality is the thing most likely to block a deployment, and it is the thing least likely to be measured. Roughly nine teams in ten can reconstruct what their agent did last Tuesday. About five in ten can tell you whether it was any good.

The instinct to instrument first is a reasonable one, and the tooling market rewarded it — tracing is a solved, buyable, drop-in problem. Evaluation is not buyable in the same way, because it requires you to state what correct means for your domain, and nobody can do that for you. Observability got adopted at the speed of a vendor integration. Evaluation adoption moves at the speed of the hardest product conversation on your team.

Why the dashboard structurally cannot tell you

It's worth being precise about why the traces above are indistinguishable, because the reason isn't an oversight anyone will patch.

The OpenTelemetry GenAI semantic conventions define the standard attribute registry for instrumenting LLM and agent workloads — the fields your platform is almost certainly collecting. The core ones:

gen_ai.request.model, gen_ai.provider.name — which model answered
gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, plus cache-read, cache-creation and reasoning token counts — how much it cost
gen_ai.response.finish_reasons — why it stopped
gen_ai.operation.name — chat, tool_call, agent_run
gen_ai.input.messages, gen_ai.output.messages — what went in and came out

Read that list looking for the field that goes red when the agent refunds the wrong order. It isn't there, and it can't be, because every one of those attributes is a property of the call rather than of the answer. Cost, latency, model version and stop reason are all knowable from inside the request/response cycle. Correctness is only knowable with reference to something outside it — a ground truth, a business rule, a resulting state you can inspect.

The conventions do include an evaluation namespace — gen_ai.evaluation.name, gen_ai.evaluation.score.value, gen_ai.evaluation.score.label, gen_ai.evaluation.explanation — and score.label accepts values like "correct". But that namespace is a slot, not a sensor. Those attributes are empty until something you wrote computes a judgement and populates them. The standard has a correctness-shaped hole in it and expects you to bring your own filling. Most teams ship the ninety per cent that auto-instruments and never fill the hole.

This is the practical consequence: your alerting is wired to signals that do not move when quality moves. Latency won't regress when the agent starts refunding the wrong order — it might even improve. Token counts won't move. Error rate stays at zero, because from the runtime's point of view nothing failed. You will find out from a customer.

Every field a trace carries is a property of the call. Correctness is a property of the outcome, and the outcome has to be checked against something the request doesn't contain.

What measuring the outcome actually looks like

The fix has a well-established precedent, and it predates the current agent wave: verify the resulting state, not the transcript.

τ-bench, the tool-agent-user benchmark from Sierra, is built on exactly this move. Its authors describe the method plainly: they "employ an efficient and faithful evaluation process that compares the database state at the end of a conversation with the annotated goal state." Not the wording of the reply. Not whether the right tool name appeared. The database, afterwards, against what the database should have contained.

Applied to Run B above, that check is unglamorous and about four lines long:

def verify_refund(conversation, db_before, db_after):
    expected = conversation.annotated_goal      # refund order #4471, $40.00
    actual   = diff(db_before, db_after)        # refunded order #4473, $2,300.00
    assert actual.refunded_order_ids == expected.refunded_order_ids
    assert actual.total_refunded_cents == expected.total_refunded_cents

That assertion fails on Run B and passes on Run A. No telemetry field distinguished them; a four-line state diff does it instantly. The same pattern generalises to whatever your agent actually touches — if it writes code, run the test suite; if it files tickets, assert on the ticket's fields; if it produces a document, check the document against the source.

τ-bench also contributes a metric worth stealing outright. Alongside the usual pass@k ("at least one of k attempts succeeded"), the paper proposes pass^k — all k attempts succeeded — "to evaluate the reliability of agent behavior over multiple trials." This is the honest metric for anything customer-facing, because your customers are not running your agent eight times and keeping the best result; they're getting one run each. The arithmetic is brutal and worth internalising: an agent that succeeds 90% of the time, if failures are roughly independent, is fully reliable across eight consecutive runs only 0.9⁸ ≈ 43% of the time.

The benchmark's own headline finding, from the abstract, is the part most teams have not absorbed: "even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail)." That was measured on a purpose-built benchmark with annotated goal states. Your production agent has neither, and the assumption that it is doing better is currently unfalsifiable — which is precisely the problem.

"We'll just add an LLM judge"

This is the standard next move, and it's the right direction, but it is not free and it is not a shortcut past defining correctness. A judge is another model whose agreement with your actual standard is an empirical question — one that has now been measured at scale, and the results argue for treating judges as instruments that need calibration rather than as oracles.

The largest systematic study of the approach to date — 21 judges from nine providers across MT-Bench, JudgeBench and RewardBench, 118 runs and roughly 541,000 individual judgments — opens by naming the methodological problem directly: "judge validation in practice relies on exact-match agreement, a metric that does not correct for chance and systematically overstates discriminative ability." Two findings in particular should change how you read a judge's score:

Chance correction matters enormously. The paper reports that "kappa deflation between exact match and Cohen's κ is universal (33–41 pp on MT-Bench)." A judge advertising 80% raw agreement with human labels may be delivering far less genuine discriminative power than that number implies, because a meaningful share of those agreements are what you'd get by guessing.
Consistency is not accuracy. The authors document a "consistency–bias paradox," finding that "high test–retest reliability (>0.95) coexists with severe position bias (>0.10)" in production judges. A judge that returns the same verdict every time looks trustworthy on a stability check and can still be reliably wrong in a direction that tracks the order you presented the options in.

None of this makes LLM-as-judge unusable — it remains the only practical option for open-ended generation where no reference string exists. It does mean a judge is a component that itself requires a labelled validation set, a chance-corrected agreement statistic, and a position-swap check before you let it gate a release. Where you can verify state instead, verify state: an assertion on a database row has no position bias.

Turning the traces you already have into the evals you don't

The useful thing about being in the 89% is that you are already sitting on the raw material. Traces are unlabelled evaluation data. The work is labelling and replaying them.

Pull fifty real traces from last month, weighted toward the unusual: long tool chains, retries, sessions a human took over, anything a customer followed up on. Not fifty happy paths — happy paths inflate every number you're about to compute.
For each one, write down what should have happened, in a form a program can check. An order id. A row count. A set of fields. An exit code. If you cannot express it as an assertion, that's the product conversation surfacing, and it is better to have it now than during an incident.
Replay them on every change — prompt edits, model version bumps, tool-schema changes, dependency upgrades — and gate the release on the result. This is the step that converts the set from a document into a regression gate. It is also, mechanically, unit testing; the only novel part is that the assertions run against end state rather than return values.
Report pass^k, not just the mean. Run each case several times and count only the cases that pass every time. The number will be worse than your average, and it is the one that corresponds to what a single user experiences.
Feed failures back. Every production failure becomes case fifty-one. The set gets stronger exactly where the system is weakest, which is the whole point of curating it by hand.

Steps 2 and 3 are where teams stall, and the reason is rarely technical — it's that "what should have happened" turns out to be genuinely contested between engineering, support and finance. Working through that with a versioned golden set and a regression gate on every change is a chunk of what SophiArch's AI Applications with LLMs course spends its validation module on, alongside the observability side that most teams have already built. The two lessons sit deliberately close together, because shipping one without the other is the failure this whole article is about.

The one-line version

Observability answers "what happened?" Evaluation answers "should it have?" The first question is solved, buyable and nearly universally adopted. The second is the one your customers are actually asking, and roughly half of teams running agents in production still have no systematic way to answer it.

If you have 89% of the stack, the remaining work isn't another dashboard. It's fifty labelled traces and an assertion that fails.

References

LangChain. (2026, May 23). State of Agent Engineering — survey of 1,340 practitioners, fieldwork 18 Nov–2 Dec 2025; source of the observability (89%), offline eval (52.4%), online eval (37.3%) and production (57%) figures. Accessed 2026-09-23.
Yao, S., Shinn, N., Razavi, P., & Narasimhan, K. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045 — source of the database-state evaluation method, the pass^k metric, and the pass^8 <25% retail result. Accessed 2026-09-23.
Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias. (2026). arXiv:2606.19544 — 21 judges, nine providers, ~541,000 judgments; source of the kappa-deflation and consistency–bias findings. Accessed 2026-09-23.
OpenTelemetry. Gen AI Attributes registry — the standard gen_ai.* span attributes, including the gen_ai.evaluation.* namespace referenced above. Accessed 2026-09-23.
SophiArch. AI Applications with LLMs — course covering the probabilistic contract, context window architecture, output validation layers, evaluation frameworks and golden sets, orchestration patterns, and LLM observability, including the regression-gate workflow described here.

Text-to-SQL Accuracy Isn't a Model Problem. It's a Schema Problem.

Jason Lau — Wed, 16 Sep 2026 08:10:15 +0000

TLDR: dbt Labs ran the same eleven questions four different ways and found that modelling the schema — with no semantic layer involved at all — moved text-to-SQL accuracy from 64.5% to 90.0%. Adding a semantic layer on top of that modelled schema moved it a further 8 points. Snowflake, using a semantic model across four BIRD databases, measured a 21-point average lift. MotherDuck pointed three frontier models at small, clean schemas with nothing but the DDL and reported 95% — though those same runs score 58–64% under BIRD's strict scoring. Read side by side, these look contradictory. They aren't. None of them is measuring the model. They're measuring how much translation work somebody already did to the schema, and how forgivingly the answers were graded.

Ask an AI assistant to write a query against your warehouse and it will, almost every time, produce something that runs. That's the trap. A query that executes and returns a number looks exactly like a query that answers the question — right up until someone downstream builds a forecast on it.

Two queries, one question, two answers

Here's the shape this takes on an ordinary warehouse, away from the benchmark leaderboards. A SaaS company wants last month's revenue. An AI assistant, asked two reasonable questions, produces two queries.

-- Query A: "what was our revenue last month?"
SELECT SUM(amount_due) AS revenue
FROM invoices
WHERE status = 'paid'
  AND billing_period = '2026-08';

-- Query B: "break down last month's revenue by product"
SELECT p.name AS product,
       SUM(i.amount_due) AS revenue
FROM invoices i
JOIN invoice_line_items ili ON ili.invoice_id = i.invoice_id
JOIN products p ON p.product_id = ili.product_id
WHERE i.status = 'paid'
  AND i.billing_period = '2026-08'
GROUP BY p.name;

-- Query A result:
revenue = 1,148,000

-- Query B result:
product            revenue
-----------------  ---------
Platform (base)    1,664,000
Extra seats          988,000
Usage overage        705,000
-----------------  ---------
total              3,357,000

Both queries ran without error. Both came from reasonable, good-faith prompts. But Query B sums i.amount_due — the invoice total — once for every line item attached to that invoice. A customer with a base plan, an extra seat and a usage overage contributes their entire invoice total three times, once under each product. "Platform (base)" doesn't show platform revenue; it shows the full value of every invoice that happened to contain a platform line.

The fix is one column: sum ili.line_amount, the line's own amount, not the invoice's. Nobody wrote a bug. The join changed what one row means, and SUM kept adding as if it hadn't. If Query B's breakdown reaches a board deck before anyone reconciles it against Query A, that takes an awkward meeting to walk back.

This is not a model failure. Point a frontier model at this schema and it will make the same mistake confidently and articulately, because nothing in the schema says that invoice_line_items has a different grain than invoices, or that amount_due is only additive at the invoice level. That knowledge lives in someone's head, or it lives in a semantic layer. If it lives in neither, which query you get is a coin flip.

The spread that looks like disagreement

Three of the most-cited text-to-SQL results around, read together, look like they're arguing. Read separately, each is internally consistent — which is the tell that they're measuring different things.

dbt Labs' April 2026 benchmark tested four configurations over an insurance dataset, eleven questions, twenty runs per model. The one people quote is text-to-SQL versus semantic layer on a modelled project: Claude Sonnet 4.6 at 90.0% against 98.2%, GPT-5.3-Codex at 84.1% against 100.0%. The more interesting comparison is the one underneath it. Against the original normalised tables, plain text-to-SQL managed 64.5% across all eleven questions; the same method against a modelled project hit 90.0%. That 25.5-point improvement came from modelling alone, with no semantic layer anywhere in the picture — and dbt built it by prompting an LLM to write "as few dbt models as possible," which produced just three. Their summary: "Adding even minimal modeling on top of raw tables improved results across the board."
Snowflake's semantic-model evaluation (March 2025, using Claude 3.5 Sonnet) is often cited as "57% to 78% on BIRD." It's really an average across four BIRD databases, and the spread inside it matters: debit_card_specializing went 52% → 83%, california_schools 63% → 80%, thrombosis_prediction 45% → 70%, toxicology 69% → 79%. Averaged, an "approximately 20% increase in accuracy" — 21 points — from adding a semantic model, not from a newer model.
MotherDuck ran the opposite experiment, pointing Claude Opus 4.5, GPT-5.2 and Gemini 3 Flash at 500 BIRD questions with, in their words, "No semantic layer. No query history. No special context. Just the schema." The reported result is 95%. That figure sits at the top of a four-tier evaluation ladder, and the bottom rung is a lot less flattering: 64.0% (train) and 58.2% (test) under strict BIRD execution matching, rising through correction of known benchmark errors and tolerance for formatting differences to 94.9%/94.4% once an LLM judges whether the answer is defensible. Their argument is two-part — that BIRD's strict scoring is itself misleading, because "to score above 62% under strict rules, you have to start reproducing the benchmark's mistakes," and that "Good data modeling is the semantic layer."

So there are two variables moving here, not one, and honest reading requires separating them. Scoring strictness explains most of the distance between MotherDuck's 95% and everyone else's numbers; you cannot line that figure up against dbt's or Snowflake's and treat the gap as real capability. Schema modelling explains the rest, and it's the variable you control. dbt moved text-to-SQL a long way with three models and no semantic layer. Snowflake moved it 21 points with a semantic model over databases they didn't restructure. MotherDuck got high marks with no layer at all — over BIRD databases that, as they note, average seven tables. None of these is a claim about which model writes better SQL.

There's a ceiling worth keeping in view too. BIRD's own paper reports that its human baseline — data engineers and database students — reached "the human result of 92.96%" execution accuracy. People who write SQL for a living, working on databases of seven tables, get roughly one question in fourteen wrong. The bar was never "flawless." It's "at least as reliable as the analyst who used to own this."

What "the schema needs modelling" actually means

"Modelling" sounds like an abstraction exercise. In practice, on a schema like the one above, it's answering three concrete questions before anything touches a model:

What is the grain of each table, in one sentence? invoices is one row per invoice. invoice_line_items is one row per line on an invoice. Join them and your result set's grain becomes "one row per line item" — so any SUM of an invoice-level column is now double-, triple- or n-counting, depending on how many lines the average invoice carries. That single fact is what would have caught Query B.

Which columns carry business meaning that isn't in their name? status = 'paid' looks unambiguous. Does it need to exclude refunds? Is a partially refunded invoice still "paid" for revenue purposes? These are policy decisions someone made once, verbally, in a meeting nobody minuted — and no assistant can recover them from the schema alone.

Is there more than one path to the same number? Revenue might come from invoices.amount_due, from summing invoice_line_items.line_amount, or from an mrr_snapshots table a finance job populates nightly. Three legitimate paths, three numbers that won't quite agree, and none of them "wrong" — they answer slightly different questions that all get asked as "what's our revenue."

"Just model your warehouse" is easy advice to give

The obvious objection to all of this: dbt's benchmark fixed its schema with three new models over an eleven-question insurance dataset. You have four hundred tables, three teams who each define revenue differently, and a migration budget of zero. "Model the schema properly" is not a thing you can do this quarter, and anyone who has tried knows that the modelling is the easy half — getting three departments to agree which definition wins is the hard one.

That constraint is exactly why semantic layers exist. They're the retrofit path: a place to encode grain, join paths and metric definitions without restructuring the warehouse underneath. Snowflake's 21-point average came this way, over databases nobody rebuilt. Treating "clean DDL" and "semantic layer" as interchangeable is true for benchmarks and false for anyone with legacy tables and a roadmap.

There's also a second argument for the retrofit that the accuracy columns actively hide. In dbt's unmodelled configuration, the semantic layer scored 0.0% on the subset of questions requiring too many joins to resolve — not because it answered them wrongly, but because it declined to answer at all. As dbt puts it: "the Semantic Layer tells you it can't answer. It never returns invalid data. Text-to-SQL will cheerfully give you a wrong number." Plain text-to-SQL scored 70–100% on that same subset, which looks like a win until you remember that nothing in those results distinguishes a correct answer from a confident one. A coverage gap you can see beats a coverage gap that quietly returns 3,357,000. That asymmetry appears in no accuracy column anywhere, and for a number headed to a board deck it may matter more than the points do.

What both routes share is scope. Neither dbt's three models nor a semantic layer requires modelling the whole warehouse — only the slice that answers the questions people actually ask. That's the practical move: take the handful of questions your team asks weekly, model just the tables those touch, and leave the other three hundred and eighty alone until someone asks them something.

The benchmarks didn't find a model problem. They found a translation-work problem, and measured what happens when someone does the translation once instead of leaving it to a fresh guess on every query.

The checklist that transfers to your warehouse

None of the studies above tells you which side of the gap your own schema sits on. Run this instead:

Take your five most-asked business questions and write down, for each table involved, its grain in one sentence. If you can't do that quickly, an assistant generating SQL against those tables can't either.
Find every column whose name promises more precision than its values deliver — status, type, category especially. If status needs a footnote to interpret correctly, that footnote has to live somewhere the assistant can read: a view, a semantic layer, a column comment. Not a Slack thread from 2024.
Ask the same business question two ways and see whether you get two numbers. That's what happened above. If it happens on your warehouse, you have a modelling gap rather than an AI-competence gap, and no model upgrade closes it.

The numbers make the same point from three directions: 90% accuracy sounds like a solved problem until you notice the missing 10% is where double-counted joins and silently redefined metrics live, invisible precisely because the query still runs. Building the judgment to catch that — reading a join and knowing what it did to the grain, reconciling a number against a second path before trusting it — is what SophiArch's SQL for Data Analysis course is built around, with a dedicated lesson on joins and double-counting early on and a closing module on auditing SQL that an AI assistant wrote.

References

Ganz, J., & Perigaud, B. (2026, April 7). Semantic Layer vs. Text-to-SQL: 2026 Benchmark Update. dbt Developer Blog. Accessed 2026-09-16.
Zayats, T. (2025, March 31). Agentic Semantic Model Improvement: Elevating Text-to-SQL Performance. Snowflake Engineering Blog. Accessed 2026-09-16.
MotherDuck. Your Data Model Is the Semantic Layer. MotherDuck Blog (undated). Accessed 2026-09-16.
Li, J., Hui, B., Qu, G., et al. (2023). Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs. arXiv:2305.03111 — source of the 92.96% human-performance baseline on BIRD-SQL. Accessed 2026-09-16.
SophiArch. SQL for Data Analysis — course covering filtering, aggregation, joins, CTEs, window functions and cohort analysis, closing with a module on auditing AI-generated SQL for the silent failure modes described here: double-counting joins, NULL traps and misplaced filters.

Long Context Didn't Kill RAG. It Killed Lazy RAG

Jason Lau — Wed, 09 Sep 2026 06:41:30 +0000

Short version: In a controlled comparison published in June 2026, long-context prompting beat semantic RAG on correctness — 73.1% against 65.4% — and cost 26 times as much per query to do it. That single row is the whole argument: long context is genuinely better at the thing people claim it's better at, and the reasons to retrieve anyway are cost, corpus size, freshness, and permissions, not accuracy. What long context actually made obsolete is the naive pipeline — fixed chunks, one embedding model, top-k by cosine, no re-ranking — which was never competitive with a big window and is barely competitive with a well-built retriever either.

Somebody says it in a planning meeting roughly once a quarter now: models take a million tokens, so why are we still maintaining a vector database? It is a fair question, asked in bad faith about half the time, and the usual response is a defensive listicle titled "RAG isn't dead." Those posts are mostly right and almost never useful, because they argue the conclusion instead of giving you a way to reach it for your own corpus.

So let's do it the other way round. Concede the strongest version of the opposing case first, then work out what's left.

The strongest case for "just put it in the window"

The most useful comparison I've found isn't a vendor benchmark, it's a preprint that ran both architectures over the same document-grounded task and reported the trade-off as a frontier rather than a winner. Hamilton et al.'s "The Token Tax of Epistemic Accuracy" (June 2026) found that "Long-context prompting achieved the highest correctness (73.1% vs. 65.4% for semantic RAG), but at 26 times the per-query token cost."

Read that without flinching. Long context won on correctness by nearly eight points. Every architectural argument that starts "but retrieval is more accurate" is arguing against a measurement, and if you have been repeating that line, it needs retiring. Stuffing the documents in works, and it works better than a mid-quality retriever, for the obvious reason: a retriever that never surfaces the right passage has capped the system's accuracy at zero for that query, whereas a big window at least has the passage in the room.

The honest framing is that retrieval buys you everything except peak accuracy on a corpus that fits.

What it doesn't buy you, and where the "26x" goes

Start with the arithmetic, because it's the part people wave at rather than compute. Take a 200,000-token prompt — a modest corpus by 2026 standards — at an illustrative $3 per million input tokens. That's $0.60 per query before the model has emitted a single output token. At 50,000 queries a month, $30,000. Answer the same question from six retrieved chunks, call it 4,000 tokens, and you're at $0.012 a query, or $600 a month. Same model, same questions, fiftyfold difference, and none of it shows up in the demo because the demo ran forty queries.

There's a real counter-argument here that RAG advocates usually miss: prompt caching. If the 200,000-token prefix is identical across queries, most providers will serve the repeated portion at a steep discount, and the gap narrows dramatically. But notice the condition. Caching pays off when the corpus is small enough to fit, stable enough not to churn, and shared across users — which is precisely the regime where nobody needed retrieval in the first place. The moment the prefix differs per user, per tenant, or per hour, the discount evaporates and you are back to paying full freight on every token you didn't need.

Then there's the accuracy claim itself, which is conditional in a way the headline number hides. Three independent lines of evidence say a long window is not a uniform one:

Chroma's Context Rot study (Hong, Troynikov and Huber, July 2025) evaluated 18 models and found that "model performance varies significantly as input length changes, even on simple tasks," with degradation worsening when distractors are present and when the question shares little wording with the passage that answers it.
NoLiMa (Modarressi et al., ICML 2025) tested 13 models that all claim 128K-plus context and reported that "at 32K, for instance, 11 models drop below 50% of their strong short-length baselines" — with GPT-4o falling from a 99.3% baseline to 69.7%.
Leng et al. found across 20 models that "while retrieving more documents can improve performance, only a handful of the most recent state of the art LLMs can maintain consistent accuracy at long context above 64k tokens."

Put those next to the Token Tax result and the picture resolves: long context wins when the relevant material fits comfortably and the query wears its keywords on its sleeve. It degrades exactly where enterprise questions live — paraphrased queries, near-duplicate distractors, corpora far past the window.

Three questions that settle it for your corpus

The reason the general debate never converges is that it's the wrong altitude. Run your own numbers instead. In order:

1. Does the material relevant to a single query fit in the window with room to spare — every time? Not the average query, the worst one. If your corpus is a 40-page product spec, retrieval is architecture you don't need. If it's 80,000 support tickets and the answer depends on which three of them are similar, no window is large enough and the question is settled.

2. What does it cost at your query volume, after caching? Do the multiplication above with your real rate and your real monthly volume. Then ask whether the prefix is genuinely shared across queries or varies per user. Shared and stable, caching rescues long context; per-user or fast-changing, it doesn't.

3. Does the corpus change, and does who's asking change what they may see? This is the axis that decides more real systems than the other two combined, and it barely appears in the online argument. A window has no concept of freshness — you rebuild the prompt or you serve stale content. It has no concept of permission either: everything in the prompt is visible to whoever's asking. An index has both. Documents are re-embedded when they change; filters are applied per user at query time. If your answer to either half is yes, you need a retrieval layer regardless of what the matrix below says about accuracy alone.

Accuracy alone points at long context. Freshness and permissions move three of the four quadrants to retrieval.

"Better retrieval" isn't a vibe, it's a short and specific list

If questions 1–3 land you on retrieval, the Token Tax result becomes a warning rather than a footnote: a mediocre retriever loses to a big window by eight points of correctness. So the pipeline that "everyone knows" — split at 512 tokens, embed, take the top five by cosine, done — isn't a starting point to defend. It's the thing long context genuinely made obsolete.

What replaces it is four changes with measurable effects, each of which gets a lesson in the Retrieval & RAG Systems course I teach, because they are the dials that actually move recall:

Hybrid retrieval. Combine lexical (BM25) and dense scoring, fused by reciprocal rank. A 2026 benchmark of ten retrieval strategies over 23,088 financial queries found the best configuration was a two-stage hybrid-plus-reranking pipeline — and, more pointedly, that "BM25 outperforms state-of-the-art dense retrieval on financial documents." Exact identifiers, product codes and figures are where embeddings are weakest and keyword matching is strongest.
Cross-encoder re-ranking. Retrieve broadly, then score query-document pairs jointly. Rosa et al. found cross-encoders beat comparable bi-encoders by over four points on BEIR, with the gap widening on domains the model wasn't trained on — and, awkwardly for the default stack, that bi-encoder first-stage retrieval "provides no gains in comparison to a simpler retriever such as BM25 on out-of-domain tasks."
Chunking as a retrieval decision. Chunk size is not a formatting preference; it decides what a single embedding is asked to represent. Splitting mid-argument is how you get a passage that scores well and answers nothing.
Measurement before cleverness. The same benchmark (Akarsu et al.) found that "query expansion methods (HyDE, multi-query) and adaptive retrieval provide limited benefit for precise numerical queries." Sophistication isn't automatically an improvement. Recall@k on your own labelled queries is what tells you which of these earned its latency.

Two retrieval signals fused by rank, then jointly scored against the query — not a single embedding search returning whatever ranks fifth by cosine.

That last one is the actual discipline. Every technique above is a hypothesis about your corpus, and the corpus gets a vote.

So: not "is RAG dead." Does the material fit, what does it cost at volume, and does it change or vary by reader. Three questions, answerable in an afternoon with your own numbers, and they will land you somewhere more defensible than either side of the argument you were being asked to join.

References

Hamilton, A., Singh, R., Wise, M., Yousif, I., Carvalho, A., Shan, Z., Mayyas, M., Cavuoto, L. A., & Megahed, F. M. (2026). The Token Tax of Epistemic Accuracy: Comparing RAG and Long-Context Architectures for Document-Grounded Generative AI Applications. arXiv:2606.20898.
Hong, K., Troynikov, A., & Huber, J. (2025). Context Rot: How Increasing Input Tokens Impacts LLM Performance. Chroma Research.
Modarressi, A., Deilamsalehy, H., Dernoncourt, F., Bui, T., Rossi, R. A., Yoon, S., & Schütze, H. (2025). NoLiMa: Long-Context Evaluation Beyond Literal Matching. ICML 2025.
Leng, Q., Portes, J., Havens, S., Zaharia, M., & Carbin, M. (2024). Long Context RAG Performance of Large Language Models. arXiv:2411.03538.
Akarsu, M., Karaman, R. K., & Mierbach, C. (2026). From BM25 to Corrective RAG: Benchmarking Retrieval Strategies for Text-and-Table Documents. arXiv:2604.01733.
Rosa, G., Bonifacio, L., Jeronymo, V., Abonizio, H., Fadaee, M., Lotufo, R., & Nogueira, R. (2022). In Defense of Cross-Encoders for Zero-Shot Retrieval. arXiv:2212.06121.
SophiArch. Retrieval & RAG Systems — course covering chunking strategy, hybrid BM25-plus-dense retrieval with reciprocal rank fusion, cross-encoder re-ranking, retrieval metrics, index freshness and permission-aware retrieval.

Prompt Engineering Won't Survive — the Skill Underneath It Will

Jason Lau — Wed, 02 Sep 2026 09:49:37 +0000

For about two years, "prompt engineer" was the job title recruiters attached to six-figure listings and career-changers scrambled to add to a resume. Then the interest mostly stopped converting into jobs. The Wall Street Journal reported that Indeed searches for the role spiked to 144 per million U.S. searches in April 2023 and have since flatlined at 20-30 per million — and, per Indeed VP of AI Hannah Calhoon, that interest was never matched by actual employer job postings in the first place: "Maybe they talked about the value of prompt engineers, but they weren't then actually hiring for that." A title that rose and mostly stalled out inside two model generations. Microsoft's Jared Spataro put the reason bluntly: "You don't have to have the perfect prompt."

That's a fair obituary for the "prompt engineer" job title. It's a bad obituary for the underlying work, and the mistake is treating both as the same thing.

What actually died was a set of tricks tuned to specific models

Open any 2023-era prompt-engineering article and it's mostly mantras: "think step by step," phrase it as a role assignment, wrap the instruction in triple quotes, promise the model a tip. These worked because they guided a specific model around a specific weakness — and weaknesses don't persist across model generations.

Chain-of-thought is the clearest case, because it's the most-cited prompting technique there is and it's now been re-measured against current models. A 2025 technical report from Wharton's Generative AI Labs ran chain-of-thought prompting against eight models spanning both non-reasoning and reasoning types. The results don't point one direction. Gemini 2.0 Flash gained 13.5% and Sonnet 3.5 gained 11.7% — real improvements. GPT-4o-mini gained 4.4%, a difference the authors couldn't call statistically significant. Gemini Pro 1.5's rate of getting every part of a problem fully correct dropped 17.2 percentage points with chain-of-thought turned on. Among the reasoning models — the ones already trained to produce their own extended reasoning — o3-mini and o4-mini picked up a marginal 2.9-3.1%, while Gemini Flash 2.5 lost 3.3%. All of this arrived with a real tax: 35-600% longer response times for non-reasoning models, 20-80% longer for reasoning ones. The paper's own framing is careful rather than dramatic — "its effectiveness depends significantly on model type and specific use case" — but read against the trend, that's a technique whose payoff is shrinking and inverting as the models it was compensating for get better at reasoning on their own.

A 2025 arXiv paper on what it calls the "Prompting Inversion" makes the same point sharper by comparing one technique across two model generations directly. The authors built "Sculpting," a heavily constrained prompting method, and pitted it against plain chain-of-thought. On GPT-4o, Sculpting won clearly — 97% accuracy against chain-of-thought's 93%. On GPT-5, the ranking flips: Sculpting drops to 94.00% while plain chain-of-thought reaches 96.36%. Their explanation is that the same constraints acting as guardrails for a mid-tier model become "handcuffs" on a more capable one — the scaffolding that stopped GPT-4o from wandering off just gets in GPT-5's way. A technique doesn't get more true as models improve; it can flip sign entirely.

The instability was already visible in how fragile the tricks were

None of this should be surprising if you look at how sensitive prompt outputs already were to changes that had nothing to do with meaning. Sclar, Choi, Tsvetkov, and Suhr measured how much purely cosmetic prompt formatting — whitespace, separator characters, capitalization — could move accuracy on the same task with the same content, and found swings of up to 76 accuracy points on LLaMA-2-13B. Critically, they found this sensitivity "remains even when increasing model size, the number of few-shot examples, or performing instruction tuning" — it isn't a small-model problem that scale quietly fixes.

That finding reframes what a lot of "prompt engineering" tips actually were. If reformatting a prompt with no change in meaning can swing accuracy by 76 points, then a technique's measured 10-point gain on some benchmark was never a stable property of the technique — it was one sample from a distribution of formatting accidents that happened to land favorably on that model, at that snapshot in time. Chasing that kind of gain by memorizing the exact phrasing that worked last week is chasing noise, and noise doesn't transfer to the next model release. You have to separate the parts of a prompt that are doing real work (the schema, the goal, the constraints) from the parts that are stylistic residue — which is exactly the split SophiArch's AI-Assisted Data Science course spends its second lesson on, precisely because the residue is what breaks first when your model evolves.

What was actually being taught the whole time

Strip out the mantras and what's left in a good prompt is closer to a specification than a sacred phrase. A 2024 peer-reviewed framework in Frontiers in Education defines prompt engineering as "the skill of communicating the problem, its context, and the constraints imposed on the desirable solution to an LLM to solve it correctly as fast as possible" — and its authors explicitly push back on the idea that this reduces to generic communication skill, arguing it's closer to task decomposition: knowing what you actually want closely enough to state it, before you ever open a chat window.

That's the part that doesn't expire when a new model ships, because it isn't a fact about models — it's a fact about the gap between what's in your head and what's on the page. Consider the difference between two requests to deduplicate a customer table:

"Clean up the duplicate customers in this table."

versus:

"Two rows are the same customer if email matches exactly (case-insensitive) OR both phone and last_name match. When rows conflict, keep the one with the most recent last_order_date; if that's tied, keep the one with more non-null fields. Do not merge rows that share only a last_name — that's a coincidence, not a duplicate. Return a mapping from removed customer_id to the customer_id it was merged into, so the removal is auditable."

Neither of these is a prompting trick. No chain-of-thought cue, no role-play framing, no promised tip. The second one just doesn't leave the model — or a junior analyst, for that matter — a single ambiguous decision to make on its own: what counts as a match, which record wins a conflict, what NOT to merge, and how the result gets checked. A model that's gotten better at reasoning will make smarter guesses at filling those gaps than last year's model did. It will still be guessing, and on a dataset that decides refund eligibility or fraud flags, "the model guessed well" is not the standard anyone should be signing off on.

That's the actual content of "prompt engineering" once you take the model-specific mantras out of it, and it's also why the skill was never new — it's requirements-writing and problem specification, wearing a 2023 label because a chat interface made the gap between vague and precise suddenly visible to everyone, all day, on every request. The interface is what's new. The gap it exposed is not, and closing it is a durable skill for exactly the reason the mantras weren't: it doesn't depend on which model happens to be running underneath.

References

Meincke, L., Mollick, E. R., Mollick, L., & Shapiro, D. (2025). Prompting Science Report 2: The Decreasing Value of Chain of Thought in Prompting. SSRN / Wharton Generative AI Labs.
You Don't Need Prompt Engineering Anymore: The Prompting Inversion (2025). arXiv.
Sclar, M., Choi, Y., Tsvetkov, Y., & Suhr, A. (2023). Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design. arXiv.
Bousquette, I. (2025). The Hottest AI Job of 2023 Is Already Obsolete. The Wall Street Journal.
Prompt Engineering as a New 21st Century Skill (2024). Frontiers in Education.
SophiArch. AI-Assisted Data Science — second lesson module on separating durable prompt structure (schema, goal, constraints) from model-specific tricks.

Your RAG Pipeline Doesn't Have an Accuracy Problem - It Has an Evaluation Problem

Jason Lau — Wed, 26 Aug 2026 03:37:34 +0000

A team builds a retrieval-augmented chatbot over the company's internal policy documents. In the demo, someone asks "how many days of parental leave do we get?" and the bot answers correctly, citing the right PDF. Someone asks about expense limits - correct again. Ten questions, ten good answers, applause, ship it.

Three months later an employee asks whether their contractor status qualifies for the health stipend, gets a confident "yes" assembled from a policy that was superseded last year, and files the claim. Nobody on the team can say when the pipeline started producing answers like that, because nothing was ever measuring whether it did.

That is the real state of most RAG systems in production. The pipeline is not unusually inaccurate - every retrieval system misses sometimes. What's missing is the apparatus that would notice. The demo was treated as the evaluation, and a demo is the one test a RAG system essentially cannot fail: the questions were chosen by the people who built the index, phrased the way the documents phrase things, asked about content everyone knew was in there.

A wrong answer and a right answer look identical

Traditional software fails loudly. A broken API call throws an exception; a bad deploy serves 500 errors; a failing test case turns red. A RAG regression does none of this. Swap the embedding model, change the chunk size, re-index after a document update - the system keeps returning fluent, well-formatted, confidently cited answers. Whether they are grounded answers is invisible in every signal you get: no exception, no latency spike, no schema violation.

This isn't an edge case of RAG engineering; it is a recurring finding in the engineering literature on it. A CAIN 2024 experience report across three RAG case studies (research, education, and biomedical domains) catalogued seven recurring failure points - missing content, missed top-ranked documents, answers retrieved but lost in consolidation, answers present in context but not extracted, wrong format, wrong specificity, incomplete answers - and its two headline takeaways are blunt: "validation of a RAG system is only feasible during operation, and the robustness of a RAG system evolves rather than designed in at the start."

Read that first clause again. You cannot fully validate this class of system before shipping it. Which means the eval harness is not a nice-to-have you add after launch - it is validation-during-operation, systematised: built from the questions real users actually asked, and run continuously as the system and its corpus change. It is the only mechanism by which you ever learn whether the thing works.

"RAG fixes hallucinations" is marketing, and there's a measurement to prove it

The reason teams skip evaluation is an ingrained assumption: retrieval grounds the model, so the hallucination problem is handled. The strongest counterexample comes from a domain with real money behind getting this right. Legal research vendors marketed their RAG products as "eliminating" or "avoid[ing]" hallucinations, even guaranteeing "hallucination-free" citations. When Stanford's RegLab ran the first preregistered empirical evaluation of these tools, the flagship products from LexisNexis and Thomson Reuters "each hallucinate between 17% and 33% of the time."

These are professional-grade systems built by teams with enormous resources over curated, authoritative corpora - the best case for RAG. Retrieval reduced hallucination relative to a bare model, but between one in six and one in three answers still contained fabrication. The gap between "we added retrieval" and "we measured what retrieval actually delivers on our queries" is exactly the gap the vendors' marketing fell into. If they can fall into it, your internal chatbot certainly can.

"Is it accurate?" is two questions wearing one trenchcoat

Suppose you accept the premise and ask: fine, how accurate is my pipeline? The question is underspecified, and that underspecification is why demo-vibes evaluation persists - there was never an agreed definition of correct to check against.

A RAG answer can fail in two independent places, and they need different fixes:

Retrieval failed. The chunks handed to the model didn't contain the answer - it wasn't indexed, didn't rank into the top-k, or got cut during context assembly. No prompt engineering will fix this; you need to change chunking, embeddings, or ranking.
Generation failed. The answer was in the retrieved context and the model ignored it, contradicted it, or embellished beyond it. Re-ranking harder won't fix this; you need to change the prompt contract, the model, or add output validation.

An aggregate "accuracy" number collapses these together and leaves you optimising blind. This decomposition is precisely what the RAGAS evaluation framework formalised into separately measurable quantities: context relevance scores the retrieval step (did the right evidence show up, without drowning in irrelevant material?), while faithfulness and answer relevance score the generation step (is every claim in the answer supported by that evidence, and does it address the question?). The Ragas library has since split the retrieval side further into context precision and context recall. The individual metric implementations have known rough edges - but the decomposition is the part that matters, because it turns "the bot was wrong" into a bug report that identifies a component.

What an actual evaluation harness looks like

None of this requires a research team. A minimum viable harness is three artefacts and a habit.

1. A golden set: your domain's definition of correct, written down. Collect 50-100 real questions - from support logs, pilot users, and the subject-matter expert who knows where the knowledge is buried. For each, record the expected answer and which document(s) it must come from:

{
  "id": "policy_031",
  "question": "Does the health stipend apply to contractors?",
  "expected_answer": "No - eligibility requires full-time employment status.",
  "must_cite": ["benefits-eligibility-2026.pdf"],
  "trap": "superseded 2024 policy still in corpus says yes"
}

That trap field is where the value lives. Easy questions inflate your score; the golden set earns its keep on superseded documents, questions whose answer spans two chunks, questions the corpus cannot answer (the correct behaviour is "I don't know" - the first failure point in the CAIN taxonomy is systems that fabricate rather than decline), and negation cases where the retrieved text says the opposite of what the surface phrasing suggests.

2. Scoring that preserves the retrieval/generation split. Score "did the must-cite document appear in the retrieved set?" separately from "is the answer faithful to what was retrieved?" For the generation side at scale you'll likely use an LLM judge - which is workable but not free of pathology: the MT-Bench study of LLM-as-a-judge found strong judges agree with human raters over 80% of the time, while also documenting systematic verbosity bias (longer answers score better regardless of quality) and signs of self-enhancement bias (judges may favour their own model's outputs). Use a judge, but spot-check it against human labels before trusting it as your regression signal.

3. A regression gate. The harness runs on every change - new embedding model, new chunking strategy, re-index, prompt edit, model version bump - and a score drop blocks the change, the same way a failing test suite blocks a merge. This is the step that converts evaluation from a one-time report into an engineering control. Without the gate, your golden set is a benchmark you ran once in a notebook; with it, "did we just get worse?" has an answer before users provide it.

The demo answered the questions you chose. The golden set answers the ones your users will actually ask - including the ones designed to make your pipeline lie. Build the second thing before you trust the first.

References

Barnett, S., Kurniawan, S., Thudumu, S., Brannelly, Z., & Abdelrazek, M. (2024). Seven Failure Points When Engineering a Retrieval Augmented Generation System. CAIN '24.
Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. (2023). RAGAS: Automated Evaluation of Retrieval Augmented Generation. arXiv.
Magesh, V. et al. Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. Stanford RegLab.
Zheng, L. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS.
Ragas documentation. List of available metrics.

If you want to go deeper than a blog post can, evaluation is where SophiArch's Building AI Applications with LLMs course plants its flag: a full module on golden sets, LLM-as-judge trade-offs, and regression gates, sitting inside a validation architecture that runs from prompt contracts through output checking to production observability.

Clustering Isn't Discovery - It's a Hypothesis You Still Have to Test

Jason Lau — Wed, 19 Aug 2026 09:56:42 +0000

An analyst runs KMeans(n_clusters=4) on dataset of customer purchase history, it gets back four groups, and he/she writes up the report as "from the data, it revealed four customer segments!" Management simply loves this - after long periods struggling to grasp at their customer base, they now have data-backed insights on real-names associated with each segment (Loyal Regulars, Bargain Hunters, At-Risk, New & Exploring), and pretty soon all the strategy meetings will silently assume those four groups are real and nobody even dares to question.

Unfortunately, those groups are not real and clustering does not "discover" hidden nuggets of insights in the data. Instead clustering imposes a partitioning based on what the analyst chooses as a similarity rule. If the analyst had chosen another reasonable rule, clustering can also fit the same dataset in a different way. Let's be clear - the four groups are a hypothesis about how to group the customers. To determine if the grouping is accurate, as in we can use this grouping to predict customer behavior, now that is a separate question that the clustering method can never answer by itself.

Same data, different "discovery"

To illustrate, try running KMeans clustering on an RFM table (recency, frequency, monetary value per customer) and only change the random seed:

from sklearn.cluster import KMeans

km_a = KMeans(n_clusters=4, n_init=1, random_state=7).fit(X_scaled)
km_b = KMeans(n_clusters=4, n_init=1, random_state=42).fit(X_scaled)

print(km_a.inertia_, km_b.inertia_)
# 8421.3   8103.9 - different local optima, neither one "wrong"

inertia: measure of how tightly grouped the data points are within the clusters

Notice that you get different inertia values. KMeans does not find the globally best grouping - it cannot. Instead the algorithm finds a local optimum based on randomly chosen initial centroids and two runs on the same data can converge to genuinely different groupings. In fact scikit-learn's user guide states "Given enough time, K-means will always converge, however this may be to a local minimum. This is highly dependent on the initialization of the centroids". As mitigation, the default behavior is to run the algorithm several times with different centroid seeds (n_init) and retain the run that scored the lowest on inertia - it does not guarantee a single answer.

What if you change to use a different method - say hierarchical clustering with average linkage on the same RFM dataset? In hierarchical clustering, you do not need to specify the number of clusters and hence, the group counts that appear "natural" in your dataset also shifts. This is because hierarchical clustering with average linkage and KMeans are optimizing on different objectives ( nested merge vs compact spherical clusters )

This dissonance between run-to-run, method-to-method is well studied under clustering stability - a collection of research whose overview by von Luxburg (2010) shows how sensitive clustering results are to the seed, sample, algorithm and why stability by itself is not sufficient for meaningful clustering.

Aren't there any methods to decide?

One might say: "We are very data-driven and we use the elbow method to pick k."

There are various metrics or methods to measure how well your clustering algorithm groups the dataset:

elbow (inertia vs k)
average silhouette method (silhouette score vs k)
gap statistic

Side-by-side comparisons of various methods on same dataset often leads to contradictory recommendations on cluster count. There is no one-metric and final decision requires human judgement based on your domain knowledge.

At times, you might have another situation:

from sklearn.metrics import silhouette_score

for k in range(2, 7):
    labels = KMeans(n_clusters=k, n_init=10, random_state=0).fit_predict(X_scaled)
    print(k, silhouette_score(X_scaled, labels))

# 2  0.41   <- highest score
# 3  0.33
# 4  0.29   <- the k the business narrative was built around
# 5  0.24
# 6  0.22

Here, the silhouette score shows k=2 is highest but you get push back from Marketing - "We need 4 separate personas to target!". Nothing wrong with that request, 2 clusters will usually lead to "Loyal" vs "Non-Loyal" segments and it is meaningless for marketing planning. 4 clusters do help in the business narrative to target specific market segments. If this is the case, a disclaimer is in order that k=4 was due to business narrative (instead of "insights from dataset" as authority)

Cluster on demographics may not mean different customer behavior

Your clustering tells you have four groups of customers and they look different based on the features you cluster on - recency, frequency, spend. Note that their behaviors that businesses care about - who leaves for a competitor, who upgrades to higher tier, who responds to coupons - may look different across those same groups. This discrepancy is what marketing teams struggle with using demographic and behavioral personas - the behavioral patterns often cut across various demographic personas.

The fix is not to have a better clustering metric but to validate externally based on your business objectives and goals (churn in the next quarter, response rate to campaign, etc) and check if they differ between clusters more than within the clusters.

What is the correct approach to clustering?

The above does not mean clustering is a useless technique to avoid. Instead what it means is that clustering is only a starting hypothesis for further investigations.

Before presenting the clustering results:

re-run with different seeds and subset of data
- this is to verify the stability of the cluster
check with multiple metrics and methods (elbow, silhouette, gap statistics)
- highlight in report when they disagree
validate externally
- select a business outcome
- confirm it varies across clusters more than within cluster
treat the chosen number of clusters as modeling decision

A clustering that complies with all four does not mean it is "true" but it does earn the right to be acted upon.

References

Kleinberg, J. (2002). An Impossibility Theorem for Clustering. NeurIPS.
von Luxburg, U. (2010). Clustering Stability: An Overview. arXiv.
scikit-learn. Clustering - K-means, User Guide - local-minimum convergence and multiple-initialization behavior.
scikit-learn. sklearn.cluster.KMeans API reference - n_init and random_state parameter definitions.
Datanovia. Optimal Number of Clusters: Elbow, Silhouette, Gap Statistic.
Ahmed, M. et al. (2022). K-Means Clustering Approach for Intelligent Customer Segmentation Using Customer Purchase Behavior Data. Sustainability, 14(12).
MarTech. Why behavioral segmentation beats personas for real personalization.

SophiArch's Unsupervised Learning & Clustering course covers k-means, hierarchical clustering, and validation together for this reason - an algorithm that returns groups is the easy half; judging whether those groups mean anything to the business is the half that determines whether the analysis survives contact with a skeptical stakeholder.

You Don't Need a Math PhD for Data Science — You Need to Stop Skipping the Boring Step

Jason Lau — Wed, 12 Aug 2026 03:08:07 +0000

Ask someone switching into data science what worries them and it's almost always the same thing: do I know enough math? So they spend weeks on gradient descent derivations and eigenvalues before they've ever wrangled a messy CSV. Then the job starts, a real dataset lands on their desk, and it turns out the model was never the hard part.

The hard part is everything upstream of it. Framing a vague question precisely enough to answer it. Checking whether the data can actually support that answer. Noticing the assumption that breaks silently instead of throwing an error. None of that needs research-level math — most of it doesn't need statistics at all, just the discipline to check before trusting. It needs the willingness to slow down exactly when it's tempting to jump to the model, because the model is the part that feels like progress.

Three moments from a fairly ordinary analytics project show what that looks like in practice

The question shows up vague, and STAYS vague
A stakeholder asks "are our customers happy?" — which is a request for a feeling, not something a query can return. Happy compared to what: last quarter, a competitor, what they expected at signup? Which customers — everyone, or just the ones active in the last 90 days? Measured by what — a survey score, a churn rate, or support-ticket volume, which could reflect a product bug as easily as unhappiness? The junior move is to pick a metric and start coding. The useful move is rewriting the question until two different analysts would agree on exactly what to compute — same population, same comparison point, same metric — even if their pulls differ by a rounding error. Nobody drills that rewrite explicitly; it doesn't sit next to "regression" and "clustering" in a syllabus, but it decides everything that follows it.
Somewhere in the pipeline, rows disappear and NOBODY notices
Say the next step is joining an orders table to a customers table:

merged = orders.merge(customers, on="customer_id")
print(len(orders), len(merged))   # 48,201  →  44,987

Three thousand orders just vanished — every one with a customer_id that didn't find a match on the other side. They might be test accounts. They might be guest checkouts. They might be exactly the segment the analysis was supposed to describe, quietly deleted by an inner join before anyone got to look at them. merge() doesn't warn either way; a silently biased dataset and a clean one produce the exact same "no errors" console output. The only defense is checking row counts before and after every join and being able to explain any drop.

Data type problems hide the same way. A column named signup_date sounds unambiguous right up until .dtype returns object instead of datetime64, and a sample of five values turns up '2024-01-15', '01/15/2024', '2024-15-01', and 'Jan 15 2024' sitting in the same column — three years of manual entry, a spreadsheet import, and an API migration, never reconciled. 01/15/2024 alone is ambiguous between January 15th and an invalid 15th month, depending on which locale wrote it. Parse that carelessly and some rows land on the wrong date, quietly skewing any monthly bucket or tenure calculation downstream. Catching it costs about five minutes: check .dtype, then eyeball a random sample of raw values before trusting what the column name implies. Skipping it costs a wrong number that someone else finds later.

A join doesn't just drop rows — it can multiply them, and a plausible-looking total hides it

Say the next step is tagging orders with any promo code they used, to break out revenue by month:

revenue_by_month = orders.merge(
  promotions,on="order_id"
).groupby("month")["revenue"].sum()
# reported total: $1,284,000 — actual total: $1,061,000

If even a handful of orders matched two promotion rows instead of one — a coupon logged twice, a duplicate entry from a batch re-run — the merge doesn't collapse them, it repeats that order's revenue row once per match. The total comes out $223,000 too high, and nothing about the output looks wrong: no error, no null, just a bigger number than reality. The fix costs nothing statistical — it's checking that the row count and the total after a join still match a number you trust from somewhere else, before either one gets reported.

None of these three moments required anything past intro-level statistics — most required none at all. Each one involved noticing something easy to miss and choosing to stop and check it rather than push on to the modeling step, which is the part that actually feels like work. That's closer to the real shape of an entry-level data science job than most course outlines suggest — the algorithms are documented, implemented, and one import away in scikit-learn; the dropped rows, the ambiguous date format, and the inflated total are not, and they're what a hiring manager is actually screening for whether or not the posting says so.

For anyone building toward a first role, that argues for a different practice list than "learn another algorithm":

Rewrite a vague business question into one with an unambiguous metric and comparison, before writing any code
Check row counts before and after every join or filter, and be able to explain any change
Inspect column dtypes and a sample of raw values before trusting what a column's name implies
After any join, check that both the row count and a key total still match a number you trust from elsewhere
Practice stating a finding's limitations in the same breath as the finding itself

None of it is glamorous, and none of it shows up as a line on a certificate. It's also the difference between an analysis that survives a second question from a skeptical stakeholder and one that quietly comes apart the first time someone re-runs it on next month's data.

SophiArch's Intro to Data Science course is built around exactly this — not another pass through the algorithm list, but deliberate practice framing questions, auditing data, and catching this class of mistake before it reaches someone else's desk.

The Most Dangerous Phrase in Analytics Is "The Data Speaks for Itself"

Jason Lau — Wed, 05 Aug 2026 08:02:13 +0000

The data speaks for itself is usually offered as a defense. It means: I didn't put my thumb on the scale, I just showed you what's there. It sounds like humility. In practice it's the opposite — it's a claim that the analyst's choices aren't part of the result, when they always are.

Data never speaks. Someone chooses the time window, the bucket size, the baseline, which outliers stay in, and which comparison gets drawn on the same axis. Those choices are what turn raw numbers into a chart anyone can read in five seconds. The phrase the data speaks for itself is what lets those choices pass without anyone — including the analyst — examining them.

Here's what that looks like in practice, and why naming the choice out loud is the actual discipline.

1. Truncated axis

ax.set_ylim(80, 100)   # "to make the trend visible"

A bar chart of conversion rate moving from 82% to 86% looks like a rounding error on a 0–100 axis. Truncate the y-axis to 80–100 and the same bars look like a dramatic surge. Neither chart is fabricated — every number on it is correct. The difference is entirely in a choice that's invisible unless someone asks what the axis range is and why.

Truncated axes aren't always wrong — sometimes the full 0–100 range genuinely hides a real, meaningful shift. The point isn't "never truncate." It's that the decision to truncate is doing real interpretive work, and "the data speaks for itself" is exactly the phrase that lets it go unstated.

2. Baseline nobody picked on purpose

df["growth"] = (df["revenue"] - df["revenue"].shift(12)) / df["revenue"].shift(12)

Year-over-year growth looks like a neutral, standard metric. It's a choice: this quarter compared to the same quarter last year, not last quarter, not a trailing average, not the pre-pandemic baseline. If last year had an unusual dip, this year's "growth" is partly an artifact of a low starting point — and the chart won't tell you that, because a percentage doesn't carry its own denominator's history along with it.

3. Outliers that got dropped "for clarity"

df = df[df["order_value"] < df["order_value"].quantile(0.99)]   # cuts the long tail

Trimming the top 1% before plotting a distribution is often reasonable — a few extreme values can flatten a histogram into something unreadable. But it's a modeling decision about what counts as signal versus noise, made silently, before the reader ever sees the shape of the full data. If those extreme orders are disproportionately from one customer segment, "clarity" just deleted the most important 1% of the story.

4. Comparison that implies causation without claiming it

"Users who completed onboarding had 3x higher retention."

Looking at the chart on the left, every word of this is true and it will be read as onboarding causes retention by nearly everyone who sees it.

What is left unmentioned are possible confounders, like for instance as shown on the right, the people who complete onboarding are typically also the people motivated enough to complete onboarding — and hence we can never establish for sure that the training caused anything.

The chart didn't lie. The framing did the arguing that implies causation when there may be none.

5. p-value treated as a verdict

stats.ttest_ind(group_a, group_b)   # p = 0.03 → "significant" → done

A p-value answers one narrow question — if there were actually no difference between these two groups, how likely would it be to see a difference this big just from random luck? A p-value of 0.03 means "about a 3% chance of seeing this by luck alone if nothing real is going on."

It says nothing about how big the difference is, whether it matters in practice, or whether the study was well-designed. It's a statement about coincidence, not about importance — and the 0.05 cutoff itself is just a convention, not a law of nature.

However, a p-value below 0.05 typically gets treated as an on/off switch: significant, ship it. When in reality, p-value is actually a statement about one specific null hypothesis, under one specific test, on one specific sample — not a verdict on whether the effect is real, large, or durable.

Reporting "p < 0.05" without reporting effect size, sample size, or how many other comparisons were run alongside this one lets a single threshold stand in for a judgment call about how much evidence is enough.

Take the example below of four studies (A, B, C, D) with roughly similar p-values (all < 0.05) but different effect sizes, measured using Cohen's d. Effect size is just "how big is the difference, really" — the gap between the two groups, scaled so it's comparable across studies. A small effect size means the two groups barely differ even if the math calls it "significant"; a large effect size means the difference is big enough to see with the naked eye.

The four studies in the chart above all clear the same p-value < 0.05 bar, but they're not telling the same story: Study D ran on 8,000 people and found a difference so tiny it's practically nothing — with a sample that large, even noise can become "significant." Study A ran on just 40 people and found a difference big enough to matter. Same headline, opposite substance.

That's why the p-value alone can't be the verdict — it only says a difference probably isn't zero, not that the difference is big enough to care about.

Why "just double-check the numbers" isn't the fix

Every example above involves numbers that are individually correct. Fact-checking the arithmetic won't catch any of them, because the distortion isn't in the math — it's in which math got chosen, and what got left off the chart. That's a different kind of review: not "is this number right," but "what would this look like if the axis, baseline, or trimming had been chosen differently, and why wasn't that shown?"

A defensible analysis — the kind that survives a skeptical stakeholder asking "wait, why does it look like that?" — means being able to answer yes to all of:

The axis range and why it was chosen would survive being stated out loud
The baseline or comparison point is named, not just implied by the metric's label
Any trimming, filtering, or exclusion is disclosed next to the result it affects
Causal language is reserved for claims that actually support causation
A significance result is reported with effect size and sample size, not p-value alone

None of this is about being more honest in some abstract sense. It's about treating every visualization and every summary statistic as an argument with premises — because that's what it is — and being willing to state the premises instead of letting "the data speaks for itself" stand in for them.

This is the discipline behind SophiArch's Statistics for Data Science course, which teaches distributions, hypothesis testing, and inference specifically through the lens of directing and auditing AI-generated analysis — including catching exactly this kind of framing choice before it reaches a stakeholder.

The 3 Kinds of Data Leakage That Make Your Model Lie to You

Jason Lau — Wed, 29 Jul 2026 03:16:55 +0000

You build a churn model. Cross-validation AUC is 0.91. You ship it. Real-world AUC is 0.76.

Nothing changed — same data, same algorithm, same infrastructure. The model just doesn't work as well as you measured it would. If this has happened to you, the most likely cause isn't a bad algorithm choice. It's data leakage: information from outside the training boundary reaching the model during development, producing an evaluation score that collapses the moment it has to generalize to genuinely unseen data.

Leakage is dangerous precisely because it's silent. The code runs. The numbers look good. Nothing throws an exception. The bug only shows up in production, weeks later, as a vague "the model's not performing like we expected" conversation.

There are three distinct patterns, and they are not equally easy to catch — which matters more than ever now that a growing share of preprocessing code is AI-generated or AI-assisted.

1. Fit-on-all: the one AI tools actually catch

This is the classic version, and it's mechanical enough that both static analysis and a decent code-review prompt can flag it: a transformer (scaler, imputer, encoder) gets fit on the entire dataset before the train/test split. The transformer has now learned statistics — mean, variance, category frequencies — from data that's supposed to be unseen. When you evaluate on the "test" set, it's been preprocessed using information from itself.

Here's the effect, measured directly rather than described:

import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

df = pd.read_csv("transactions_merged_features.csv")
X = df[["monthly_spend", "support_tickets", "tenure_days", "plan_encoded"]]
y = df["churned"]

# --- The leaked version ---
scaler = StandardScaler()
X_scaled_all = scaler.fit_transform(X)          # fit on ALL data, including test

X_train_l, X_test_l, y_train, y_test = train_test_split(
    X_scaled_all, y, test_size=0.2, random_state=42
)
model = LogisticRegression().fit(X_train_l, y_train)
auc_leaked = roc_auc_score(y_test, model.predict_proba(X_test_l)[:, 1])
print(f"AUC (leaked):  {auc_leaked:.3f}")   # 0.912

# --- The honest version ---
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

scaler_honest = StandardScaler()
X_train_scaled = scaler_honest.fit_transform(X_train)   # fit on train only
X_test_scaled  = scaler_honest.transform(X_test)         # transform only

model_honest = LogisticRegression().fit(X_train_scaled, y_train)
auc_honest = roc_auc_score(y_test, model_honest.predict_proba(X_test_scaled)[:, 1])
print(f"AUC (honest):  {auc_honest:.3f}")   # 0.841

Same data, same model, same random seed. AUC drops from 0.912 to 0.841 — a 7-point gap caused entirely by fitting the scaler before the split. At a business level, that's the difference between a model you'd confidently deploy and one you'd send back for more work.

The reason AI coding assistants can catch this one: it's a pattern-matchable rule — "does fit or fit_transform appear before train_test_split in the same variable's lineage?" Ask any capable AI assistant to review a preprocessing script for this specific issue and it will generally find it, because it's a syntactic property of the code, not a fact about your business.

2. Target leakage: the one that requires knowing your business, not your syntax

This is where AI code review quietly stops being useful, because target leakage isn't a code mistake — it's a feature design problem. The code is syntactically fine. The leakage is in what the column means.

# Feature audit — ask for each column: would I have this at prediction time?
feature_audit = {
    "tenure_days":          "YES — calculated from signup_date, known at any time",
    "monthly_spend":        "YES — last billing cycle is known",
    "support_tickets":      "YES — CRM records exist at prediction time",
    "plan_encoded":         "YES — current plan is known",
    "days_to_first_ticket": "NO — some customers haven't opened a ticket yet",
    "cancellation_flag":    "NO — only exists for customers already mid-cancellation",
}

cancellation_flag is the textbook case. The model "predicts" churn by noticing that churned customers have a cancellation flag set. That's not a prediction, it's a tautology — and it will produce a beautiful AUC in development, because the leaked feature is, definitionally, almost perfectly correlated with the label.

An AI assistant reviewing this code has no way to know that cancellation_flag is only populated after a customer has already initiated cancellation, unless that constraint is written down somewhere it can read — a schema comment, a data dictionary, a docstring. Column names that are suggestive (churn_date, days_since_cancellation) sometimes get flagged. Column names that are business-specific and non-obvious (at_risk_flag, populated by a CS rep's note that may or may not predate the churn event) will not.

3. Temporal leakage: the one that hides in aggregation windows

The third pattern is the hardest to spot in a code diff, because the leaking value is often computed correctly in isolation — the leakage is in the time window the aggregation covers relative to the prediction point.

Example: a 12_month_avg_spend feature used to predict whether a customer churned in month 3. The average is computed over the full 12 months — which includes months 4 through 12, all of which happened after the prediction point. The feature is real, the arithmetic is correct, and it's still leakage, because at the moment you'd actually need this prediction (month 3, for a live customer), months 4–12 haven't happened yet.

This is the one that survives code review most often, because there's no syntax cue at all — just a groupby().mean() that looks completely ordinary. Catching it requires drawing the prediction-point boundary explicitly and checking every feature against it:

Every feature derived from data on the wrong side of the prediction point — regardless of how predictive it looks in training — is leakage.

The one question that catches all three

For every feature, ask: at the moment I make this prediction for a live customer, would this value already exist?

If the answer is "no," or "only sometimes," the feature needs to be dropped or rebuilt using only historically-available data. This question doesn't require a tool — it requires knowing your data's timeline, which is exactly the part an AI assistant wasn't in the room for.

The structural fix for at least one of these

Fit-on-all leakage — pattern #1 — has an actual structural fix, not just a discipline-based one: sklearn.pipeline.Pipeline. Wrapping your scaler, imputer, and model in a single Pipeline makes it impossible to call fit_transform on the full dataset by accident, because the pipeline controls the fit/transform order for you. It doesn't help with target or temporal leakage — those require the feature audit above — but it closes off the one failure mode that's purely mechanical.

If you want the full walkthrough — building the Pipeline, composing ColumnTransformers for mixed-type data, and a systematic protocol for auditing AI-generated preprocessing code specifically — it's the subject of SophiArch's Feature Engineering & Pipelines course, including the lesson this article's timeline diagram and code examples are drawn from.

Your AI's EDA Looks Right. Here's How to Tell When It Isn't.

Jason Lau — Wed, 22 Jul 2026 02:51:22 +0000

Ask an AI coding assistant to run exploratory data analysis on a new dataset and it will produce, in seconds, something that looks like a competent analyst's first pass: .describe(), a correlation matrix, some distribution plots, a groupby or two, and a paragraph of narrative summarizing what it found.

The problem is not that this output is wrong. Often it's correct. The problem is that it looks identical whether it's correct or not — and the failure modes that make it wrong are exactly the ones a quick skim won't catch, because nothing throws an error and nothing looks incomplete.

Here are six patterns that show up constantly in AI-generated EDA, each one silently producing output that reads as clean.

1. Profiling before type conversion

# The AI commonly generates this order — the problem is invisible in the output
print(df.describe())                                    # order_date excluded — it's still a string
df["order_date"] = pd.to_datetime(df["order_date"])     # too late

.describe() silently drops non-numeric columns. If a date column is still sitting as a string object when you profile, it just... isn't in the summary. Nothing errors. You get a clean-looking table with one fewer column than you meant to check, and no indication anything was skipped.

The fix is ordering, not code complexity: type conversions belong in the very first block, before any statistic is computed.

2. Reaching for `.mean()` on a right-skewed column

AI assistants default to .mean() for basically every "typical value" question, because it's the statistically obvious first move. For revenue, spend, session duration — anything with a long right tail — that default quietly overstates what's typical:

df.groupby("channel")["revenue"].mean()   # a handful of large B2B orders pull this up

A groupby that reports mean without also reporting median, count, and std produces a number that's technically correct and practically misleading. The fix is cheap — always pull all four together — but it requires knowing to ask, since the AI won't flag its own choice of statistic as a judgment call.

3. `.dropna()` before you've profiled the missingness

df = df.dropna()          # applied early, "to make analysis cleaner"
# profiling happens on what's left

Profiling exists to characterize how and where values are missing — whether it's random, or concentrated in a specific segment, or correlated with another column. Drop the rows first and that pattern is gone before you ever looked at it. The output afterward looks complete. It's complete for a dataset that no longer represents the one you started with.

4. The silent subset filter

This is the one most likely to change your conclusions without you noticing:

df = df[df["status"] == "delivered"]     # ← added mid-script, nothing downstream is labeled

Every correlation, every groupby, every chart that runs after this line applies to delivered orders only — but nothing in the output says so. A stakeholder reading "42% of orders come from the enterprise channel" has no way to know that's 42% of delivered orders, unless someone happened to scroll up to a filter line with no comment attached to it.

5. Narrative summaries that quietly assert causation

AI-generated EDA often ends with a paragraph of plain-language commentary. That commentary is where causal language creeps in — "unit price drives revenue," "the strong correlation confirms X" — from analysis that only ever established correlation. The numbers above the paragraph are usually fine. The prose interpreting them is where scope creep happens, because narrative generation and statistical rigor are not the same task, even when the same model produces both in one response.

Read the numbers. Treat the narrative paragraph as a first draft to revise, not a finding to cite.

6. Treating "no issues found" as "the data is clean"

If you ask an AI to check for data quality issues and it reports none, that's not confirmation. It means the checks it ran didn't find anything — which says nothing about the checks it didn't run. Cardinality on string columns, structural missingness patterns, multivariate outliers: these don't show up unless something specifically goes looking for them.

All six patterns share the same shape: the output looks complete, and completeness is exactly what a quick review checks for.

Why "just review the code" isn't quite the answer

Reviewing AI-generated code before running it catches some of this — type conversions before statistics, mean vs. median, subset filters — because those are visible in the script itself. What code review alone won't catch is #5 and #6: causal language buried in a narrative summary, and the false confidence of a clean-looking "no issues found." Those require reading the output, not just the code, and holding it against what you already know about the domain.

A defensible EDA — AI-assisted or not — means you can answer yes to all of:

Type conversions happened before any statistic was computed
Missingness was profiled before anything was dropped
Every group summary includes count and spread, not just mean
No subset filter changed scope without being documented
Causal language has been replaced with correlational language where causation wasn't established
"No findings" was treated as "the checks I ran found nothing," not as "the data is clean"

None of this is really about AI. It's the same discipline that's always separated a defensible analysis from a plausible-looking one. What's changed is the speed at which plausible-looking output gets produced — which means the checklist matters more, not less.

This is the framework behind SophiArch's Exploratory Data Analysis course, including a lab where you're handed a full AI-generated EDA notebook and asked to find exactly these kinds of issues before they reach a stakeholder.

DEV Community: Jason Lau

Your Agent Has Observability. It Doesn't Have Evals.

Two traces

The gap has a number attached to it

Why the dashboard structurally cannot tell you

What measuring the outcome actually looks like

"We'll just add an LLM judge"

Turning the traces you already have into the evals you don't

The one-line version

References

Text-to-SQL Accuracy Isn't a Model Problem. It's a Schema Problem.

Two queries, one question, two answers

The spread that looks like disagreement

What "the schema needs modelling" actually means

"Just model your warehouse" is easy advice to give

The checklist that transfers to your warehouse

References

Long Context Didn't Kill RAG. It Killed Lazy RAG

The strongest case for "just put it in the window"

What it doesn't buy you, and where the "26x" goes

Three questions that settle it for your corpus

"Better retrieval" isn't a vibe, it's a short and specific list

References

Prompt Engineering Won't Survive — the Skill Underneath It Will

What actually died was a set of tricks tuned to specific models

The instability was already visible in how fragile the tricks were

What was actually being taught the whole time

References

Your RAG Pipeline Doesn't Have an Accuracy Problem - It Has an Evaluation Problem

A wrong answer and a right answer look identical

"RAG fixes hallucinations" is marketing, and there's a measurement to prove it

"Is it accurate?" is two questions wearing one trenchcoat

What an actual evaluation harness looks like

References

Clustering Isn't Discovery - It's a Hypothesis You Still Have to Test

Same data, different "discovery"

Aren't there any methods to decide?

Cluster on demographics may not mean different customer behavior

What is the correct approach to clustering?

References

You Don't Need a Math PhD for Data Science — You Need to Stop Skipping the Boring Step

The Most Dangerous Phrase in Analytics Is "The Data Speaks for Itself"

1. Truncated axis

2. Baseline nobody picked on purpose

3. Outliers that got dropped "for clarity"

4. Comparison that implies causation without claiming it

5. p-value treated as a verdict

Why "just double-check the numbers" isn't the fix

The 3 Kinds of Data Leakage That Make Your Model Lie to You

1. Fit-on-all: the one AI tools actually catch

2. Target leakage: the one that requires knowing your business, not your syntax

3. Temporal leakage: the one that hides in aggregation windows

The one question that catches all three

The structural fix for at least one of these

Your AI's EDA Looks Right. Here's How to Tell When It Isn't.

1. Profiling before type conversion

2. Reaching for .mean() on a right-skewed column

3. .dropna() before you've profiled the missingness

4. The silent subset filter

5. Narrative summaries that quietly assert causation

6. Treating "no issues found" as "the data is clean"

Why "just review the code" isn't quite the answer

2. Reaching for `.mean()` on a right-skewed column

3. `.dropna()` before you've profiled the missingness