DEV Community: Jason Lau

You Don't Need a Math PhD for Data Science — You Need to Stop Skipping the Boring Step

Jason Lau — Wed, 12 Aug 2026 03:08:07 +0000

Ask someone switching into data science what worries them and it's almost always the same thing: do I know enough math? So they spend weeks on gradient descent derivations and eigenvalues before they've ever wrangled a messy CSV. Then the job starts, a real dataset lands on their desk, and it turns out the model was never the hard part.

The hard part is everything upstream of it. Framing a vague question precisely enough to answer it. Checking whether the data can actually support that answer. Noticing the assumption that breaks silently instead of throwing an error. None of that needs research-level math — most of it doesn't need statistics at all, just the discipline to check before trusting. It needs the willingness to slow down exactly when it's tempting to jump to the model, because the model is the part that feels like progress.

Three moments from a fairly ordinary analytics project show what that looks like in practice

The question shows up vague, and STAYS vague
A stakeholder asks "are our customers happy?" — which is a request for a feeling, not something a query can return. Happy compared to what: last quarter, a competitor, what they expected at signup? Which customers — everyone, or just the ones active in the last 90 days? Measured by what — a survey score, a churn rate, or support-ticket volume, which could reflect a product bug as easily as unhappiness? The junior move is to pick a metric and start coding. The useful move is rewriting the question until two different analysts would agree on exactly what to compute — same population, same comparison point, same metric — even if their pulls differ by a rounding error. Nobody drills that rewrite explicitly; it doesn't sit next to "regression" and "clustering" in a syllabus, but it decides everything that follows it.
Somewhere in the pipeline, rows disappear and NOBODY notices
Say the next step is joining an orders table to a customers table:

merged = orders.merge(customers, on="customer_id")
print(len(orders), len(merged))   # 48,201  →  44,987

Three thousand orders just vanished — every one with a customer_id that didn't find a match on the other side. They might be test accounts. They might be guest checkouts. They might be exactly the segment the analysis was supposed to describe, quietly deleted by an inner join before anyone got to look at them. merge() doesn't warn either way; a silently biased dataset and a clean one produce the exact same "no errors" console output. The only defense is checking row counts before and after every join and being able to explain any drop.

Data type problems hide the same way. A column named signup_date sounds unambiguous right up until .dtype returns object instead of datetime64, and a sample of five values turns up '2024-01-15', '01/15/2024', '2024-15-01', and 'Jan 15 2024' sitting in the same column — three years of manual entry, a spreadsheet import, and an API migration, never reconciled. 01/15/2024 alone is ambiguous between January 15th and an invalid 15th month, depending on which locale wrote it. Parse that carelessly and some rows land on the wrong date, quietly skewing any monthly bucket or tenure calculation downstream. Catching it costs about five minutes: check .dtype, then eyeball a random sample of raw values before trusting what the column name implies. Skipping it costs a wrong number that someone else finds later.

A join doesn't just drop rows — it can multiply them, and a plausible-looking total hides it

Say the next step is tagging orders with any promo code they used, to break out revenue by month:

revenue_by_month = orders.merge(
  promotions,on="order_id"
).groupby("month")["revenue"].sum()
# reported total: $1,284,000 — actual total: $1,061,000

If even a handful of orders matched two promotion rows instead of one — a coupon logged twice, a duplicate entry from a batch re-run — the merge doesn't collapse them, it repeats that order's revenue row once per match. The total comes out $223,000 too high, and nothing about the output looks wrong: no error, no null, just a bigger number than reality. The fix costs nothing statistical — it's checking that the row count and the total after a join still match a number you trust from somewhere else, before either one gets reported.

None of these three moments required anything past intro-level statistics — most required none at all. Each one involved noticing something easy to miss and choosing to stop and check it rather than push on to the modeling step, which is the part that actually feels like work. That's closer to the real shape of an entry-level data science job than most course outlines suggest — the algorithms are documented, implemented, and one import away in scikit-learn; the dropped rows, the ambiguous date format, and the inflated total are not, and they're what a hiring manager is actually screening for whether or not the posting says so.

For anyone building toward a first role, that argues for a different practice list than "learn another algorithm":

Rewrite a vague business question into one with an unambiguous metric and comparison, before writing any code
Check row counts before and after every join or filter, and be able to explain any change
Inspect column dtypes and a sample of raw values before trusting what a column's name implies
After any join, check that both the row count and a key total still match a number you trust from elsewhere
Practice stating a finding's limitations in the same breath as the finding itself

None of it is glamorous, and none of it shows up as a line on a certificate. It's also the difference between an analysis that survives a second question from a skeptical stakeholder and one that quietly comes apart the first time someone re-runs it on next month's data.

SophiArch's Intro to Data Science course is built around exactly this — not another pass through the algorithm list, but deliberate practice framing questions, auditing data, and catching this class of mistake before it reaches someone else's desk.

The Most Dangerous Phrase in Analytics Is "The Data Speaks for Itself"

Jason Lau — Wed, 05 Aug 2026 08:02:13 +0000

The data speaks for itself is usually offered as a defense. It means: I didn't put my thumb on the scale, I just showed you what's there. It sounds like humility. In practice it's the opposite — it's a claim that the analyst's choices aren't part of the result, when they always are.

Data never speaks. Someone chooses the time window, the bucket size, the baseline, which outliers stay in, and which comparison gets drawn on the same axis. Those choices are what turn raw numbers into a chart anyone can read in five seconds. The phrase the data speaks for itself is what lets those choices pass without anyone — including the analyst — examining them.

Here's what that looks like in practice, and why naming the choice out loud is the actual discipline.

1. Truncated axis

ax.set_ylim(80, 100)   # "to make the trend visible"

A bar chart of conversion rate moving from 82% to 86% looks like a rounding error on a 0–100 axis. Truncate the y-axis to 80–100 and the same bars look like a dramatic surge. Neither chart is fabricated — every number on it is correct. The difference is entirely in a choice that's invisible unless someone asks what the axis range is and why.

Truncated axes aren't always wrong — sometimes the full 0–100 range genuinely hides a real, meaningful shift. The point isn't "never truncate." It's that the decision to truncate is doing real interpretive work, and "the data speaks for itself" is exactly the phrase that lets it go unstated.

2. Baseline nobody picked on purpose

df["growth"] = (df["revenue"] - df["revenue"].shift(12)) / df["revenue"].shift(12)

Year-over-year growth looks like a neutral, standard metric. It's a choice: this quarter compared to the same quarter last year, not last quarter, not a trailing average, not the pre-pandemic baseline. If last year had an unusual dip, this year's "growth" is partly an artifact of a low starting point — and the chart won't tell you that, because a percentage doesn't carry its own denominator's history along with it.

3. Outliers that got dropped "for clarity"

df = df[df["order_value"] < df["order_value"].quantile(0.99)]   # cuts the long tail

Trimming the top 1% before plotting a distribution is often reasonable — a few extreme values can flatten a histogram into something unreadable. But it's a modeling decision about what counts as signal versus noise, made silently, before the reader ever sees the shape of the full data. If those extreme orders are disproportionately from one customer segment, "clarity" just deleted the most important 1% of the story.

4. Comparison that implies causation without claiming it

"Users who completed onboarding had 3x higher retention."

Looking at the chart on the left, every word of this is true and it will be read as onboarding causes retention by nearly everyone who sees it.

What is left unmentioned are possible confounders, like for instance as shown on the right, the people who complete onboarding are typically also the people motivated enough to complete onboarding — and hence we can never establish for sure that the training caused anything.

The chart didn't lie. The framing did the arguing that implies causation when there may be none.

5. p-value treated as a verdict

stats.ttest_ind(group_a, group_b)   # p = 0.03 → "significant" → done

A p-value answers one narrow question — if there were actually no difference between these two groups, how likely would it be to see a difference this big just from random luck? A p-value of 0.03 means "about a 3% chance of seeing this by luck alone if nothing real is going on."

It says nothing about how big the difference is, whether it matters in practice, or whether the study was well-designed. It's a statement about coincidence, not about importance — and the 0.05 cutoff itself is just a convention, not a law of nature.

However, a p-value below 0.05 typically gets treated as an on/off switch: significant, ship it. When in reality, p-value is actually a statement about one specific null hypothesis, under one specific test, on one specific sample — not a verdict on whether the effect is real, large, or durable.

Reporting "p < 0.05" without reporting effect size, sample size, or how many other comparisons were run alongside this one lets a single threshold stand in for a judgment call about how much evidence is enough.

Take the example below of four studies (A, B, C, D) with roughly similar p-values (all < 0.05) but different effect sizes, measured using Cohen's d. Effect size is just "how big is the difference, really" — the gap between the two groups, scaled so it's comparable across studies. A small effect size means the two groups barely differ even if the math calls it "significant"; a large effect size means the difference is big enough to see with the naked eye.

The four studies in the chart above all clear the same p-value < 0.05 bar, but they're not telling the same story: Study D ran on 8,000 people and found a difference so tiny it's practically nothing — with a sample that large, even noise can become "significant." Study A ran on just 40 people and found a difference big enough to matter. Same headline, opposite substance.

That's why the p-value alone can't be the verdict — it only says a difference probably isn't zero, not that the difference is big enough to care about.

Why "just double-check the numbers" isn't the fix

Every example above involves numbers that are individually correct. Fact-checking the arithmetic won't catch any of them, because the distortion isn't in the math — it's in which math got chosen, and what got left off the chart. That's a different kind of review: not "is this number right," but "what would this look like if the axis, baseline, or trimming had been chosen differently, and why wasn't that shown?"

A defensible analysis — the kind that survives a skeptical stakeholder asking "wait, why does it look like that?" — means being able to answer yes to all of:

The axis range and why it was chosen would survive being stated out loud
The baseline or comparison point is named, not just implied by the metric's label
Any trimming, filtering, or exclusion is disclosed next to the result it affects
Causal language is reserved for claims that actually support causation
A significance result is reported with effect size and sample size, not p-value alone

None of this is about being more honest in some abstract sense. It's about treating every visualization and every summary statistic as an argument with premises — because that's what it is — and being willing to state the premises instead of letting "the data speaks for itself" stand in for them.

This is the discipline behind SophiArch's Statistics for Data Science course, which teaches distributions, hypothesis testing, and inference specifically through the lens of directing and auditing AI-generated analysis — including catching exactly this kind of framing choice before it reaches a stakeholder.

The 3 Kinds of Data Leakage That Make Your Model Lie to You

Jason Lau — Wed, 29 Jul 2026 03:16:55 +0000

You build a churn model. Cross-validation AUC is 0.91. You ship it. Real-world AUC is 0.76.

Nothing changed — same data, same algorithm, same infrastructure. The model just doesn't work as well as you measured it would. If this has happened to you, the most likely cause isn't a bad algorithm choice. It's data leakage: information from outside the training boundary reaching the model during development, producing an evaluation score that collapses the moment it has to generalize to genuinely unseen data.

Leakage is dangerous precisely because it's silent. The code runs. The numbers look good. Nothing throws an exception. The bug only shows up in production, weeks later, as a vague "the model's not performing like we expected" conversation.

There are three distinct patterns, and they are not equally easy to catch — which matters more than ever now that a growing share of preprocessing code is AI-generated or AI-assisted.

1. Fit-on-all: the one AI tools actually catch

This is the classic version, and it's mechanical enough that both static analysis and a decent code-review prompt can flag it: a transformer (scaler, imputer, encoder) gets fit on the entire dataset before the train/test split. The transformer has now learned statistics — mean, variance, category frequencies — from data that's supposed to be unseen. When you evaluate on the "test" set, it's been preprocessed using information from itself.

Here's the effect, measured directly rather than described:

import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

df = pd.read_csv("transactions_merged_features.csv")
X = df[["monthly_spend", "support_tickets", "tenure_days", "plan_encoded"]]
y = df["churned"]

# --- The leaked version ---
scaler = StandardScaler()
X_scaled_all = scaler.fit_transform(X)          # fit on ALL data, including test

X_train_l, X_test_l, y_train, y_test = train_test_split(
    X_scaled_all, y, test_size=0.2, random_state=42
)
model = LogisticRegression().fit(X_train_l, y_train)
auc_leaked = roc_auc_score(y_test, model.predict_proba(X_test_l)[:, 1])
print(f"AUC (leaked):  {auc_leaked:.3f}")   # 0.912

# --- The honest version ---
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

scaler_honest = StandardScaler()
X_train_scaled = scaler_honest.fit_transform(X_train)   # fit on train only
X_test_scaled  = scaler_honest.transform(X_test)         # transform only

model_honest = LogisticRegression().fit(X_train_scaled, y_train)
auc_honest = roc_auc_score(y_test, model_honest.predict_proba(X_test_scaled)[:, 1])
print(f"AUC (honest):  {auc_honest:.3f}")   # 0.841

Same data, same model, same random seed. AUC drops from 0.912 to 0.841 — a 7-point gap caused entirely by fitting the scaler before the split. At a business level, that's the difference between a model you'd confidently deploy and one you'd send back for more work.

The reason AI coding assistants can catch this one: it's a pattern-matchable rule — "does fit or fit_transform appear before train_test_split in the same variable's lineage?" Ask any capable AI assistant to review a preprocessing script for this specific issue and it will generally find it, because it's a syntactic property of the code, not a fact about your business.

2. Target leakage: the one that requires knowing your business, not your syntax

This is where AI code review quietly stops being useful, because target leakage isn't a code mistake — it's a feature design problem. The code is syntactically fine. The leakage is in what the column means.

# Feature audit — ask for each column: would I have this at prediction time?
feature_audit = {
    "tenure_days":          "YES — calculated from signup_date, known at any time",
    "monthly_spend":        "YES — last billing cycle is known",
    "support_tickets":      "YES — CRM records exist at prediction time",
    "plan_encoded":         "YES — current plan is known",
    "days_to_first_ticket": "NO — some customers haven't opened a ticket yet",
    "cancellation_flag":    "NO — only exists for customers already mid-cancellation",
}

cancellation_flag is the textbook case. The model "predicts" churn by noticing that churned customers have a cancellation flag set. That's not a prediction, it's a tautology — and it will produce a beautiful AUC in development, because the leaked feature is, definitionally, almost perfectly correlated with the label.

An AI assistant reviewing this code has no way to know that cancellation_flag is only populated after a customer has already initiated cancellation, unless that constraint is written down somewhere it can read — a schema comment, a data dictionary, a docstring. Column names that are suggestive (churn_date, days_since_cancellation) sometimes get flagged. Column names that are business-specific and non-obvious (at_risk_flag, populated by a CS rep's note that may or may not predate the churn event) will not.

3. Temporal leakage: the one that hides in aggregation windows

The third pattern is the hardest to spot in a code diff, because the leaking value is often computed correctly in isolation — the leakage is in the time window the aggregation covers relative to the prediction point.

Example: a 12_month_avg_spend feature used to predict whether a customer churned in month 3. The average is computed over the full 12 months — which includes months 4 through 12, all of which happened after the prediction point. The feature is real, the arithmetic is correct, and it's still leakage, because at the moment you'd actually need this prediction (month 3, for a live customer), months 4–12 haven't happened yet.

This is the one that survives code review most often, because there's no syntax cue at all — just a groupby().mean() that looks completely ordinary. Catching it requires drawing the prediction-point boundary explicitly and checking every feature against it:

Every feature derived from data on the wrong side of the prediction point — regardless of how predictive it looks in training — is leakage.

The one question that catches all three

For every feature, ask: at the moment I make this prediction for a live customer, would this value already exist?

If the answer is "no," or "only sometimes," the feature needs to be dropped or rebuilt using only historically-available data. This question doesn't require a tool — it requires knowing your data's timeline, which is exactly the part an AI assistant wasn't in the room for.

The structural fix for at least one of these

Fit-on-all leakage — pattern #1 — has an actual structural fix, not just a discipline-based one: sklearn.pipeline.Pipeline. Wrapping your scaler, imputer, and model in a single Pipeline makes it impossible to call fit_transform on the full dataset by accident, because the pipeline controls the fit/transform order for you. It doesn't help with target or temporal leakage — those require the feature audit above — but it closes off the one failure mode that's purely mechanical.

If you want the full walkthrough — building the Pipeline, composing ColumnTransformers for mixed-type data, and a systematic protocol for auditing AI-generated preprocessing code specifically — it's the subject of SophiArch's Feature Engineering & Pipelines course, including the lesson this article's timeline diagram and code examples are drawn from.

Your AI's EDA Looks Right. Here's How to Tell When It Isn't.

Jason Lau — Wed, 22 Jul 2026 02:51:22 +0000

Ask an AI coding assistant to run exploratory data analysis on a new dataset and it will produce, in seconds, something that looks like a competent analyst's first pass: .describe(), a correlation matrix, some distribution plots, a groupby or two, and a paragraph of narrative summarizing what it found.

The problem is not that this output is wrong. Often it's correct. The problem is that it looks identical whether it's correct or not — and the failure modes that make it wrong are exactly the ones a quick skim won't catch, because nothing throws an error and nothing looks incomplete.

Here are six patterns that show up constantly in AI-generated EDA, each one silently producing output that reads as clean.

1. Profiling before type conversion

# The AI commonly generates this order — the problem is invisible in the output
print(df.describe())                                    # order_date excluded — it's still a string
df["order_date"] = pd.to_datetime(df["order_date"])     # too late

.describe() silently drops non-numeric columns. If a date column is still sitting as a string object when you profile, it just... isn't in the summary. Nothing errors. You get a clean-looking table with one fewer column than you meant to check, and no indication anything was skipped.

The fix is ordering, not code complexity: type conversions belong in the very first block, before any statistic is computed.

2. Reaching for `.mean()` on a right-skewed column

AI assistants default to .mean() for basically every "typical value" question, because it's the statistically obvious first move. For revenue, spend, session duration — anything with a long right tail — that default quietly overstates what's typical:

df.groupby("channel")["revenue"].mean()   # a handful of large B2B orders pull this up

A groupby that reports mean without also reporting median, count, and std produces a number that's technically correct and practically misleading. The fix is cheap — always pull all four together — but it requires knowing to ask, since the AI won't flag its own choice of statistic as a judgment call.

3. `.dropna()` before you've profiled the missingness

df = df.dropna()          # applied early, "to make analysis cleaner"
# profiling happens on what's left

Profiling exists to characterize how and where values are missing — whether it's random, or concentrated in a specific segment, or correlated with another column. Drop the rows first and that pattern is gone before you ever looked at it. The output afterward looks complete. It's complete for a dataset that no longer represents the one you started with.

4. The silent subset filter

This is the one most likely to change your conclusions without you noticing:

df = df[df["status"] == "delivered"]     # ← added mid-script, nothing downstream is labeled

Every correlation, every groupby, every chart that runs after this line applies to delivered orders only — but nothing in the output says so. A stakeholder reading "42% of orders come from the enterprise channel" has no way to know that's 42% of delivered orders, unless someone happened to scroll up to a filter line with no comment attached to it.

5. Narrative summaries that quietly assert causation

AI-generated EDA often ends with a paragraph of plain-language commentary. That commentary is where causal language creeps in — "unit price drives revenue," "the strong correlation confirms X" — from analysis that only ever established correlation. The numbers above the paragraph are usually fine. The prose interpreting them is where scope creep happens, because narrative generation and statistical rigor are not the same task, even when the same model produces both in one response.

Read the numbers. Treat the narrative paragraph as a first draft to revise, not a finding to cite.

6. Treating "no issues found" as "the data is clean"

If you ask an AI to check for data quality issues and it reports none, that's not confirmation. It means the checks it ran didn't find anything — which says nothing about the checks it didn't run. Cardinality on string columns, structural missingness patterns, multivariate outliers: these don't show up unless something specifically goes looking for them.

All six patterns share the same shape: the output looks complete, and completeness is exactly what a quick review checks for.

Why "just review the code" isn't quite the answer

Reviewing AI-generated code before running it catches some of this — type conversions before statistics, mean vs. median, subset filters — because those are visible in the script itself. What code review alone won't catch is #5 and #6: causal language buried in a narrative summary, and the false confidence of a clean-looking "no issues found." Those require reading the output, not just the code, and holding it against what you already know about the domain.

A defensible EDA — AI-assisted or not — means you can answer yes to all of:

Type conversions happened before any statistic was computed
Missingness was profiled before anything was dropped
Every group summary includes count and spread, not just mean
No subset filter changed scope without being documented
Causal language has been replaced with correlational language where causation wasn't established
"No findings" was treated as "the checks I ran found nothing," not as "the data is clean"

None of this is really about AI. It's the same discipline that's always separated a defensible analysis from a plausible-looking one. What's changed is the speed at which plausible-looking output gets produced — which means the checklist matters more, not less.

This is the framework behind SophiArch's Exploratory Data Analysis course, including a lab where you're handed a full AI-generated EDA notebook and asked to find exactly these kinds of issues before they reach a stakeholder.