DEV Community: Atticus Li

How I Debug SaaS Cold Outreach Before Sending More Emails

Atticus Li — Wed, 12 Aug 2026 15:30:54 +0000

When a feature fails, I do not fix it by sending more traffic through the same broken path. I inspect inputs, state changes, assumptions, and failure conditions.

Cold outreach deserves the same treatment.

If a developer sends emails and gets silence, the common reaction is to buy a larger list or generate more variants. That changes throughput. It does not tell you whether the prospect has a reason to care today.

I now start with a buying trigger: an observable event that might create a painful workflow for a specific owner. The trigger is a hypothesis, not permission to invent the rest of the prospect's story.

The bug is usually upstream of the email copy

An outreach system has a simple path:

observable change
  -> suspected workflow problem
  -> person who owns the problem
  -> narrow offer
  -> meaningful commitment

When the path breaks at the first or second step, changing the subject line is like polishing an error message while the wrong function keeps running.

A company raising money is not automatically a buying trigger for your developer tool. A team hiring its first salesperson might be a trigger if your product removes a workflow that moves to that role. A public migration notice might be a trigger if your tool addresses a known implementation job. The event must connect to the job you handle.

The catch is that public evidence is incomplete. You can observe the change. You usually cannot observe the internal cost, urgency, or authority. Your first message should test those missing fields.

What paid service work taught me about inputs

I ran Jobsolv as a white-glove service before investing in software. The selected cohort included 26 clients paying $2,000 to $3,000 and approximately $58K+ in founder-reported services revenue.

Doing the work exposed different requirements. Some clients wanted to approve each application. Others wanted more automation. Many needed recalibration around what the market would support.

The lesson was narrower than "service first always works." That cohort informed the software, but it did not validate the later SaaS outcomes. The useful evidence came from seeing the workflow and receiving a paid commitment.

This is why I separate three states in outreach:

State	What it tells me	What it does not tell me
Reply	The message reached a relevant concern	The prospect will pay
Workflow detail	The suspected problem exists in some form	The problem is urgent
Paid next step	The buyer will commit resources now	The segment will repeat at scale

Each state lets you update the hypothesis without pretending you have reached the next one.

Build a trigger record

Before writing the first email, create one small record for the segment:

trigger: observable event or change
friction: suspected manual workaround
owner: role responsible for the outcome
offer: smallest useful job the product can do
commitment: action that would count as buyer evidence
disconfirming_answer: response that means the hypothesis is wrong

Here is how to fill it:

Start with evidence you can verify, such as a job post, product launch, public migration, or workflow complaint.
Map that event to one task your product handles.
Name the person likely to own that task.
Offer a narrow result instead of a tour of every feature.
Write the answer that would prove your assumption wrong.

That final field prevents you from treating every polite response as progress.

Route failures instead of adding volume

Use the responses to locate the failure:

if prospect does not recognize the trigger:
    revise the segment or trigger source
else if prospect recognizes the event but not the pain:
    revise the problem hypothesis
else if prospect recognizes the pain but rejects the next step:
    revise the offer or timing
else:
    test the same record with another well-matched prospect

This is a diagnostic, not a conversion benchmark. The repository evidence does not show that a particular cold email sequence caused Jobsolv's service revenue, so I would not claim one. Referrals were one recorded acquisition source, and the paid work provided direct access to the workflow.

My decision rule is simple: if the trigger record cannot explain why this person might care now, then I do not expand the list. I first find a stronger event or a narrower segment.

The constraint is manual research time. Reviewing each prospect costs more attention than exporting contacts. That trade-off is acceptable while the core hypothesis is uncertain. Once several real buyers confirm the same path, automation can help with collection and record keeping. It should not manufacture personalized pain or fake familiarity.

Which field in your outreach record is currently based on the weakest evidence?

Disclosure: AI assistance was used to research, structure, and edit this article. The author is responsible for the sources, examples, and final wording.

Calibration Training: What 20 Years of Forecasting Tournaments Say About Trusting Your Gut

Atticus Li — Mon, 03 Aug 2026 09:10:53 +0000

Write Down the Number Before You Know the Answer

You make predictions all day. "This refactor will cut the p99." "The bottleneck is the N+1, not the index." "This'll take two days." "That library will bite us in six months."

Some of those are right. You remember the ones that were spectacularly right, and the ones that were embarrassingly wrong. What you almost certainly don't have is the aggregate: of all the calls you were about 80% sure about, how many actually landed? Not anecdotally — measured.

I work in growth, where the same gap shows up as "this test will win" instead of "this index will help," and I've come to think it's the most underrated skill in any job that involves judgment under uncertainty. There's twenty years of research on it, it came out of an intelligence-community forecasting tournament, and the practical version costs about two minutes per prediction.

Here's what the research found and how to run it on yourself.

Measuring judgment instead of theorizing about it

In the early 2010s, IARPA — the intelligence community's advanced-research arm — ran a multi-year forecasting tournament pitting research teams against each other on real geopolitical and economic questions. Will a specific country default within six months? Will a particular ceasefire hold? Will a named leader remain in power through year-end?

Philip Tetlock's team, the Good Judgment Project, recruited thousands of ordinary volunteers with no special access to classified information and had them submit probability forecasts on hundreds of these questions over multiple years.

The result that made the project famous: a small subset of these ordinary volunteers — "superforecasters" — didn't just do well. They substantially and consistently outperformed both random chance and, in the published results, trained intelligence analysts with access to classified information on the same questions.

That's the finding that matters here. Good judgment under uncertainty isn't primarily about domain expertise or access to better information. It's a measurable, learnable discipline, separable from subject-matter knowledge — and the tournament identified exactly what the discipline consists of.

Which is good news if you've ever suspected the most confident person in the architecture review isn't reliably the most correct one.

The three habits that separated them

Breaking big questions into sub-questions with knowable base rates. Superforecasters rarely answered the headline question directly. Asked whether a leader would remain in power, they'd decompose it: how often do leaders in structurally similar situations get removed within a year? What's specifically different about this case, and how much should that shift the estimate?

The engineering translation is nearly one-to-one. "Will this migration go smoothly" invites a gut read. "How many of our last eight migrations of this size slipped, and what's different about this one" invites an actual calculation. You almost always have the base rate. It's in the repo history.

Updating in small, frequent increments — not big dramatic reversals. The best forecasters revised their estimates often, usually by a few points at a time, as new information arrived. They didn't sit on a confident call and then flip entirely when disconfirming evidence became undeniable.

This is the business-legible version of Bayesian updating: confidence should move a little with each new piece of evidence, continuously, rather than staying frozen until the evidence forces a reversal you can no longer avoid. If you've ever watched a team defend an architecture decision for two quarters and then abandon it in a single meeting, you've seen the failure mode.

Tracking their own calibration, explicitly, over time. This is the one with the most direct application and the one almost nobody outside forecasting research has adopted. Superforecasters didn't just make predictions — they scored themselves against outcomes using a Brier score, a standard measure of forecast accuracy that penalizes both overconfidence and underconfidence.

Someone who says "90% chance" should be right roughly nine times out of ten on similar calls — not seven, not ten. A Brier score, tracked over enough predictions, tells you whether your gut runs hot, cold, or well-calibrated. It's the only way to actually know rather than assume.

Why the absence of this log matters more than it looks like it should

Confidence and accuracy are two different things, and without a calibration log you only ever get feedback on the first one.

A person who is consistently, wrongly, 90% confident sounds exactly as convincing in the room as someone genuinely well-calibrated at 90% — right up until somebody tracks the record. Most organizations never track the record. So conviction gets rewarded as a proxy for judgment, with no actual check on whether the two are the same thing for this particular person on this particular type of call.

Individual wins and losses get remembered anecdotally. The aggregate pattern — was I actually right 80% of the time when I said I was 80% sure, or closer to 50% — never gets computed, because the log that would let you compute it was never kept.

The lightweight practice

You don't need a research program. You need one recurring habit cheap enough to stick.

Step	What it looks like	Why it matters
1. Log confidence before results	Before a change ships or a bet launches, write down a specific percentage — "I'm 70% confident this beats control by a meaningful margin" — not a vague "I think this will work"	Vague confidence can't be scored later. A number can be checked against what actually happened.
2. Resist rounding to 50/50 or to extremes	Use real numbers across the range — 60%, 70%, 85% — rather than defaulting to "I'm sure" (95%+) or hedging everything to "who knows" (50%)	Superforecasters used fine-grained probabilities; most people compress toward the extremes or the middle, which destroys the signal in the score
3. Don't touch the log once written	The forecast is locked before results come in — no retroactive editing once you know the outcome	Post-hoc "I sort of knew that" edits are exactly the hindsight bias that makes untracked judgment feel better calibrated than it is
4. Review quarterly, in aggregate	Pull every logged prediction from the quarter: of the calls you rated 80% confident, did roughly 80% actually hit?	A single hit or miss tells you nothing. The pattern across dozens of calls tells you whether your gut runs overconfident, underconfident, or well-tuned
5. Adjust your stated confidence, not just your decisions	If the review shows your "90% confident" calls only hit 60% of the time, distrust your own top-end confidence specifically — don't become vaguely more cautious everywhere	Calibration is often uneven — someone can be well-calibrated at 60% and badly overconfident at 95%. Only a logged record reveals where the miscalibration lives

A plain text file with one line per prediction is enough. Date, the claim, the number. That's the whole tool.

The payoff connects to how you size a bet in the first place: a hypothesis you logged at 55% confidence and one you logged at 90% shouldn't get the same resourcing commitment, and a running calibration score is what tells you whether your stated 90% has historically earned that trust.

The uncomfortable part

The likely result of running this for a year, for most people, is discovering their gut runs more overconfident than they'd assumed — particularly on the calls that felt most obvious going in.

That's not a flattering finding, which is exactly why almost nobody generates the data that would reveal it. Tetlock's forecasters weren't more naturally gifted than intelligence professionals with more information and more experience. They were more willing to keep score on themselves and adjust based on what the score showed, rather than what their conviction in the moment suggested.

That's a discipline, not a talent. It's available to anyone willing to write the number down before they know the answer.

FAQ

How is a Brier score actually calculated? It's the squared difference between your stated probability and the outcome (scored as 1 for "happened," 0 for "didn't"), averaged across all predictions — lower is better, and 0 means perfect calibration. You don't need to compute it precisely to get the benefit; even an informal quarterly "how often did my 70%-confidence calls hit" captures most of the value.

Isn't this just journaling with extra steps? The difference is the number and the review discipline. Journaling captures reasoning; a calibration log captures a falsifiable claim that gets checked against reality on a schedule. Most journaling never gets scored against outcomes, which is exactly the step that produces the learning.

What if I don't have enough major decisions to build a sample? Log smaller calls too — estimates, bug-cause guesses, minor predictions about team output. Superforecasters built calibration on volume across many modest questions, not a handful of huge ones. The skill transfers.

Does this replace statistical rigor? No — it operates one level up. Statistical testing tells you whether a specific result is likely real. Calibration tracking tells you whether your own pre-test intuition about which hypotheses will pan out is trustworthy, which affects how much weight to give your gut when deciding what to try in the first place.

How long before it's useful? Meaningful patterns usually need a few dozen logged predictions before the aggregate signal beats noise — roughly a couple of quarters of consistent logging. The point isn't a fast verdict; it's building the first real dataset you've ever had on your own judgment.

Bottom line

Tetlock's tournament didn't find that some people are naturally gifted forecasters and everyone else isn't. It found that good judgment is a trackable, improvable discipline, and the single habit that improves it fastest is one almost nobody practices: writing down a real confidence number before you know the answer, and checking it later.

It costs a few minutes per call and a quarterly review. What it buys is the only honest answer to the question every confident estimate implicitly asks: how much should anyone actually trust this gut, based on its track record rather than its tone.

If you start a calibration log this week, the first entry can be a prediction about whether you'll still be keeping it in three months. Write down the number.

I write about experimentation, judgment under uncertainty, and growth decisions at atticusli.com.

A/B Test Sample Size & MDE Calculator Guide

Atticus Li — Mon, 03 Aug 2026 09:10:52 +0000

Check This Table Before You Build the A/B Test

Here is a sprint I've watched get burned more than once, and it usually starts with a ticket rather than a bad idea.

Someone asks for an A/B test. You wire up the feature flag, split the traffic, build the variant, add the events, ship it. Six weeks later the readout comes back "no significant difference," everyone shrugs, and the variant gets deleted. The work was real. The answer was never obtainable with the traffic that page gets — and that was knowable on day one, in about thirty seconds, before a single line of test code existed.

I'm a growth person, not an engineer, but this is the part of my job that most directly wastes engineering time, so it's the part worth handing over. Before I run any experiment I calculate the required sample size — not as a formality, as a feasibility check. At least 30% of the test ideas I've seen would have required six months to run properly given the available traffic. Knowing that upfront saves time, money, and false confidence.

So here's the table. Screenshot it, put it in the ticket template, and make "which cell is this?" the first question asked in refinement.

The sample size table most calculators won't show you

Every sample size calculator asks for four inputs and returns one number. That's useful when you already know your baseline. It's useless for the question that actually comes first: is this test worth planning at all?

Each cell is the visitors needed per variant at 95% confidence and 80% power, for a given baseline conversion rate and a given relative minimum detectable effect (MDE).

Baseline CVR	5% MDE	10% MDE	20% MDE	50% MDE
1%	637,000	163,000	42,500	7,700
2%	315,000	80,500	21,000	3,800
3%	208,000	53,000	14,000	2,500
5%	122,000	31,000	8,200	1,500
10%	58,000	14,500	3,800	700

Three things to read off it before you build anything.

Double every number for a two-arm test. These are per-variant figures. A control and one variant at a 3% baseline chasing a 10% lift needs 53,000 each — 106,000 total. This single misreading is the most common planning error I see, and it's why so many tests get called at roughly half the power their owners believe they have.

The relationship is quadratic, not linear. Halving your MDE roughly quadruples the sample size. Look across the 3% row: 53,000 at a 10% MDE becomes 208,000 at a 5% MDE. Same test, same page, same traffic — asking for twice the sensitivity costs four times the visitors.

Low baselines are brutal. A 1% baseline needs roughly eleven times the traffic of a 10% baseline for the same relative lift. If you're optimizing a low-converting flow, your MDE has to be large or the test will never finish.

If your traffic can't support the cell you land in, the answer isn't to run the test anyway. It's to change the MDE, change the page, or change the method.

Why an underpowered test is worse than no test

An underpowered test doesn't have enough data to reliably detect the effect you're looking for — even when that effect is real.

Say you're testing a new product page design. The new design genuinely improves conversion by 8% relative. But the test only reached 40% of the required sample size before someone called it. It shows no significant result. You kill the variant. You just deleted a real winner.

That's a Type II error — a false negative. And unlike a false positive (shipping a winner that wasn't), a false negative is invisible. Nobody files a bug for it. Nobody knows what was lost.

The rate of false negatives is controlled by statistical power. At 80% power and your required sample size, you'll correctly detect a true effect 80% of the time. At 40% of the required sample size, power falls to roughly 40–45%. You are worse than a coin flip at finding effects that are genuinely there.

This is the argument for treating the sample size calculation as a build gate rather than a reporting detail. It's the cheapest check in the whole pipeline and it runs before any code.

The four inputs, and how each one gets fumbled

Every credible calculator — Optimizely's, Evan Miller's, AB Testguide — needs exactly four inputs. Get one wrong and the whole number is wrong.

1. Baseline conversion rate. The current rate of the metric you're optimizing, on the specific page being tested — not a site-wide average. Testing checkout? Use checkout conversion rate, not overall site CVR. This is the input most often pulled from the wrong dashboard.

2. Minimum Detectable Effect (MDE). The smallest relative lift you want the test to be able to detect. Set it at the minimum lift you'd actually ship. Check whether your calculator uses relative or absolute MDE — this guide is relative throughout. Mixing the two silently changes the answer by a lot.

3. Statistical significance threshold. Almost always 95% (alpha = 0.05, two-tailed). Some teams use 90% for exploratory tests to reduce sample size. Use 99% when stakes are high — pricing tests, checkout changes on high-revenue pages.

4. Statistical power. Almost always 80% (beta = 0.20). Use 90% for high-stakes tests where missing a real effect is costly.

Worked example

New checkout page layout on an e-commerce site.

Baseline CVR: 3.0%
MDE: 10% relative (detect a lift from 3.0% to 3.3% or larger)
Significance: 95% (two-tailed)
Power: 80%

Through a standard two-proportion z-test: ~53,000 visitors per variation (~106,000 total across both arms).

Now convert to duration: divide by your weekly traffic per arm, then round up to whole weeks so you capture weekly cycles. If that lands past your planning horizon, you've just saved the sprint.

A few things that surprise people

A 5% MDE is expensive. On a 3% baseline you need roughly 208,000 visitors per variant — about 416,000 total. On a 10% baseline it drops to about 58,000 per variant. Because sample size scales quadratically with MDE, a 5% MDE costs roughly four times what a 10% MDE costs on the same page. Most teams chasing a 5% MDE don't have the traffic to resolve it and would be better served testing a bolder change.

More variants cost more than you'd guess, but not the way most people guess. Every additional variant needs its own full per-variant allocation, and multiple comparisons raise false-positive risk unless you apply a correction such as Bonferroni. Adding a third variation costs about 21% more per variant — roughly 1.8× the total traffic, not 3×.

"We'll run it for two weeks and see" isn't a plan. Duration alone tells you nothing. Two weeks on a high-traffic homepage may be well powered; the same two weeks on a 500-session-per-week page produces a fraction of the data you need. Calculate the sample size first, convert it to weeks at your traffic level, and run for that long.

Stopping early inflates false positives no matter what the p-value shows at the moment you peek. Significance isn't a fixed visitor count — it's the point at which your observed difference is unlikely to be noise given your sample size and effect.

The one-line version for your ticket template

Which cell of the table is this test in, and does our weekly traffic reach it inside the planning horizon?

If the answer is no, the useful output isn't a test. It's a bigger change worth a bigger MDE, or a different method entirely. Either way you found out before the feature flag, which is the whole point.

I built free tools for exactly this: a sample size calculator to find how many visitors each variant needs, and an MDE calculator to check whether the lift you're chasing is even detectable given your traffic. Both are free, no signup.

Correction (July 2026): an earlier version of this guide reported the worked-example sample sizes as per-variant figures when they were closer to the combined two-arm totals — understating the per-variant requirement by roughly half. It also overstated the cost of a third variation as 3× the per-variant sample size; the correct figure is about 21% more per variant, or roughly 1.8× total traffic. Every number here has been recalculated from the two-proportion z-test at 95% confidence and 80% power, and cross-checked against a power simulation.

The Meta-Analysis Your Experimentation Program Is Missing

Atticus Li — Wed, 29 Jul 2026 18:29:02 +0000

The Meta-Analysis Your Experimentation Program Is Missing

Meta description: Most programs audit individual tests, almost none audit the program itself. A quarterly portfolio audit answers what leadership actually wants asked.

TL;DR

A product manager says: "Users want better deals." A brand marketer says: "TV is driving more direct demand." A performance marketer says: "This channel has a strong ROAS." Finance says: "But is this incremental?" Product says: "Will this hurt user trust?" Leadership says: "Should we scale this?" — every one of those claims is being made about an individual initiative. Almost nobody is asking the equivalent question about the program making all these calls.
Most experimentation teams rigorously audit individual tests but never audit the portfolio of tests as a system, the way a portfolio manager audits a fund's overall performance rather than any single trade.
The Experimentation Portfolio Audit is a named framework with four components: calibration tracking, pipeline health, hit-rate trend, and idea-source diversification.
This is the level of question CEOs, CFOs, and CMOs actually want answered about a testing program, but rarely know how to ask for explicitly — "is this program getting better at making decisions" rather than "did this test win."
Run it quarterly. It's a different cadence and a different unit of analysis than a test readout, and conflating the two is why most programs never see their own blind spots.

A product manager says: "Users want better deals." A brand marketer says: "TV is driving more direct demand." A performance marketer says: "This channel has a strong ROAS." Finance says: "But is this incremental?" Product says: "Will this hurt user trust?" Leadership says: "Should we scale this?"

Every one of those is a claim about a single initiative, and every experimentation program in existence has a process for adjudicating exactly this kind of claim — run the test, read the result, make the call. What almost no program has is a process for asking the same set of questions about itself. Is our program's hit rate actually improving, or does it just feel that way? Is finance's skepticism about incrementality something we should be more worried about than we currently are, across the portfolio, not just on this one campaign? Nobody schedules that conversation, because it doesn't fit the cadence of any single test readout. It needs its own cadence, and its own framework.

The blind spot: everyone audits the trade, nobody audits the fund

Picture how a portfolio manager operates versus how most experimentation programs operate. A portfolio manager doesn't just ask "did this trade make money." They track, across every position: was the original thesis right, and by how much? Is the pipeline of new investment ideas healthy, or is the fund running out of genuinely new theses and just re-trading old ones? Is the win rate trending up, flat, or down over the last several quarters? Are ideas coming from diversified research, or is the whole fund making correlated bets off one analyst's worldview?

Now look at how most experimentation programs operate. Team runs test. Test wins or loses. Team reports the individual result. Team moves to the next test. Multiply that by a hundred tests a quarter, and what you have is a hundred individually-audited trades and a completely un-audited fund. Nobody is asking whether the program itself is improving as an instrument for making decisions — only whether this quarter's crop of tests happened to win.

That's the actual gap. It's not that companies lack a testing culture. Many have a mature one at the level of the individual test — proper randomization, reasonable sample sizes, clean reporting. What's missing is the equivalent discipline one level up: treating the whole set of tests as a portfolio with its own health metrics, separate from any individual result.

The Experimentation Portfolio Audit

This is the named framework: a structured, quarterly review of the program as a system, built from four components.

1. Calibration tracking

Before every test ships, someone had a belief about how it would go — a predicted lift, or at minimum an implicit probability of success ("I'm pretty confident this wins"). Calibration tracking means logging that prediction before the result comes in, then comparing it systematically against what actually happened, across every test in the portfolio, over time.

This is the direct, practical payoff of the "winner's curse" phenomenon well-documented in experimentation literature — the pattern where the estimated effect size of a test that clears significance is, on average, inflated relative to the true effect, precisely because it cleared a significance threshold partly by chance as well as by real effect. Most programs know about the winner's curse as a concept. Almost none of them have measured their own inflation factor. Calibration tracking answers a concrete, useful question: when your team predicts a moderate lift, does reality typically come in lower? By how much, roughly, and is that gap shrinking or growing over time? A program that has never measured this has no idea whether its own forecasts — the ones finance is quietly building projections on top of — are honest or systematically optimistic.

2. Pipeline health

Win rate is the metric everyone tracks. Almost nobody tracks whether the pipeline of hypotheses feeding that win rate is healthy. A program can maintain a perfectly respectable win rate while quietly running out of genuinely novel ideas — re-testing minor variants of the same three hypotheses because that's what's easy to generate, rather than sourcing new ones.

Pipeline health asks: over the last few quarters, what fraction of new tests represent a genuinely new hypothesis, as opposed to a small variation on something already tested? A declining rate of novel hypotheses is a leading indicator of trouble that a stable win rate will hide for a surprisingly long time — because a program that keeps re-testing safe, well-understood variants can sustain a good win rate right up until it has nothing left to test that matters.

3. Hit-rate trend over time

Not the win rate in isolation — the trend of the win rate, tracked over enough quarters to see a direction. Is the program actually getting better at picking winners as it accumulates institutional knowledge about what works in this specific business, or is it flat, or quietly declining?

A flat or declining hit-rate trend, even with an acceptable absolute win rate, is worth investigating on its own. It can mean the easy wins have already been captured and the program hasn't adjusted its hypothesis quality to compensate, or it can mean the same idea sources are being mined past the point of diminishing returns (which connects directly to component four).

4. Idea-source diversification

Where are hypotheses actually coming from? A healthy portfolio draws from a mix: quantitative funnel analysis, qualitative research (support tickets, sales calls, user interviews), and competitive observation. An unhealthy one is quietly dominated by one source — often whichever team or individual is loudest, or whichever data is easiest to pull — and the correlated nature of that source means the whole hypothesis pipeline is making a version of the same bet repeatedly, even when it looks diversified on a roadmap slide.

Component	Question it answers	Why win rate alone misses it
Calibration tracking	Is our forecasted lift honest, or systematically inflated?	Win rate says a test won; it says nothing about whether the predicted magnitude was realistic
Pipeline health	Are we still generating genuinely new hypotheses?	A shrinking pipeline can sustain a fine win rate for several quarters before it shows up as a problem
Hit-rate trend	Are we getting better at this, or coasting?	A single quarter's win rate can't show a trend; only a multi-quarter view can
Idea-source diversification	Are we making one correlated bet repeatedly?	A long roadmap can look diverse while every hypothesis traces back to the same source

Why this is the question leadership actually wants asked

CEOs, CFOs, and CMOs rarely ask for "a portfolio audit of the testing program" in those words, because most of them have never seen the framework named. What they do ask, constantly, in slightly different phrasing, is some version of: "Is this program actually getting smarter, or are we just running a lot of tests?" That is precisely the question a pile of individual test readouts cannot answer, no matter how many of them you hand over. A hundred clean individual readouts tells leadership that a hundred decisions were made carefully. It tells them nothing about whether the underlying instrument making those decisions is improving, plateauing, or drifting toward false confidence. For more on why individual test rigor doesn't guarantee program-level rigor, Trustworthy Online Controlled Experiments — written by the team that ran experimentation at Microsoft, Bing, and LinkedIn — remains the standard reference for what rigor looks like at scale.

This is also the natural extension of the Confidence Tier Model: that framework governs how much certainty a single bet needs before you size it. The Portfolio Audit is the same discipline applied one level up — not "how confident should we be in this test," but "how confident should we be in our own confidence, given how this program's predictions have actually performed over time."

Running it quarterly

The cadence matters as much as the content. A quarterly audit is frequent enough to catch a declining pipeline or a calibration drift before it becomes a credibility problem with finance, but infrequent enough that you're looking at a real trend rather than noise from a handful of recent tests. Trying to run this monthly usually just re-measures the same few tests repeatedly and mistakes short-term variance for a trend. Running it annually means a full year can pass with a quietly degrading pipeline before anyone notices.

In practice, the audit is a standing quarterly agenda item, distinct from any individual test readout: pull every prediction and result from the quarter for calibration tracking, tag each new hypothesis by source and novelty for pipeline health and diversification, and plot the trailing win rate against the last several quarters for the trend. None of the four components require new tooling most programs don't already have — the tests were already logged. What's missing is almost never data. It's the standing habit of looking at the portfolio as a portfolio.

FAQ

How is this different from a normal quarterly business review?

A normal QBR typically reports outcomes — revenue impact, notable wins, roadmap for next quarter. The Portfolio Audit is specifically about the health of the decision-making system itself: is it calibrated, is its pipeline healthy, is its hit rate trending in a good direction, is it drawing from diverse sources. It can feed into a QBR, but it's a different unit of analysis than a results summary.

What's a healthy hit rate, and does this framework require a specific number?

No, deliberately. A "good" hit rate varies enormously by vertical, maturity of the product, and how bold the hypotheses being tested are — a program running only safe, incremental tests should have a higher win rate than one deliberately testing bolder bets, and a higher win rate in that case isn't necessarily better. The framework cares about the trend and the calibration, not a universal benchmark number.

We don't have enough historical data to see a trend yet. Is this still worth doing?

Start the calibration logging and source-tagging now, even before you have enough history to see a trend — the audit's value compounds. A program that starts tracking predicted-versus-actual lift this quarter will have a genuinely useful calibration read within two or three quarters. The programs that never start never get that visibility at all.

Doesn't tracking calibration create an incentive to sandbag predictions to look good later?

It's a real risk, which is why the predicted lift needs to be logged before the result is known, ideally in a system the predictor can't quietly edit after the fact, and why the audit should be framed as improving the instrument rather than grading individuals. The goal is an honest read on the program's forecasting accuracy, not a performance review of whoever made the prediction.

Who should own running this audit?

Whoever owns the experimentation program's methodology, not whoever owns any individual test. It needs someone with visibility across the whole portfolio and enough seniority that the findings — especially an uncomfortable calibration gap or a shrinking pipeline — get acted on rather than filed away.

Bottom line

Every program already knows how to audit a test. Far fewer know how to audit themselves, because the muscle for it doesn't get built by running more tests — it gets built by deliberately stepping back from any single result and asking whether the whole system is improving. The Experimentation Portfolio Audit is that step back, made concrete: four questions, one quarterly cadence, and an honest look at whether the program is actually getting better at making bets or just staying busy making them.

If you're building or auditing an experimentation program and want an outside read on this, get in touch.

Sequential Testing and the SPRT: How to Stop a Test Early Without Cheating

Atticus Li — Mon, 27 Jul 2026 06:46:56 +0000

Sequential Testing and the SPRT: How to Stop a Test Early Without Cheating

Meta description: Peeking at a fixed-sample A/B test inflates false positives. Sequential testing lets you check results repeatedly and stop early without cheating.

TL;DR

Fixed-sample testing assumes you'll wait for a pre-calculated sample size before looking at results. Checking early and stopping the moment you see significance — "peeking" — quietly inflates your real false-positive rate, often far above the 5% you think you're getting.
Abraham Wald's Sequential Probability Ratio Test (SPRT), developed for wartime quality control, is the mathematically rigorous alternative: a procedure built to be checked repeatedly, with pre-calculated boundaries that keep the false-positive rate honest by construction.
The difference between the SPRT and peeking isn't willpower — it's that the SPRT's stopping rule is part of the math from the start, so stopping early doesn't cost you anything in error-rate control.
Sequential design is the right call when traffic is limited, the cost of running a test too long is high, or the business genuinely can't commit to waiting for a fixed horizon — not a substitute for rigor, but a different kind of rigor suited to a different constraint.
This is a methodology choice, not a shortcut — and it's one input into the broader question of how much certainty a given bet needs, covered in the Confidence Tier Model.

Every experimentation program eventually hits the same moment: a test has been live for four days, the dashboard shows a lift, and someone — a stakeholder, a PM, sometimes you — asks "can we call it?" The honest answer depends entirely on what kind of test you designed, and most teams don't have a clean answer, because most teams designed a fixed-sample test and are now trying to read it like a sequential one.

Those are not interchangeable. Knowing the difference, and choosing deliberately between them before the test starts, is the actual skill — not "wait long enough" or "look for significance," but designing the test so that looking whenever you want doesn't break the statistics.

Why fixed-sample testing assumes a promise you probably won't keep

Classical A/B testing — the kind taught in most CRO courses — works like this: you calculate a required sample size in advance, based on your baseline conversion rate, the minimum effect size you care about detecting, and your desired confidence level. You commit to collecting exactly that much data. Then, and only then, you look at the result and check whether it clears your significance threshold.

The statistics behind that threshold are only valid under one condition: that you looked exactly once, at the pre-agreed sample size. The 5% false-positive rate on a standard significance test is a promise about a single decision point, not about a process of repeated glances.

That promise is almost never what actually happens inside a real company. A test goes live, and a dashboard exists, and someone checks it — not out of malice, but because that's what dashboards are for. The test "looks good" on day three, someone mentions it in a meeting, and the pressure to end it right there builds before the pre-calculated sample size has been reached.

Why peeking inflates false positives — the mechanism, not just the warning

This isn't a hypothetical concern about discipline — it's a well-documented statistical problem, most famously explained in Evan Miller's widely-cited essay "How Not to Run an A/B Test." The mechanism is simpler than it sounds.

A significance test asks: "if there were truly no difference between A and B, how likely is it that random noise alone would produce a gap this large?" At a single, pre-specified sample size, that probability is genuinely 5% (assuming a standard threshold). But random noise doesn't move in one direction and stay there — it wanders. A test's measured lift will drift above and below the "true" effect throughout its run, the same way a coin-flip tally can briefly show more heads than tails even from a fair coin.

If you check that wandering result once, there's a small chance you catch it at a noisy peak. Check it every day for two weeks and you get many chances to catch one — and stopping specifically when you see one is the failure mode. Each additional look is another opportunity for pure noise to cross the significance line, and standard significance math was never built to account for that multiplicity. The stated 5% false-positive rate can realistically climb well past 20-30% for a test checked repeatedly and stopped opportunistically — not because anyone cheated, but because the checking itself changes the odds.

The insidious part is that this happens with zero bad intent. Nobody sets out to defraud their own experimentation program. They just look at a live dashboard the way anyone would, and stop at the first moment it validates what they hoped to see.

What the SPRT actually is

Long before online A/B testing existed, statistician Abraham Wald developed the Sequential Probability Ratio Test in the 1940s for a different problem: quality-control sampling, where the cost of drawing each additional sample was real and testing had to be as fast as it could responsibly be. His procedure was one of the first to formally solve a specific tension — that the only way to be allowed to look repeatedly without inflating your error rate is to build the repeated-looking into the test's design from the start.

Conceptually, the SPRT works like this: instead of a fixed sample size, you pre-calculate two boundaries — an "accept the effect is real" boundary and an "accept there's no effect" boundary — based on the false-positive and false-negative rates you're willing to tolerate. As data comes in, you track a running measure of how strongly the evidence favors one hypothesis over the other, checked after every new data point. The test ends the moment the evidence crosses either boundary — and not a moment before.

The mathematical guarantee is what makes this different from peeking: because the boundaries account in advance for the fact that you'll be checking constantly, crossing one of them means the false-positive rate is still controlled at the level you specified — by construction, not by hoping nobody looked too early. You get to look as often as you want. The looking was priced into the design, rather than bolted on as an afterthought.

Sequential testing versus peeking: same behavior, opposite math

	Peeking at a fixed-sample test	Sequential testing (SPRT-based)
When you're "allowed" to look	Only once, at the pre-calculated sample size (in theory)	Continuously, from the start
What happens if you stop early on a good-looking result	Real false-positive rate is inflated — often severely	No inflation — the stopping boundary already accounts for repeated looks
What the statistical guarantee depends on	Strict discipline never to act on an early look	The test design itself, not the operator's willpower
Best suited for	High-traffic tests where a fixed horizon is genuinely affordable	Limited traffic, urgent decisions, high cost of running too long
Common failure mode	Stakeholders check the dashboard and stop early anyway	Choosing sequential boundaries so loose they approve weak effects too easily

The table makes the actual difference clear: it was never about whether you look. It's about whether the math you're using was built to be looked at.

When a business should actually choose sequential over fixed-sample

Sequential testing isn't strictly better — it's a different tool suited to a different constraint. Three situations make it the right call:

Limited traffic. If a fixed-sample calculation says you need six weeks to reach adequate power, and you don't have six weeks of stable traffic, a sequential design lets you stop as soon as the evidence is strong enough — for a genuinely large effect, considerably faster than waiting out a fixed horizon built around detecting a smaller one.

Need to move fast. Some decisions carry real time-value: a pricing test during a narrow competitive window, or a homepage change ahead of a seasonal peak. Waiting for a textbook-perfect fixed sample size while the business context shifts underneath the test can mean the answer arrives after it stopped mattering. Sequential design front-loads the chance to stop early when the true effect is large, without giving up rigor to get there.

High cost of running a test too long. Every day a test runs, some portion of traffic sees a variant that might be worse. If the downside is expensive — a checkout flow, a pricing page, a high-intent conversion point — a method that ends the test the moment sufficient evidence exists reduces exposure to that downside compared to a design that mechanically runs its full pre-calculated course regardless of how the evidence is trending.

None of this is a reason to abandon fixed-sample testing everywhere. A high-traffic page with no urgency and no real cost to a slightly longer runtime is a perfectly reasonable candidate for a classic fixed-horizon design — simpler to explain to stakeholders, and the "wait for the number" discipline is easy to enforce when there's no pressure pushing against it. The choice is about matching the method to the constraint, not defaulting to whichever one sounds more sophisticated.

The senior judgment call underneath this

Choosing sequential over fixed-sample isn't a statistics decision made in isolation — it's a business-constraint decision that happens to have a statistical implementation. The judgment call is diagnosing, honestly, which constraint you're actually under: is it traffic, is it time-value, is it downside risk, or is it none of those and you'd simply prefer not to wait? Only the first three are legitimate reasons to reach for a sequential design. The fourth is impatience wearing a statistical costume, and a good sequential design won't rescue a decision that hasn't earned enough evidence yet — it will just tell you, honestly and a bit faster, that the evidence isn't there.

That's the real value of understanding the SPRT conceptually, even if you never derive the boundary math yourself: it lets you tell the difference between a team that's moving fast because it chose the right tool for its constraint, and a team that's moving fast because it stopped looking as soon as the dashboard agreed with them.

FAQ

Is sequential testing the same thing as a "Bayesian" test?

Not exactly, though they're often discussed together and can be used for similar goals. The SPRT is a classical (frequentist) method built specifically around a likelihood-ratio boundary. Bayesian sequential methods exist too and update a probability distribution rather than tracking a likelihood ratio against fixed boundaries. Both are legitimate ways to handle repeated looking; the SPRT is the historical origin point and the clearest place to build intuition for why sequential design works at all.

Can I just apply a stricter significance threshold to compensate for peeking?

That's a rough patch, not a real fix. A tighter threshold can reduce the damage, but it doesn't carry the same guarantee as a method purpose-built for repeated checking, and it's easy to get wrong in either direction — too loose to help, or so strict the test becomes needlessly slow. If you know you'll want to check repeatedly, design for it from day one rather than retrofitting a fixed-sample test after the fact.

Does sequential testing require more traffic than fixed-sample testing?

Often less, for a real effect — that's much of the appeal. Because the test can stop the moment sufficient evidence accumulates, a genuinely strong effect can be confirmed faster than a fixed-horizon design that's committed to running its full pre-calculated course regardless. For a true null result (no real effect), a well-designed sequential test still runs to a sensible stopping point rather than dragging on forever — that's part of what the pre-calculated boundaries manage.

What's the actual downside of sequential testing, if it sounds strictly better?

Two things. First, it's a more complex methodology to explain and to implement correctly — a poorly-configured sequential design (boundaries set too loosely) can approve weak effects faster than a rigorous fixed-sample test would, which is a real risk if the setup isn't done carefully. Second, some existing tools and stakeholder mental models are built around the "wait for the number" fixed-sample framing, so introducing sequential design also means investing in explaining it well.

How does this connect to deciding how much certainty a bet actually needs?

Sequential testing is one lever for reaching a high-certainty result faster under a real constraint — but it doesn't change how much certainty a given decision requires in the first place. That's a separate question, and it's the one the Confidence Tier Model is built to answer: sequential design can help you earn a Proven-tier result under time or traffic pressure, but it's not a way to skip earning it.

Bottom line

Peeking and sequential testing look identical from the outside — someone checking a dashboard and making a call before a pre-calculated sample size is reached. The difference is entirely in whether the underlying math was built to allow it. Fixed-sample testing makes a promise about looking exactly once; sequential testing, grounded in Wald's SPRT, makes a different and equally rigorous promise about looking as often as you need to. Choosing between them is a constraint-matching decision, not a rigor-versus-speed trade-off — and knowing which constraint you're actually solving for is what separates a fast, disciplined program from one that's just impatient with better vocabulary.

If you're building or auditing an experimentation program and want an outside read on this, get in touch.

The Confidence Tier Model: How to Decide When Your Data Isn't Enough

Atticus Li — Mon, 27 Jul 2026 06:46:55 +0000

The Confidence Tier Model: How to Decide When Your Data Isn't Enough

Meta description: Most testing programs are built for traffic they don't have. Three confidence tiers — proven, directional, speculative — each with its own bet-sizing rule.

TL;DR

Fixed-sample A/B testing assumes you can wait for statistical significance. Most teams can't — traffic is too thin, or the market is moving too fast to wait.
The fix isn't lowering your standards. It's replacing the binary "significant / not significant" gate with three explicit confidence tiers — Proven, Directional, Speculative — each with its own evidence bar and its own bet-sizing rule.
Underpowered tests systematically overestimate effect size (the "winner's curse"). A confidence tier that accounts for this is more honest than a p-value that pretends otherwise.
The way to move a learning up a tier isn't more of the same test — it's triangulation: stacking correlated, individually-weak signals until they converge.
This is a methodology choice, not a compromise. Teams that name their confidence tier explicitly make faster, more defensible decisions than teams that either wait for certainty they'll never reach, or ship everything with false confidence.

Six people, six kinds of evidence, and a decision that needs to get made this quarter — not whenever a test finally clears p<0.05. This is the actual job: not running tests, but converting six competing claims into one evidence base leadership can act on. Most experimentation methodology is written for a world where you have the traffic to wait for a clean answer. Most companies don't live in that world.

The problem classic A/B testing doesn't solve

Fixed-sample significance testing has a hidden assumption baked in: that you can collect enough data to reach a conclusive answer before the business needs to decide. For a handful of companies with tens of millions of users, that's true. For almost everyone else, it isn't — the page doesn't get enough traffic, the conversion event is too rare, or the market is moving fast enough that by the time you'd reach significance, the question itself has changed.

The instinct in that situation is usually one of two bad options. Either wait for certainty that will never arrive, and ship nothing while competitors move — or quietly abandon rigor altogether and ship on vibes, dressed up as "directional data." Neither is a methodology. Both are the absence of one.

There's a third option, and it's the one senior practitioners actually use: decide, explicitly, how much certainty this specific bet requires, and size the bet to match the certainty you actually have.

The Confidence Tier Model

Instead of one gate ("is this significant, yes or no"), use three tiers. Each tier has its own evidence bar, its own typical methodology, and — critically — its own rule for how big a bet you're allowed to make on it.

Tier	Evidence bar	Typical methodology	Bet-sizing rule
Proven	Statistically significant and replicated at least once, or converging evidence from multiple independent tests	Fixed-sample or sequential test that reached its pre-registered threshold; repeated in a second context	Full rollout; safe to use as the basis for resourcing decisions and forward planning
Directional	Positive (or negative) trend that hasn't reached traditional significance, but is supported by at least one converging secondary signal	Underpowered test read with a Bayesian posterior ("70% probability B beats A by at least X%") or a proxy-metric result	Ship to a meaningful but bounded segment; treat as provisional; schedule a revisit before treating it as permanent
Speculative	A single weak signal — one underpowered test with no corroboration, or qualitative-only evidence	Single test, session-replay observation, a handful of support tickets	Cheap, reversible test only; never the sole basis for a resourcing ask; explicitly flagged as unproven when reported up

The point of naming the tier isn't bureaucracy — it's honesty. A "directional" learning presented as "proven" is how a program loses credibility with finance the first time a confidently-reported win doesn't hold up in real revenue. A tier system forces that distinction into the open before it becomes an expensive surprise.

Why the tiers need different methodology, not just different confidence

Moving a learning between tiers isn't a matter of squinting harder at the same p-value. It requires deliberately choosing a different way of generating evidence:

From Speculative to Directional: triangulation over isolation. A single underpowered test can't reach significance on its own — but it doesn't have to stand alone. Stack it against session replays, support-ticket language, a related test on an adjacent page, or a qualitative signal from sales calls. No one source is conclusive; the convergence across several weak, correlated sources is the evidence. This is closer to how a diagnostic works in medicine, or how an analyst builds a case in intelligence work, than how a single A/B test report is usually read.

From Directional to Proven: pooling, not just repeating. Ten underpowered tests on a related hypothesis, analyzed together for consistency of direction and rough magnitude, can be strong evidence even when none of them individually clears significance. This is an internal meta-analysis — treating a family of related tests as one larger dataset rather than ten independent verdicts.

Choosing sequential over fixed-sample when the business can't wait. If the constraint is time rather than traffic, a sequential testing design (built to be checked repeatedly without inflating the false-positive rate, unlike casually peeking at a fixed-sample test) can produce a Proven-tier result faster than a fixed-horizon design — the methodology choice itself is what earns the tier, not just patience.

That methodology choice — which of these three moves you reach for, and why — is exactly the kind of senior judgment call that a junior analyst wouldn't know to make, and it's the actual differentiator between a program that's fast because it's careful about what it bets on, and one that's fast because it's guessing.

The part CFOs actually care about

Finance doesn't ask "was this test significant." Finance asks "how sure are we, and how much should that certainty cost us." The Confidence Tier Model is the direct answer to that question, because it ties bet size to evidence quality explicitly instead of leaving it implicit:

A Proven learning can justify a headcount or budget request — the evidence would survive scrutiny.
A Directional learning justifies a bounded pilot, not a resourcing ask — you're buying more information, not committing capital.
A Speculative learning justifies a cheap, reversible test — nothing more, and reporting it as more than that is where a program's credibility erodes.

FAQ

Isn't this just a fancy way of saying "ship stuff you're not sure about"?

No — it's the opposite. It's a way of being explicit about how sure you are, so the size of the bet matches the size of the evidence. The failure mode this replaces is treating every shipped test as equally certain, which either produces false confidence (reporting a Directional result as Proven) or paralysis (refusing to act on anything short of full significance).

What if leadership wants a single yes/no answer, not three tiers?

Give them the tier and the bet-sizing rule that comes with it, not the underlying statistics. "This is Directional — we're confident enough to pilot it on 20% of traffic, not confident enough to roll out fully yet" is a complete, decision-ready answer. The tiering is what lets you say yes without overclaiming.

How do you decide the specific threshold for each tier in a low-traffic context?

Set it relative to your own minimum detectable effect, not a generic industry rule. If your traffic can only reliably detect a large effect, your Proven bar should reflect that — and it's a signal you should be testing bigger, bolder changes rather than small ones your traffic can't resolve anyway.

Does a Speculative-tier learning ever get to skip straight to Proven?

Rarely, and only when the signal is unusually strong relative to a well-understood mechanism (a known, stable behavioral principle, not a novel or context-specific claim). Treat that as the exception that still needs to earn its way to Directional first in almost every real case.

How is this different from just using Bayesian statistics?

Bayesian methods are one of the tools that can move a learning between tiers — the posterior probability approach is a natural fit for the Directional tier especially. But the tier model is broader: it also covers triangulation, pooling, and sequential design, and it adds the organizational layer (bet-sizing rules) that a purely statistical framework doesn't specify on its own.

Bottom line

The teams that make the best bets under real-world constraints aren't the ones with the most traffic — they're the ones who've stopped pretending every decision needs the same amount of certainty. Naming the tier, out loud, before you size the bet, is the entire discipline. I turn uncertain business decisions into testable hypotheses, then turn test results into investment decisions — this framework is the mechanism in between.

If you're building or auditing an experimentation program and want an outside read on where your own methodology is quietly overclaiming certainty it doesn't have, get in touch.

Triangulation Over Isolation: Building Confidence from Weak, Convergent Signals

Atticus Li — Mon, 27 Jul 2026 06:39:09 +0000

Triangulation Over Isolation: Building Confidence from Weak, Convergent Signals

Meta description: A single underpowered test never proves anything alone. How senior practitioners stack weak, independent signals until they converge into real confidence.

TL;DR

No single weak signal — one underpowered test, one heatmap, one batch of support tickets — is ever conclusive on its own. Treating it as if it were is the most common overclaiming mistake in experimentation.
Real-world confidence is built by triangulation: stacking multiple weak signals that independently point the same direction, until the convergence itself becomes the evidence.
The strength of triangulation depends entirely on whether your signals are genuinely independent or just the same underlying data restated three times. Independence is the whole game.
The practical rule: before calling a learning Directional, name at least two independent corroborating signals. Before calling it Proven, replicate the test or show consistent direction across several related tests.
This is the mechanism that actually moves a learning up a tier in the Confidence Tier Model — triangulation is how Speculative becomes Directional in practice, not just in theory.

A single underpowered test lands on your desk. The lift looks good — not significant, but positive, and the sample is what it is given the traffic you have. Someone asks: "So can we ship this?" The honest answer is never "yes, the test says so." The honest answer is "let me see what else agrees with it."

That question — what else agrees with it — is the actual discipline. Most experimentation writing treats each test as a self-contained unit of truth: it either reached significance or it didn't. But that's not how confidence actually gets built in practice, and it's not how any adjacent evidence-based discipline works either. A single lab result doesn't diagnose a patient. A single data point doesn't move an intelligence assessment. A single witness doesn't close a case. Confidence comes from convergence — multiple imperfect sources of evidence, each individually insufficient, pointing at the same conclusion from different angles.

Why isolation fails

An underpowered test isn't just "less certain" than a fully-powered one — it's structurally unable to answer the question you're asking it. If your traffic can only reliably detect a large effect, and the true effect is moderate, no amount of squinting at that one dataset will resolve it. You're not looking at low-confidence evidence of a real effect; you're looking at noise that happens to lean positive, and noise leans positive about half the time by construction.

The same is true for any single non-quantitative source. One session replay showing a user getting confused at checkout is an anecdote, not a pattern — that user might have been on a bad wifi connection, distracted, or simply unusual. One support ticket complaining about pricing might be a single loud customer, not the market. The mistake isn't using these sources. It's using one of them alone and calling it evidence.

The signals a growth team actually has

Before you can triangulate, it helps to have an honest inventory of what "a signal" even includes. Most growth teams under-use the sources they already have access to because they've been trained to think only the quantitative funnel data counts as real evidence.

Signal type	What it tells you	Typical weakness alone
Quantitative funnel/cohort data	Where and how much drop-off happens	Underpowered on its own below significance; can't explain why
Session replays and heatmaps	How users actually behave, moment to moment	Small, potentially unrepresentative sample; observer bias in interpretation
Support ticket language	What's frustrating people enough to write in	Selection bias — only captures people upset enough to contact you
Sales call notes	What prospects object to or ask about before they buy	Anecdotal, filtered through the rep's framing, small n
A related test on an adjacent page or flow	Whether the same mechanism shows up elsewhere	Different context might mean a different underlying cause
Competitor moves	What the market is collectively betting works	You don't see their data, only their action — could be a mistake they haven't caught yet

None of these six, alone, would survive scrutiny as proof. But notice what happens when three or four of them line up: a funnel test shows a soft positive trend on a redesigned pricing page, session replays show users hesitating on the same screen, and sales call notes mention the same objection independently. No single source moved. But together, they've told you something that none of them could tell you alone.

What makes signals actually independent

Here's the part that separates rigorous triangulation from a comforting illusion of it: the value of a second signal depends entirely on whether it's independent of the first, not just whether it's a different-looking chart.

Two signals are correlated, not independent, when they trace back to the same underlying data-generating process. A drop in checkout completion, a heatmap showing rage-clicks on the payment button, and a spike in cart-abandonment analytics events are three visualizations of the same behavior. They'll always agree with each other, because they're measuring the same thing through three different lenses. Stacking them and calling it triangulation is double-counting — it feels like three pieces of evidence, but it's structurally closer to one, restated.

Genuine independence means the signal was generated by a different mechanism, a different population, or a different method entirely. A quantitative funnel signal (users) and a sales call objection (prospects who haven't converted yet) and a competitor's product change (a totally separate company's internal research, filtered through their own incentives) are independent in the way that actually matters — if all three land on the same conclusion, they didn't get there by sharing a data source. That convergence is real evidence, in the same way that three witnesses who weren't in contact with each other telling a consistent story is stronger than one witness saying the same thing three times.

The practical test: ask "if my first signal were simply wrong — biased, coincidental, or misread — would my second signal still exist?" If the answer is no, because the second signal is downstream of the same system that produced the first, you don't have two signals. You have one signal wearing two hats.

The Triangulation Framework

This is the operational version — the two thresholds worth naming explicitly before you act on a learning:

Before treating a Speculative-tier learning as Directional: name at least two independent corroborating signals. Not two charts from the same analytics platform. Two signals that could plausibly have disagreed with each other, and didn't. A soft quantitative trend plus a qualitative pattern from a genuinely different source (support language, sales notes, session behavior) clears this bar. A soft quantitative trend plus a second cut of the same underlying dataset does not.

Before treating a Directional learning as Proven: either replicate the test, or show consistent direction and rough magnitude across several related tests. Replication means running the same test again, ideally in a different context or time window, and seeing the effect hold. The related-tests path is a form of internal meta-analysis: if five variations on the same hypothesis, tested across different pages or segments, all lean the same direction and roughly the same size, that consistency is stronger evidence than any one of them individually reaching significance — even though none of them alone would clear a rigorous bar.

Both thresholds share the same underlying logic: a single source, however clean-looking, is a claim. Multiple independent sources that agree are a finding.

Why this matters more as traffic gets thinner

Triangulation isn't a workaround for teams with bad data — it's the load-bearing method for any team operating below the traffic level where classic fixed-sample testing works cleanly, which in practice is most teams outside a handful of the largest consumer platforms. The instinct when traffic is thin is to either wait indefinitely for a test that will never reach significance, or to ship on a single unconvincing result dressed up with confident language. Triangulation is the actual third option: it lets you build real confidence out of evidence that was never going to be individually sufficient, by being disciplined about what counts as genuine corroboration versus an echo of the same source.

This is also why triangulation is a genuine skill rather than a checklist. Recognizing that a heatmap and a funnel metric are correlated but a heatmap and a sales call note are independent requires understanding where each piece of data actually comes from — the instrumentation, the population, the collection method. That judgment is exactly the kind of thing a dashboard can't do for you.

FAQ

How many independent signals are "enough"?

There's no universal number, but two independent, genuinely uncorrelated signals pointing the same direction is a reasonable minimum bar for Directional, and three or more — especially if they span quantitative and qualitative sources — is a strong case. The bigger driver than count is independence quality: two truly independent signals beat five correlated ones every time.

What if my signals point in different directions?

That's a real finding too, and often a more useful one than false convergence. Disagreement between independent signals usually means the effect is context-dependent — it's real in one segment or scenario and absent or reversed in another. The next step isn't to average them into a mushy middle conclusion; it's to figure out what's different about the contexts where they diverge.

Isn't this just confirmation bias with extra steps?

It's the opposite, if done honestly. Confirmation bias is selectively noticing signals that agree with what you already believed and ignoring ones that don't. Triangulation done properly requires actively seeking out independent sources — including ones that could disconfirm the hypothesis — and being willing to report "the signals didn't converge" as a legitimate outcome. The discipline is in checking independence rigorously, not in stopping as soon as you've found two things that agree.

Can qualitative signals ever outweigh quantitative ones?

Yes, in specific cases — most often when the quantitative sample is too small to be informative but the qualitative pattern is unusually consistent and specific (the same exact objection, worded similarly, from unrelated sources). Quantitative data isn't inherently more trustworthy than qualitative data; it's differently biased. A thin quant signal plus a strong, specific, repeated qualitative pattern can outweigh a thin quant signal alone.

How does this relate to the Confidence Tier Model?

Triangulation is the mechanism, the tier model is the destination. The Confidence Tier Model tells you what evidence bar each tier requires and what you're allowed to bet once you're there. Triangulation is how you actually clear the bar between Speculative and Directional in a low-traffic environment where a single test will never get there alone.

Bottom line

No single weak signal is going to save you from having to make a judgment call — and that's fine, because that was never the job of any one signal in the first place. The job is recognizing which sources are genuinely independent, stacking enough of them that a real pattern would show up as convergence, and being honest when they don't agree. That's not a lesser version of rigor. For most real-world traffic levels, it's what rigor actually looks like.

If you're building or auditing an experimentation program and want an outside read on this, get in touch.

Why Most 'Wins' Don't Replicate: The Winner's Curse, Applied to Growth Teams

Atticus Li — Mon, 27 Jul 2026 06:38:58 +0000

Why Most "Wins" Don't Replicate: The Winner's Curse, Applied to Growth Teams

Meta description: The winner's curse means shipped A/B test wins systematically overstate their true effect. The fix: track predicted lift against realized lift over time.

TL;DR

The "winner's curse" is a real, well-documented statistical phenomenon: when you select the best-looking result out of many noisy estimates, that selected estimate is systematically inflated relative to the true effect — purely because of how selection interacts with noise, not because anyone did anything wrong.
The mechanism is two ideas working together: regression to the mean (extreme results tend to be closer to average next time) and selection effects (picking the "winner" specifically selects for noise that happened to point in a favorable direction).
This is a structural reason a shipped "win" often doesn't fully hold up in real revenue afterward — the test result was never purely signal, and the part that was noise doesn't ship with the variant.
The fix isn't more skepticism in the abstract — it's calibration tracking: recording predicted lift against realized lift for every shipped test, over time, to learn your own program's typical inflation factor and discount future estimates accordingly.
This is also the seed of a larger practice — auditing your own program's track record systematically — which deserves its own deeper treatment beyond what's covered here.

A test finishes. The dashboard shows a lift that clears significance. Everyone is pleased, the win gets shipped, and it gets written up as a case study. Three months later, someone asks finance to reconcile that lift against actual revenue, and the number that shows up is smaller — sometimes much smaller — than what the test reported.

The default explanation reaches for something operational: a seasonal effect wore off, a different team shipped something that interacted with it, the market shifted. Sometimes that's true. But there's a more fundamental reason this happens constantly across virtually every serious experimentation program, and it has nothing to do with anything going wrong afterward. It's baked into the act of selecting a "winner" in the first place.

The winner's curse, defined properly

The winner's curse is a well-documented phenomenon originally described in economics — first in the context of competitive bidding (auction winners tend to have overpaid, precisely because winning required having the most optimistic estimate among many bidders), and it generalizes far beyond auctions to any situation where you select the best-looking result from a set of noisy estimates.

Applied to experimentation, the logic is this: every test result is a combination of a true effect and noise. Most tests you run — whether across many variants, many segments, or many attempts at a hypothesis — will produce results that scatter around the true effect, some higher, some lower, purely due to random variation. When you then select the single best-performing result to ship (the highest lift, the most significant p-value, the test that happened to look good right when you needed to make a call), you aren't randomly sampling from that scatter. You're specifically selecting for the cases where noise pointed in the favorable direction, on top of whatever true effect exists.

The result: the measured effect size of whatever you selected is systematically higher than the true effect, not because of bad luck or bad practice, but because selection and noise interact in exactly this way, every time, by mathematical necessity — not team-specific bad luck.

The mechanism in two ideas

Regression to the mean. An unusually extreme result — in either direction — is more likely to be followed by a result closer to average than by another equally extreme result, because part of what made it extreme in the first place was a favorable noise draw that isn't guaranteed to repeat. This applies to that suspiciously excellent test result the same way it applies to a rookie athlete's outlier season, or a fund's best-ever quarter: some of it was real skill or real effect, and some of it was a draw from the tails of the noise distribution that won't reliably recur.

Selection effects. The winner's curse isn't just "noise exists." It's that the specific act of picking the best-looking option out of several actively filters for the cases where noise was most flattering. If you ran five variants and shipped the one with the highest measured lift, you didn't just measure that variant's true effect — you measured its true effect plus whatever noise happened to be most generous to it that week, because that combination is exactly what made it win the selection in the first place.

Put together: the more candidates you're choosing among (more variants, more segments, more looks at a test in progress, more hypotheses tried), the larger this inflation tends to be, because a bigger set gives noise more chances to produce an extreme, flattering outlier somewhere in the pile.

Why a shipped "win" often doesn't fully hold up

This is the direct link to the frustrating pattern every growth team eventually notices: a change ships based on a test result, and the real-world impact — measured honestly against a proper baseline months later — comes in lower than what the test predicted. Often meaningfully lower.

It's tempting to treat every one of these gaps as evidence something specific broke — a novelty effect wearing off, a competitor reacting, seasonality. Those explanations are sometimes correct, and they're worth checking. But even in a world where nothing external changed at all, part of the gap between predicted and realized lift is structurally guaranteed by the selection process itself. The test result was never a clean read of the true effect — it was the true effect plus a specific, favorable noise draw that got selected precisely because it was favorable. That noise component doesn't ship with the variant into production. Only the true effect does.

This matters most in exactly the situations that feel best at the time: a surprisingly large lift, a test that "barely" cleared significance right before a deadline, or a winner chosen from a large batch of variants or segments. These are the cases where the selection pressure — and therefore the expected inflation — is highest, which is close to the opposite of how they usually get reported internally.

What the fix actually looks like: calibration tracking

The fix isn't blanket skepticism toward every result, which just trades one bias for another (unjustified doubt instead of unjustified confidence). The fix is measurement: track predicted lift against realized lift, systematically, across every shipped test, over enough time to see the pattern rather than any single instance of it.

Approach	What it does	What it misses
Trust every test result at face value	Fast, simple, no extra process	Systematically overstates real impact; erodes trust with finance when results don't hold up
Blanket discount ("everything's probably overstated")	Cheap gut-check correction	Not calibrated to your specific program; could over- or under-correct
Calibration tracking (predicted vs. realized lift, logged over time)	Learns your program's actual typical inflation factor, grounded in your own data	Requires discipline to log every result and revisit it later — an easy step to skip when the team's already moving to the next test

Calibration tracking works like this: for every test you ship, record the lift the test reported at the moment of the ship decision. Then, independently, measure the realized impact against a proper post-ship baseline once enough time has passed for the number to stabilize. Do this consistently, across enough tests, and a pattern emerges — your program's own typical ratio of realized-to-predicted lift. That ratio is specific to your traffic levels, your typical number of variants per test, how often results get read under time pressure, and how your organization tends to make ship calls. It will not match another company's ratio, and it shouldn't be expected to.

Once you know your own typical inflation factor, you can use it honestly: report a shipped test's predicted lift alongside a discounted, more-likely-realistic figure, calculated from your program's own track record rather than a hopeful guess. That's a meaningfully more credible number to hand to finance than the raw test result, and it's also a more useful number internally — it keeps the team from over-resourcing a strategy based on a lift that was partly a statistical mirage.

The senior diagnostic most teams stop short of

Most teams that notice the predicted-versus-realized gap stop at the anecdote: "that one didn't hold up, weird." Very few build the actual longitudinal record that would let them answer the more useful question — is this gap a one-off, or is it the expected size of gap given how many variants we typically test and how our team makes ship decisions? Answering that requires treating your own historical test-to-reality gap as data worth collecting on purpose, not a mildly annoying anomaly to explain away test by test. That habit — logging the gap instead of narrating it — is the difference between a program that slowly gets better at forecasting its own impact and one that re-discovers the same surprise every quarter.

That habit is also the seed of a broader practice: systematically auditing your own program's track record isn't limited to lift inflation — it extends to which types of tests tend to hold up, which methodologies produce more reliable estimates, and where a program's blind spots cluster. That's a deeper subject in its own right, worth treating separately from the calibration mechanism covered here.

FAQ

Does the winner's curse mean A/B testing results can't be trusted?

No — it means a single result, especially one selected as "the best" from several options, should be read as somewhat optimistic rather than as an unbiased estimate of the true effect. The test still contains real signal. The fix is accounting for the known direction and rough scale of the bias, not discarding the method.

Is this the same thing as p-hacking?

Related but distinct. P-hacking usually refers to actively manipulating an analysis (trying different cuts of data, stopping at a convenient moment) to manufacture significance. The winner's curse happens even with completely honest, correctly-run tests — it's a structural consequence of selecting the best result from a noisy set, not a sign anyone did anything improper.

Does running fewer variants per test reduce the winner's curse?

Directionally, yes — the curse's magnitude scales with how many candidates you're selecting among, so a head-to-head test between fewer variants has less room for a flattering outlier to emerge purely from noise than a test with many simultaneous variants or many segment cuts. It doesn't eliminate the effect, but a smaller candidate set reduces it.

How long should we wait before comparing predicted lift to realized lift?

Long enough for the metric to stabilize past any short-term novelty effect and to capture at least one full relevant business cycle (weekly, seasonal, or otherwise, depending on what you're measuring). The specific window depends on your business, but the discipline matters more than the exact number — pick a consistent window and apply it the same way every time so your calibration data is comparable across tests.

Isn't discounting future estimates just another form of guessing?

No — the difference is that it's grounded in your own program's documented track record rather than an intuition. A discount factor derived from a real, logged history of predicted-versus-realized outcomes is a measured correction; an ungrounded gut-check discount is just a different unverified guess wearing a more humble tone.

Bottom line

The winner's curse isn't a flaw in your testing program — it's a predictable, well-understood consequence of selecting the best result from a noisy set, and it shows up in every serious experimentation practice that ships more than one variant or reads results under any time pressure. Expecting every shipped win to fully replicate in real revenue is expecting a level of precision the selection process was never capable of providing. The senior move isn't to distrust every result — it's to measure the gap on purpose, learn your own program's typical inflation factor, and report lift estimates that already account for it. That's a more credible number than the raw test result, and it's one your own history can actually justify.

If you're building or auditing an experimentation program and want an outside read on this, get in touch.

Decide What Counts as a Win Before You Test

Atticus Li — Thu, 23 Jul 2026 19:14:22 +0000

Decide What Counts as a Win Before You Test

Meta description: Medicine proved that picking your primary metric after seeing the data is a structural bias. The five-minute fix most experimentation programs skip.

TL;DR

A test finishes. The metric everyone agreed to watch is flat. But average order value moved, or a secondary funnel step improved, and the room quietly reframes what "the win" was — after seeing which number happened to move.
The cleanest large-scale evidence that this specific sequence is a real bias, not a harmless judgment call, comes from medicine: large NIH-funded trials that hadn't registered a primary outcome in advance reported a significant benefit far more often than the same kind of trials after registration became mandatory.
The fix isn't fewer metrics. It's naming, in writing, which one metric decides ship or kill, what bar it has to clear, and what happens for each outcome — before the test starts, when no result yet exists to make one candidate metric look more flattering than another.
Every other metric still matters. It just can't retroactively become the proof for a test it wasn't named to decide — it can only generate the next hypothesis.
The actual skill isn't precommitting itself. It's choosing, in advance, the metric that's genuinely closest to the business decision at stake — not the one that happens to be easiest to move.

A test wraps. The metric everyone agreed mattered when the test was planned — checkout conversion, say — is flat, within noise, a clean null. In the same readout, average order value ticked up. Or time-to-first-value improved. Or a downstream retention cohort looks slightly better, though it's early. Someone reasonably says: "isn't that actually what we cared about?" The room nods. The test gets written up as a win, on the metric that happened to move, and nobody in the room did anything that feels like misconduct. That's exactly what makes this failure mode durable — it doesn't require anyone to act in bad faith, only to make a completely natural decision in the wrong order.

The cleanest evidence this is a real bias, not a judgment call

Medicine ran, unintentionally, one of the largest natural experiments available on this exact question. Before the year 2000, researchers running large clinical trials were not required to publicly register which outcome they considered primary before the trial began. After 2000, prospective registration on a public registry — ClinicalTrials.gov — became standard practice for major trials. Kaplan and Irvin's 2015 analysis in PLOS ONE, "Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time," found that 17 of 30 trials (57%) published before 2000 reported a statistically significant benefit on their primary outcome — compared to just 2 of 25 (8%) published after 2000.

The biology being studied didn't change in 2000. The treatments, the patient populations, the underlying science — none of it shifted because a registry requirement went into effect. What changed was narrower and more specific: researchers lost the ability to look at everything they'd measured and, after the fact, designate whichever outcome looked best as "the" primary result. Prospective registration doesn't make anyone more honest in the moment they're analyzing data. It removes the moment of choice from after the data arrives and forces it to before, which is the entire mechanism. The authors are careful to note this is a strong association rather than an experimentally proven cause — other things about trial conduct changed across those decades too — but the size and direction of the shift, on exactly the outcome you'd predict from the mechanism, makes this some of the best real-world evidence available for a bias that's otherwise mostly discussed in the abstract.

The same mechanism, running quietly in a normal analytics stack

Almost no CRO or growth program operates with anything as formal as the pre-2000 clinical-trial process — but most operate with its structural equivalent. A test launches against a loosely stated goal ("improve checkout"), and a modern analytics stack surfaces a dozen or more numbers that moved by the time it's read out: conversion, AOV, time on page, scroll depth, a funnel step two clicks downstream, a retention cohort that's too early to trust but is right there in the dashboard anyway. Nothing was pre-registered as the metric. So when the obvious one is flat, there's no structural obstacle to the room settling on whichever of the other dozen moved in the direction everyone was hoping for. That's not a smaller, informal version of what pre-2000 trials did. It's the identical mechanism, running by default because nobody removed the moment of choice from after the results arrived.

It's worth being precise about how this differs from a related bias already worth watching for in any testing program: an already-selected "winning" metric being inflated relative to its true effect, simply because it was chosen as the best-looking result out of several noisy candidates. That's a real and separate phenomenon. This one happens earlier and is arguably more consequential — it's not about the size of the metric that won being overstated, it's about which metric gets to be called the win in the first place, chosen after the fact from whichever one happened to move. In practice the two compound: outcome-switching selects a flattering metric to call primary, and the standard selection-inflation problem then overstates how large that metric's true effect actually is. Two separate biases, stacking in the same direction, from two different moments in the same process.

The fix: name it before you look

The fix costs nothing to run and doesn't require new tooling — it requires deciding three things in writing before the test launches, not after the readout:

The one metric that decides ship or kill. Not the one metric you'll look at — the one metric whose result determines the decision. Everything else is still worth watching; it just doesn't get a vote on this test's verdict.
The bar it has to clear, stated as a specific direction and magnitude, not "it should go up."
What happens for each outcome, decided for all three realistic cases: it clearly clears the bar, it clearly doesn't, or it lands in the ambiguous middle. Naming the action for the ambiguous case in advance matters most, because that's the case most likely to trigger exactly the after-the-fact metric search this whole practice exists to prevent.

None of this requires ignoring the other numbers on the dashboard. A secondary metric moving in an interesting direction is a legitimate, valuable thing to notice — it's a candidate hypothesis for the next test, where it can earn the same precommitment treatment before being trusted. What it can't do is retroactively become the proof for the test that was actually designed and powered around a different question. Keeping that boundary explicit is the entire practice. Blurring it is how a null result quietly becomes a win without anyone deciding, on purpose, that it should.

The judgment call underneath the mechanics

Precommitting a metric is not, by itself, a hard skill — anyone can lock in a number in a planning doc. The actual skill is choosing, in advance, which metric is genuinely closest to the business decision this test is supposed to inform, rather than whichever one is easiest to move or most likely to look good.

A team optimizing for a good-looking readout will gravitate toward a shallow, upstream metric — a click, a scroll, a micro-conversion — because those move easily and often, and an easy win is more pleasant to report than an honest null. A team optimizing for a decision that will actually hold up gravitates toward the metric closest to the outcome the business cares about, even when that metric is noisier, slower to move, and more likely to come back flat. That choice has to be made before the test runs, precisely because it's much harder to make honestly once a shallow metric has already moved in a flattering direction and a deeper one hasn't. The senior version of this practice isn't the discipline of precommitting. It's the judgment of knowing which metric deserved the commitment in the first place.

FAQ

Doesn't locking in a metric early just mean I might miss something important?

It doesn't mean missing it — it means categorizing it correctly. Anything unexpected that moves is still visible, still worth discussing, and still a legitimate seed for the next test. Precommitment only determines what's allowed to close out this test's verdict. An interesting secondary movement earns its own precommitted test before it gets to be called a finding rather than an observation.

What if the primary metric is flat but something big and unexpected shows up elsewhere?

Treat it exactly as valuable as it is: a strong new hypothesis, not a result. Write it down, and if it's promising enough, design a test where it's the named primary metric next. What it shouldn't do is get reported as this test's outcome — that's the precise substitution this practice exists to prevent, and it's also usually the moment where a program's reported win rate quietly stops meaning what people assume it means.

How is this different from choosing when to stop a test?

They're separate decisions about separate questions. Stopping-rule methodology — covered in depth elsewhere in this series — governs when you're allowed to look at a result without inflating your false-positive rate. This is about what you're allowed to call the result once you do look — which of possibly many measured numbers gets to be the verdict. A test can have a perfectly rigorous stopping rule and still fall into outcome-switching if the primary metric wasn't named until after the data arrived.

How does this relate to deciding how much evidence a bet needs?

They answer different questions at different points in the process. The Confidence Tier Model helps size a bet against however much evidence you actually have, once you know what the evidence says. This practice determines what's allowed to count as "what the evidence says" in the first place. Skipping this step doesn't just risk a wrong tier assignment — it risks tiering a metric that was never the right one to be measuring the decision against.

Does this only apply to formal A/B tests?

No — it applies to any decision where more than one plausible metric could be used to declare success, which is most consequential business decisions, not just controlled experiments. A new hire's first-quarter review, a feature launch, a pricing change rolled out without a holdout — all of them have multiple candidate metrics available after the fact, and all of them benefit from the same fix: name the one that decides the verdict before you have a result that could bias the choice.

Bottom line

Medicine didn't fix its outcome-switching problem by asking researchers to be more honest. It fixed it by moving one decision — which metric counts — to before the data existed to bias it, and the reported result changed by a wide enough margin, at a large enough scale, to make the mechanism hard to argue with. Nothing about a growth or product team's version of this problem is different in kind, only in formality. Naming the metric, the bar, and the action for every outcome, in writing, before the test starts, is a five-minute habit with the same effect: it doesn't make anyone more honest in the room. It just removes the moment where honesty would have been the only thing standing between a flat result and a win that was chosen, not found.

If you're building or auditing an experimentation program and want an outside read on this, get in touch.

How to Tell If a Growth Hire Understands Risk

Atticus Li — Thu, 23 Jul 2026 19:14:21 +0000

How to Tell If a Growth Hire Understands Risk

Meta description: A great win story tells you almost nothing about judgment. Two borrowed interview probes — from forecasting research and intelligence tradecraft — do.

TL;DR

A confident story about a big win is one of the weakest signals available in a growth-leadership interview — it's vivid, it's memorable, and it tells you almost nothing about the judgment that produced it versus the luck that carried it.
Most interviews test for articulateness under a friendly, low-stakes question. Almost none test for the specific skill that actually separates a good experimentation leader from a confident one: how someone reasons under uncertainty, in real time.
Two techniques, borrowed from forecasting research and intelligence tradecraft, convert that skill from a vibe into something you can actually probe for in forty-five minutes.
Neither probe can be prepared for with a rehearsed answer, because neither asks the candidate to tell a story — both ask them to do something, live, that a good story can't substitute for.
The candidate with the thinner resume who passes both probes is very often the better long-term hire. The skill you're actually buying is how their next ten ambiguous calls go, not how their best past one turned out.

A hiring manager sits across from a growth-lead candidate who tells a great story: a redesign that lifted conversion, a pricing change that held up, a launch that hit its number. It's specific, it's confident, it's told with the easy fluency of someone who's told it many times before. Everyone in the room nods. It's also almost worthless as evidence, and not because the candidate is lying — because a vivid, well-rehearsed story about a past win is exactly the kind of input that's easiest to be fooled by and hardest to verify in the room. It could be the record of genuinely excellent judgment. It could be one lucky call, retold with the confidence of a pattern. From the interview seat, the two are indistinguishable, because the story format itself doesn't carry the information that would let you tell them apart.

That's a real gap, not a minor one. Growth and product leadership is a job that consists almost entirely of making calls under incomplete evidence — and most interview processes have no mechanism for testing that skill directly. They test for confidence, for fluency, for a track record that may or may not have been earned by the judgment being evaluated. What follows are two techniques that test the actual thing.

Why the resume story is the wrong evidence to weigh heavily

The reason a compelling war story is a weak signal isn't specific to hiring — it's a well-documented feature of how people process vivid information generally. A single, detailed, emotionally satisfying account is disproportionately persuasive regardless of whether it's been corroborated, because coherence feels like truth even when it isn't evidence of it. Intelligence analysts are trained specifically to resist this, because the same trap — a single vivid, uncorroborated source outweighing several duller, more reliable ones — is one of the best-documented causes of analytical failure. A hiring interview has the identical structure: one polished anecdote, delivered with total conviction, against no corroborating information at all.

The practical implication isn't to distrust every story a candidate tells. It's to stop treating story quality as a proxy for judgment quality, because the two are only loosely correlated — a genuinely lucky call and a genuinely well-reasoned one get told with the same confident cadence after the fact, especially by someone who's had years to polish the retelling. If the story is the only evidence on the table, you're not evaluating judgment. You're evaluating narrative skill, which is a real skill, just not the one the role actually requires.

Probe one: ask for a number, not a story

Forecasting research — most famously the Good Judgment Project's multi-year tournaments — has spent years measuring what actually separates good judgment from confident judgment, and the finding that matters most here is almost embarrassingly simple: people with genuinely good judgment under uncertainty routinely assign real probabilities to their beliefs and check those probabilities against what actually happened. Almost everyone else just remembers whether they were right, in the vague, self-flattering way memory tends to work.

The interview version: when a candidate describes a call they made with real conviction, stop them and ask two follow-up questions. First — "What percentage would you have put on that, at the time, before you knew the answer?" Not "were you confident," a number. Second — "How did you find out whether you were right, and when?"

Watch for three things. Whether they can produce an actual number instead of reaching for "pretty confident" — most candidates, asked for the first time, visibly struggle with this, and that struggle is itself informative. Whether the number sounds calibrated rather than reflexively extreme — someone who says "95%" about a genuinely uncertain call before they had evidence is telling you something different than someone who says "65%." And whether they have an actual answer to the second question, or whether the honest answer is that they moved on to the next project and never checked. That last gap is the most common finding, and it's the more important one — a candidate who has never once gone back and scored their own past confidence against reality has never built the one habit most reliably associated with good judgment in this specific line of work.

Probe two: give them an ambiguous drop and listen for the alternatives

The second technique comes from intelligence tradecraft rather than forecasting research, and it targets a different failure mode: not overconfidence, but premature certainty about why something happened.

Give the candidate a short, realistic scenario: "A core conversion metric dropped by a meaningful margin the week after a launch. Walk me through how you'd figure out what happened." Then just listen, and count how many genuinely distinct explanations they generate before they commit to one. The specific, structured version of this — Analysis of Competing Hypotheses: listing every plausible explanation before looking closely at any of them, then actively hunting for evidence that would rule each one out rather than evidence that confirms the first one — is a real technique taught throughout the intelligence community precisely because unaided judgment reliably jumps to the first plausible story and then spends its remaining effort defending it rather than testing it.

A weak answer commits early: "sounds like the new checkout flow broke something," followed by a description of how they'd confirm that specific theory. A strong answer holds off: the launch, yes, but also a tracking regression, a traffic-mix shift from a paid campaign change that landed the same week, a seasonal pattern, a competitor promotion — several candidate explanations, named before any of them gets investigated, with a specific idea of what evidence would rule each one out rather than just support the favorite. The gap between those two answers is not intelligence or experience. It's a discipline, and it's one of the more reliable predictors of whether someone's future root-cause calls will be trustworthy or just confidently plausible.

What each source of evidence actually tells you

Evidence source	What it reveals	What it can't tell you
A resume win story	That something good happened once, and that the candidate can narrate it persuasively	Whether the outcome reflected their judgment or favorable noise they got credit for
The calibration probe	Whether they've ever quantified their own confidence and checked it against reality	Domain expertise, or how they'll perform on a problem type they haven't faced before
The competing-hypotheses probe	Whether their default mode under ambiguity is to search for the truth or to defend the first plausible story	Whether they'll actually apply that discipline under real time pressure, not just in an interview

None of the three is sufficient alone. But the first one is the only one most interview processes actually collect, which is precisely backwards — it's the one that's easiest for a good storyteller to pass regardless of the judgment underneath it.

The diagnostic catch this surfaces

Here's the finding that makes this worth doing, stated plainly: a candidate with an excellent-sounding track record can fail both probes, and a candidate with a noticeably thinner one can pass both. That's not a paradox — it's exactly what you'd expect once you separate outcome from process. A strong track record can be a small number of calls that happened to land well, retold with the confidence of a pattern, by someone who's never once gone back to check their own calibration or practiced distinguishing a favorite explanation from a verified one. A thinner track record can belong to someone earlier in their career who nonetheless already has both habits — and habits, unlike a specific past win, are the thing that transfers to your business, under your constraints, on problems that look nothing like the ones on their resume.

That's the actual hire you're making. Not their best past call. How their next ten ambiguous ones are likely to go, somewhere you can't yet see, on evidence you haven't yet collected. A win story can't tell you that. A number and a list of ruled-out alternatives, produced live under a question they couldn't have rehearsed, comes a lot closer.

FAQ

Isn't this too abstract to actually use in a normal interview?

No — both probes are two follow-up questions, not a separate exercise. Ask a candidate for a story you'd ask for anyway, then push with "what percentage, at the time" and "how did you find out." Ask a candidate a scenario question you'd ask for anyway, then just count the alternatives before they commit. The discipline is in what you listen for, not in redesigning the interview.

What if the candidate has never explicitly tracked their calibration? Is that disqualifying?

Not by itself — almost nobody has, which is exactly the point of asking. What matters is what happens next: do they recognize the gap immediately and reason about why it matters, or do they deflect with a version of "I just have good instincts." The first response is a promising sign in someone who hasn't built the habit yet. The second is the more useful warning, regardless of how strong their resume looks.

Does this work for evaluating an existing team, not just a candidate?

Yes, and arguably it's more useful there, because you have real, current decisions to probe instead of a rehearsed history. Pick a live ambiguous result from the last quarter and ask the same two questions of whoever owned the call. The same gaps show up, and unlike an interview, you can act on what you find.

How is this different from a standard behavioral interview question?

A standard behavioral question ("tell me about a time you...") asks for a story, which is exactly the format that's easiest to prepare and hardest to verify. Both probes here ask for something a rehearsed story doesn't contain — a specific number produced in the moment, or a live count of alternative explanations — which is much harder to fake convincingly on the spot.

What's the actual pass/fail signal here?

There isn't a hard pass/fail — the value is comparative. A candidate who produces a real number and checks it, and who generates several genuine alternatives before committing to one, is demonstrating a specific, transferable discipline. A candidate who can only offer conviction and a single confident theory is demonstrating storytelling. Weigh the difference the same way you'd weigh any other job-relevant skill you tested for directly.

Bottom line

A great story is the easiest thing for a strong candidate to prepare and the hardest thing for an interviewer to verify — which makes it close to the worst evidence available for the judgment a growth-leadership role actually requires. Asking for a number instead of a feeling, and counting alternatives instead of accepting the first plausible one, tests something a rehearsed answer can't fake. What you're really buying isn't the candidate's best past call. It's how the next ambiguous one, on your business, is likely to go — and that's a specific, probeable habit, not a vibe you pick up from a good story.

If you're trying to tell whether a growth hire — or a fractional leader you're evaluating — actually has this judgment, not just a good story about it, get in touch.