DEV Community: Dron Pancholi

How to Audit a Vendor’s Backtest Before You Pay a Single Rupee

Dron Pancholi — Tue, 03 Nov 2026 07:23:50 +0000

You are not the one who built the strategy — you are the one being asked to pay for it. That makes you the auditor. Here is the buyer-side checklist, in order.

Almost everything written about backtest validation is written for the person building the strategy. But if you are considering paying for one — a bot, a signal service, a course strategy, a “proven” model — you are not the builder. You are the buyer. And the buyer needs a different checklist, because the builder’s incentives run the other way.

Here it is, in order. Any vendor who fails the first three is done.

1. Show me the raw predictions

Not a screenshot. Not an equity curve. The raw file: predictions with timestamps, and the actual outcomes. If you cannot recompute the accuracy yourself — or have it recomputed — from raw predictions, you have nothing verifiable. A claimed-accuracy column in a vendor’s file is the easiest number in finance to inflate, which is why any serious check ignores it and recomputes from the predictions themselves.

2. How many did you try — and where are the losers?

The trial count is the single most predictive question in this whole field. A vendor who tried 500 versions and shows you the best is showing you a lottery winner. Ask for the number, and ask to see the discarded versions. A vendor who cannot or will not answer has answered.

3. What do the dumb baselines score?

Always-bullish and no-change, on the same data. If the vendor’s model cannot beat both, it adds nothing — it is riding the base rate. On a bare CSV with no price history, those two baselines are the honest ones; anything claiming five baselines without price history is faking three of them.

4. Show me the folds, not the average

A single overall accuracy hides decay. Ask for walk-forward results fold by fold, earliest to latest. A model decaying from 80% to 56% across folds is a dying edge with a flattering average. Also ask which version of “walk-forward” they mean: period stability (accuracy across time periods) or purged, embargoed training (leak-proof fold construction). Vendors who say “walk-forward validated” without saying which have told you nothing precise.

5. What did it cost — in the backtest?

Commissions, spread, slippage, taxes. A backtest with zero costs is a fantasy. Ask for net-of-cost results, and ask at what slippage the edge dies (the break-even cost per trade). If the edge survives only at zero friction, it does not survive.

6. What market regime was this?

A strategy that only worked in one regime — one trend, one volatility environment — is a bet on that regime repeating. Ask for results sliced by year and by regime. Concentration in one slice is a warning, not a track record.

7. Can I forward-test before paying?

The strongest answer a vendor can give is: “Here is a forward-test window — paper-trade it live for 30 days and watch.” The weakest is pressure to pay now for a backtest you cannot reproduce. Time is the cheapest audit there is.

The sentence that ends most pitches

“I’d like the raw prediction file, the trial count, the baseline scores, and the fold-by-fold results. If those check out, I’ll forward-test before paying.”

Most vendors will not survive that sentence. The ones who do are the only ones worth a second conversation — and they are rare, which is exactly why the sentence works.

BuildIT Index exists so you can run that sentence mechanically: hand it a prediction file, get a recomputed accuracy, baseline scores, walk-forward folds, leakage gate, multiple-testing correction, calibration, and a verdict with an exit code. The tool does the audit; you make the decision.

The buyer who asks the right questions rarely needs to pay for the wrong answer.

This article is educational, not financial advice. Past backtest results do not predict future returns.

Originally published at startbuildit.online

MinTRL: How Long a Track Record Do You Actually Need?

Dron Pancholi — Tue, 03 Nov 2026 07:23:11 +0000

A Sharpe ratio of 2.0 on 18 months of data proves almost nothing. Minimum Track Record Length turns “impressive Sharpe” into “how many years would this take to believe” — with the numbers.

A strategy shows a Sharpe ratio of 2.0. Your instinct says “excellent.” The correct response is a question: over how long?

Minimum Track Record Length (MinTRL) is the answer to that question, formalised. It asks: given an observed Sharpe ratio, how many years of track record would you need before you could believe, with 95% confidence, that the true Sharpe is above zero — that the strategy is better than nothing at all?

The idea comes from the same research tradition as the deflated Sharpe ratio (Bailey and López de Prado), and it is brutally clarifying, because it converts a sexy number into a boring one: years.

Why short track records prove so little

Returns are noisy. A high Sharpe over a short window can easily be luck — a few good months, a favourable regime, a lucky draw. The shorter the window, the wider the uncertainty around the Sharpe, and the longer the track record you would need to shrink that uncertainty to something believable.

The relationship runs the way you would hope it does not: the more modest the Sharpe, the longer the record needed. As an anchored example from the reference literature, a Sharpe of around 0.95 on daily data needs roughly 3 years of history to clear the 95% confidence bar. Lower Sharpes need longer; higher Sharpes need less — but even a Sharpe of 2.0 on a year and a half of data is thinner evidence than it looks, and once you correct for the number of strategies tried (the multiple-testing problem), the required length grows further. In some published multi-market reviews, the required length for the observed Sharpes was effectively infinite — the data could not support the claim at any reasonable horizon.

The trial-count trap inside MinTRL

Here is the part vendors never mention: MinTRL assumes the Sharpe you see is the Sharpe of one strategy, honestly measured. If the vendor tried 200 strategies and shows you the best, the relevant Sharpe is deflated first — and the deflated Sharpe needs a longer track record than the advertised one. A Sharpe of 2.0 that deflates to 0.6 under a multiple-testing correction does not need months of belief. It needs years.

So the two questions always travel together: “What is the Sharpe?” and “How many did you try?” A MinTRL quoted without a trial count is a half-answer.

How to use MinTRL tomorrow

When a vendor quotes a Sharpe, ask for the track record length in years and the number of trials. Then ask what MinTRL their own numbers imply. If they cannot answer, they have not done this math — and you should do it for them, conservatively.

Treat any Sharpe above 3.0 on a short record as a red flag, not a selling point. Extreme Sharpes on short samples are the signature of overfitting, not genius.

Remember that MinTRL is about statistical belief, not profitability. A strategy can clear MinTRL and still lose money to costs, slippage, and regime change. It is one gate among several: recompute, baselines, walk-forward folds, leakage, trial correction, and then track-record length.

The honest summary: most strategies you are shown have not existed long enough, on their own numbers, to be believed. MinTRL just makes that sentence quantitative — and once you see it in years, you cannot unsee it.

A Sharpe of 2.0 on 18 months is a claim. A Sharpe of 2.0 on 5 years with a known trial count is evidence. MinTRL turns the first into the second.

This article is educational, not financial advice. Past backtest results do not predict future returns.

Originally published at startbuildit.online

Backtest vs live: why execution friction kills good-looking strategies

Dron Pancholi — Tue, 03 Nov 2026 07:16:20 +0000

Your strategy’s logic was never the problem. Latency, slippage, and costs are the gap between a beautiful backtest and a losing live account — and the gap is measurable.

Assume your edge is real. Assume it survives out-of-sample testing, beats its baselines, and holds up across market regimes. It can still lose money live. The reason is not mysterious, and it is not your logic. It is execution friction — the gap between the price your backtest books and the price the market actually gives you.

Latency becomes slippage becomes losses

A backtest books the decision price as the fill price. Live markets deliver neither instant fills nor requested prices. Latency is the delay between signal and fill; slippage is the worse price you get because the market moved during that delay. They are one chain, not two problems: cut the delay, and most of the slippage goes with it.

The damage is invisible per trade and only shows up in aggregate — which is why it surprises people. Take a system backtesting at a small average profit per trade. Add routine retail slippage per round trip and the edge shrinks by more than half. Push slippage higher in a volatile session and expectancy turns negative. Across a year of trades, ordinary slippage alone can erase a year’s expected profit. The strategy did not break. The market charged it rent the backtest forgot to include.

The full friction list

Slippage and latency, worst during news and high volatility — exactly when backtests stay silent.
Brokerage, taxes, and exchange charges, per trade, every trade.
Spread and market impact, which grow with size and thin liquidity.
Partial fills and missed trades, which quietly change the strategy you thought you were running.
Outages and disconnections, which turn an automated strategy into an unmonitored bet while positions are live.

How to close the gap before risking capital

Model friction explicitly. Re-run the backtest with realistic slippage and full costs. If the edge survives, it is an edge worth discussing. If it does not, you have learned something cheaply.
Measure expectancy net of everything. Gross profit per trade is a vanity metric. Net expectancy per trade, after all friction, is the number that decides whether to trade.
Forward-test on paper first. Live data flow exposes latency, fill inconsistency, and data quirks that no historical simulation contains. Backtesting tells you if a strategy could have worked; forward testing tells you if it actually functions.
Watch the backtest-live divergence. A strategy whose live results shrink steadily against its backtest, with no explanation beyond “market conditions,” is usually confessing over-optimisation after the fact.

The sequence that works

Backtest to screen. Validate to filter — recompute, baselines, walk-forward folds, leakage, trial-count correction. Forward-test to confirm function. Then, and only then, trade small. Each stage exists to kill a different kind of false confidence, and skipping one means paying for its lesson with capital instead of time.

The validation stage is where BuildIT Index sits, and where the seven checks apply. It does not test execution, pick models, or promise a future return. It answers a narrower question that has to be answered first: does the record in the file support the number on the page?

If the answer is no, the live account was never the place to find out.

This article is educational, not financial advice. Past backtest results do not predict future returns.

Originally published at startbuildit.online

How to Audit Someone Else’s Backtest in 10 Minutes

Dron Pancholi — Tue, 03 Nov 2026 07:15:44 +0000

Four questions expose most weak backtests in minutes: trial count, costs, sample span, and what chose the parameters. Plus the baseline test every claim must pass.

You do not need a quant degree to audit a backtest. You need four questions, asked in order, and the willingness to walk away when the answers are vague. Most weak backtests fail at least two of these in the first ten minutes.

Question 1: How many trials produced this result?

If a vendor tried 500 parameter combinations and shows you the best, the headline number is partly luck. The more alternatives explored, the less a naive performance statistic means — unless it is adjusted for multiple testing. Honest answers sound like “we tried 12 configurations, here is the correction.” Evasive answers sound like “our AI found the optimal settings.” Count the trials, or discount the claim to zero.

Question 2: Were costs modelled realistically?

Commissions, spread, slippage, borrow costs, market impact. A backtest that books fills at the decision price is testing a market that does not exist. Ask for net-of-cost results specifically. If the edge disappears under realistic friction, there was no edge — there was an accounting choice. Execution friction is measurable, so an honest backtest can include it.

Question 3: Does the sample span enough market conditions?

A strategy tested only on a bull market has not been tested; it has been photographed in good light. Demand results sliced by year and by regime — uptrend, downtrend, high and low volatility. Concentration of returns in one narrow slice is a warning sign, not a track record. And remember: a strategy that looks great on six trades is a small sample wearing a big number. Six trades is a caution flag, not a victory lap.

Question 4: Did the same data choose and validate the parameters?

This is the fastest way to expose a weak backtest. If the full dataset informed the design, the features, or the tuning, then the “validation” on that same data is contaminated — even if it carries a validation label. Parameters must be chosen on training data and judged on data that played no part in choosing them. Purged, embargoed folds are the strict version of this; at minimum, demand genuine out-of-sample periods.

The baseline test (30 seconds)

Ask what the dumbest rule scored on the same data. If a directional model claims 70% accuracy but always predicting “up” scores 68%, the model added almost nothing. On a bare CSV with no price history, two honest baselines exist: always-bullish and no-change. Any claim that cannot beat those has nothing to say, however sophisticated the model sounds.

What a clean audit looks like

A credible result survives this sequence: accuracy recomputed from raw predictions (claimed columns ignored), baselines beaten, walk-forward folds stable rather than decaying, leakage gate passed, trial count disclosed and corrected for. That is the standard BuildIT Index applies mechanically — it returns a verdict and an exit code, so a failing backtest fails a pipeline instead of persuading a meeting.

Print these four questions and keep them beside every pitch deck, vendor demo, and course sales page. They will save you more money than any indicator ever will.

No trial count, no net-of-cost results, no out-of-sample period, no baseline. Any one missing answer is enough to stop the conversation.

This article is educational, not financial advice. Past backtest results do not predict future returns.

Originally published at startbuildit.online

PBO, Explained in Plain English: The Probability Your Best Backtest Is a Fluke

Dron Pancholi — Tue, 03 Nov 2026 06:09:23 +0000

You tested 8 strategies and one looks brilliant. PBO asks the question most backtests never face: among everything you tried, how likely is the winner just the luckiest loser?

You ran 8 versions of a strategy. Seven were mediocre. One was brilliant. You ship the brilliant one. Question: did you find an edge, or did you run a lottery and crown the winning ticket?

That is exactly what PBO — the Probability of Backtest Overfitting — measures. And it is one of the least explained ideas in trading, because the people who understand it write papers, and the people who write blog posts mostly skip it. Here it is in plain English, with a worked example.

The core idea

PBO does not ask “is this strategy good?” It asks something sneakier: if you pick the best performer from a set of candidates, how likely is it to disappoint out of sample? It answers by splitting history into many pieces, ranking the candidates in each piece, and checking whether the in-sample winner keeps winning out of sample — or whether the winner keeps changing, which is what luck looks like.

The formal method is called Combinatorially Symmetric Cross-Validation (CSCV). You do not need the maths to use the intuition: a real edge is stable. A fluke is a rotating cast of winners.

A worked example

Imagine 8 configurations of the same strategy, tested over 4 years.

In-sample, configuration #5 wins: Sharpe 2.1, best by a clear margin.

Split history into halves, quarters, and many combinations. In most out-of-sample slices, #5 is mid-pack. The winner changes from slice to slice: #2 here, #7 there, #5 rarely.

PBO comes out at 0.62. Interpretation: a 62% chance that the in-sample winner underperforms out of sample — the selection process itself is likely what produced the “winner.”

Now the same setup, different outcome: configuration #3 wins in-sample, and it also ranks near the top in most out-of-sample slices. The winner is stable. PBO comes out at 0.14. That does not prove the strategy will make money — nothing proves that — but it does mean the selection was probably not a lottery.

As rough bands used by validation services: below about 0.20 looks clean, 0.20–0.50 is caution, above 0.50 means the winner is more likely than not a fluke. Treat the bands as guidance, not law.

The two honesty notes nobody states

First, PBO needs a candidates file — the full set of versions you tried. With a single model and no candidates, PBO is null. It is not zero, it is not “passed,” it is undefined, and any tool that reports a PBO for one lonely model is inventing a number. This is why BuildIT Index states it plainly: PBO is null for single models, and never a gate.

Second, PBO is only as honest as the candidate list. If you tried 200 versions and hand over 8, the computation is being fed a curated history. The trial count problem never goes away; it just moves. Demand the full list, or discount the score.

How to use PBO

Before you trust a “best” strategy, ask for its PBO and the candidate count behind it.

If PBO is above 0.50, the correct response is not to tune harder — it is to suspect the whole selection process and start over with fewer degrees of freedom.

If a vendor cannot produce a PBO at all, that is not a fail — but ask why, and ask what else they ran instead: walk-forward folds, deflated Sharpe, a leakage gate. One missing check is fine. All of them missing is the answer.

A backtest that survives recomputation, beats its baselines, holds up fold by fold, and carries a low PBO has earned a forward test. A “winner” with a high PBO has earned nothing but suspicion — and suspicion, in this business, is free.

A backtest that survives every check has earned a forward test. A backtest that fails one has saved you the money you would have spent learning it live.

This article is educational, not financial advice. Past backtest results do not predict future returns.

Originally published at startbuildit.online

Why Most Backtests Lie — and the 7 Checks That Catch Them

Dron Pancholi — Tue, 03 Nov 2026 05:04:52 +0000

A beautiful backtest is the easiest thing in trading to fake — usually by accident. Here are the seven ways backtests mislead, and the check that catches each one.

A backtest is a story about the past, told by someone who already knows how it ends. That is why it lies so well — usually without anyone intending to lie.

Every trader has lived the same disappointment: a strategy that looked unstoppable in testing falls apart the moment real money touches it. The usual explanation is “the market changed.” Sometimes that is true. More often, the backtest was never evidence in the first place. It was a fitting exercise that found noise and dressed it as signal.

Here are the seven ways backtests mislead, and the check that catches each one.

1. Overfitting: tuned to noise, not signal

Try enough parameter combinations and one of them will look brilliant by pure chance. That is not a discovery; it is what happens when you search long enough. The more variants you tried, the less the best one means — unless you correct for the number of trials. Ask one question: how many configurations were tested before this one was shown? If nobody counted, the headline number is uninterpretable.

2. Look-ahead bias: tomorrow’s information, today’s decision

The classic version uses a price or indicator value that was not available at decision time — tomorrow’s open to decide today, a revised earnings figure in place of the original. It is usually a one-line indexing mistake, not fraud, and it is devastating: a strategy that “predicts” with future data will fail on day one of live trading.

3. Survivorship bias: the losers are missing

Backtest only on stocks that exist today and you have silently deleted every bankruptcy and delisting from history. The market you tested never contained real failures, so it looks far kinder than the market you will actually trade.

4. Data snooping: mining until something works

This is overfitting’s quieter cousin. You did not tune parameters — you tried indicators, universes, and time windows until a pattern appeared. A pattern found by searching thousands of possibilities is not the same as a pattern predicted in advance. The check is the same: count the trials, and demand out-of-sample confirmation.

5. Ignored costs: the edge that fees eat

A backtest that assumes fills at the decision price, with no spread, slippage, brokerage, or taxes, is testing a fantasy market. Small per-trade frictions compound: a strategy earning a thin gross edge per trade can go negative once realistic costs land on every single trade. If the edge does not survive costs, there is no edge.

6. Regime dependence: one good period doing all the work

A strategy can look strong in aggregate while depending entirely on one favourable stretch — one year, one volatility environment, one trend. Slice the results by year and by market regime. If the edge lives in a single slice, you do not have a robust strategy; you have a bet on that regime repeating.

7. No baseline: beating nothing

A model that claims 70% accuracy sounds impressive until you learn that always predicting “up” scores 68% on the same data. Every result needs a baseline: what would the dumbest possible rule have scored? On a bare CSV with no price history, two honest baselines exist — always-bullish and no-change. If a model cannot beat those, it has nothing to say. The harness runs those two baselines on every bare CSV, and never fakes the three that need price history.

The real purpose of a backtest

Here is the reframe that matters: a backtest’s true value is not validating winning strategies. It is discarding losing ones. A strategy that survives recomputation from raw predictions, beats its baselines, holds up fold by fold across time, passes a leakage gate, and survives a multiple-testing correction has earned the right to be forward-tested. One that fails any of those checks has saved you money — that is a successful backtest too.

That is exactly what BuildIT Index does: it recomputes accuracy from raw predictions (any claimed-accuracy column is ignored), checks the result against baselines, measures walk-forward consistency fold by fold, and returns a verdict with an exit code — so a bad backtest can fail a pipeline instead of a meeting.

In its verified demo, a sample model claimed 78% and recomputed to 69.33% (416 of 600), with folds decaying from 80.00% to 55.83%. Verdict: OVERFIT. The demo is the point: the tool’s job is to say no.

A backtest that survives every check has earned a forward test. A backtest that fails one has saved you the money you would have spent learning it live.

Print these seven checks and keep them beside every pitch deck, vendor demo, and course sales page. They will save you more money than any indicator ever will.

This article is educational, not financial advice. Past backtest results do not predict future returns.

Originally published at startbuildit.online