DEV Community: Bot LAB

How to Detect Look-Ahead Bias in Your Own Backtest

Bot LAB — Wed, 14 Oct 2026 04:06:23 +0000

Look-ahead bias is the failure where your backtest uses information that did not exist yet at the moment it made a decision. You detect it mechanically, not by reading your code: compute your indicator twice — once over the full price history the way a vectorised backtest does, and once bar by bar on history truncated at that bar the way a live bot sees it — then compare the two values at the same bar. If they differ anywhere, something in your pipeline peeked at the future. That is the whole method, and it takes about ten lines.

This matters because look-ahead is the one backtest failure that never announces itself. Overfitting at least leaves a trail: good in-sample, bad out-of-sample. A leak looks like skill in every period, including your out-of-sample window, because the leak travels with the data. Reading the code does not reliably catch it either — the leak is usually not in a line that says shift(-1). It is in a .mean() over a whole column, a resample that joins a daily close onto hourly bars, a center=True you typed six months ago, or a gap-filling call that borrows the next bar's price.

The test

python full = signal(df) # vectorised, whole file for i in sampled_bars: inc = signal(df.iloc[:i+1]) # only data up to and including bar i assert full[i] == inc[-1] # any difference = look-ahead

Two details make it honest. First, truncate at the bar, not at the row index of some already-merged frame — if your feature table was built by joining timeframes, rebuild the join inside the truncated call. Second, sample bars from across the whole history, not just the end; some leaks only fire where the data has gaps, or where a higher-timeframe boundary falls.

What it looks like on real data

We ran this on 3,000 closed BTCUSDT 1-hour candles from Bybit (2026-06-11 03:00 UTC to 2026-10-14 02:00 UTC), auditing 120 randomly sampled bars with lookahead_analysis from our MIT-licensed library. Each row is a signal written the way people actually write signals.

| Signal construction | Bars checked | Mismatches | Leaks? | |---|---|---|---| | EMA(20) over EMA(50) cross | 120 | 0 | no | | z-score vs trailing 200-bar mean/std | 120 | 0 | no | | close.shift(-1) > close | 120 | 64 | yes | | Daily close joined onto 1h bars, not shifted | 120 | 57 | yes | | rolling(21, center=True).mean() | 120 | 32 | yes | | Min-max normalised over the whole file | 120 | 31 | yes | | z-score vs full-sample mean/std | 120 | 4 | yes |

Three things are worth reading off that table.

The obvious bug (shift(-1)) and the subtle one (a daily close joined onto hourly bars without shifting it forward) are equally fatal: 64 and 57 bad bars out of 120. The second is far more common in real code, because resampling and merging feels like data plumbing rather than a modelling decision.

Normalisation is a leak. Min-max scaling over the whole file mismatched on 31 of 120 bars, because the minimum and maximum of a column are facts about the future. Anyone who has scaled features before splitting train and test has done this.

A leak can be small and still be a leak. The full-sample z-score mismatched only 4 times in 120 bars — the mean and standard deviation of four months of hourly closes barely move when you add one more bar. It is still disqualifying: the value at bar i depends on bars after i, so the number your backtest traded on is a number your bot cannot compute.

The boolean trap

Here is the result that changed how we run this check. We took the same series with every seventh bar deleted, filled the holes with bfill() — backward fill, which copies the next available price into the gap — and computed an EMA(20) over it.

| What we audited | Bars checked | Mismatches | |---|---|---| | The EMA value itself | 120 | 15 | | The boolean close > EMA built from it | 120 | 0 |

Same bug, same bars. Audit the final true/false signal and the test passes. Audit the number underneath and the leak is plainly there. A boolean is a lossy summary: a contaminated value has to cross a threshold before the comparison flips, so most of the corruption is invisible at the signal level — until a day when it is not. Audit every intermediate value you compute, not only the entry flag.

Multi-timeframe and warm-up

If your strategy reads a 4h or daily candle while deciding on a 1h bar, the rule is simple: every higher-timeframe row you touch must have closed by the close of the bar you are evaluating. A partial candle is legal only if you build it yourself from lower-timeframe candles that have closed. Most exchange APIs return the currently-forming candle as the first element of the response, so this bug gets written by accident constantly. Our own fetch helper drops any candle whose close time is in the future, and the audit then checks the surviving rows against the decision timestamp.

A related failure produces the same symptom — backtest results your bot cannot reproduce — without any future data at all. Long indicators depend on how much history you loaded. Computing EMA(200) on the real BTC series above with only 100 bars of warm-up, versus 1,000 bars, moved the value by a median of 0.48% and up to 2.18%. That sounds harmless until you look at the trend boolean it feeds: close > EMA(200) flipped on 25% of the bars we evaluated (n = 20 evaluation bars, so treat the exact percentage as indicative, not precise). With 300 bars of warm-up the drift fell to 0.07% median and nothing flipped. Your backtest loads years of history; your live bot usually loads whatever its first API call returned.

What a leak is worth

To put a size on it, we ran one trading rule twice on the same 3,000 bars, with the same exits (stop at max(ATR14, 0.6%), target 1.5x the stop, 25-bar time stop) and the same cost assumption (0.055% taker fee per side, charged in R). The only difference is the entry condition: the second run compares the next close to the current one.

| Entry condition | Trades | Mean R | Win rate | Total R | |---|---|---|---|---| | EMA(20)/EMA(50) cross (clean) | 24 | +0.044R | 50.0% | +1.1R | | Next close > current close (leak) | 232 | +0.240R | 57.8% | +55.6R |

One bar of hindsight turns a rule with no detectable edge into one that looks like a business. The trade counts differ — the leaky condition triggers far more often — so this is an illustration of magnitude, not a controlled comparison. The direction is the point: leaks do not nudge results, they manufacture them.

What this method does not prove

A clean look-ahead audit is a licence to keep testing, nothing more.

It does not prove your edge is real. It proves only that your numbers were computable at the time. Overfitting, too few trades, concentration in two coins and ignored costs are separate failures with separate tests. For scale: across the 128 strategy books we track, as of 2026-10-14, zero pass our six readiness gates — and almost all of them are look-ahead clean.
It only covers the code you hand it. If the leak lives in how your trade ledger was assembled, or in a cached feature file built earlier over the whole history, auditing the signal function will not see it.
It is a sampled test. We checked 120 bars, not 3,000. Zero mismatches means "no leak at those bars", which is strong evidence, not proof. Increase the sample when a result looks surprisingly good.
Your data vendor can leak. Prices revised after the fact, timestamps labelled with a candle's close instead of its open, or funding rates stamped at the start of a settlement period rather than the end, will all pass a self-consistency check and still be wrong.
Nothing here says anything about fills. A signal that was computable at 14:00 can still be impossible to fill at the price your backtest assumed.

What this means for your own bot

Write a signal(df) -> one value per bar function with no hidden state and no reads outside df. If you cannot, that is your first finding.
Run the full-versus-truncated comparison on at least 100 bars spread over your whole history. Any mismatch stops everything else until it is fixed.
Audit values, not booleans. Every indicator, every normalised feature, every merged higher-timeframe column — and the entry flag last.
Grep for the usual suspects as a second pass, never as the first: shift(-, center=True, bfill, backfill, .interpolate(), resample(...).last(), and any .mean(), .std(), .min() or .max() taken over a full column.
Drop unclosed candles at the source. Timestamp every input with the moment it became available, and compare that timestamp to the decision time rather than trusting row order.
Fix warm-up before comparing backtest to live. Load the same amount of history live as your backtest effectively used, and measure the sensitivity rather than assuming it is small.
Re-run the audit after every change to the feature pipeline. Leaks arrive with refactors, not with new ideas.

The Python implementation of all of the above — the full-versus-truncated audit, the warm-up sensitivity test, the multi-timeframe closed-candle check, plus the cost model and the statistical gates we use on our own books — is MIT-licensed at github.com/botlabtrade/botlab-oss. Install numpy and pandas, point it at your own data, and you get the same report we run on ours. If you would rather not install anything, there is a browser version of the checks at botlab.org.uk/#tool, which runs locally on a CSV of your trades.

This is educational material about testing methodology. Nothing here is financial advice, and no result above is a live trading record.

Paper trading results are simulated. Past results don't predict future results. Crypto trading can lose money, including all capital. Not financial advice.

Originally published on BOTLAB. Score your own bot's trade log free: Check your bot.

NEAR Protocol Trading Bot: What to Test First

Bot LAB — Wed, 14 Oct 2026 04:06:22 +0000

Before you point a trading bot at NEAR Protocol or any other coin that is trending right now, test four things first: real trading costs at today's spreads, a large enough sample of trades, how the strategy did on data it never saw, and how it behaves when the hype fades. A backtest that only looks good during a hot streak usually reflects the streak, not an edge.

This guide shows how to run those checks step by step. It does not say anything about where NEAR's price is going, and it won't. The goal is to judge the bot, not the coin.

Why trending coins are a trap for backtests

When a coin like NEAR Protocol, Injective or Optimism shows up on trending lists, three things usually change at once:

Volatility jumps. Bigger candles mean bigger wins and bigger losses per trade.
Volume and attention change quickly. Order books can look deep at one moment and thin the next.
The recent past is unusual. A bot tuned on the last few weeks is tuned to an unusual regime.

A strategy that "would have made money" on a trending coin may only be showing that the coin moved a lot in one direction. That is market beta, not skill. A simple buy-and-hold over the same window would often show the same thing.

Step 1: Count the trades before you look at returns

The first number to check is not profit. It is the trade count.

A trending window is often short, so a bot may only have taken 15–30 trades on it. With that few trades, one or two lucky outliers can create the whole result. We cover this in depth in How Many Trades Does a Backtest Need?. In short:

Under ~30 trades: treat any result as anecdotal.
30–100 trades: early signal at best, wide uncertainty.
Several hundred trades across different market conditions: now you can start comparing.

If the coin doesn't have enough history for a meaningful sample, test the same rules across several similar coins and pool the results. If the rules only work on one hot coin, that is a warning sign.

Step 2: Put realistic costs in first

Costs decide whether a strategy survives, especially on fast timeframes. During a trend spike, two costs rise:

Spread and slippage. Your fill price can be noticeably worse than the candle close your backtest assumed.
Funding (for perpetual futures). When a coin is crowded on one side, funding payments can become a real drag on positions held for hours or days.

Fees in USDT add up quickly on small accounts too. Always use your exchange's actual fee tier, and assume taker fees unless your bot genuinely posts limit orders that get filled. Our Lab News post Costs come first explains why we subtract costs before looking at anything else.

Step 3: Split the data and test out-of-sample

Never judge a bot on the same data used to choose its settings. A basic honest setup:

In-sample (e.g. first 60–70% of the data): pick parameters here.
Out-of-sample (the remaining 30–40%): run once, no tweaking.
Walk-forward (better): repeat the split on rolling windows, so every period gets tested on data the parameters never saw.

If performance collapses out-of-sample, the strategy was probably fitted to noise. The more parameter combinations you tried, the more likely the best one looks good by chance. That is what the Deflated Sharpe Ratio is designed to correct for.

Step 4: Check the regime, not just the total

Split results by market condition:

Trending up
Trending down
Sideways / choppy
High-volatility spikes (the "trending coin" moments)

A trend-following bot will often look great in a strong move and bleed slowly in sideways markets. A mean-reversion bot often does the opposite. Neither is wrong, but you need to know which one you have, and what happens to it when the trending phase ends.

Step 5: Look past win rate

A high win rate can hide a bad bot if the few losses are large. A low win rate can be fine if winners are much bigger than losers. Look at these together:

Profit factor (gross wins ÷ gross losses, after costs)
Average win vs average loss
Max drawdown and how long it took to recover
Longest losing streak

We compare the first two in Profit Factor vs Win Rate for Trading Bots.

Worked example (illustration only)

The numbers below are made up to show the method. They are not BOTLAB results and do not describe any real bot or coin.

Imagine a 1-hour breakout bot backtested on a trending altcoin over six weeks:

Trades: 60
Average gross result per trade: +0.60%

Now subtract realistic costs, assuming:

Fees: 0.06% per side → 0.12% round trip
Slippage during volatile hours: 0.15% per side → 0.30% round trip
Total cost per trade: 0.42%

Net per trade: 0.60% − 0.42% = +0.18%. Two-thirds of the apparent edge disappeared.

Next, split the trades by time:

First 40 trades (in-sample, during the strongest trend): +0.45% net per trade
Last 20 trades (out-of-sample, after the trend cooled): −0.36% net per trade

The "profitable" backtest was really one strong move. On data it hadn't seen, the bot lost money after costs. With only 20 out-of-sample trades, you can't even say confidently how bad it is. The honest conclusion: not proven, keep paper trading or drop it.

Step 6: Paper trade on live candles before any real money

A backtest replays the past. Paper trading (simulated money, real market prices) runs the bot forward on candles it has never seen, with no chance of hindsight leaking in. It also exposes practical problems a backtest misses: delayed signals, missed fills, API errors and funding at the moment it actually applies.

Our view is simple: every bot flies the simulator first. For a trending coin, paper trade long enough to see the trend end, not just while it's hot.

How we judge bots at BOTLAB

We run every bot through 6 statistical gates (G1–G6), defined on the site. A bot is only marked "ready for micro live" after passing all six.

Passing the gates isn't the same as being proven. Of the 159 BOTLAB bots tracked when the site was last built (2026-10-14), 1 is marked ready for micro live, 1 is on watch and 157 are not ready. The one that passed, mr4h/long, has 80 paper trades from 2026-08-16 to 2026-10-13 (58 days) with a 73.8% win rate, and our own verdict is still "sample too small to conclude." Another bot, Atlas/ichimoku-4h, passed 4 of 6 gates over 229 paper trades (2026-08-08 to 2026-10-13) and is flagged as a suspected overfit. Those are paper trading results, and they don't predict future results.

If you have your own bot's trade history, you can run it through the same kind of checks with the free tool: Check your bot.

A quick checklist before running a bot on a hot coin

[ ] At least several dozen trades, ideally hundreds across different conditions
[ ] Fees, slippage and funding included at realistic levels
[ ] Out-of-sample or walk-forward results reviewed, not just in-sample
[ ] Results split by regime (trend, chop, spike)
[ ] Profit factor, drawdown and losing streaks checked, not only win rate
[ ] Paper traded on live candles, including after the trend cooled
[ ] Exchange API key set to trade-only, withdrawals off, IP-restricted
[ ] You've decided in advance the maximum loss you can accept, including losing all of it

FAQ

Is it a good idea to run a trading bot on a trending coin like NEAR Protocol?

That depends on the bot, not the trend. A trending coin gives you a short, unusual sample with higher costs and sharper reversals. Test the rules out-of-sample and across several coins before trusting any result. This guide makes no prediction about NEAR or any other coin.

How long should I backtest a bot on a new trending coin?

Long enough to include more than one market regime and enough trades to be meaningful. If the coin's history is too short, test the same rules on similar coins with longer histories and pool the trades.

Why does my bot look profitable in the backtest but lose in paper trading?

The most common reasons are underestimated costs (especially slippage in volatile hours), parameters fitted to the in-sample period, and a regime change after the backtest window. Paper trading on live candles is designed to catch exactly this.

Does a high win rate mean the bot is safe?

No. A high win rate with occasional large losses can still lose money overall and produce deep drawdowns. Check profit factor, average loss size and max drawdown alongside win rate.

Key takeaways

Judge the bot, not the coin. Trending markets inflate backtests.
Count trades first; small samples prove very little.
Subtract fees, slippage and funding before reading any result.
Only out-of-sample and forward (paper) results count as evidence.
Even a bot that passes every gate can still have too small a sample to conclude anything.
Use trade-only API keys with withdrawals off, and never risk money you can't afford to lose.

Paper trading results are simulated. Past results don't predict future results. Crypto trading can lose money, including all capital. Not financial advice.

Originally published on BOTLAB. Score your own bot's trade log free: Check your bot.

Deflated Sharpe Ratio Explained: Why Your Backtest Looks Better Than It Is

Bot LAB — Mon, 12 Oct 2026 16:06:37 +0000

A Sharpe ratio tells you how much return a strategy produced per unit of wobble. The Deflated Sharpe Ratio asks a harder question: given how many versions of this strategy you tried before picking this one, how impressive is that number really?

The gap between the two is where most backtests quietly die. This guide explains what the deflation does, how to compute it, and what number you should expect your own bot to produce.

The problem in one example

Take a strategy with no edge at all. Now test 50 variants of it: different moving-average lengths, different stop distances, different hours of the day. Keep the best one.

That winner will have a good-looking equity curve and a respectable Sharpe ratio. Not because it works, but because you ran 50 lotteries and kept the winning ticket. The ordinary Sharpe ratio has no way of knowing this happened. It sees one strategy and one track record, and it answers the question you asked rather than the question you should have asked.

The Probabilistic Sharpe Ratio (PSR) and the Deflated Sharpe Ratio (DSR), both from work by Marcos López de Prado and David Bailey, close that hole.

PSR: is this Sharpe ratio distinguishable from zero?

PSR(0) is the probability that a strategy's true Sharpe ratio is above zero, given the Sharpe you observed, the number of observations, and the shape of the return distribution.

It rises with sample size, and it falls when returns are negatively skewed or have fat tails — which is exactly the profile of a strategy that wins small and often and then loses big. Two bots can show the same Sharpe ratio and have very different PSR values if one of them earned it over 1,000 trades and the other over 60, or if one of them has a hidden tail.

A common working threshold is PSR(0) ≥ 0.95: at least a 95% probability the true Sharpe is positive.

DSR: the same question, after counting your attempts

DSR takes PSR and deflates the benchmark it is measured against. Instead of asking whether the Sharpe beats zero, it asks whether the Sharpe beats the Sharpe you would expect the best of N trials to produce by pure chance.

That expected maximum grows with the number of trials, and it grows faster than people's intuition. The intuition to carry around is the one from order statistics: the expected largest value from N independent standard-normal draws rises roughly with the square root of twice the natural log of N. Sixty trials put the bar near 2.9 standard deviations; a thousand trials push it past 3.7. The number of attempts is not a footnote to the result. It is part of the result.

Two things follow:

DSR is always lower than PSR, and the gap widens with every variant you try.
A strategy can have PSR(0) = 0.99 and DSR = 0.30 at the same time. That combination is the signature of overfitting, and it is the single most informative pair of numbers you can compute about a backtest.

What counts as a trial

This is where almost everyone under-counts. A trial is any variant you looked at and could have chosen, including:

every parameter combination in a grid search, not just the ones you wrote down;
every entry filter you added and then removed because it did not help;
every coin universe you tried;
every version of the strategy from earlier sessions, if the current version descends from it;
every timeframe you tested before settling on the one you kept.

If you ran a 10 by 10 grid, that is 100 trials, not 1. If you then added a filter and ran it again, that is 200. The honest number is almost always larger than the number you remember, and the correct response to uncertainty is to use the larger estimate. Deflating by too many trials makes you slow. Deflating by too few makes you confident about noise.

A worked comparison

Two of our own demonstration strategies, both audited the same way:

| | Honest trend-follower | Curve-fitted variant | |---|---|---| | What it is | EMA crossover with a band, past bars only | the best of 50 variants: a crossover that only trades on two weekdays, at one hour, held for exactly 36 bars | | Variants tried | 3 | 50 | | PSR(0) | 0.95 | 0.98 | | DSR | 0.74 | 0.39 | | Verdict | not yet enough evidence | not ready, and the gap says why |

The curve-fitted one has the better ordinary statistics. It is the one you would pick if you only looked at PSR, or at the equity curve, or at the win rate. Its DSR is what gives it away: once the 50 attempts behind it are priced in, most of its apparent edge is explained by having tried 50 times.

Note also that the honest strategy does not pass either. That is normal and it is the point. A real edge usually needs more trades before it can be told apart from luck, and "not yet" is a legitimate verdict.

How to use these numbers in practice

Write down your trial count before you compute anything. Deciding it afterwards, when you can see which number you need, is how the count ends up at 1.
Report PSR and DSR together. PSR alone flatters. DSR alone hides whether the problem is sample size or selection.
Set the bar in advance. Ours: PSR(0) ≥ 0.95 and DSR ≥ 0.5, alongside a minimum trade count, a recency check, concentration limits and a drawdown limit. Six fixed gates, decided before the result is known.
Expect to fail. Across the hundred strategy books we track, mostly paper, none currently passes all six gates at our 300-trade floor. That is the honest base rate, and a tool that frequently tells you yes is not measuring anything.

What these statistics cannot do

DSR works on your list of trades. It cannot see your code. So it will not catch:

look-ahead bias, where the backtest used information that did not exist yet — that needs a separate audit that recomputes the signal on truncated history and checks it does not change;
wrong cost assumptions — if your R values were computed with optimistic fees, every statistic downstream inherits the optimism;
a backtest that never could have been filled at the prices it assumed.

Get the trades right first. Then deflate.

Compute it on your own trades

The implementation we use — PSR, DSR, minimum track record length, the six gates, the cost model and the look-ahead audit — is MIT-licensed Python at github.com/botlabtrade/botlab-oss. There is also a checker that runs in your browser: paste a CSV of closed trades, enter how many variants you tried, and read the six gates with your own numbers. Nothing is uploaded.

Paper trading results are simulated. Past results don't predict future results. Crypto trading can lose money, including all capital. Not financial advice.

Originally published on BOTLAB. Score your own bot's trade log free: Check your bot.

Do Funding Rate and Open Interest Filters Improve a Crypto Bot?

Bot LAB — Mon, 12 Oct 2026 16:06:35 +0000

Exchange data is sold as the missing ingredient for crypto bots. Funding rate, open interest, the long/short account ratio, the perp premium: add one as an entry filter, the pitch goes, and your signal starts skipping the bad trades.

We tested that claim properly and the answer was mostly no. Out of 60 filter-and-dataset combinations, one cleared the bar that separates a real effect from the best of sixty coin flips. This guide shows the test, the numbers, and the reason the result is so one-sided.

What we tested

Five trade lists, all on 15 USDT perpetual contracts on Bybit:

| Dataset | What it is | Trades | Baseline mean R per trade | |---|---|---|---| | 15m | a 20-indicator voting scalp signal | 113,013 | -0.188R | | 30m | the same signal on 30-minute bars | 62,585 | -0.151R | | 4h (all) | a 4-hour mean-reversion bot's full ledger | 6,159 | -0.111R | | 4h (subset) | the part that bot actually trades | 424 | +0.101R | | 1D | a Donchian-20 daily breakout, 2 ATR stop | 886 | +0.205R |

R is the result of a trade measured in units of its own risk: risk 1% of the account, make 2%, that is +2R. Every R above is after costs — taker fee 0.055% per side, slippage measured by walking a real order book for the trade size, and average funding multiplied by holding time.

Twelve filters, each with fixed rules taken from tools that already existed, nothing tuned for this test:

Price and trend: BTC daily regime against its 200-day EMA, multi-timeframe EMA agreement on 4h, the Ichimoku cloud, Supertrend, ADX between 20 and 35, position in the 20-day Donchian range, Choppiness below 55.
Exchange data: funding not crowded on our side of the trade, open interest rising with price over 24 hours, no open-interest spike above 8% in 4 hours, the long/short account ratio contrarian against its 30-day median, the perp premium contrarian against its 7-day mean.

How we avoided fooling ourselves

Three decisions did most of the work.

Every data point carries the moment it became knowable. Funding is stamped at its settlement time, open interest and the long/short ratio at the end of their hour, candle-derived values at candle close. A trade may only read rows that were available strictly before its entry. We re-checked a random sample of 48 trades against the live exchange API and they matched. Without this rule, a filter that quietly reads the current hour's open interest will look brilliant and be worthless.

The split is by time, not at random. The oldest 60% of entries are the training period, the newest 40% the test period. A filter counts as helping only if the trades it agrees with beat the baseline in both periods, with at least 30 agreeing trades in each. Random splits leak: a trade from June sitting in training while its neighbour from the same June afternoon sits in test is not an out-of-sample test of anything.

We counted our own attempts. Twelve filters across five datasets is 60 trials. If all 60 were pure noise, the largest t-statistic you would expect to see by chance is roughly the square root of twice the natural log of 60, about 2.86. That is the bar. A filter scoring t = 2.0 in a 60-trial study has not found anything; it is simply the luckiest of sixty.

Because trades opened on the same day share the same market, we also computed a daily-clustered statistic: the average of each day's agree-minus-disagree difference, rather than treating every trade as independent. This matters more than most people expect. A filter that tags 400 trades across 30 days has roughly 30 independent observations, not 400.

What came back

One combination cleared 2.86. Multi-timeframe EMA agreement on the 4-hour bot: both the fast pair and the slow pair pointing the same way as the trade. It helped in training, it helped in test, and it cleared the noise bar. Everything else either failed one period, or beat the baseline by less than the study's own luck threshold.

The funding and open-interest guardrails almost never disagree. Across all ten cells where they applied, they agreed with 84% to 100% of trades, and in six of those ten they were effectively constant — they say "fine, go ahead" nearly every time. That is a useful property for a risk check, which should only ever stop you in a genuinely extreme state. It is a useless property for a signal, because a filter that never says no cannot improve anything.

The long/short ratio changed its mind by dataset. It helped on 15m, on 30m and on the 4-hour ledger, overfit on the 4-hour subset, and did nothing on the daily. A filter whose direction depends on which dataset you show it is describing the dataset, not the market.

The reason, which is simpler than the statistics

Look at the fee arithmetic on the 15-minute signal. Fees alone are 0.055% twice, so 0.11% per round trip, before slippage and before funding. The minimum stop is 0.6%. So 0.11 divided by 0.6 is about 0.18R of pure cost per trade — almost exactly the size of that signal's loss per trade of -0.188R.

The 15-minute signal does not have a filtering problem. It has a cost problem. No entry filter can rescue an edge smaller than the cost of taking it; the best a filter can do is make you lose more slowly by trading less.

On the daily breakout, with a stop measured in multiples of ATR, the same 0.11% is a rounding error. This is why the same filter can look useless on one timeframe and plausible on another: the thing that changed was the denominator.

What this means for your own bot

Check costs before you check filters. Divide your round-trip cost by your typical stop distance. If the answer is a meaningful fraction of your expected edge per trade, no filter will save the strategy.
Count every variant you tried, including the ones you abandoned, and raise your evidence bar accordingly. The honest number is almost always larger than the one you remember.
Treat exchange data as a guardrail, not as alpha. Funding and open-interest extremes are good reasons to stand aside. They are poor reasons to enter.
Check how often your filter disagrees. If it agrees with 95% of your trades, it cannot be doing much, however good its numbers look.

What we are not claiming

The 60-trial count covers this study alone. These five datasets came from strategies that were themselves selected after earlier experiments, so the true number of attempts behind them is larger and every figure here is, if anything, flattering. The 15m and 30m lists are backtests of our own scalp signal rather than live bots. The order-book slippage uses a recent snapshot, which understates the cost of trading in thinner past markets. And the split is a single cut in time, not a walk-forward.

The one surviving result is a reason to keep testing multi-timeframe trend agreement on new trades. It is not a reason to trade it.

Run the numbers yourself

The statistics behind all of this — the probabilistic and deflated Sharpe ratios, the minimum track record length, the six readiness gates, the cost model and the look-ahead audit — are MIT-licensed Python at github.com/botlabtrade/botlab-oss. Point it at your own trade list and you can reproduce the method on your own data. There is also a browser version that needs no install.

Paper trading results are simulated. Past results don't predict future results. Crypto trading can lose money, including all capital. Not financial advice.

Originally published on BOTLAB. Score your own bot's trade log free: Check your bot.

Is a Trading Bot Profitable? How to Test One Honestly

Bot LAB — Mon, 12 Oct 2026 04:06:30 +0000

Is a trading bot profitable? Some can be for a while, but most crypto trading bots are not profitable once fees, slippage and changing market conditions are counted, and no bot can promise future profit. The only honest way to find out is to test a specific bot with enough trades, real costs and a forward test on prices it has never seen.

This guide shows you how to run that test yourself, step by step, without relying on screenshots or marketing claims.

Is a trading bot profitable? The short, honest answer

A trading bot is a set of rules that buys and sells automatically. It is not a source of profit by itself. A bot only makes money if its rules have a real edge, meaning a repeatable advantage that survives:

Trading costs: exchange fees, funding rates on perpetual futures, and slippage.
Different market regimes: trending, sideways and high-volatility periods.
Time: results that keep holding on new data, not just on the past.

Most bots fail at least one of these. Sometimes a bot looks profitable because the market went up and it was simply holding long positions. That is market exposure (often called "beta"), not skill. When the market turns, the "profit" turns too.

So a better question than "is a trading bot profitable?" is: "Does this bot have evidence of an edge, and how strong is that evidence?"

Why most bots that look profitable are not

1. The sample is too small

Twenty winning trades in a row feel convincing, but they can easily be luck. You usually need many trades across different market conditions before a result means much. We cover this in detail in How Many Trades Does a Backtest Need?.

2. The backtest is overfit

Overfitting happens when a strategy is tuned so tightly to past data that it "memorises" history. Warning signs:

Many parameters (for example, five indicators, each with custom settings).
Very smooth backtest equity curves.
Results that collapse when you change a setting slightly.
Great results on one coin or one time window only.

3. Costs are ignored or underestimated

A strategy that makes many small trades can look excellent before costs and lose money after them. Fees and slippage hit high-frequency and scalping bots hardest.

4. Win rate is mistaken for profitability

A high win rate says nothing on its own. A bot can win 75% of trades and still lose money if its losses are much bigger than its wins. See Profit Factor vs Win Rate for Trading Bots.

How to test whether a trading bot is profitable

Use this checklist on any bot: one you built, one you found online, or one someone is trying to sell you.

Step 1: Get the full trade list, not a summary

Ask for (or export) every trade: entry time, exit time, side, size, price and fees. A summary like "+40% last month" is not evidence. If a seller cannot or will not share a trade list, treat that as a red flag.

Step 2: Add realistic costs

Re-calculate every trade with:

Your exchange's actual taker/maker fees.
A slippage estimate (more for small-cap coins and fast markets).
Funding payments if the bot holds perpetual futures positions.

Step 3: Check the sample size and period

How many trades are there in total?
Over how many days or months?
Does the period include a downtrend, an uptrend and a sideways stretch?

A few dozen trades over a few weeks is an early signal at best.

Step 4: Look at risk, not just return

Max drawdown: the largest peak-to-trough fall. Could you sit through it without switching the bot off?
Average loss vs average win.
Longest losing streak.
Leverage used: high leverage turns ordinary drawdowns into liquidations.

Step 5: Compare against simply holding

If the bot only makes money when the market rises, compare it with just holding the same coin. If buying and holding did about as well, the bot may not be adding anything.

Step 6: Forward test with paper trading

Run the bot on live prices with simulated money (paper trading) for a period it has never seen. This is the closest you can get to real conditions without risking capital. Backtests describe the past; forward tests show whether the rules still behave the same way on new data.

Step 7: Set pass/fail rules before you look

Decide in advance what counts as "good enough": a minimum number of trades, a maximum drawdown, a minimum result after costs. Writing the rules first stops you from moving the goalposts when a result looks exciting.

At BOTLAB we apply this idea through 6 statistical gates (G1–G6), defined on the site. A bot is only labelled "ready for micro live" after passing all six.

Worked example: a bot that wins most trades but still loses

Illustration only. These are made-up numbers to show the maths. They are not BOTLAB results and do not describe any real bot.

Imagine a bot that made 100 trades in a backtest:

Wins: 60 trades, averaging +1.0% each
Losses: 40 trades, averaging −1.4% each
Round-trip fees and slippage: 0.12% per trade

Before costs:

Wins: 60 × 1.0% = +60%
Losses: 40 × 1.4% = −56%
Gross result: +4% (simple sum, ignoring compounding)

After costs:

Costs: 100 × 0.12% = −12%
Net result: 4% − 12% = −8%

A 60% win rate and a positive gross backtest, yet the bot loses money once costs are included. Now imagine the same bot run with x20 leverage: every number above is multiplied, and a bad streak could wipe out the account. This is why testing must always include costs and leverage.

What real paper-trading data looks like

Honest testing usually produces more "no" answers than "yes" answers. At BOTLAB, as of 12 October 2026, we track 142 bots in paper trading (simulated money, real market prices): 0 are rated ready for micro live, 2 are on watch and 140 are not ready.

Two examples from that paper-trading data:

mr4h/long: 77 paper trades over 57 days (16 Aug – 12 Oct 2026), win rate 72.7%, passes 5 of 6 gates. Status: WATCH. Verdict: sample too small to conclude.
Atlas/live: 392 paper trades over 65 days (8 Aug – 12 Oct 2026), win rate 40.6%, passes 1 of 6 gates. Status: NOT_READY.

Even the best-looking bot here is not considered proven, because 77 trades is not enough to rule out luck. Publishing failures is part of the method; see Failure is the method and the mr4h/long paper update.

If you want a structured second opinion on your own bot's numbers, you can run them through our free tool: Check your bot.

Red flags when someone claims their bot is profitable

Fixed monthly return promises (for example "10% every month").
Only screenshots, no exportable trade history.
No mention of drawdown, fees or number of trades.
Countdown timers or "limited spots" pressure.
Requests for your exchange password, 2FA codes, seed phrase, or an API key with withdrawal permission.

If you ever connect a bot to an exchange, the API key should be trade-only, withdrawals off, and IP-restricted.

FAQ

Are trading bots legit?

The software itself is a legitimate tool; many traders automate rules they already use. What is often not legitimate is the marketing around bots, especially promises of fixed returns. Judge a bot by its full trade history, costs and drawdown, not by claims.

What is a realistic trading bot success rate?

"Success rate" usually means win rate, and it can be misleading. A bot with a 40% win rate can do well if its wins are much larger than its losses, and a bot with a 75% win rate can lose money. Look at win rate together with average win, average loss, costs and number of trades.

Is a trading bot worth it for beginners?

A bot does not remove the need to understand risk. For beginners, the most useful first step is paper trading: run a bot with simulated money on live prices, learn how drawdowns feel, and only then decide whether automation fits you. Never use money you cannot afford to lose.

How long should I paper trade a bot before trusting it?

There is no magic number, but aim for enough trades to cover different market conditions, usually more than a few weeks of data. Decide your pass/fail rules in advance and stick to them.

Key takeaways

A trading bot is only as good as its edge after fees, slippage and funding.
High win rates and short winning streaks are not proof of profitability.
Test with full trade lists, realistic costs, enough trades and a forward paper test.
Watch drawdown and leverage as closely as returns.
Most bots fail honest testing, and that is useful information, not a setback.

Not available in restricted jurisdictions (see the Service Agreement). You trade with your own exchange account; BOTLAB never holds funds.

Paper trading results are simulated. Past results don't predict future results. Crypto trading can lose money, including all capital. Not financial advice.

Originally published on BOTLAB. Score your own bot's trade log free: Check your bot.

Profit Factor vs Win Rate for Trading Bots

Bot LAB — Sun, 11 Oct 2026 12:07:12 +0000

Profit factor is usually the more useful of the two, because it measures whether a bot's wins are bigger than its losses. Win rate only tells you how often it wins. Neither number is enough alone, though. A bot can win 80% of its trades and still lose money, and a bot that wins 35% of the time can come out ahead. This guide explains both metrics, how they relate, and how to use them together without fooling yourself.

What win rate actually measures

Win rate is the share of closed trades that ended in profit:

Win rate = winning trades ÷ total trades

It is easy to understand and easy to market, which is why bot sellers love to quote it. But it ignores two things that decide whether a strategy makes or loses money:

How big the wins are compared with the losses
How much each trade costs in fees, funding and slippage

A high win rate often comes from taking profit quickly and letting losers run. That produces lots of small wins and a few large losses. The equity curve looks smooth for weeks, then one bad trade wipes out a month of gains.

What profit factor actually measures

Profit factor compares the money made on winning trades with the money lost on losing trades:

Profit factor = gross profit ÷ gross loss

How to read it:

Below 1.0: the bot lost money over the sample
Exactly 1.0: break-even before anything else goes wrong
Above 1.0: winners outweighed losers over the sample

Profit factor captures both how often a bot wins and by how much. That makes it a better first summary than win rate. It still has weaknesses, which we cover below.

The link between them: expectancy

The cleanest way to connect the two metrics is expectancy, the average result per trade:

Expectancy = (win rate × average win) − (loss rate × average loss)

From it you can work out the break-even win rate for any payoff profile:

Break-even win rate = average loss ÷ (average win + average loss)

This one formula answers most "is this win rate good?" questions. A 70% win rate is excellent if the average loss equals the average win. It is a losing system if the average loss is five times the average win.

Worked example: two bots, opposite profiles

Illustration only. The numbers below are made up to show the maths. They are not BOTLAB results and do not describe any real bot.

Imagine two bots, each tested on 100 trades.

Bot A: high win rate, small wins, big losses

Win rate: 80% (80 wins, 20 losses)
Average win: $10
Average loss: $50

Calculations:

Gross profit = 80 × $10 = $800
Gross loss = 20 × $50 = $1,000
Profit factor = 800 ÷ 1,000 = 0.80
Expectancy = (0.80 × 10) − (0.20 × 50) = 8 − 10 = −$2 per trade
Break-even win rate = 50 ÷ (10 + 50) = 83.3%

Bot A wins four trades out of five and still loses money. It needs to win more than 83% of the time just to break even.

Bot B: low win rate, big wins, small losses

Win rate: 35% (35 wins, 65 losses)
Average win: $30
Average loss: $10

Calculations:

Gross profit = 35 × $30 = $1,050
Gross loss = 65 × $10 = $650
Profit factor = 1,050 ÷ 650 ≈ 1.62
Expectancy = (0.35 × 30) − (0.65 × 10) = 10.5 − 6.5 = +$4 per trade
Break-even win rate = 10 ÷ (30 + 10) = 25%

Bot B loses almost two trades out of three. Over this sample its winners more than covered the losers.

Now add costs

Suppose fees and slippage cost $1 per round trip (again, an illustration):

Bot A expectancy falls from −$2 to −$3 per trade
Bot B expectancy falls from +$4 to +$3 per trade

Costs hurt every strategy, but they hurt high-frequency, small-target strategies the most. If a bot's average win is only a little larger than its round-trip cost, a modest rise in fees or slippage can flip it negative. Always check whether a reported profit factor is net of fees.

Why each metric can mislead you

Win rate traps

Hidden tail risk. Strategies with no stop-loss, or with averaging-down, can show very high win rates until one large loss arrives.
Survivorship in short samples. A few weeks without a big adverse move can make a fragile bot look brilliant.
Psychological comfort. Frequent small wins feel good, which makes people overlook the size of the losses.

Profit factor traps

Small samples. With 20 trades, one lucky outlier can push profit factor from 1.1 to 2.5. The number is only as reliable as the number of trades behind it. See How Many Trades Does a Backtest Need?.
Outlier dependence. If removing the single best trade drops profit factor below 1.0, the "edge" may be one lucky event.
Gross vs net. A profit factor calculated before fees can be very different from what you'd actually see.
Overfitting. Tune enough parameters on one dataset and you can make almost any profit factor look good. It usually falls apart on new data.
Market beta. A long-only bot in a rising market can show a strong profit factor simply because prices went up. That is exposure to the market, not a proven edge.

How to judge a bot using both numbers

Use this checklist when you look at any bot's results, including your own:

Get the trade count and the period first. No trade count, no conclusion.
Read win rate together with average win and average loss. Calculate the break-even win rate and compare.
Check profit factor is net of costs. Ask what fee level and slippage were assumed.
Remove the best one or two trades. Recalculate profit factor. If it collapses, be cautious.
Look at the worst losing streak and max drawdown. A good profit factor with a drawdown you couldn't sit through is not usable.
Compare to simply holding the asset. If the bot did roughly what the market did, it may have no edge of its own.
Prefer out-of-sample or forward results. A backtest is a hypothesis. Paper trading on live prices is a better test, and it is still not proof. Our Paper trading, explained post covers the difference.

How we apply this at BOTLAB

We don't treat a high win rate as evidence by itself. Every bot we track has to pass 6 statistical gates (G1–G6), defined on the site, before it is labelled "ready for micro live". As of our latest build, 101 of the 102 bots we track are marked not ready.

A real example of why win rate alone isn't enough: in paper trading (simulated money, real market prices) from 2026-08-16 to 2026-10-03, our mr4h/long bot recorded 53 trades with an 81.1% win rate and passed 6 of 6 gates. Our own verdict is still "sample too small to conclude". Another bot, botGPT/dir2s, had a 39.0% win rate over 41 paper trades (2026-09-17 to 2026-10-06), and our verdict there is "market beta, no proven edge". A high or low win rate doesn't settle the question either way. You can follow the mr4h/long paper update.

If you want to run your own bot's trade list through these kinds of checks, try the free tool: Check your bot.

FAQ

What is a good profit factor for a crypto trading bot?

There is no universal "good" number. A net profit factor above 1.0 means winners outweighed losers in that sample. How much that matters depends on the trade count, whether fees are included, how much the result depends on a few outliers, and whether it holds up on data the bot was not tuned on. A high profit factor on a small sample deserves extra suspicion.

Can a bot with a 90% win rate lose money?

Yes. If the average loss is more than nine times the average win, a 90% win rate loses money. Use the break-even formula, average loss ÷ (average win + average loss), to check.

Is a low win rate a red flag?

Not by itself. Trend-following strategies often win less than half their trades and rely on occasional large winners. The real questions are whether the average win is large enough to cover the losses and costs, and whether you could tolerate the losing streaks that come with a low win rate.

Should I use profit factor or expectancy?

Use both. Profit factor is a ratio and is easy to compare across bots. Expectancy tells you the average result per trade in money or R terms, which makes the impact of fees easier to see. Both need a sufficient trade count to mean anything.

Key takeaways

Win rate tells you how often a bot wins. Profit factor tells you whether the wins outweigh the losses.
Always calculate the break-even win rate from average win and average loss.
Use profit factor net of fees and slippage, and check that it survives removing the best trades.
Small samples make both metrics unreliable. Look at trade count and period first.
A strong-looking metric can just be market beta or overfitting. Forward and paper results are better evidence, but they are still not proof.

Not available in restricted jurisdictions (see the Service Agreement). You trade with your own exchange account; BOTLAB never holds funds.

Paper trading results are simulated. Past results don't predict future results. Crypto trading can lose money, including all capital. Not financial advice.

Originally published on BOTLAB. Score your own bot's trade log free: Check your bot.

How Many Trades Does a Backtest Need? A Practical Guide

Bot LAB — Sun, 11 Oct 2026 12:07:07 +0000

There is no single magic number. As a rough guide, under 30 trades tells you almost nothing, around 100 starts to be informative, and several hundred trades across different market conditions is where a backtest becomes hard to dismiss. The number you actually need depends on how big the bot's average edge is compared with how noisy its individual trades are. This guide shows you how to work that out.

Why trade count matters so much

Every backtest is a small sample of a strategy's possible outcomes. If you flip a fair coin 10 times, getting 7 heads is not unusual. Trading results work the same way. A bot with zero real edge can produce a beautiful equity curve over 20 or 40 trades purely by luck.

The question a backtest should answer is not "did it make money?" but "how likely is it that this result came from luck?" Trade count is the main thing that shrinks luck's influence.

The core idea: noise shrinks slowly

The uncertainty around a bot's average result per trade shrinks with the square root of the number of trades. That has an uncomfortable consequence:

To halve your uncertainty, you need 4x the trades.
To cut it to a third, you need 9x the trades.

This is why 50 trades feels like a lot but usually isn't.

Rough tiers for trade count

These are rules of thumb, not laws. Treat them as a starting point:

Under 30 trades: Anecdote. Useful for spotting obvious bugs, not for judging an edge.
30–100 trades: Early signal. You can rule out strategies that are clearly broken, but a profitable-looking result can still easily be noise.
100–300 trades: Informative, if trades are reasonably independent and the edge per trade is not tiny.
300+ trades across several market regimes: Strong enough to take seriously, provided you didn't test hundreds of variants to find it.

What changes the number you need

1. Size of the edge versus the noise

Measure each trade in R (profit or loss divided by the amount risked). A bot averaging +0.5R per trade with consistent outcomes needs far fewer trades to prove itself than one averaging +0.05R with wild swings. Small edges need big samples.

2. Skewed payoffs

Strategies with a high win rate and occasional large losses are the hardest to judge. A short sample may simply not include the rare big loss yet. The reverse is also true: trend-following bots with many small losses and rare big wins can look terrible over a short window.

3. Market regimes

200 trades all taken during one strong uptrend is really one experiment repeated. You want trades spread across trending, sideways and volatile periods. Count regimes, not just trades.

4. How many variants you tested

If you tried 100 parameter combinations and kept the best, that winner's backtest is inflated. The more variants you test, the higher the bar should be. This is the heart of overfitting.

5. Trade independence

Ten positions opened on the same day on correlated coins behave more like one or two trades than ten. Clustered trades reduce your effective sample size.

Worked example: how many trades would this bot need?

Illustration only. These are made-up numbers to show the method, not results from any real bot.

Imagine a backtest with:

Trades: 40
Average result per trade: +0.15R
Standard deviation of trade results: 1.2R

Step 1: Standard error of the average. 1.2 ÷ √40 ≈ 1.2 ÷ 6.32 ≈ 0.19R

Step 2: Compare the edge to the noise. 0.15 ÷ 0.19 ≈ 0.79. A common rough bar is about 2. At 0.79, the result is well within what luck alone could produce.

Step 3: A rough plausible range. Average ± 2 standard errors: 0.15 ± 0.38, so roughly −0.23R to +0.53R. The true edge could easily be negative.

Step 4: How many trades to reach the bar of 2? n ≈ (2 × 1.2 ÷ 0.15)² = 16² = 256 trades

Step 5: Now add realistic costs. Suppose fees and slippage cut the average to +0.10R. Then: n ≈ (2 × 1.2 ÷ 0.10)² = 24² = 576 trades

A third less edge more than doubled the trades needed. This is why fee assumptions matter so much, especially for high-frequency bots.

The high win rate trap

A high win rate feels reassuring, but it says little on its own.

Illustration only: a bot wins 9 out of 10 trades at +0.3R each, then loses 1 trade at −3R. Win rate: 90%. Net result over those 10 trades: 9 × 0.3 − 3 = −0.3R. Over a short sample that happened to avoid the loss, it would have looked flawless.

Always look at the average win, average loss, and the worst loss, not just the win rate.

A real example of "good numbers, not enough trades"

Here is one from our own lab. At BOTLAB, the bot mr4h/long recorded, in paper trading (simulated money, real market prices), 53 trades over 48 days (2026-08-16 to 2026-10-03) with an 81.1% win rate. It passed all 6 statistical gates (G1–G6), defined on the site, and is marked ready for a micro live test. Even so, its verdict is still "sample too small to conclude". You can see the paper update for mr4h/long.

That combination is the point of this article: passing filters and having a high win rate does not replace a larger sample. It only means the bot has earned the right to keep being tested.

A checklist before trusting any backtest

[ ] Trade count: At least 100, ideally several hundred.
[ ] Regimes: Trades spread across trending, sideways and volatile periods.
[ ] Costs: Realistic fees, funding and slippage included.
[ ] Edge vs noise: Average R divided by its standard error is comfortably above about 2.
[ ] Payoff shape: Average win, average loss and worst loss reviewed, not only win rate.
[ ] Variants tested: You know how many combinations were tried before this one.
[ ] Out-of-sample: Some data was held back and never used for tuning.
[ ] Independence: Clustered or correlated trades are not counted as fully separate.

From backtest to forward testing

Even a large, clean backtest is still a test on data you've already seen. The next step is forward testing on live prices without real money, usually called paper trading. It is the honest check against overfitting because the bot meets data that didn't exist when it was built. If you're new to the idea, see Paper trading, explained.

The same sample-size rules apply to forward tests. A few weeks of good paper results is still a small sample; let the trade count build up before drawing conclusions.

If you want a quick sanity check of your own bot's numbers, the free BOTLAB tool can help: Check your bot.

FAQ

Is 30 trades enough for a backtest?

Usually not. 30 trades can reveal bugs or a clearly broken strategy, but the uncertainty around the average result is too wide to confirm an edge unless that edge is very large and consistent.

Does more historical data always help?

More trades help, but only if the data is relevant. Very old data from a different market structure can mislead. Prefer more trades across varied, reasonably recent regimes, and keep some data aside for out-of-sample testing.

Does a higher win rate mean I need fewer trades?

Not necessarily. High win rate strategies often hide rare large losses, which a short sample may miss entirely. What matters is the average result per trade relative to its variability, plus the size of the worst loss.

How long should I paper trade a bot after backtesting?

Think in trades, not days. Run it until it has built up a meaningful number of trades across different market conditions, and compare those forward results with the backtest. Large gaps between the two are a warning sign.

Key takeaways

There is no magic number, but under 30 trades is anecdote and 100+ is a sensible minimum.
Uncertainty shrinks with the square root of trades: halving it needs 4x the data.
Smaller edges, higher costs and skewed payoffs all raise the trades you need.
Count regimes and tested variants, not just trades.
A high win rate or passing filters does not replace a larger sample.
Confirm backtests with forward paper trading before drawing conclusions.

Paper trading results are simulated. Past results don't predict future results. Crypto trading can lose money, including all capital. Not financial advice. Not available in restricted jurisdictions (see the Service Agreement). You trade with your own exchange account; BOTLAB never holds funds.

Originally published on BOTLAB. Score your own bot's trade log free: Check your bot.