Your Trading Bot's Backtest Is Lying to You - Here Are the 4 Ways

Aurora Systems — Sat, 11 Jul 2026 09:40:11 +0000

A strategy I once tested quoted +740% in the backtest. When it actually traded, it made about $2.81.

That gap - between what a backtest promises and what a live market pays - is where most trading bots quietly die. I've spent three years building bots in Python, and the most valuable thing I've learned isn't a strategy. It's that a backtest is the easiest thing in the world to rig in your own favor, usually by accident.

Here are the four ways it lies, each with a real number from my own testing, and how to catch each one before it costs you money.

The baseline: an honest test of the "hello world" strategy

First, a calibration. The classic moving-average crossover - SMA(9/21), the strategy every tutorial starts with - backtested honestly on 1,000 hours of real BTC/USDT, with decisions using only past, closed bars:

No fees: -6.3%
With a realistic 0.1% fee per trade: -11.3%
Buy & hold over the same window: -21.4%

Notice two things. It loses before a single fee. And it "beat buy & hold" while still losing money - those are not the same sentence, and any backtest report that only shows relative performance is already halfway to lying.

Lie #1: Lookahead - your strategy can see the future

The most common bug, and it's usually one careless line:

signal = strategy(prices)         # WRONG - sees the whole series, including the future
signal = strategy(prices[:i+1])   # RIGHT - only the past, one bar at a time

Hand the strategy the entire price series instead of only the data up to the current bar, and every historical "decision" secretly knows how the story ends. The equity curve goes vertical. The returns are fiction. Subtler versions hide in feature scaling (normalizing by a series max that includes future data) and in fill assumptions (deciding at the close of a bar and getting filled at that same close).

How to catch it: run the backtest walk-forward - feed the strategy a growing window, never the whole series. And treat any curve that looks too smooth as a bug until proven otherwise. Real edges are messy.

Lie #2: Overfitting - you tuned until it looked perfect

A real experiment from my logs. I grid-searched MA parameters on the first half of my data. The best combination returned +2.4%. Then I ran those exact same parameters on the second half - data the search had never seen: -1.5%.

The profit was a property of the search, not the market. Run enough parameter combinations against one slice of history and something will always look brilliant - you're not discovering an edge, you're memorizing noise. That green-in-sample, red-out-of-sample flip is the fingerprint of overfitting.

I later re-ran this lesson at a bigger scale: parameters tuned on one half-year, validated on the next, then scored on a final unseen period across five coins. One of five survived. The other four evaporated exactly the way the +2.4% did.

How to catch it: judge a strategy only on data it was never tuned on. Optimize on one slice, score on another. No exceptions, no peeking, no "just one more tweak" after seeing the holdout.

Lie #3: Ignored costs - you assume you trade for free at the price you see

Fees alone nearly doubled the baseline loss above (-6.3% -> -11.3%). And fees are the polite cost. Slippage is the rude one. Walking a market order through the real BTC order book:

$1,000: ~0.000% price impact
$1,000,000: ~0.035%
$10,000,000: ~0.132% - and it exhausts the entire visible book

That's on the deepest market in crypto. On a thin pair, the visible book collapses after the first few coins - which is exactly how a backtest can quote +740% on a freshly launched pool while the tradeable reality is about $2.81. The price you see is the price for the first few coins. Everything after that is slippage.

How to catch it: model fees and a slippage assumption from the very first backtest. The thinner the market and the bigger your size, the more the quote lies about the fill.

Lie #4: Survivorship - you only tested the winners

Backtest "all coins" using today's coin list and you've silently deleted every project that went to zero and got delisted. Your universe is pre-filtered to survivors, so of course the results look great. The same trap hits stock strategies tested on the current index members - the companies that failed out of the index are exactly the ones your strategy would have bought on the way down.

How to catch it: use data that includes the dead - delisted assets, failed projects, all of it. And be suspicious of any result that claims "tested on everything."

The honest checklist

Run this on every idea:

Express it as "prices in -> signal out." If you can't, it isn't testable yet.
Backtest with realistic fees and slippage from day one.
Score it only on data it was never tuned on.
Read absolute return and max drawdown - "beat the market" while losing money is not a win.
Distrust anything that looks great; hunt for the bug first.
If it survives all that, paper-trade it on a testnet. For a long time.
Size positions with a real risk layer and keep a kill-switch armed.
Never run money you can't afford to lose to zero.

The goal was never to find a magic strategy. It's to build a backtester honest enough to kill your bad ideas cheaply - so the rare good one is the only thing left standing.

I write about the honest engineering behind trading bots - including the losing numbers. If this was useful:

The full version is a book: Backtests Lie - why most bots lose, and what building one really teaches you.
Free companion guide (PDF): The 4 Ways Your Backtest Is Lying To You
Day-to-day notes on X: @AuroraSystemsX

Educational only. Not financial advice. Trading carries real risk, including total loss.