DEV Community

AgentChip
AgentChip

Posted on

Your Backtest Is Lying to You. Here's How to Prove It.

#trading #python #datascience #productivity

If you've spent any time building trading strategies, you've seen the pattern: a backtest that looks amazing, a forward test that looks okay, and a live account that quietly bleeds out. The strategy didn't fail — the backtest was lying, and nobody built in a way to catch it.

The problem isn't bad data or bad code. It's that most backtesting workflows are built to confirm a strategy, not to disprove it. Curve-fit parameters, no benchmark, no out-of-sample check, no significance test — you get a pretty equity curve and a false sense of edge.

What I built: a backtester that refuses to flatter you

A single-file Python backtest engine (pure standard library, zero dependencies) with the honesty checks built into the core, not bolted on:

  • 70/30 in-sample / out-of-sample split — the strategy has to survive data it never saw
  • Parameter heat-map overfitting scan — if the profitable parameter region is narrow, it's flagged FRAGILE, not marketed as robust
  • Rolling-window stability — edge has to persist across time slices, not just one lucky stretch
  • Shuffle random control — does the strategy actually beat a randomly shuffled version of the same data?
  • t-test significance — is the edge statistically distinguishable from noise?
  • Buy-and-hold benchmark — the humblest baseline; if you can't beat it, it says so

Instead of a vanity report, you get a verdict in three honest tiers:

CONFIRMED  — edge survives out-of-sample, parameter scan, shuffle, and benchmark
MIXED      — real signal, but fragile in places; know exactly where
REJECTED   — what you actually need to hear before funding it with real money
Enter fullscreen mode Exit fullscreen mode

Four strategies ship built-in (SMA cross, Donchian, RSI, momentum) and you can drop in your own as a Python function.

# Run the built-in demo on synthetic data with an edge
python honest_backtester.py --demo

# Test your own strategy on a CSV of OHLC data
python honest_backtester.py --data my_data.csv --strategy my_strategy.py --report md
Enter fullscreen mode Exit fullscreen mode

Why honesty is the feature

The trading tools market is full of products that sell you confirmation. The ones that tell you "no edge here, don't trade this" are rare — because that's the answer most people don't want to pay for. But it's the answer that saves you from the real cost: funding a strategy that only worked in the past.

If you've ever wondered whether your edge is real or just a well-fit curve, the full kit (engine + overfitting checks + benchmark + docs) is at AgentChip.

The market doesn't owe you a strategy. A good backtester at least tells you the truth about the one you have.

Originally published on the AgentChip blog.

Top comments (0)