DEV Community: joeschatzman

Why Your Backtest Is Lying to You (and How to Close the Backtest-to-Live Gap)

joeschatzman — Wed, 23 Sep 2026 00:46:47 +0000

I once built a strategy with a beautiful backtest. Smooth equity curve, healthy Sharpe, a win rate that made me feel clever. I put it live.

It proceeded to lose money at roughly an 8% win rate and cost me five figures before I pulled the plug.

Nothing about the code was "wrong." The backtest was just lying to me — quietly, in the specific ways backtests always do. This post is the checklist I wish I'd run before going live. None of it is exotic; all of it is the difference between a number that flatters you and a process you can trust.

Why backtests overstate reality

A backtest is a simulation, and every simulation makes assumptions. The optimistic ones stack up:

Overfitting. If you tune parameters until the curve looks great, you haven't found an edge — you've memorized the noise in one slice of history. The more knobs you turn, the more certain this becomes.
Lookahead bias. Using information that wouldn't have been available at decision time. The classic version: computing a signal on today's close and then "buying at the close." In reality you'd act on the next bar. It's easy to leak the future without noticing.
Slippage and fills. Backtests love to fill you at the exact price you wanted. Live markets don't. On anything less than deeply liquid instruments, the gap between assumed and actual fills eats returns.
Survivorship bias. Testing on today's index members ignores every ticker that got delisted. Your universe is quietly pre-filtered for winners.
Regime dependence. A strategy tuned on a 2023–2024 bull run has never seen a real drawdown. It looks robust because it was never stressed.

Individually, each nudges results up a little. Together, they can turn a break-even system into a "genius" backtest.

How to actually close the gap

The goal isn't a prettier backtest. It's a strategy that behaves live roughly like it did in the test. Here's the sequence that gets you there.

1. Split your data — and mean it

Tune on one stretch of history, then evaluate on data the strategy has never touched. The simplest version: build on 2023–2024, validate on 2025. Better: walk-forward — repeatedly optimize on a window, test on the next unseen window, and roll it forward. If performance falls off a cliff out-of-sample, you found an artifact, not an edge.

# Conceptual walk-forward loop
for train_window, test_window in rolling_windows(data):
    params = optimize(train_window)      # fit on the past
    result = run(test_window, params)    # judge ONLY on unseen data
    oos_results.append(result)
# The out-of-sample curve is the one that matters.

2. Stress the trade order, not just the trades

A single equity curve is one lucky (or unlucky) ordering of your trades. Monte Carlo it: reshuffle the trade sequence thousands of times and look at the distribution of outcomes. If the 5th-percentile path is a catastrophe, your "safe" strategy isn't — you just got a friendly ordering the first time.

3. Model slippage and commissions on purpose

Add realistic per-trade cost and slippage assumptions and re-run. If a small, honest slippage estimate erases the edge, the edge was never real — it lived in the frictionless fantasy of a naive fill model.

4. Paper trade before real capital

Run the exact same logic against live market data with no money on the line. This is where lookahead bugs and data-feed quirks surface — the ones no historical test can catch because they only exist in the seam between "backtest engine" and "live engine." Ideally the same code runs both, so there's no second implementation to drift.

5. Put the guardrails in before you need them

Even a validated strategy meets a market it didn't expect. Decide your limits up front and let the machine enforce them:

position sizing rules (fixed fractional, volatility-scaled — pick one and stick to it),
a hard per-trade risk cap,
and a drawdown circuit breaker that halts the strategy automatically if it hits your max-loss line.

The point isn't to predict the bad day. It's to make sure the bad day can't compound while you're asleep.

The honest checklist

Before anything goes live, I want a "yes" to all of these:

[ ] Does it survive out-of-sample / walk-forward, not just the fitted window?
[ ] Is the Monte Carlo distribution acceptable at the 5th percentile, not just the median?
[ ] Does it still work with realistic slippage + commissions?
[ ] Did it run in paper trading and behave like the backtest?
[ ] Are position sizing, risk limits, and a drawdown halt actually wired in?

If any answer is "no," I don't have an edge yet. I have a hypothesis and a pretty chart.

A note on tooling

You can do all of this by hand — walk-forward loops, a Monte Carlo reshuffler, a slippage model, a paper-trading harness, a risk monitor. I eventually got tired of rebuilding that scaffolding for every idea and built a self-hosted framework where the same engine backtests and trades live, so there's no second implementation to drift. That's AlgoDeploy if you're curious — but the method above is the thing that matters, whatever you run it with.

The one-sentence version

A great backtest is easy; a real edge is hard — and the entire job of a serious process is to tell them apart before you fund the account, not after.

Backtested and hypothetical results have inherent limitations and do not guarantee future performance. Nothing here is investment advice — it's about software and process. Trade your own decisions.

I built a self-hosted algo-trading stack in Python — backtest to live, no black box

joeschatzman — Wed, 29 Jul 2026 12:47:41 +0000

Every "algo trading platform" I tried made me pick one of two bad options:

A hosted black box — upload your strategy to someone else's cloud, run it on their runtime, pay a monthly fee, and hope their fills resemble reality.
Roll your own — weeks of plumbing (broker APIs, data feeds, order lifecycle, risk management) before you place a single live trade.

I wanted a third option: bring my own Python strategy and run it on infrastructure I control — my machine, my broker keys, no cloud lock-in. Here's what that actually takes, the parts that bit me, and the tool I ended up building.

The strategy is the easy part. It's everything around it that's hard.

1. Backtest-to-live parity (the #1 way backtests lie)

The most common way a backtest lies to you: your live code path differs from your backtest code path. If your backtest fills at the bar close but live fills at next-bar-open with slippage, your equity curve is fiction.

The fix is architectural: the same engine runs both. Signal evaluation, position sizing, and risk checks share one implementation — only the data source and the execution target swap out. A vectorized pass gives you speed for research; an event-driven pass gives you realism (next-bar fills, slippage, commissions) and is the exact code that runs live.

If your "live trader" is a separate implementation from your backtester, you don't actually know what you're deploying.

2. Broker abstraction — Alpaca vs Interactive Brokers

These two brokers could not be more different to integrate:

Alpaca is a clean REST API. Keys in, JSON out.
IBKR is a socket API through TWS or IB Gateway that has to be running, with session quirks, contract qualification, and a forced daily restart.

If your strategy code talks directly to either one, you're locked in. The fix is a thin broker interface — place_order, get_positions, get_quote, get_bars — with adapters behind it. Strategy code never sees broker-specific objects.

A few real IBKR gotchas I hit building the adapter, in case they save you time:

Contracts must be "qualified" (populate conId) before you can request market data or place orders — otherwise reqTickers throws.
IBKR returns 1.7976931348623157e+308 (max double, its "UNSET" sentinel) for empty price fields — a market order will report a bogus ~1.8e308 limit price if you don't strip it.
The daily gateway restart means a 24/7 live strategy needs reconnection handling.

None of that is in the strategy. All of it has to be abstracted away so the strategy stays portable.

3. Risk management as a first-class citizen — not an afterthought

Bugs happen. A runaway loop or a bad signal shouldn't drain an account. So risk lives in the
execution path:

Pre-trade checks — position limits, buying power, max order size.
A runtime drawdown circuit breaker — halts trading when equity draws down past a threshold.

If risk is a wrapper you can forget to call, it's not risk management.

4. Be honest about what's still hard

No backtest fully models slippage, fill timing, or market impact — configurable slippage and commission modeling narrows the gap but doesn't close it. And live-trading reliability (the IBKR gateway restart, reconnection, monitoring) is genuine operational work. Anyone selling you "backtest = live" is selling you something.

What I ended up building: AlgoDeploy

I packaged all of the above into AlgoDeploy — backtest,
risk-manage, and go live on your own Alpaca or Interactive Brokers account (US equities,
single-leg options, crypto). You build strategies in a no-code dashboard, a YAML config, or
Python — all the same engine underneath. It runs on your machine with your keys; nothing is
routed through a hosted backend.

It's a one-time license (you own the version you buy), source-available, with a 7-day free trial. Honest scope: IBKR is US equities + single-leg options + crypto today; multi-leg spreads, futures, and forex are on the roadmap.

If you've ever wanted to run your own strategies without renting a platform or building all
the plumbing yourself, I'd genuinely value your feedback — especially on what's missing for
your workflow.

👉 https://algo-deploy.com

How I Built a Python Backtesting Engine with 623 Tests

joeschatzman — Mon, 08 Jun 2026 23:16:17 +0000

I spent the last year building a backtesting and live trading engine in Python. It started as a personal tool — I was tired of rewriting the same plumbing every time I wanted to test a strategy. Here's how it turned into a real product.

The Problem

Every time I wanted to backtest a trading strategy, I ended up writing the same boilerplate:

Pull OHLCV data from somewhere
Wire up indicators
Write entry/exit logic
Track positions, P&L, drawdowns
Build some kind of report
Hope nothing broke when I changed one thing

I tried the existing tools. Some were abandoned. Some forced you into their API patterns. Some were notebooks-only with no path to live execution. None of them let me go from "I have an idea" to "it's running live" without rewriting half the code.

So I built my own.

What I Built

AlgoDeploy is a Python engine that covers the full loop:

Strategy definition → Backtesting → Risk management → Live execution

The key design decision was making strategies declarative. Most strategies don't need custom Python — they're just indicator conditions, entry rules, exit rules, and position sizing. So I built a YAML config layer:

symbol: SPY
start: 2020-01-01
end: 2025-01-01
capital: 100000

entry:
  - close < sma(40)
  - rsi(14) < 30

exit:
  hard_stop: 2%
  trailing: 3%
  take_profit: 8%

position_size: 5%

That config is a complete strategy. No Python required. But if you need custom logic, you can drop to Python and use the same engine programmatically.

Architecture

The engine has a few distinct layers, and keeping them separate was the most important decision I made:

Indicators

11 built-in indicators (ATR, EMA, SMA, RSI, MACD, Bollinger, VWAP, Beta, OBV, Stochastic, ADX), all computed as pandas Series from OHLCV data. The indicator layer is stateless — it takes a DataFrame and returns Series. This means you can test indicators independently, swap them out, or add custom ones without touching anything else.

Comparators

Entry conditions are built from composable comparator functions: greater_than, less_than, crosses_above, crosses_below, within_pct_of, between. In YAML, you write close > sma(20). Under the hood, it's parsed into a comparator chain that evaluates on each bar.

Exit Rules

Hard stops, trailing stops, take-profit targets, time-based exits, and scale-outs. Each exit rule is independent — you can stack them, and the first one that triggers wins. This was important because exit logic is where most backtesting frameworks get messy.

Risk Layer

This runs independently from strategy logic. Drawdown limits, daily loss caps, position-level risk caps, exposure constraints, and a "require stop" rule that rejects any order without a stop price. The risk layer can veto any trade the strategy wants to make.

Keeping risk separate from strategy means you can swap strategies without re-implementing risk checks, and you can tighten risk rules without touching strategy code.

Position Sizing

Kelly Criterion, volatility-scaled (target a dollar risk per trade), and fixed fractional (risk X% of equity per trade). Each sizer takes the current portfolio state and returns a position size. Again, independent from everything else.

Testing

The project has 623 automated tests. That's not a vanity metric — it's a survival mechanism.

When you're dealing with financial calculations, a subtle bug in position sizing or exit logic can silently produce wrong results. You won't see it in a stack trace. You'll see it when your backtest returns don't match reality.

So I test aggressively:

Unit tests for every indicator, comparator, exit rule, position sizer, and risk check
Integration tests for the backtest engine (known inputs → verified outputs)
End-to-end tests for the full pipeline: config → data → backtest → report
Edge cases: zero-volume bars, gaps, splits, single-bar strategies, empty universes

The test suite runs in CI on every commit. If I change how trailing stops calculate, I know within seconds whether anything else broke.

The Dashboard

I built a web UI on top of the engine because not everything needs to be done in code. The dashboard lets you:

Configure strategies with dropdowns and form fields (no YAML or Python needed)
Run backtests and see results in real-time via WebSocket updates
View metrics: total return, CAGR, Sharpe, Sortino, max drawdown, win rate, profit factor
Compare against benchmarks (SPY, QQQ, etc.)
Save and load strategy configurations

The dashboard is a local web app — it talks to the same Python engine underneath. Nothing goes to the cloud.

Reports

Backtest results export as self-contained HTML files. One file, no server needed, you can email it or archive it. They include equity curves, drawdown charts, trade logs, and all the summary metrics.

Live Trading

The engine connects to Alpaca for live execution. The same strategy that ran in a backtest can run live with one config change. The live runner follows a state machine: scan universe → evaluate entry rules → risk check → size position → execute. It logs every decision for audit.

There's also a paper trading mode that logs orders without placing them, so you can validate before risking real money.

What I Learned

Declarative first, escape to code. Most users don't want to write Python for a simple moving average crossover. But some users need custom logic. Supporting both from the same engine was the right call.

Risk is not part of strategy. This was the biggest architectural win. When risk checks are independent, you can reason about them separately. "Does my strategy generate good signals?" and "Am I risking too much?" are different questions.

Test financial code like your money depends on it. Because eventually, it does. A 0.1% rounding error in position sizing compounds fast.

Reports need to be portable. Nobody wants to spin up a Jupyter server to share backtest results. Self-contained HTML files solved this completely.

Current Status

The engine is feature-complete and I'm looking for beta testers to stress-test it before public launch. If you're interested in algo trading and want to try it, the site is at algo-deploy.com.

Happy to answer questions about the architecture, the testing approach, or anything else in the comments.