Most traders lose money even when they’re right about direction.
The problem isn’t their thesis. It’s that they trade on one signal, one indicator, one gut feeling — without a true probability framework. Markets punish that reliably.
Neural networks solve a different problem entirely. They learn the conditional expectation E[Y|X] — the statistical relationship between what you can observe right now and what the market is most likely to do next — across thousands of variables simultaneously.
This is how Two Sigma runs 10,000+ live signals, how Citadel powers its quant desks, and how Renaissance built the Medallion Fund (66% annualized before fees for 30+ years).
Here’s the complete framework you can implement today.
Part 1: What a Neural Network Actually Computes
Key insight: When trained to minimize squared error, the network learns the conditional expectation E[Y | X].
Proof sketch: expanding the loss shows the optimal ( f(X) ) is exactly E[Y | X]. The network isn’t guessing the next outcome — it’s computing the mathematically optimal expected value given the inputs.
Part 2: Why Direct Price Prediction Fails (And the Fix)
Feed 500 days of closing prices into an LSTM to predict day 501 → beautiful in-sample, useless out-of-sample.
This isn’t model failure. It’s non-stationarity. Financial data distributions shift across regimes, so the learned conditional expectation becomes invalid.
Solution: Engineer stationary features:
- Log returns over multiple windows:
- Volatility ratios:
- Momentum normalized by volatility:
- Volume z-scores, spread signals, regime indicators
Test every feature with the Augmented Dickey-Fuller test (p < 0.05 = stationary).
Target variable: binary direction (positive risk-adjusted return) or z-scored returns — far more stable than raw prices.
Part 3: LSTM — The Right Architecture for Sequential Market Data
Market data has temporal dependencies. Standard feedforward nets ignore them.
Start with lookback of 10–20 periods for daily data (or 24 for 5-min bars) and tune empirically.
Part 4: Training Without Fooling Yourself
Use a sequential three-way split (never random shuffle):
- Training → Validation (early stopping) → Test (used only once)
Implement walk-forward validation for realistic out-of-sample results.
Expected directional accuracy for a good model: 52–57%.
Paired with proper Kelly sizing and consistency, this compounds into serious edge.
Part 5: The Complete Production Pipeline
- Data → Polygon.io / yfinance
- Stationary feature engineering + ADF tests
- Sequential split + scaling
- LSTM training with early stopping + gradient clipping
- Signal → Half-Kelly position sizing
- Live monitoring (KS statistic) + rolling retraining every 30 days
Summary
Neural networks don’t give you a crystal ball. They give you a mathematically rigorous way to extract conditional expectations from data — if you use stationary features, the right architecture, disciplined training, and proper risk management.
The math is learnable. The code is buildable in a weekend. The only difference between you and the hedge funds is following the framework without shortcuts.
Drop your answer in the comments:
If you had to add exactly one new feature to your model that no other systematic trader is using, what would it be and why?
Further reading / resources:
- Author’s full quant roadmap (linked in original thread)
- Research the universal approximation theorem and non-stationarity in financial time series
Top comments (0)