I tried to beat the S&P 500 with machine learning. The bugs taught me more than the model.

rahairi rani — Sun, 18 Oct 2026 09:48:52 +0000

The short version

I could not build a stock picker I can prove beats the S&P 500, and the most useful thing I found was a list of bugs that made weak results look strong. Over several weeks I tested CAN SLIM rules, a momentum study, a volatility-targeting overlay, a machine-learning ranking model and, last, candlestick patterns. Every test had its pass/fail bar written down before it ran.

The ranking model has a small, real-looking signal: a rank correlation of about +0.03 to +0.04, with a t-statistic of 2.5. As a portfolio it earned roughly 20% to 25% a year against SPY's 13.8%. It did that with about 1.4 times SPY's market exposure and a worst drawdown near -50%. Its risk-adjusted return was statistically indistinguishable from SPY's.

So I froze one version and started a forward paper test on Shariah-compliant stocks, benchmarked to SPUS. It began on 16 October 2026, so it has no verdict yet. Candlestick patterns, tested afterwards both inside the model and as plain trading rules, added nothing. This post is about how I got here, what broke along the way, and what I will and will not claim.

The setup: write the bar first, then run the test

The goal was simple: beat SPY after costs, on a risk-adjusted basis, with rules I could run unattended. I used daily prices from EODHD and company filings from SEC EDGAR. The machine-learning work ran on a point-in-time table of about 450 liquid US stocks per date, including delisted names, with roughly 1.8 million rows from 2010 onward.

The discipline mattered more than any model. Before each test I wrote down what would count as a pass. Usually that meant beating SPY in both 2015-2020 and 2021 onward, a worst drawdown no deeper than SPY's, and a bootstrap confidence interval for the Sharpe difference that stayed above zero.

Then I ran it once and reported the result, including the failures. After dozens of tests on the same history, something will look good by luck. Fixed bars are the only defence I know against fooling myself.

Attempt 1: CAN SLIM rules lost to doing nothing

On today's S&P 500 members from 2020 to 2026, my price-only version of CAN SLIM lost 12.1% over six years while SPY gained 158%. I coded the method as I understood it: a market-direction gate, relative strength, base patterns, breakout volume, a 7% stop and an 8-week hold rule. That test list already flatters the strategy, because today's members are the survivors.

The trade statistics showed why. Of 255 trades, only 16% won. Just 37% ever reached +5%, and only 6% reached +20%. Most breakouts simply did not run.

Two changes helped. Dropping a rule that moved stops to breakeven when the market was "under pressure" cut the loss to 3.5%. Allowing new buys in that pressure state, while still blocking them below the 200-day average, lifted a 2015-2019 hold-out from +3.8% to +34.3% and cut the worst drawdown from 23.3% to 13.4%. It was better in four of five years, but one year (2017) made most of the profit, and SPY made +72.9%.

The real surprise was the gate. Holding SPY only in confirmed uptrends returned +24.6% over 2020-2026, against +158% for buy and hold. Skipping only the days below the 200-day average kept +108% and cut the worst drawdown from 33.7% to 19.8%. The 200-day line was the useful part of the gate.

Adding strict earnings and sales filters removed 96% of candidates and left 18 trades, too few to say anything.

Attempt 2: momentum faded, the volatility overlay survived

Rank momentum passed its first test and then fell apart under attack. I ranked stocks on 12-month return, skipping the latest month. It cleared its entry bar with a pooled t-statistic of 3.3, but the edge vanished when I removed the five best industries in 2010-2014. It also failed a 2003-2009 hold-out (t of -0.2 and a -61% drawdown).

A by-year breakdown explained the original result: 62% of its excess return came from 2024-2026, only 12% of the months. That is exactly where a list of today's index members is most biased toward winners.

The volatility-targeting overlay held up better, though not as a way to beat the market. The rule scales SPY exposure toward a 15% volatility target using 20-day volatility, capped at 1x, with a 10-point band and trades at the next close. Against a constant 80% SPY mix its Sharpe ratio was 0.10 higher, but the confidence interval (-0.03 to +0.26) just missed excluding zero.

The drawdown benefit was consistent, though. The worst drawdown was 34.7% against 49.7% for the constant mix, and in 2008 it was 34.2% against 54.8% for SPY. It is a sound risk control, not a source of alpha.

Attempt 3: a ranking model with a t-statistic of 4.5

My first machine-learning result looked excellent, and I nearly stopped there. The model was LightGBM, trained to predict each stock's 20-day return rank among its peers on the same date. It used 28 features: momentum, volatility, distance from highs, liquidity, beta, skew and five built from SEC filings.

I evaluated it walk-forward. Every January it retrained on past data only, with a 20-day purge, and then it was scored on non-overlapping 20-day holds from 2015 to 2026.

The first run gave a rank correlation of +0.053 and +0.041 in the two halves. The best fifth beat the worst fifth by 1.45% and 1.21% per 20 days, with a pooled t-statistic of 4.5. It beat a fixed composite of nine textbook signals and plain momentum. A t of 4.5 is the kind of number that should make you suspicious, and it should have.

Three bugs that made the model look better than it was

Auditing the data builder cut the model's pooled t-statistic from 4.5 to about 2.5. Each bug was quiet, and each one flattered the result.

Delisting losses booked twice. When a stock vanishes, the table books a -30% loss on its last row. The builder booked it on the last two rows of every series that ended, including acquisitions at a premium. The giveaway was 383 such rows across 194 tickers, exactly two per ticker. Because the model learned to avoid stocks about to disappear, fixing this alone removed roughly a third of the measured edge.
A leak test that could not see the leak. My original self-test checked that prices after a cut date could not change earlier features. When I planted a look-ahead in the fundamentals, the old test passed it. I wrote a new test that removes every filing made after the cut date, and it failed on my real data. A trailing four-quarter sum was treated as public on the date of its newest quarter, even if an older quarter in the sum had been first reported later. Now a trailing sum becomes available only when all four quarters are public.
Missing data that predicted death. Delisted companies mostly had no match in the SEC's current ticker file, so their fundamentals were blank. Among stocks that later left the universe, 62.3% of rows had no fundamentals, against 9.1% for stocks that stayed. A model can read "blank" as "will disappear". Missing values are now filled with random ranks, so they carry no information.

The model barely needed the fundamentals anyway. Removing all five SEC-based features left its edge intact, so the final headline specification uses price features only.

The honest numbers: more return, the same return per unit of risk

The cleaned-up model earned more than SPY, but almost all of the extra came from taking more risk. The table shows 2015-2026 for a 20-stock portfolio drawn from the 100 most liquid names, with idle money held in SPY under my volatility rule. Costs are 5 basis points per side.

Portfolio	Return a year	Sharpe	Worst drawdown	Beta to SPY
SPY buy and hold	13.8%	0.83	-33.7%	1.00
SPY under the volatility rule	11.1%	0.88	-18.4%	0.69
Model portfolio	24.8%	0.92	-52.4%	1.37
Model, sold to SPY after two high-volatility days	21.5%	0.97	-35.9%	0.92

The model's alpha was +6.5% a year with a t-statistic of 1.5, which is not significant. The confidence interval for its Sharpe advantage over the volatility rule ran from -0.33 to +0.39, so zero is well inside it.

The edge also faded. In 2015-2020 the alpha was +12.5% a year (t of 2.7). From 2021 onward it was -1.0% (t of -0.1), and the worst drawdown in that period was 52.4%, against 24.5% for SPY.

Restricting to Shariah-compliant stocks made the backtest look better, and I do not trust it

I wanted the system to trade only stocks I could hold under Shariah screening, so I limited it to the holdings of SPUS, the S&P 500 Shariah ETF. The backtest improved sharply: 28.6% a year against SPY's 13.8%, with an alpha of +10.2% (t of 2.7).

Then I looked at the baseline. An equal-weight basket of the same stocks, with no model at all, also beat SPY by 3.6 points a year (17.4% against 13.8%). A diversified basket should not do that unless the list is built from hindsight.

The list is today's holdings, which means it contains the companies that stayed large, listed and compliant. The model's ranking skill inside the list was about what it was before (rank correlation of +0.025 and +0.028), so the extra return is mostly the list, not the model.

The lesson is blunt: no backtest on today's holdings list can tell you what you would have earned. Only forward data can. I also cannot certify that any stock is compliant. The test trades whatever list I give it, so I rely on the fund's published holdings and refresh them when the index changes.

Everything I tried to improve it failed the same bar

Six variants of gates and exits all failed my pre-registered bar. From 2021 onward none beat SPY under my volatility rule on Sharpe ratio, and none kept its worst drawdown within SPY's. Every bootstrap interval for the Sharpe difference included zero.

Variant	Return a year	Sharpe	Worst drawdown
V0 control (the frozen test)	24.8%	0.92	-52.4%
V1 buy only when the volatility gate is on	22.2%	0.89	-48.9%
V2 V1, plus sell everything after two gate-off days	21.5%	0.97	-35.9%
V3 CAN SLIM style: top 5, 8% stop, 50-day exit	27.6%	1.01	-41.9%
V4 V1, only stocks with beta below 1.2	15.8%	0.97	-32.0%
V6 two-stage gate	20.8%	0.87	-47.7%

Three things stood out:

The volatility gate was off on only 17% of days, so blocking new buys changed little. Retreating to SPY (V2) cut the drawdown but cost return.
V3, the design I preferred, had the best alpha overall (+14.7% a year, t of 2.2). All of it came from 2015-2020 (+26.5%, t of 3.2). From 2021 it was +1.6% with a t of 0.2.
Changing the model itself also failed. Training on beta-adjusted returns cut the top-ranked fifth's beta by 0.25 to 0.40, but the pooled t fell from 2.5 to 0.6. Raw volatility and beta levels made it worse (0.1), and averaging five random seeds changed nothing (2.5). About a third of the model's post-2021 edge was extra beta.

The forward test: freeze the model and let time decide

Because every year of history has been inspected many times, the only clean evidence left is data that does not exist yet. So I froze one version, called V0, and started trading it on paper.

The rules are fixed. A LightGBM model with price-only features scores only SPUS names. Every five trading days it sells a holding if its rank falls below 60 or it is down 25%, then buys the best-ranked names it does not hold, up to 20. Orders fill at the next close with 5 basis points per side, and the benchmark is SPUS.

I fixed the success bar before the first trade. After at least 50 decision days, about a year, the model's rank correlation with what stocks actually did over the next 5 days must average +0.02 or more, with a t-statistic of at least 2. I chose that measure because each decision day scores about 100 names, which gives far more observations than a portfolio return does.

Each week the automated job runs the same cycle:

Refresh prices and compute the live features after Friday's US close.
Compare the live features with the training table on the dates they share.
Score the Shariah-compliant list and apply the buy and sell rules.
Append the decisions to an append-only ledger.
Rebuild the paper portfolio from the ledger and fresh prices, and compare it with SPUS.
Back up the ledger and record its SHA-256 hash, so any later edit shows up.

The job runs on a small server, uses a lock so two runs cannot overlap, stops when the price data is stale, and sends a Telegram alert on any failure.

The safeguard earned its keep on day one. The server refused to record its first decision because one live feature differed from the table by 0.862 on a scale of -1 to +1. A fuller report showed that only 4 of 448 rows differed materially, 0.04% of cells, and that the model scored both feature sets almost identically (rank correlation 0.991). It was a harmless data quirk, but a single number could not have told me that, so the check now judges whether the model's answer changes.

One caution about what the first decision shows. Twelve of its 20 stocks were semiconductor, memory, storage or optical-hardware names, which makes it close to one bet on that theme. Early paper returns will say more about that theme than about the model.

Candlestick patterns: two tests, the same answer

Candlestick patterns added nothing, whether I fed them to the model or traded them as plain rules. I ran two separate tests after the forward test started, each with its bar written first.

The first test was a machine-learning ablation. I gave the frozen model's price-only feature set 17 candle-shape features: body size, upper and lower wicks, where the close sits in the day's range, the opening gap, and the same values for the two previous days. The bar was a gain in rank correlation above zero in both periods, with a paired t of at least 2.

The rank correlation fell slightly, from +0.034 to +0.030 (paired t of -1.2). At a 5-day horizon the candles actively hurt (t of -2.4). On their own, the candle features had a small positive signal in 2015-2020 that turned negative from 2021. An effect that flips sign is not one you can trade.

The second test used no machine learning at all. I coded four classic bullish patterns as fixed rules: hammer, bullish engulfing, piercing line and morning star. They only counted in an uptrend, right after a fresh 10-day low. Each was compared with the same situation without a pattern, so the candle had to beat the pullback itself.

The universe was about 2,000 US stocks including delisted ones, ranked to the 450 most liquid each day, from 2010 to 2026. The outcome was the 10-day return after entry minus SPY's. Because four patterns were tested, the bar was a t of 2.5, counted over 10-day blocks rather than individual events, since patterns cluster on sell-off days.

Pattern	Events	Excess over the no-pattern baseline (10 days)	t	Verdict
Hammer	6,630	+0.15%	1.2	Fail
Bullish engulfing	6,727	-0.13%	-1.0	Fail
Piercing line	2,465	+0.01%	0.0	Fail
Morning star	5,377	0.00%	0.0	Fail

The setting itself had no edge either. Any day in an uptrend after a fresh 10-day low beat SPY by just +0.07% over the next 10 days.

The first run of this test also produced the most obviously wrong number of the whole project. The no-pattern baseline beat SPY by an average of 88.6% in 10 days. A few broken price series caused it, one of them showing a 29,000-fold gain in two weeks. A data check removed 861 events near impossible one-day price jumps, plus 34 warrants and units, and the baseline fell to the +0.07% above. The real outliers that remained, such as GameStop in January 2021, stayed in.

What I learned, and what I will not claim

The main lesson is to treat any good-looking result as a bug until it survives an attack. These are the habits that paid off:

Write the pass/fail bar before the test. It stops you from moving the goalposts when the answer disappoints.
Test your tests. Plant a known leak and check that your detector catches it. Mine did not, until I rewrote it.
Look for fingerprints. "Exactly two rows per ticker" was the clue to the double-counted delistings. A baseline that beat SPY by 88% in 10 days, with a win rate no higher than the patterns', pointed straight at a few broken price series.
Judge risk-adjusted return, not raw return. A higher return from a beta of 1.4 is not skill.
Distrust any list built today. Current index members and ETF holdings carry hindsight.
Build alarms that explain themselves. A single scary number cost me a day. A report that shows where the difference is saved it.

What I will not claim: that this model beats the market, that its past edge will repeat, or that anyone should trade it. After the fixes it is a weak ranking signal that faded after 2020, and I have no proof of an edge.

The forward test continues, and I will write up the first honest read after about 50 decision days. If the rank correlation comes in near zero, the practical answer is the unglamorous one: hold a plain index, perhaps with a volatility rule to cap drawdowns.

All figures come from my own backtests with adjusted prices and 5 basis points of cost per side, and ignore taxes and market impact. SPY stands in for SPUS before its 2019 launch. This is a personal research log, not financial advice.

DEV Community: rahairi rani