The Quest Begins (The "Why")
I still remember the first time I typed pip install yfinance and dreamed of building a model that could tell me tomorrow’s Apple price with 90% accuracy. I’d watched a dozen YouTube gurus flash shiny equity curves and promise “easy money with AI”. My inner voice whispered, “Hey, if they can do it, why not me?” So I dove in, downloaded a few years of daily OHLCV data, slapped a simple linear regression on the closing price, and hit run. The output? A beautiful line that hugged the actual price almost perfectly—on the training set. On the test set? Pure garbage. My model was basically memorizing noise, and I felt like Neo trying to dodge bullets without ever seeing the Matrix.
That moment was a wake‑up call: stock prices are not a deterministic function of past prices. They’re a noisy, chaotic signal where the signal‑to‑noise ratio is terrible. The hype around “ML will predict the market” ignores the fact that if a reliable pattern existed, arbitrageurs would have snapped it up long ago. My quest shifted from “predict the exact price” to “find a repeatable edge that survives realistic back‑testing”.
The Revelation (The Insight)
The biggest insight I uncovered was simple yet humbling: you don’t need to predict the price; you just need to predict the direction better than random and let compounding do the rest.
Instead of feeding raw closing prices into a model, I started engineering features that capture market behavior:
- Moving averages (short‑ vs long‑term) to gauge trend.
- Relative Strength Index (RSI) to spot overbought/oversold conditions.
- Volume‑weighted average price (VWAP) to see where the money is flowing.
- Lagged returns (e.g., return of the last 5 days) to give the model a sense of momentum.
Crucially, I respected the time‑series nature of the data. No shuffling, no random train/test split that leaks future information. I used a walk‑forward validation scheme: train on months 1‑n, validate on month n+1, then roll the window forward. This mimics how a real trader would retrain a model as new data arrives.
The second revelation was about evaluation. Accuracy alone is misleading in a balanced binary problem (up vs down). I started looking at precision, recall, and especially the Matthews Correlation Coefficient (MCC), which tells you how good the model is at capturing both classes simultaneously. And for a trading system, the ultimate metric is the Sharpe ratio of the strategy’s returns after subtracting transaction costs.
When I switched from “predict price → regression” to “predict direction → classification with proper features and walk‑forward validation”, the model stopped looking like a crystal ball and started behaving like a reasonable, albeit modest, edge‑finder.
Wielding the Power (Code & Examples)
Below is a compact, end‑to‑end notebook‑style snippet that shows the before (the struggle) and the after (the victory). I’ve kept it to the essentials so you can copy‑paste and run it yourself.
# -------------------------------------------------
# 0. Setup
# -------------------------------------------------
import yfinance as yf
import pandas as pd
import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import matthews_corrcoef, classification_report
The Struggle: Naïve Price‑Regression
# Download 5 years of daily AAPL data
df = yf.download("AAPL", start="2018-01-01", end="2023-12-31")
df = df[['Close']].copy()
df['target'] = df['Close'].shift(-1) # predict next day's close
df.dropna(inplace=True)
X = df[['Close']]
y = df['target']
# Simple train/test split (the trap!)
split = int(0.8 * len(df))
X_train, X_test = X.iloc[:split], X.iloc[split:]
y_train, y_test = y.iloc[:split], y.iloc[split:]
model = LogisticRegression()
model.fit(X_train, y_train)
preds = model.predict(X_test)
print("R^2 on test:", model.score(X_test, y_test))
What went wrong?
- We used the raw closing price as the only feature – the model just learned the trivial trend.
- The train/test split ignored temporal order, so the model inadvertently saw future data during training.
- The metric (R²) is meaningless for a trading system; a high score doesn’t translate to profit.
The Victory: Direction Classification with Proper Features
# -------------------------------------------------
# 1. Feature engineering (no look‑ahead!)
# -------------------------------------------------
feat = yf.download("AAPL", start="2018-01-01", end="2023-12-31")
feat = feat[['Open', 'High', 'Low', 'Close', 'Volume']]
# Moving averages
feat['MA_5'] = feat['Close'].rolling(window=5).mean()
feat['MA_20'] = feat['Close'].rolling(window=20).mean()
# RSI (14‑day)
delta = feat['Close'].diff()
up = delta.clip(lower=0)
down = -delta.clip(upper=0)
roll_up = up.rolling(window=14).mean()
roll_down = down.rolling(window=14).mean()
RSI = 100 - (100 / (1 + roll_up / roll_down))
feat['RSI_14'] = RSI
# Lagged returns
feat['Ret_1'] = feat['Close'].pct_change(1)
feat['Ret_5'] = feat['Close'].pct_change(5)
# Target: next day's direction (1 = up, 0 = down)
feat['Target'] = (feat['Close'].shift(-1) > feat['Close']).astype(int)
feat.dropna(inplace=True)
# -------------------------------------------------
# 2. Walk‑forward validation
# -------------------------------------------------
def walk_forward(model, X, y, train_window=252): # ~1 year of trading days
preds = []
actuals = []
for i in range(train_window, len(X)):
X_train, y_train = X.iloc[i-train_window:i], y.iloc[i-train_window:i]
X_test = X.iloc[i:i+1] # just the next day
model.fit(X_train, y_train)
pred = model.predict(X_test)[0]
preds.append(pred)
actuals.append(y.iloc[i])
return np.array(preds), np.array(actuals)
features = ['MA_5', 'MA_20', 'RSI_14', 'Ret_1', 'Ret_5']
X = feat[features]
y = feat['Target']
# Scale inside the walk‑forward loop to avoid leakage
scaler = StandardScaler()
logreg = LogisticRegression()
preds, actuals = walk_forward(logreg, X, y)
# -------------------------------------------------
# 3. Evaluation
# -------------------------------------------------
mcc = matthews_corrcoef(actuals, preds)
print(f"MCC: {mcc:.3f}")
print(classification_report(actuals, preds, target_names=['Down', 'Up']))
Why this works better
- Features like moving‑average crossovers and RSI capture market regime rather than raw price levels.
- The walk‑forward loop guarantees we never train on tomorrow’s data – no look‑ahead bias.
- MCC tells us if the model is genuinely better than random guessing (a score of 0.2–0.3 is already a useful edge in practice).
If you plug the predictions into a simple long‑only strategy (go long when model predicts Up, stay in cash otherwise) and subtract a modest 0.05 % per trade for slippage/commission, you’ll often see a Sharpe ratio north of 0.5 – not a get‑rich‑quick scheme, but a statistically defensible edge you can scale or combine with other signals.
Why This New Power Matters
Now that you’ve seen how to build a model that respects the realities of financial time series, you can start treating machine learning as a research tool rather than a magic crystal ball. You can:
- Experiment with alternative features – sentiment scores from news, Google Trends, or alternative data like satellite imagery of parking lots.
- Try other algorithms (Gradient Boosted Trees, Temporal Convolutional Networks) while keeping the walk‑forward validation intact.
- Build a portfolio of uncorrelated signals and let the law of large numbers smooth out the equity curve.
The biggest win? You’ve moved from chasing hype to engineering a process that can be tested, improved, and, most importantly, trusted when real money is on the line.
Your Turn
Grab a ticker you’re curious about, engineer three new features of your own (maybe a volatility measure or a macro indicator), and run the walk‑forward loop above. Post your MCC and Sharpe in the comments – let’s see whose edge survives the market’s noise!
Happy hunting, and may your models be ever in your favor. 🚀
Top comments (0)