Market regimes — bull, bear, high-volatility, credit-stress — are not directly observable. They must be inferred from noisy, multi-dimensional price data across equities, credit, and volatility. Traditional rule-based approaches (e.g., "VIX above 30 means fear") are brittle and incomplete. A hybrid machine learning approach that combines dimensionality reduction, unsupervised clustering, and probabilistic state modeling offers a far more robust framework for identifying the hidden structural states that drive asset returns.
In this article, we build a full regime detection pipeline from scratch using Python. We pull cross-asset data for SPY, IWM, HYG, LQD, and VIX from Yahoo Finance, engineer a rich feature set including credit spreads, realized volatility, and multi-horizon returns, then apply PCA for dimensionality reduction, K-Means with silhouette-based cluster selection, and visualize the resulting regimes over time. Every step is runnable end-to-end in a standard Python environment.
Most algo trading content gives you theory.
This gives you the code.3 Python strategies. Fully backtested. Colab notebook included.
Plus a free ebook with 5 more strategies the moment you subscribe.5,000 quant traders already run these:
Subscribe | AlgoEdge Insights
This article covers:
- Section 1 — What Is a Market Regime and Why ML?** Explains the conceptual motivation for regime detection, the limitations of rule-based methods, and the intuition behind using PCA + clustering on cross-asset features.
- Section 2 — Python Implementation:** Step-by-step code covering setup and imports (2.1), data ingestion and feature engineering (2.2), PCA and silhouette-optimized K-Means (2.3), and regime visualization (2.4).
- Section 3 — Results and Interpretation:** What the regimes actually look like, how to read cluster statistics, and what performance characteristics to expect.
- Section 4 — Use Cases:** Practical applications including regime-conditional allocation, signal filtering, and risk overlays.
- Section 5 — Limitations and Edge Cases:** Honest assessment of where this pipeline breaks down and what to watch for in production.
1. What Is a Market Regime and Why Machine Learning?
A market regime is a persistent statistical environment — a period during which asset returns, volatility, and correlations exhibit consistent structural behavior. Risk-on regimes are characterized by rising equities, tight credit spreads, and suppressed volatility. Risk-off regimes flip that picture: equities fall, high-yield credit widens, and the VIX spikes. The challenge is that these states are latent — they cannot be read directly from a single price series. They emerge from the joint behavior of multiple assets simultaneously.
Think of it like weather forecasting. You do not declare "winter" by measuring temperature alone. You observe temperature, humidity, pressure, and wind patterns together. The same logic applies here: a complete picture of market stress requires observing equities (SPY, IWM), credit risk (HYG, LQD), and implied volatility (VIX) as a system, not in isolation.
Rule-based regime filters — such as a 200-day moving average crossover or a fixed VIX threshold — fail because they rely on a single signal and use arbitrary cutoffs. Machine learning allows the data itself to reveal the natural cluster structure. PCA compresses the high-dimensional feature space into its most informative directions, removing noise and multicollinearity. K-Means then partitions the compressed data into coherent groups. The silhouette score acts as an objective measure of cluster quality, letting the algorithm choose the optimal number of regimes rather than hard-coding an assumption.
The result is a data-driven regime label for each trading day — a label that reflects the full cross-asset environment and can be used to condition portfolio allocation, filter trading signals, or manage risk exposures dynamically.
2. Python Implementation
2.1 Setup and Parameters
We use yfinance for data, scikit-learn for PCA and K-Means, and matplotlib for visualization. The key configurable parameters are the lookback windows for rolling features, the PCA variance threshold, and the range of K values to search over during silhouette optimization.
# pip install yfinance scikit-learn pandas numpy matplotlib
import yfinance as yf
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
import warnings
warnings.filterwarnings("ignore")
# ── Parameters ──────────────────────────────────────────────
TICKERS = ["SPY", "IWM", "HYG", "LQD", "^VIX"]
START_DATE = "2007-01-01"
END_DATE = "2024-12-31"
# Rolling window lengths (trading days)
SHORT_VOL_WIN = 21 # ~1 month realized vol
MED_VOL_WIN = 63 # ~1 quarter realized vol
RETURN_WINS = [1, 21, 63, 126] # momentum horizons
# PCA: keep components explaining this fraction of variance
PCA_VAR_THRESHOLD = 0.95
# Silhouette search range for K-Means
K_MIN, K_MAX = 2, 8
RANDOM_STATE = 42
2.2 Data Ingestion and Feature Engineering
We download adjusted closing prices for all five instruments, align them into a single DataFrame, and then construct a multi-dimensional feature set. The features capture momentum across four horizons, realized volatility, the credit spread between HYG and LQD as a proxy for risk appetite, SPY versus IWM relative performance, VIX level transformations, and rolling drawdown depth.
# ── Download and align price data ───────────────────────────
raw = yf.download(TICKERS, start=START_DATE, end=END_DATE,
auto_adjust=True, progress=False)["Close"]
raw.columns = ["HYG", "IWM", "LQD", "SPY", "VIX"]
raw.dropna(how="all", inplace=True)
raw.ffill(inplace=True)
df = pd.DataFrame(index=raw.index)
# ── Multi-horizon log returns ────────────────────────────────
for asset in ["SPY", "IWM", "HYG", "LQD"]:
for w in RETURN_WINS:
df[f"{asset}_ret_{w}d"] = np.log(
raw[asset] / raw[asset].shift(w)
)
# ── Realized volatility (rolling std of daily log returns) ──
for asset in ["SPY", "IWM"]:
daily_ret = np.log(raw[asset] / raw[asset].shift(1))
df[f"{asset}_rvol_{SHORT_VOL_WIN}d"] = (
daily_ret.rolling(SHORT_VOL_WIN).std() * np.sqrt(252)
)
df[f"{asset}_rvol_{MED_VOL_WIN}d"] = (
daily_ret.rolling(MED_VOL_WIN).std() * np.sqrt(252)
)
# ── Credit spread proxy (HYG underperformance vs LQD) ───────
df["credit_spread"] = (
np.log(raw["LQD"] / raw["LQD"].shift(21)) -
np.log(raw["HYG"] / raw["HYG"].shift(21))
)
# ── SPY vs IWM relative return (large vs small cap) ─────────
df["spy_iwm_rel"] = (
np.log(raw["SPY"] / raw["SPY"].shift(21)) -
np.log(raw["IWM"] / raw["IWM"].shift(21))
)
# ── VIX features ─────────────────────────────────────────────
df["vix_level"] = raw["VIX"]
df["vix_log"] = np.log(raw["VIX"])
df["vix_chg_1d"] = raw["VIX"].diff(1)
df["vix_vs_21d_avg"] = raw["VIX"] / raw["VIX"].rolling(21).mean() - 1
df["vix_vs_rvol"] = (
raw["VIX"] / 100 /
(np.log(raw["SPY"] / raw["SPY"].shift(1))
.rolling(SHORT_VOL_WIN).std() * np.sqrt(252) + 1e-8)
)
# ── Rolling drawdown depth (SPY) ─────────────────────────────
roll_max = raw["SPY"].rolling(126, min_periods=1).max()
df["spy_drawdown"] = raw["SPY"] / roll_max - 1
# ── Drop NaN / Inf rows before modeling ──────────────────────
df.replace([np.inf, -np.inf], np.nan, inplace=True)
df.dropna(inplace=True)
print(f"Feature matrix shape: {df.shape}")
print(f"Date range: {df.index[0].date()} → {df.index[-1].date()}")
2.3 PCA Dimensionality Reduction and Silhouette-Optimized K-Means
After standardizing all features to zero mean and unit variance, we compress the feature matrix with PCA retaining 95% of explained variance. We then search over K values from 2 to 8, computing the silhouette score for each, and select the K that maximizes cluster cohesion. The final K-Means model is fit on the PCA-reduced data, and regime labels are mapped back to the original date index.
# ── Standardize ──────────────────────────────────────────────
scaler = StandardScaler()
X_scaled = scaler.fit_transform(df.values)
# ── PCA ──────────────────────────────────────────────────────
pca = PCA(n_components=PCA_VAR_THRESHOLD, svd_solver="full",
random_state=RANDOM_STATE)
X_pca = pca.fit_transform(X_scaled)
n_components = X_pca.shape[1]
explained = pca.explained_variance_ratio_.cumsum()[-1]
print(f"PCA: {n_components} components explain {explained:.1%} of variance")
# ── Silhouette search ─────────────────────────────────────────
sil_scores = {}
for k in range(K_MIN, K_MAX + 1):
km = KMeans(n_clusters=k, random_state=RANDOM_STATE, n_init=20)
labels = km.fit_predict(X_pca)
sil_scores[k] = silhouette_score(X_pca, labels, sample_size=5000,
random_state=RANDOM_STATE)
print(f" K={k} silhouette={sil_scores[k]:.4f}")
best_k = max(sil_scores, key=sil_scores.get)
print(f"\nOptimal K: {best_k} (silhouette={sil_scores[best_k]:.4f})")
# ── Fit final model ───────────────────────────────────────────
km_final = KMeans(n_clusters=best_k, random_state=RANDOM_STATE, n_init=50)
df["regime"] = km_final.fit_predict(X_pca)
# ── Regime statistics ─────────────────────────────────────────
regime_stats = df.groupby("regime").agg(
count=("spy_drawdown", "count"),
avg_spy_ret_21d=("SPY_ret_21d", "mean"),
avg_vix=("vix_level", "mean"),
avg_credit_spread=("credit_spread", "mean"),
avg_drawdown=("spy_drawdown", "mean")
).round(4)
print("\nRegime Summary Statistics:")
print(regime_stats.to_string())
2.4 Visualization
The chart below plots SPY closing price over time with each trading day shaded by its detected regime. This allows an immediate visual audit: do the cluster boundaries align with known market stress periods such as 2008–2009, 2020, and 2022?
# ── Align regime labels with SPY price ───────────────────────
spy_aligned = raw["SPY"].loc[df.index]
regime_labels = df["regime"]
palette = ["#4FC3F7", "#EF5350", "#66BB6A", "#FFA726",
"#AB47BC", "#26C6DA", "#D4E157", "#FF7043"]
plt.style.use("dark_background")
fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(14, 8),
gridspec_kw={"height_ratios": [3, 1]})
fig.patch.set_facecolor("#0D0D0D")
# ── Top panel: SPY price colored by regime ───────────────────
for regime_id in sorted(regime_labels.unique()):
mask = regime_labels == regime_id
ax1.scatter(spy_aligned.index[mask], spy_aligned.values[mask],
s=1.5, color=palette[regime_id],
label=f"Regime {regime_id}", alpha=0.85)
ax1.set_yscale("log")
ax1.set_ylabel("SPY Price (log scale)", color="white", fontsize=11)
ax1.set_title("Cross-Asset Market Regime Detection: SPY Colored by ML Cluster",
color="white", fontsize=13, pad=12)
ax1.legend(loc="upper left", fontsize=9, framealpha=0.3,
markerscale=5)
ax1.xaxis.set_major_formatter(mdates.DateFormatter("%Y"))
ax1.tick_params(colors="white")
for spine in ax1.spines.values():
spine.set_edgecolor("#333333")
# ── Bottom panel: VIX time series ────────────────────────────
ax2.fill_between(raw["VIX"].loc[df.index].index,
raw["VIX"].loc[df.index].values,
alpha=0.6, color="#EF5350")
ax2.axhline(20, color="#FFA726", linewidth=0.8, linestyle="--",
label="VIX = 20")
ax2.axhline(30, color="#EF5350", linewidth=0.8, linestyle="--",
label="VIX = 30")
ax2.set_ylabel("VIX", color="white", fontsize=10)
ax2.legend(loc="upper right", fontsize=8, framealpha=0.3)
ax2.tick_params(colors="white")
ax2.xaxis.set_major_formatter(mdates.DateFormatter("%Y"))
for spine in ax2.spines.values():
spine.set_edgecolor("#333333")
plt.tight_layout(h_pad=1.5)
plt.savefig("regime_detection.png", dpi=150, bbox_inches="tight",
facecolor="#0D0D0D")
plt.show()
Figure 1. SPY daily closing prices (log scale, top) colored by ML-detected market regime alongside the VIX time series (bottom) — stress regimes visually cluster around the 2008 financial crisis, 2020 COVID shock, and 2022 rate-hike drawdown.
Enjoying this strategy so far? This is only a taste of what's possible.
Go deeper with my newsletter: longer, more detailed articles + full Google Colab implementations for every approach.
Or get everything in one powerful package with AlgoEdge Insights: 30+ Python-Powered Trading Strategies — The Complete 2026 Playbook — it comes with detailed write-ups + dedicated Google Colab code/links for each of the 30+ strategies, so you can code, test, and trade them yourself immediately.
Exclusive for readers: 20% off the book with code
MEDIUM20.Join newsletter for free or Claim Your Discounted Book and take your trading to the next level!
3. Results and Interpretation
On a dataset spanning 2007 to 2024, the silhouette optimization typically converges on four to five regimes. A representative run reveals a clear "crisis" cluster — roughly 8–12% of trading days — characterized by a mean VIX above 35, deeply negative 21-day SPY returns around -6% to -10%, a strongly positive credit spread (HYG dramatically underperforming LQD), and average SPY drawdown below -15%. This cluster captures the 2008–2009 financial crisis, the March 2020 COVID shock, and the late 2022 drawdown with high consistency.
A second cluster, representing approximately 50–60% of all trading days, reflects the canonical risk-on regime: VIX averaging 14–17, positive multi-horizon returns across SPY and IWM, tight credit spreads, and minimal drawdown. A third and fourth cluster typically represent transitional environments — moderately elevated volatility, mixed credit signals, and flat-to-mildly-negative returns. These transitional regimes are the most difficult to act on but are important for risk management: they often precede the full crisis cluster by several weeks.
The PCA step usually retains 7–10 components to achieve 95% explained variance from a feature matrix of 25–30 columns. The compression is meaningful — it removes correlated noise between overlapping return windows and volatility measures — without discarding the structural signal needed to separate regimes. Silhouette scores in the range of 0.15–0.30 are typical for financial time-series clustering; higher scores (>0.35) may indicate overfitting to a specific sample period and should be treated with skepticism.
4. Use Cases
Regime-conditional asset allocation. Map each detected regime to a target equity allocation — full risk in the risk-on cluster, reduced exposure in transitional regimes, and maximum defensiveness (cash, long volatility, short credit) in the crisis cluster. This acts as a systematic overlay on any underlying strategy.
Signal filtering. Mean-reversion and momentum signals behave differently across regimes. Momentum tends to work in trending, low-volatility environments; mean-reversion tends to work in high-volatility, choppy regimes. Regime labels can be used to activate or suppress specific signal types dynamically.
Risk and drawdown management. Position sizing models can consume the current regime label as a feature. In the crisis cluster, leverage is mechanically reduced; in the risk-on cluster, it is restored. This avoids the lag inherent in reactive volatility-scaling approaches.
Macro factor research. Regime labels derived from this pipeline can be used as a dependent variable to study which leading indicators (yield curve slope, ISM PMI, credit impulse) best predict regime transitions, building toward a forward-looking regime forecasting model.
5. Limitations and Edge Cases
Look-ahead bias in rolling features. All rolling return and volatility features are constructed using only past data, but the PCA and K-Means models are fit on the full sample. In a live deployment, these models must be retrained on an expanding window or walk-forward basis to avoid implicitly using future information in the regime labels.
Regime label instability. K-Means cluster assignments are not ordered or semantically stable across retraining runs. Regime "0" in one calibration period may correspond to the crisis state; in the next, it may correspond to the risk-on state. A production system requires a post-processing step that maps cluster centroids to interpretable economic labels using known reference periods.
Non-stationarity. Financial return distributions shift structurally over time. A model trained on 2007–2015 data may misclassify the 2020–2024 low-rate, high-dispersion environment. Regular recalibration — quarterly at minimum — is necessary to maintain classification quality.
Cluster count sensitivity. The silhouette score is a local optimality measure and does not guarantee the economically most meaningful partition. A K that scores slightly lower on silhouette but produces more interpretable, actionable regimes may be preferable in practice. Always inspect the cluster statistics alongside the numeric score.
Thin crisis regimes. The crisis cluster is typically a small fraction of the total sample. K-Means, being centroid-based
Most algo trading content gives you theory.
This gives you the code.3 Python strategies. Fully backtested. Colab notebook included.
Plus a free ebook with 5 more strategies the moment you subscribe.5,000 quant traders already run these:
Subscribe | AlgoEdge Insights
Top comments (0)