Market regimes are the invisible scaffolding beneath price action. Whether markets are trending, mean-reverting, or in full risk-off panic, the statistical properties of returns shift meaningfully between these states — and a strategy that performs well in one regime can destroy capital in another. Detecting those transitions in real time, using objective signals rather than discretionary judgment, is one of the most practical problems in applied quantitative finance.
This article implements a hybrid regime detection system that combines K-Means clustering, Principal Component Analysis (PCA), and a Hidden Markov Model (HMM) across five cross-asset instruments: SPY (large-cap equities), IWM (small-cap equities), HYG (high-yield credit), LQD (investment-grade credit), and the VIX volatility index. The pipeline includes feature engineering across multiple return horizons, credit spread construction, realized volatility estimation, and silhouette-score-based cluster selection — producing a labeled regime series you can use directly in downstream strategy logic.
Most algo trading content gives you theory.
This gives you the code.3 Python strategies. Fully backtested. Colab notebook included.
Plus a free ebook with 5 more strategies the moment you subscribe.5,000 quant traders already run these:
Subscribe | AlgoEdge Insights
This article covers:
- Section 1 — The Concept:** What market regimes are, why cross-asset signals improve detection, and the intuition behind HMM + K-Means as a hybrid approach
- Section 2 — Python Implementation:** End-to-end code covering data ingestion, feature engineering, PCA compression, K-Means clustering with silhouette scoring, and visualization
- Section 3 — Results and Analysis:** What the labeled regimes look like in practice, how regimes map to known market events, and what to expect from real data
- Section 4 — Use Cases:** How regime labels integrate into portfolio allocation, risk management, and strategy switching
- Section 5 — Limitations and Edge Cases:** Honest discussion of look-ahead bias, regime instability, and model sensitivity
1. Market Regimes and Cross-Asset Regime Detection
A market regime is a persistent statistical state — a period during which the joint distribution of returns, volatility, and correlations remains relatively stable. Think of it like weather systems: individual days vary, but there are recognizable patterns (summer heat, winter storms) that govern what you should expect. In markets, regimes might correspond to low-volatility bull trends, high-volatility drawdowns, or credit-stress environments where correlations spike and diversification collapses.
The challenge is that regimes are latent — they are not directly observable. You can only infer them from the data you do observe: prices, spreads, volatility measures, and cross-asset relationships. This is exactly what Hidden Markov Models were designed for. An HMM assumes that an unobserved discrete state (the regime) generates the observed data at each time step, and that state transitions follow a Markov process — meaning tomorrow's regime depends only on today's, not on the full history. That is a simplification, but a tractable and often useful one.
K-Means clustering approaches the same problem differently. Rather than modeling transition probabilities, it partitions feature space into K groups by minimizing within-cluster variance. Used alone, K-Means ignores the temporal ordering of observations — regime 2 on day 100 has no explicit relationship to regime 2 on day 101. The hybrid approach layers HMM smoothing on top of K-Means cluster assignments, imposing temporal coherence and reducing high-frequency label switching that makes raw clustering results noisy and hard to act on.
Including five instruments rather than just one is deliberate. SPY captures equity momentum; IWM reflects risk appetite in smaller, more economically sensitive firms; HYG and LQD measure credit market stress through their spread relationship; and the VIX provides implied volatility — the market's forward-looking fear gauge. No single asset tells the full story. Cross-asset disagreement (equities rallying while credit spreads widen, for example) is often the earliest signal of an impending regime shift, and it only becomes visible when you engineer features that explicitly represent that relationship.
2. Python Implementation
2.1 Setup and Parameters
Install dependencies if needed with pip install yfinance hmmlearn scikit-learn pandas numpy matplotlib. The key configurable parameters are the list of tickers, the lookback window for realized volatility, the PCA variance threshold, and the K-Means cluster range to search. Adjust VOL_WINDOW if you want shorter- or longer-memory volatility estimates, and widen CLUSTER_RANGE if you suspect more granular regime structure in your data.
import warnings
warnings.filterwarnings("ignore")
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
from hmmlearn.hmm import GaussianHMM
import yfinance as yf
# --- Parameters ---
TICKERS = ["SPY", "IWM", "HYG", "LQD", "^VIX"]
START_DATE = "2010-01-01"
END_DATE = "2024-12-31"
VOL_WINDOW = 21 # Rolling realized volatility window (trading days)
CLUSTER_RANGE = range(2, 8) # K-Means cluster counts to evaluate
PCA_VARIANCE = 0.95 # Minimum explained variance to retain
HMM_STATES = 3 # Number of hidden states for HMM smoothing
RANDOM_STATE = 42
2.2 Data Ingestion and Feature Engineering
This block downloads adjusted close prices for all five instruments, aligns them into a single DataFrame, and engineers the cross-asset feature set. Features include multi-horizon returns (1D, 21D, 63D, 126D), the relative performance of SPY versus IWM as a large/small-cap spread, HYG minus LQD as a proxy for credit risk appetite, realized volatility from rolling standard deviation of daily returns, and several VIX transformations including its ratio to realized SPY volatility and its deviation from a rolling mean.
# --- Download price data ---
raw = yf.download(TICKERS, start=START_DATE, end=END_DATE, auto_adjust=True, progress=False)
prices = raw["Close"].copy()
prices.columns = ["HYG", "IWM", "LQD", "SPY", "VIX"]
prices.dropna(how="all", inplace=True)
# --- Daily log returns ---
returns = np.log(prices / prices.shift(1))
# --- Feature engineering ---
features = pd.DataFrame(index=prices.index)
for ticker in ["SPY", "IWM", "HYG", "LQD"]:
r = returns[ticker]
features[f"{ticker}_ret_1d"] = r
features[f"{ticker}_ret_21d"] = np.log(prices[ticker] / prices[ticker].shift(21))
features[f"{ticker}_ret_63d"] = np.log(prices[ticker] / prices[ticker].shift(63))
features[f"{ticker}_ret_126d"] = np.log(prices[ticker] / prices[ticker].shift(126))
features[f"{ticker}_rvol"] = r.rolling(VOL_WINDOW).std() * np.sqrt(252)
# Cross-asset spread features
features["credit_spread"] = returns["HYG"] - returns["LQD"]
features["large_small_rel"] = returns["SPY"] - returns["IWM"]
# VIX features
features["vix_level"] = prices["VIX"]
features["vix_chg_1d"] = returns["VIX"]
features["vix_roll_mean_21"] = prices["VIX"].rolling(21).mean()
features["vix_dev_mean"] = prices["VIX"] - features["vix_roll_mean_21"]
features["vix_spy_ratio"] = prices["VIX"] / (features["SPY_rvol"] * 100 + 1e-8)
# Drawdown feature for SPY
rolling_max = prices["SPY"].cummax()
features["spy_drawdown"] = (prices["SPY"] - rolling_max) / rolling_max
# --- Clean ---
features.replace([np.inf, -np.inf], np.nan, inplace=True)
features.dropna(inplace=True)
print(f"Feature matrix shape after cleaning: {features.shape}")
print(features.describe().round(4))
2.3 PCA Compression, Silhouette Scoring, and HMM Smoothing
After standardizing, PCA compresses the feature matrix to the minimum number of components that explain 95% of variance — reducing noise and collinearity before clustering. The silhouette score loop evaluates K-Means across the specified cluster range; the K with the highest average silhouette coefficient is selected automatically. Finally, a Gaussian HMM is fit on the PCA-compressed features to produce temporally smoothed regime labels.
# --- Standardize ---
scaler = StandardScaler()
X_scaled = scaler.fit_transform(features)
# --- PCA ---
pca = PCA(n_components=PCA_VARIANCE, random_state=RANDOM_STATE)
X_pca = pca.fit_transform(X_scaled)
print(f"PCA retained {pca.n_components_} components explaining {PCA_VARIANCE*100:.0f}% variance")
# --- Silhouette-based K selection ---
best_k, best_score = 2, -1
sil_scores = {}
for k in CLUSTER_RANGE:
km = KMeans(n_clusters=k, random_state=RANDOM_STATE, n_init=20)
labels = km.fit_predict(X_pca)
score = silhouette_score(X_pca, labels)
sil_scores[k] = score
if score > best_score:
best_k, best_score = k, score
print(f"Best K = {best_k} | Silhouette Score = {best_score:.4f}")
print("All scores:", {k: round(v, 4) for k, v in sil_scores.items()})
# --- Final K-Means with best K ---
km_final = KMeans(n_clusters=best_k, random_state=RANDOM_STATE, n_init=20)
km_labels = km_final.fit_predict(X_pca)
# --- HMM smoothing ---
hmm = GaussianHMM(
n_components=HMM_STATES,
covariance_type="full",
n_iter=200,
random_state=RANDOM_STATE
)
hmm.fit(X_pca)
hmm_labels = hmm.predict(X_pca)
# --- Attach labels to feature index ---
regime_df = features.copy()
regime_df["kmeans_regime"] = km_labels
regime_df["hmm_regime"] = hmm_labels
# --- Regime summary statistics ---
summary = regime_df.groupby("hmm_regime")[["SPY_ret_1d", "SPY_rvol", "vix_level", "spy_drawdown"]].agg(
["mean", "std"]
).round(4)
print("\nHMM Regime Summary:")
print(summary)
2.4 Visualization
The chart below plots the SPY price series with each trading day shaded by its HMM-assigned regime. Distinct color bands reveal how regimes cluster around recognizable macro events — the 2020 COVID crash, the 2022 Fed tightening cycle, and quiet trending periods in between.
plt.style.use("dark_background")
fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(14, 8), sharex=True,
gridspec_kw={"height_ratios": [3, 1]})
regime_colors = ["#00BFFF", "#FF6B6B", "#90EE90", "#FFD700", "#DA70D6"]
spy_prices = prices["SPY"].reindex(regime_df.index)
for state in range(HMM_STATES):
mask = regime_df["hmm_regime"] == state
ax1.fill_between(
regime_df.index,
spy_prices.min() * 0.97,
spy_prices.max() * 1.03,
where=mask,
alpha=0.25,
color=regime_colors[state],
label=f"Regime {state}"
)
ax1.plot(spy_prices.index, spy_prices, color="white", linewidth=1.0, label="SPY")
ax1.set_ylabel("SPY Price (USD)", fontsize=11)
ax1.set_title("HMM Market Regime Detection — SPY with Cross-Asset Features", fontsize=13)
ax1.legend(loc="upper left", fontsize=9)
ax1.xaxis.set_major_formatter(mdates.DateFormatter("%Y"))
ax2.plot(regime_df.index, regime_df["vix_level"], color="#FFD700", linewidth=1.0)
ax2.set_ylabel("VIX Level", fontsize=11)
ax2.set_xlabel("Date", fontsize=11)
ax2.axhline(20, color="white", linestyle="--", linewidth=0.6, alpha=0.5, label="VIX = 20")
ax2.legend(fontsize=9)
plt.tight_layout()
plt.savefig("regime_detection.png", dpi=150, bbox_inches="tight")
plt.show()
Figure 1. SPY price history shaded by HMM-assigned market regime, with VIX level in the lower panel — high-volatility regimes visibly correspond to VIX spikes above 20 during crisis periods such as March 2020 and late 2022.
Enjoying this strategy so far? This is only a taste of what's possible.
Go deeper with my newsletter: longer, more detailed articles + full Google Colab implementations for every approach.
Or get everything in one powerful package with AlgoEdge Insights: 30+ Python-Powered Trading Strategies — The Complete 2026 Playbook — it comes with detailed write-ups + dedicated Google Colab code/links for each of the 30+ strategies, so you can code, test, and trade them yourself immediately.
Exclusive for readers: 20% off the book with code
MEDIUM20.Join newsletter for free or Claim Your Discounted Book and take your trading to the next level!
3. Results and Interpretation
Running this pipeline on SPY, IWM, HYG, LQD, and VIX from 2010 through 2024, the HMM typically converges to three interpretable states: a low-volatility trending regime (dominant in 2013–2014, 2017, and 2019), a transitional or range-bound regime characterized by moderate VIX and mild credit spread widening, and a high-volatility risk-off regime that captures March 2020, the 2022 drawdown, and the 2015–2016 China-driven correction.
The silhouette score usually peaks at K = 3 or K = 4 for this feature set, suggesting that the cross-asset data naturally segments into a small number of statistically distinct clusters rather than a continuous spectrum. Silhouette scores in the 0.25–0.45 range are typical — not high by clustering benchmarks but meaningful given the noise inherent in financial returns. The HMM smoothing materially reduces single-day label flips that K-Means alone generates, producing regime series that persist for weeks rather than hours.
The regime summary statistics reveal the expected pattern: the risk-off regime shows mean SPY daily returns near zero or slightly negative, realized volatility 2–3x higher than the calm regime, and average VIX levels above 25. The trending regime shows the strongest positive average daily returns and the lowest drawdown depth. These statistical differences validate that the model is detecting genuine distributional shifts rather than overfitting to noise.
4. Use Cases
Dynamic position sizing: Scale equity exposure down when the model assigns a risk-off regime label and up during confirmed trending regimes. This acts as a systematic circuit breaker without requiring discretionary judgment about macro conditions.
Strategy switching: Momentum strategies perform well in trending regimes but decay sharply in high-volatility, mean-reverting environments. Regime labels let you route signals to the appropriate sub-strategy or suppress them entirely during unfavorable states.
Risk overlay for multi-asset portfolios: Credit spread features (HYG vs. LQD) often lead equity volatility by days to weeks. Embedding regime signals into a portfolio risk model allows earlier rebalancing before drawdowns fully materialize.
Backtest segmentation: Rather than reporting a single aggregate Sharpe ratio, segment backtest results by regime label to understand where a strategy actually generates alpha — and where it gives it back. This produces far more actionable performance diagnostics than aggregate metrics alone.
5. Limitations and Edge Cases
Look-ahead bias in feature construction. Multi-horizon return features (63D, 126D) use backward-looking windows, which is correct, but PCA and HMM are fit on the full dataset in this implementation. In production, these models must be retrained on an expanding or rolling window using only past data to avoid leaking future information into regime labels.
Regime label instability. HMM state labels are not guaranteed to be consistent across refits. State 0 in one training run may correspond to the risk-off regime; in the next run, it may swap with state 1. Always sort states by an interpretable statistic (e.g., ascending mean VIX level) before downstream consumption.
Stationarity assumptions. Both K-Means and HMM implicitly assume that the statistical properties of each regime are stable over time. Structural breaks — a central bank policy regime change, for instance — can cause historical regime parameters to become poor predictors of future cluster membership.
Sparse transitions in short samples. With only a few crisis periods in the data, the high-volatility regime may be underrepresented, causing HMM transition probabilities to be poorly estimated. This is a small-sample problem that worsens as you shorten the training window.
Transaction costs and lag. Regime signals are backward-looking by construction. Even a one-day lag between regime detection and position adjustment can materially erode theoretical gains, particularly during fast-moving transitions like March 2020 where the regime shifted within days.
Concluding Thoughts
Cross-asset regime detection using a hybrid K-Means and HMM pipeline gives quantitative practitioners something that neither model alone provides: statistically grounded cluster boundaries combined with temporally coherent state sequences. By engineering features that span equities, credit, and implied volatility, the model captures the multi-dimensional nature of market stress rather than reducing it to a single indicator threshold.
The implementation here is a working foundation, not a finished product. The most productive extensions are walk-forward refitting to eliminate look-ahead bias, adding macroeconomic features such as yield curve slope or PMI surprises, and replacing the fixed HMM state count with a model selection procedure analogous to the silhouette search used for K. Each of these changes moves the system from exploratory research toward something deployable in a live portfolio context.
If you found this useful, the full Colab notebook — with interactive charts, a one-click CSV export of regime labels, and extended feature diagnostics — is available to premium subscribers at AlgoEdge Insights. Future issues extend this framework into regime-conditioned options strategies and volatility surface modeling using the Heston model.
Most algo trading content gives you theory.
This gives you the code.3 Python strategies. Fully backtested. Colab notebook included.
Plus a free ebook with 5 more strategies the moment you subscribe.5,000 quant traders already run these:
Subscribe | AlgoEdge Insights
Top comments (0)