Portfolio Optimization in Python: From Markowitz to Machine Learning

Every quant faces the same question: how do you allocate capital across assets to maximize returns while minimizing risk? This is portfolio optimization — and it's the backbone of modern quantitative finance.

In this post, we'll build portfolio optimization from scratch in Python, starting with Markowitz's classical mean-variance framework and progressing to machine learning approaches used on actual trading desks.

The Markowitz Framework

Harry Markowitz's 1952 paper revolutionized finance by formalizing diversification. The core idea: don't just pick individual stocks — optimize the entire portfolio.

The Math Behind It

Given a portfolio of $n$ assets with:

• Expected return vector: $\mu$
• Covariance matrix: $\Sigma$
• Weight vector: $w$

The portfolio return and risk are:

$$R_p = w^T \mu$$

$$\sigma_p^2 = w^T \Sigma w$$

The optimization problem becomes:

$$\min_w \quad w^T \Sigma w \quad \text{subject to} \quad w^T \mu = \mu_{target}, \quad w^T \mathbf{1} = 1$$

Python Implementation

import numpy as np
import pandas as pd
from scipy.optimize import minimize

def portfolio_stats(weights, mean_returns, cov_matrix):
    """Calculate portfolio return and volatility."""
    portfolio_return = np.sum(mean_returns * weights)
    portfolio_volatility = np.sqrt(np.dot(weights.T, np.dot(cov_matrix, weights)))
    return portfolio_return, portfolio_volatility

def minimize_volatility(target_return, mean_returns, cov_matrix, n_assets):
    """Find minimum volatility portfolio for a target return."""
    args = (mean_returns, cov_matrix)

    constraints = (
        {'type': 'eq', 'fun': lambda x: np.sum(x) - 1},
        {'type': 'eq', 'fun': lambda x: portfolio_stats(x, mean_returns, cov_matrix)[0] - target_return}
    )

    bounds = tuple((0, 1) for _ in range(n_assets))
    result = minimize(
        fun=lambda x: portfolio_stats(x, mean_returns, cov_matrix)[1],
        x0=np.array([1/n_assets] * n_assets),
        method='SLSQP',
        bounds=bounds,
        constraints=constraints
    )
    return result

# Generate sample data
np.random.seed(42)
n_assets = 5
tickers = ['AAPL', 'MSFT', 'GOOGL', 'AMZN', 'TSLA']

# Simulate returns
mean_returns = np.array([0.12, 0.10, 0.15, 0.11, 0.20])
cov_matrix = np.array([
    [0.04, 0.006, 0.008, 0.005, 0.01],
    [0.006, 0.03, 0.007, 0.004, 0.008],
    [0.008, 0.007, 0.06, 0.009, 0.015],
    [0.005, 0.004, 0.009, 0.035, 0.007],
    [0.01, 0.008, 0.015, 0.007, 0.10]
])

# Find efficient frontier
target_returns = np.linspace(0.08, 0.20, 50)
frontier_volatilities = []

for target in target_returns:
    result = minimize_volatility(target, mean_returns, cov_matrix, n_assets)
    frontier_volatilities.append(result.fun)

print("Efficient frontier computed!")
print(f"Minimum volatility: {min(frontier_volatilities):.4f}")

The Efficient Frontier

The efficient frontier is the set of portfolios offering the highest expected return for each level of risk. Any portfolio below the frontier is suboptimal — you could get more return for the same risk.

Portfolio	Return	Volatility	Sharpe Ratio
Min Variance	10.2%	14.8%	0.42
Max Sharpe	14.5%	18.2%	0.63
Equal Weight	13.6%	19.1%	0.50

Beyond Markowitz: Black-Litterman

Classical Markowitz has a problem: it's extremely sensitive to input estimates. Small changes in expected returns lead to wildly different allocations. The Black-Litterman model fixes this by combining market equilibrium with investor views.

The Key Insight

Instead of estimating returns directly, Black-Litterman starts with implied equilibrium returns derived from market capitalization weights:

$$\Pi = \delta \Sigma w_{mkt}$$

where $\delta$ is the risk aversion coefficient and $w_{mkt}$ is the market portfolio.

Python Implementation

def black_litterman(cov_matrix, market_weights, risk_aversion, views, view_confidences):
    """
    Black-Litterman model implementation.

    Parameters:
    - cov_matrix: covariance matrix
    - market_weights: market cap weights
    - risk_aversion: risk aversion coefficient
    - views: matrix of investor views (k x n)
    - view_confidences: confidence in each view (k x k)
    """
    n = len(market_weights)

    # Implied equilibrium returns
    pi = risk_aversion * cov_matrix @ market_views

    # View uncertainty (Omega)
    tau = 0.05
    sigma_views = view_confidences

    # Black-Litterman expected returns
    P = views
    Q = np.array([0.10, 0.15])  # View returns

    M1 = np.linalg.inv(tau * cov_matrix)
    M2 = P.T @ np.linalg.inv(sigma_views) @ P
    bl_returns = np.linalg.inv(M1 + M2) @ (M1 @ pi + P.T @ np.linalg.inv(sigma_views) @ Q)

    return bl_returns

Machine Learning Approaches

Modern quant desks use ML for portfolio optimization in several ways:

1. Reinforcement Learning for Dynamic Allocation

import torch
import torch.nn as nn

class PortfolioAllocator(nn.Module):
    def __init__(self, n_assets, lookback=60):
        super().__init__()
        self.lstm = nn.LSTM(n_assets, 64, batch_first=True)
        self.fc = nn.Linear(64, n_assets)
        self.softmax = nn.Softmax(dim=-1)

    def forward(self, returns_history):
        # returns_history: (batch, lookback, n_assets)
        lstm_out, _ = self.lstm(returns_history)
        weights = self.fc(lstm_out[:, -1, :])
        return self.softmax(weights)

2. Variational Autoencoders for Scenario Generation

Instead of assuming normal returns, VAEs learn the actual distribution of market scenarios, capturing fat tails and regime changes.

3. Graph Neural Networks for Sector Allocation

GNNs model relationships between assets, capturing contagion effects and sector rotations that traditional correlation matrices miss.

Practical Considerations

Transaction Costs

Real portfolios face trading costs. The optimization must account for:

def portfolio_with_transaction_costs(weights, new_weights, transaction_cost=0.001):
    """Add transaction cost penalty."""
    turnover = np.sum(np.abs(new_weights - weights))
    cost = turnover * transaction_cost
    return cost

Rebalancing Frequency

Frequency	Pros	Cons
Daily	Captures signals	High transaction costs
Weekly	Balance	Moderate costs
Monthly	Low costs	May miss opportunities

Risk Constraints

# Maximum position size
constraints.append({'type': 'ineq', 'fun': lambda x: 0.20 - x})

# Maximum sector exposure
sector_exposure = np.sum(weights[sector_indices])
constraints.append({'type': 'ineq', 'fun': lambda x: 0.40 - sector_exposure})

Real-World Application

Here's a complete pipeline you can use:

import yfinance as yf
from sklearn.covariance import LedoitWolf

# Fetch data
data = yf.download(['AAPL', 'MSFT', 'GOOGL', 'AMZN', 'TSLA'], 
                    start='2020-01-01', end='2024-01-01')
returns = data['Adj Close'].pct_change().dropna()

# Shrinkage estimator for covariance
lw = LedoitWolf().fit(returns)
cov_matrix = lw.covariance_
mean_returns = returns.mean() * 252  # Annualized

# Optimize
result = minimize_volatility(0.12, mean_returns, cov_matrix, 5)
print(f"Optimal weights: {dict(zip(tickers, result.x.round(3)))}")

Key Takeaways

1. Markowitz is the foundation — understand mean-variance optimization before moving to advanced methods
2. Black-Litterman fixes sensitivity — use market equilibrium as a prior
3. ML adds value — but only when you have enough data and proper validation
4. Transaction costs matter — theoretical optimal ≠ practical optimal
5. Risk constraints are essential — never optimize without limits

Want to Learn More?

If you're preparing for quant interviews or want to deepen your portfolio theory knowledge, check out our Quantitative Finance for Absolute Beginners course — it covers portfolio optimization, risk management, and derivatives pricing with hands-on Python projects.

---

Built with Python, NumPy, SciPy, and PyTorch. Full code available on GitHub.