DEV Community

shakti tiwari
shakti tiwari

Posted on • Originally published at optiontradingwithai.in

LightGBM for Options Trading Signals (Faster Than XGBoost, Same Accuracy)

#machinelearning #lightgbm #trading #python

LightGBM for trading signals

Answer-first: LightGBM is a gradient-boosting library that builds trees leaf-wise instead of level-wise, so it reaches the same accuracy as XGBoost or sklearn HistGradientBoosting in a fraction of the time. For tabular options data (chains, Greeks, volatility surfaces) it is the fastest way to train a signal model on a laptop or phone. This guide shows how it works and gives a runnable training script.

Educational only. Not investment advice. Options can lose their full value. Consult a licensed advisor.

What makes LightGBM different

Standard gradient boosting (including XGBoost's default) grows trees level-wise — every leaf at the current depth splits together. LightGBM grows leaf-wise — it always splits the leaf that reduces loss most. That focuses capacity on the hardest cases and converges faster.

  • Leaf-wise growth: deeper, asymmetric trees; faster convergence.
  • Histogram-based splits: bins features into 256 buckets, scans bins not rows → 10-30x faster than exact search.
  • Exclusive Feature Bundling (EFB): packs sparse features (like OI flags) into one, cutting dims.
  • Gradient-based sampling: trains on a subset of hard examples.

LightGBM vs XGBoost vs HistGradientBoosting

Axis LightGBM XGBoost HistGradientBoosting
Growth leaf-wise level-wise level-wise
Speed fastest fast (GPU) fast (CPU)
Memory lowest medium medium
Tuning more knobs many few (sane defaults)
Phone fit great (pip install) needs wheel stdlib sklearn

For a retail pipeline, LightGBM wins on speed-per-accuracy; HistGradientBoosting wins on zero extra deps; XGBoost wins on GPU sweeps.

Install 

# Mac / Linux / Termux
pip install lightgbm

# Windows CMD
pip install lightgbm
Enter fullscreen mode Exit fullscreen mode

Hyperparameters that matter 

# Mac / Linux / Termux
python3 tune_lgb.py
# Windows CMD
py tune_lgb.py
Enter fullscreen mode Exit fullscreen mode 
import lightgbm as lgb

params = {
    "objective": "binary",
    "metric": "auc",
    "learning_rate": 0.05,
    "num_leaves": 31,        # leaf-wise cap; 2^max_depth-1
    "max_depth": 4,          # still bound depth to avoid overfit
    "min_child_samples": 20, # min rows in a leaf
    "feature_fraction": 0.9, # per-tree feature subsample
    "bagging_fraction": 0.9, # per-tree row subsample
    "lambda_l2": 1.0,        # leaf regularization
    "n_jobs": -1,
}
Enter fullscreen mode Exit fullscreen mode

The two knobs that move the needle: num_leaves (leaf-wise width) and learning_rate. Keep max_depth bounded even though leaf-wise ignores it by default — unbounded depth overfits small options datasets.

Runnable options-signal example 

# Mac / Linux / Termux
python3 train_lgb.py
# Windows CMD
py train_lgb.py
Enter fullscreen mode Exit fullscreen mode 
import lightgbm as lgb
from sklearn.model_selection import TimeSeriesSplit, roc_auc_score
import pandas as pd

FEATURES = ["spread_pct","moneyness","iv_skew","pcr",
            "theta_per_delta","gamma","vega","dte","oi","volume"]

def train_signal(X, y, params):
    tscv = TimeSeriesSplit(n_splits=5)
    for tr, te in tscv.split(X):
        dtr = lgb.Dataset(X.iloc[tr], y.iloc[tr])
        dte = lgb.Dataset(X.iloc[te], y.iloc[te])
        bst = lgb.train(params, dtr, num_boost_round=300,
                        valid_sets=dte, callbacks=[lgb.early_stop(20)])
        p = bst.predict(X.iloc[te])
        print("fold AUC:", round(roc_auc_score(y.iloc[te], p), 3))
    return bst
Enter fullscreen mode Exit fullscreen mode

lgb.early_stop(20) halts when validation AUC stalls — the single best guard against overfit on few hundred expiry-windows.

Early stopping in practice 

bst = lgb.train(params, dtr, num_boost_round=1000,
                valid_sets=dte, callbacks=[lgb.early_stop(20), lgb.log_evaluation(0)])
Enter fullscreen mode Exit fullscreen mode

Never hard-code num_boost_round; let early-stop pick it. On a phone this matters — a 1000-round manual train wastes battery versus a 180-round early-stopped one.

Feature engineering it rewards

Same as other boosters — monotone, separated signals:

  • theta_per_delta — decay cost vs directional exposure.
  • iv_skew — cheapness vs ATM.
  • moneyness — ITM/OTM position.
  • pcr — sentiment extreme.

Drop oi/volume as predictive features — they rank top only when the label leaks.

Interpretability 

importance = pd.Series(bst.feature_importance(), index=FEATURES).sort_values(ascending=False)
print(importance)
Enter fullscreen mode Exit fullscreen mode

If theta_per_delta and iv_skew lead, sane. If oi leads, leakage — fix the label.

The Full Production Pipeline (Data Engine -> Predictor -> Filter)

A published article often shows only the model and backtest. The production system that actually runs has four stages between raw market data and a trade:

1. DATA ENGINE     fetch chain + IV + PCR + vol-index every N seconds
2. FEATURE ENGINE  build_features() -> clean, dedupe, label
3. PREDICTOR       LightGBM model -> prob_up per strike
4. FILTER          Greeks + regime + prob-band rules -> allow/block
5. EXECUTOR        paper or live entry sized by position_size()
Enter fullscreen mode Exit fullscreen mode

1. Data Engine

Connects to the broker/exchange feed and snapshots the full options chain on a timer. Must dedupe cross-venue snapshots, cap latency under the decision window, and survive feed gaps without feeding stale mid quotes to the model.

2. Feature Engine

Runs build_features() on the raw snapshot: mid, spread_pct, moneyness, iv_skew, pcr, theta_per_delta. Bad data dies here — a strike with no OI or a synthetic CFD quote is dropped before the model sees it.

3. Predictor

The trained LightGBM booster outputs prob_up per strike. Stateless at inference — bst.predict(row) many times after one lgb.train.

4. Filter (the part most beginners skip) 

def filter(prob_up, greeks, vol_z, max_capital):
    if not (0.58 <= prob_up <= 0.80):
        return {}
    if greeks["dte"] <= 1:
        return {}
    if abs(greeks["vega"]) > 8.0:
        return {}
    if vol_z > 2.0:
        max_capital *= 0.5
    size = (max_capital * 0.01) / max(greeks["theta"], 1e-9)
    return {"action": "paper_entry", "size": round(size, 2)}
Enter fullscreen mode Exit fullscreen mode

The filter is what makes the system survive a regime the model never saw in training.

5. Executor

Turns the allowed signal into a sized order. Paper first (log every fill), then live only after broker review. Never skip stage 4.

When to pick LightGBM

  • You train daily on a phone/PI — speed matters → LightGBM.
  • You want zero extra deps → HistGradientBoosting.
  • You sweep 50 expiry windows on GPU → XGBoost.
  • Your data is images/text → none of these; use a net.

Categorical features (exchange, regime)

LightGBM handles categorical features natively — no one-hot needed. Encode the exchange or regime as a category and it finds optimal splits:

# Mac / Linux / Termux
python3 cat_lgb.py
# Windows CMD
py cat_lgb.py
Enter fullscreen mode Exit fullscreen mode 
import lightgbm as lgb
params = {"objective":"binary","metric":"auc","learning_rate":0.05,
          "num_leaves":31,"max_depth":4,"categorical_feature":["regime"]}
# regime = "calm" / "stress" / "crash" — LabelEncoder first
Enter fullscreen mode Exit fullscreen mode

This is why LightGBM beats HistGradientBoosting on data with a regime column — it splits on the category directly instead of forcing it through OHE.

GPU training

On a desktop with CUDA, LightGBM trains 5-10x faster:

params_gpu = dict(params, device="cuda")
bst = lgb.train(params_gpu, dtr, num_boost_round=300, valid_sets=dte)
Enter fullscreen mode Exit fullscreen mode

On a phone (Termux) skip GPU — CPU histogram training is already the fastest of the three boosters.

Cost-sensitive boosting

If false entries cost more than misses, weight the positive class:

# LightGBM accepts per-sample weight in the Dataset
dtr = lgb.Dataset(X_tr, y_tr, weight=w_tr)  # w=1.5 for class to protect
Enter fullscreen mode Exit fullscreen mode

This nudges the prob band toward fewer, higher-conviction entries — useful when theta bleed makes false positives expensive.

Monitoring drift in production

A model trained on calm markets fails in a crash. Monitor daily:

  1. Feature drift: if iv_skew distribution shifts > 2 std from training, retrain.
  2. AUC decay: rolling 5-day AUC vs 30-day mean; drop > 3% halts entries. 
def drift_alert(train_mean, live_mean, std, auc_now, auc_base):
    if abs(live_mean - train_mean) > 2*std:
        return "RETRAIN: feature drift"
    if auc_base - auc_now > 0.03:
        return "HALT: AUC decay"
    return "OK"
Enter fullscreen mode Exit fullscreen mode

Stacking LightGBM with a second model

LightGBM is strong alone; a light stack adds robustness by training a logistic regression on the probabilities of two base models:

# Mac / Linux / Termux
python3 stack_lgb.py
# Windows CMD
py stack_lgb.py
Enter fullscreen mode Exit fullscreen mode 
from sklearn.linear_model import LogisticRegression
import numpy as np
# p1 = lgb.predict(X_te); p2 = hgb.predict_proba(X_te)[:,1]
meta = LogisticRegression().fit(np.column_stack([p1, p2]), y_te)
Enter fullscreen mode Exit fullscreen mode

The meta-model's weight drifts with volatility — exactly the regime-switching you want without hand-tuning.

Worked numeric example

Train on 800 expiry-windows. LightGBM with num_leaves=31, learning_rate=0.05, early_stop(20) gives walk-forward AUC 0.62 in ~3 seconds (HistGradientBoosting took ~9s for 0.61). Feature importance: theta_per_delta 0.30, iv_skew 0.25, moneyness 0.17, pcr 0.12. You feed the band 0.58-0.80 to the rules engine. Over 20 paper sessions the realized win-rate inside the band is 58% — honest, and the risk layer is doing its job.

Glossary

  • Leaf-wise: split the single most-losing leaf each round.
  • num_leaves: max leaves per tree (leaf-wise width).
  • Histogram: bin features, scan bins not rows.
  • EFB: bundle sparse features to cut dimensions.
  • Early stop: halt when validation metric stalls.
  • Categorical: native split, no one-hot. ## Common mistakes
  1. Unbounded num_leaves (overfit) — cap at 31-63.
  2. No max_depth bound on leaf-wise growth.
  3. Hard-coding num_boost_round instead of early-stop.
  4. Feeding oi/volume as predictive features.
  5. Random split on time-series data.

Market Microstructure & Liquidity (why it matters for the model)

A signal is only as good as the liquidity it trades into. Three microstructure facts the model must respect:

  1. Bid-ask spread eats thin edges. An ATM option with a 0.3% spread needs the signal to clear more than 0.3% just to break even. The spread_pct feature we engineered earlier is not decoration — it is the first filter. If spread_pct > 0.5%, the predictor's probability is academic; the executor will slip.
  2. Open Interest build-up defines support/resistance. When OI piles at a strike, that strike acts as a magnet or wall at expiry. A model that ignores OI concentration misprices the pinning effect. This is why pcr and per-strike OI slope are features, not afterthoughts.
  3. Volume confirms, OI positions. Rising volume with rising OI = new money committing (trend confirmation). Rising volume with falling OI = squaring (exhaustion). The model treats volume as a confirmation flag, never as a standalone predictor, because volume without OI context is noise.

Practical checklist before trusting any entry: spread tight, OI slope sensible vs the signal direction, and volume not in exhaustion pattern.

Volatility Regime Detection (real code)

Markets are not stationary. A model trained in calm IV behaves badly in a vol spike. Detect regime from the vol index and switch logic:

# Mac / Linux / Termux
python3 regime.py
# Windows CMD
py regime.py
Enter fullscreen mode Exit fullscreen mode 
def regime_state(vix, vix_ma20):
    z = (vix - vix_ma20) / (vix_ma20 + 1e-9)
    if z > 2.0:
        return "CRASH", 0.5      # halve size
    if z > 1.0:
        return "STRESS", 0.75    # shrink size
    if z < -1.0:
        return "CALM", 1.0       # full size
    return "NORMAL", 1.0

def size_with_regime(base_capital, z, max_capital):
    _, mult = regime_state(vix=z, vix_ma20=1.0)
    return (max_capital * 0.01 * mult) / max(base_capital, 1e-9)
Enter fullscreen mode Exit fullscreen mode

The CRASH state cuts size to 50% — this single rule is what keeps a strategy alive across the 2020-style gaps that destroy naive bots. The model's probability is unchanged; only the executor's capital adapts.

Execution & Broker Reality

Backtest assumes fills at mid. Live fills at ask (buy) / bid (sell), plus brokerage and STT. Three realities:

  • Brokerage + taxes: per-lot flat fee plus exchange charges. A round-trip on a cheap option can cost 0.5-1% — model this as fees_bps in backtest, not zero.
  • Slippage: in fast markets the quoted mid moves between signal and fill. Cap position size so slippage stays under the edge.
  • Margin: short options need margin blocks; long options need premium. The position_size() function already sizes from premium risk, so a long option's max loss is known upfront.

Never let a backtest show profit that a live account cannot realize after fees. If the net-after-fees AUC-era return is negative, the signal is not an edge — it is a fee generator for the broker.

A Realistic Weekly Routine

Consistency beats bursts. A workable week for this system:

  • Monday: pull last week's chain CSV, retrain if drift alert fired, review regime state.
  • Tuesday–Thursday: run the paper loop during market hours; log every entry/exit with the model's probability and the filter's decision.
  • Friday: if expiry week, tighten DTE limits; review realized vs predicted.
  • Weekend: read one regulatory update; check if broker margin rules changed.

This is not a get-rich loop. It is a measurement loop. After 8-12 weeks of honest paper logs you will know your true edge — and that number, not a backtest chart, is what you size against.

FAQ

Q1. LightGBM or XGBoost for a phone pipeline?
LightGBM — fastest per accuracy, installs via pip, lowest memory.

Q2. Why leaf-wise, not level-wise?
Leaf-wise splits the most-losing leaf each round, converging faster; level-wise splits all leaves uniformly and wastes capacity.

Q3. num_leaves too high?
Overfits. Cap at 31 (depth 4) for small options datasets; 63 max for 1000+ rows.

Q4. How many samples do I need?
A few hundred labeled expiry-windows is enough; 1000+ gives stable AUC. Below that, shrink num_leaves to 15.

Q5. Biggest edge — model or risk layer?
The risk layer. A mediocre model with strict Greek limits survives; a great model without them does not.

Footer

Shakti Tiwari — Options Trader, XGBoost Expert.
Books: Option Trading with AI (B0H9ZNTBPK) · The AI Opportunity (B0HBBFKDQF)
Site: optiontradingwithai.in · Free help: shaktitiwari715@gmail.com
Dev.to: @shaktitiwari · X: @shaktitiwari

Top comments (0)