Huber regression had the strongest predictive ranking in my crypto model comparison: mean out-of-sample Spearman IC +0.055. Its selected trades nevertheless averaged −7.74 basis points after modeled costs.
The tree model with the least negative result was an XGBoost configuration at −1.73 bps. Its uncertainty interval crossed zero. LightGBM was close behind at −1.93 bps. Neither result establishes a profitable strategy.
Those numbers came only after an uncomfortable discovery: the first version of the backtest had a boundary leak and treated gaps in price history as consecutive bars. We fixed both and reran the comparison.
Research status: this is exploratory, retrospective research on history already examined in earlier work. Prices are OHLC proxies, costs are assumed, the universe was selected retrospectively, and there is no executable portfolio P&L. It is not an independent holdout or a live-trading result.
Signal is not a strategy
The question began with three features: 1-, 5-, and 20-bar log returns on 15-minute Bybit perpetual candles. Could a model use them to rank the next return? Could a trading rule survive costs?
An information coefficient, or IC, describes rank correlation between a score and the future return. To assess a trading rule, we also need to know how often it trades, what each trade earns and what execution costs.
Here IC is one Spearman correlation pooled across available symbol-bar observations per fold, then averaged equally across folds. It is not a daily cross-sectional IC; pooled correlations can reflect differences between symbols as well as within-symbol ranking. We did not estimate an IC confidence interval in this run.
Why can the rankings differ? IC uses all scored observations to ask about ordering. The trading result uses only observations beyond a model-specific threshold, assigns a direction and subtracts costs. A model can rank the whole panel well yet select a less rewarding set of extreme scores. The two statistics also weight observations differently, so this experiment does not isolate a single cause of the gap.
An earlier OLS study reported 4,218 trades and +5.45 bps mean net using a fixed 12 bps round-trip assumption and a fixed ±12 bps signal threshold. Its bootstrap implementation also had a bug: repeated sampled days were discarded. The reported interval could not be relied on. A positive number did not answer whether another model, evaluated under the same rules, would do better.
Seven candidates, one historical protocol
The original comparison used reversal, OLS, Ridge, Huber and constrained LightGBM. I later added XGBoost and CatBoost as a post-hoc extension. The seven configurations share the same time splits, universe, features available to learned models, signal-selection rule and modeled costs. The reversal baseline uses only the one-bar feature.
- Data: 16 Bybit USDT perpetuals; 15-minute bars; symbols enter as data becomes available. Selecting this universe retrospectively creates survivorship risk.
- Evaluation: six expanding walk-forward folds, 2022 H1 through 2024 H2; training starts in 2021.
- Target: next-bar close-to-close return in bps. The model and scaler are fitted on training data only.
- Signal: a long or short when the score exceeds the 99th percentile of its own training score magnitude in either direction.
- Economics: next-bar open-to-close return, less modeled round-trip costs of 13, 15 or 19 bps by liquidity tier.
The cost assumption is 5.5 bps taker fee per side plus 1, 2 or 4 bps assumed slippage per side. These are inputs to the historical proxy, not measured historical fees or fills.
The threshold helps make differently scaled scores comparable, but it does not force models to trade equally often in the evaluation period. Models may select different symbols and dates, so their mean per-trade returns are not returns on an equally allocated portfolio.
The tree configurations are fixed and untuned here; they use different algorithms and regularization. This is a comparison of seven configurations, not a universal ranking of model families.
The later boosters each use 200 trees, depth 4 and learning rate 0.03. XGBoost uses min_child_weight=2000, L1=1 and L2=10; CatBoost uses L2=10. Ridge uses a strong fixed alpha=1,000,000 on standardized features, so its effective regularization changes relative to the growing training sample. A different tuning budget could change the ranking.
Five equations and one timing problem
For bar close C[t] and open O[t]:
features: x[t, k] = log(C[t] / C[t-k]), k ∈ {1, 5, 20}
train label: y[t] = 10,000 × (C[t+1] / C[t] − 1)
threshold: q = 99th percentile of |training score|
signal: s[t] = +1 if score > q; −1 if score < −q; else 0
trade net: p[t] = s[t] × 10,000 × (C[t+1] / O[t+1] − 1) − cost
The last equation applies to selected trades only. A basis point is 0.01% of position notional.
The model learns close-to-close returns, while the economic proxy measures the next candle's open-to-close move. The next open can occur before a system reacts to a confirmed close, so the OHLC price does not establish that a trade could have filled there.
The timing diagram is illustrative, not a measured latency distribution. A later bid/ask observation does not establish that an order filled.
The corrected results
| Model | Mean fold IC | Trade-weighted net / trade | 95% block interval, bps | Trades |
|---|---|---|---|---|
| XGBoost* | +0.012 | −1.73 bps | [−7.41, +3.63] | 10,885 |
| LightGBM | +0.012 | −1.93 bps | [−7.64, +3.65] | 11,006 |
| CatBoost* | +0.011 | −5.73 bps | [−11.54, +0.83] | 11,341 |
| Ridge | +0.044 | −7.14 bps | [−14.34, −0.76] | 12,685 |
| Huber | +0.055 | −7.74 bps | [−15.34, −0.88] | 12,487 |
| OLS | +0.036 | −9.82 bps | [−16.00, −4.42] | 12,461 |
| One-bar reversal | +0.040 | −12.55 bps | [−18.39, −6.48] | 11,497 |
XGBoost and CatBoost were added after the first run. “Out-of-sample” means out of each fold's fitting window, not a clean final holdout. IC values are descriptive means, not claims of statistical significance.
Net is weighted by the number of trades. Averaging the six fold-level means instead gives LightGBM −3.09 bps: another valid summary, but a different weighting. Neither measure is portfolio return.
Intervals use 2,000 resamples of 10-calendar-day blocks. All trades within a sampled day stay together; the same days are sampled for each model. Fitted models and selected trades stay fixed during resampling; the models are not refitted. The intervals do not account for model selection or multiple comparisons. The XGBoost and LightGBM intervals crossing zero neither prove profit nor rule it out. The 0.20 bps difference between their point estimates is not a basis for declaring a winner.
Look inside the average
LightGBM's fold means range from −12.49 to +4.29 bps. Its three positive half-years are small beside its two largest negative ones. All seven configurations lose in 2022 H2 under these assumptions. This is regime sensitivity, not a clean prediction of what happens next.
The day-block bootstrap gives an exploratory LightGBM-minus-OLS interval of [+3.71, +12.90] bps per trade. The models select different trades, the research is post-selection, and this interval is not adjusted for testing multiple candidates. It is not proof that LightGBM generally beats OLS.
What happens when positions share capital?
The table weights each trade equally. A model that selects many simultaneous markets may look different when those positions have to share capital. In a separate post-hoc portfolio proxy, we divide 1× gross notional equally among selected symbols on each active bar. The proxy enters at the next open, exits at the following close and pays the same modeled round-trip costs. It holds cash when there is no valid selected signal.
With the original in-sample threshold, the mean daily additive proxy is −53.68 bps for LightGBM and −54.62 bps for XGBoost. LightGBM has the less negative number under this aggregation, reversing their per-trade ordering. The paired 10-day block-bootstrap interval for XGBoost minus LightGBM is [−14.95, +11.59] bps per day: it includes zero.
Does the threshold matter?
The original 99th percentile comes from predictions on the same observations used to fit each model. For a sensitivity check, we fit another model on the first 80% of each outer training window, purge labels crossing the boundary, and calculate the percentile on the remaining 20%. We then refit on the full outer training window before evaluating the next fold. Refitting can change score scale; this is one post-hoc chronological split, not a tuned or independently validated rule.
With the training-holdout threshold, the mean daily additive proxy changes to −80.98 bps for LightGBM and −79.67 bps for XGBoost. Their ordering switches again. The paired interval for XGBoost minus LightGBM is [−12.15, +13.46] bps per day, again including zero. Both intervals use 2,000 resamples and common 10-day blocks. Models are not refitted, the historical period was already examined, and the intervals are not adjusted for multiple comparisons.
These numbers sum bar-level bps by next-bar entry day and divide by all calendar days, including days in cash. Notional resets to 1× whenever a signal is active; returns are not compounded. The proxy omits funding, impact, order-book depth, margin, liquidation and changing equity. Missing or invalid symbol-bars are treated as unavailable signals, and equal gross notional is not equal risk. This is not an executable portfolio backtest.
The cost cliff
For the original five candidates, I held signals fixed and scaled the assumed fee plus slippage:
LightGBM's selected trades averaged +16.15 bps gross proxy and 18.08 bps modeled cost. At half the assumed cost, net is +7.11 bps; at baseline, −1.93; at twice the cost, −20.01. The break-even multiplier is about 16.15 / 18.08 = 0.89.
That number is an arithmetic result for this historical trade set; it does not establish an obtainable fee quote. The sensitivity scales both fees and slippage, keeps the trades fixed, and does not model queueing, market impact or latency. XGBoost and CatBoost are not included in this cost chart.
To put the units in context: on a hypothetical $10,000 position, one basis point is $1. The LightGBM averages correspond to roughly $16.15 gross proxy, $18.08 assumed costs and −$1.93 net per selected trade. These are scaled averages, not cash flows from a funded account; actual position sizes, simultaneous trades and fills are not modeled in that calculation.
With 5- and 20-day bootstrap blocks, LightGBM's intervals still cross zero: approximately [−7.86, +3.83] and [−7.88, +3.80] bps. Other dependence assumptions could produce different intervals.
Two bugs that changed the experiment
The train/test boundary. The first script selected training rows by timestamp but constructed their targets using the next row. A final training row could therefore use a price from the evaluation period. The corrected split requires each training label to finish by the evaluation boundary. A regression test changes future prices and verifies that training data does not change.
# Bar timestamps denote bar opens; label_end is the next bar's close.
train = panel[(panel.ts >= train_start) & (panel.label_end <= eval_start)]
eval_ = panel[(panel.ts >= eval_start) & (panel.label_end <= eval_end)]
Compare the end of the label with the evaluation boundary, not just the timestamp on the feature row. This rule matches this next-bar target; longer holding periods or overlapping labels need their own purge.
Missing bars. A row shift is not necessarily a 15-minute shift. The corrected run requires 20 consecutive historical intervals and one consecutive future interval. Across loaded history it detected 1,631 gaps and excluded 34,587 rows, including warm-up and incomplete outcomes. The downloadable result lists exclusions and full-file hashes by symbol.
The old +5.45 bps OLS estimate and this rerun changed both the threshold and cost convention. We cannot attribute their difference to costs alone. We can say that the earlier positive result did not carry over to the new decision and cost contract.
What would change my mind?
An executable-price simulation with point-in-time universe rules, realistic order sizing and a portfolio risk model could change this assessment. So could a genuinely new period.
If you are building a similar comparison, keep three questions separate:
- Can the score rank returns? Compare models on the same untouched time windows, and state whether the IC is pooled or cross-sectional.
- Does the decision rule survive costs? Freeze the threshold, timing, sizing and fee assumptions; report gross and net results together.
- Would the portfolio have been executable? Check simultaneous exposure, missing data, available liquidity and actual decision-to-order timing. An OHLC open is not a fill.
This experiment answers the first two only under its stated historical assumptions. It does not establish the third.
We have frozen seven fitted candidates for a prospective shadow period from 1 November 2026 to 1 February 2027 UTC, with LightGBM versus OLS as the primary comparison. As of 30 October 2026, that period had not started. It uses the same exchange and local infrastructure, so it is independent in time, not an external audit. A separate observer will record post-decision bid/ask references with at least 500 ms delay and mark missed windows; the quotes are not guaranteed fills.
Before the prospective test, we checked public Bybit top-of-book snapshots during a short window on 29 October 2026, 14:30–18:00 UTC. The collector recorded 224 confirmed symbol-bars across 16 markets; 222 had a retained bid/ask snapshot. Median quoted spread across the snapshots was 1.06 bps. On our host clock, the median confirmed-candle receipt was 724 ms after the scheduled boundary. This is a small pre-forward diagnostic: quotes can precede a decision, local clock error affects the delay, and no order was sent. It does not measure actual execution. Diagnostic summary.
Separately, between 18:57 and 18:59 UTC on 29 October, we submitted five 0.001 BTCUSDT market-order round trips to Bybit's demo environment. In the first, simulated fills were 80,851.8 USDT on entry and 80,845.9 on exit; fees totalled 0.08893374 USDT. All five closed with a loss of between 0.08730960 and 0.10012708 USDT including fees and the price move while holding each position. The anonymized records include pre-entry quotes and execution timestamps. This short, one-symbol demo sample contains no model signals: it is not a representative cost estimate, evidence of live fills, or validation of the historical backtest.
A separate demo-only pilot is configured to attempt at most one 0.001 BTCUSDT round trip per primary model on eligible forward signals, sequentially on a shared account. It can stop for manual review, and connectivity failures can leave a demo position open. Signals can be missed, and the two trades are neither paired nor a statistically useful comparison. Demo fills are simulated; they cannot establish live execution. The pilot does not change the primary shadow protocol.
Download the research code and results · Reproduction instructions. We reran all seven models from the extracted bundle using the original files and reproduced the reported aggregates. The bundle includes a public-API downloader and a semantic fingerprint of the original candle slice, including missing timestamps. We separately verified all 140,255 BTC rows and 1 January 2024 for all 16 markets (96 bars each); the other 15 markets have not yet been independently downloaded in full. The downloader stops if the data available from Bybit do not match.
References: LightGBM paper, XGBoost paper, CatBoost paper, and scikit-learn's time-aware cross-validation example.
Top comments (0)