The last post was about making a number trustworthy. Leakage geometry, purge widths, de-overlap, a baseline that could not cheat. It ended with a minute-scale ceiling that held at 52% across seven configurations and a model family swap.
This one is about what happens after you trust the number. Because a probability you are going to bet on is a different object from a probability you are going to report.
The probabilities are not decorative
The path-passage classifier is a three-class LightGBM. It returns p_up, p_down, p_none. Those go straight into the expected-value score that decides whether to take a trade and how big:
long_score = p_up * (B - C) + p_down * (-B - C) + p_none * (-C)
short_score = p_up * (-B - C) + p_down * (B - C) + p_none * (-C)
B is the barrier, C the cost. Read the arithmetic. Every term is linear in a probability. Scale p_up by 1.2 and you scale the long score by very nearly 1.2.
So miscalibration does not stay in the model. It becomes a bet-sizing error, in proportion, in the bins where the gate actually fires. A classifier that is right 70% of the time while claiming 90% is not 20 points wrong. It is sizing every position in that bin as though the edge were far larger than it is.
Boosted trees are known for uncalibrated softmax output. I had been consuming it as if it were a probability.
The audit
Seven live assets. For each one, fit an Inductive Venn-Abers wrapper on the time-ordered older 80% of that model's training data, 6,988 rows, and evaluate against a 500-row uniform-random sample of the newer 20%, seed 42. The LightGBM models are reloaded from disk and left alone. Only the wrapper is fit.
Measure Expected Calibration Error and log-loss, before and after.
| Asset | ECE before → after | ECE Δ | Log-loss Δ |
|---|---|---|---|
| BTC | 0.1272 → 0.0621 | -51.2% | -5.5% |
| ETH | 0.1795 → 0.0298 | -83.4% | -11.5% |
| SOL | 0.1680 → 0.0386 | -77.0% | -10.6% |
| XRP | 0.2219 → 0.0645 | -70.9% | -17.7% |
| ADA | 0.1419 → 0.0369 | -74.0% | -8.0% |
| LINK | 0.1260 → 0.0737 | -41.5% | -1.2% |
| LTC | 0.1508 → 0.0603 | -60.0% | -14.1% |
Every asset has a real gap. That settles the first question, which was whether this was one bad model or a property of the setup. It is systematic.
The second question is the interesting one.
The direction is asset-specific, and that rules out the easy fix
ETH fails the way boosted trees are supposed to fail. Its worst reliability bin is [0.90, 1.00]. Eleven samples. Mean stated confidence 93.8%. Empirical accuracy 45.5%.
Most certain, least reliable. That is the bin where the EV gate fires hardest and the position sizes are largest.
The other six fail in the opposite direction.
XRP, in the [0.60, 0.70] bin: 93.1% accuracy at 64.9% stated confidence, n=116. A 28-point understatement.
LTC, same bin: 90.4% accuracy at 64.7% confidence, n=115.
That is a suppressed-signal failure. The gate does not fire often enough, because the stated confidence lags what the model actually delivers. It costs money quietly, by declining trades that were good.
One asset over-confident. Six under-confident.
Which kills the convenient answer. Platt scaling and temperature scaling apply one monotone correction. They cannot pull ETH's tail down and push XRP's middle up at the same time, because those are corrections in opposite directions. A single global calibrator fits the average of two failure modes and helps neither.
Per-asset Venn-Abers works here because it fits each asset's own reliability curve and does not assume a shape.
Where the split came from
This is the part I did not expect, and it is the reason I keep the training params in the same document as the audit.
The hardened LightGBM settings were min_data_in_leaf=400, num_leaves=8, max_depth=3. I introduced those specifically to stop the ETH-style saturation, where terminal leaves go to 1.0 and the model claims certainty it has not earned.
They worked. They also worked too well.
Constraining the leaves prevented the over-confidence failure and produced a structural under-confidence pattern across the rest of the universe. The fix for one failure mode manufactured the opposite failure mode in six assets.
ETH is the lone holdout that still saturates, because in the rare cases where it is genuinely certain its leaves still reach the top of the range. Eleven samples in that bin tells the story: sparse and extreme.
That is a straight tradeoff I made without knowing I was making it. Reliability diagrams are what showed it. An accuracy score would have shown a modest improvement and nothing else, because averaging is exactly the operation that hides a bin.
The bar was set before the numbers came back
ECE has to improve by at least 50%, and log-loss by at least 5%. Both, not either.
Six assets clear it. LINK-USD misses both, at -41.5% and -1.2%.
LINK is not a data problem. It has the same 8,736 training rows as everything else, so this is model quality rather than availability. The LightGBM may simply be better calibrated for LINK already, in which case there is less for the wrapper to do and the small lift is honest. Or the calibrator needs different hyperparameters. Both are worth a follow-up and neither is resolved today.
So the rollout is selective. Six of seven, and LINK stays on raw softmax until somebody investigates it.
The mechanism is deliberately boring. The loader falls back to raw softmax for any asset with no calibrator on disk, so excluding LINK means either leaving its pickle off the deployment image or adding an allow-list:
allow = set(os.environ.get(
"VENN_ABERS_ASSETS",
"BTC-USD,ETH-USD,SOL-USD,XRP-USD,ADA-USD,LTC-USD",
).split(","))
One environment variable and three lines in the loader. A rollout that cannot express "these six and not that one" ends up shipping the failure with the successes.
What this audit does not prove
The status line at the top of the integration guide says prototype, not integrated into the live inference path. That is still true, and the caveats are worth having in the open.
The holdout is a 500-row uniform sample, seed 42. The 80/20 split is on the model's own training data rather than on live signals.
There has been no live-data refit, because signals_history does not yet carry realized 24-hour outcomes for the p_up/p_down/p_none rows. Migration 028 landed on 2026-05-21 to start collecting them.
BTC's barrier is 150 bps and the rest are 200, so the ECE deltas compare cleanly but the absolute ECE values across assets carry a class-balance shift. I treat those as informal.
LINK's failure in particular should be re-run once live outcomes exist. A training-data fit can hide a different live picture, and that is exactly the asset where I would expect it.
Why this is the second half of the last post
The geometry work answered whether the measurement could be trusted. Purge, embargo, de-overlap, a split that cannot leak. All of that gets you a number you can believe.
Believing the number is not the same as being able to bet on it. The ceiling post established that the minute-scale direction signal is capped near 52% and that the cap is real rather than an artifact. This audit establishes something narrower and more immediate: on the horizon where there is signal, the probability the model hands the sizer is not the probability it should act on, and the correction is different for every asset.
Accuracy averages. A reliability diagram does not. The bin where the model was most confident and least correct is worth more attention than any figure computed across the whole set, because it is the bin where the money goes.
🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series
Top comments (0)