DEV Community

Cover image for I built a ghost detector for backtest edges. It was 99% confident and 80% right.
rytjy
rytjy

Posted on Edited on Originally published at dev.to

I built a ghost detector for backtest edges. It was 99% confident and 80% right.

Hacktoberfest Open-Source AI Challenge: Ghost in the Machine Submission ๐Ÿ‘ป๐ŸŒผ

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine.

Every strategy in my archive looked alive once. A few still do. The rest are ghosts: equity curves that fell apart when the data changed, parameters that were tuned on the same bars they were judged on, an edge that existed only inside one sample.

On Dรญa de los Muertos you set a table for the ones who are gone and say their names. This post is that table for my dead configs โ€” a small ofrenda built out of code โ€” and the detector is the thing that tells me, before I spend money, which of the living are already ghosts.

The clearest case is my best-looking config: an in-sample Sharpe of 0.981, the highest in the set. The detector's verdict on it: 0.101. And the first time I ran the tool on my own archive, all 29 strategies scored exactly 0.000 โ€” a fact about the tool, not about the strategies.

I wanted a detector that answers one question โ€” given only what you could have known in-sample, how likely is this edge to survive? โ€” and, more importantly, a detector honest about its own confidence. The second part turned out to be the hard one.

And it is overconfident exactly where confidence matters: when it says almost certainly real it is right about 80% of the time, not 99%. That gap is the reason this post exists.

What I Built

ghostscore is a small CLI. You give it a strategy's in-sample statistics; it returns a probability that the edge survives out of sample. It is trained entirely on synthetic markets where the answer is known, then applied to real configs from my archive.

It's for anyone who already has a strategy and a backtest and wants to know whether the edge is real before risking money on it.

$ python3 -m ghostscore.cli real --n 900 --seed 7
spearman(ghost_score, OOS Sharpe) = 0.512   p = 0.0045   n = 29
top-5 mean OOS Sharpe 0.482 ยท bottom-5 mean OOS Sharpe 0.220
Enter fullscreen mode Exit fullscreen mode

On 29 real configs โ€” MACD variants, Donchian, Bollinger, RSI, time-series momentum, AO, each long/short with and without an EMA200 gate โ€” the ranking correlates with realised out-of-sample Sharpe at 0.51. Not a crystal ball. Enough to be useful as a filter.

The most instructive row is the one that looks best in-sample:

config IS Sharpe OOS Sharpe ghost score
Donchian (20) 0.981 0.455 0.101
MACD 12/26/5 + gate 0.475 0.496 0.865
MACD 19/39/9 0.631 0.709 0.536

Donchian (20) has the highest in-sample Sharpe of the entire set and a survival probability of 0.101. That is the ghost the tool is for.

Two things are true about that 0.512, and only one of them is flattering. The 29 real configs never entered training โ€” the model only ever saw synthetic markets, so this is a genuine out-of-distribution check rather than a fit. But I did use those 29 rows to diagnose the two bugs described below, which means they are not a pristine holdout either. The honest reading is "one archive, one sample", not "validated".

How I use it

As a triage band, not a probability: below 0.2 I do not spend money on the config; 0.2โ€“0.6 I paper-trade; above 0.6 I look at the cost model before anything else. The bands come from these 29 rows and will move with a second archive.

Demo

ghostscore demo

A real terminal recording, not a mock-up. This clip runs with --n 300 to keep it short, so its holdout line (AUC 0.860, ECE 0.178) differs slightly from the numbers in this post, which come from --n 900. Regenerate either with scripts/record_demo.py or ghostscore.cli real.

Same run as text, for anyone who cannot play the clip:

$ python3 -m ghostscore.cli real --n 900 --seed 7
synthetic holdout: AUC 0.850  Brier 0.180  ECE 0.141  base 0.400  backend sklearn-hgb
   family            config  is_sharpe  oos_sharpe  ghost_score
     MACD     12/26/5 +gate      0.475       0.496        0.865
       AO           (5, 34)      0.282       0.608        0.823
  RSI-rev      (2, 10, 90)      0.206       0.475        0.793
Enter fullscreen mode Exit fullscreen mode

Code

github.com/rytjy/ghostscore โ€” MIT, Python, no API keys, CPU only. The full run takes ~12 s on one CPU core.

git clone https://github.com/rytjy/ghostscore && cd ghostscore
python3 -m ghostscore.cli run  --n 1500 --seed 7   # synthetic only
python3 -m ghostscore.cli real --n 900  --seed 7   # score real configs
python3 scripts/plot_calibration.py results/holdout.json
Enter fullscreen mode Exit fullscreen mode

How I Built It

A generator where the truth is known

Three market regimes:

  • noise โ€” iid returns, nothing to exploit but costs;
  • trend โ€” positively autocorrelated returns, momentum pays;
  • revert โ€” negatively autocorrelated, momentum bleeds.

Returns are Student-t (fat tails) with occasional negative jumps, because Gaussian innovations put every real config outside the training support โ€” more on that below. Half the strategy population uses an EMA200 gate, matching the archive and producing the long flat stretches that real PnL series actually contain.

A strategy is sign(trailing k-bar return), filled at the close, costing 1/2/5 bps per edge. In-sample length is sampled from 900โ€“1,900 bars so the model sees the length of the real configs (~1,094).

My first labels were pure noise (AUC 0.489)

This was my first failure. Using a single out-of-sample window as the label gave AUC 0.489 โ€” chance. A 480-bar OOS Sharpe has a standard error near 1.0; the label was mostly noise. The correlation between in-sample and out-of-sample Sharpe was โˆ’0.02.

The fix: average OOS Sharpe over three independent continuation paths and strengthen the regimes. Same data, same features, back to 0.85.

Features that survive scale changes

Eleven in-sample features, all scale-free: Sharpe, skew, kurtosis, hit rate, turnover, max drawdown, profit factor, t-stat, lag-1 autocorrelation, Sharpe standard error, bar count. Absolute mean and volatility are deliberately excluded: synthetic markets run at 1% daily vol, crypto runs at 3โ€“5%, and a tree model reads that difference as a regime.

One report, one named backend

TabPFN is the intended classifier โ€” a tabular foundation model, few-shot, CPU, no training loop. A gradient-boosting fallback keeps the demo runnable. Every report prints backend, so a number can never be attributed to the wrong model.

The honest part: 99% confident, 80% right

Synthetic holdout, n=270:

AUC 0.850 ยท Brier 0.180 ยท ECE 0.141 ยท base rate 0.400
Enter fullscreen mode Exit fullscreen mode
predicted bin n predicted observed
0.0โ€“0.1 110 0.013 0.136
0.9โ€“1.0 71 0.988 0.803

Reliability diagram: predicted probability vs observed frequency, 10 fixed-width bins

When it says "almost certainly real" it is right 80% of the time, not 99%. When it says "almost certainly a ghost" it is right 14% of the time, not 1%. Ranking and calibration are different abilities; this tool is good at the first and overconfident at the second. That gap is the finding.

The second failure: every real score was 0.000

The first application to the archive returned 0.000 for all 29 configs. Two causes:

  1. max_dd divided by a peak that can sit near zero, producing values like โˆ’4.8e11;
  2. Gaussian innovations (kurtosis โ‰ˆ 3) against real crypto kurtosis of 10โ€“240.

Fixes: multiplicative drawdown bounded to [โˆ’1, 0], Student-t innovations with negative jumps, and the EMA200 gate in half the synthetic population. After that the scores spread from 0.003 to 0.865 and the rank correlation appeared.

I kept both failures in the repository history. A detector that was chance, then zero, then useful is a more honest artefact than one that arrived useful.

Why Open Matters Here

The model is not a black box. The classifier is an open-source gradient-boosting implementation running on this machine, and every report prints backend=sklearn-hgb so a number can never be attributed to the wrong model. I wanted TabPFN โ€” an open-weights tabular foundation model โ€” but its install failed here (repeated read timeouts from the package index). Rather than claim a model I did not run, I shipped the fallback and printed its name.

You can see the failures. A closed endpoint would hand you an AUC and an invoice. It would not show you a calibration table where the top bin says 0.988 and delivers 0.803, and it would not have kept the two earlier versions of this tool that scored chance (AUC 0.489) and then zero (all 29 configs at 0.000). Both are still in the repository history.

The regimes are auditable. Every label comes from code you can read, not from a vendor's "AI score". You can disagree with my generator and rerun the experiment โ€” that is the whole difference between an audit and a rating.

The data never leaves. Strategies are the one thing in this field people do not upload. Local, CPU only, no API key, no account.

The Spirit of It

I keep a folder of dead configs. Some were killed by costs, some by a regime that ended, one by a drawdown I did not see because my accounting was wrong. This project is an ofrenda for them โ€” not a memorial, a post-mortem. The detector's job is to say, before the money is spent, which of the living are already ghosts.

The Donchian row is why I trust it more than its AUC. It had the best in-sample Sharpe I have ever produced, and the tool looked at it and said: probably a ghost.

Limits, Stated Up Front

  • 29 real configs, one archive. Everything about real markets here rests on 29 rows from my own strategy folder โ€” six families (MACD, Donchian, Bollinger, RSI, time-series momentum, AO), long/short, with and without an EMA200 gate. It is not a benchmark.
  • The label is Sharpe. "Survives" is defined as out-of-sample Sharpe staying positive. A tool trained on that will miss an edge that shows up as convexity or as fat tails instead.
  • The regimes are mine. Noise, trend and revert, with my autocorrelation and jump settings. If real markets have a fourth shape, the generator never taught it.
  • Calibration is the weak end. At the two extremes the tool is off by about 12 and 19 points. Use it to rank, not to size positions.
  • Costs are 1/2/5 bps per edge. No funding, no borrow, no slippage beyond that, no capacity limit.

The test I owe this tool: run it on a second archive I did not build. If the rank correlation there comes in under 0.3, the detector is the ghost and this post is wrong.

Prize Categories

None claimed. The TabPFN path is implemented in the repository, but its install failed on this machine, so every number here comes from the sklearn fallback and every report prints backend=sklearn-hgb.

About the Model, Honestly

No partner category is claimed. The TabPFN path is implemented in the repository, but its install failed on this machine (repeated read timeouts from the package index), and every number in this post comes from the gradient-boosting fallback โ€” the reports print backend=sklearn-hgb for that reason. Claiming a partner category on a backend I did not run would be exactly the kind of in-sample story this tool is built to catch.

Build Log

Six commits, all inside the challenge window (first 2026-10-27 05:08Z, last 2026-10-27 10:04Z): commit history. The two failures in this post are in that history, not rewritten. A DevRelay session is not included because the tool does not support this environment; this log replaces it.

All numbers are reproducible from the repository: python3 -m ghostscore.cli real --n 900 --seed 7 and python3 scripts/plot_calibration.py results/holdout.json. Written with an AI assistant; every figure comes from code in the repo.

Top comments (0)