DEV Community: rytjy

37 public leaderboard strategies, one screen: the arithmetic holds, the concentration does not

rytjy — Sat, 07 Nov 2026 08:19:14 +0000

I parsed every strategy on a public leaderboard this week - 37 ids from Collective2's server-rendered page, monthly tables only, no login - and recomputed each one locally before looking at any ranking.

What I recomputed: yearly monthly cells compounded against the printed YTD; annualised return as eq^(365/days)-1; the share of total log return carried by the best three months; annual return against reported max drawdown; record length; win rate against the break-even win rate implied by average win and average loss.

The arithmetic reconciles. Across 36 active strategies, every year's monthly cells compound to the printed YTD within 0.4pp, and the annual figure matches calendar-day annualisation within -1.6 to +0.2pp on the five strategies I checked by hand. That rules out "the platform cannot compute". Any criticism has to be about what the number means, not whether it adds up.

What varies (36 active):

19 carry more than 70% of their total log return in the best three months. One carries 194% - outside those months the record is net negative.
11 have an annual return above 2.5x the reported max drawdown.
11 have less than 18 months of history behind an annualised number.
3 have a win rate under 2 standard errors above break-even.
14 carry no flag at all. A clean result is also a result.

Two examples, both public:

152974419: 14 months, +55.6% cumulative, 49.2% annualised, 7.5% max drawdown. One month - 2026-07 at +24.1% - carries 84% of the log return. 469 trades, 91.7% win rate against a 58.3% break-even: the per-trade edge looks real, the annual figure is one month.
13202557: 22 years, +6,979%, and only 12% of it in the best three months. Concentration is not the issue; a 77% max drawdown is.

The limits, since a screen is not an audit: the leaderboard is a selection - it shows winners, and the strategies that disappeared are not measured; everything is hypothetical per the platform's own disclaimer; no broker statements were seen; the edge test assumes a fixed average win and loss.

One page with the table and both examples: https://github.com/rytjy/backtest-audit-kit/raw/main/docs/c2-screen-37.pdf
Parsed table: https://github.com/rytjy/backtest-audit-kit/raw/main/docs/c2-screen-37.table.txt
Repo: https://github.com/rytjy/backtest-audit-kit

A vendor claimed +90.2% annualised. Their own monthly table says 34.8% without three months.

rytjy — Sat, 07 Nov 2026 06:06:03 +0000

Last week a trading-platform newsletter landed in my inbox with a strategy of the week.

The headline: Hypothetical Annual Return (Compounded) +90.2%. Somewhere above it, in smaller type: "All results are hypothetical."

I do not know whether the strategy works. I do know that a monthly percentage table is enough to check the arithmetic, so I checked it.

The table

Two and a bit years of monthly cells, printed right there in the email:

Year	months (%)
2024	-3.8, +8.3, -0.5, -7.9, +5.3, +3.9, +4.5
2025	+8.5, +0.1, +0.6, +50.7, +7.6, +4.5, +3.3, +10.9, -, +6.2, +12.4, +10.7
2026	+14.1, +23.4, -20.9, +20.3, -6.1, -8.9, -3.9, +2.8, +9.3, +18.5, +2.2, +50.1

Compound them:

2024 (7 months): +9.16%
2025 (11 months, September missing): +180.27% — the email prints +180.4%
2026 (11 months): +49.9% — the email prints +50.1%
Full span: roughly +350%, which annualises to ~90.3%

So the +90.2% is not a typo and not a fabrication. The arithmetic holds. That is the part almost nobody checks, and it passed.

What the arithmetic hides

Three questions turn "consistent" into "informative".

1. How much rests on a few months?

The three largest cells are +50.7%, +23.4%, +50.1%. Remove those three months and the annualised figure falls to about 34.8%. The strategy is not a smooth 90% engine; it is a handful of good months carrying a decade of averages.

2. What is the worst peak-to-trough on monthly closes?

-21.8%. And 2026 contains -20.9%, then -6.1%, -8.9%, -3.9% in a row. Anyone sizing this needs to know that sequence exists before they read the annualised number.

3. What exactly is being measured?

The vendor's own disclaimer: "there is no single real-life trading account that looks exactly like the track record shown here." Fees are estimated, not taken from a broker statement. The track record is a model account.

None of that is fraud. It is a hypothetical number, clearly labelled as such, in 8-point type.

The three-question check

For any seller quoting an annualised return:

Ask for the monthly cells, not the annual figure.
Compound them yourself. If it does not match, stop and ask why. If it matches, continue.
Drop the three best months and recompute; compute the worst monthly peak-to-trough.

Two minutes, a calculator, and you learn whether the number is durable or decorative.

I keep the working notes and a small standard-library toolkit for this kind of check in backtest-audit-kit. The write-up for this case is case 15.

If you sell a strategy and your numbers survive these three questions, say so — that is rarer than a 90% headline.

Fifty Authorizations Is Not an Attack: 30 Blocks of EIP-7702 Data

rytjy — Fri, 30 Oct 2026 10:08:28 +0000

On 2026-10-30 a major exchange's hot wallets were drained. Public reporting says the attackers never saw a private key — they got the signing infrastructure to sign 50 EIP-7702 authorizations in one transaction. I wrote a decoder to look for that shape on-chain. I found it in the first 30 blocks. It was a product, not an attack.

The claim I wanted to test

Reports described the pattern as: one transaction, dozens of authorization_list entries, nonce values in sequence, all landing in a single block. If that shape is the signature of the attack, then a simple rule — more than N authorizations in one transaction = suspicious — should find it.

So I built the smallest tool that could check: RLP decoding, keccak256, secp256k1 recovery, no web3 dependency. EIP-7702 authorizations are signed over

keccak256(0x05 || rlp([chain_id, address, nonce]))

Recover the signer, compare it with the transaction's from, and you can answer the only three questions that matter structurally: who handed execution rights to which contract, does that contract have code, and how many authorizations are packed into this one transaction.

The sample

Blocks 26,303,375 – 26,303,404 — about six minutes of mainnet time. Every type-4 transaction:

Metric	Value
type-4 transactions	133 (~4.4 per block)
authorizations	319
unique signers	308
unique delegates	104
`chain_id = 0` (replayable everywhere)	0
delegate with no code	1 (pointing at `0x0`)
auth signer != tx sender	38 / 133

Distribution of authorizations per transaction: 120 transactions carry exactly one. The rest are 2, 3, 4, 5, 6, 8, 21, 30, 32, 34 — and one with 50.

The transaction that broke my rule

Block 26,303,378, transaction 0x65f116e6…2e23:

50 authorizations, 50 distinct signers
all pointing at the same delegate 0xe6b9…43ed (772 bytes of code)
submitted by a single sender 0x59aa…e16b
the same sender filed 34 / 32 / 30 / 21 / 8 / 5-authorization transactions in adjacent blocks

That is precisely the shape the incident was described with. It is also, almost certainly, a batch delegation service — one operator, steady cadence, every signer handing execution rights to the same deployed contract, and no assets leaving in the same window.

A rule of "5+ authorizations in one transaction = high severity" would have fired five times in six minutes. All five would have been wrong. An alarm that is 100% false positives is not an alarm.

What the rule should look like instead

Structure alone is not enough. You need behavior layered on top, ordered by how cheaply it can be verified:

Is the delegate new? A first-seen contract plus a batch of authorizations starts to look like an attack. A delegate with months of history does not.
Did assets move in the same window? In the reported incident, the authorizations were not the payload — the transfers three minutes later were. Authorization and transfer have to be evaluated together.
Do the signers have any history with the sender? 38 of 133 transactions had signer != sender, which is a legitimate relayer pattern. It stops being unremarkable when the signers have never interacted with that sender before.

I implemented layer 1 only. Layers 2 and 3 need an indexer.

Update: I found the attack transaction

After publishing the first version of this post I went back and scanned blocks 26,300,750 – 26,300,800 for every transaction touching Binance 14 (0x28c6…1d60). It is in there.

block 26,300,753 — 2026-10-29 20:20:47 UTC
tx 0xad99264d…298b, type 4, from 0x7c8601…5ca0, to Binance 14
50 authorizations, every one of them signed by Binance 14
all delegating to 0x63e842b0…6fae (1,674 bytes of code), chain_id = 1
nonces 18,914,824 down to 18,914,775 — contiguous, 50 in a row

The receipt is status: 1, 1,258,562 gas, and zero logs. That transaction moved nothing. It installed the delegation, and the transfer came afterwards.

The discriminator is the nonce, not the count. The batch-delegation product had 50 authorizations from 50 different signers. The attack had 50 authorizations from one signer with consecutive nonces — a stack of pre-signed delegations to replay one after another. You can detect that without an indexer: group authorizations by signer, and flag any signer that appears N times with contiguous nonces. A product doing mass migration never repeats a signer; an attacker building a replay stack always does.

That rule is now code, not a paragraph: w7702.py flags any signer appearing three or more times over contiguous nonces. Run against both transactions: the attack fires replay stack: 50 authorizations from 0x28c6… over contiguous nonces 18914775..18914824; the 50-signer product fires nothing. Eight unit tests, four of them for this rule alone.

And then I measured how often it fires on ordinary traffic. 300 randomly sampled blocks spanning seven days: 1,015 type-4 transactions, 16 replay stacks — about 1.6%. Every one of them carried 3 or 5 authorizations; none carried more than five. The attack carried 50.

So contiguity alone is a weak signal. A wallet pre-signing its next five delegations looks structurally identical to an operator pre-signing fifty, and the small version is routine: one sender in that sample filed three separate 5-nonce stacks back to back, which is exactly what an operational hot wallet does when it tops up its delegation budget.

The separating dimension in this data is scale, not shape. That is a worse answer than the one I wrote an hour ago, and it is the one the numbers support.

The delegate itself is purpose-built. Disassembling its 1,674 bytes yields exactly one embedded address — 0x7c8601…5ca0, the operator who sent the transaction. The destination is hardcoded into the contract, not supplied as an argument. A batch-delegation product is generic; this one already knows where the money goes.

One more thing the scan showed: across blocks 26,300,750 – 26,300,821 (about fourteen minutes after the delegation landed), no transaction calls into Binance 14 at all — only the hot wallet's own outbound traffic. So whatever moved the reported 167,464 ETH did not do it by calling the delegated account directly in that window.

Binance 14 returns 0x today — no code, delegation cleared.

My own tool was wrong first

The unit tests failed on the first run: my RLP encoder wrote integer 0 as 0x00. RLP encodes zero as the empty string 0x80.

The consequence is not cosmetic — a nonce = 0 authorization hashes differently, and recovery returns a fake signer. Nonce zero is exactly what a first-time authorization from a fresh account looks like, so the bug would have produced wrong answers on the transactions most worth inspecting.

Fixed, with a regression test. The same test round produced a real validation: in 0x73af…756f, the authorization signer is not the transaction sender, and the tool flags no authorization belongs to the transaction sender. That is a relayer submitting on someone's behalf — legal, and exactly the kind of "valid on-chain, unclear intent" case a human should look at.

What I did not verify

The figures — $5.8B of outflows, $3B frozen, 167,464 ETH in the first transfer — are still from public reporting. I confirmed the delegation transaction; I did not reconstruct the transfers themselves, and I have not traced the delegate contract's internal calls.
The 30-block sample above and the incident block are different windows; the earlier numbers (133 type-4 transactions) do not include the attack.
30 blocks is roughly six minutes. "4.4 type-4 transactions per block" does not extrapolate to a day.
I did not audit any delegate bytecode. The tool emits structural warnings; deciding whether a delegate is malicious is still a human judgement.
Zero chain_id = 0 authorizations means zero in this window, not zero on mainnet.

Why I am publishing the negative result

The interesting output of six minutes of scanning is not a detection — it is a disproof of a detection rule I was about to write. Shape-based rules are cheap to state and expensive to be wrong about in public. The data said no, so the rule changed.

Code and full numbers: security/w7702/ — decoder, incremental scanner, five tests, and the raw report with an explicit scope statement.

I tried to reproduce four Collective2 Calmar ratios. None of them matched.

rytjy — Wed, 28 Oct 2026 05:03:52 +0000

I spent an evening recomputing the headline statistics on four Collective2 strategy pages. The profit factors mostly hold up. The Calmar ratios do not — none of the four.

This is not a fraud hunt. It is a reproducibility check, the same one I would run on any backtest: take the numbers a page publishes, recompute the derived metrics from them, and see whether they agree.

The two definitions I used

Calmar ratio = annualized return ÷ maximum drawdown
Profit factor = (win % × average win) ÷ ((1 − win %) × average loss)

Both are standard. If a page shows all five inputs, its Calmar and profit factor are arithmetic, not opinion.

Results

Four pages had complete summary blocks on 2026-10-27 (twelve candidates were listed, nine fetched; five showed all-zero stats and were excluded as inactive).

Strategy	Annual return	Max DD	Calmar (page)	Calmar (recomputed)	page ÷ recomputed
131145247	32.4%	14.1%	3.52	2.30	1.53×
132169961	22.3%	31.8%	0.95	0.70	1.35×
146455121	56.9%	29.7%	2.76	1.92	1.44×
146649042	22.8%	13.4%	2.06	1.70	1.21×

Profit factor on the same pages:

Strategy	PF (page)	PF (recomputed)	Difference
131145247	1.50	1.46	+2.8%
132169961	2.70	2.35	+14.8%
146455121	3.10	3.03	+2.4%
146649042	1.70	1.69	+0.9%

What I take from it

1. The summary blocks are mostly coherent. Three of four profit factors land within 3% — rounding level. That is what a well-formed summary looks like. It also means the pages are not sloppy in general, which makes the Calmar result more interesting, not less.

2. The Calmar overshoot is not a constant. 1.21× to 1.53×. If the platform used one different definition consistently, you would see a roughly fixed multiplier. You do not. So it is not "Calmar but annualized monthly" or "Calmar but with a rolling drawdown" — or if it is, the effect varies by strategy in a way the page never tells you.

3. One page fails both checks. 132169961's profit factor is 14.8% above what its own win rate and average win/loss imply. Either one input is stale, or the profit factor is computed over a different trade set than the other fields. Same page, same summary block.

4. The practical problem is comparability, not honesty. If two strategies both show Calmar 2.0, nothing on the page tells you they were computed the same way. Ranking subscriptions by that number is ranking by a quantity whose definition you cannot see. The same applies to any platform metric you cannot recompute.

What this does not say

It does not say any seller is lying. The likeliest explanation is a platform-side definition difference — the profit-factor check mostly passing is evidence for that.
I had no trade-level data and no equity curve. So I cannot say which field is wrong, and I cannot recompute max drawdown independently.
I did not determine what Collective2's Calmar actually uses. That needs the equity series, which these pages do not expose publicly.
Nothing here is a claim about future performance.

Reproduce it

The checker is four rows of published fields and a handful of divisions:

ROWS = [
    # id, ann_ret_pct, max_dd_pct, pf_page, win_pct, avg_win, avg_loss, calmar_page
    ("131145247", 32.4, 14.1, 1.50, 35.6, 903.0, 342.0, 3.52),
    ("132169961", 22.3, 31.8, 2.70, 58.0, 315.0, 185.0, 0.95),
]

for sid, ann, dd, pf_page, win, aw, al, cal_page in ROWS:
    cal_calc = ann / dd
    w = win / 100.0
    pf_calc = (w * aw) / ((1 - w) * al)
    print(sid, cal_page / cal_calc, pf_page / pf_calc - 1)

Full script and tables: backtest-audit-kit/audits.

Why I published this

If you are paying a monthly fee for a strategy, the headline ratio is part of what you are buying. It costs one afternoon to check whether that ratio can be recomputed from the page that advertises it. If it cannot, ask the seller which definition produced it — and treat "whatever the platform shows" as an answer.

Scope: verified — the arithmetic relationship between six summary fields on four public pages on 2026-10-27. Not verified — trade-level accuracy, drawdown computation, equity curve, platform metric definitions, future returns. No strategy seller was contacted.

I built a ghost detector for backtest edges. It was 99% confident and 80% right.

rytjy — Tue, 27 Oct 2026 10:08:31 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine.

Every strategy in my archive looked alive once. A few still do. The rest are ghosts: equity curves that fell apart when the data changed, parameters that were tuned on the same bars they were judged on, an edge that existed only inside one sample.

On Día de los Muertos you set a table for the ones who are gone and say their names. This post is that table for my dead configs — a small ofrenda built out of code — and the detector is the thing that tells me, before I spend money, which of the living are already ghosts.

The clearest case is my best-looking config: an in-sample Sharpe of 0.981, the highest in the set. The detector's verdict on it: 0.101. And the first time I ran the tool on my own archive, all 29 strategies scored exactly 0.000 — a fact about the tool, not about the strategies.

I wanted a detector that answers one question — given only what you could have known in-sample, how likely is this edge to survive? — and, more importantly, a detector honest about its own confidence. The second part turned out to be the hard one.

And it is overconfident exactly where confidence matters: when it says almost certainly real it is right about 80% of the time, not 99%. That gap is the reason this post exists.

What I Built

ghostscore is a small CLI. You give it a strategy's in-sample statistics; it returns a probability that the edge survives out of sample. It is trained entirely on synthetic markets where the answer is known, then applied to real configs from my archive.

It's for anyone who already has a strategy and a backtest and wants to know whether the edge is real before risking money on it.

$ python3 -m ghostscore.cli real --n 900 --seed 7
spearman(ghost_score, OOS Sharpe) = 0.512   p = 0.0045   n = 29
top-5 mean OOS Sharpe 0.482 · bottom-5 mean OOS Sharpe 0.220

On 29 real configs — MACD variants, Donchian, Bollinger, RSI, time-series momentum, AO, each long/short with and without an EMA200 gate — the ranking correlates with realised out-of-sample Sharpe at 0.51. Not a crystal ball. Enough to be useful as a filter.

The most instructive row is the one that looks best in-sample:

config	IS Sharpe	OOS Sharpe	ghost score
Donchian (20)	0.981	0.455	0.101
MACD 12/26/5 + gate	0.475	0.496	0.865
MACD 19/39/9	0.631	0.709	0.536

Donchian (20) has the highest in-sample Sharpe of the entire set and a survival probability of 0.101. That is the ghost the tool is for.

Two things are true about that 0.512, and only one of them is flattering. The 29 real configs never entered training — the model only ever saw synthetic markets, so this is a genuine out-of-distribution check rather than a fit. But I did use those 29 rows to diagnose the two bugs described below, which means they are not a pristine holdout either. The honest reading is "one archive, one sample", not "validated".

How I use it

As a triage band, not a probability: below 0.2 I do not spend money on the config; 0.2–0.6 I paper-trade; above 0.6 I look at the cost model before anything else. The bands come from these 29 rows and will move with a second archive.

Demo

A real terminal recording, not a mock-up. This clip runs with --n 300 to keep it short, so its holdout line (AUC 0.860, ECE 0.178) differs slightly from the numbers in this post, which come from --n 900. Regenerate either with scripts/record_demo.py or ghostscore.cli real.

Same run as text, for anyone who cannot play the clip:

$ python3 -m ghostscore.cli real --n 900 --seed 7
synthetic holdout: AUC 0.850  Brier 0.180  ECE 0.141  base 0.400  backend sklearn-hgb
   family            config  is_sharpe  oos_sharpe  ghost_score
     MACD     12/26/5 +gate      0.475       0.496        0.865
       AO           (5, 34)      0.282       0.608        0.823
  RSI-rev      (2, 10, 90)      0.206       0.475        0.793

Code

github.com/rytjy/ghostscore — MIT, Python, no API keys, CPU only. The full run takes ~12 s on one CPU core.

git clone https://github.com/rytjy/ghostscore && cd ghostscore
python3 -m ghostscore.cli run  --n 1500 --seed 7   # synthetic only
python3 -m ghostscore.cli real --n 900  --seed 7   # score real configs
python3 scripts/plot_calibration.py results/holdout.json

How I Built It

A generator where the truth is known

Three market regimes:

noise — iid returns, nothing to exploit but costs;
trend — positively autocorrelated returns, momentum pays;
revert — negatively autocorrelated, momentum bleeds.

Returns are Student-t (fat tails) with occasional negative jumps, because Gaussian innovations put every real config outside the training support — more on that below. Half the strategy population uses an EMA200 gate, matching the archive and producing the long flat stretches that real PnL series actually contain.

A strategy is sign(trailing k-bar return), filled at the close, costing 1/2/5 bps per edge. In-sample length is sampled from 900–1,900 bars so the model sees the length of the real configs (~1,094).

My first labels were pure noise (AUC 0.489)

This was my first failure. Using a single out-of-sample window as the label gave AUC 0.489 — chance. A 480-bar OOS Sharpe has a standard error near 1.0; the label was mostly noise. The correlation between in-sample and out-of-sample Sharpe was −0.02.

The fix: average OOS Sharpe over three independent continuation paths and strengthen the regimes. Same data, same features, back to 0.85.

Features that survive scale changes

Eleven in-sample features, all scale-free: Sharpe, skew, kurtosis, hit rate, turnover, max drawdown, profit factor, t-stat, lag-1 autocorrelation, Sharpe standard error, bar count. Absolute mean and volatility are deliberately excluded: synthetic markets run at 1% daily vol, crypto runs at 3–5%, and a tree model reads that difference as a regime.

One report, one named backend

TabPFN is the intended classifier — a tabular foundation model, few-shot, CPU, no training loop. A gradient-boosting fallback keeps the demo runnable. Every report prints backend, so a number can never be attributed to the wrong model.

The honest part: 99% confident, 80% right

Synthetic holdout, n=270:

AUC 0.850 · Brier 0.180 · ECE 0.141 · base rate 0.400

predicted bin	n	predicted	observed
0.0–0.1	110	0.013	0.136
0.9–1.0	71	0.988	0.803

When it says "almost certainly real" it is right 80% of the time, not 99%. When it says "almost certainly a ghost" it is right 14% of the time, not 1%. Ranking and calibration are different abilities; this tool is good at the first and overconfident at the second. That gap is the finding.

The second failure: every real score was 0.000

The first application to the archive returned 0.000 for all 29 configs. Two causes:

max_dd divided by a peak that can sit near zero, producing values like −4.8e11;
Gaussian innovations (kurtosis ≈ 3) against real crypto kurtosis of 10–240.

Fixes: multiplicative drawdown bounded to [−1, 0], Student-t innovations with negative jumps, and the EMA200 gate in half the synthetic population. After that the scores spread from 0.003 to 0.865 and the rank correlation appeared.

I kept both failures in the repository history. A detector that was chance, then zero, then useful is a more honest artefact than one that arrived useful.

Why Open Matters Here

The model is not a black box. The classifier is an open-source gradient-boosting implementation running on this machine, and every report prints backend=sklearn-hgb so a number can never be attributed to the wrong model. I wanted TabPFN — an open-weights tabular foundation model — but its install failed here (repeated read timeouts from the package index). Rather than claim a model I did not run, I shipped the fallback and printed its name.

You can see the failures. A closed endpoint would hand you an AUC and an invoice. It would not show you a calibration table where the top bin says 0.988 and delivers 0.803, and it would not have kept the two earlier versions of this tool that scored chance (AUC 0.489) and then zero (all 29 configs at 0.000). Both are still in the repository history.

The regimes are auditable. Every label comes from code you can read, not from a vendor's "AI score". You can disagree with my generator and rerun the experiment — that is the whole difference between an audit and a rating.

The data never leaves. Strategies are the one thing in this field people do not upload. Local, CPU only, no API key, no account.

The Spirit of It

I keep a folder of dead configs. Some were killed by costs, some by a regime that ended, one by a drawdown I did not see because my accounting was wrong. This project is an ofrenda for them — not a memorial, a post-mortem. The detector's job is to say, before the money is spent, which of the living are already ghosts.

The Donchian row is why I trust it more than its AUC. It had the best in-sample Sharpe I have ever produced, and the tool looked at it and said: probably a ghost.

Limits, Stated Up Front

29 real configs, one archive. Everything about real markets here rests on 29 rows from my own strategy folder — six families (MACD, Donchian, Bollinger, RSI, time-series momentum, AO), long/short, with and without an EMA200 gate. It is not a benchmark.
The label is Sharpe. "Survives" is defined as out-of-sample Sharpe staying positive. A tool trained on that will miss an edge that shows up as convexity or as fat tails instead.
The regimes are mine. Noise, trend and revert, with my autocorrelation and jump settings. If real markets have a fourth shape, the generator never taught it.
Calibration is the weak end. At the two extremes the tool is off by about 12 and 19 points. Use it to rank, not to size positions.
Costs are 1/2/5 bps per edge. No funding, no borrow, no slippage beyond that, no capacity limit.

The test I owe this tool: run it on a second archive I did not build. If the rank correlation there comes in under 0.3, the detector is the ghost and this post is wrong.

Prize Categories

None claimed. The TabPFN path is implemented in the repository, but its install failed on this machine, so every number here comes from the sklearn fallback and every report prints backend=sklearn-hgb.

About the Model, Honestly

No partner category is claimed. The TabPFN path is implemented in the repository, but its install failed on this machine (repeated read timeouts from the package index), and every number in this post comes from the gradient-boosting fallback — the reports print backend=sklearn-hgb for that reason. Claiming a partner category on a backend I did not run would be exactly the kind of in-sample story this tool is built to catch.

Build Log

Six commits, all inside the challenge window (first 2026-10-27 05:08Z, last 2026-10-27 10:04Z): commit history. The two failures in this post are in that history, not rewritten. A DevRelay session is not included because the tool does not support this environment; this log replaces it.

All numbers are reproducible from the repository: python3 -m ghostscore.cli real --n 900 --seed 7 and python3 scripts/plot_calibration.py results/holdout.json. Written with an AI assistant; every figure comes from code in the repo.

Same code, same question, three layouts: the verdict moved, the code did not

rytjy — Mon, 26 Oct 2026 10:56:22 +0000

I built a small harness to answer one narrow question: when a local model answers
"is there a lookahead bug in this snippet?", how much of that answer is about the code?

Short answer: on a 2B model, not much. Move the question above the code and the
verdict flips - deterministically, on both runs.

Setup

6 snippets from real backtest-review work: 3 with a genuine defect (lookahead, omitted fees, grid-search maximum reported as strategy performance), 3 that match a suspicious pattern but are correct.
3 prompt layouts, identical content otherwise:
- E = evidence first: code, then question
- Q = question first: question, then code
- H = evidence first plus "if the code does not contain enough information, answer UNSURE. Do not guess."
2 runs per cell, temperature 0.2, 16-token cap, CPU only, llama.cpp b11804.
Two models: gemma-2-2b-it Q4_K_M and qwen2.5-coder-1.5b-instruct Q4_K_M.
72 calls total.

Ground truth is my own reading of each snippet, recorded before the runs.

Result 1: the 1.5B model answers YES to everything

qwen2.5-coder-1.5b returned YES on 36 of 36 calls, under all three layouts.

Exactly half the items are positive, so an always-YES model scores 50%. It scored
50.0%. Zero abstentions, zero flips: perfectly stable and perfectly uninformative.
Stability and usefulness are different properties, and a fixed-answer model gets a
perfect score on the stability one.

Result 2: the 2B model lands inside the trivial baseline

model	accuracy	always-YES baseline	YES rate	abstentions
gemma-2-2b-it	19/36 = 52.8%	18/36 = 50.0%	75.0%	0
qwen2.5-coder-1.5b	18/36 = 50.0%	18/36 = 50.0%	100.0%	0

One correct answer above the always-YES baseline, on 36 calls. That is not a
capability, it is rounding.

The UNSURE instruction changed nothing: 0 abstentions across 72 calls. Both models
answered with full confidence every single time.

Result 3: 4 of 6 items change verdict when only the layout changes

Per item, in order E | Q | H, one character per run:

item	truth	E	Q	H
lookahead	defect	YY	NN	YY	layout-sensitive
fee_missing	defect	YY	NN	YY	layout-sensitive
grid_select	defect	YY	YY	YY	stable
fee_named_other	clean	NY	NN	YY	layout-sensitive
rolling_ok	clean	YY	YY	NN	layout-sensitive
label_shift	clean	YY	YY	YY	stable

Two of these are not noise. lookahead and fee_missing were answered YES twice with
the code first, and NO twice with the question first. Same model, same text, same
temperature - only the order changed, and the flip reproduced.

Adding the UNSURE hint moved rolling_ok from YES, YES to NO, NO.

Only one cell in the whole grid is plain run-to-run noise: fee_named_other under E,
which gave NO then YES.

What I take from it

If a verdict changes when you move the question above the code, that verdict is not
a measurement of the code. It is a measurement of the prompt.

Concretely: one ask per finding is not enough for anything automated. Either ask the
same item under several layouts and report the disagreement as a confidence signal,
or keep the model as a pointer and let a human read the snippet.

Also: consistency is not evidence. The most consistent model here never looked at
the code.

A bug I found in my own harness

The first version of this bench printed YES for all 36 gemma calls. Not because the
model agreed with itself, but because the parser never read the model.

This llama.cpp build still echoes the prompt next to the answer
(--no-display-prompt notwithstanding). The prompt contains the sentence "Answer with
exactly one word: YES, NO, or UNSURE", and my parser took the first YES/NO/UNSURE
token in stdout - which is that sentence, not the model output.

After the fix, 9 of 36 cells changed, every one of them from YES to NO: each was a NO
the model had actually produced and the parser had overwritten. Rows from the buggy
parser are kept in runs/*.buggy1026 and are counted nowhere above.

Fix: keep full stdout, cut at the last Answer: marker, parse from there. Everything
above comes from the fixed parser, and the harness now stores complete output so any
row can be re-checked.

The lesson is not "check your parser". It is that a harness with this bug reports a
perfectly stable, perfectly wrong model, and nothing in the output looks broken.

Limits

6 items. A smoke test, not a benchmark.
Ground truth is my own reading, not outside adjudication.
Two small 4-bit models. Nothing here says anything about 7B+ models.
No seed control, so run-to-run flips include sampling noise.
One machine, CPU only.

Reproduce

python3 -m flipbench.runner --model gemma --runs 2 --out runs/gemma.jsonl
python3 -m flipbench.runner --model qwen  --runs 2 --out runs/qwen.jsonl
python3 -m flipbench.analyze runs/gemma.jsonl runs/qwen.jsonl

Roughly 8s per call for gemma, 6s for qwen, about 4 minutes per model here.

Code: https://github.com/rytjy/flipbench

A 1.5B model is a triage signal, not a verdict

rytjy — Sat, 24 Oct 2026 06:03:08 +0000

I run an auditor that reads trading-strategy repos while I sleep. A 1.5B model is part of it, and for a long time I described that part wrong.

The tool runs five deterministic checks over the source, then hands each finding to a local model with exactly six lines of context and one yes/no question: is this actually true here? Nothing leaves the machine. No API key, no upload, no account. The model is Qwen2.5-Coder-1.5B-Instruct Q4_K_M, loaded through llama.cpp, on a laptop CPU.

The interesting part is not that it works. It is where it stops working, and what I changed once I measured that.

The 48-second constraint

Four model checks on the bundled sample take about 48 to 50 seconds on this laptop. That is roughly twelve seconds per question. The deterministic pass, --no-llm, takes under a second.

Twelve seconds per question is the fact that shaped everything else. You cannot put that inside a loop over every file in a repository. You cannot put it behind a save button. What you can do is run it at night, or while you make coffee, and then read a report that is already sorted.

Latency also removes a whole class of bad ideas for free. I stopped considering "ask the model to look around and tell me what seems off" not because it is philosophically wrong but because it costs twelve seconds and returns a paragraph, and I do not want a paragraph at that price. I want a pointer.

One question, six lines, three answers

The model never sees the repository. It sees the finding's evidence line plus five lines around it, and it can answer YES, NO, or UNSURE. It is never asked to name a file, never asked for a line number, never asked for a summary.

That last restriction came from a failure. In an early build I asked it to summarise the file. The output was fluent, specific, and wrong about which function did what — the kind of answer that is expensive to read because you have to verify every clause. A checker that points at line 9 and a model that confirms or denies a claim about line 9 is a pipeline. A model that writes prose about code is a different product, and not one a 1.5B can deliver.

So the division is blunt:

Locations come from code, never from the model.
The model answers one narrow question per finding, and "UNSURE" is a real answer, not a failure.
The report prints the evidence line verbatim, so the reason to read a finding is never the model's prose.

The instability I could not engineer away

Here is the measurement that changed my design.

I asked Gemma 2 2B IT Q4_K_M (temperature 0.2) the same yes/no question about the same code, three times, in isolated contexts. The answers were NO, NO, YES.

Same weights, same prompt, same evidence line. Two of three said the cost bug was not there. It was there; the checker had already printed the ledger line with no fee term in it.

I also run both models and print their answers side by side, as a confidence signal. Two runs of the same audit gave 3 of 4 agreement and then 4 of 4. It would be tidy to call that sampling variance, but it is not: the code changed between the runs — I moved the context window to the evidence line — so the difference is confounded. I would rather write that down than round it into a nicer story.

Both runs are committed in the repository, and so are the three re-asks. That is the only claim I am willing to make: at 2B, on this task, the verdict is not a stable measurement. It is a hint about where to look, from a reader who has read six lines.

Once you accept that, the design stops fighting the model.

Five rules that follow

The checker owns the finding. Every path:line in the report comes from code that matched a pattern. The model cannot create a finding, move one, or delete one.
One question per finding. Narrow scope keeps a 1.5B inside the task. Broad scope turns it into a confident narrator.
The model is allowed to say nothing. "UNSURE" and "nothing to add" are valid outputs, and the report still finishes without them.
Agreement is a hint, not a vote. Two models agreeing means neither disputed the checker. It does not mean the finding is real, and the report never prints a confidence score I cannot compute.
Absence checks print what they searched. "No fee term in this file" is a weaker statement than "no match for fee, commission, slippage, or spread across 214 lines". For negative findings the evidence is the search, so the search has to be in the output.

The blind spot is written in fixed text, not generated: the tool does not run the strategy, does not see your data, and cannot tell you whether the edge is real. All it can do is point at the places where a backtest usually lies and ask a small model whether the pointer is correct.

What I have not measured yet

The honest gap is the threshold. Right now a finding either fires or it does not, and the model's answer is read by a human. If I ever wanted to automate the decision — skip the ones the model denies, say — I would need two numbers I do not have:

the margin around the cutoff, meaning the distribution of scores for true findings versus near-misses, not three anecdotes;
the flip rate under prompt shift, meaning the same question with a quoted line added or the field order changed.

Both are cheap to measure and both are missing from this post. That is the difference between a demo and something I would let run unattended on a repository I care about. I would rather say that here than discover it in production.

When to skip the model entirely

--no-llm runs the deterministic checks only, instantly, and it is the right mode more often than you would think:

when you need the same output twice, byte for byte;
when the repository is small enough that reading the report is faster than waiting;
when you are about to argue with someone about a number and cannot afford a witness who changes its mind.

The model is worth its twelve seconds when the finding is ambiguous and you want a second reader at 3am. It is not worth it when you already know what to look at.

Try it

pip install -e .
slothaudit audit samples/broken_lookahead            # ~50s, four model checks
slothaudit audit samples/broken_lookahead --compare   # both models, one report
slothaudit audit samples/broken_lookahead --no-llm    # deterministic only, instant
python3 -m slothaudit audit samples/broken_lookahead  # works without install

A clean repository gets a deliberately un-reassuring answer: no v0 check fired; that means this run had nothing to point at, not that the strategy survives an audit.

Code: github.com/rytjy/slothaudit

Disclosure: I used an AI assistant to help draft and edit this post. The measurements, the code, and the command output are mine, and the output above is pasted verbatim from runs on this machine.

slothaudit: a night-shift auditor for your backtest, on a 1.5B model in your own laptop

rytjy — Thu, 22 Oct 2026 10:35:36 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

slothaudit is a command-line auditor for trading-strategy repos that runs entirely offline. Point it at a folder; it runs five deterministic source checks, then a local 1.5B open-weight model reads the code each finding points at and answers one narrow question about it: is this finding actually true here?

You start it before you go to bed. In the morning you read six lines.

Why a night shift and not a chat window: the bugs that kill a backtest are boring and repetitive — a fill taken on the signal bar, a ledger that never mentions fees, one market event counted as eleven samples, a drawdown that resets its own peak. You find them when you already suspect them, which is exactly when you don't need help. So the boring pass runs while you're asleep, at sloth speed, on your own CPU.

Demo

Rendered from a real run's output on 2026-10-23 — CPU only, network interface idle.

$ slothaudit audit samples/broken_lookahead
# slothaudit report — samples/broken_lookahead
Generated 2026-10-23 12:30, offline.

## Morning summary

FIX FIRST: samples/broken_lookahead/strategy.py:10 — fill price read on the signal bar
ALSO: samples/broken_lookahead/strategy.py:12 — no fee or slippage term in the ledger
ALSO: samples/broken_lookahead/strategy.py:8 — entries without a cooldown

## Findings

### CO-01 [high] no fee or slippage term in the ledger
samples/broken_lookahead/strategy.py:12
This file records PnL but never mentions fee, commission, slippage or spread. Searched 24 lines: 0 matches for fee|commission|slippage|spread|cost.
Evidence:
    samples/broken_lookahead/strategy.py:12  pnl = (exit_ - entry) * 100
Model check: **YES**

A clean repo gets the other answer, and it is deliberately not reassuring:

$ slothaudit audit samples/clean_strategy
No v0 check fired. That is not the same as clean: it means this run had
nothing to point at, not that the strategy survives an audit.

Both runs: under a minute on a laptop CPU, no network.

Code

github.com/rytjy/slothaudit — Python standard library only, no dependencies beyond the local model binary.

The split of labour is the point:

Locations come from code, never from the model. The report prints path:line from the checker. The model is never asked to name a file or a line.
Every finding prints the line it matched. The checker already knows the exact source line, so the report shows it under Evidence:. For an absence check (no fee term, no cooldown) it prints the ledger line and the number of lines searched. The model's prose is never the reason to open a file.
The model answers one question per finding, with the surrounding six lines as context, and must start its reply with yes, no or unsure. That is a task a 1.5B model can do. Asking it to "write a summary" produced confident nonsense — so I stopped asking.
Every report ends with the blind spot, in fixed text: this tool does not run the strategy, does not see your data, and cannot tell you the edge is real.

How I Built It

Prior work, credited. The five checks are distilled from my own earlier kit, backtest-audit-kit, which documents the failing cases they came from. slothaudit is a new project written this week during the challenge; the checks were rewritten as source-level heuristics and the model layer is new.

Models: two open-weight GGUFs, both served by llama.cpp build b11804 as a subprocess — llama-cli in and out, one process per audit, nothing left running: Qwen2.5-Coder-1.5B-Instruct (1.1 GB) and Gemma 2 2B IT Q4_K_M (1.6 GB). --model qwen or --model gemma picks one; --compare asks both.
Checks: five regex/AST-level heuristics distilled from real audit cases: same-bar fills, missing cost terms, entries without a cooldown, peak resets in drawdown, and where(x > 0, 0) turning NaN into zero.
Everything else: Python 3.12 standard library. No key, no account, no telemetry, no upload.

Four decisions came from being honest about what a small local model can do:

Few-shot beat instruction. "Reply with VERDICT: YES | NO | UNSURE" made it echo the template back verbatim. A worked example about different code fixed it.
When the model has nothing to add, the tool says so. An empty summary is rendered as "deterministic findings only", not as a confident paragraph.
An unstable verdict is a finding, not noise. Asked the same yes/no question about the same code three times, the 2B answered NO, NO, YES (temperature 0.2). That is why every finding now prints the source line the check matched: the model's answer is a second opinion, the line is the evidence — and when the two disagree, the line is what you act on.
The slow path is the design. Four checks through the 1.5B took about 48 seconds on this laptop's CPU. That number is why the tool is built to run overnight and print six lines in the morning, instead of pretending to be a chat window.

The Lazy Payoff

An audit of the broken sample takes about a minute and produces four findings, one of which — the missing cost term — is the kind of thing I would otherwise notice three weeks later, after wondering why a strategy that "works" loses money.

The honest comparison is not a minute vs 20 minutes of expert review. It is a minute vs the zero minutes I actually spend looking, because after the fifth project nobody re-reads the fill logic. That is the effort it saves: the pass that never happened.

I ran it on both sample repos this week; the clean one fired nothing, which is the result that makes the other one trustworthy.

Prize Categories

Best Use of Gemma. slothaudit asks the same yes/no question of two open-weight models on the same laptop — Qwen2.5-Coder-1.5B-Instruct and Gemma 2 2B IT — both through llama.cpp, with no API key and no network. --compare prints both answers for each finding and marks whether they agreed:

Model check: qwen YES · gemma YES
Agreement: yes

That is a confidence signal, not proof: agreement means neither model disputed the checker's reading of those six lines; disagreement points you at the line to read yourself. Neither model is asked where the bug is, and neither writes the report — they only answer the one question the deterministic checker cannot answer alone.

On the bundled broken sample, two runs of the same audit gave 3 of 4 and then 4 of 4 agreement. The code changed between them, but asking Gemma the cost question again on its own returned NO in both variants — at 2B the verdict is not stable enough to be a metric. So the report leads with the checker's evidence line and treats agreement as a hint. Both runs are committed: 2026-10-22 and 2026-10-23.

Asked on its own on the same code, Gemma returned:

$ slothaudit audit samples/broken_lookahead --model gemma
### LA-01 [high] fill price read on the signal bar
Model check: gemma **YES** — the code fills on the bar that produced the signal.

### CO-01 [high] no fee or slippage term in the ledger
Model check: gemma **YES**

### SI-01 [medium] entries without a cooldown
Model check: gemma **YES**

### MD-01 [medium] peak reset inside the drawdown loop
Model check: gemma **YES** — this code calculates a running peak by adding the current pnl to the previous peak.

Same code, same question, different run: this one says YES on the cost finding, while the 10-22 run and one isolated re-ask said NO. That flip is why the report prints the evidence line instead of a confidence score.

Full report: docs/gemma_only.md.

Why Open Innovation Matters

Three concrete reasons, not slogans:

The code never leaves the machine. A trading-strategy repo is a person's edge, and its data includes fills and account sizes. Uploading that to a hosted API to get a lint report is a bad trade. Here the audit runs on the same laptop the repo lives on — I tested it with the network interface down.
It costs nothing per run. A nightly audit is 20 model calls a day. On a hosted API that is a meter running; locally it is the laptop you already own, asleep at 3 a.m.
The model is swappable. The interface is one function — chat(prompt, model) — with llama.cpp as the local path and an Ollama HTTP server as the fallback. Qwen 1.5B and Gemma 2B are the same three lines of wiring. When a better small model appears, I drop in a GGUF. No re-platforming, no migration, no vendor.

The 1.5B is not a placeholder for a bigger model I wish I could afford. It is a deliberate choice: the checks find the problem, the model only has to answer "is this true here?" — and a small model that answers a narrow question honestly beats a large one that writes a beautiful paragraph I cannot verify.

Built during Week 3. The repo includes the two sample strategies used above, and the audit output is reproducible byte-for-byte on any machine with the same GGUF.

AI disclosure: the code and the draft of this post were produced with an AI coding assistant (OpenClaw), working from my direction. Every number here — the timings, the verdicts, the two-model comparison — comes from runs committed in the repo, and the demo GIF is rendered from real output, not staged.

Four ways your backtest lies — and a dependency-free Python kit that catches them

rytjy — Fri, 16 Oct 2026 15:03:28 +0000

My strategy looked fine in the backtest and lost money live.\n\nIf you've done this for a while, you already know the suspects. What surprised me is how mechanical they are — each one is a specific, checkable defect, not a vague "the market changed":\n\n- Look-ahead — signals silently shifted onto future bars.\n- A ledger that doesn't balance — cash/position conservation breaks, and the PnL is fiction.\n- Fees and funding counted zero times (or twice) — the cost model quietly drops out.\n- Exit logic that scans the whole window instead of the current bar.\n\nI packaged those four checks into a small, dependency-free Python toolkit (no numpy, no pandas):\n\n

\npip install backtest-audit-kit\n

\n\n- a look-ahead detector\n- ledger invariants (cash/position conservation)\n- a fee / funding double-count detector\n- mutation testing, to find blind spots in your own test suite\n\n*A worked example.* cases/ ships a real BTC backtest whose cost model was zero — reported PnL came out at roughly 5.7x the principal. But the structural problem was worse than the bug: the take-profit needed +0.08% while a round-trip taker fee is 0.10%, so every winning trade net-loses by construction. No amount of parameter tuning fixes that.\n\nRepo: https://github.com/rytjy/backtest-audit-kit\n\n**What check would you add?** I'm collecting the ones I haven't automated yet.