DEV Community

stooq1
stooq1

Posted on

I wrote down the pass mark before the test. Then I failed it.

A trend rule, two markets, one day of work, and three moments where my own checking was the thing that was broken. Code and data: trend_daily.py, strategy-graveyard.

The setup

The rule is the most boring one in the book. On daily bars, take the sign of the return over the last N bars, size the position so that each instrument contributes the same volatility, cap the leverage, rebalance weekly or when the sign flips, charge costs per side and pay funding on perpetuals. The version I declared before any of this ran was an ensemble of three windows: 60, 120 and 250 days.

I had already tested it on Binance perpetuals and on three Russian futures families, written the result down as "a positive-expectancy crash hedge, not income", and moved on. Then I went back and finished the job on the remaining six futures families. What follows is one day of work, in order, including the parts where I was wrong.

Wrong the first time: a prediction I had written in my own journal

The journal said, in plain words: the other six families will not change the picture, they will only add diversification of the same shape. That was a real prediction about a test I had not run, which is the only kind worth grading.

Nine families instead of three took the portfolio from Sharpe 0.56 at t 1.5 to 0.83 at t 2.3, and cut the maximum drawdown from 19% to 12%. Better still, the six families the rule had never seen, taken alone, gave 0.88 at t 2.4 — cleanly out of sample, and better than the three the rule had been developed on. The usual direction of travel is the opposite.

Wrong the second time: I was cutting the best days from one side only

The project has a standing rule that every positive result gets decomposed by its top five days, because an edge that is really three good days is exposure, not edge. The daily trend had 49% of its total in five days, all of them in early 2022, and I had filed that as a mark against it.

Then I cut the best days from the benchmark too, which is the obvious thing to do and which I had not done.

full minus top 5 minus top 10 minus top 20
trend ensemble 0.88 0.55 0.40 0.13
vol-scaled long 0.35 −0.03 −0.18 −0.45

Removing five days out of 1,741 takes the benchmark from +23.5% to −1.9%. Concentration was a property of the market, not a defect of the rule, and the rule was in fact less concentrated than buy-and-hold. My check had been asymmetric, and asymmetric checks always flatter whichever side you forget to cut.

Wrong the third time: a column in my own script was lying

Next I mapped the rule over a grid of windows from 30 to 300 days, and then stress-tested costs. The first cost run came back with the ensemble at Sharpe 1.28 where the baseline run had said 0.88 — at higher costs. A number that improves when you make the world worse is not a discovery, it is a bug.

It was a bug. The ensemble column in my script was labelled "ensemble 60/120/250" as a hardcoded string, while the ensemble it actually computed used whatever windows the --lookbacks flag was given. I had passed six windows and read the result as if it were three. One line to fix, and worth more as a habit than as a fix: when a number is suddenly better than the baseline, check that it is the same quantity before you check anything else.

The result that looked very good

With those three corrections in place, the picture on Russian futures was strong.

The window map was a plateau, not a spike: every window from 30 to 200 days was positive, and only the annual windows died. Costs were a non-issue — you have to multiply the real 0.3–1.0 basis points per side by about fifty before the rule falls to the level of just being long. And the single best window, 30 days, held up on every cut I could think of: Sharpe 1.29, and 1.34 at t 3.5 with February and March 2022 removed entirely, which is to say it was not a crash hedge after all. Removing its own top 20 days still left it positive while the benchmark was deeply negative.

t 3.5 clears the bar I set for this project at the start. Nothing else in four months had cleared it.

It is also exactly the moment to be suspicious, because I had found that window by scanning a grid after the fact, and the six "new" families had taken part in the scan, so they were no longer clean for this particular claim.

So I wrote the threshold down first

Before running anything on crypto, into the journal:

Single window N=30, same vol targeting, same weekly rebalance, 7.5 bp per side, funding charged. If the shape from futures is real, N=30 gives at least 0.7 on the sixteen Binance perpetuals and at least 0.9 on the top four by liquidity with the longs held in spot, and beats the three-window ensemble on the same samples. Below 0.5 and N=30 is written off as fitted to one market.

Then I ran it. 0.59 and 0.91 on the full samples; 0.55 and 0.73 compared on the same days as the benchmark. One threshold met on the headline number and missed once the samples were aligned, one threshold missed outright. N=30 did beat the ensemble on all four slices, so the direction survived. The magnitude did not: the same window that gave 1.26 to 1.34 on futures gives 0.55 to 0.73 here, and on the sixteen perpetuals it fails to beat a vol-scaled long position.

The test that settled it

Levels are noisy. Shape is harder to fake, so I mapped the same nine windows on crypto and compared the shapes by rank correlation.

sample 30 45 60 90 120 160 200 250 300 long
16 perps 0.59 0.37 0.55 0.30 0.55 0.01 0.32 0.38 0.59 0.70
top-4 + spot 0.75 0.36 0.59 0.24 0.70 0.37 0.48 0.56 0.64 0.63
futures, all 9 1.29 1.18 0.95 0.84 0.94 1.08 0.65 0.16 −0.19 0.21

Rank correlation of each shape with the futures shape: +0.90 for the six new futures families, −0.12 and +0.03 for the two crypto samples. Average of the short windows minus average of the long ones: +0.86 on futures, +0.02 and −0.07 on crypto. On the sixteen perpetuals a vol-scaled long beats all nine windows in a row.

So the gradient replicates across two samples of one market and does not exist on the other market at all. Which means the 0.91 that squeaked past my threshold was the best draw from a flat surface — and first place out of nine on flat ground is worth very little, particularly when you arrive already knowing which window to hope for.

What I actually believe now

Not "a trend premium exists". The honest sentence is: this rule worked on one market in one regime. On Russian futures from 2019 to 2026 it is strong and survives every cut I know how to make. On crypto — the only venue I can actually trade — what remains is alpha of 0.58 to 0.78 against a vol-scaled long at roughly zero correlation. That is a real diversifier to a long position and it is not an income: about 7% a year at 10% volatility, which on a ten-thousand-dollar account is a few hundred dollars and a 13% drawdown.

The most plausible explanation for the gap is unglamorous. Russian futures spent 2022 to 2026 in a long decline, where the short side pays; crypto spent the same years going up.

What I would ask of any daily-trend backtest

Say which period you used and admit what it was. Include funding. Fix the instrument list before the run, or set it by a rule that cannot see the outcome. Cut the top days from the benchmark as well as from the strategy. Map the parameter instead of reporting one value, and check whether the shape replicates rather than whether the peak does. And when you go to a second market, write the pass mark down before you look — not because it makes you right, but because it makes you unable to negotiate with yourself afterwards.

That last one is the only reason this write-up says what it says. I had a t of 3.5 and every incentive to believe it.

Full journal, code and all the numbers above: github.com/stooq1/strategy-graveyard

Top comments (0)