DEV Community: Tomozo / HAWK Backtester

I let Claude Code search for a trading strategy, but hid the last two years of data from it

Tomozo / HAWK Backtester — Thu, 22 Oct 2026 13:45:02 +0000

I gave Claude Code ten years of USD/JPY daily bars and asked it to find a trading rule. It ran backtests through an API, as many as it wanted up to 30, but the last 20% of the data was hidden from it the whole time. At the end it got exactly one look at the hidden part, and it had to write down what it expected before it looked.

The point wasn't to find a strategy. It was to see what happens to "the best backtest an AI found" when the AI can't quietly try its way into it.

The setup

Data: USD/JPY daily, April 2015 to April 2025, 2,604 bars. The last 520 bars (from 2023-04-26) were cut off on the server, so the agent's runs never received those prices.
Agent: Claude Opus 5.5 in Claude Code, 2026-10-19, working through Hawk Backtester's Agent API. It wrote each strategy as a Python file and ran it with a small script.
Rules: at most 30 search runs. After each run, a note saying what it changed and why. When done, pick one strategy, write a prediction for the hidden period, then run it once on the hidden period.
The ledger: every run goes into one study. Each result comes back with the trial number and a deflated Sharpe ratio: the probability that the run beats the best you'd expect from that many tries of pure noise.

What it tried

It used 13 of its 30 runs and then stopped on its own: "further searching would mostly have added selection bias."

#	Rule	Visible return	Max DD	Trades	Deflated Sharpe
1	50-day breakout, fixed stop and target, 1,000 units	−53.9%	−55%	3	0.13
2	same, sized to risk 2% per trade	+14.6%	−16.2%	35	0.06
3	50-day breakout, 3-yen trailing stop, no target	+12.0%	−7.5%	42	0.003
4	trailing stop scaled to recent volatility	−7.3%	−21.4%	75	0.0
5	#3 plus a 200-day trend filter	+6.2%	−9.5%	37	0.002
6	6-month momentum, always in the market	+14.6%	−21.0%	62	0.0
7	hold when 3 of 4 lookbacks agree	−2.4%	−26.2%	123	0.0
8	12-month momentum, checked monthly	+14.4%	−15.7%	16	0.04
9	buy 5-day lows above the 200-day average	−5.5%	−17.1%	119	0.0
10–13	#3 with a 100-day / 20-day lookback, 2-yen / 4-yen trail	−3.3% to +9.9%		32–73	≤ 0.007

Run 1 was a sizing bug, not a test of the signal, and the agent said so in its next note. Runs 10 to 13 weren't new ideas; it was checking whether #3 sat on a plateau or a spike. The lookback held up (20, 50 and 100 days were all positive). The trailing-stop distance didn't (2 yen lost money, 4 yen made a third as much), and it noted that "3 yen is partly a lucky value."

Here is the dashboard's research note at the end. The labels are Japanese: hidden period 20% (520 bars), hidden period used once, 13 trials, and the run list with each trial's deflated Sharpe in the last column.

What the ledger said

Every search run came back "within chance, given the number of trials." The luck threshold (the per-trade Sharpe you'd expect from the best of N noise strategies) jumped to about 0.5 by the third trial and stayed between 0.47 and 0.53. The pick, #3, had a per-trade Sharpe of 0.127 and a deflated Sharpe of 0.003.

The agent agreed with it. From its notes before the final run:

Ledger says chance, and I agree: I expect roughly break-even, not a confirmation.

The prediction, then the hidden period

Written before the run: about +1% over the hidden two years (range −6% to +8%), max drawdown about 6%, about 10 trades, shorts losing money again.

What happened:

Equity over the hidden period: +1.2% (11,201 at the cut to 11,334), max drawdown −7.4%.
Trades entered in the hidden period: 24, net +245, profit factor 1.31, win rate 46%, per-trade Sharpe 0.065 (about half the visible 0.127).
Trade count: missed. It expected about 10 and got 24. The hidden period had far more breakouts and stop-outs than the visible average.

Its own summary:

The pick didn't fall apart, but it didn't prove anything either. It earned a small positive return in the hidden period, per trade about half as good as on the visible data. That fits a weak edge, or no edge plus some luck. I would not call this a working strategy.

What I take from it

The interesting part isn't the strategy. It's that the agent didn't oversell it. Without the ledger, the story writes itself: "AI finds a USD/JPY breakout system, +12% with a 7.5% drawdown." With the ledger in every response, the agent kept reading "within chance," stopped searching early, ran sensitivity checks instead of new ideas, and predicted the hidden period close to what happened.

Two things I changed after watching it:

The holdout response mixed the periods. A holdout run reported a total return of +13.3% because the outcome covered the whole run. The agent noticed and worked out the hidden part itself. Holdout runs now also return research.hidden_period with the equity change and drawdown for the hidden period alone.
Agent Mode was broken in production before this run (the browser runner rejected every session with "runId is missing"), and a failure in the browser came back to the agent as an empty success. Both are fixed: a browser-side failure now comes back as failed with the message, and a run no browser picks up says so after 30 seconds.

Limits

One dataset, one period, one model. Daily bars, no swap or carry costs.
The model's training data includes general knowledge of USD/JPY history up to 2025. It said so in its summary, and its picks followed the visible runs, but I can't rule out that it helped.
A 2-year holdout can't confirm a weak edge. All this shows is that this pick wasn't an overfit illusion that collapsed.

The ledger, the hidden period and the single holdout are part of Hawk Backtester's Agent Mode. Strategies are plain Python files that run on your machine (pip install hawk-bt).

If you let agents run backtests, what do you use to keep them from picking the lucky run?

A reader said my "exit problem" detector could be fooled by noise. I added a random-entry control

Tomozo / HAWK Backtester — Mon, 19 Oct 2026 13:50:05 +0000

My MT5 analyser splits losing trades into two kinds: the ones that were in profit first (an exit problem) and the ones that went wrong right away (an entry problem). A reader, @arhancanli, left a comment on an earlier post that took that split apart in two sentences:

With a stop of 1% and noisy bars, a random entry will often show a small positive MFE before it is stopped out, so "most losers were once in profit" can come out high even when the entry has no edge.

They suggested two fixes: measure MFE as a multiple of the stop distance, and compare against the same exits applied to random entry times. If the real losers don't beat the random ones, it isn't an exit problem.

That was right, so I built it. This is what came out, including the part where I got it wrong first.

The control

For each losing trade, the tool replays 100 entries at random bar opens, with:

the same side (buy or sell),
the same loss distance, used as a stop,
the same holding time.

When one bar reaches both the stop and a new high, the stop is taken first, so random entries aren't flattered. Real trades are measured without the range of the bar they opened in (the fill was somewhere inside it), so the random ones are measured the same way.

The random trades that end in a loss are then classified with the same rule as the real ones: "in profit first" if the open profit reached at least half the loss distance, "wrong right away" if it stayed under 15% of it. Then a one-sided test (about 95%) asks whether the real share is higher than the random share.

If it isn't, the tool no longer says "exit problem" or "entry problem". It says the share is no higher than random entries give, and drops the suggestion that goes with it.

The part I got wrong

My first version measured each random trade's open profit against its own loss. A random trade that timed out down 2 pips with 3 pips of open profit along the way counted as "in profit first". On a test with flat, quiet bars, 98% of random losers came out as exit problems.

On the bundled sample, that version said the exit finding was noise too: 34% for the real losers, 43% for random entries.

The fix is what the comment actually said: measure against the real loser's loss distance, like a multiple of R. A random trade only counts as "in profit first" if it went as far your way as the real trade's loss was.

What the sample says now

The sample is a breakout EA on synthetic EURUSD M15 data, 136 trades, 90 losers.

Kind of loser	This EA	Random entries
In profit first	34%	25%
Went wrong right away	33%	39%
Went wrong right away (3 pips cost)	44%	52%

The exit finding holds (z ≈ 1.96). The entry finding doesn't: random entries go wrong right away at least as often as this EA does.

That second line changed something I had published. The guide on reading MT5 reports said that once you subtract a 3-pip cost, this EA's losses "lean towards the entry" (40 entry problems against 22 exit problems). Without a control, that sounded fair. With one, 44% against 52% for random entries says the opposite: the entries are no worse than random. The cost moves both groups the same way. I took the sentence out.

Other things the comments changed

The same reader left a second comment on the post about re-running on save, and a few smaller things came out of both:

Same-bar ties. When a bar touches both the target and the stop, the engine assumes the stop came first. The exit analysis now counts those trades and shows how much profit would change if the target had come first. On the demo's hourly bars with 1% levels the count is 0, since an hourly bar there moves far less than the 2% between the levels. On tighter levels it won't be.
Size-only changes in the diff. The strategy in that post sizes from equity, so one changed exit resizes every later trade. The diff only counted a trade as "changed" when its exit moved, so those weren't in the count, but the PnL line mixed them in. They now have their own line: same bars, size changed, with their PnL.
Heatmaps say how many cells the best one was picked from. More cells means a higher best cell by luck alone.

Limits of the control

Random entries are drawn from the whole test period. If your EA only trades London open, the control doesn't match that. A control matched on time of day is the obvious next step.
With 90 losers the test is rough. It can tell "clearly more than random" from "not clearly", but a share 5 points above random won't pass.
It says whether the split is real, not what to change. A real exit problem still needs a tested change to the exit, on a different period than the one you found it on.

The analyser is free, runs in the browser, and the report never leaves your machine:

https://app.hawk-backtester.com/en/tools/mt5-report/?utm_source=devto&utm_medium=article&utm_campaign=control

If you test EAs, I'd like to know whether a time-of-day matched control would change your numbers, and what else you would compare against.

I built a backtester that shows why a trading rule lost, not just how much

Tomozo / HAWK Backtester — Tue, 13 Oct 2026 13:40:02 +0000

Most backtests end with one number. Total return, maybe a drawdown and a win rate. I kept getting stuck at that point: the number told me a rule was bad, but not which part of it to change.

So I built Hawk Backtester around a different question. Not "how much did this rule make", but "why did it win, and why did it lose". This is the first time I'm showing it outside Japan, so here is the whole thing, including what it can't do yet.

Same total, opposite fixes

Two strategies can both end at -2.8% and need completely different changes.

In one, most losing trades were in profit at some point, then reversed and hit the stop. That's an exit problem.
In the other, most losers were never in profit at all. That's an entry problem, often entering in the wrong kind of market.

If you only look at the total, you tweak the take profit in both cases, and in one of them you're fixing the part that wasn't broken. Everything in the app is built to make that split visible.

Try it without signing up

The demo runs in your browser with no account. The engine is C++ compiled to WebAssembly, and a 1,500-bar run takes about 15 milliseconds, so it re-runs every time you move a slider.

It uses synthetic USD/JPY-like hourly data where trends, ranges and shocks take turns. Pick a market type and a built-in strategy (MA cross, Donchian breakout, Bollinger, RSI) and move the sliders.

https://app.hawk-backtester.com/en/demo/?utm_source=devto&utm_medium=article&utm_campaign=showdev

What it tells you about a run

Which markets it won and lost in. Trades are grouped by the market regime at entry. The same breakout rule made about +10% in trending data and lost about 6% in ranging data. The total hid that completely.

Whether the entry or the exit is the problem. Every trade records its best open profit (MFE), its worst open loss (MAE) and why it closed: take profit, stop loss or the opposite signal. A few hints are generated from those trades, and clicking one filters the trade list down to the trades behind it.

Whether a change really helped. Re-run after a change and the app shows which trades changed compared to the previous run. When I moved a take profit from 1% to 2%, the trade count stayed at 60, but 20 trades ended differently and about half of those got worse. The total said "slightly better".

Whether a parameter is just luck. Sweep two parameters and you get a heatmap. The average of the best cell's neighbours is shown next to it. If the neighbours are good too, it's a plateau. If they aren't, the best cell was probably noise.

There's also a button that sends the aggregated numbers (not your code, not raw data) to an LLM and gets back what to look at and what to test next. It's told not to give trading advice.

Your own strategy, in Python

pip install -U hawk-bt
hawk-bt run my_strategy.py

from hawk_bt import Strategy, Context, Param

class MaCross(Strategy):
    fast = Param(10, min=3, max=50, step=1)
    slow = Param(50, min=10, max=200, step=5)

    async def step(self, ctx: Context) -> None:
        ...  # read ctx.state, call ctx.engine.place_ticket(...)

Your Python process talks to the browser over a localhost websocket, so the strategy code never leaves your machine. Every Param shows up as a slider, and saving the file re-runs the backtest and shows the diff against the previous run.

How fills work

I found four fill-model bugs in my own engine. Fixing them moved results by up to 4% and changed which strategy ranked first, so I write the assumptions down now:

TP and SL fill at their level, or at the open if the bar gaps through
if one bar touches both, the stop is assumed to come first
spread, fees and slippage are charged on every fill
you can enter at the signal bar's close or the next bar's open

https://app.hawk-backtester.com/en/simulation-model/?utm_source=devto&utm_medium=article&utm_campaign=showdev

Data

No market data is bundled. You can upload a CSV, generate synthetic data, or import bars with your own data-provider API key (Twelve Data, for example). The key is stored encrypted and only used for your imports.

What it can't do yet

one instrument per backtest, no portfolios
bar data only, no ticks
the app is built for desktop; on a phone only the demo really works

Pricing

The free plan has unlimited backtests (built-in strategies, Python and watch mode), 5 saved results, sweeps up to 25 runs and 3 AI summaries a month. Paid plans raise those limits.

I also wrote up the ideas behind each view: MFE and MAE, splitting results by regime and avoiding overfitting.

https://app.hawk-backtester.com/en/?utm_source=devto&utm_medium=article&utm_campaign=showdev

Almost nobody uses this yet. If you backtest in Python today, what do you look at after the total return, and what would you need to see before trying this on a real strategy?

Is your MT5 EA losing on its entries or its exits? A free report analyser

Tomozo / HAWK Backtester — Sun, 11 Oct 2026 23:33:18 +0000

The MT5 Strategy Tester gives you a lot of numbers: profit factor, drawdown, win rate, consecutive losses. What it doesn't tell you is which part of the EA to change. A profit factor of 0.58 says the EA is losing. It doesn't say whether the entries are wrong or the exits are.

So I made a small free tool that reads the Strategy Tester's HTML report and answers that question. It runs in the browser, needs no account, and the report never leaves your machine.

https://app.hawk-backtester.com/en/tools/mt5-report/?utm_source=devto&utm_medium=article&utm_campaign=mt5

Two kinds of losing trade

Every losing trade belongs somewhere on a line between two cases.

It was in profit first. The price moved your way, then came back and hit the stop. The entry was fine; the exit gave it away. Work on the take profit, a trailing stop or moving the stop to break even.
It went wrong right away. The price never moved your way. No exit rule would have saved it. Work on the entry: the signal itself, or the market conditions you enter in.

To tell them apart you need each trade's best open profit (MFE) and worst open loss (MAE). The MT5 report doesn't have those, so the tool asks for one more file: a bar CSV exported from MT5 for the same symbol and period. With the bars it walks every trade from entry to exit and records how far it went each way.

On the sample report (a breakout EA on synthetic EURUSD M15 data), 90 of 136 trades lost:

31 had been in profit first, by 0.00204 on average, before losing. Exit problem.
30 were barely ever in profit. Entry problem.
29 were somewhere in between.

That is an even split, which is itself useful: changing only the take profit would address a third of the losses at most.

Is the stop wider than it needs to be?

The tool also looks at the winners. In the sample, 90% of winning trades never went more than 0.00218 against you, and the median stop was 0.00250, so the stop is not wider than winners needed. If it were 0.00500, you would be paying for room no winner used.

This check can't tell you a stop is too tight, though. A trade that needed more room was stopped out and counts as a loser, so it never shows up among the winners. (Updated after a reader pointed this out in the comments.)

Where it makes and loses money

With the bars, each trade also gets the market regime just before entry: uptrend, downtrend or range, from ADX(14) on hourly bars. In the sample, ranging markets accounted for 93 trades and -685 of the -965 total. For a breakout EA, that points to a filter rather than a new exit rule.

There are also tables by exit reason, holding time and hour of day, and every finding links to the trades behind it.

How it works

The report is UTF-16 HTML with labels in whatever language your terminal uses. The parser finds the deals table by its shape (13 columns, buy/sell and in/out values) rather than by header names, so a Japanese or German report works the same as an English one.
Deals are rebuilt into trades first in, first out, so partial closes and reversals become separate trades.
The order of prices inside a bar is unknown, so MFE and MAE use the highs and lows of bars that lie entirely between entry and exit. A trade opened mid-bar would otherwise be credited with moves from before it existed, so for the entry and exit bars only the fill prices count. That errs low; M1 or M5 bars recover more of the path. (Also changed after the same comment.)
Everything runs in plain JavaScript in the page. Nothing is uploaded unless you sign in and choose to save.

Comparing two versions of an EA

If you sign in (free), you can save reports and compare two of them. Trades are matched by side and entry time, so you can see which trades a change actually affected: which ones it saved, which ones it broke, and which were new. When you tweak an EA, the total usually moves a little; the per-trade view shows whether that came from the change you meant to make.

Limits

MT5 Strategy Tester HTML reports only. MT4 is not supported yet.
Without the bar CSV you get the basic statistics and exit reasons, but not MFE/MAE or regimes.

If you build EAs, I would like to know whether this matches how you debug them, and whether MT4 support would matter to you.

This tool is part of Hawk Backtester, a backtester built around the question of why a rule won or lost. The ideas behind the split are in the guide on MFE and MAE.

My backtest re-runs when I save the strategy file, and shows which trades changed

Tomozo / HAWK Backtester — Sun, 11 Oct 2026 04:03:41 +0000

I'm building a backtesting web app called Hawk Backtester on the side. This post is about one feature, a watch mode for local Python strategies.

The loop that annoyed me most was edit the strategy, run it in a terminal, open the results, then try to remember what the previous run looked like. The last step was the worst. If PnL went up a little, I couldn't tell whether my change did it or a couple of trades just happened to land differently.

So now the backtest runs again the moment I save, and it tells me which trades changed.

Using it

On your machine:

pip install -U hawk-bt
hawk-bt run my_strategy.py

A strategy looks like this. Anything declared with Param shows up in the browser as a slider.

from hawk_bt import Strategy, Context, Param

class MaCross(Strategy):
    fast = Param(10, min=3, max=50, step=1)
    slow = Param(50, min=10, max=200, step=5)

    async def step(self, ctx: Context) -> None:
        snap = ctx.state.snapshot
        close = ctx.state.candles.close[: int(snap.step)]  # last element is the current bar
        if len(close) < self.slow + 1:
            return
        fast, slow = close[-self.fast:].mean(), close[-self.slow:].mean()
        prev_fast, prev_slow = close[-self.fast - 1:-1].mean(), close[-self.slow - 1:-1].mean()
        if prev_fast <= prev_slow and fast > slow:
            side = "buy"
        elif prev_fast >= prev_slow and fast < slow:
            side = "sell"
        else:
            return
        price = float(close[-1])
        await ctx.engine.place_ticket(side=side, units=int(snap.equity / price),
                                      take_profit=price * 0.01, stop_loss=price * 0.005)

In the browser you pick a dataset and press "Connect to Local Strategy" once. After that you just go back to your editor and save. Moving a slider also re-runs your local Python with the new value.

The diff sits above the results. When I changed the take profit from 1% to 2% and saved, it said: trades 60→60, added 0, removed 0, changed 20, PnL +406.55. Same number of trades, but 20 of them ended differently, and about half of those 20 actually got worse. If I'd only looked at the total I would have called it "a bit better" and moved on.

How it's put together

The execution engine is C++ compiled to WebAssembly and runs in the browser. Your strategy runs in your own Python process and talks to the browser over a localhost websocket, so the strategy code is never uploaded. I decided that early, because I wouldn't want to upload my own strategies to somebody's server either.

The watch part is simple. hawk-bt run polls the file's mtime, reloads the module when it changes, and asks the browser to start a new run. On each run, Python sends the strategy name and its Param definitions, and the browser answers with the current slider values. For the diff, trades from the previous run and the new run are matched on entry time and side.

One bug took me a while. The browser sometimes received the trade log for the same run twice, treated the second copy as "the next run", and every diff came out as zero. Tagging each run with an ID and ignoring repeats fixed it.

Two other things in the same screen

You can pick two parameters, sweep the grid, and look at it as a heatmap. The single best cell is easy to overfit, so the average of its neighbours is shown next to it.

In this example the best cell is +17.01%, but its neighbours average +9.31%. The left column (10-bar lookback) is around +15% for every take profit, so the lookback matters much more than the TP here.

There's also a button that sends the aggregated numbers to an LLM and gets back what to look at and what to try next. Only the aggregates go out, not your code or raw data, and it's told not to give trading advice.

In the run above it picked up the diff too, and its first suggested experiment was to work out why 20 trades changed before comparing the two versions.

Limits

One instrument per backtest. Bar data only, no ticks. When TP and SL are both touched inside one bar it assumes the stop came first. Portfolio support is what I'm working on next.

Trying it

Honestly almost nobody uses this yet, so even a short "I tried it and X was confusing" helps a lot. There's a demo that needs no account (synthetic data).

What would you want the diff to show that it doesn't yet?