HFM's exit simulator evaluated 12,720 configurations over 14,399 qualifying events across 15 crypto symbols. One selected configuration reported an average of +25.98 basis points per event after the simulator's 10 bps fee deduction. That is about 0.26% per event, enough to warrant a closer look.
Removing one session changed the average to −3.69 bps. The exit search had also used the validation split. The archive contained summaries rather than the joint outcome record needed to test the whole search.
The configuration behind +25.98 bps
The audited rule used a momentum exit: a 50 bps initial stop, a 60-second momentum window, a 30 bps reversal threshold, and a maximum hold of 600 seconds. The backtest covered September 2025 through September 2026. These are simulated outcomes with modeled fills and fees, not live-trading returns:
| Metric | Audited configuration |
|---|---|
| Qualifying events | 14,399 across 15 symbols |
| Mean net outcome per event | +25.98 bps |
| Median net outcome per event | −16.2 bps |
| Win rate | 34% |
| Mean net outcome excluding 2025-10-10 | −3.69 bps |
The median event lost money, while a small part of the sample pulled the mean positive. The separate day-block audit treats the market session as the relevant unit: removing 10 October 2025 removed the positive event-level average. This is a sensitivity diagnostic. It does not justify discarding an inconvenient day and calling the remainder an unbiased estimate.
These are statistics for the selected rule's events. They are not the best and median scores across all 12,720 configurations; the archived material available for this article does not establish that distribution.
12,720 rules used the same history
The grid contained 240 fixed-stop configurations, 8,640 trailing stops, 1,680 break-even-plus-trail rules, and 2,160 momentum exits. Changing a stop or trailing distance did not produce an independent sample: every configuration was evaluated on the same history.
A test of one rule specified in advance asks a different question from a test of a rule chosen after inspecting thousands. For the search, the null is that none of the candidates beats a specified benchmark. The Reality Check tests that family-level claim; testing only the selected column leaves out how it was selected.
The trailing-stop family alone contributed 8,640 configurations. Nearby settings can share trades and market shocks.
# A significance threshold, not an observed p-value.
alpha = 0.05
n_rules = 12_720
per_rule_alpha = alpha / n_rules
print(f"{per_rule_alpha:.2e}")
# 3.93e-06
The 3.93×10⁻⁶ value is the per-test significance threshold for a 5% Bonferroni family-wise error bound, not a p-value measured in HFM. The bound does not require independent rules, but each input p-value must already handle dependence in the observations. Dividing alpha by 12,720 cannot repair a test that treats events from the same market session as independent.
VAL had already entered the search
The event-score threshold came from development scores. That is a normal fitting step, but results on those development events are in-sample. The exit-rule search then used both DEV and VAL, including the September 2026 validation period. There was no remaining untouched split for the selected configuration.
Reusing VAL is a separate problem from the number of exit rules. A Reality Check can address a documented candidate search under its assumptions without requiring a separate holdout. It does not make VAL independent again or automatically account for earlier decisions about event filters, models, and which experiments to pursue.
| Split | How it was used |
|---|---|
| DEV | Fit the event-score threshold; included in the exit search |
| VAL, September 2026 | Included in exit selection |
| Untouched holdout | None remained for final evaluation |
Where the simulator lost the audit trail
In exit_sim3.py, every candidate produces an array of event outcomes. The grid's add() helper sends that array to stats() and keeps a dictionary of means, quantiles, win rate, and parameters. It does not keep the underlying array. Earlier, load_pass_paths() has already dropped dates and symbols when it returns the numeric paths.
We therefore could not tell from the summaries which rules made money on the same dates. Two rules can have similar means and standard deviations but very different joint behavior. A bootstrap of the maximum needs that joint record, with every candidate carried through the same resampled time blocks.
Without the retained rule-level outcome matrix, we could not reproduce a search-aware test from the archived results alone. The source code can recompute outcomes from events, model artifacts, and prices; an instrumented rerun could restore the record if the original inputs and versions are recoverable. That reconstruction has not been performed here.
A proposed output contract for the rerun
I would retain one row per event and candidate before computing summaries. The following is a proposed schema, not a change already implemented in HFM:
run_id, event_id, detected_at_utc, symbol, candidate_id,
candidate_net_bps, benchmark_net_bps
# run manifest:
# code revision + input hashes + model/threshold artifacts
# complete grid + benchmark policy + fees/fills + ranking objective
# expected calendar + missing-input checks + split-use history
For a concrete benchmark, this proposed rerun would use a fixed exit 600 seconds after the same entry, with no optimized stop or target. Both policies would use the same event cohort, price-path construction, and 10 bps fee deduction. The benchmark must be fixed before the rerun; it was not a documented comparison behind the original +25.98 bps figure.
The objective here is mean excess bps per event, not portfolio return. Each event has equal weight; overlapping signals are evaluated as separate hypothetical trades, without pretending that each can receive the same live capital. For every calendar date and candidate, save the sum of event-level excess outcomes and the event count. Dates without signals contribute zero to both; missing market data must be flagged rather than treated as a no-signal day.
Saving the sum and count preserves the event-weighted objective when days contain different numbers of signals. The bootstrap sketch below centers each candidate under a zero-mean excess-outcome null, then resamples whole date blocks jointly. Resampling the raw positive-mean outcomes would retain the observed advantage instead of imposing that null; see the least-favorable-null discussion in the INE review of predictive-ability tests.
import numpy as np
# Proposed event-mean test; not an HFM test result.
# daily_sum: dates x candidates, summed excess bps for each date.
# daily_count: events per date, shared cohort across all candidates.
s = np.asarray(daily_sum, dtype=float)
n = np.asarray(daily_count, dtype=float)
assert s.ndim == 2 and n.shape == (s.shape[0],)
assert s.shape[0] > 1 and s.shape[1] > 0
assert np.isfinite(s).all() and np.isfinite(n).all()
assert (n >= 0).all() and n.sum() > 0
assert (s[n == 0] == 0).all()
mean_excess = s.sum(axis=0) / n.sum()
observed = mean_excess.max()
centered = s - n[:, None] * mean_excess # zero excess per event
rng = np.random.default_rng(42)
B, block_len = 2_000, 7 # illustrative, not calibrated for HFM
n_days = len(n)
n_blocks = (n_days + block_len - 1) // block_len
null_max = []
while len(null_max) < B:
starts = rng.integers(0, n_days, size=n_blocks)
idx = ((starts[:, None] + np.arange(block_len)) % n_days)
idx = idx.ravel()[:n_days]
if n[idx].sum() == 0:
continue
null_max.append((centered[idx].sum(axis=0) / n[idx].sum()).max())
null_max = np.asarray(null_max)
p_hat = (1 + np.count_nonzero(null_max >= observed)) / (B + 1)
The code is an unstudentized, Reality-Check-style illustration for the proposed event-level objective, not a validated test implementation. The block-length rule, replication count, and stationarity and dependence assumptions still need checking. Hansen's SPA uses studentization and a different, data-dependent construction of the null distribution.
Which correction answers which question?
| Question | Appropriate starting point | What it controls |
|---|---|---|
| Can any pre-specified rule be called significant? | Holm or Bonferroni | Family-wise error; valid under arbitrary dependence between tests, given valid input p-values |
| How many discoveries can we tolerate in a broad screen? | Benjamini–Hochberg under its assumptions; Benjamini–Yekutieli under arbitrary dependence | False discovery rate, not the chance the best rule is real |
| Did the best rule from a time-series search beat a benchmark? | Search-aware block bootstrap, Reality Check, or SPA | A family-level comparison with the benchmark, accounting for time dependence under the method's assumptions |
FDR fits a screen that reports many discoveries. This exit-rule audit instead asks whether any candidate beats its chosen benchmark. Even rejection of the family null would establish evidence for at least one candidate, not the future profitability of a particular configuration.
Why the result was not promoted
The audit did not establish +25.98 bps as an edge ready for trading. Its positive mean depended on one session, the selected rule had no untouched final split, and the archived summaries could not support a test of the candidate family. The execution audit also identified optimistic fill assumptions, outside the statistical correction discussed here.
Before considering promotion, I would require the versioned outcome record above, a defined benchmark and ranking criterion, and an evaluation period that had not informed development. The candidate would have to be fixed before opening that period, then checked with a capital-constrained execution model. No rerun or new holdout result is reported here.
Part of the HFM Research methodology series.
Top comments (1)
The median of −16.2 bps next to a +25.98 bps mean says the most here. At a 34% win rate, a few events from one session carrying the whole average is exactly what a per-day breakdown should catch before anyone reaches for p-values. Did the archive keep enough per-event detail to split the selected rule's result by symbol as well? That would show whether 10 October was market-wide or driven by one or two coins.