14,399 Backtest Events Gave Me t = 8.20. Equal-Weighting Days Gave 1.66

A crypto-perpetual breakout simulation produced 14,399 events across 15 symbols from September 2025 to September 2026. Mean net return was +25.98 bps per event, with a naive event-level t-statistic of +8.20.

But the result depended heavily on how observations were weighted. Returns were concentrated in a small number of market sessions, something the aggregate statistic did not reveal.

Pooled over events, the mean is positive. Per day, it is negative. The gap is entirely Oct 10.

Same events, different unit of analysis

I grouped events by calendar day, computed one mean event return per day, then ran a one-sample t-test on those 365 daily means:

event-level:  t = mean(event returns) / (sd / √N)
               = +25.98 / (380.2 / √14,399)  =  +8.20

day-level:    t = mean(daily means)   / (sd / √N)
               = −4.67  / (53.82 / √365)     =  −1.66

These are different estimands. The first weights every event equally; the second weights every trading day equally. The sign reversal does not prove the naive SE was wrong — it shows that the result is sensitive to the weighting choice, which is reason enough to report both. The second is a diagnostic, not a replacement for clustered standard errors or a block bootstrap. For any strategy that fires multiple times a day on correlated moves, it's a useful sanity check.

The day behind the result

October 10, 2025 accounted for most of the positive event-level return. DOTUSDT traded between $0.94 and $4.29 during a market-wide flash crash. ATOMUSDT also experienced extreme price dislocations. Whether the simulated trades could have been executed at those prices is unclear. The simulation's 434 events that day averaged +981 bps. Remove that one day and the event-level mean changes from +25.98 to −3.69 bps.

The median event lost −16.2 bps. Win rate was 34%. The maximum event return was +8,756 bps. The top 1% of events were enough to flip the mean. Only one day made it positive.

October 10, 2025 contributed 434 of 14,399 events, but those events averaged +981 bps. Removing that one day changes the event-level mean from +25.98 to −3.69 bps.

Statistic	Unit / N	Mean net	SD	t-stat	Note
Naive t-test	14,399 events	+25.98 bps	380.2 bps	+8.20	Assumes independent events
Daily diagnostic	365 days	−4.67 bps	53.8 bps	−1.66	Two-sided p ≈ 0.098
Sensitivity: without Oct 10	364 days	−7.37 bps	14.9 bps	−9.42	Post-hoc; exposes concentration

Event-level SD is 380 bps; day-level SD is 54 bps — a 7× collapse. Both t-stats use the same formula (mean / SE), but they're measuring different dispersions. The event-level SE is tiny because N=14,399; the day-level SE is larger because the daily means are themselves variable, and one day (Oct 10, mean +981 bps) inflates both the event-level mean and the day-level SD in opposite directions.

The sensitivity row is not a confirmatory result. Removing an outlier and calling the rerun clean is p-hacking. It is there to show how much of the claim rests on one day.

The diagnostic

import numpy as np
import pandas as pd
from scipy import stats

def day_block_t(returns_bps: np.ndarray, timestamps: pd.Series) -> dict:
    """Equal-weighted daily diagnostic for event returns.

    Groups events by calendar day, computes the mean return per day,
    and tests whether the distribution of daily means differs from zero.

    This changes the unit of analysis from event to trading day.
    It is NOT a cluster-robust standard error for the event-level mean.
    """
    returns_bps = np.asarray(returns_bps, dtype=float)
    days = pd.to_datetime(pd.Series(timestamps), utc=True).dt.floor("D").to_numpy()
    if len(returns_bps) != len(days):
        raise ValueError("returns_bps and timestamps must have equal length")
    if not np.isfinite(returns_bps).all():
        raise ValueError("returns_bps must be finite")
    frame = pd.DataFrame({"return_bps": returns_bps, "day": days})
    daily = frame.groupby("day")["return_bps"].mean()
    if len(daily) < 2:
        raise ValueError("at least two trading days are required")
    t_stat, p_value = stats.ttest_1samp(daily, popmean=0)

    return {
        "n_events": len(frame),
        "n_days": len(daily),
        "daily_mean_bps": round(daily.mean(), 2),  # input already in bps
        "t_stat": round(float(t_stat), 4),
        "p_value": round(float(p_value), 4),
        "daily": daily,   # keep the series — inspect it, don't trust the summary
    }

# Usage
result = day_block_t(event_returns_bps, event_timestamps)
print(f"Event-level t = {event_returns_bps.mean() / (event_returns_bps.std(ddof=1) / len(event_returns_bps)**0.5):.2f}")
print(f"Day-level   t = {result['t_stat']}  (N={result['n_days']} days)")
# Event-level t = +8.20
# Day-level   t = -1.6579  (N=365 days)

Keep the daily series in the result. A t-statistic is one number; the series shows where the result is coming from.

Inspecting the concentration

import matplotlib.pyplot as plt

daily = result["daily"]  # already in bps

fig, ax = plt.subplots(figsize=(10, 3))
colors = ["#ef4444" if v > 200 else ("#4ade80" if v > 0 else "#f87171")
          for v in daily]
ax.bar(range(len(daily)), daily.values, color=colors, alpha=0.7, width=1)
ax.axhline(0, color="#6b7280", linewidth=1)
ax.set_xlabel("Trading day")
ax.set_ylabel("Daily mean return (bps)")
ax.set_title(f"Daily mean return — {result['n_days']} days, t = {result['t_stat']:.2f}")

# Flag days > 3σ
mean, std = daily.mean(), daily.std()
outliers = daily[(daily - mean).abs() > 3 * std]
for idx, val in outliers.items():
    ax.annotate(f"{val:.0f} bps", xy=(daily.index.get_loc(idx), val),
                xytext=(0, 8), textcoords="offset points",
                ha="center", fontsize=8, color="#ef4444")
plt.tight_layout()

Leave-one-day-out

Removing October 10 explicitly is post-hoc. The full diagnostic runs the same removal for all 365 days and plots t-stat vs dropped day.

LOO diagnostic across all 365 trading days. Dropping any day other than Oct 10 barely moves the t-stat (−1.3 to −1.8 range). Dropping Oct 10 alone moves it to −9.42. One spike, standing alone — concentration, not fat tails.

If several days had moved the t-stat materially, the process would be fat-tailed and the right test would be a week-block bootstrap. That's not what happened here: one day, one event cluster, and it's gone when you drop Oct 10.

Limits

There is no single correct return series. If capital is allocated to a portfolio, construct portfolio returns. If the estimand is per-event alpha, preserve that estimand and use an estimator that accounts for dependence: clustered standard errors, a block bootstrap, or HAC/Newey–West where it applies.

Nor is deleting October 10 a valid way to obtain a clean p-value. The primary result keeps every day. The removal is explicitly a post-hoc sensitivity analysis: it tells us how fragile the claim is, not what a pre-specified experiment would have concluded.

The daily t-test still assumes independence across days. It does not automatically account for serial correlation, which is another reason it is a diagnostic rather than the final inference procedure.

The 14,399 events weren't 14,399 independent observations. Most of the positive return came from 434 events on a single day. The naive t-stat didn't reveal that concentration. The daily breakdown did.

---

Part of the HFM Research methodology series — six things we found when we audited our own backtest results.

Comments

[Arhan Canli]

I checked the sensitivity row against your own numbers. Dropping Oct 10 moves the daily mean from −4.67 to −7.37 (consistent), but t goes from −1.66 to −9.42, which implies the daily sd fell from 53.8 to about 15 bps. So most of that −9.42 is the sd shrinking once the one day that inflated it is deleted, not extra evidence of a negative edge. I'd read that row as "the dispersion was one day" and stop there.

Two things on the weighting question that the post doesn't cover:

• A day with 1 event and a day with 434 events get the same weight in the day-level test, so the 365 daily means have very different variances. A week-block bootstrap, or weighting by √n, would show whether −1.66 holds.
• 434 events across 15 symbols in one crash is closer to one market bet counted 434 times. Grouping by day fixes that day, but not the dependence with the days around it. Did Oct 11-12 look different from the median day?

[Pavel Kazantsev 💎]

Sorry for the longread, but you're right on all three and I want to show the actual numbers.

The SD one is the most important to get right. Back it out from the table: mean −4.67, t −1.66, n=365 gives SD = 53.8 bps. Drop Oct 10: mean −7.37, t −9.42, n=364 gives SD = 14.9 bps. So the daily SD fell by 3.6×. The t-jump isn't stronger negative evidence — the thing generating the variance is gone. Oct 10 wasn't just an outlier in returns, it was carrying most of the day-level noise. The row label "sensitivity: without Oct 10" implies I'm showing robustness; what it actually shows is that the distribution was Oct 10 + 364 days of something else entirely. Should've reported the SD in that table — it would've been obvious immediately.

On √n weighting, I honestly don't know which direction it goes without running it. If busy days tend to be negative, it pushes t down further. But if Oct 10 (434 events, +981 bps) dominates the high-count days, you're giving more weight to the big positive outlier and the mean could flip back. The real check is a scatter of n_events vs daily_mean — if there's no pattern, equal-weighting is fine as a diagnostic, if there is one the WLS version matters. Didn't do that.

The cross-day thing is probably the most under-addressed. A flash crash doesn't end at midnight — Oct 11–12 had positions unwinding, elevated vol, weird spreads. Treating them as two fresh i.i.d. draws from the same process as a random Tuesday is wrong. Week-block bootstrap would absorb that. Or just look at whether the t-stat is stable when any 5-day window around an extreme day is excluded — not as a result, just as another concentration check.

One thing I didn't explain well in the post itself: the mean flipping sign, not just the significance. In event space Oct 10 contributes 434 × 981 bps ≈ 425k bps-events of positive weight. In day space it gets one vote. The other 364 days average −7.37 and outvote it. That's not an artifact — that's actually the finding. One session generated enough lucky fills to flip the aggregate mean positive while everything else was quietly losing. Just needed that framing.

[Arhan Canli]

The event-space versus day-space framing is the finding, and I'd put it in the post as one sentence: pooled over events the mean is positive, per day it is negative, and the gap is entirely Oct 10. That says more than either t-stat.

Two cheap checks that need no new model. Report the SD next to every t (you saw why). And run leave-one-day-out for all 365 days, not only the extreme one, then plot the t-stat against which day was dropped. If one spike stands alone, that is concentration; if several days move it, the process is fat-tailed and the week-block bootstrap is the right test.

[Pavel Kazantsev 💎]

Yeah, the one-sentence version is cleaner — I basically split one finding into two sections when I didn't need to.

The full LOO plot I didn't run. Pulled Oct 10 because the numbers made it obvious, but plotting t vs dropped day for all 365 would actually show the shape of the concentration rather than just the worst case. Going to do that and update the post.

SD point is fair — it's in the math block but easy to miss. The whole story of why the two t-stats differ is sitting in those two numbers (380 vs 54) and I didn't make that explicit enough.

[Pavel Kazantsev 💎]

Both done. Added SD next to every t — the 7× collapse from 380 to 54 bps is in the table now, not buried two paragraphs down.

Ran the LOO for all 365 days. 364 of them barely move the t-stat (−1.3 to −1.8). Only Oct 10 does anything — drops it to −9.42. One outlier, not fat tails. Block bootstrap would be the wrong test here.

The one-sentence framing is in the lede now. Agreed it lands better than the individual stats.