Defense contract awards looked like a 10% edge. It was the sector.

A public, free, daily signal that sounds made for AI trading agents - tested properly, against four yardsticks instead of one.

Every weekday, the U.S. Department of Defense publishes the contracts it awarded the day before. It is public, it is
free, and it sounds like exactly the kind of signal an AI trading agent would love: a company wins a large government
contract, so its stock should go up.

We tested it properly. The answer is no - and the reason why is the most useful lesson in backtesting I know.

The test

We froze the rules before running anything: which awards count, which holding periods (from 1 to 250 trading days),
which company sizes, which costs, and what "passing" means. The plan was committed before the first number came out,
so nothing could be tuned after seeing results. Every one of the 480 combinations was recorded, passing or not, so the
multiple-testing correction counts all of them.

Each result was compared against four yardsticks, not one:

• the small-cap index,
• the defense-sector ETF,
• cash,
• and the same stocks bought on random dates.

What it looked like

New contract awards, held for a year, beat the small-cap index by +10.1% after costs (2021-2026, 509 awards across
87 companies; confidence interval +5.6% to +17.2%).

If you stopped there, you would have an edge. A backtest that only compares against cash - or against a broad index -
stops there.

What it really was

The same awards lost to the defense ETF by 2.8%, and lost to the same companies bought on random days by 2.2%.

Owning defense contractors paid over those years. Buying them because they had just won an award added nothing. The
"edge" was the sector.

0 of the 480 combinations passed. Not tradeable.

Two more things the data said

• Small companies react overnight, then fade. For companies under $2 billion, an award worth at least 1% of the company gapped up about +1.0% at the next open - before anyone trading on daily data could act - and then lost 5.4% against the small-cap index over the next 20 trading days.
• The sign flipped between periods. In 2015-2020 the same small-company cells looked positive. In 2021-2026 they were negative. An effect that changes sign with the calendar is not an effect you can rely on.

The lesson

A result is only as good as the yardstick you compare it with. "Beats the index" is not enough if the stocks were all
in a sector that beat the index. The two comparisons that caught this one cost almost nothing to add:

1. the sector the stocks belong to, and
2. the same stocks on random dates (a placebo).

If your AI agent or backtest reports an edge, ask it for both before you believe it.

What I don't know

• Whether awards carry information the market has not priced in for companies too small to have a sector ETF that fits them.
• Whether the overnight move is reachable at all for anyone who isn't watching the release minute by minute.

A question for you

What yardstick do you hold a backtest against - and has the placebo test ever killed one of yours?

Want this checked on your own backtest? Backtest & Data Audit, one-week pilot: https://altdataforge.com/audit

Research and education only - not investment advice. Award data: U.S. Department of Defense daily contract
announcements (public domain).