My stock scanner's AI was a momentum echo. Are today's models?

This is a submission for the Kaggle Benchmarking Challenge

TL;DR

• One of 9 models on Kaggle got more bullish the further a stock had already run, by more than the real outcomes justified, on 351 real cases. That was GPT-5.4 mini. Next came Qwen3 235B Instruct 2507, which leaned on the run-up clearly harder than the other 7, but not by enough to count as chasing. The other 7 stayed inside the test's noise.
• Rally headlines split them. After accounting for the run-up, 5 of the 9 models gave those headlines clearly lower odds than company news, and 1 clearly higher.
• None of the 9 models clearly beat a flat guess of 31%, and none ranked the stocks that jumped clearly better than a coin flip.
• My old scanner's AI was a momentum echo: the bigger the run-up, the more excited it got (rank correlation +0.55), though the run-up said nothing about direction. Its most excited calls ended more than 5% lower 41% of the time; its calmest, 18%.
• The twist: one grading choice nearly erased a finding. Graded from that day's close, its excited calls closed higher 11.9 points less often than its calm ones; graded from the previous day's close, the gap shrinks to 0.9. They still ended more than 5% down more often under every rule, by 13 points even on the previous close.

The benchmark: Hype Trap on Kaggle.

I built a stock scanner and ran it for one season, February to May 2026. It watched small, heavily talked-about US stocks, mostly under $10 a share, and for many of them it asked an AI model for a quick call: a "takeoff" number from 0 to 100 for the stock's momentum over the next 1 to 3 days. One stock had jumped 60% in five trading days, on headlines about a retail-trader frenzy. The AI gave it a 72, in its most excited band. Five trading days later the stock was more than 5% lower. The scanner saved each call the moment it was made, before anyone knew how the stock would do.

That gave me something a synthetic test set can't: real predictions, made before the outcome existed, with outcomes I can look up now. So I built a benchmark around one question: does an AI get more excited the further a stock has already run, when the run-up says nothing about which way it goes next? I call that run-up chasing. An AI that does it is a momentum echo: it repeats what the price already did and calls it a forecast. The second question is whether a "the stock is rallying" headline makes it worse. I named the benchmark Hype Trap.

Nothing here is investment advice. The scanner only ever traded practice money.

What I Benchmarked

Where the cases come from

The scanner saved 15,606 scored rows; 829 made it through every check, and 351 of those became the benchmark. The first cut: when the AI step was skipped or failed, the app saved a fixed placeholder score as if the AI had given it. That was 31% of all scored rows, and none of them are in here. About 1 in 10 headline sets turned out to be about some other company, so those went too.

How 15,606 rows became 351 cases

Step	Rows left
Scored rows the scanner saved	15,606
Real AI calls (placeholders removed)	10,808
Has a stored headline and enough price history	9,715
The scanner saw the stock again about a week later	3,673
Likely split errors, extreme single days and frozen quotes removed	3,466
One call per stock per 5 days (keeps the earliest)	880
Has official daily closes (51 had no daily price data)	829
Balanced benchmark sample	351

Every outcome comes from official daily closing prices. I never used the bot's own trade records, which had bugs of their own.

The wrong-company headlines. While drawing cases I found 41 of about 418 whose headlines covered a different business or general market news. The scanner searched news by ticker symbol, and a short symbol also matches ordinary words, people's names, companies with the same symbol on foreign exchanges, and market roundups. The live AI read those headlines too and gave a takeoff number anyway. I threw those cases out.

Each case shows a model two things: the stock's move over the previous 5 trading days, rounded to a whole percent, and one sentence summarizing its headlines, rewritten with names and other identifying details removed. No social media text. Here is that +60% case, exactly as a model sees it:

Stock's move over the previous 5 trading days: +60%
News: The stock skyrocketed on heavy volume in a retail-trader frenzy, prompting articles asking whether to chase it.

On this exact case, the 9 models on Kaggle gave it a 20% to 55% chance of a gain of more than 5%, and GPT-5.4 mini, the one model that counts as chasing, went highest.

I tried to break my own anonymization

Before publishing, I had a separate AI try to name the companies. On the first wording it named 7 of 15, mostly by pooling cases from the same company, so the public copy drops the month and the company column, and 85 sentences were made more generic. Two later passes named 1 of 15 and 1 of 20, and those sentences were rewritten too. After rounding the biggest run-ups, every case still matched at least 12 stocks.

Two yardsticks

• Closed higher: the close 5 trading days later is above the starting close. I use it for the scanner's own AI, because it is the plainest reading of a momentum call.
• Up more than 5%: the close 5 trading days later is more than 5% above the starting close. The Kaggle task asks this, because for stocks this jumpy, ending a hair higher is mostly noise. In the natural mix, 31.0% of cases went up by more than 5%.

To compare the old AI with today's models, I use rank correlations and AUC, which work on either yardstick.

Starting price, labels, weights and news kinds

• Starting close: for scans outside market hours, the last close at or before the scan (the scanner's own quote matched it, median difference 0.0%). For scans during market hours, that day's close. I did not use the previous close, because the 5-day run-up the AI saw already included the day's move up to the scan, and starting from the previous close would count that part twice. The twist in Findings shows how much this choice matters.
• Labels: up above +5%, down below -5%, flat in between. Prices are split-adjusted.
• Balance: 3 outcomes times 3 bands of the old AI's takeoff number, 40 cases per group, except one group (high takeoff, flat) that only had 31 usable cases. Every case carries a weight that turns the balanced sample back into the natural mix. The weights run from 1.03 to 4.30, so the 351 cases carry about as much information as 277 unweighted ones. Results below are weighted unless a table says raw.
• News kinds: each case also carries one of four news labels.

News kind	What it means	Cases
company_fact	Something happened at the company: results, a deal, financing, a trial readout, an analyst rating, a lawsuit	192
mixed	A real company fact plus price-move framing ("shares soared after the company won a contract")	90
price_action	The news is mostly that the stock is moving ("why is it surging", "hits a 52-week high")	46
promo	Tip-sheet style hype or automated rating pages with no new company information	23

Models Tested

The reference: the scanner's own live AI. The scanner didn't log which model answered each call; its code defaulted to Claude Sonnet 4.5 with GPT-4o mini as a fallback, so the honest label is "most likely Claude Sonnet 4.5, unconfirmed." It also saw more than the benchmark shows: the ticker, the price, three headlines with sources, social media summaries, and the scanner's own formula score, which already included price momentum. So when the old AI and a new model differ, the extra context could be the reason. Claude Haiku 4.5, from the same family as the old AI's likely model, followed the run-up at +0.31 with this stripped prompt, against the old AI's +0.69 on the same 351 cases (+0.55 across all 829 calls), with the scanner's full context. I can't split model from context here, so the lesson I trust is about the pipeline: check what your prompt hands the model.

The models on Kaggle all got the same prompt, one fresh chat per case, and each model's numbers here come from one run at its provider's default settings.

Nine models from six labs ran the main task, four of them reasoning models. No call errored.

The models, with lab, reasoning, training cutoff and answers

Model	Lab	Reasoning	Training cutoff	Answered (of 351)	Parse failures	Output tokens
Qwen3 235B Instruct 2507	Alibaba	no	not published (released July 2025)	351	0	6k
Claude Haiku 4.5	Anthropic	no	July 2025	351	0	9k
Claude Opus 5.5	Anthropic	yes	June 2026	351	0	24k
Gemini 2.5 Flash	Google	yes	January 2025	351	0	499k
Gemini 3.7 Flash	Google	yes	March 2026	351	0	123k
GPT-5.4	OpenAI	no	August 2025	351	0	6k
GPT-5.4 mini	OpenAI	no	August 2025	351	0	8k
GLM-5	Zhipu (Z.ai)	yes	not published (released February 2026)	351	0	776k
Grok 4.20 Non-Reasoning	xAI	no	not published (released March 2026)	351	0	5k

Every model here ran on version 2 of the main task. Kaggle's model proxy reserves the most a call could possibly cost before it starts, and version 1 sent 4 calls at a time, so those reservations didn't always fit in the free daily AI quota and the proxy refused most of Claude Opus 5.5's calls. Version 2 sends 2 at a time and changes nothing else. Even so, the proxy refused GPT-5.5 on 321 of the 351 cases, so GPT-5.4 stands in for OpenAI's bigger models. Grok 4.20 Reasoning finished on version 1, but its version 2 run stalled when the quota ran low, so it isn't in the tables either. The task page may still list those runs. A refused call counts as 50 there, so ignore the score of any run with refused calls.

The exact prompt and run settings

No system message. Kaggle's model proxy doesn't pass a temperature, so each model ran at its provider's default sampling. There is no reason field, and I read every reply that had text outside the JSON before making the runs public; none names a company. A reply that can't be parsed counts as 50. Google says requests to Gemini 3.7 Flash have gone to Gemini 3.8 Flash since October 8, and my runs came after that, so its row may be 3.8 Flash answering. Google's model cards give both the same March 2026 training cutoff.

The prompt, with line breaks added for reading:

You are reading news about a small, frequently hyped US-listed stock (most trade under
$10 a share). The company's name is hidden. You see the stock's move over the previous
5 trading days and a one-sentence summary of the recent news.

Stock's move over the previous 5 trading days: {move}
News: {news}

Give two answers:
1. p_up: your probability, from 0 to 100, that 5 trading days from now the stock's price
   will be more than 5% above its price right now.
2. read: what this news mostly is. Pick exactly one word:
   company = something happened at or to the company (results, a deal, financing, a trial
   result, an analyst rating, an insider or fund trade, a lawsuit, a listing notice)
   price = the news is mostly that the stock is moving (why it is surging, a new high, a
   list of movers)
   promo = promotional or automated hype with no new company information
   mixed = a real company fact plus clear price-move framing

Reply with JSON only and no other text: {"p_up": <integer 0-100>, "read": "<company|price|promo|mixed>"}

Two twin tasks change one thing each on the same 351 cases. hype_trap_no_move hides the run-up number, to measure that one number's pull (a headline can still say the stock soared). hype_trap_no_read drops the read question, in case defining "price move" news in the prompt makes a model warier of hype. hype_trap_no_move ran on Claude Haiku 4.5, Gemini 3.7 Flash, Qwen3 235B Instruct 2507, and GPT-5.4 mini; hype_trap_no_read ran on Claude Haiku 4.5, Gemini 3.7 Flash, and Qwen3 235B Instruct 2507.

How it's scored

• Run-up chasing, the main test. A model's extra lift is how many points its answer rises from the smallest run-up to the biggest, minus how much the real rate of a gain of more than 5% rose over the same range. It counts as chasing when the 95% range of its extra lift sits entirely above zero. The tables also give its rank correlation with the run-up (+1 means bigger run-up, bigger number, every time).
• Brier skill, the leaderboard number. 0 means no better than always guessing the 31% base rate; below 0 is worse. Bias is the average answer minus 31, in points. AUC is the chance that a stock that jumped got a higher number than one that didn't; 0.5 is a coin flip.
• Recognition probe. On 30 cases, plus 10 made-up placebos, an extra call asks only whether the model recognizes the company.

Definitions, for checking my math

• Run-up chasing, in full. The rank correlation (-1 to +1) between the model's probability and the run-up it was shown is unweighted, and says how closely its answers follow the run-up. On these 351 cases the old AI's takeoff number scores +0.69 (+0.55 across all 829 calls; the 351 were drawn to hold about twice the natural share of high-takeoff calls, so the two aren't directly comparable). The real rate's own lift is small (-0.5 points here, weighted; the top and bottom bands differ by about 8 points, but a dip in the +1 to +10% band flattens the straight line). A 95% range that includes zero means the result could be noise.
• Why not just compare correlations? The real outcomes' rank correlation with the run-up is +0.03, but a yes-or-no outcome's correlation is shrunk by noise. A made-up forecaster that knew the true odds in each run-up band would score +0.39 and look like a chaser. On the extra lift it scores 0.0. But with 351 yes-or-no outcomes, the real lift is only known to within about 20 points either way, so only a strong chase clears the bar.
• Brier skill: 1 minus the model's Brier score (the average squared miss; 0 is perfect) divided by the Brier score of always guessing the 31% base rate. It mostly rewards getting the average right: a model that answers 35 for every case scores -0.007 without telling any two cases apart, and one that answers 50 every time scores -0.169. That is why the leaderboard also shows bias and AUC. AUC works on any scale, so the old AI's takeoff number gets one too.
• Rally-headline gap: the model's average probability on price_action cases minus company_fact cases. The real gap is +3.3 points. Rally headlines came after bigger run-ups (median +8% against +2%), so a model that only looks at the run-up shows a gap without reading the headline at all. A fake model from my test suite that does exactly that scores +3.5. So I also report the gap after a fit on the run-up, where that fake model scores +0.9 and the real outcomes +1.7. The hype-trap score is the model's gap minus the real gap.
• Read check: how often the model's one-word read matches my label.
• Recognition probe: it never asks for a name, and the 10 placebo summaries fit hundreds of companies, so a yes on one is a claim with nothing to recognize. Since the probe can't prove memory either way, I read it against each model's training cutoff (or, when none is published, its release month). A model trained before February 2026 can't have seen what these stocks did next, so a yes from it means the summary still hints at a company or the model claims more than it knows. A model that may have been trained into the season gets a separate check under Limits, whatever it answers.
• Brier = weighted mean of (p - outcome)^2. It splits exactly into a wrong-average part, (mean p - base rate)^2, and a case-to-case part, the rest. The task reports both (brier_wrong_average, brier_case_to_case).
• AUC is weighted by the case weights; ties count half.
• Extra lift = the weighted least-squares slope of the probability on the percentile rank of the run-up (0 to 1, so the slope reads as points across the whole range), minus the same slope for the outcome. The slope of a yes-or-no outcome on the rank equals the slope of its true chance, so a model whose answers match the real odds at every run-up scores about 0, whatever shape the real curve has. On the full 829-call pool the outcome's own lift is +8.1 points (95% range -3.0 to +18.8); the 351 cases' -0.5 sits inside that noise.
• Ties: every model answered the same cases, so I compare models with a paired bootstrap (resampling the same cases for both models) and only call one better when the 95% range of the difference excludes zero.
• The run-up fit is a weighted least-squares fit of the probability on a cubic in the percentile rank of the run-up, plus news-kind dummies. The reported gap is the price_action coefficient (rally_gap_runup_adjusted_pts).
• Names in the task's summary.json: rally-headline gap = bullishness_gap_pts, real gap = real_gap_pts, hype-trap score = hype_trap_score_pts, run-up correlation = runup_chasing.spearman_p_vs_prior_move, extra lift = runup_chasing.extra_lift_pts, chases = runup_chasing.chases.
• The task's 95% ranges come from 2,000 resamples of cases with a fixed seed (the old AI's pool numbers in Finding 5 use 2,000 to 4,000). Ranges from resampling whole companies are computed offline with a private company key and reported where they differ.

Findings

1. Run-up chasing: what today's models did

One of the 9 models counts as chasing on this test: GPT-5.4 mini. Extra lift ran from +3.0 points for Gemini 3.7 Flash to +32.8 for GPT-5.4 mini. Compared pair by pair, GPT-5.4 mini leaned on the run-up clearly harder than every other model, and Qwen3 235B Instruct 2507 clearly harder than the remaining 7. Rank correlations with the run-up ran from +0.01 to +0.73 across the models, against +0.69 for the scanner's old AI on the same cases. One caveat: the read question in my prompt made all 3 models that ran the no_read twin follow the run-up more closely, and GPT-5.4 mini didn't run it (see Limits).

Did it chase?

Model	Run-up corr.	Extra lift, pts (95% range)	Chases?	Probe yes (placebo yes)
GPT-5.4 mini	+0.73	+32.8 (+12.4 to +51.6)	yes	2 of 30 (1 of 10)
Qwen3 235B Instruct 2507	+0.46	+18.6 (-2.8 to +38.3)	no	9 of 30 (1 of 10), claims recognition
Grok 4.20 Non-Reasoning	+0.16	+7.7 (-12.4 to +26.8)	no	5 of 30 (0 of 10), may overlap the season
Gemini 2.5 Flash	+0.01	+6.9 (-16.7 to +29.0)	no	3 of 30 (0 of 10)
GPT-5.4	+0.20	+6.3 (-14.4 to +25.8)	no	3 of 30 (0 of 10)
Claude Haiku 4.5	+0.31	+5.8 (-14.3 to +25.0)	no	1 of 30 (0 of 10)
Claude Opus 5.5	+0.35	+5.4 (-14.9 to +24.4)	no	0 of 30 (0 of 10), trained past the season
GLM-5	+0.09	+4.3 (-16.5 to +24.3)	no	2 of 30 (0 of 10), may overlap the season
Gemini 3.7 Flash	+0.21	+3.0 (-17.4 to +21.9)	no	0 of 30 (0 of 10), trained into the season
Reference: always 31 (the base rate)	none	+0.5	no
Reference: always 50	none	+0.5	no
Reference: fake model that reads only the run-up, plus noise	+0.60	+22.9 (+2.8 to +42.1)	yes
Reference: the scanner's old AI, same cases	+0.69	not a probability

Two of the 9 models said yes to more than 3 of the 30 real cases, and 1 of those was trained before these calls. No model said yes to more than 1 of the 10 made-up ones. None of those 2 ranked the cases clearly better than chance, so whatever they thought they recognized didn't clearly help them.

In the "above +30%" band, the models' average answers ran from 27.3% (Claude Opus 5.5) to 61.6% (GPT-5.4 mini), against a real rate of 42.8% for a gain of more than 5% (37 cases, weighted); in the "0% or lower" band the real rate was 34.6%.

Big run-ups swing harder both ways: of the 58 calls in the full pool already up more than 30%, 43% gained more than 5% and 40% lost more than 5%.

Gemini 3.7 Flash is the one model whose published cutoff falls inside the season (March 2026), so I split its cases there. On the 61 from before that month, which it could have read about, it ranked the stocks no better than on the other 290: AUC 0.512 (95% range 0.332 to 0.701) against 0.519. Its run-up chasing didn't clearly differ either. With only 61 cases, that rules out a strong memory, not a faint one.

2. The leaderboard

None of the 9 showed it could tell which of these stocks would jump: no Brier skill range sits above zero, and every AUC range includes 0.5.

Model	Brier skill	Bias, pts	AUC
Claude Haiku 4.5	+0.013	-2.1	0.549
Gemini 3.7 Flash	-0.002	-1.4	0.523
Claude Opus 5.5	-0.008	-5.3	0.522
Grok 4.20 Non-Reasoning	-0.025	+5.8	0.538
GPT-5.4	-0.026	-1.7	0.507
GLM-5	-0.090	+5.7	0.470
Qwen3 235B Instruct 2507	-0.127	+11.6	0.480
GPT-5.4 mini	-0.190	+17.0	0.503
Gemini 2.5 Flash	-0.225	+10.2	0.492
Reference: always 31 (the base rate)	0.000	0.0	0.500
Reference: always 50	-0.169	+19.0	0.500
Reference: fake model that reads only the run-up, plus noise	-0.021	+2.6	0.545
Reference: the scanner's old AI, same cases	not a probability		0.535

The 95% ranges, and which models can be told apart

Model	Brier skill (95% range)	AUC (95% range)
Claude Haiku 4.5	+0.013 (-0.019 to +0.034)	0.5492 (0.4804 to 0.61797)
Gemini 3.7 Flash	-0.002 (-0.034 to +0.021)	0.5226 (0.4513 to 0.58904)
Claude Opus 5.5	-0.008 (-0.045 to +0.014)	0.5221 (0.4551 to 0.5884)
Grok 4.20 Non-Reasoning	-0.025 (-0.091 to +0.021)	0.5383 (0.46798 to 0.60599)
GPT-5.4	-0.026 (-0.067 to +0.007)	0.5073 (0.4378 to 0.5755)
GLM-5	-0.090 (-0.168 to -0.033)	0.4697 (0.3997 to 0.5366)
Qwen3 235B Instruct 2507	-0.127 (-0.232 to -0.058)	0.48004 (0.4151 to 0.5451)
GPT-5.4 mini	-0.190 (-0.331 to -0.089)	0.5035 (0.4414 to 0.5692)
Gemini 2.5 Flash	-0.225 (-0.366 to -0.117)	0.4924 (0.4233 to 0.5642)
Reference: fake model that reads only the run-up, plus noise	-0.021 (-0.078 to +0.025)	0.545
Reference: the scanner's old AI, same cases	not a probability	0.535 (0.476 to 0.598)

On Brier skill, the tied pairs: Claude Haiku 4.5 with Gemini 3.7 Flash, Claude Haiku 4.5 with Claude Opus 5.5, Claude Haiku 4.5 with Grok 4.20 Non-Reasoning, Gemini 3.7 Flash with Claude Opus 5.5, Gemini 3.7 Flash with Grok 4.20 Non-Reasoning, Gemini 3.7 Flash with GPT-5.4, Claude Opus 5.5 with Grok 4.20 Non-Reasoning, Claude Opus 5.5 with GPT-5.4, Grok 4.20 Non-Reasoning with GPT-5.4, GLM-5 with Qwen3 235B Instruct 2507, and GPT-5.4 mini with Gemini 2.5 Flash, and every other pair has a clear gap. On AUC, most pairs are tied; the clear gaps: Claude Haiku 4.5 above GLM-5 and Claude Haiku 4.5 above Qwen3 235B Instruct 2507.

3. Twin tests: hide the run-up, drop the read question

For GPT-5.4 mini, hiding the run-up number cut its extra lift from +32.8 to +6.6 points: showing the run-up raised its answers by 4.6 on stocks already up more than 30% and lowered them by 12.6 on stocks that hadn't gone up. So seeing the number mostly cut its odds on stocks that hadn't gone up. The no_move twin ran on the one model that counts as chasing, plus 3 that don't.

The no_move twin, model by model

Model	Extra lift with the run-up shown / hidden, pts	Rank corr. shown / hidden	Change on stocks up more than 30%, pts	Change on stocks that hadn't gone up, pts
GPT-5.4 mini	+32.8 / +6.6	+0.73 / +0.29	+4.6 (+1.8 to +7.5)	-12.6 (-14.3 to -10.7)
Qwen3 235B Instruct 2507	+18.6 / +8.9	+0.46 / +0.18	-6.2 (-9.3 to -2.9)	-9.7 (-11.8 to -7.9)
Claude Haiku 4.5	+5.8 / +5.7	+0.31 / +0.18	-6.5 (-9.7 to -3.9)	-5.5 (-7.0 to -4.1)
Gemini 3.7 Flash	+3.0 / +3.6	+0.21 / +0.17	-0.1 (-1.0 to +0.7)	+0.1 (-0.6 to +0.9)

Extra lift is the chase table's measure, with the run-up shown (the main task) and hidden (this twin). A change is the answer with the number shown minus the answer with it hidden.

Asking for the read lowered the average answer of all 3 models that ran the no_read twin, by 1.5 to 6.0 points, mostly on stocks that hadn't gone up, so with it their answers followed the run-up more closely. Both twins point the same way: the extra run-up following came mostly from lower answers on stocks that hadn't gone up.

The no_read twin, model by model

Model	Average change, pts	On stocks up more than 30%	On stocks that hadn't gone up	Rank corr. with / without	Rally gap after run-up fit, with / without
Claude Haiku 4.5	-6.0 (-7.1 to -4.9)	-1.6 (-3.1 to 0.0)	-7.8 (-9.8 to -6.1)	+0.31 / +0.06	-3.3 / -7.8
Gemini 3.7 Flash	-1.5 (-1.9 to -1.1)	0.0 (-0.8 to +0.8)	-2.4 (-3.0 to -1.8)	+0.21 / +0.05	-0.7 / -0.3
Qwen3 235B Instruct 2507	-1.7 (-2.3 to -1.2)	+1.7 (-1.1 to +4.9)	-2.9 (-3.8 to -2.0)	+0.46 / +0.32	-4.5 / -5.3

A change is the answer with the read question minus the answer without it.

4. Rally headlines: what the models did with them

After accounting for the run-up, 5 of the 9 models clearly gave rally headlines lower odds than company news and 1 clearly gave them higher, while the old AI leaned slightly the other way (+1.9 takeoff points, a different scale) and the real outcomes ran +1.7. My prompt defines that kind of news right next to the question, so this could be the prompt talking. The no_read twin drops that question. Without it, Claude Haiku 4.5 rated rally headlines lower than with the question, and Gemini 3.7 Flash and Qwen3 235B Instruct 2507 stayed within noise. So the definition in the prompt didn't clearly make any of them warier of rally headlines. The one-word read matched my labels 72.1% to 89.5% of the time.

The real gap, and each model's gap and read check

On the Kaggle question, rally headlines came out +3.3 points ahead of company news, weighted, but the 95% range runs from -13.2 to +20.6. After the fit on the run-up, +1.7. So this sample can't say whether those headlines are a trap at all, which makes the models' reaction to them the interesting part.

Model	Rally gap after run-up fit, pts (95% range)	Read matched my label
Claude Haiku 4.5	-3.3 (-4.5 to -2.1)	77.8%
Gemini 3.7 Flash	-0.7 (-1.7 to +0.4)	88.6%
Claude Opus 5.5	+2.0 (+0.9 to +3.1)	88.6%
Grok 4.20 Non-Reasoning	-5.7 (-8.4 to -3.0)	72.1%
GPT-5.4	-2.9 (-4.8 to -1.0)	78.9%
GLM-5	-2.2 (-5.6 to +1.2)	89.5%
Qwen3 235B Instruct 2507	-4.5 (-7.3 to -1.9)	82.9%
GPT-5.4 mini	-1.3 (-4.5 to +1.8)	72.4%
Gemini 2.5 Flash	-10.1 (-15.5 to -4.2)	86.6%

On cases a model itself called "price", its average answer (unweighted) differed from cases it called "company" by -8.6 to +3.3 points across the 9 models that used both words, and was higher for 2 of them.

5. The old scanner AI, live: a momentum echo

Its excitement tracked the past. Across all 829 clean calls in their natural mix, made before the outcomes existed (the part of this post nobody can rerun), the takeoff number had a rank correlation of +0.55 with the run-up the stock already had, and -0.06 with the next 5 days' return. The run-up itself told you nothing about which way the next 5 days would go: about 0.00. The obvious objection: "You asked it for momentum, and it gave you momentum." It was asked about the next 1 to 3 days, and leaning on the last 5 only helps if the past move predicts the next one. For these stocks it didn't. Nor was it just copying the scanner's formula score, which already contained price momentum: holding that score fixed, the link to the run-up only drops from +0.55 to +0.43.

Old AI's takeoff	Calls	Already ran (median)	Closed higher	Up more than 5%	Down more than 5%
Low (25 or below)	309	-1%	59.5%	27.8%	18.1%
Mid (26 to 50)	394	+4%	52.8%	32.7%	23.6%
High (above 50)	126	+21%	47.6%	33.3%	41.3%

Its most excited calls were the riskiest ones. Using my starting-price rule (the twist below shows the other two), high-takeoff calls closed higher 11.9 points less often than low ones and ended more than 5% down 23 points more often. On the Kaggle yardstick, a gain of more than 5%, they did about the same as the calm calls, and the takeoff number's AUC for that gain was 0.53, which can't be told apart from a coin flip. On whether the stock closed higher at all, it did worse than a coin flip: 0.45. This high-versus-low comparison was the first question I asked, before any relabeling. If takeoff meant a big move either way, it got that part right: 75% of its excited calls moved more than 5% up or down, against 46% of its calm ones. It got the direction wrong, and big run-ups swing harder anyway, so some of that is the run-up again.

It leaned toward rally headlines too, but mostly through the run-up those stories came with. So for my old AI, the hype trap was mostly the run-up itself.

The 95% ranges, and the old AI by news kind

The 95% ranges for the numbers above:

Number	Value	95% range
Takeoff vs the run-up the stock already had (rank correlation)	+0.55	+0.50 to +0.60
Takeoff vs the next 5 days' return	-0.06	-0.14 to +0.01
The run-up vs the next 5 days' return	about 0.00	-0.08 to +0.07
Takeoff vs the run-up, holding the scanner's score fixed	+0.43	+0.37 to +0.48
High minus low takeoff: closed higher, pts	-11.9	-22.0 to -1.8
High minus low takeoff: ended more than 5% down, pts	+23	+14 to +33
High minus low takeoff: gained more than 5%, pts	+5.5	-4.5 to +15.3
Takeoff's AUC for a gain of more than 5%	0.53	0.49 to 0.57
Takeoff's AUC for closing higher	0.45	0.42 to 0.49

The rally-headline numbers use the 351 benchmark cases, the only ones with news labels. The old AI gave price_action headlines a takeoff about 5 points higher than company news (+5.1, 95% range +0.4 to +10.3, and -0.1 to +10.8 resampling whole companies). After the fit on the run-up, that drops to +1.9 (-1.6 to +5.6).

By news kind, weighted to the natural mix (closed higher at all):

News kind	Cases	Closed higher	Median 5-day return	Already ran (median)	Old AI's average takeoff
company_fact	192	55.4%	+0.5%	+2%	35.1
price_action	46	67.4%	+0.8%	+8%	40.1
mixed	90	47.6%	-2.2%	+8%	49.8
promo	23	58.6%	+0.4%	+2%	30.3

The mixed group is the one I'd watch: the old AI was most excited by it (14.7 takeoff points above company news), and it closed higher least often (7.8 points below company news, 95% range -21.9 to +6.3, so a pattern only).

6. The twist: one grading choice nearly erased a finding

Grading from the previous day's close would have nearly erased the gap between the excited and the calm calls. Here is one market-hours call in order:

previous close -> morning run-up (inside the 5-day move the AI saw) -> scan -> that day's close -> 5 trading days later

Start from that day's close (my rule) and the excited calls closed higher 11.9 points less often. Start from the scanner's own quote at scan time and it's -9.7. Start from the previous close and it nearly disappears (-0.9), because that rule credits the AI with the morning run-up it had already seen, and the excited calls came on days that were already running. Under every rule the excited calls still ended more than 5% down more often, by 13 points even on the previous close. Under that rule they also gained more than 5% more often, by 13 points: the excited calls swung harder in both directions. If I had graded the generous way, I would have written a much friendlier post about that AI.

The price source mattered too. My first version graded each call with the scanner's own later price quotes, taken whenever a later scan happened to run. Rebuilt from official daily closes, 20.6% of the cases in both versions changed label, the high-versus-low gap went from -6.0 to -11.9 points, and the rally-headline comparison flipped sign. Neither rally gap was clear of zero, so I treat rally headlines as an open question.

The relabeling, in numbers

Of the 339 cases in both versions, 70 changed label (20.6%) and 48 flipped between "closed higher" and not. Rally headlines first looked 3.9 points worse than company news at closing higher and ended up 12.0 points better (on the Kaggle question, a gain of more than 5%, the gap is +3.3). Under the previous-close rule, the gap in drops of more than 5% has a 95% range of +4 to +22.

What I'd tell anyone feeding news to an LLM

1. Save every call before the outcome exists, with the model's name. Mine saved the calls but not which model made them.
2. Strip fallback and placeholder scores before grading anything. They were 31% of my rows.
3. Check that each headline is about the right company. About 1 in 10 of mine weren't.
4. Choose your starting price before you look at results. Mine moved a finding from -11.9 to -0.9.
5. Compare the model's score with the run-up it was shown. If they move together and the run-up predicts nothing, you have an expensive momentum echo.

Limits and what I'd measure next

• The chase test is blunt. The real lift across the run-up is only known to within about 20 points on 351 yes-or-no outcomes, so a model that leans a little harder than reality won't be flagged.
• The prompt asks two things at once. The main prompt defines "price move" news next to the probability question. With the read question, the 3 models that ran the no_read twin followed the run-up more closely (rank correlation +0.21 to +0.46 with it, +0.05 to +0.32 without), so the main task may overstate run-up chasing. The other 6 models weren't checked this way, including GPT-5.4 mini, the one model that counts as chasing.
• Memory. A model trained on 2026 news might recognize an event even with names removed, and the probe is a self-report on 30 cases, so a "no" doesn't prove a model never saw the story. Gemini 3.7 Flash was trained into the season, Claude Opus 5.5 was trained past it, and Grok 4.20 Non-Reasoning and GLM-5 don't publish a cutoff and came out during it, so the probe can't clear them. A model that remembered how these stocks did would rank them well; their AUCs ran from 0.470 to 0.538, none clearly above 0.5, so there is no clear sign of that.
• One season, one run behind each model's numbers, and a survivorship filter. The 351 cases are one kind of stock, from 235 companies, in one season (February to May 2026). Each model's numbers come from one run at its provider's default sampling, so small differences between models may be run-to-run noise. And a call needed a later scanner quote about a week out to enter the pool; 62% of otherwise usable calls had none.

The other limits: data, labels, grading

The data.

• Only calls the scanner let through: it asked the AI only about stocks in a middle band of its formula score, and never saved the AI's "skip this" calls.
• Survivorship: stocks drop out when the buzz fades or they pass the price cap, so the calls with no later quote aren't random, and I don't know which way that bias points.
• To guard against split errors I dropped calls with any single-day move beyond 80% either way, which also removes some real hype days.
• Some companies appear more than once, which is why the main ranges are also checked by resampling whole companies.

The labels.

• One labeler: the paraphrases and news labels were written by an AI (Claude) in one pass and checked by script. Claude models are among those tested, so watch for a home-team edge. Of the 351 cases, 287 were labeled without seeing their outcome and 64 knowing which outcome group they were filling. No model ranked the cases clearly better than chance, so if those 64 sentences leak anything, no model made much use of it. Still, all 9 models ranked those 64 better than the 287 blind ones, by 0.10 in AUC on average, though that gap's 95% range (-0.03 to +0.24) includes zero, so it may be noise or a small leak. On the 287 blind cases alone, the AUCs ran from 0.44 to 0.52, and every model's extra lift stayed within 3 points of its full-set number, so the chase result holds without the 64.
• I checked 20 cases by hand: I couldn't name the company in any of them, and I'd have labeled 2 of them differently.
• Small groups: 46 price_action and 23 promo cases.

The grading.

• The old AI was asked about the next 1 to 3 days and graded on 5.
• Every model is graded from that day's close, the rule I picked for the old AI, though the prompt says "right now".
• The leaderboard number leans on one season's real rate (31%). In a hotter or colder market, models that guess high or low by habit would move up or down the board without reading the cases any better.

What I'd measure next.

• Rebuild without the survivorship filter: daily closes now exist for most of the 62% the scanner never saw again.
• Use the 20-day outcome every case already has; the task only uses 5 days.
• Ask the read question in a separate call, so it can't prime the probability at all.
• Grade the chase against the real curve from all 829 calls, which pins it down better than the 351 cases alone.

My Benchmark

The scanner and its logs are older; the case set, the Kaggle tasks and this write-up are new, built for this challenge.

https://www.kaggle.com/benchmarks/saltysteve/hype-trap

Hype Trap has 1 task on the 351 cases: hype_trap (the leaderboard task).

The case file is a private Kaggle dataset, so it can't be downloaded. The outcome labels sit only in that private dataset, and the per-case output files leave them out, so the full answer key isn't published. The cases carry no tickers, company names, dates or prices.

If you have built anything that feeds news to an LLM: did you check what it was really reacting to?

Comments

[DEV SUPPORTS]

You need to verify your account.

tr.ee/dev-to