DEV Community

dustin liang
dustin liang

Posted on

Rolling Percentile Spike Detection for Stock News Attention: A pandas Walkthrough

Rolling percentile spike detection flags a day as unusual when its value is above a high percentile, say the 95th, of the values in a trailing window that excludes the day itself. For daily counts of how often a stock is mentioned in news and commentary, this works better than a mean-and-standard-deviation rule, because attention counts are skewed, bursty and full of quiet weekends.

The short recipe in pandas is s.rolling(60, min_periods=40).quantile(0.95).shift(1), compared against the current value, plus an absolute floor so that "5 mentions instead of 2" never counts as a spike. The rest of this walkthrough explains each piece, shows why the shift(1) matters, compares the result with a z-score, and turns flagged days into episodes you can report. The data is synthetic and the stock is a method demonstration, not a view on any company.

Why attention counts need percentiles, not z-scores

Daily mention counts for a single stock behave like this: most days sit in a narrow band, weekends drop to a fraction of weekdays, and a few days jump by five or ten times when results, announcements or a theme rotation bring the name into the conversation. That shape breaks two assumptions behind a z-score:

  • The distribution is not symmetric. A standard deviation computed on skewed counts understates how often large values occur on the upside and gives meaningless thresholds on the downside.
  • Spikes contaminate the baseline. One large day inside the trailing window inflates the standard deviation for the next 60 days. The day after a spike, or a second spike two weeks later, can look ordinary.

A trailing percentile is rank-based. One extreme value moves the 95th percentile of a 60-day window by at most one position, so the threshold stays stable after a burst.

Build a test series

To make every number reproducible, generate a series with a known shape: Poisson noise around 30 mentions per day, weekends at 40% of weekday volume, and four injected bursts.

import numpy as np
import pandas as pd

rng = np.random.default_rng(7)
days = pd.date_range("2026-07-01", "2026-11-16", freq="D", tz="Asia/Shanghai")
base = rng.poisson(30, len(days)).astype(float)
base[days.day_of_week >= 5] *= 0.4                     # quieter weekends
bursts = {"2026-08-28": 140, "2026-10-30": 170, "2026-10-31": 90, "2026-11-12": 180}
for d, extra in bursts.items():
    base[days.get_loc(pd.Timestamp(d, tz="Asia/Shanghai"))] += extra
s = pd.Series(base.round(), index=days, name="mentions")
Enter fullscreen mode Exit fullscreen mode

Real data has the same columns once you aggregate records to one row per calendar day: a date index in the market's timezone and an integer count. If your source gives hourly counts, sum them per day in Asia/Shanghai before doing anything else; summing in UTC splits the A-share session across two dates.

The core rule: trailing quantile, shifted

WINDOW, Q, MIN_OBS, FLOOR = 60, 0.95, 40, 50

threshold = (s.rolling(WINDOW, min_periods=MIN_OBS)
               .quantile(Q, interpolation="higher")
               .shift(1))
spike = (s > threshold) & (s >= FLOOR)
Enter fullscreen mode Exit fullscreen mode

Each argument is a decision you should be able to defend:

  • WINDOW = 60 calendar days covers about two months, long enough to contain a few ordinary bursts and short enough to follow a slow change in baseline coverage.
  • Q = 0.95 means about three days in any 60 are above the threshold by construction. That is the false-alarm rate you are accepting before the floor is applied.
  • interpolation="higher" returns an actual observed value rather than a blend of two, which makes the threshold easy to explain: "higher than the third-largest day of the previous 60".
  • MIN_OBS = 40 prevents thresholds computed from a handful of days at the start of the series. In this sample the first valid threshold appears on 2026-08-10.
  • shift(1) excludes today from its own baseline. Without it, a spike raises the threshold it is being compared against, and in a backtest the threshold would contain information from the moment being tested.
  • FLOOR = 50 is an absolute minimum. Percentile rules are scale-free, which is a problem for thinly covered stocks where the 95th percentile might be 4.

Printing flagged days:

out = pd.DataFrame({"mentions": s, "p95_prev60": threshold, "spike": spike})
print(out[out["spike"]])
#                            mentions  p95_prev60  spike
# 2026-08-28 00:00:00+08:00     170.0        38.0   True
# 2026-10-30 00:00:00+08:00     204.0        39.0   True
# 2026-10-31 00:00:00+08:00     101.0        39.0   True
# 2026-11-12 00:00:00+08:00     210.0        41.0   True
Enter fullscreen mode Exit fullscreen mode

All four injected bursts are found, and the threshold barely moved between them, from 38 to 41, even though the window contained earlier spikes.

Add a percentile rank for reporting

A boolean flag is enough for alerts. For research notes and dashboards, a continuous score is more useful. Rolling.rank(pct=True) gives today's rank inside a window that includes today:

pct_rank = s.rolling(WINDOW + 1, min_periods=MIN_OBS + 1).rank(pct=True)
Enter fullscreen mode Exit fullscreen mode

The window is WINDOW + 1 so that it holds the previous 60 days plus today. A value of 1.0 means today is the highest of the 61; 0.984 means one earlier day was higher. In the sample, 2026-10-31 scores 0.984 because 2026-10-30 is in its window. The rank stays readable when you compare stocks with very different coverage, since it never mentions raw counts.

Compare with a rolling z-score

Here is the z-score version with the same window and the same shift:

mu = s.rolling(WINDOW, min_periods=MIN_OBS).mean().shift(1)
sd = s.rolling(WINDOW, min_periods=MIN_OBS).std().shift(1)
z = (s - mu) / sd
print(z[z > 3].round(2))
# 2026-08-28    15.41
# 2026-10-30    18.54
# 2026-11-12     6.73
Enter fullscreen mode Exit fullscreen mode

Two things happen, both caused by contamination:

  1. The day after a spike is missed. 2026-10-31 has 101 mentions on a Saturday, more than triple a normal weekday, yet its z-score is 2.92 because the previous day's 204 inflated the standard deviation.
  2. Later spikes are discounted. 2026-11-12 is the largest day in the sample, but its z-score of 6.73 is less than half that of 2026-10-30, because the earlier burst is still inside its 60-day window.

Neither problem exists for the percentile threshold. If you need a z-score for a model, compute it on log1p(mentions) with a median and MAD (median absolute deviation) instead of mean and standard deviation; it behaves much closer to the percentile rule.

Handle weekends and holidays

Weekend volume in the sample is about 40% of weekdays. A single threshold computed across all days is dominated by weekdays, so a weekend spike has to be very large to cross it. Two options:

  • Separate baselines. Compute thresholds on weekdays and weekends independently, with a shorter window and lower floor for weekends since there are fewer of them.
  • Trading-day index. Drop non-trading days, or roll weekend mentions into the next trading day, then compute the threshold on trading days only. This matches how most price data is indexed.
is_weekend = s.index.day_of_week >= 5
thr_split = pd.Series(np.nan, index=s.index)
for weekend in (False, True):
    part = s[is_weekend == weekend]
    win, mobs = (30, 15) if weekend else (WINDOW, MIN_OBS)
    thr_split[part.index] = part.rolling(win, min_periods=mobs).quantile(Q).shift(1)
spike_split = (s > thr_split) & (s >= 20)
Enter fullscreen mode Exit fullscreen mode

In this sample the split version catches the same four bursts and adds one marginal weekday, a reminder that every extra degree of freedom also adds false positives. Public holidays such as the October Golden Week behave like long weekends; if they matter for your universe, keep a holiday calendar and treat those days like weekends.

Turn flagged days into episodes

Consecutive flagged days usually describe one event. Group them so you report "a two-day episode peaking at 204" instead of two separate alerts:

grp = (spike != spike.shift()).cumsum()
episodes = (out[spike].groupby(grp[spike])
              .agg(start=("mentions", lambda x: x.index[0].date()),
                   end=("mentions", lambda x: x.index[-1].date()),
                   days=("mentions", "size"),
                   peak=("mentions", "max"))
              .reset_index(drop=True))
print(episodes)
#         start         end  days   peak
# 0  2026-08-28  2026-08-28     1  170.0
# 1  2026-10-30  2026-10-31     2  204.0
# 2  2026-11-12  2026-11-12     1  210.0
Enter fullscreen mode Exit fullscreen mode

For alerting, add a cool-down: once an episode starts, suppress new alerts for the same stock for N days unless the count exceeds the episode's peak. That keeps a three-day story from producing three notifications.

Intraday version

The same rule works on hourly counts with two changes. Compare each hour with the same hour of previous days rather than with the previous 60 hours, since 10:00 and 03:00 have completely different baselines. And use a larger floor relative to the hourly median, because hourly counts are smaller and noisier. A practical baseline is the 95th percentile of the same clock hour over the last 20 trading days, shifted by one day.

Reading a published heat number the same way

The method is easier to reason about when you check it against a real public series. For example, Lava (熔岩), a service that monitors A-share news-flow heat from institutions and KOLs for research and quant workflows, publishes per-stock monitoring pages with daily mention trends. Its Cambricon (688256.SH) page, at 2026-11-16 13:31 Beijing time, showed 470 mentions over seven days, up 59.3% on the prior seven days, a seven-day daily average of 67.1, a single-day high of 208, and 1,057 mentions across the 30 days from 2026-10-18. Those are statistics about discussion volume, not about price.

A week-on-week change answers "is this week louder than last week". A rolling percentile answers a different question: "is today unusual relative to this stock's own recent history". A single-day high of 208 against a seven-day average of 67.1 is exactly the kind of skew where a percentile threshold is easier to interpret than a z-score. You can look at the daily trend chart on the Cambricon news heat monitor and try reading it with the rules above. The stock is used here only as a demonstration of the method.

FAQ

What window length should I use for rolling percentile spike detection?
Start with 60 calendar days or 40 trading days for daily data. Shorter windows react faster to changes in baseline coverage but give noisier thresholds; longer windows are stable but slow to adapt when a stock's coverage permanently increases.

Why use shift(1) instead of closed="left"?
On a gap-free daily index they give the same result, and recent pandas versions accept closed="left" for both fixed and time-based windows. shift(1) is easier to spot in code review and behaves the same across versions. Whichever you pick, add a test that today's value is never inside today's threshold.

Does a spike in mentions mean anything for the price?
Not by itself. A spike is a description of discussion volume. Whether it relates to anything else is a separate research question that needs its own data and testing.

How do I compare spikes across stocks with different coverage?
Use the percentile rank, not the raw count or the ratio to threshold. A rank of 0.99 means the same thing for a heavily covered large cap and a thinly covered small cap.

Conclusion

For bursty, skewed attention counts, rolling percentile spike detection is a small amount of pandas with a large effect on reliability. Use a trailing quantile shifted by one period, require a minimum number of observations, add an absolute floor, and treat weekends separately when they behave differently. Report percentile ranks for comparison and group consecutive flags into episodes. Compared with a rolling z-score, the percentile rule keeps detecting the day after a burst and does not discount later spikes, which are exactly the cases that matter in news-flow data.

中文简介:本文讲用滚动分位数检测个股消息热度的异常放大:用前 60 天(不含当天)的 95 分位作阈值,加最少样本数和绝对下限,并与滚动 z 分数对比,说明为什么一次爆量会让 z 分数漏检随后几天。文中附可复现的 pandas 代码、周末分开处理与连续异常合并成事件的方法;寒武纪仅作方法演示,不涉及任何买卖判断。

Top comments (0)