DEV Community

dustin liang
dustin liang

Posted on

CUSUM Change Detection in Python: Catching Persistent Shifts in Daily Stock News Attention

CUSUM change detection in Python takes about fifteen lines: standardise each new observation against a baseline, subtract a small allowance k, add the result to a running sum that is never allowed to go below zero, and raise an alarm when the sum crosses a threshold h. Because the sum accumulates evidence over several days, CUSUM catches a persistent move of one standard deviation or less, the kind a single-day z-score threshold almost never sees.

That is the gap this article is about. Daily counts of how often a stock is mentioned in news and research notes have two kinds of change. One is the spike: a single announcement day with three times the usual volume. Spike detectors handle it well. The other is a step: coverage settles at a higher level and stays there for weeks, because a new theme now includes the company or more institutions have started following it. Each individual day of a step looks unremarkable. Here is how to catch it with pandas and NumPy, how to keep the single spike from triggering the same alarm, and how to pick k and h with a quick simulation instead of guesswork.

All data is synthetic. The series stands for one unnamed stock; this is a method demonstration, not a view on any company.

A series with one spike and one step

120 business days. The expected count is 8 per day for the first 70 days and 12 afterwards, which is a 50% rise in baseline attention. Day 30 gets one extra burst of 25 mentions.

import numpy as np
import pandas as pd

rng = np.random.default_rng(42)
n = 120
dates = pd.bdate_range("2026-05-04", periods=n)
lam = np.where(np.arange(n) < 70, 8.0, 12.0)    # persistent shift from day 70
counts = rng.poisson(lam).astype(float)
counts[30] += 25                                 # one-day spike
s = pd.Series(counts, index=dates, name="mentions")

print(dates[70].date(), dates[30].date())        # 2026-08-10 2026-06-15
Enter fullscreen mode Exit fullscreen mode

Step 1: a baseline that does not peek

Standardise against a trailing baseline. Two choices matter here:

  • Use the rolling median, not the mean, so a spike does not lift the baseline for the next 40 days.
  • Shift by one day, so today's value is never part of its own baseline.

For the scale, count data gives you a shortcut: for a Poisson-like process the variance is about the mean, so the square root of the baseline median is a sensible standard deviation. If your counts are overdispersed (variance well above the mean, which is common for attention data), use a rolling MAD instead.

W = 40
base_mu = s.rolling(W, min_periods=20).median().shift(1)
base_sd = np.sqrt(base_mu.clip(lower=1))        # Poisson: var ~ mean
z = (s - base_mu) / base_sd

print(z[z > 3].round(2))
Enter fullscreen mode Exit fullscreen mode
2026-06-15    7.07
2026-08-05    3.83
2026-08-31    3.26
Enter fullscreen mode Exit fullscreen mode

This is the single-day rule in action. It finds the spike on 06-15 (z = 7.07), fires once on 08-05, which is before the step and therefore noise, and once on 08-31, three weeks after the step began. A reader looking at these three alerts would have no idea that the stock's baseline coverage changed on 08-10.

Step 2: the one-sided CUSUM

The upward CUSUM statistic is

S_t = max(0, S_{t-1} + z_t − k)

with an alarm when S_t > h. The allowance k is the amount of drift you are willing to ignore; a common rule is half the shift you want to detect, in standard-deviation units. h sets the trade-off between false alarms and delay.

def cusum_up(z: pd.Series, k: float = 0.5, h: float = 5.0, clip: float | None = None):
    z = z.copy()
    if clip is not None:
        z = z.clip(upper=clip)
    S = 0.0
    out_S, alarms = [], []
    for t, x in z.items():
        if np.isnan(x):
            out_S.append(np.nan)
            continue
        S = max(0.0, S + x - k)
        out_S.append(S)
        if S > h:
            alarms.append(t)
            S = 0.0                                # reset after an alarm
    return pd.Series(out_S, index=z.index), alarms

S, alarms = cusum_up(z)
print([d.date() for d in alarms])
# [2026-06-15, 2026-08-18, 2026-08-21, 2026-09-10]
Enter fullscreen mode Exit fullscreen mode

The step is now detected on 08-18, six business days after it started. Look at the z-scores from 08-10 to 08-18: -1.41, 2.37, -1.41, 1.77, 2.12, 1.06, 1.89. None of them would pass a threshold of 3, but each adds 1 to 2 to the running sum, which is exactly the kind of evidence CUSUM is built to collect.

There are two problems left in that output, though: the spike on 06-15 also triggered an alarm, and the step produced three alarms instead of one.

Step 3: keep the spike out of the step detector

A single value of z = 7.07 adds 6.57 to the sum in one day, which crosses h = 5 immediately. If you already run a separate spike detector, you do not want CUSUM duplicating it. Clip the input:

S2, alarms2 = cusum_up(z, clip=3)
print([d.date() for d in alarms2])
# [2026-08-18, 2026-08-21, 2026-09-11]
print(S2.iloc[28:34].round(2).tolist())
# [0.0, 0.0, 2.5, 1.29, 0.09, 0.0]
Enter fullscreen mode Exit fullscreen mode

With the input capped at 3, the spike contributes at most 2.5 and decays within three days. The step still alarms on 08-18. This split of responsibilities works well in practice: a percentile or z-score rule for "today is unusual", CUSUM for "the normal level has moved".

Step 4: one event per step, with an onset estimate

The repeated alarms after 08-18 happen because the 40-day rolling median takes about 20 days to absorb the new level; until then, every day still looks slightly high. Two standard additions fix the output:

  1. A cooldown after each alarm, roughly half the baseline window, during which no new alarm is raised.
  2. An onset estimate: the day after the sum was last at zero. CUSUM's alarm date is always late; the last zero tells you approximately when the change began.
def cusum_events(z, k=0.5, h=5.0, clip=3.0, cooldown=20):
    z = z.clip(upper=clip)
    S, last_zero, quiet_until = 0.0, None, None
    events = []
    for t, x in z.items():
        if np.isnan(x):
            continue
        if quiet_until is not None and t <= quiet_until:
            continue
        S = max(0.0, S + x - k)
        if S == 0.0:
            last_zero = t
        if S > h:
            pos = z.index.get_loc(t)
            onset = (z.index[z.index.get_loc(last_zero) + 1]
                     if last_zero is not None else z.index[0])
            events.append({"alarm": t.date(), "onset_est": onset.date(),
                           "days_late": int(np.busday_count(onset.date(), t.date()))})
            S, last_zero = 0.0, t
            quiet_until = z.index[min(pos + cooldown, len(z) - 1)]
    return pd.DataFrame(events)

print(cusum_events(z))
Enter fullscreen mode Exit fullscreen mode
        alarm   onset_est  days_late
0  2026-08-18  2026-08-05          9
Enter fullscreen mode Exit fullscreen mode

One event. The onset estimate is 08-05, three business days before the true change on 08-10, because the noisy 08-05 value started the accumulation. That is typical: treat the onset as approximate within a few days, not as an exact date.

Step 5: choose k and h by simulation

Textbook tables give average run lengths for normal data, but your z-scores come from a rolling median on Poisson counts, so it is quicker and more honest to simulate. Run the detector on many series with no change to count false alarms, and on many series with a known step to measure delay:

def false_alarms(k, h, clip=3, trials=300, n=250):
    total = 0
    for i in range(trials):
        r = np.random.default_rng(1000 + i)
        x = pd.Series(r.poisson(8.0, n).astype(float))
        mu = x.rolling(W, min_periods=20).median().shift(1)
        zz = (x - mu) / np.sqrt(mu.clip(lower=1))
        total += len(cusum_up(zz, k, h, clip)[1])
    return total / trials          # alarms per 250 trading days

def detection_delay(k, h, clip=3, trials=300):
    delays = []
    for i in range(trials):
        r = np.random.default_rng(5000 + i)
        lam = np.where(np.arange(140) < 70, 8.0, 12.0)
        x = pd.Series(r.poisson(lam).astype(float))
        mu = x.rolling(W, min_periods=20).median().shift(1)
        zz = (x - mu) / np.sqrt(mu.clip(lower=1))
        hits = [t for t in cusum_up(zz, k, h, clip)[1] if t >= 70]
        delays.append(hits[0] - 70 if hits else np.nan)
    d = pd.Series(delays)
    return d.median(), d.isna().mean()
Enter fullscreen mode Exit fullscreen mode

Results for a step from 8 to 12 per day:

k h False alarms per 250 days Median delay (days) Missed
0.5 4 1.84 3 0.7%
0.5 5 0.89 4 1.0%
0.5 8 0.17 7 4.3%
1.0 4 0.14 6 17%

For comparison, the single-day z > 3 rule produced 1.72 false alarms per 250 days on the same no-change data and, as shown above, did not reliably mark the step at all. k = 0.5, h = 5 is a reasonable default for a one-stock monitor: under one false alarm a year with a median delay of four trading days. For a watchlist of 200 names, the false alarms add up (about 180 a year at that setting), so either raise h towards 8 or require the alarm to persist before it reaches a human.

Practical notes

  • Run it downward too. A mirrored statistic, S = max(0, S - z - k), catches coverage that fades, which is just as informative for a monitoring log.
  • Overdispersion breaks the Poisson scale. If the variance of your counts is much higher than the mean, the z-scores are too large and false alarms rise. Check s.var() / s.mean() on a quiet period; above about 2, switch to a rolling MAD.
  • Non-trading days. Either fold weekend mentions into the next trading day or use a calendar-day index throughout. Mixing the two shifts the baseline every Monday.
  • Re-baseline after an event. Once a step is confirmed, the old baseline is stale. The cooldown handles this crudely; a cleaner option is to restart the rolling window from the onset estimate.
  • Log the parameters. Store W, k, h, the clip value and the cooldown with each alarm, so you can compare runs later.

FAQ

What is the difference between CUSUM and a rolling z-score?
A rolling z-score judges each day on its own, so it only reacts to large single-day moves. CUSUM accumulates small deviations over several days and reacts to a sustained change in level, even when no single day stands out.

Is there a CUSUM implementation in statsmodels?
statsmodels has CUSUM tests for regression residuals (for example breaks_cusumolsresid in statsmodels.stats.diagnostic), which test for structural breaks after the fact. For online monitoring of a daily series, the short loop above is easier to control and to explain.

Can CUSUM tell me why attention shifted?
No. It tells you when the level probably changed. Finding out why means reading what was published around the onset date.

Does a CUSUM alarm say anything about the share price?
No. It describes discussion volume only. Whether attention shifts relate to anything else is a separate research question that needs its own testing.

Conclusion

Spikes and steps are different problems. A threshold on daily z-scores finds spikes; a one-sided CUSUM with a clipped input, a cooldown and an onset estimate finds steps, typically within a few trading days and with a false-alarm rate you can set by simulation. Running both side by side gives a monitoring log that separates "one loud day" from "this stock is now discussed more than it used to be". For background on how daily stock heat is tracked across names, see stock heat monitoring in Lava.

中文简介:本文讲如何用 Python(pandas、NumPy)实现单边 CUSUMï¼Œæ£€æµ‹ä¸ªè‚¡æ¯æ—¥æ–°é—»æåŠé‡çš„æŒç»­æ€§æŠ¬å‡ã€‚å…ˆç”¨æ»šåŠ¨ä¸­ä½æ•°å¹¶æ»žåŽä¸€å¤©å»ºç«‹åŸºçº¿ï¼ŒæŒ‰æ³Šæ¾æ–¹å·®æ ‡å‡†åŒ–ï¼›å†ç´¯åŠ åç¦»ã€è¶…è¿‡é˜ˆå€¼æŠ¥è­¦ã€‚æ–‡ä¸­è¯´æ˜Žå•æ—¥ z>3 è§„åˆ™ä¸ºä½•æ¼æŽ‰é˜¶è·ƒå˜åŒ–ï¼Œå¦‚ä½•å¯¹è¾“å…¥æˆªå°¾é¿å…å•æ—¥å°–å³°è¯¯æŠ¥ï¼Œå¦‚ä½•åŠ å†·å´æœŸå¹¶ä¼°è®¡å˜åŒ–èµ·ç‚¹ï¼Œå¹¶ç”¨æ¨¡æ‹Ÿæ¯”è¾ƒä¸åŒ k、h çš„è¯¯æŠ¥çŽ‡ä¸Žæ£€æµ‹å»¶è¿Ÿã€‚æ•°æ®ä¸ºåˆæˆæ ·æœ¬ï¼Œä»…ä½œæ–¹æ³•æ¼”ç¤ºã€‚

Top comments (0)