DEV Community

dustin liang
dustin liang

Posted on

pandas ewm with halflife and times: Smoothing Irregular Stock Sentiment Snippets Without Hidden Lag

Series.ewm(halflife=..., times=...) computes an exponentially weighted moving average where the weight of each observation depends on how much clock time has passed, not on how many rows ago it was. With halflife=pd.Timedelta("2h"), a snippet that is two hours old counts half as much as one that arrived just now, whether there were fifty snippets in between or none.

That is exactly what you want for sentiment scores attached to news items, research notes or forum posts about a single stock. They arrive in irregular bursts: dozens during the trading session, a few in the evening, almost nothing overnight. A row-based EWM treats a 3 a.m. post and a 10:01 post as equally "one step" apart. A time-based EWM does not.

This walkthrough uses pandas and NumPy only and synthetic scores in the range -1 to 1, so you can plug in any classifier. It covers the API, a manual check of the weights, three traps (resampling labels, sparse overnight data and the missing "decay to now"), and a per-stock groupby. The labels S1 and S2 are placeholders for a method demonstration, not a view on any company.

Build an irregular snippet stream

Three days of snippets: 25 random timestamps during each trading day, 4 in the evening and overnight, and a small cluster of negative scores on the afternoon of 2026-11-12.

import numpy as np
import pandas as pd

rng = np.random.default_rng(11)
TZ = "Asia/Shanghai"

def snippets(day: str, n_session: int, n_evening: int) -> pd.DatetimeIndex:
    """Random timestamps: dense during the trading day, sparse in the evening."""
    start = pd.Timestamp(f"{day} 09:30", tz=TZ)
    evening = pd.Timestamp(f"{day} 16:00", tz=TZ)
    a = start + pd.to_timedelta(rng.uniform(0, 330, n_session), unit="min")
    b = evening + pd.to_timedelta(rng.uniform(0, 900, n_evening), unit="min")
    return a.append(b)

idx = (snippets("2026-11-11", 25, 4)
       .append(snippets("2026-11-12", 25, 4))
       .append(snippets("2026-11-13", 25, 4))
       .sort_values())
score = rng.normal(0.1, 0.5, len(idx)).clip(-1, 1)

neg = (idx > pd.Timestamp("2026-11-12 13:30", tz=TZ)) & \
      (idx < pd.Timestamp("2026-11-12 15:00", tz=TZ))
score[neg] = rng.normal(-0.6, 0.2, neg.sum()).clip(-1, 1)
s = pd.Series(score, index=idx, name="score")

print(s.index.to_series().diff().quantile([0.5, 0.9, 1.0]))
# median gap ~11 min, 90th percentile ~3.5 h, max ~5.8 h
Enter fullscreen mode Exit fullscreen mode

The gap distribution is the whole argument for using times: the median gap is about 11 minutes, but one gap in ten is longer than three and a half hours.

Row-based versus time-based EWM

by_count = s.ewm(halflife=6).mean()                                   # 6 rows
by_time = s.ewm(halflife=pd.Timedelta("2h"), times=s.index).mean()    # 2 hours

view = pd.DataFrame({"score": s, "by_count": by_count, "by_time": by_time})
print(view.loc["2026-11-12 14:00":"2026-11-13 09:45"].round(3))
Enter fullscreen mode Exit fullscreen mode
timestamp (CST) score by_count by_time
11-12 14:58 -0.690 -0.167 -0.169
11-12 20:37 -0.249 -0.176 -0.203
11-13 00:49 -0.144 -0.181 -0.179
11-13 04:41 -0.465 -0.212 -0.373
11-13 09:35 -0.174 -0.208 -0.216
11-13 09:38 0.054 -0.180 -0.096
11-13 09:38 0.226 -0.135 0.003

Both versions agree during the dense afternoon session, because rows and time move at roughly the same rate there. They split at the edges:

  • Overnight, the lone 04:41 snippet arrives almost four hours after the previous one. In the time-based version everything older has decayed heavily, so that single post pulls the average to -0.373. The row-based version barely notices.
  • The next morning, three snippets in three minutes move the time-based average from -0.216 to 0.003, while the row-based one still remembers yesterday's afternoon.

Neither behaviour is a bug. The time-based EWM is the more honest description of "what has the recent tone been", but it also tells you that it can be driven by very few observations. The section on effective sample size below deals with that.

API rules worth knowing

A few constraints, checked against pandas 3.0:

  • With times, the decay must be given as halflife and it must be timedelta-like ("2h" or pd.Timedelta("2h")). Passing span or com together with times raises ValueError: halflife must be a timedelta convertible object.
  • Without times, a timedelta halflife is rejected.
  • Only .mean() is implemented with times. .std() and .sum() raise NotImplementedError.
  • Sort first. In the version tested, unsorted times did not raise an error and simply produced wrong weights. Call sort_index() before you smooth.
  • Missing values: with the default ignore_na=False, a NaN still lets time pass, so older points keep decaying. Usually that is what you want for time-based weights.

Check the weights by hand

It is worth verifying once that pandas does what the halflife implies. For the final row, each observation's weight is 0.5 ** (age_hours / 2):

age_h = ((s.index[-1] - s.index).total_seconds() / 3600).to_numpy()
w = 0.5 ** (age_h / 2)
manual = (w * s.to_numpy()).sum() / w.sum()
print(round(manual, 6), round(by_time.iloc[-1], 6))   # 0.387401 0.387401
Enter fullscreen mode Exit fullscreen mode

They match. This also gives you the formula you need when you want to extend the method, as in the next sections.

Trap 1: resampling to an hourly grid can leak the future

Dashboards and joins often want one value per hour. The tempting line is by_time.resample("1h").last(). By default pandas labels each hourly bin with its left edge, so the row labelled 14:00 contains the last value inside 14:00–15:00, including the 14:58 snippet.

left = by_time.resample("1h").last()
right = by_time.resample("1h", label="right", closed="right").last()
print(pd.DataFrame({"default": left, "right": right})
        .loc["2026-11-12 13:00":"2026-11-12 15:00"].round(3))
Enter fullscreen mode Exit fullscreen mode
hour label default (left) label/closed = right
13:00 0.075 0.030
14:00 -0.169 0.075
15:00 NaN -0.169

If you join the default version to anything stamped "as of 14:00", you are using information from 14:58. With label="right", closed="right", the value at 14:00 only uses snippets up to and including 14:00. After that, ffill() is safe to fill empty hours.

Trap 2: a smooth line built from two snippets

The overnight value of -0.373 looks as authoritative as the afternoon value, but it rests on very little data. The effective sample size (ESS) of a weighted average, (sum w)^2 / sum(w^2), shows how many equally weighted observations it is worth:

def ess_series(s: pd.Series, halflife_hours: float) -> pd.Series:
    t = s.index
    out = []
    for i in range(len(t)):
        age = ((t[i] - t[: i + 1]).total_seconds() / 3600).to_numpy()
        w = 0.5 ** (age / halflife_hours)
        out.append(w.sum() ** 2 / (w ** 2).sum())
    return pd.Series(out, index=t, name="ess")

ess = ess_series(s, 2)
gated = by_time.where(ess >= 3)
Enter fullscreen mode Exit fullscreen mode
timestamp (CST) by_time ESS gated
11-12 14:58 -0.17 17.76 -0.17
11-13 00:49 -0.18 2.88 NaN
11-13 04:41 -0.37 2.01 NaN
11-13 09:35 -0.22 1.55 NaN
11-13 09:38 0.00 3.51 0.00

At the close of the negative cluster the average is backed by about 18 effective snippets. Overnight it falls to about two, and the first morning snippet is worth barely 1.5. Show ESS next to the smoothed value, or hide values below a threshold as above. The loop is O(n²), which is fine for one stock and one day; for long histories, restrict each step to the last few halflives.

Trap 3: pandas does not decay to "now"

An EWM only updates when a new row arrives. At 20:00 on 11-12, five hours after the last snippet, by_time still reports -0.169, exactly the value at 14:58. If your monitor asks "what is the tone right now?", that value is stale.

A small function evaluates the decayed average at any wall-clock time, with an optional neutral prior so that a long silence drifts back toward zero instead of freezing:

def decayed_mean(s: pd.Series, now: pd.Timestamp, halflife_hours: float,
                 prior: float = 0.0, prior_weight: float = 1.0) -> float:
    past = s[s.index <= now]
    age = ((now - past.index).total_seconds() / 3600).to_numpy()
    w = 0.5 ** (age / halflife_hours)
    return float((w @ past.to_numpy() + prior * prior_weight) / (w.sum() + prior_weight))

for now in ["2026-11-12 15:00", "2026-11-12 20:00", "2026-11-13 09:25"]:
    now = pd.Timestamp(now, tz=TZ)
    last_ewm = by_time[by_time.index <= now].iloc[-1]
    print(now, round(last_ewm, 3), round(decayed_mean(s, now, 2), 3))
Enter fullscreen mode Exit fullscreen mode
now (CST) last EWM row decayed mean with prior
11-12 15:00 -0.169 -0.153
11-12 20:00 -0.169 -0.106
11-13 09:25 -0.373 -0.083

Before the open on 11-13, the last EWM row still says -0.373 because of one overnight post. The decayed mean says the tone is mildly negative and weakly supported, which is a fairer summary. prior_weight=1.0 behaves like one imaginary neutral snippet that never ages; raise it if you want quiet periods to pull back to neutral faster.

Many stocks: groupby with times

For a stream that covers several stocks, smooth each one separately. Pass the group's own timestamp column as times:

df = pd.DataFrame({"ts": s.index,
                   "code": rng.choice(["S1", "S2"], len(s)),
                   "score": s.to_numpy()})
df = df.sort_values(["code", "ts"])
df["ewm_2h"] = (df.groupby("code", group_keys=False)
                  .apply(lambda g: g["score"].ewm(halflife=pd.Timedelta("2h"),
                                                  times=g["ts"]).mean()))
Enter fullscreen mode Exit fullscreen mode

Sorting by code and timestamp first protects you from the silent unsorted-times problem. group_keys=False keeps the original index so the result aligns back onto df.

Choosing the halflife

  • Match it to the question. One to three hours is an intraday "current tone". Twelve to twenty-four hours is a daily narrative. Several days is background mood.
  • Check how often the ESS gate fires. If most overnight values are hidden, the halflife is too short for your data density.
  • Keep the raw counts. A smoothed score of -0.2 based on 40 snippets and one based on 3 snippets are different facts. Store the count and ESS beside the score.

FAQ

What is the difference between halflife with and without times?
Without times, halflife is measured in rows. With times, it is measured in clock time and must be a timedelta such as "2h".

Can I use span or alpha with times?
No. pandas only accepts a timedelta halflife when times is set. You can convert: a row-based span corresponds to a halflife of roughly (span + 1) × ln(2) / 2 rows, which you can multiply by a typical gap to get a time halflife, but specifying the halflife directly is clearer.

Why can't I get an exponentially weighted standard deviation with times?
It is not implemented in pandas when times is passed. Compute it manually from the same weights: the weighted mean of squared deviations from the weighted mean.

Is the EWM value at a timestamp free of look-ahead?
Yes, each row only uses that row and earlier ones. Look-ahead usually enters later, through left-labelled resampling or by joining on the wrong side of a bin.

Conclusion

For irregular sentiment snippets, ewm(halflife=pd.Timedelta(...), times=...) gives a smoothed tone whose memory is measured in hours rather than rows. Sort before smoothing, verify the weights once by hand, resample with right-closed, right-labelled bins, show effective sample size beside the value, and decay explicitly to the current time when a monitor needs a "now" reading. The same steps apply to any single-stock opinion stream; for a worked example of hourly sentiment tracking on one name, see Lava's CATL sentiment monitor.

中文简介:本文讲如何用 pandas 的 ewm(halflife, times) 按真实时间间隔平滑不规则到达的个股舆情情绪分,并与按行数衰减的 EWM 对比。文中给出手算权重校验、小时重采样的未来函数陷阱(label/closed 设为 right)、有效样本量门槛,以及“衰减到当前时刻”的带先验函数和按股票 groupby 的写法。全部为合成数据的方法演示,不涉及任何买卖判断。

Top comments (0)