11. Trending products, part 2: predictive models

Originally published at 4thwithme.dev.

In part 1 we ran four models that only look back. They made the same mistake. The Gadget fad and the Hoodie holiday spike looked the same. The Tumbler, the one real trend, was mostly missed. Now we try models that look ahead. They ask a better question: is this more than we expected?

[ THE DATA, THREE YEARS LONG ]

A season needs history. In 120 days you cannot see a yearly pattern. So I added 975 days before the old data. The last 120 days are exactly the same as in part 1. A check in the code confirms it.

In the new history, the Hoodie has a holiday bump every year. The Tumbler sits flat at 30 sales a day. The other three are quiet. One catch: in this fake world the season repeats on the same day every year. Real holidays move.

[ MODEL 5: FOURIER WAVES ]

Any repeating pattern can be built from simple waves. This is the Fourier series. Add a few waves of the right size and you copy the season. A tool called the FFT tells you which waves are in your data.

the Hoodie hides a yearly wave and a weekly wave

This chart is a periodogram. The tall sticks at 1, 2, 3 and beyond are yearly waves. The stick near 52 is the weekly wave. The holiday bump is a sharp spike, so it needs many yearly waves to copy.

season(t) = a₀ + Σ [ aₖ·sin(2πkt/P) + bₖ·cos(2πkt/P) ]

P is the wave length: 7 for weekly, 365 for yearly. We add K waves for each. The sizes aₖ and bₖ come from least squares. In Python it is short:

def waves(t, period, k):
    return [f(2 * np.pi * i * t / period) for i in range(1, k + 1) for f in (np.sin, np.cos)]

X = np.column_stack([np.ones_like(t)] + waves(t, 7, 3) + waves(t, 365, K))
beta, *_ = np.linalg.lstsq(X, sales, rcond=None)
season = X @ beta

The big choice is K, the number of yearly waves. I picked it on a test year the model had not seen. I trained on year 1 and tested on year 2. The average miss per day for the Hoodie was 9.6 with K = 0 or 3, then 7.1 with K = 10 and 6.5 with K = 20. For the other four products, zero yearly waves won. Too many waves would only copy noise (overfitting).

K = 3 reaches 63, K = 10 reaches 106, K = 20 reaches 130. The real peak is 130

We fit the season on the 975 history days only. It never sees the last 120 days.

Signal 1: surprise

Now we use the season as a forecast. Our guess for next week is the season plus the recent level. The level is the average leftover of the last 14 days. We make the guess 7 days early. Then we see how wrong we were.

z = (actual − expected) / √expected / s

This is the part 1 z-score with one change. "Expected" now comes from the forecast, not a flat average. And s is the real spread, measured on the history years. It ran 1.2 to 1.4 times the Poisson guess. In part 1 I said to measure it, so here it is.

Take the Hoodie on day 100, the holiday peak. We expected 846 sales for the week. It sold 775. That gives (775 − 846) / √846 = −2.4. Divide by its spread of 1.26 and z = −1.9. In part 1 the same day scored 24.0. The model saw the holiday coming, so it was not a surprise.

same Hoodie spike: part 1 sees a rocket, the forecast sees a calendar

The Gadget still lights up. It was flagged on days 59 to 77. The Hoodie never crossed 3. The Tumbler was flagged on only 6 days. A steady climb is not a surprise for long, because the 14-day level catches up.

Signal 2: projected growth

Surprise cannot see slow climbs. So we add a second signal. Remove the season. Draw a straight line through the last 56 days of what is left. The slope tells us how fast sales climb.

growth = 28 × slope / level

That is the growth we expect over the next 28 days. I flag a product when growth is 15% or more and the slope is clearly not noise (a t-score of 3 or more).

the Tumbler sits above the line all the way

The Tumbler gets what it missed in part 1. Its growth is 21% on day 45, 20% on day 60 and 14% on day 119. It is flagged on 78 of 85 days. The Gadget peaks at +66%, but it stays flagged until day 89. That is 19 days after its peak, because a 56-day line turns slowly. The Hoodie and the Pin are never flagged.

Fourier models have limits. They need two or more years of history. They assume the season has the same size each year. And they know nothing about fads. Prophet is this same idea plus trend breaks and holidays. For more, see the chapter on Fourier terms.

[ MODEL 6: LIGHTGBM ]

Fourier fits one product at a time. LightGBM learns from many products together. It builds many small decision trees. Each new tree fixes the mistakes of the ones before it. This is called gradient boosting. The LightGBM paper explains how it runs fast.

IN SIMPLE WORDS

Fourier is one expert who knows one product very well. LightGBM is a big room of beginners. Each one asks a simple yes-or-no question. Together they learn rules like "after a huge jump, sales usually fall."

It is not a time series model. It wants a table. Each row is one product on one day. Each column is a fact we know on that day. The answer column is what we want to predict: how much sales change over the next 28 days.

Five products are not enough to learn from. So I made 200 fake ones: 40 each of trends, seasonal products, fads, flat products and tiny ones. They have random sizes and start dates, and they share the same noise. The fads rise and then fall, so the model can learn the fall. That gives 109,200 training rows.

Each row has 10 inputs. Six come from the recent past: the part 1 scores (z-score, velocity, decay), the change of the last 28 days against the 28 before, size, and the day of the week. Two say what happened in the same month last year and two years ago. Two more give the day of the year. The answer is the change in the next 28 days against the last 28.

model = lgb.LGBMRegressor(n_estimators=300, learning_rate=0.05, num_leaves=15)
model.fit(X_train, y_train)            # y = ln(next 28 days / last 28 days)
growth = model.predict(X_today)        # flag if growth >= ln(1.15)

One rule matters a lot: no peeking at the future (data leakage). I used only rows whose 28-day future ends before day 975, the start of the story data. Always split by time, never at random. This is how time series testing works.

the part 1 scores add up to 43%, last year's pattern to 18%

So part 1 was not wasted. Its scores are now the model's inputs. Last year's pattern adds 18%.

What did it do? On the Gadget it was flagged on days 59 to 67. Then it stopped on day 68, two days before the peak. On day 70 it predicted −45% for the next 28 days. The real number was −8%. On day 74 it predicted −51% and the real number was −66%. The size is rough, but the direction is right. It also had 3 stray flags at the very end (days 117 to 119).

On the Hoodie it was flagged on days 67 to 98. That is 33 days before the peak, and 21 days before part 1's z-score. It learned from last year. But it says "going up", not "going up more than the calendar says". On the Tumbler it was flagged on only 18 of 85 days. Real growth averaged +22% a month. Its guesses sat mostly between +9% and +17%, just under the line. Trees average things out, so slow climbs look smaller than they are.

[ HEAD TO HEAD ]

All models on the same data. The rules were fixed before I ran anything: z of 3 or more, growth of 15% or more, and LightGBM growth of 15% or more.

no model is best on all three products

Model	Gadget (fad)	Hoodie (season)	Tumbler (real trend)
Part 1 z ≥ 3	days 59 to 79	days 88 to 110	24 of 85 days, mostly noise
Surprise z ≥ 3	days 59 to 77	never	6 days
Growth ≥ 15%	days 63 to 89 (and day 35)	never	78 of 85 days
LightGBM ≥ 15%	days 59 to 67	days 67 to 98	18 of 85 days

The Pin was never flagged by any model. Good. The Tote, which really does climb slowly, was flagged on 14 days by growth and 5 or fewer by the others.

[ WHAT TO USE WHEN ]

There is no winner. Each model answers a different question:

• Surprise asks: "Is this more than the calendar explains?" It fixes the Hoodie.
• Growth asks: "Is it climbing steadily?" It finds the Tumbler. It is slow to notice that a fad is over.
• LightGBM asks: "Where is it going next month?" It ends fads early and sees seasons coming. It under-calls slow climbs.

Here is my rule of thumb. It is a suggestion, not something I tested. Buy deep only when surprise and growth both say yes and LightGBM does not say "down." For the Gadget, LightGBM turned off on day 68. That is the day to stop buying.

[ THE SMALL PRINT ]

This is a demo, not a benchmark. One seed, one run, five products. The season repeats exactly. Every fake fad has the same shape, which is the best case for LightGBM. Real catalogs are messier, so test on your own products and your own seasons before you trust a number from this post.

[ ELSEWHERE ]

GitHub · LinkedIn · Dev.to · Substack · 4thwithme.dev/blog

May the --force be with you. See you next week.