On one Walmart store, a machine-learning forecast was 3.9 percent worse than a 28-day moving average on absolute error, and the worse forecast for 58 percent of the 823 items. Split by demand class, the result is two opposite ones: it wins on the steady sellers and loses by about 10 percent on the intermittent and lumpy items, which are most of the store.
The usual way to choose a forecast is to compare one error number per model and pick the smallest. This note shows, on real retail data, why that number alone can point the wrong way for most of the catalogue. Everything below was measured with forelint, the open-source library I wrote for auditing forecast backtests, and the script that produces every figure is in its repository.
The setup
The data is the public M5 competition set: daily unit sales from Walmart. I took one store, CA_1, and one department, FOODS_3: 823 items. Each forecast covers the next 28 days, and the backtest repeats it at ten forecast origins 28 days apart, so the ten forecasts do not overlap and every comparison rests on ten separate results.
Two forecasts compete. One is a gradient-boosting model from scikit-learn, trained on all items together, with six features: the item, the day of the week, sales 28 and 35 days earlier, and 7- and 28-day averages of those. It is deliberately plain. The other is a 28-day moving average: each item's mean daily sales over its last four weeks, repeated forward.
Overall
On absolute error the model is worse. Its MAE was 1.742 against 1.676 for the moving average, a forecast value added of −3.9 percent, and it was the worse of the two on 58 percent of items.
On squared error it looks slightly better: RMSE 3.44 against 3.51. But a Diebold-Mariano test across the ten origins gives p = 0.20, so that difference is not established. Squared error weights big misses heavily, and big misses happen on the items that sell a lot; the RMSE versus MAE note works through why the two can disagree.
By demand class
The Syntetos-Boylan classification sorts each item by how often it sells and how much the sale sizes vary, using its history before the first forecast: smooth, erratic, intermittent or lumpy. Breaking the same backtest down by class, on absolute error:
| Demand class | Items | Value added vs moving average | Items where the model is worse |
|---|---|---|---|
| Smooth | 152 | +5.0% | 23% |
| Erratic | 33 | +1.4% | 33% |
| Intermittent | 463 | −10.5% | 68% |
| Lumpy | 171 | −9.2% | 69% |
The model earns its keep on the steady sellers and loses about a tenth of its accuracy on everything that sells in bursts.
The model wins on the 152 smooth items and loses on the 463 intermittent and 171 lumpy ones, which together are 77 percent of the store. On about two thirds of the intermittent and lumpy items, the moving average was the better forecast.
That changes the decision. The overall number says the moving average is a little better. The breakdown says use the model for the steady sellers, where it is better, and a simpler forecast or a method built for intermittent demand for the rest. One model for every item is the choice the overall number hides.
Two more things worth knowing
Bias, item by item. Across the store the model's bias was small, −0.6 percent of sales. But its mean error was significantly different from zero on 24 percent of items, against under 1 percent for the moving average: too high on some items and too low on others, cancelling out in the total. Item-level bias is what turns into empty shelves and overstock, as the forecast bias note works through.
Concentration. The 82 items with the largest error, a tenth of the store, held 74 percent of the model's squared error. Improving the squared-error score mostly means fixing that list, and the audit names the items.
What I would check on any forecast
- Value added against a simple baseline, on the error measure that matches the cost of a miss.
- The same comparison per demand class. Steady items and intermittent items reward different methods, and one average mixes them.
- Whether a difference is real: a Diebold-Mariano test or a model confidence set across several forecast origins whose windows do not overlap. The days of one forecast are not independent, and overlapping windows share target days.
- Bias per item, not only in total, and which items carry the error.
forelint does all four in one call, on the cross-validation table most forecasting code already produces. It installs with pip install forelint. The full M5 script is in its repository, and the story of how the library started is in the note on how my validation ranked five models backwards.
Updated 8 November 2026: re-measured with forelint 0.2.0 at ten forecast origins whose windows do not overlap. Earlier versions of this note used a single origin, then overlapping origins, neither of which supports a valid significance test.
First published on davidmashiah.com.
Top comments (0)