If your model predicts sequences and you evaluated it on a random split, your accuracy number is probably lying to you.
Here's the leak. You shuffle your data and carve out a random test set. But if behavior unfolds over time, a random split drops future events into the training set and past events into the test set. Your model gets to peek at what comes later, then gets graded on what came before. The score looks fantastic and means very little.
If instead you split on time — train on the past, test strictly on the future, the way the model will actually be used — the number drops. And that lower number is the honest one. It's the only one that tells you whether the thing will work in production.
I held that line building a sequential recommender: evaluated on a temporal, leave-last-out split so nothing from the future ever leaked backward, with a simpler baseline kept alongside so every claim was measured against something real.
An unflattering metric you trust beats a beautiful one you can't. When a result looks too good, suspect your split before you celebrate.
More ML notes and projects → www.divyakush.com
Top comments (1)
This is exactly where the evaluation loss function matters. A random split is often testing interpolation over a shuffled history, while production asks for extrapolation into the next slice of time. I like the baseline point too because it makes the embarrassment measurable.