Summary
Can you tell, a few clicks into a shopping session, whether the visitor will buy? I built a ladder of eight models, from guessing the base rate to a transformer with self-supervised pre-training, and scored 4.9 million anonymous shopping sessions after 1, 2, 3, 5 and 10 clicks. The comparisons were fixed before the test window was opened, and it was opened once. In the first few clicks, gradient boosting on about two dozen hand-built summary features beat every neural model; by ten clicks, every sequence model had passed it. The validation window showed a similar trend.
Key findings
- At one click, every model ties. All four are within about 1% of each other: a single click carries little signal, and no model can do more with it.
- Gradient boosting's lead is largest at two clicks: the sequence models trail by 6–7% on the test window and 5–6% on the validation window.
- At three clicks, the pre-set test favors gradient boosting. The pre-trained transformer is 3.6% behind (Holm-adjusted p = 0.002), and the LSTM and plain transformer also trail. On the validation window, the models are level there.
- By ten clicks, all three sequence models lead. The LSTM is 6.9% ahead on the test window and 6.2% on the validation window.
- Masked-event pre-training gave no detectable gain: −0.2% at three clicks with all labels (p = 0.66), and +0.7% with 10% of the labels (Holm-adjusted p = 0.23).
- What wins is a model plus its inputs. Gradient boosting reads hand-built features with exact timing; the sequence models read click tokens. The results compare those two approaches, not the models alone.
What's new here. Earlier studies found that hand-built features do well early and sequence models gain on longer sessions. To my knowledge, this is the first study to measure that crossover on identical session prefixes at fixed click counts, showing the sequence models actually pass gradient boosting and that its edge peaks at two clicks, and the first to test masked pre-training for in-session purchase prediction on short anonymous sessions. Section 6 sets out how it differs from prior work.
1. Background
A visitor lands on an online store and clicks three times. Will they buy?
That question sits behind a lot of what stores want to do: which visitors to show an offer, which to leave alone, which page changes to test on whom. It is also an old benchmark. The RecSys Challenge 2015 asked teams to predict purchases from anonymous click sessions (Ben-Shimon et al., 2015), and the winning entry was built on gradient boosting over hand-built features (Romov and Sokolov, 2015). Since then, models of behavior have moved to sequences: recurrent networks for sessions (Hidasi et al., 2016), then self-attention (Kang and McAuley, 2018; Sun et al., 2019), then transformers pre-trained on large behavior logs before being tuned for a task (Li et al., 2022; Pancha et al., 2022). Recent work on purchase intention uses transformers too (Feng and Liu, 2026). So in 2026 the reflex answer is a transformer. Sessions are sequences, transformers eat sequences, so feed it the clicks.
There are reasons to doubt the reflex. Tree ensembles still beat deep networks on many tabular problems (Grinsztajn et al., 2022), and a careful re-run of recent neural recommenders found that simple, well-tuned baselines often matched or beat them (Ferrari Dacrema et al., 2019). The pre-training gains were also reported for users with long histories, not for anonymous sessions a few clicks long.
I wanted to know whether the reflex holds up on short, anonymous sessions, with every model given a fair chance and the test data opened only once.
The interesting part is not who won. It's that the answer depends on how far into the session you ask.
2. Setup
Data. A public, research-only e-commerce data set released for the SIGIR eCom 2021 data challenge (Tagliabue et al., 2021): anonymous browsing sessions from a real online shop. After cleaning, about 4.9 million sessions, and about 1 in 93 ends in a purchase.
Task. Score each session after its first k clicks, for k = 1, 2, 3, 5 and 10, using only those clicks. The label is whether that session ends in a purchase. A session that bought within its first k clicks has nothing left to predict, so it drops out at that cut-off.
Split by time, not at random. The data covers about three months. Train on the first nine weeks (3.8 million sessions), tune on the next two weeks (the validation window, 0.5 million sessions), and test on the final two weeks (the test window, 0.6 million sessions), which stayed closed until every decision was made. Sessions that cross a window boundary are dropped, and a one-hour gap separates the windows (3,334 sessions removed in all), so no session lands in two windows. A random split lets the model peek at the future; a time split is what a live system actually faces.
Metric. Buyers are rare, so accuracy is useless ("nobody buys" is 99% accurate) and ROC-AUC flatters weak models. I report what I'll call average lift. Lift is the familiar e-commerce ratio: the conversion rate in a group you pick, divided by the average. Here I rank all sessions by the model's score, take the lift at each buyer's position in that ranking, and average over all buyers. Mathematically it's average precision divided by the base rate. An average lift of 1.0 means no better than guessing; 4.0 means four times better. Neural models are averaged over five seeds.
Statistical tests, fixed in advance. Before opening the test window, I wrote down which comparisons would be tested, all at three clicks:
-
Two main comparisons, with p-values adjusted for testing two things at once (Holm's method):
- the pre-trained transformer against gradient boosting
- the pre-trained transformer against the same transformer without pre-training, with only 10% of the labels
-
Four secondary comparisons, without that adjustment:
- the LSTM against gradient boosting
- the plain transformer against gradient boosting
- the pre-trained transformer against the plain one, with all labels
- the pre-trained transformer against gradient boosting, with 10% of the labels
- How each test works: a paired bootstrap. Resample the test sessions 1,000 times, recompute both models' scores on the same resample, and read a 95% interval and a p-value off the spread of the differences.
- Everything else, including all results at other cut-offs, is reported as descriptive.
3. The model ladder
Each rung has to beat the one below it to justify its cost.
- Base rate. Every session gets the same score.
- Lookup rules. Score a session by the purchase rate of sessions like it: same last action, cart or no cart, same time spent, or the exact sequence of the first few clicks.
- Logistic regression on summary features: action counts, the last action, the last two actions, cart status, timing (total time, longest gap, average gap), and how often the shopper returns to a product they've already seen.
- Gradient boosting on the same features.
- A decaying action state. Each click fades the memory of earlier ones and reinforces its own kind. It's adapted from a user-interest model I built in 2014, and it has two settings, both tuned on validation.
- LSTM. One layer, 64 units, reading the clicks in order.
- Small transformer. Two layers, four attention heads, trained from scratch.
- The same transformer with masked-event pre-training. First it learns to fill in hidden clicks on about a million unlabeled sessions, then it's fine-tuned to predict purchases.
Every click becomes one token that combines three things: the action (page view, product view, add to cart, or remove from cart), how long the shopper paused before it (six bins, from first click and under 5 seconds up to over 10 minutes), and how the product relates to earlier clicks (no product, the same product as the previous click, one seen earlier in the session, or a new one). That gives 96 possible tokens.
What's actually being compared. Every model starts from the same click data, but not the same inputs. Gradient boosting gets hand-built summary features, including exact timing in milliseconds. The neural models get the tokens above, with pauses in six bins, and learn their own features from them. So this compares two complete approaches, model plus its inputs, not the model alone. When one wins, the credit is shared between the model and what it was given.
4. Results
4.1 At three clicks, gradient boosting wins
On the test window, at three clicks:
| Model | Average lift |
|---|---|
| Gradient boosting | 3.85 |
| LSTM | 3.76 |
| Transformer | 3.72 |
| Pre-trained transformer | 3.71 |
| Logistic regression | 3.12 |
The first main comparison was the pre-trained transformer against gradient boosting. The transformer came out 3.6% behind, with an interval that excludes zero (Holm-adjusted p = 0.002). So the gap is real: gradient boosting is better here, by a small margin. The plain transformer and the LSTM also trailed.
The reason isn't mysterious. After three clicks there is barely a sequence to model. Whether the shopper has added something to the cart, what they did last, and how fast they're moving already carry most of the signal, and a tree ensemble over well-chosen summaries extracts it cheaply.
These are test-window numbers. On the validation weeks the three-click gap was much smaller, close to a tie; I come back to that in Section 4.3.
4.2 The crossover
Now ask the same question later in the session. Here is each sequence model's gap to gradient boosting in average lift, on the test window:
| Model | 3 clicks | 5 clicks | 10 clicks |
|---|---|---|---|
| LSTM | −2.4% | +2.7% | +6.9% |
| Transformer | −3.4% | 0.0% | +2.8% |
| Pre-trained transformer | −3.6% | +0.4% | +4.1% |
At one click every model is within 1% of gradient boosting. At two clicks the sequence models trail most, about 6–7% behind, then close the gap by five clicks and lead at ten.
By five clicks the field is level. By ten, all three sequence models lead, and the LSTM is about 7% ahead.
Statistical significance tests. As planned (Section 2), the tests are at three clicks. Tests chosen after seeing the results, also called post hoc tests, would give unreliable p-values at other cut-offs. At three clicks all three gaps were significant (LSTM p = 0.036; transformer and pre-trained transformer p = 0.002). The larger gaps at ten clicks are untested, not non-significant. The five- and ten-click gaps are descriptive. They are consistent across seeds and across both windows, but I didn't test them, so I'd call them a strong pattern rather than a result.
Also note what the comparison is: by cut-off point, not by session length. It says sequence models gain as you watch more of a session. It does not say they work better on long sessions in general.
4.3 Does it hold up? A similar trend on the validation window
One test window is one stretch of weeks, and a pattern could belong to those weeks rather than to the problem. So here is the same chart on the validation weeks, the period just before the test window, drawn on the same scale:
The shape repeats. At one click the models are within 1% of each other. At two clicks the sequence models trail most, 5 to 6% behind. By ten clicks all three lead again, in the same order as on the test window: the LSTM first, then the pre-trained transformer, then the plain one. The gaps are close in size too, even though every model scored about 25% higher on validation than on test. The weeks got harder; the ranking didn't move.
The one place the two windows disagree is three clicks. On the test window gradient boosting led by 2 to 4%, and the pre-set test confirmed it. On validation the four models are within 1% of each other. So gradient boosting's early edge shows at two clicks in both windows, but at three clicks in only one. I'd read Section 4.1 as "at three clicks, gradient boosting is ahead or level," and treat the dip at two clicks as the clearer sign of where hand-built features are strongest.
A caution on what this check can do. The validation weeks were used for tuning, such as the two settings of the decaying action state, so they aren't a second untouched test. They can't confirm a number. What they can show is whether the shape survives a change of weeks, and it does.
4.4 Pre-training: no detectable gain
Masked-event pre-training is the part I expected to matter most. Against the same transformer without it, on the test window:
| 3 clicks | 5 clicks | 10 clicks | |
|---|---|---|---|
| Gain from pre-training | −0.2% | +0.4% | +1.3% |
Three clicks is the tested point, and there the difference is well within noise (p = 0.66).
The other place pre-training is supposed to help is with scarce labels, and this was the second main comparison. I trained both versions on only 10% of the labels; at three clicks the pre-trained one was 0.7% ahead, with an interval that included zero (Holm-adjusted p = 0.23). Against gradient boosting trained on the same 10%, the pre-trained transformer was 1.3% behind, also not significant (p = 0.29). With scarce labels, the pre-trained transformer was neither clearly ahead of nor clearly behind either model.
The pre-training did learn something: a simple linear model on top of the frozen pre-trained representation beat logistic regression on the hand-built features (4.49 vs 4.00 on validation, at three clicks). It just didn't learn enough that the fine-tuned model couldn't learn on its own from about 2.1 million labeled prefixes, drawn from 820,000 training sessions.
The gain does creep up as sessions get longer, which is where you'd expect a model of click sequences to start earning its keep. With this much labeled data, though, it never became large.
5. What I'd do with real traffic
If I were building purchase-intent scoring for a live store, this is how I'd use these results:
- Start with gradient boosting. For the first few clicks it's the best model I tested, and it's cheap to train, fast to serve and easy to explain.
- Look at the session-length distribution before reaching for a sequence model. The case for an LSTM or a transformer lives in the later clicks. If most of your sessions end in five clicks, there's little to gain. If a good share run past ten, that's where to test one.
- Decide for yourself what size of gain is worth a switch. Whether a 3% or a 7% gain justifies a heavier model depends on your traffic, what a missed buyer costs, and the cost of serving the model. That bar belongs to the system you're building, which is why I report each difference with its size and interval rather than a pass or fail.
- Measure per store, not just overall. Base rates differ enormously between a grocery site and a furniture site, and a pooled score hides where a model fails.
- Use the score to read experiments, not just to target. Split visitors into low, middle and high intent and compare the A and B arms within each group. A change that helps hesitant shoppers can look like nothing on average.
The broader lesson is the one I keep relearning: the right model depends on where in the problem you are. Watching a session unfold, the useful question isn't "which model is best?" but "best at which click?"
6. Related work
Sequence models for sessions. Recurrent networks were applied to anonymous sessions to recommend the next item (Hidasi et al., 2016). Self-attention followed, reading the sequence left to right (Kang and McAuley, 2018) or, with a fill-in-the-blank objective, in both directions (Sun et al., 2019). My rung 8 borrows that masked objective, applied to session events instead of items. These models predict the next item; here the target is whether the session ends in a purchase.
Pre-training on behavior. UserBERT pre-trains a BERT-style model on e-commerce users' long- and short-term behavior and reports gains on several downstream tasks (Li et al., 2022). PinnerFormer learns user representations from long action sequences in production at Pinterest, training them to predict a user's engagement over the following weeks rather than the next action (Pancha et al., 2022). Both work with identified users and long histories. Masked training has also been used on anonymous sessions, but to predict the next item (de Souza Pereira Moreira et al., 2021). This study uses it for a different target, whether a short anonymous session ends in a purchase, and also tests it with scarce labels; here it gave no detectable gain.
Purchase prediction from clicks. The RecSys Challenge 2015 made session purchase prediction a public benchmark (Ben-Shimon et al., 2015), and its winning entry was built on gradient boosting (Romov and Sokolov, 2015). Sheil et al. (2018) compared LSTMs with gradient boosting on that data, broken down by session length, and found the LSTM closing the gap on sessions longer than ten clicks. On an earlier Coveo data set, Requena et al. (2020) found that hand-built features beat deep models when predicting from the first 5 to 14 clicks, while deep models did better on full sessions; their split was random. A recent leakage-aware benchmark found gradient boosting ahead of LSTMs and transformers when only pre-purchase clicks are used (Goetzinger et al., 2026). Mcformer uses a multi-channel transformer over clickstream attributes to predict purchase intention (Feng and Liu, 2026); it works with identified users on a marketplace, predicting whether the next action is a purchase, a different setup from in-session prediction on anonymous traffic.
How strong are simple baselines? Tree ensembles remain hard to beat on tabular data (Grinsztajn et al., 2022), and many neural recommenders lose to well-tuned simple baselines when re-evaluated (Ferrari Dacrema et al., 2019).
How this study differs. The direction of the main result is not new: hand-built features do well early, and sequence models gain as sessions get longer (Sheil et al., 2018; Requena et al., 2020). To my knowledge, this study adds three things:
- The crossover, measured on identical prefixes. Every model scores the same sessions after exactly 1, 2, 3, 5 and 10 clicks, so each comparison is like for like. Earlier work compared whole sessions of different lengths or started at five clicks. Measured this way, the sequence models do not just close the gap but pass gradient boosting, and gradient boosting's edge is largest at two clicks, after a tie at one.
- A test of masked pre-training for in-session purchase prediction. With all labels and with 10% of them, pre-training gave no detectable gain on short anonymous sessions.
- A design built to hold up: a time-based split with gaps between windows, tests fixed before the test window was opened once, paired bootstrap intervals on average precision, and the same shape on a second time window.
7. Scope
- One data set, one shop, one time split. The crossover point will move on other stores.
- Trained and tested, not deployed. No real-time serving or live traffic.
- Anonymous sessions, in-session purchases. The task is whether a session ends in a purchase. Sessions can't be linked to shoppers, so a purchase on a later visit is outside the task, and the model doesn't predict it.
- Modest neural models. Larger transformers or more pre-training data could shift the picture; with about 2.1 million labeled prefixes from 820,000 training sessions, these small ones were already competitive.
8. Conclusion
Which model predicts a purchase best depends on how far into the session you ask. On 4.9 million anonymous shopping sessions, gradient boosting on hand-built features led in the first few clicks, with its largest edge at two clicks. By ten clicks, the LSTM and both transformers had passed it. The pattern held on the test window and on the validation window before it. Masked pre-training, the step I expected to matter most, gave no detectable gain, even with scarce labels.
The direction agrees with earlier work. What this study adds is a measurement of the crossover on identical prefixes at fixed click counts, a test of masked pre-training for in-session purchase prediction, and a design meant to hold up: a time-based split, tests fixed in advance, and a test window opened once.
Two things come next. The comparison here is between packages, each model plus its inputs, so the next step gives every model the same inputs to separate the model from its features. The full method and the remaining analyses will go into a longer write-up on arXiv.
Data: Coveo, released for the SIGIR eCom 2021 data challenge (Tagliabue et al., 2021), used under its research terms. Code available on request.
References
- Ben-Shimon, D., et al. (2015). RecSys Challenge 2015 and the YOOCHOOSE dataset. Proceedings of the 9th ACM Conference on Recommender Systems (RecSys '15), 357–358.
- de Souza Pereira Moreira, G., Rabhi, S., Lee, J. M., Ak, R., and Oldridge, E. (2021). Transformers4Rec: Bridging the gap between NLP and sequential / session-based recommendation. RecSys '21, 143–153. https://doi.org/10.1145/3460231.3474255
- Feng, J., and Liu, H. (2026). Next, browse or purchase: A sequential and interpretable deep learning approach for predicting customer purchase intention. Information Sciences, 123144. https://doi.org/10.1016/j.ins.2026.123144
- Ferrari Dacrema, M., Cremonesi, P., and Jannach, D. (2019). Are we really making much progress? A worrying analysis of recent neural recommendation approaches. RecSys '19. https://doi.org/10.1145/3298689.3347058
- Goetzinger, P., Topić, P., Noy, S., and Huffstadt, K. (2026). Sequence-aware models for predicting online purchase conversion from clickstream data: A leakage-aware benchmark study with calibration, value, and interpretability. International Journal of Data Science and Analytics, 22(1). https://doi.org/10.1007/s41060-026-01199-2
- Grinsztajn, L., Oyallon, E., and Varoquaux, G. (2022). Why do tree-based models still outperform deep learning on typical tabular data? NeurIPS 2022, Datasets and Benchmarks Track. arXiv:2207.08815
- Hidasi, B., Karatzoglou, A., Baltrunas, L., and Tikk, D. (2016). Session-based recommendations with recurrent neural networks. ICLR 2016. arXiv:1511.06939
- Kang, W.-C., and McAuley, J. (2018). Self-attentive sequential recommendation. IEEE ICDM 2018, 197–206. arXiv:1808.09781
- Li, T., Cevahir, A., Cho, D., Gong, H., Nguyen, D., and Stenger, B. (2022). UserBERT: Modeling long- and short-term user preferences via self-supervision. arXiv:2202.07605
- Pancha, N., Zhai, A., Leskovec, J., and Rosenberg, C. (2022). PinnerFormer: Sequence modeling for user representation at Pinterest. KDD '22. https://doi.org/10.1145/3534678.3539156
- Requena, B., Cassani, G., Tagliabue, J., Greco, C., and Lacasa, L. (2020). Shopper intent prediction from clickstream e-commerce data with minimal browsing information. Scientific Reports, 10, 16983. https://doi.org/10.1038/s41598-020-73622-y
- Romov, P., and Sokolov, E. (2015). RecSys Challenge 2015: Ensemble learning with categorical features. Proceedings of the 2015 International ACM Recommender Systems Challenge.
- Sheil, H., Rana, O., and Reilly, R. (2018). Predicting purchasing intent: Automatic feature learning using recurrent neural networks. SIGIR 2018 Workshop on eCommerce.
- Sun, F., Liu, J., Wu, J., Pei, C., Lin, X., Ou, W., and Jiang, P. (2019). BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. CIKM '19, 1441–1450. https://doi.org/10.1145/3357384.3357895
- Tagliabue, J., et al. (2021). SIGIR 2021 E-Commerce Workshop Data Challenge. arXiv:2104.09423


Top comments (0)