DEV Community

Multigrid
Multigrid

Posted on • Originally published at multigrid.ai

Reconstructing a User Session From Clickstream Events

There is no session in your data. There is a stream of timestamped events with an identifier attached, and a session is whatever rule you apply to cut it up. The rule is almost always an inactivity timeout, and every metric you report depends on the number you choose.

A session is a definition, not a fact

The standard definition is: a session is a maximal run of events from one identity in which no two consecutive events are separated by more than the timeout. Google Analytics 4 uses 30 minutes as its default session timeout, adjustable in the property settings — see Google’s own documentation. That 30 minutes is a convention inherited from early web analytics, not a measured property of human attention, and it is the reason “sessions” from two tools rarely agree.

It matters because the timeout sits in the denominator of everything. Sessions per user, pages per session, conversion rate per session, average session duration — halve the timeout and sessions go up, pages per session goes down, and conversion rate per session goes down, all without any change in user behaviour. Before comparing a session metric across tools or across a migration, check that both sides used the same timeout, because a difference there explains more discrepancies than any other single cause.

Sessionising a worked sequence

One user, one day, with a 30-minute timeout. Times are event time in the user’s own clock, and gaps are shown between consecutive events.

event                        time      gap from previous
1  /home                     09:00:00   —
2  /search?q=boots           09:00:41   41 s
3  /product/884              09:02:10   1 m 29 s
4  /product/884  (scroll)    09:03:55   1 m 45 s
5  /cart                     09:31:20   27 m 25 s   < 30 m -> same session
6  /checkout                 10:05:02   33 m 42 s   > 30 m -> NEW session
7  /checkout/payment         10:06:30   1 m 28 s
8  /order/confirm            10:08:11   1 m 41 s

session A: events 1-5   09:00:00 -> 09:31:20   duration 31 m 20 s
session B: events 6-8   10:05:02 -> 10:08:11   duration  3 m  9 s
Enter fullscreen mode Exit fullscreen mode

Look at what that split did. The user browsed, added to cart, went to lunch, came back and bought. The rule assigns the purchase to session B, so session A — the one that contains all the product research — converted at 0%, and session B converted at 100% with a single page of “discovery”. Any attribution model reading sessions as units of intent has now credited the wrong one. This is not a bug in the sessionisation; it is the cost of the definition, and it is why user-level and session-level analyses answer different questions.

Note also that the duration of session A is 31 minutes 20 seconds even though the user was gone for 27 of them, and that the duration of a one-event session is zero by construction — there is no second timestamp to subtract. That single fact is why average session duration is systematically dragged down by bounce traffic and why “engaged sessions” with a minimum-duration or minimum-event-count filter exist at all.

Doing it in SQL and in a stream

In batch, sessionisation is three window functions and no joins. The pattern is worth memorising because it is the same in every SQL dialect that has LAG and a cumulative SUM.

WITH gapped AS (
  SELECT
    user_id,
    event_time,
    page,
    CASE WHEN event_time - LAG(event_time) OVER w > INTERVAL '30 minutes'
           OR LAG(event_time) OVER w IS NULL
         THEN 1 ELSE 0 END AS is_new_session
  FROM events
  WINDOW w AS (PARTITION BY user_id ORDER BY event_time)
)
SELECT
  user_id,
  SUM(is_new_session) OVER (PARTITION BY user_id ORDER BY event_time
                            ROWS UNBOUNDED PRECEDING) AS session_index,
  event_time, page
FROM gapped;
Enter fullscreen mode Exit fullscreen mode

The cumulative sum over a 0/1 “new session” flag is the whole trick: it increments exactly at the boundaries and is constant between them, so it is the session number. Hash it with the user id to get a stable session id.

In a stream processor, the same thing is a session window with a 30-minute gap, which is one line — but the semantics are different in a way that matters. A session window cannot be emitted until the gap has fully elapsed with no further events, so a live session is never available; and a late-arriving event can merge two sessions that were already emitted, which requires a sink that supports retraction. The mechanics of that merge are worked through in windowing strategies. If you need a live view of an in-progress session, keep the state keyed by user with a 30-minute TTL and read it directly, rather than waiting for the window to close.

Four cases that break the count

  • Client clock skew. If event times come from the browser, some fraction of your users have clocks minutes or years wrong, and a single event timestamped 1970 or 2038 destroys the ordering for that user. The standard fix is to record both the client time and the server receipt time, compute the offset per session from the first event, and correct the rest — but never to trust client time unadjusted for boundary decisions.
  • Background activity. A tab left open that polls, a service worker that syncs, or an autoplaying video keeps emitting events with no human present, so the timeout never trips and one “session” runs for nine hours. Exclude non-interaction events from the gap calculation — they can still be recorded, they just must not reset the clock.
  • Midnight and timezone splits. Many pipelines partition by day, and a session crossing the partition boundary is cut in two by the storage layout rather than by the rule. If daily partitioning is unavoidable, decide explicitly whether a session is attributed to the day it started or the day it ended, and apply it everywhere.
  • Bots and synthetic traffic. A crawler produces perfectly regular inter-event gaps, often faster than any human, and it will happily generate the highest pages-per-session in your dataset. Filter before sessionising, not after, because a bot included in the baseline distorts every percentile you might later use to detect one.

Identity is the harder half

Everything above assumes a stable identity to partition by. In practice you have a device-scoped cookie or an app install id before login and a user id after, and the same person on a phone and a laptop is two identities that later resolve to one. Sessionising on the pre-login id and then stitching means a session can change its owner retroactively, which is fine for a warehouse that can recompute and disastrous for a streaming aggregate that already emitted.

The workable design is to key sessions on the anonymous id — which never changes mid-session — and carry the resolved user id as an attribute that can be filled in later. That keeps the session boundary decision independent of identity resolution, so a late login updates an attribute rather than re-partitioning the stream. Session identity also has legal weight: a session id joined to a persistent user id is personal data in most jurisdictions, and retention rules apply to the joined table even where the raw event stream is considered pseudonymous. Decide the retention period at the same time you decide the timeout.

Once sessions exist, the ordered page sequence inside each one is the input to next-action modelling, which is where the boundary rule stops being a reporting convention and starts affecting a prediction.

Related

Top comments (0)