Archiving Analytics with D1 and Workers

About three months into building my behavioral analytics SaaS, I ran into a good problem: the data was piling up. I wanted to preserve its value while keeping my storage costs as low as possible. Recent activity gives me a detailed snapshot, while older data contributes to a longer historical record.

Let's say someone is running my SDK on their domain and gets 500 human visits a day. I track behavior separately for each page. If someone visits the landing page, then /pricing, then /tacos, that's three D1 rows. At an average of ten page views per visit, 500 visits becomes 5,000 new rows.

You can see how this adds up quick.

The payload for each row is rather small. It doesn't contain raw clicks or individual interaction events, just a compact JSON summary of behavioral shapes and metrics. Still, small rows stop feeling small when you keep creating them forever.

I designed two Cron-driven processes to handle this: the Archivist and the Janitor. The Archivist preserves useful historical statistics, while the Janitor removes detailed behavioral rows once they are safe to delete.

When does the data become safe to archive?

First, let's look at the timing of these two processes. The SDK observes interaction signals during a page visit. When change_view occurs, meaning the visitor leaves or replaces the current view, those signals are run through the Shape Engine and then purged. A compact summary, typically less than 1 KB, is sent through a first-party route back to D1.

Each summary belongs to a Page Session, and several Page Sessions can belong to the same Domain Session. The client keeps that Domain Session identity for up to two hours, while the server allows additional time for its page summaries to arrive and become final.

Once the server knows the Domain Session is complete, it can close it and make it eligible for the Archivist.

Here is a simplified version of the client-side lifetime check:

const LIFETIME = 2 * 60 * 60 * 1000;
const now = Date.now();

let domainSession = JSON.parse(
  sessionStorage.getItem("domain_session")
);

if (!domainSession || now - domainSession.startedAt >= LIFETIME) {
  domainSession = {
    id: crypto.randomUUID(),
    startedAt: now
  };

  sessionStorage.setItem(
    "domain_session",
    JSON.stringify(domainSession)
  );
}

Using sessionStorage keeps the Domain Session tied to a single browser tab instead of accidentally combining activity across multiple tabs.

The Cron schedule itself does not make data safe to archive. It only decides when the Archivist wakes up. The database state decides what the Archivist is allowed to process.

That distinction is important. A Cron execution might run late, run twice, or retry after a failure. The Archivist should get the same answer each time it asks: which Domain Sessions are complete, and which of them have not been applied to the archive?

The Archivist starts with a bounded query similar to this:

SELECT domain_id, domain_session_id, closed_at
FROM open_domain_sessions
WHERE status = 'CLOSED'
  AND closed_at IS NOT NULL
ORDER BY closed_at
LIMIT 50;

Once the Archivist successfully records the contribution, the Domain Session moves from CLOSED to APPLIED, so it will not be selected again.

The same principle applies to the Janitor. An old row is not automatically a deletable row. The Janitor requires two separate facts:

1. The row has passed its retention period.
2. Its contribution to the historical archive has already been applied successfully.

If either condition is false, the detailed row stays in D1.

This gives the data two separate lifetimes. Detailed analytics remain available while they are recent and useful for inspecting individual activity. The derived statistics can remain useful much longer without requiring me to retain every detailed source row.

With the timing and eligibility rules established, we can look at what the Archivist actually preserves.

What does the Archivist preserve?

The Archivist works in daily windows. Once a window is ready, it takes the completed Domain Sessions within that period and rolls their contributions into a durable daily statistical archive.

It does not create a historical copy of every Page Session. The detailed Page rows are temporary. Their useful statistics survive, but the individual browsing detail does not.

The archive preserves daily totals and statistical dimensions such as:

• Page and Domain Session counts
• qualifying, Undetermined, filtered, and capacity-dropped outcomes
• zero-event Page counts
• total and behaviorally eligible event counts
• device composition
• weighted behavioral shapes and metrics
• schema and calculation versions

This is more useful than saving one grand total. If I stored only an average shape or a label like mixed, I would lose the information needed to combine days accurately later.

For the behavioral values, I preserve weighted numerators and their denominator:

historicalAverage =
  totalWeightedNumerator / totalEligibleEvents;

That means daily archives can be combined into weeks, months, or longer ranges without retaining every detailed Page Session that produced them.

The Archivist also records an immutable application fact for each completed Domain Session it processes. That fact is not another copy of the browsing history. It is proof that a specific closed contribution was applied to a specific daily archive exactly once.

Updating the daily archive, creating that application proof, and moving the Domain Session from CLOSED to APPLIED happen atomically.

That is what makes retries safe.

Once that proof exists and the retention period expires, the Janitor can safely remove the detailed rows. The Archivist preserves their statistical value; the Janitor makes sure I don't pay to store the original detail forever.