DEV Community: Anthony Yerhot

Archiving Analytics with D1 and Workers

Anthony Yerhot — Thu, 03 Sep 2026 17:05:13 +0000

About three months into building my behavioral analytics SaaS, I ran into a good problem: the data was piling up. I wanted to preserve its value while keeping my storage costs as low as possible. Recent activity gives me a detailed snapshot, while older data contributes to a longer historical record.

Let's say someone is running my SDK on their domain and gets 500 human visits a day. I track behavior separately for each page. If someone visits the landing page, then /pricing, then /tacos, that's three D1 rows. At an average of ten page views per visit, 500 visits becomes 5,000 new rows.

You can see how this adds up quick.

The payload for each row is rather small. It doesn't contain raw clicks or individual interaction events, just a compact JSON summary of behavioral shapes and metrics. Still, small rows stop feeling small when you keep creating them forever.

I designed two Cron-driven processes to handle this: the Archivist and the Janitor. The Archivist preserves useful historical statistics, while the Janitor removes detailed behavioral rows once they are safe to delete.

When does the data become safe to archive?

First, let's look at the timing of these two processes. The SDK observes interaction signals during a page visit. When change_view occurs, meaning the visitor leaves or replaces the current view, those signals are run through the Shape Engine and then purged. A compact summary, typically less than 1 KB, is sent through a first-party route back to D1.

Each summary belongs to a Page Session, and several Page Sessions can belong to the same Domain Session. The client keeps that Domain Session identity for up to two hours, while the server allows additional time for its page summaries to arrive and become final.

Once the server knows the Domain Session is complete, it can close it and make it eligible for the Archivist.

Here is a simplified version of the client-side lifetime check:

const LIFETIME = 2 * 60 * 60 * 1000;
const now = Date.now();

let domainSession = JSON.parse(
  sessionStorage.getItem("domain_session")
);

if (!domainSession || now - domainSession.startedAt >= LIFETIME) {
  domainSession = {
    id: crypto.randomUUID(),
    startedAt: now
  };

  sessionStorage.setItem(
    "domain_session",
    JSON.stringify(domainSession)
  );
}

Using sessionStorage keeps the Domain Session tied to a single browser tab instead of accidentally combining activity across multiple tabs.

The Cron schedule itself does not make data safe to archive. It only decides when the Archivist wakes up. The database state decides what the Archivist is allowed to process.

That distinction is important. A Cron execution might run late, run twice, or retry after a failure. The Archivist should get the same answer each time it asks: which Domain Sessions are complete, and which of them have not been applied to the archive?

The Archivist starts with a bounded query similar to this:

SELECT domain_id, domain_session_id, closed_at
FROM open_domain_sessions
WHERE status = 'CLOSED'
  AND closed_at IS NOT NULL
ORDER BY closed_at
LIMIT 50;

Once the Archivist successfully records the contribution, the Domain Session moves from CLOSED to APPLIED, so it will not be selected again.

The same principle applies to the Janitor. An old row is not automatically a deletable row. The Janitor requires two separate facts:

The row has passed its retention period.
Its contribution to the historical archive has already been applied successfully.

If either condition is false, the detailed row stays in D1.

This gives the data two separate lifetimes. Detailed analytics remain available while they are recent and useful for inspecting individual activity. The derived statistics can remain useful much longer without requiring me to retain every detailed source row.

With the timing and eligibility rules established, we can look at what the Archivist actually preserves.

What does the Archivist preserve?

The Archivist works in daily windows. Once a window is ready, it takes the completed Domain Sessions within that period and rolls their contributions into a durable daily statistical archive.

It does not create a historical copy of every Page Session. The detailed Page rows are temporary. Their useful statistics survive, but the individual browsing detail does not.

The archive preserves daily totals and statistical dimensions such as:

Page and Domain Session counts
qualifying, Undetermined, filtered, and capacity-dropped outcomes
zero-event Page counts
total and behaviorally eligible event counts
device composition
weighted behavioral shapes and metrics
schema and calculation versions

This is more useful than saving one grand total. If I stored only an average shape or a label like mixed, I would lose the information needed to combine days accurately later.

For the behavioral values, I preserve weighted numerators and their denominator:

historicalAverage =
  totalWeightedNumerator / totalEligibleEvents;

That means daily archives can be combined into weeks, months, or longer ranges without retaining every detailed Page Session that produced them.

The Archivist also records an immutable application fact for each completed Domain Session it processes. That fact is not another copy of the browsing history. It is proof that a specific closed contribution was applied to a specific daily archive exactly once.

Updating the daily archive, creating that application proof, and moving the Domain Session from CLOSED to APPLIED happen atomically.

That is what makes retries safe.

Once that proof exists and the retention period expires, the Janitor can safely remove the detailed rows. The Archivist preserves their statistical value; the Janitor makes sure I don't pay to store the original detail forever.

From Traffic to Behavior: Why I Built Shape Beacon

Anthony Yerhot — Tue, 18 Aug 2026 01:05:30 +0000

I deployed my first public domain earlier this year. I learned something very quickly: traffic according to Cloudflare is defined by something as small as a wispy breeze coming from a crawling script, not the stampede of human feet.

I got pretty pumped up by the stats Cloudflare was feeding me. But I was skeptical and wanted more.

I started with a simple KV counter. It just counted how many views my splash page had.

Then I put those numbers next to Cloudflare's.

My worker: 23 page views.

Cloudflare: 649 unique visits.

Same site. Same week. A very different story.

I still wanted more data.

Not bounce rate, clicks, or page views.

I wanted something that actually measured the physical interaction happening on my site.

So I created my own layer of analytics: Shape Beacon.

With this tool, I can look at that same week and get a different picture. Shape Beacon detected meaningful behavioral interaction in only 10 sessions during that same period.

So what is it actually doing?

Shape Beacon temporarily collects browser interaction, reduces it into behavioral signals, and then throws away the raw event buffer.

_handleChunkLimit() {
    const events = this.collector.getRawEvents();
    this.chunkHistory.push(this._analyzeBuffer(events));

    this.collector.reset();
}

The important part isn't the event counter. It's what happens afterward: the raw interaction is used as input to the calculation, not retained as the analytics product.

From there, the engine looks at the physical characteristics of the interaction.

const dx = ev.x - prev.x;
const dy = ev.y - prev.y;
const dist = Math.hypot(dx, dy);

totalDist += dist;
velocities.push(dist / dt);

Distance, velocity, timing, and directional changes become behavioral signals.

Shape Beacon doesn't need to know what the user is looking at to describe how they interact with the page.

Those signals are reduced into four behavioral dimensions:

Intensity
Rhythm
Exploration
Coherence

The engine compares the four-dimensional point against tuned behavioral shapes.

for (const [shape, ideal] of Object.entries(this.tune.shapes)) {
    const dist = Math.hypot(
        metrics.intensity - ideal[0],
        metrics.rhythm - ideal[1],
        metrics.exploration - ideal[2],
        metrics.coherence - ideal[3]
    );

    distances[shape] = 1 / (dist + 0.0001);
}

The session is treated as a point in behavioral space and compared against the shapes that exist within that space.

The result is a small behavioral fingerprint.

What gets sent?

When the session ends, Shape Beacon sends a <1KB JSON payload containing the processed behavioral signal.

No raw coordinates, clicks, or keystrokes are stored.

It's a lightweight JavaScript SDK that can also be routed through a first-party proxy.

The goal is to extract more meaning from less data.

The problem I'm exploring with Shape Beacon is:
What did that interaction look like?

I've recently deployed Shape Beacon as a free tool. I'm also exploring where this approach fits within modern edge platforms and have submitted Shape Beacon for consideration to Cloudflare, Vercel, and Netlify. Those applications are currently pending, so we'll see where this experiment goes.

The goal was simple: respect the end-user's browser while still giving developers meaningful behavioral data.

Cheers,
Anthony

Shape Beacon

Behavioral Analysis: The Challenge of the Tab

Anthony Yerhot — Tue, 11 Aug 2026 12:02:52 +0000

Currently working on a Saas that reads a user’s behavior on a site. The goal is keeping it lightweight and privacy centered.

All interactions are recorded in the browser. When the browser hides the page, a compact summary is calculated and sent to a cloudflare worker for insertion into a D1 DB.

The exit events are strictly bound using this function.

// Bind exit triggers strictly once
if (this.config.endpoint && !this._hasSetupTriggers) {
    const handleExit = (event) => {
        if (this._hasSent) return; 
        if (document.visibilityState === 'hidden' || event.type === 'pagehide' || event.type === 'beforeunload') {
            this._hasSent = true;
            this.forceSend();
        }
    };
    document.addEventListener('visibilitychange', handleExit);
    window.addEventListener('pagehide', handleExit);
    window.addEventListener('beforeunload', handleExit);
    this._hasSetupTriggers = true;
}

2 attempts are made to send.

1st

const success = navigator.sendBeacon(this.config.endpoint, payload);

And 2nd fallback

if (!success) {
    fetch(this.config.endpoint, { method: 'POST', body: payload, keepalive: true }).catch(e => console.error(e));
}

But what about if the user returns to the tab?

The function keeps recording when they return. But it won't fire again because of the way I set up handleExit.

const handleExit = (event) => {
    if (this._hasSent) return; // <-- THE LOCK
    if (document.visibilityState === 'hidden' || event.type === 'pagehide' || event.type === 'beforeunload') {
        this._hasSent = true;      // <-- LOCKING IT
        this.forceSend();
    }
};

Because it locks hasSent to true I can't fire another beacon for the session.

I need to turn this._hasSent = false when they return so I can fire again.

const handleVisibilityChange = (event) => {
    // 1. User hides the tab, navigates away, or closes the browser
    if (document.visibilityState === 'hidden' || event.type === 'pagehide' || event.type === 'beforeunload') {
        if (this._hasSent) return; // Prevent duplicate immediate fires
        this._hasSent = true;      // Lock the beacon
        this.forceSend();          // Fire navigator.sendBeacon
    } 
    // 2. User returns to the tab
    else if (document.visibilityState === 'visible') {
        this._hasSent = false;     // Unlock the beacon so it can fire again later
    }
};

document.addEventListener('visibilitychange', handleVisibilityChange);
window.addEventListener('pagehide', handleVisibilityChange);
window.addEventListener('beforeunload', handleVisibilityChange);

But this creates another problem, multiple writes for the same session ID (primary key).

I use write or replace for duplicate session IDs. This takes the latest data and scrubs the old.

// --- 7. DATA INGESTION (shape_db) ---
// Force the Project ID to be the securely retrieved domain_whitelist.
const permanentProjectId = authData.domain_whitelist || 'unknown_domain';

// UPDATED: INSERT OR REPLACE handles tab-switching updates for the same sessionId
await env.DB.prepare(`
    INSERT OR REPLACE INTO sessions (
        id, project_id, schema_version, url, timestamp, 
        session_duration_ms, event_count, engine_version, tune_version, device, metrics, shapes
    ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`).bind(
    data.sessionId, 
    permanentProjectId, 
    data.payloadSchema,                 
    data.url || '',
    data.timestamp || new Date().toISOString(),
    data.sessionDurationMs,
    data.eventCount || 0,
    data.engineVersion || '',
    data.tuneVersion || '',
    data.device || 'unknown',
    JSON.stringify(data.metrics, null, 2), 
    JSON.stringify(data.shapes, null, 2)
).run();

Full project here:

Repository: View the code on GitHub
Live Site: Check out the SaaS

Hopefully my experience with users and their desire to open tabs and jump around like jack rabbits help.

Cheers