Counting visitors without cookies or storing IP addresses

We wanted three numbers for RadarMakers: how many pages were viewed, roughly how many people viewed them each day, and where they came from. We did not want a cookie banner, a third-party script, or a table full of IP addresses. This post explains the implementation, a Go backend with Postgres and a Next.js frontend, and its limits.

What gets stored

We do not store a separate event row for every page view. Every page view increments counters in a set of aggregate tables, each keyed by the UTC day: views and visitors per day, views per hour, per page, per referrer host, per country, per device class, per campaign, and per pair of internal pages for "came from this page to that page". A page view becomes "+1" in several of those tables, and there is no row that ties them together.

The one table that is not a pure counter is analytics_seen:

CREATE TABLE IF NOT EXISTS analytics_seen (
    day     DATE NOT NULL,
    visitor TEXT NOT NULL,
    PRIMARY KEY (day, visitor)
);

It exists only to tell whether a visitor was already counted today. The visitor column holds a daily hash, never an IP address.

The browser side: a beacon and nothing else

A small client component in the Next.js layout attempts one beacon each time the path changes to a page we count, skipping a repeat of the same path:

// components/PageTracker.tsx, excerpt
const first = countedPath === null;
const ref = first ? document.referrer : window.location.origin + countedPath;
// ... campaign comments omitted
const params = new URLSearchParams(window.location.search);
const src = first ? params.get("ref") || params.get("utm_source") || undefined : undefined;
countedPath = pathname;
const body = JSON.stringify({ path: pathname, ref, src });
navigator.sendBeacon("/api/v1/track", new Blob([body], { type: "application/json" }));

It writes no cookie and nothing to local storage. The only state is two module variables that live as long as the loaded page. In a single page app document.referrer stays the same after client side navigation, so the real referrer is sent only for the first counted page; after that the component sends the previous counted page of our own site. The server records that as an internal move when the previous page is a valid, counted path different from the current one.

Before sending anything, the component checks a few things:

// lib/track-optout.ts, called from components/PageTracker.tsx
export function skipTracking(nav: TrackNavigator): boolean {
  return nav.doNotTrack === "1" ||
    nav.globalPrivacyControl === true ||
    nav.webdriver === true;
}

If Do Not Track is "1", Global Privacy Control is true or navigator.webdriver is true, nothing is sent. Browsers driven by automation, such as Playwright or Selenium, set navigator.webdriver; we added that check after our own page tests started to inflate the counts. MDN marks navigator.doNotTrack as deprecated and non-standard, part of a discontinued specification, and points to Global Privacy Control as one of the alternatives browsers are exploring. We choose to skip counting when either privacy signal is present. The component also waits until it knows whether the user is logged in and sends nothing for administrators; that exclusion lives in this client component, not in the public endpoint. Both the component and the server skip the admin panel, account pages and the API.

The server side: filter, derive, forget

The Go handler answers 204 No Content, also when it drops a request, so the response does not reveal the filter rules; the route's rate limiter in front of it can still answer 429. The handler reads up to 2048 bytes of the body for parsing, then reads and discards up to another 1 MiB before answering. It drops empty and bot-like User-Agents, such as anything containing bot, crawl, headless or curl, and paths it does not count. It cuts the query string and fragment off the path, so /products/x?secret=1 is counted as /products/x.

Referrers are reduced to a host. A visit from a search results page is stored as google.com, not as the full URL with the search terms. A referrer from our own site is not a source at all; it becomes the "from" side of an internal move.

Then come the two personal inputs, the IP address and the User-Agent. They are used in memory for three things and then dropped:

• the country, looked up in a local database file, or ?? when unknown;
• the device class, desktop, mobile or tablet, from a few substrings of the User-Agent;
• the daily visitor hash.

// internal/handler/analytics.go
func visitorHash(secret string, at time.Time, ip, ua string) string {
    sum := sha256.Sum256([]byte(secret + "|" + at.UTC().Format("2006-01-02") + "|" + ip + "|" + ua))
    return hex.EncodeToString(sum[:])[:32]
}

The hash covers an application secret, the UTC date, the IP address and the User-Agent, and only the first 32 hex characters are kept. Requests with the same IP address and the exact same User-Agent string get the same hash within a UTC day, as long as the secret does not change, and a different hash on the next day, because the date is part of the input.

Counting a unique visitor with one insert

The repository records a page view in one transaction. The first statement decides whether this is a new visitor for the day:

// internal/repository/analytics.go, excerpt
res, err := tx.ExecContext(ctx,
    `INSERT INTO analytics_seen(day, visitor) VALUES($1::date, $2) ON CONFLICT DO NOTHING`, day, e.Visitor)
if err != nil {
    return err
}
n, err := res.RowsAffected()
if err != nil {
    return err
}
newVisitor := int(n)

If the row is new, RowsAffected is 1; if the hash was already there, the conflict is ignored and it is 0. That number is then added to the daily visitors counter and the per-country visitors counter, while views always get 1:

INSERT INTO analytics_daily(day, views, visitors) VALUES($1::date, 1, $2)
ON CONFLICT (day) DO UPDATE SET views = analytics_daily.views + 1, visitors = analytics_daily.visitors + $2

The other counters follow the same INSERT ... ON CONFLICT DO UPDATE pattern. Since paths and referrer hosts come from strangers, the tables for pages, referrers, internal moves and traffic sources have a daily threshold of distinct values, for example 5000 paths. Once a threshold is reached, new values go into a single "other" row instead. The check and the insert are separate queries, so concurrent requests can overshoot a threshold a little; it is a guard against flooding, not an exact limit.

Cleaning up

A goroutine started with the server runs the cleanup at startup and then once an hour. Each run deletes every analytics_seen row from an earlier UTC date and aggregate rows older than 400 days. This follows UTC dates, not an exact 24 hour lifetime: a hash written at 23:59 UTC stays until the first successful run after midnight, which is usually within the hour.

What this does and does not promise

• It is an estimate. Two people whose requests come from the same IP address with the exact same User-Agent string count as one visitor. One person whose IP address changes during the day, for example when switching from Wi-Fi to mobile data, usually counts as two. A visitor who opts out or blocks the beacon is not counted at all.
• The hash is not anonymous to us. Anyone with the application secret and a guess of someone's IP address and User-Agent could recompute that day's hash and see whether it is in the table. That is part of why the hashes are deleted by the first cleanup after the UTC day ends. The aggregate tables hold no hashes.
• The claim is about the analytics tables. The IP address and User-Agent are not written to any of them. Your reverse proxy, web server or hosting provider may still keep access logs with IP addresses; check those separately if you make a promise to your users.
• The counters use no stable visitor key across days. They do not report returning visitors or individual browsing histories, only single internal moves from one page to another.

How we test it

The handler test runs against a real disposable Postgres database. It sends rejected requests and checks that nothing was counted, sends realistic ones, moves an injected clock to the next day, and finally reads every row of the analytics tables it lists as text and fails if any of them contains the test IP address or the strings Firefox or Mozilla from the test User-Agent:

// internal/handler/analytics_test.go, excerpt
for r.Next() {
    var line string
    r.Scan(&line)
    rows++
    if strings.Contains(line, testIP) || strings.Contains(line, "Firefox") || strings.Contains(line, "Mozilla") {
        t.Errorf("%s stores personal data: %s", tbl, line)
    }
}

It also fails if there were no rows to inspect, because a check that looks at an empty database proves nothing. A separate unit test checks that the hash is stable within a day and changes with the date, the IP address, the User-Agent and the secret.

Takeaways

• Store counts, not visits. A row per page view is a log, whatever you call it.
• If you need daily uniques, a hash of a secret, the date, the IP and the User-Agent, deleted after the day ends, plus an INSERT ... ON CONFLICT DO NOTHING, gives you an estimate without a cookie.
• Reduce referrers to hosts and strip query strings from paths before they reach the database.
• Put a threshold on distinct values in tables that strangers can fill with made-up paths and hosts.
• Write the test that greps your tables for the test IP address and User-Agent. It can catch accidental storage of those values in the tables it scans.