DEV Community

Daniel Pertu
Daniel Pertu

Posted on

Four caps on our local data, three different units, and the one directory with no cap is the only one measured in megabytes

Notifio is a desktop app that watches rental listing search pages and alerts you when a new listing appears. It runs on your own machine, which means everything it keeps is on your disk and nothing about it is my problem until a user emails me about it.

So I went looking for what bounds it. Not "what does it store", which the privacy audit I did earlier covered, but what stops each of those things growing forever.

The census

Everything lives under app.getPath('userData'), which is %APPDATA%\Notifio on Windows and ~/Library/Application Support/Notifio on macOS. There is one module that knows all of it:

export const DATA_DIR         = path.join(USER_DATA_DIR, 'data');
export const BROWSER_DATA_DIR = path.join(USER_DATA_DIR, 'browser-data');
export const CONFIG_PATH      = path.join(USER_DATA_DIR, 'config.json');
export const PROFILE_PATH     = path.join(USER_DATA_DIR, 'profile.json');
export const ENTITLEMENT_PATH = path.join(USER_DATA_DIR, 'entitlement.json');
export const RECIPES_PATH     = path.join(USER_DATA_DIR, 'recipes.json');
export const LEDGER_DIR       = path.join(USER_DATA_DIR, 'ledger');
export const FINDS_PATH       = path.join(USER_DATA_DIR, 'finds.json');
Enter fullscreen mode Exit fullscreen mode

Nine things once you expand the directories, and here is what holds each one down:

What Bounded by Unit Rough worst case
finds.json MAX_FINDS = 200 items 70 KB
ledger/replies.ndjson 8,000 rows, compacted to 2,000 rows 4 MB
config.json MAX_SEARCHES = 15 searches a few KB
data/<hash>.json one file per search, so also 15 files 80 KB
profile.json it is one boolean n/a bytes
entitlement.json one cached record n/a bytes
recipes.json nothing
logs.txt nothing
browser-data/ nothing tens of MB per site

Four caps. One counts items, one counts rows, one counts searches, one counts files. Not one of them counts bytes, which is the only unit the disk filling up is actually measured in.

That is not automatically wrong. Counting items is easier to reason about and easier to write a test for, and in a store whose records are all roughly the same size, items and bytes are the same cap with a constant factor. It is worth knowing the constant factor, though, so I worked them out.

What the caps are worth

A Find is six fields, mostly URLs:

export interface Find {
  id: string;        // origin + path
  siteName: string;
  siteUrl: string;   // the search URL
  title: string;
  url: string;
  foundAt: number;
}
Enter fullscreen mode Exit fullscreen mode

Call it 350 bytes of JSON with keys. Two hundred of those is about 70 KB, and the comment on the constant says what the 200 is for:

/**
 * How many finds to keep. A busy hunt produces a few hundred a week; this is
 * enough to cover "what did I miss overnight" without the file growing forever.
 */
const MAX_FINDS = 200;
Enter fullscreen mode Exit fullscreen mode

That is a product reason, not a disk reason. Good: 200 was chosen because it is roughly a week of history, and the 70 KB is a consequence nobody needs to think about.

The reply ledger is the big one. A ReplyRecord is 13 fields including two ISO timestamps and three URLs, so around 550 bytes a row, and the cap is 8,000 rows before compaction down to 2,000. That is about 4.4 MB at the trigger point and 1.1 MB after.

But 8,000 rows is not 8,000 replies, and this is the part worth being careful about. The ledger is append-only, and an update is a new row with the same id:

/**
 * NDJSON rather than sqlite on purpose: no native module, so the existing
 * electron-builder pipeline stays untouched. Updates are appended as a new row
 * with the same id and readers fold last-wins, which keeps writes atomic by
 * construction.
 */
Enter fullscreen mode Exit fullscreen mode

A reply that goes queued, in_progress, sent is three rows. So the cap is really "about 2,500 replies' worth of history", and the thing it is protecting is not disk at all: every read of that file parses every row and folds them, and isListingHandled() is called on the hot path. The 8,000 is a parse-time budget wearing a disk cap's clothing. I would rather it said so.

The compaction only runs on one of the two write paths

Here is the finding I did not expect. Rows are created here:

export function createReply(input): ReplyRecord {
  const record: ReplyRecord = { ...input, id: crypto.randomUUID(), ... };
  appendRow(REPLIES_FILE, record);
  return record;
}
Enter fullscreen mode Exit fullscreen mode

And updated here:

export function updateReply(id: string, patch: Partial<ReplyRecord>): ReplyRecord | null {
  // ...
  appendRow(REPLIES_FILE, next);
  maybeCompactReplies();
  return next;
}
Enter fullscreen mode Exit fullscreen mode

maybeCompactReplies() is called from updateReply and not from createReply. The only path that enforces the cap is the one that is not the path that grows the file from empty.

In practice it fires, because a reply always transitions status at least once, so updates vastly outnumber creates. The cap holds. What bothers me is that it holds for a reason that lives in a different module: it depends on the reply engine's behaviour, not on anything the ledger can see. Move the engine to a design that writes a terminal row in one shot, which is a perfectly reasonable thing to do, and this file quietly loses its only cap with no test going red.

One line in createReply fixes it, and the fix costs nothing because maybeCompactReplies already reads before it decides.

The directory that has no cap and is the only one that matters

Three of the four caps between them govern maybe 5 MB in the worst case I can construct. Meanwhile:

const userDataDir = path.join(BROWSER_DATA_DIR, hostname.replace(/\./g, '_'));
const context = await chromium.launchPersistentContext(userDataDir, { /* ... */ });
Enter fullscreen mode Exit fullscreen mode

One persistent browser profile per site you monitor, so your sign-in to a listing site survives a restart and you do not have to log in again every morning. A browser profile is a real browser profile: a cookie database, local storage, a disk cache, service worker storage. Tens of megabytes each, and there is no cap on it anywhere, for up to 15 sites.

So the honest summary of this audit is that I have written four caps, in three units, with comments explaining each one, around the data that accounts for a low single-digit percentage of what the app occupies. The thing that uses real disk is bounded only by how many sites a user adds.

It is not unbounded growth, which is what I was looking for and did not find: a profile for a given site reaches a steady state and the only way to get more of them is to add more searches, which is capped at 15. It is also the only store whose size I cannot state from the code, which is its own kind of answer.

What goes on the list:

  • A size readout in the app. Not a cap, a number. "Notifio is using 340 MB, mostly browser sessions for 6 sites" turns an invisible cost into an informed one, and it is a du-equivalent walk of one directory.
  • Clear the profile when a search is removed. Today removing a search deletes its diffing snapshot, because that one is explicit:
  /**
   * Delete the snapshot for a URL. Used when a search's URL changes so the old
   * baseline doesn't linger on disk.
   */
Enter fullscreen mode Exit fullscreen mode

The browser profile for that site's hostname stays, which is right if another search uses the same site and pure waste if none does. That check is a filter over the config.

Takeaways

  • Write down the unit next to the cap. Items, rows, lines and bytes are four different promises, and a reviewer cannot tell which one you meant from a number called MAX_SOMETHING.
  • A cap enforced on one of two write paths is a cap that depends on caller behaviour. Put the check where the file is written, not where you expect the traffic.
  • When a cap's real purpose is parse time rather than disk, say so in the comment. Someone will eventually "optimise" it in the wrong direction.
  • Audit by size, not by file count. Four careful caps on 5 MB and no opinion at all about the 300 MB next to it is a plausible place to end up, and you only see it by measuring.

The app is on the download page, the privacy policy is where the on-device claims live, and the help page is where a storage readout would be explained once it exists.

Top comments (0)