"Your file never leaves your device": how to test that claim (and the Chromium catch)

Lots of web tools now say "runs in your browser, nothing is uploaded". I make one of them
(Drible, 53 file tools, 48 of which run client-side), and at some point I
realised the claim was just a sentence on a page. Nothing would fail if a refactor, a new
dependency or an analytics script started sending files somewhere.

So I turned the promise into a test. This post is how it works, including a gotcha in Chromium
that makes the obvious version of the test pass when it shouldn't.

The idea

Run a real job in a real browser (sign a PDF, merge two PDFs, compress a photo, convert a HEIC)
and record every network request the page makes while doing it. If any request could have
carried the file, fail.

"Could have carried the file" needs a definition. I used two checks:

1. Does the body contain a file signature? PDFs start with %PDF, JPEGs with FF D8 FF, PNGs with 89 50 4E 47, and HEIC files contain ftypheic. If any of those bytes appear in a request body, the file (or part of it) went out.
2. Is it anything other than a plain GET? Even without a visible body, a POST or PUT is a way to send data.

The only request allowed through is the analytics beacon, and only if it is small.

The test (Playwright)

const FILE_MAGIC: [string, Buffer][] = [
  ['PDF', Buffer.from('%PDF')],
  ['JPEG', Buffer.from([0xff, 0xd8, 0xff])],
  ['PNG', Buffer.from([0x89, 0x50, 0x4e, 0x47])],
  ['HEIC', Buffer.from('ftypheic')],
];

const BEACON = /\/cdn-cgi\/rum\b/; // Cloudflare Web Analytics, cookieless
const BEACON_MAX_BYTES = 4096;

function watchRequestBodies(page: Page) {
  const sent: { method: string; url: string; bytes: number; carries: string[] }[] = [];
  page.on('request', (r) => {
    const body = r.postDataBuffer();
    const method = r.method();
    if (['GET', 'HEAD'].includes(method) && (!body || body.length === 0)) return;
    sent.push({
      method,
      url: r.url().split('?')[0],
      bytes: body ? body.length : 0,
      carries: body ? FILE_MAGIC.filter(([, m]) => body.includes(m)).map(([n]) => n) : [],
    });
  });
  return sent;
}

const possibleUploads = (sent) =>
  sent.filter((s) => s.carries.length > 0 || !(BEACON.test(s.url) && s.bytes <= BEACON_MAX_BYTES));

And a job:

test('Merge PDF', async ({ page }) => {
  const sent = watchRequestBodies(page);
  await page.goto('/pdf/merge');
  // upload two fixtures, click Merge, wait for the download...
  expect(possibleUploads(sent), 'only the analytics beacon may send data').toEqual([]);
});

I picked one tool per technique, because that is where an accidental upload would come from:
the PDF editor (Sign PDF), a pdf-lib tool (Merge), a Canvas tool (Compress Image) and a
WebAssembly tool (HEIC to JPG, which runs libheif compiled to WASM).

The catch: the test passed when it should have failed

A test that never fails proves nothing, so I added a control: run one of our five tools
that does upload (PDF compression with Ghostscript on the server) and assert that the detector
catches it.

In Firefox the control passed. In Chromium it failed: the detector saw no upload at all.

The reason: when a page sends a file as a Blob or File (which is how FormData uploads and
most XHR/fetch uploads go out), Chromium does not expose that body through the DevTools protocol
that Playwright uses, so postDataBuffer() comes back empty. If the test had only looked for
file signatures in bodies, every upload in Chromium would have been invisible, and the "privacy
test" would have been green forever.

That is why the detector also treats any non-GET request as suspicious, whatever its visible
body. The upload shows up as a POST to /api/... even when Chromium hides its contents, and the
control now passes in every browser:

test('control: a server tool is detected as uploading', async ({ page }) => {
  const sent = watchRequestBodies(page);
  await page.goto('/pdf/compress');
  // upload sample.pdf and click Compress...
  await expect
    .poll(() => possibleUploads(sent).some((s) => s.url.includes('/api/')), { timeout: 90_000 })
    .toBe(true);
});

If you write a test like this for your own tool, add the control first. Without it you are
testing your detector's blind spots, not your app.

Where it runs

The suite runs every night against production (not a local build) in Chromium, Firefox and
WebKit, so it also catches things that only exist in production: a CDN script, an injected
analytics tag, a header change. Four jobs plus the control take about 15 seconds per browser.

What it does not prove

Honest limits, because "provably private" gets thrown around a lot:

• It covers the jobs it runs. A different tool, or a code path the test doesn't click, is not covered (that's why it samples one tool per technique).
• It watches the page's network requests. It is not a substitute for a tight Content-Security-Policy, which is the other half: ours limits where the page may connect to a short allowlist (our own origin, the analytics endpoint, and the CDNs that serve fonts and the background-removal model).
• It says nothing about the 5 tools that are server-side by design. Those are labelled on the page, and the server deletes the file right after responding.

Check any site yourself in a minute

You don't need Playwright to check a site that makes this claim (including mine):

1. Open the tool and DevTools (F12) > Network.
2. Run a job with a harmless file.
3. Look for any POST or PUT, especially one roughly the size of your file. In Chromium, don't expect to see the file in the payload tab; the request itself is the tell.
4. Optional: load the page, turn off Wi-Fi, and run the job again.

If you build one of these tools, I'd love to see more of them ship a test like this. It's about
60 lines and it turns a marketing sentence into something that fails CI.

More on how Drible splits browser and server processing: How it works.

Comments

[DEV SUPPORTS]

You need to verify your account.

tr.ee/dev-to