"Runs offline" is the easiest claim in software to make and one of the least frequently checked. I build an offline AI app, so I'm on the hook for the claim — here's the testing I actually run, which is the same thing you could run against anything else that claims it.
1. Disable the network adapter, not just Wi-Fi.
Airplane mode and the Wi-Fi toggle leave more running than people assume, and on a laptop with an ethernet port you've only closed one door. Disabling the adapter in the OS is the version that's actually airtight, and it's the one I film.
2. Watch the filesystem, not the app.
On Windows, Process Monitor filtered to the process plus the host drive, running for a whole session, tells you what the app wrote and where. On macOS, sudo fs_usage does the equivalent. This catches the thing a network check doesn't: an app that writes cached data, logs or model files somewhere you didn't expect, which matters a lot if the selling point is portability with no trace left behind.
3. Search the whole machine after unplugging.
Full-disk filename and content search for anything the app might have left — config folders, temp directories, registry entries on Windows. Zero results is the only passing grade. Someone did this to my app before buying it and told me afterwards, which is exactly the right instinct.
4. Check it in a state you didn't design for.
The interesting failures aren't in the happy path. Launch it with no network at all from a cold boot, rather than disconnecting after it's already running and warm — plenty of software quietly fetches what it needs on first launch and then appears to work offline forever after.
What I got wrong first time:
I tested on my own development machine, which had every model file already downloaded and every dependency already resolved. It proved nothing. The test only means something on a machine that has never seen the app before, which is why the video version of this is filmed on a different laptop.
Why bother being this strict about it?
Because the people who care about offline AI are, by definition, people who check. The security consultant who messaged me about this app had already run his own trace search before he got in touch. If the claim doesn't survive that, it was never worth making.
If you've got a sharper method than the four above — particularly for catching network calls a process makes before you've started watching it — I'd like to hear it.
The app, if you want to run these tests against it yourself: getpocketcore.com
Top comments (0)