A Japanese version of this is on Zenn.
I build and sell a macOS network-security app as a solo dev. The landing page says:
Zero Telemetry — no outbound traffic, no data collection
Words are cheap, and that's just a claim. To a user, this is "an unknown solo dev's app that rewrites your firewall with root, saying 'trust me, I send nothing.'" There's no reason to believe it.
And this app sees sensitive stuff, constantly: the gateway MAC and fingerprint, listening ports, the Wi-Fi SSID you're on, FileVault / SIP / firewall state, files it scanned, URLs it checked. If any of that leaked, it would betray exactly the people who picked it for privacy.
So I decided to turn the claim into a measurement. tcpdump and free tools only — wiretap my own app. This post is the method and the places I tripped.
Why reading the code isn't enough
"The MCP server and detection logic are open source — just read them." A few holes:
1. The main app isn't open source. Only the read-only MCP server and detection logic are public; the privileged helper, pf control, and licensing stay closed (that surface helps attackers). Code alone doesn't cover the whole thing.
2. No App Sandbox. To be upfront: RoamSwitch does not run inside App Sandbox — it controls the firewall and sharing daemons, so it can't. The consequence is that entitlements don't restrict outbound traffic. codesign -d --entitlements showing "no network-client key" is not proof it can't send. You have to show behaviour.
3. Regressions. Even if this build is truly silent, a future version could add an analytics SDK, a crash reporter, a "check for news" feature. A one-time check is worthless — this needs to be a procedure you can re-run every release.
So instead of "read the code," I hand people "run this procedure."
What counts as a pass
Update (the build that adds Link Guard)
Link Guard adds a fifth path to the table below: once a day RoamSwitch itself
fetches a signed list of phishing sites fromlafine.net/updates/v1. Automatic,
like path #2 — but nothing goes the other way. No query string, no cookies,
nothing that identifies the machine. Just a download.The signature is checked against a key dedicated to the feed — separate from the
app-update (appcast) key. The feed is re-signed every day by server-side CI, so
that private key has to live in CI; keeping it feed-only means a leak can only
serve a bad blocklist (no code, no app update). The app-update signing key still
never leaves the Mac keychain.So the "automatic, non-OS destination" count goes from one at the time of this
test (just the appcast, i.e.lafine.net) to two — appcast + feed, both
lafine.net, both receive-only. Turn off Link Guard's auto-update and the feed
one stops.I decided "sends nothing" and "does pull updates" are worth stating as
separate things, so the Linux whitepaper (v1.1) now spells that out.
The only traffic allowed out is the four paths documented in the whitepaper (§7):
| # | destination | process | when |
|---|---|---|---|
| 1 | lafine.net/api/v1/license/* |
RoamSwitch | license activate / deactivate only |
| 2 | lafine.net/updates/appcast.xml |
RoamSwitch | Sparkle (launch + every 24 h) |
| 3 | ClamAV mirrors |
freshclam (not the app) |
only with ClamAV installed |
| 4 |
HEAD to a target URL |
RoamSwitch | only when the in-app Link Safety sheet is used |
Anything else out of RoamSwitch / RoamSwitchHelper / RoamSwitchMCPServer is a fail.
Tools (no Little Snitch)
I don't own Little Snitch, so:
- tcpdump (system) — every packet that leaves the NIC
-
tshark (
brew install wireshark) — pull destinations and TLS SNI from the pcap - nettop / lsof (system) — tcpdump doesn't say which process sent a packet; these map socket → process
- LuLu (Objective-See, free, open source) — per-process outbound events, from the unified log
LuLu is a fine Little Snitch substitute, and it's by a well-known name in Mac security — which, for a write-up, only helps.
Trip 1: I didn't need to capture for 25 hours
I'd convinced myself that "Sparkle's periodic check is every 24 h, so I have to capture for 25 h straight to span it." I set up for an overnight run.
Then I looked at Info.plist: SUScheduledCheckInterval = 86400, and SULastCheckTime sits in defaults in the clear. So there's no waiting:
BID=com.tetsuharu.RoamSwitch
defaults delete $BID SULastCheckTime
osascript -e 'quit app "RoamSwitch"'; sleep 2; open -a RoamSwitch
# launch sees "overdue" and checks within seconds to a couple of minutes
The appcast HTTPS GET is byte-for-byte identical whether it comes from this, the "Check for Updates" menu item, or a fresh first launch.
The only reason to run long is to catch a wall-clock "once a day" telemetry beacon — an unusual design, and there's no analytics SDK in the app anyway (open source, checkable). So a few hours of idle is plenty. Total active work: ~3–4 h, one sitting.
Trip 2: on a normal machine, attribution is hell
tcpdump tells you "a SYN went to 140.82.112.x." It does not tell you whether RoamSwitch, Chrome, Dropbox, or the OS sent it.
On a daily-driver machine, even idle, there's a flood of non-Apple traffic: Safe Browsing, component updaters, cloud sync, chat keep-alives. Picking RoamSwitch's packets out of that by timestamp and port is genuinely error-prone — you end up waving things away as "probably Chrome."
So you make the machine silent first. A dedicated account or spare Mac is ideal; quieting the same login also works:
- quit every GUI app
-
launchctl bootoutthe third-party agents (reverts on reboot) -
softwareupdate --schedule off, analytics off, Spotlight suggestions off - turn off iCloud Private Relay — otherwise it obscures SNI and the analysis is useless
Then, before launching RoamSwitch, run tcpdump for 10 minutes and check that every remaining destination is Apple (17.0.0.0/8, etc.). If a non-Apple destination is still there, something's still running — find it with lsof -nP -iTCP@<IP>, stop it, start over.
Only once that baseline is clean can you lean on "any non-Apple traffic after this ≈ RoamSwitch."
Don't open this procedure in a browser on the test machine. Chrome especially is a pile of non-Apple traffic (Safe Browsing, a GCM keep-alive). Use a second device, print it, or pre-cache the page and use Safari.
The scenario: public Wi-Fi + maximum lockdown
Reproduce the situation RoamSwitch is built for: you join a café / public Wi-Fi and the app locks down automatically.
- pin RoamSwitch's security level to Maximum Lockdown (manual override — not Trusted/Open, not Standard Protection)
- connect to a network you haven't registered as trusted (real café Wi-Fi is best; a phone hotspot works)
At max lockdown, RoamSwitch's pf blocks other apps' egress too, so the capture is even quieter and its own traffic stands out. Crucially, appcast and license still get through — RoamSwitch puts its own traffic in a pf exception. Part of the point of the test is to confirm that exception is minimal and only points where it's supposed to (i.e. it didn't quietly whitelist a telemetry endpoint for itself).
Three monitoring layers
# layer 1: full capture (local / DNS / mDNS filtered out)
sudo tcpdump -i any -n -w cap-%Y%m%d-%H%M.pcap -G 3600 -W 72 -Z root \
'(ip or ip6) and not net 127.0.0.0/8 and not net 169.254.0.0/16 \
and not net 224.0.0.0/4 and not port 5353 and not port 53 and not port 67 and not port 68'
# layer 2: socket -> process, sampled every 5s, in a root shell
sudo bash -c 'while :; do ts=$(date -u +%FT%TZ); \
lsof -nP -iTCP -sTCP:ESTABLISHED +c0 2>/dev/null \
| awk -v ts="$ts" "NR>1{print ts,\$1,\$2,\$9}"; sleep 5; done' | tee lsof.log
# layer 3: LuLu events from the unified log
log stream --style syslog --predicate 'subsystem BEGINSWITH "com.objective-see"' > lulu.log &
The MCP server only speaks over stdio, so I drive every tool through it and separately confirm its PID never opens an outbound socket.
Analysis and verdict
After the idle hold and the phases (run every feature by hand once; deliberately fire the four known paths), pull destinations from the pcap and cross-reference with the process log by timestamp:
for f in cap-*.pcap; do
tshark -r "$f" -Y 'tcp.flags.syn==1 && tcp.flags.ack==0' -T fields -e ip.dst
done | sort -u
awk '$2 ~ /^RoamSwitch/ {for(i=1;i<=NF;i++) if($i ~ /->/){split($i,a,"->"); print $2,a[2]}}' \
lsof.log | grep -vE '127\.0\.0\.1|::1' | sort -u
Pass criteria:
-
RoamSwitchHelper— zero outbound flows -
RoamSwitchMCPServer— zero outbound flows (stdio only) -
RoamSwitch— nothing outsidelafine.net(and the URL you hit for path #4) - path #3 is attributed to
freshclam, not the app - no new destination appears during the feature-exercise phase
What actually came out
Verdict: PASS. Across a ~2-hour capture (forced Sparkle check, every feature exercised once, DNS Threat Guard toggled, a network transition, security level pinned to Maximum Lockdown):
Zero non-local outbound connections attributed to RoamSwitch, RoamSwitchHelper, or RoamSwitchMCPServer.
The only sockets the MCP server opened were 127.0.0.1:5000 and 127.0.0.1:7000 — the get_exposed_ports tool checking HTTP headers on locally-listening dev servers. Nothing left the machine.
Here is every HTTPS destination in the whole capture (TLS SNI):
bag.itunes.apple.com gspe35-ssl.ls.apple.com proxy-safebrowsing.googleapis.com
configuration.apple.com init.push.apple.com setup.icloud.com
dns.quad9.net mask.icloud.com swdist.apple.com
gateway.icloud.com mesu.apple.com swscan.apple.com
gsa.apple.com ocsp2.apple.com tether.edge.apple
p117-contacts.icloud.com courier.push.apple.com swallow.apple.com
fbs.smoot.apple.com
lafine.net <- the only non-OS destination
Strip out the OS's own services (iCloud, push, software update, OCSP, Safari Safe Browsing) and one destination remains: lafine.net — Sparkle fetching appcast.xml (egress path #2). dns.quad9.net shows up because the DNS Threat Guard switched the system resolver; the encrypted lookups are mDNSResponder's, not RoamSwitch's. swallow.apple.com / fbs.smoot.apple.com sit on AWS IPs but are Apple services.
entitlements:
$ codesign -d --entitlements :- /Applications/RoamSwitch.app/Contents/MacOS/RoamSwitchHelper
<dict></dict>
$ codesign -d --entitlements :- /Applications/RoamSwitch.app/Contents/MacOS/RoamSwitchMCPServer
<dict></dict>
Empty — not even com.apple.security.network.client (which, as noted, wouldn't enforce anything without App Sandbox, hence the behavioural check).
LuLu logged no RoamSwitch-related outbound events for the duration.
It's all in one shell script, in the audit/ folder of the roamswitch-support repo. ./rs-zerotel-audit.sh all --idle 2h --clam runs prep, capture, forced Sparkle check, MCP exercise, analysis, a PASS/FAIL verdict, and a shareable FINDINGS.md. The only manual bits are helper approval, pinning max lockdown, one network transition (a Wi-Fi off/on is enough), and a few menu actions.
What this audit can't show
Being upfront:
- Not "no traffic at all". The four paths are real. Zero Telemetry here means no collection or transmission of usage / diagnostic data.
- Not "never". The conclusion is scoped to the observed window — a few hours of normal and adversarial use.
- Payloads aren't inspected (destination + SNI only). To go further, put a mitmproxy in the path and confirm the appcast and license requests carry only "UA + version" and "key + device hash."
But "a claim" versus "a claim + a procedure anyone can run + an honest statement of the limits" is a very different weight of trust. The more skeptical someone is, the more they want something they can check themselves.
Script and run sheets (EN/JA): github.com/lafine1211/roamswitch-support, audit/
Whitepaper (Zero Telemetry scope is §7): https://lafine.net/security.html
Top comments (2)
The move from "read the code" to "here is a procedure you can re-run" is the part worth stealing, and it generalizes well past macOS. Your regression point is the strongest thing in the post and I think it is underrated. A one-time audit proves a build, not a policy. The realistic failure is not a dev who lied in 1.0; it is the crash reporter someone adds in 2.3 without ever connecting it back to a landing-page claim made two years earlier. Making the check cheap enough to re-run is what turns "zero telemetry" from a slogan into something with a test behind it. The SULastCheckTime trick is a nice catch, and the general shape transfers: find the persisted timer, delete it, relaunch, stop waiting on wall-clock. The Android equivalent for anything gated behind Doze or a 24h window is forcing the state with adb shell dumpsys deviceidle force-idle rather than leaving a device alone overnight. One question as a reader: does the capture catch a path that only phones home on a trigger you did not think to exercise? Your allowlist has "only when the Link Safety sheet is used," so you clearly considered it. Is the capture window driven by a scripted walk through every feature, or is it passive running plus spot-checks?
You nailed the real reason why this test exists. It is never about proving good intentions in version 1.0, but keeping ourselves honest in version 2.3 when someone wants to slap in a quick crash-reporting SDK without remembering a promise made years earlier.
To your question: it is an automated script rather than passive listening. The test harness steps through security mode toggles, forces the Sparkle update check via the SULastCheckTime trick you mentioned, and explicitly triggers the URL safety probe.
That said, the bigger safeguard against "triggers I forgot to exercise" is the architecture itself:
First, there are zero third-party telemetry or analytics SDKs compiled into the binary. No Firebase, no Sentry, no tracking frameworks. You can run otool -L on the executable and see only standard Apple frameworks plus Sparkle.
Second, actual outbound socket code in the Swift codebase is physically localized to just two places: Sparkle for update XMLs and an ephemeral URLSession in the link scanner when the user explicitly clicks check. The rest of the app (the pf packet filter rules, ARP table checks, port scanners, clipboard inspection) runs entirely on Darwin and POSIX APIs in-memory with zero network code attached to them.
So even if there is an edge case path the script failed to hit, the binary simply does not have the network plumbing to phone home anyway.
Loved your Android dumpsys force-idle tip. Forcing timers instead of waiting on wall clocks is a great pattern. Thanks again for the sharp feedback!