How I Used an Uncensored Local LLM for Adversarial QA on My macOS App

When you build a security app as a solo developer, the hardest engineering problem isn't writing the defense code — it's QA and resilience testing.

Security software operates under a bizarre constraint: nothing happening is the normal state. The software only does its job when something hostile or anomalous occurs.

The trouble is, when you write your own test scripts, you subconsciously test what you expect. You launch a test server, watch the port get blocked, nod, and mark it green. But real attackers and weird production environments don't follow the developer's script. They poke at edge cases, state machine bugs, and race conditions between subsystems.

To break my own bias, I set up a local uncensored model (qwen3.8-27b-uncensored) running locally on my Mac via OpenCode as a dedicated Adversarial QA Tester, pointed it at a disposable macOS VM (Tart), and told it to find holes in my app, RoamSwitch.

Here is what that stress test surfaced, the edge cases it broke, and how I fixed them.

---

1. Why an uncensored local LLM for QA?

If you ask a commercial cloud LLM to write live packet-injection scripts or craft ransomware-like file tampering patterns, safety filters will reject the prompt.

In defensive QA, though, you need real hostile traffic and weird filesystem operations to verify fail-closed behavior.

Running an uncensored model locally on my Mac meant I could run unrestricted automated stress tests: continuous Scapy ARP injection, deceptive dotfile placement, and multi-port backdoor attempts directly against the target VM.

graph LR
    subgraph Host["Host Mac (Adversarial QA Runner)"]
        LLM["Local LLM (Qwen 27B uncensored)<br/>via OpenCode"]
        Payloads["Dynamic Stress Test Suite<br/>(Scapy / ARP / EICAR / kqueue)"]
        LLM --> Payloads
    end

    subgraph VM["Target macOS VM (Tart)"]
        RS["RoamSwitch (Defense Engine)"]
        Engine["・pf packet filtering<br/>・PortAnomalyGuard<br/>・ClamAV real-time quarantine<br/>・RansomwareCanaryGuard"]
        RS --- Engine
    end

    Payloads -->|"1. ARP Reply flooding (MitM)"| VM
    Payloads -->|"2. 0.0.0.0 Backdoor listeners"| VM
    Payloads -->|"3. Quarantine evasion payloads"| VM
    Payloads -->|"4. Canary decoy tampering"| VM

---

2. Three subtle edge cases the QA suite caught (v1.5.5)

Here are three real architectural edge cases that standard happy-path unit tests missed.

---

1. Flagging a binary on one port implicitly trusted it on all other ports

RoamSwitch has a feature called PortAnomalyGuard that catches processes binding to 0.0.0.0 and immediately blocks inbound traffic via macOS's packet filter (pf).

1. The test launched an HTTP server on port 8765 with Homebrew Python (python3 -m http.server 8765 --bind 0.0.0.0).
2. RoamSwitch flagged port 8765 and injected a pf drop rule.
3. In follow-up testing, the runner launched a second listener on port 9877 using the same Python binary.

The result: The second listener on port 9877 was NOT blocked.

• The Root Cause: To prevent duplicate alert spam, PortAnomalyGuard.evaluate() automatically added flagged binaries to KnownExecutablesV2 by binary name alone ("Python"). The second listener on port 9877 saw "Python" already in the known set and skipped inspection entirely.
• The Fix (v1.6.2): Generic script interpreters (Python, Node.js, Ruby, Netcat, Bash) are now strictly scoped to executablePath:port (e.g., /opt/homebrew/bin/python3:8765). Flagging port 8765 gives zero trust to port 9877. A startup migration purges legacy port-less records.

---

2. Duplicate filenames broke ClamAV quarantine and left malware alive

RoamSwitch watches Downloads, Desktop, and Documents with FSEvents and scans new files with ClamAV, moving threats to ~/Library/Application Support/RoamSwitch/Quarantine.

The test dropped 4 tricky variations of the EICAR test string:

• Hidden dotfiles (.hidden_eicar.txt)
• Incomplete download extensions (malware.tmp)
• Terminal copies lacking com.apple.quarantine attributes (copied_eicar.txt)
• Standard downloads (normal_eicar.txt)

All 4 were caught and isolated. But then the test ran a collision test: dropping the exact same malware filenames a second time.

Because files with those exact names already existed in Quarantine, ClamAV's --move failed with an error. The new malware was left completely ALIVE in the user's Downloads folder. Quarantined files were also left at normal 0644 permissions.

• The Fix (v1.6.0–v1.6.3):
  1. If an identically named file already exists in Quarantine, RoamSwitch catches the collision and moves it under a timestamped unique name (malware_1725068200.tmp).
  2. Upon isolation, it immediately applies chmod 000 (posixPermissions: 0o000), physically stripping read and execute permissions.

---

3. Missing canaries were prematurely regenerated on app restart

RoamSwitch places decoy canary files in target directories and monitors them with kqueue (DispatchSource). Any unauthorized tampering or renaming triggers an immediate Air-Gap containment (block drop all) and terminates suspicious processes.

Renaming a canary to /tmp/stolen_canary.xlsx triggered Air-Gap in milliseconds.

However, the post-test QA review caught a lifecycle issue:

"When the app restarts, setupCanaryBait() sees the decoy is missing and automatically creates a new one. If an attacker modified a canary and forced an app relaunch, the altered file would become the new baseline hash."

• The Fix (v1.6.4):
  1. All expected canary SHA-256 hashes are persisted to disk (UserDefaults).
  2. Routine app launches no longer recreate missing files; they preserve the violation against the recorded baseline.
  3. Self-healing regeneration from clean embedded templates only runs when the user explicitly clicks "Release Emergency Containment" in the UI.

---

3. Verified QA Results (v1.6.4)

After applying these fixes in v1.6.0 through v1.6.4, I re-ran the full test suite across all 7 defense domains:

#	Defense Domain	Stress Scenario	v1.5.5 (Initial)	v1.6.4 (Final)	Verdict
T1	ARP Spoofing Auto-Containment	Injected rogue gateway ARP replies via Scapy	✅ Air-Gap triggered	✅ Air-Gap triggered	PASS
T2	Unknown Port Auto-Isolation	Multi-port backdoor listeners on 0.0.0.0	❌ Secondary port bypassed	✅ path:port strictly isolated	FIXED
T3	ClamAV Download Quarantine	4 placement vectors + collision re-tests	⚠️ Move failed on collision (ALIVE)	✅ Unique rename & chmod 000	FIXED
T4	Ransomware Canary Bait	Decoy tampering & baseline reset tests	⚠️ Premature file recreation	✅ Air-Gap & Baseline persisted	FIXED
T5	Helper XPC Boundary	Unsigned binary connecting to Mach service	✅ Rejected by Team ID check	✅ Strict check maintained	PASS
T6	DNS Threat Protection Guard	Untrusted network DNS hijacking	✅ Quad9 Secure DNS enforced	✅ Secure DNS maintained	PASS
T7	Air-Gap State Hygiene	Post-release timestamp file inspection	⚠️ Timestamp file lingered	✅ Clean unlink on release	FIXED

A full TCP SYN port scan (nmap -Pn -sS) across all 65,535 ports confirmed complete stealth isolation (filtered (no-response)).

---

4. Takeaways for solo QA

Using an uncensored local model for adversarial testing gave me a practical way to run enterprise-grade stress tests on my own:

1. Breaks Developer Bias: The model doesn't care about your code structure; it tests the awkward seam between features.
2. Fast and Private: Running locally on a Mac against a local VM meant I could run aggressive tests without cloud rate limits or safety filter blocks.
3. Resilience over Green Builds: Real software quality isn't about passing the tests you wrote for yourself — it's about holding up when something hostile tries to break your assumptions.

The full reproduction procedures, scripts, and verified audit reports are open source:

• Penetration Test Report (GitHub): https://github.com/lafine1211/roamswitch-support/blob/main/audit/RESULTS-PENTEST-2026-08-31.md
• Security Whitepaper: https://lafine.net/security.en.html
• RoamSwitch Homepage: https://lafine.net/