What I learned building a screenshot organizer that never sends a screenshot to a server

I'm Bazi, and on the side I build Pizazoo, an app that sorts screenshots into categories (Receipts, Chats, Code, Travel and so on) on iPhone, iPad and Mac. The one rule I set at the start: screenshots never leave the device. No account, no upload, no server-side model. Sync goes through the user's own iCloud.

That rule shaped almost every technical decision. Here's what I learned, including the bugs that taught me the most.

The pipeline, all on-device

The Mac and iOS apps share one Swift processing pipeline. For each screenshot it collects several cheap signals and lets them vote:

• Text, from Apple's Vision text recognizer.
• Image labels and a feature print from Vision (a feature print is a compact vector you can compare between images).
• Semantic similarity between the recognized text and each category's description, using NaturalLanguage sentence embeddings.
• Keyword signals for things with obvious tells (currency totals, stack traces, flight numbers).
• A nearest-neighbour vote from screenshots the user already placed or corrected, so it learns from fixes.
• Apple's on-device Foundation Models, when the device supports them.

None of this needs a network connection, and none of it costs me anything per user.

Design for the device that can't run the model

My own test iPhone reports deviceNotEligible for Apple Intelligence, and so do plenty of real customers' phones. On those devices the Foundation Models path simply never runs.

So the "fallback" is actually the main path. I stopped treating the language model as the classifier and started treating it as a bonus that can break ties when it's there. If your app leans on on-device AI, test seriously on a device without it, because a good share of your users have exactly that.

Visual similarity is not classification

This one produced my favourite bug: at one point every screenshot a user took landed in the same custom category.

The cause was the visual nearest-neighbour vote. VNFeaturePrintObservation distances are great for "find screenshots that look like this one" (that's what powers the Related screen), but screenshots all look alike to it: white backgrounds, text, status bars. They cluster tightly. Whichever category had the most example screenshots won the vote almost every time, and a lenient acceptance rule rubber-stamped it.

The fix was a rule, not a threshold: a visual vote may only corroborate a real text, label or keyword match. It can never decide on its own.

Never let one slow framework block the queue

macOS 27 brought a nasty surprise. Vision's text recognizer sometimes goes into a bad system-wide state: cold loads of 26 to 106 seconds, errors on images that read fine an hour later, and once a request that ignored cancellation for more than 29 minutes. Vision also holds a process-wide lock while loading, so image labels and feature prints stall behind it.

Waiting for text inside the per-item timeout meant screenshots piled up unsorted. What fixed it:

1. Give text recognition a fixed budget (20 seconds) in the first pass. If it isn't back, classify with the signals you have and keep going.
2. Let the slow request finish in the background, and backfill the text later.
3. Warm the recognizer up with an image that actually contains text. A blank warm-up image only loads the detector, so the first real request still paid the full cost.

Let the recognizer detect the language

At one point I replaced automatic language detection with an explicit list (the user's preferred languages plus English), hoping it would avoid the macOS 27 stall. It didn't help with the stall: a controlled run with an explicit ["en-US"] stalled for 62 seconds anyway. It did cause a real regression, though: for a user with an English system language, screenshots of Chinese or Arabic text were no longer recognized at all. They weren't searchable, and the auto-redact feature couldn't see personal info in them.

automaticallyDetectsLanguage = true is back, with a comment in the code saying not to narrow it again without measuring both effects.

Sync without a server of your own

Pizazoo syncs through CloudKit with its own sync engine and custom records, not SwiftData's built-in mirroring. Three lessons from that:

Pin SwiftData away from CloudKit. Once the app has the iCloud entitlement, ModelConfiguration defaults to mirroring the store through CloudKit, which requires every attribute to be optional and forbids unique constraints. Our schema has neither, so the store failed to load at launch (NSCocoaErrorDomain 134060). The fix is one argument, cloudKitDatabase: .none, and it now has a comment that nobody should remove it.

Make conflict resolution deterministic. Reinstalling the iOS app used to duplicate every screenshot in iCloud: the reinstall re-imported photos under new IDs while the old records synced back. Now rows with the same SHA-256 of the image bytes are merged, and the survivor is the row with the smallest UUID string. Every device picks the same survivor without talking to the others. The rule I wrote next to it: never make this order-dependent, or two devices can each delete the other's copy.

Change the hash when you change the pixels. The sync engine re-uploads an image only when its checksum changes. Editing a screenshot in place (markup, blur) and forgetting to recompute the checksum means the edit quietly stays on one device. Every code path that overwrites pixels now has to update the checksum.

The OS will change underneath you

On macOS 27, defaults write com.apple.screencapture location ... runs without any error and does nothing. Screenshots keep landing on the Desktop, because the active setting moved to a new key, location-screenshot. The app read the old key, so its settings screen showed the wrong folder. It now reads the key for the running OS version and writes both. I also wrote it up for other people, because a silent no-op like that wastes a lot of afternoons.

What I'd tell myself at the start

• Keep everything on-device if you can. It removes a whole category of cost and liability, and users notice.
• Treat on-device AI as an upgrade, not a dependency.
• Be suspicious of any single signal that "almost always" wins.
• Put a budget on every system framework call that can stall.
• Leave comments that explain why a line must not change. Future you will try to change it.

If you're curious, Pizazoo is at pizazoo.com. It's free to organize, search and mark up, with a one-time Pro upgrade. I'm happy to go deeper on any of this in the comments.

Disclosure: I'm the developer of Pizazoo. This post was drafted by an AI agent working from my own code and engineering notes.