DEV Community: Cool Light Shop Co,. LTD

On-Device AI in iOS: How I Built a Screenshot Classifier Without Any Cloud Calls

Cool Light Shop Co,. LTD — Tue, 26 May 2026 04:29:48 +0000

Why On-Device AI Matters for Screenshots

Screenshots are sensitive. They contain prices, flight details, personal conversations, banking info. Uploading them to a cloud AI is a non-starter for most users.

The good news: iOS gives you everything you need to build a capable AI pipeline that runs entirely on-device. Here's how I did it for Snaap, an AI screenshot cleaner.

The Pipeline

Step 1: Find the Screenshots

let options = PHFetchOptions()
options.predicate = NSPredicate(
    format: "mediaType == %d AND (mediaSubtypes & %d) != 0",
    PHAssetMediaType.image.rawValue,
    PHAssetMediaSubtype.photoScreenshot.rawValue
)
options.sortDescriptors = [NSSortDescriptor(key: "creationDate", ascending: false)]
let screenshots = PHAsset.fetchAssets(with: options)

iOS natively tags screenshots — no ML model needed for detection.

Step 2: Extract Text with Vision OCR

let request = VNRecognizeTextRequest { request, error in
    let text = request.results?
        .compactMap { $0 as? VNRecognizedTextObservation }
        .compactMap { $0.topCandidates(1).first?.string }
        .joined(separator: "\n") ?? ""
}
request.recognitionLevel = .accurate
request.usesLanguageCorrection = true

VNRecognizeTextRequest with .accurate level catches even small text on product screenshots. Processing ~600 images takes about 90 seconds on iPhone 14 Pro.

Step 3: Rule-Based Classification

func classify(ocrText: String) -> Category {
    let text = ocrText.lowercased()

    // Travel: look for flight codes like VN123
    if text.contains("boarding pass") || text.contains("flight") ||
       text.range(of: "[A-Z]{2}\\d{3,4}", options: .regularExpression) != nil {
        return .travel
    }

    // Receipt: price patterns + keywords
    let pricePattern = "[\$£€¥]\s*\d+[\.,]\d{2}"
    if text.contains("total") && 
       text.range(of: pricePattern, options: .regularExpression) != nil {
        return .receipt
    }

    // Recipe: multiple cooking keywords
    let recipeWords = ["ingredients", "tbsp", "preheat", "bake", "simmer"]
    if recipeWords.filter({ text.contains($0) }).count >= 2 {
        return .recipe
    }

    // Code: programming keywords
    let codeWords = ["func ", "const ", "import ", "async", "await"]
    if codeWords.filter({ text.contains($0) }).count >= 2 {
        return .code
    }

    return .other
}

The key insight: screenshots of the same category share highly predictable vocabulary. A flight booking always says "boarding pass" or "gate." A receipt always has a price and the word "total." You don't need an LLM for this — domain-specific heuristics work better.

Step 4: Context Generation

func generateSentence(for screenshot: Screenshot) -> String {
    switch screenshot.category {
    case .travel:
        if isDatePast(screenshot.extractedDate) {
            return "Flight to \(destination). You already landed."
        }
        return "Flight to \(destination) — \(formatDate(screenshot.extractedDate))."
    case .product:
        if weeksAgo(screenshot.createdAt) > 4 {
            return "\(product) · \(price). Saved \(weeks) weeks — still want it?"
        }
        return "\(product) · \(price) from \(source)."
    // ... etc
    }
}

The sentences are designed to prompt a decision. "You already landed" makes it safe to delete. "Still want it?" keeps the door open. The goal isn't perfect accuracy — it's removing the fear of deleting.

Step 5: Duplicate Detection with Perceptual Hashing

func computeHash(for image: UIImage) -> String? {
    // Resize to 8x8 grayscale
    // Compute average brightness
    // Build 64-bit string: each bit = pixel > average
    // Hamming distance < 10 = duplicate
}

pHash catches visually identical screenshots even if one is slightly cropped or has a different timestamp. Found 42 duplicates in my library that I never knew existed.

Why Not Use an LLM?

Speed: Rule-based classification is instant. No API latency.
Privacy: Nothing leaves the device. Critical for screenshot content.
Cost: $0 vs. paying per token.
Reliability: No hallucinations, no API outages.
Offline: Works on airplanes, in subways, anywhere.

For a constrained domain like screenshot classification, LLMs are overkill. The vocabulary is predictable, the categories are well-defined, and the cost of a misclassification is low (user just taps "other").

Results & App

Snaap is free on the App Store: https://apps.apple.com/app/snaap-voucher-reminder-ai/id6770817204

The entire AI pipeline — OCR, classification, context generation, duplicate detection, expiry checking — runs in about 0.15 seconds per screenshot on device. No network calls, no backend, no user accounts.

If you're building an iOS app that touches user data, I'd strongly recommend exploring on-device AI first. The frameworks are solid, the privacy story is compelling, and users genuinely appreciate it.

Built with Vision, PhotoKit, GRDB, SwiftUI + UIKit. iOS 16+.

I Built an AI That Reads Your Screenshots and Tells You Why You Saved Them

Cool Light Shop Co,. LTD — Tue, 26 May 2026 04:29:34 +0000

The Problem

I had 2,847 screenshots on my iPhone. Boarding passes from 2023. Products I screenshotted but never bought. Recipes I saved and forgot. Memes I already sent to everyone.

Apple's Photos app treats them as... photos. Not as the half-finished intentions they actually are.

The Insight

Every screenshot is an externalized intention — something you meant to do, buy, watch, or remember. But the camera roll is a terrible task manager.

I realized the reason people don't delete screenshots isn't laziness — it's fear of deleting something important. If you knew what each screenshot was, you'd decide in one second. The problem is information, not motivation.

What I Built

Snaap — an iOS app that reads every screenshot with on-device AI and generates a one-sentence explanation:

"Boarding pass for flight VN123 — departed Feb 14. Trip is over."

"Nike Air Max $89 from Instagram. Saved 3 months ago — not bought yet."

"Pasta recipe with 8 ingredients. Saved 4 weeks ago — never cooked."

Once you know why you saved it, the decision to keep or delete becomes instant.

How It Works (Tech Stack)

1. Screenshot Detection

iOS exposes PHAssetMediaSubtype.photoScreenshot — no ML needed to find screenshots. PhotoKit handles ingestion and change observation so new screenshots appear automatically.

2. On-Device OCR

Apple's Vision framework (VNRecognizeTextRequest) extracts all text from each screenshot. Runs on a background queue, results cached to SQLite.

3. Rule-Based Classification

No GPT, no cloud, no API calls. A keyword + regex classifier buckets screenshots into:

Receipt (price patterns, "total", "invoice")
Travel (flight codes, "boarding pass", "gate")
Recipe ("ingredients", "tbsp", "preheat")
Product ("add to cart", "buy now", "sale")
Code ("func", "const", "import")
Meme/Other (fallback)

4. Context Generation

A template engine turns classification + extracted data into human-readable sentences. Dates, prices, source apps, and time-since-saved are all woven in.

5. Duplicate Detection

Perceptual hashing (pHash): resize to 8x8 grayscale, compute bit string, compare with Hamming distance < 10.

6. Expiry Detection

Regex extracts dates from travel screenshots, compares with current date. Expired boarding passes = safe to delete.

Architecture

SwiftUI (Splash, Scan, Home, Settings)
    +
UIKit (Inbox card stack with UIPanGestureRecognizer swipes)
    |
PhotoKit → Vision OCR → Classifier → Sentence Engine → GRDB/SQLite

No backend. No API calls. No user accounts.
100% on-device. Screenshots never leave the phone.
Hybrid SwiftUI + UIKit. SwiftUI for static screens, UIKit for the gesture-heavy card stack.

Results

My first real session:

634 screenshots scanned in ~90 seconds
89 expired items auto-detected
42 duplicates found
612 cleaned in 4 minutes
Liberated ~1.2 GB

What I Learned

Rule-based AI is underrated. For a constrained domain like screenshots, regex + keywords outperform LLMs on speed, cost, and privacy. No hallucinations either.
The swipe UX is the product. The AI just enables fast decisions — the card stack + gesture interface is what makes it feel good.
"Why you saved it" > "organize." Users don't want another filing system. They want closure on unfinished intentions.
iOS has great primitives. PHAsset.photoScreenshot, Vision OCR, GRDB — the OS gives you everything you need for a privacy-first AI app.

Try It

Snaap is free on the App Store: https://apps.apple.com/app/snaap-voucher-reminder-ai/id6770817204

Would love feedback from the dev community — especially on the classification approach and swipe UX!

Built with Swift, SwiftUI, UIKit, Vision, PhotoKit, and GRDB. iOS 16+.