On-Device OCR in SwiftUI: A Practical Guide to the Vision Framework

Cloud OCR APIs are the default reach for text-in-image problems, and for most of what I build they're the wrong default: they add a network dependency, a per-request cost, and a round trip for something the phone in your hand can already do, offline, in under a second. The Vision framework's text recognition has been production-ready for years. Here's the version of the guide that skips the "hello world" and goes straight to the parts that actually matter in a shipping app.

The minimum viable request

import Vision

func recognizeText(in image: CGImage) async throws -> [String] {
    let request = VNRecognizeTextRequest()
    request.recognitionLevel = .accurate
    request.usesLanguageCorrection = true

    let handler = VNImageRequestHandler(cgImage: image, options: [:])
    try handler.perform([request])

    guard let observations = request.results else { return [] }

    return observations.compactMap { observation in
        observation.topCandidates(1).first?.string
    }
}

That's a working OCR pipeline in about fifteen lines. .recognitionLevel = .accurate matters — the .fast option trades meaningful accuracy for speed you usually don't need, since this whole pipeline typically finishes in well under a second on-device regardless.

Getting the region, not just the text

The naive version above throws away something you'll almost always want back: where on the image the text actually was. If you're building anything like a "draw a box, get the text in that box" feature, you need the bounding geometry, not just the string:

func recognizeText(in image: CGImage, croppedTo rect: CGRect? = nil) async throws -> [(text: String, box: CGRect)] {
    let request = VNRecognizeTextRequest()
    request.recognitionLevel = .accurate

    if let rect = rect {
        request.regionOfInterest = rect // normalized 0...1 coordinates, not pixels
    }

    let handler = VNImageRequestHandler(cgImage: image, options: [:])
    try handler.perform([request])

    return (request.results ?? []).compactMap { observation in
        guard let candidate = observation.topCandidates(1).first else { return nil }
        return (candidate.string, observation.boundingBox)
    }
}

regionOfInterest is the detail that makes a "select an area to scan" feature fast instead of slow — you're telling Vision to only process the part of the image you actually care about, rather than running full-frame recognition and discarding most of the output.

The coordinate system will get you once

boundingBox comes back in Vision's coordinate space: normalized 0 to 1, with the origin in the bottom-left. SwiftUI's coordinate space has its origin in the top-left. If you draw a box directly using Vision's raw values, it will appear vertically flipped and you will spend twenty minutes convinced your math is wrong before remembering this. The fix is a single flip on the y-axis:

func convert(_ boundingBox: CGRect, in viewSize: CGSize) -> CGRect {
    CGRect(
        x: boundingBox.minX * viewSize.width,
        y: (1 - boundingBox.maxY) * viewSize.height,
        width: boundingBox.width * viewSize.width,
        height: boundingBox.height * viewSize.height
    )
}

Write this helper once, and every future OCR overlay you build stops being a coordinate-math debugging session.

What actually matters once you ship this

Downsample large images before recognition. A full-resolution photo from a modern iPhone camera is far more pixels than text recognition needs. Resizing to roughly 2000px on the long edge before running the request cuts processing time meaningfully with no accuracy loss for normal document or screen text.

usesLanguageCorrection is not free. It improves accuracy on natural-language text by leaning on a language model, but it can actively hurt accuracy on short strings, codes, or anything that isn't real words — serial numbers, for instance. Turn it off for those cases specifically rather than leaving it on globally.

Handle the empty-result case as a real UI state, not an afterthought. Blurry screenshots, glare, text at a bad angle — recognition legitimately returns nothing sometimes. A "no text found, try again" state is not optional polish, it's the difference between a tool that feels reliable and one that feels broken the first time someone points it at a low-contrast label.

Test against real screens, not just documents. Text rendered on an LCD or OLED screen — which is a common target if you're building anything like a screen-capture-to-text tool — behaves differently than ink on paper: sub-pixel rendering, moiré patterns from photographing a screen, and anti-aliasing all degrade recognition in ways a flat scanned document never will. If that's your use case, test it directly instead of assuming document-OCR quality carries over.

The whole pattern costs you an import and about thirty lines of real code, runs faster than any network call could, and never sends a user's document, screenshot, or handwritten note anywhere off their device. For most consumer OCR use cases, that's not a tradeoff — it's strictly better on every axis that matters.