Local Apple Vision OCR for uncopyable macOS errors and two-column PDFs

#macos #swift #ocr #vision

The problem: the text exists, but you cannot select it

On macOS, a lot of the text that blocks you is drawn, not selectable:

Unity's console output (NullReferenceException: Object reference not set to an instance of an object)
Blender's status/error dialogs (Error: Cannot read file "...": No such file or directory)
Xcode's red inline diagnostics
Gatekeeper and permission dialogs ("AppName" cannot be opened because the developer cannot be verified.)
Any English text baked into a screenshot or a scanned PDF

Copy-paste translation pipelines fail on all of these, because there is no text layer to copy. And text-extraction order from a two-column PDF frequently interleaves the columns, so pasting into a translator produces scrambled sentences even when extraction succeeds.

So the job is: read what is on screen, in place, without shipping the screenshot anywhere.

Approach: on-device OCR, then translate the text only

The split I settled on:

Stage	Where it runs	What leaves the machine
Screen capture → text recognition	On-device (Apple Vision, `VNRecognizeTextRequest`)	nothing — pixels stay local
Recognized text → translation	Online	the recognized string only

This is deliberate. Sending screenshots of internal code, customer names, or unreleased assets to a cloud OCR endpoint is a decision most of us are not authorized to make. On-device recognition removes that decision from the critical path.

It is not a "100% offline" tool. Translation needs the network. That boundary is worth stating plainly, because NSURLErrorDomain error -1009 (offline) is a real failure mode, not a bug.

Implementation notes that actually matter

import Vision

func recognize(_ cgImage: CGImage, orientation: CGImagePropertyOrientation = .up) throws -> [String] {
    let request = VNRecognizeTextRequest()
    request.recognitionLevel = .accurate       // small UI text needs .accurate
    request.recognitionLanguages = ["en-US", "ja-JP"]
    request.usesLanguageCorrection = false     // never "fix" identifiers
    request.minimumTextHeight = 0.0

    let handler = VNImageRequestHandler(cgImage: cgImage, orientation: orientation)
    try handler.perform([request])

    return (request.results ?? []).compactMap { $0.topCandidates(1).first?.string }
}

Three things that cost me the most time:

usesLanguageCorrection must be false for code. With correction on, E0382 and dmodel get "helpfully" rewritten, and you can no longer search for them.
Pass the orientation. Rotated scans silently return garbage unless you hand VNImageRequestHandler a CGImagePropertyOrientation.
Rebuild reading order from boundingBox. For two-column layouts, cluster observations into rows by y, then sort each cluster by x. Reading order is a layout problem, not an OCR problem.

Two-column PDFs: why region capture helps

A region capture preserves the visual reading path, so the left column's continuation does not get glued to the right column's opening. Nothing magic here — it is a different extraction route, and that difference is what fixes the symptom.

What it is

This is a small macOS app I built for my own workflow and now distribute:

macOS 13.0+ / Apple Silicon (arm64) only — does not run on Intel Macs
No account, no sign-up
Free to use

Download page (with campaign parameters): https://www.zipp-mac.com/?utm_source=devto&utm_medium=social

Source and builds: https://github.com/dayu-999/zipp-mac

First launch on an unsigned/not-notarized build: right-click → Open (the standard Gatekeeper path — the app is not "damaged").

Limits worth repeating

Translation is online; offline means no translation.
The screenshot is not uploaded, but the recognized text is.
Apple Silicon only.
I am not claiming a latency figure here. Numbers on a dev machine do not transfer to yours.

Closing

If your blocker is "the error is on screen but I can't copy it", local OCR is the shortest path that does not require a policy decision about sending your screen to someone else's server. The translation step can stay online; the pixels do not have to leave.

DEV Community