DEV Community

Zane — 個人開発メモ
Zane — 個人開発メモ

Posted on

Reading unselectable Xcode/Unity errors: rectangle OCR with Apple Vision on macOS

Xcode and Unity errors are often visible on screen but impossible to select. Runtime crash dialogs, plugin warnings rendered by native UI, Electron renderer panes — ⌘C returns nothing because these surfaces are not text views. If your translation workflow depends on the clipboard, it silently dies there.

This is an implementation note on reading those errors with Apple Vision (VNRecognizeTextRequest) from a screen rectangle: how to capture, how to convert coordinates, how to reduce misreads, and where the approach honestly stops.

Capturing the screen: CGWindowList vs ScreenCaptureKit

CGWindowListCreateImage lets you grab a single window, which is less noisy than the full desktop. It is deprecated from macOS 14 onward, so treat it as a fallback.

SCScreenshotManager / SCStream is the current path, available per-window and per-display. It is async, which makes cancellation easier when the user re-triggers the shortcut.

Both require Screen Recording permission, and both fail silently without it: the call succeeds and returns an image of the wallpaper. Guard with a cheap check — if the image is nearly uniform, treat it as a failed capture.

Retina: points are not pixels

Window coordinates are points; the captured image is in pixels. On a Retina display the factor is 2, so using points directly crops half the region.

let scale = window.backingScaleFactor
let pixelRect = CGRect(x: pointRect.minX * scale,
                       y: pointRect.minY * scale,
                       width: pointRect.width * scale,
                       height: pointRect.height * scale)
Enter fullscreen mode Exit fullscreen mode

Two flips then stack up:

  1. Window coordinates are top-left origin; CGImage cropping is also top-left.
  2. Vision's boundingBox is normalized with a bottom-left origin.

To map a result back to the image:

let x = obs.boundingBox.minX * CGFloat(width)
let y = (1 - obs.boundingBox.maxY) * CGFloat(height)
Enter fullscreen mode Exit fullscreen mode

Mixing these two flips produces the classic "top edge lines up, bottom edge drifts" bug. Keep the transform in one function and test it by placing known glyphs in the four corners.

Request configuration that survives error logs

let request = VNRecognizeTextRequest()
request.recognitionLevel = .accurate
request.recognitionLanguages = ["en-US", "ja-JP"]
request.usesLanguageCorrection = false
request.minimumTextHeight = 0.008
Enter fullscreen mode Exit fullscreen mode
  • usesLanguageCorrection = false — stack traces and paths are full of tokens that are not dictionary words; correction rewrites them into something else.
  • minimumTextHeight — the default silently drops small text. Measure what 11pt looks like as a fraction of image height in your capture.
  • recognitionLanguages — order matters for mixed-language UI.

Rectangle hygiene

Accuracy is mostly decided by padding, not by the model.

  • Too tight: glyph antialiasing is cut, rn becomes m, 1 becomes l.
  • Too loose: adjacent columns and line numbers leak in.

At 2x scale, 4–8px of padding worked. Three preprocessing steps earned their keep: grayscale (subpixel color fringing), contrast normalization (translucent dark-mode windows), and 2x upscaling with bicubic interpolation for small captures.

Reassembling lines

VNRecognizedTextObservation arrives per line. Group by vertical gaps: within 1.5x the glyph height is the same paragraph, beyond that is a new block. For stack traces, keep boundingBox.minX so indentation can be restored — depth is information.

Where this stops working

  • Text too small to resolve on screen (zooming does not create information)
  • Low contrast: pale grey on white, overlays, watermarks
  • Rotated or heavily italic text
  • Lines of pure punctuation (}, -->) that need context
  • Math: sub/superscript association is lost in any linear reading

If a human cannot read it on screen, the rectangle path will not either. Zoom first, then select.

See it running

Demo of macos
swift
ocr
xcode
the rectangle workflow (YouTube):
https://www.youtube.com/watch?v=4_tQzj7VB_I

Reference tool: Zipp (Mac)
https://www.zipp-mac.com/?utm_source=devto&utm_medium=article&utm_campaign=xcode_vision_ocr

Takeaway

Unselectable errors are not "untranslatable" — they are simply not on the clipboard. Get capture permission and failure detection right, keep points and pixels separate, and isolate the Vision coordinate flip. Everything after that is padding and minimumTextHeight tuning.

Top comments (0)