Xcode and Unity errors are often visible on screen but impossible to select. Runtime crash dialogs, plugin warnings rendered by native UI, Electron renderer panes — ⌘C returns nothing because these surfaces are not text views. If your translation workflow depends on the clipboard, it silently dies there.
This is an implementation note on reading those errors with Apple Vision (VNRecognizeTextRequest) from a screen rectangle: how to capture, how to convert coordinates, how to reduce misreads, and where the approach honestly stops.
Capturing the screen: CGWindowList vs ScreenCaptureKit
CGWindowListCreateImage lets you grab a single window, which is less noisy than the full desktop. It is deprecated from macOS 14 onward, so treat it as a fallback.
SCScreenshotManager / SCStream is the current path, available per-window and per-display. It is async, which makes cancellation easier when the user re-triggers the shortcut.
Both require Screen Recording permission, and both fail silently without it: the call succeeds and returns an image of the wallpaper. Guard with a cheap check — if the image is nearly uniform, treat it as a failed capture.
Retina: points are not pixels
Window coordinates are points; the captured image is in pixels. On a Retina display the factor is 2, so using points directly crops half the region.
let scale = window.backingScaleFactor
let pixelRect = CGRect(x: pointRect.minX * scale,
y: pointRect.minY * scale,
width: pointRect.width * scale,
height: pointRect.height * scale)
Two flips then stack up:
- Window coordinates are top-left origin;
CGImagecropping is also top-left. - Vision's
boundingBoxis normalized with a bottom-left origin.
To map a result back to the image:
let x = obs.boundingBox.minX * CGFloat(width)
let y = (1 - obs.boundingBox.maxY) * CGFloat(height)
Mixing these two flips produces the classic "top edge lines up, bottom edge drifts" bug. Keep the transform in one function and test it by placing known glyphs in the four corners.
Request configuration that survives error logs
let request = VNRecognizeTextRequest()
request.recognitionLevel = .accurate
request.recognitionLanguages = ["en-US", "ja-JP"]
request.usesLanguageCorrection = false
request.minimumTextHeight = 0.008
-
usesLanguageCorrection = false— stack traces and paths are full of tokens that are not dictionary words; correction rewrites them into something else. -
minimumTextHeight— the default silently drops small text. Measure what 11pt looks like as a fraction of image height in your capture. -
recognitionLanguages— order matters for mixed-language UI.
Rectangle hygiene
Accuracy is mostly decided by padding, not by the model.
- Too tight: glyph antialiasing is cut,
rnbecomesm,1becomesl. - Too loose: adjacent columns and line numbers leak in.
At 2x scale, 4–8px of padding worked. Three preprocessing steps earned their keep: grayscale (subpixel color fringing), contrast normalization (translucent dark-mode windows), and 2x upscaling with bicubic interpolation for small captures.
Reassembling lines
VNRecognizedTextObservation arrives per line. Group by vertical gaps: within 1.5x the glyph height is the same paragraph, beyond that is a new block. For stack traces, keep boundingBox.minX so indentation can be restored — depth is information.
Where this stops working
- Text too small to resolve on screen (zooming does not create information)
- Low contrast: pale grey on white, overlays, watermarks
- Rotated or heavily italic text
- Lines of pure punctuation (
},-->) that need context - Math: sub/superscript association is lost in any linear reading
If a human cannot read it on screen, the rectangle path will not either. Zoom first, then select.
See it running
Demo of macos
swift
ocr
xcode
the rectangle workflow (YouTube):
https://www.youtube.com/watch?v=4_tQzj7VB_I
Reference tool: Zipp (Mac)
https://www.zipp-mac.com/?utm_source=devto&utm_medium=article&utm_campaign=xcode_vision_ocr
Takeaway
Unselectable errors are not "untranslatable" — they are simply not on the clipboard. Get capture permission and failure detection right, keep points and pixels separate, and isolate the Vision coordinate flip. Everything after that is padding and minimumTextHeight tuning.
Top comments (0)