How Mac apps add text suggestions to every app: the Accessibility API explained

Disclosure: I build DraftKey, an on-device writing assistant for Mac that's built on the APIs below. It comes up once at the end. This post is about the platform, not the product.

Grammarly, autocomplete tools, text expanders, window managers and screen readers all have the same problem on macOS. They need to work inside apps they didn't write. Nearly all of them solve it with the Accessibility API, the same interface VoiceOver uses. If you've ever wondered why one of these tools works in Mail but not in some editor, or why it asks for a scary-sounding permission, this is how the pieces fit together.

Step 0: become a trusted client

Your app can't read other apps' UI until the user adds it under System Settings → Privacy & Security → Accessibility. You check for that and ask for it with one call:

import ApplicationServices

let opts = [kAXTrustedCheckOptionPrompt.takeUnretainedValue() as String: true] as CFDictionary
let trusted = AXIsProcessTrustedWithOptions(opts)   // shows the system prompt if needed

AXIsProcessTrustedWithOptions returns whether your process is a trusted accessibility client. With the prompt option set, macOS shows its own dialog that points the user to System Settings. You can't grant this permission in code, and that's on purpose.

The permission is broad. A trusted client can read the contents and structure of almost any app's windows. That's why it's worth knowing who made any tool you grant it to.

Step 1: find the focused text field

Everything in the Accessibility API is an AXUIElement, an opaque reference to a UI object in another process, with named attributes you can read. Start at the system-wide element and ask what has focus:

func focusedElement() -> AXUIElement? {
    let system = AXUIElementCreateSystemWide()
    var value: CFTypeRef?
    guard AXUIElementCopyAttributeValue(system, kAXFocusedUIElementAttribute as CFString,
                                        &value) == .success else { return nil }
    return (value as! AXUIElement)
}

From there, these are the attributes a writing tool cares about:

• kAXRoleAttribute / kAXSubroleAttribute: is this a text field, a text area, or something else?
• kAXValueAttribute: the full text of the field.
• kAXSelectedTextRangeAttribute: the selection as a character range (not bytes). An empty range gives you the caret position. Apple's docs say this attribute is required for all editable text elements, which is why it's the most reliable one to build on.
• kAXSelectedTextAttribute: just the selected text.

With the value and the caret position, you have the context a model needs: the text before the cursor, and maybe a little after it.

Step 2: find where the caret is on screen

To draw ghost text or a popup next to the cursor, you need screen coordinates. That's what parameterized attributes are for. kAXBoundsForRangeParameterizedAttribute takes a range and returns the bounding rectangle of that text as it appears on screen:

func caretRect(_ el: AXUIElement, location: Int) -> CGRect? {
    var range = CFRange(location: location, length: 0)
    guard let axRange = AXValueCreate(.cfRange, &range) else { return nil }
    var out: CFTypeRef?
    guard AXUIElementCopyParameterizedAttributeValue(
            el, kAXBoundsForRangeParameterizedAttribute as CFString, axRange, &out) == .success
    else { return nil }
    var rect = CGRect.zero
    AXValueGetValue(out as! AXValue, .cgRect, &rect)
    return rect
}

Two practical notes:

• Some apps return an empty rectangle for a zero-length range. A common fallback is to ask for the bounds of the character just before the caret and use its right edge.
• The rectangle uses top-left-origin screen coordinates, while AppKit windows use bottom-left. Flip the y value against the main screen's height before you position your overlay.

Step 3: react to typing without polling

You don't want to read the whole field 60 times a second. AXObserver lets you subscribe to notifications from another process:

var observer: AXObserver?
AXObserverCreate(pid, { _, element, notification, _ in
    // debounce, then re-read value + selection and request a suggestion
}, &observer)

let app = AXUIElementCreateApplication(pid)
AXObserverAddNotification(observer!, app, kAXFocusedUIElementChangedNotification as CFString, nil)
AXObserverAddNotification(observer!, app, kAXValueChangedNotification as CFString, nil)
AXObserverAddNotification(observer!, app, kAXSelectedTextChangedNotification as CFString, nil)

CFRunLoopAddSource(CFRunLoopGetCurrent(),
                   AXObserverGetRunLoopSource(observer!), .defaultMode)

You need one observer per app process, so watch NSWorkspace for app activation and move your observer as the user switches apps. Value-changed tells you the user typed. Selected-text-changed tells you the caret moved, which is often your cue to throw away a stale suggestion.

Step 4: put the text back

Once the user accepts a suggestion, there are two common ways to insert it:

1. Set the selected text. Writing to kAXSelectedTextAttribute with AXUIElementSetAttributeValue replaces the current selection, or inserts at the caret if nothing is selected. It's clean when it works, and it leaves the clipboard alone. Not every app supports writing it, so check the result code.
2. Paste it. Put the text on the pasteboard and post a synthesized ⌘V with CGEvent. It works in more places, but you have to save and restore the user's clipboard, and some apps handle pasted text differently from typed text.

Many tools try the first and fall back to the second.

Why it doesn't work everywhere

This is the part users actually notice.

Apps expose different amounts. Native AppKit text views implement everything above. Custom-drawn editors (some code editors, canvas-based web apps, games) may expose little or nothing. Your tool can only see what the app reports.

Electron and Chromium build their tree on demand. Per Electron's accessibility docs, Electron apps turn on accessibility support automatically when assistive technology like VoiceOver is running. A third-party tool can request it by setting the AXManualAccessibility attribute to true on the app's element. Until then, the text you're looking for may not be in the tree at all.

Secure fields are off-limits, and should be. Password fields have the subrole kAXSecureTextFieldSubrole, which Apple describes as a field for sensitive data that shows input as bullets. Skip them. Separately, when an app turns on Secure Event Input, which NSSecureTextField does automatically and Terminal and iTerm let you keep on, Apple's TN2150 explains that keyboard event taps stop receiving keystrokes. You can check this with IsSecureEventInputEnabled() and show the user why suggestions paused.

The Mac App Store is mostly out. Apple's App Sandbox documentation lists "use of accessibility APIs in assistive apps" among the things the sandbox forbids, and Mac App Store apps must be sandboxed. That's why most system-wide writing tools are downloads signed with Developer ID, not App Store apps. (Event taps that need only Input Monitoring are a narrower exception.)

Debugging tips

• Accessibility Inspector (Xcode → Open Developer Tool) shows exactly which attributes an app exposes. Check it before you blame your code.
• During development, macOS ties the permission to your code signature, so a rebuilt binary with a different signature can quietly lose trust. tccutil reset Accessibility <bundle-id> gives you a clean slate.
• Every AX call is a cross-process message to the target app. Keep them off the main thread, and set a timeout with AXUIElementSetMessagingTimeout so a frozen app can't freeze yours.

Where the model fits

All of the above is plumbing. The model only sees a string of context and returns a string. That separation is what makes on-device suggestions practical. The Accessibility layer collects the text locally, and if inference also runs locally, the text never has to leave the Mac.

That's the design I went with for DraftKey. It uses Accessibility to work in compatible text fields, skips secure fields, has per-app controls, and runs its writing model on the Mac. It hits the same limits described above, so it can't work in editors that don't expose their text.

If you're building something in this space, start with Accessibility Inspector and a single AXObserver. You'll learn most of what you need from watching which notifications actually fire.