Moving focus and cursor between monitors with head pose on macOS

I have two monitors, and I kept typing into the window on the screen I'd just looked away from. Focus stays where it was, but my eyes have moved on. I made a menu bar app to fix it. It moves cursor and focus to the display you're facing.

The code is at https://github.com/arefbhrn/FocusFollow (MIT). This post is about how it works and what didn't.

Why head pose and not eye tracking

Proper eye tracking needs hardware a Mac doesn't have. But to tell two monitors apart you don't need to know where your eyes land. If you look at the left screen, your head turns left. Yaw and pitch are enough.

Vision gives you those for free. VNDetectFaceRectanglesRequest (revision 3) returns yaw, pitch and roll for a face. I run the camera at 720p and cap it at 15 fps through AVCaptureSession. That's plenty for head direction and the CPU barely notices. Each frame is analysed and thrown away.

Calibration and classifying

For each display you look at it for about 20 seconds, and I store the mean and standard deviation of yaw and pitch. The calibration is saved per display arrangement, so plugging in a monitor invalidates it.

At runtime, each pose gets a distance to every screen, measured in standard deviations:

for (id, screen) in stats {
    let yaw = (pose.yaw - screen.meanYaw) / max(screen.stdYaw, minimumStd)
    let pitch = (pose.pitch - screen.meanPitch) / max(screen.stdPitch, minimumStd)
    distances[id] = (yaw * yaw + pitch * pitch).squareRoot()
}

The nearest screen wins, unless it's still too far away, in which case the answer is "away" and nothing happens. That covers looking at your phone or your keyboard.

The minimumStd floor (2 degrees) came from a bug. If you sit very still during calibration the deviation gets tiny, and then any small movement makes the classifier think you're looking somewhere else. A floor keeps it forgiving.

Making it not annoying

A plain classifier flips back and forth constantly, so the first version was unusable. Two things fixed most of it.

One is a dwell filter. A result only counts after it's been stable for a configurable delay, so glances get ignored. The other is that it holds while you're typing or moving the mouse. I get that from NSEvent global monitors and only keep timestamps, never what was typed. It also stops on sleep, lock and screen saver.

Moving focus

Once a screen is confirmed I look up the last window you used on it with CGWindowListCopyWindowInfo, activate that app, and raise the window through the Accessibility API. Moving the cursor too is optional, with CGWarpMouseCursorPosition.

What doesn't work

It couldn't separate two windows on the same screen perfectly. I tried a window mode with a 25 dot gaze grid per screen and a fit on [1, yaw, pitch]. It works for big windows that don't overlap, and not much beyond that, so it's experimental and off by default.

Accuracy also depends a lot on where the camera is and how it's lit. Minimized, hidden and other-Space windows are never focused.

Privacy

It's a camera app, so I wanted this to be checkable. Frames stay in memory, there's no networking code in the project, and Accessibility is used to see when you type, not what. The whole thing is small enough to read.

Try it

macOS 14 or newer, free. The release is Apple silicon. It isn't notarized, so the first launch needs Open Anyway in Privacy & Security. If you try it, I'd like to know how it does with other camera positions and monitor layouts.