How I built auto-zoom for screen recordings that you can actually edit (Swift + ScreenCaptureKit)

I make small Mac apps you buy once. The newest is MacMuse Recorder, a screen recorder whose main feature is automatic zoom: you click around while recording, and the finished video pushes in on the part that matters.

Screen Studio made that look popular. What I wanted on top of it was control: every zoom should be an ordinary block on the timeline, so when the guess is wrong you drag it, retime it or delete it instead of fighting it.

This post is how that works under the hood. Everything is Swift on macOS 15+, and the snippets are the real code.

1. Record more than pixels

While ScreenCaptureKit writes the video, the app keeps three other streams:

• Cursor samples.
• Clicks, from an NSEvent global monitor for mouse-down events.
• Activity. ScreenCaptureKit attaches dirty rects to each frame: the regions that changed since the last one. I boil them down to about 10 samples a second, each a bounding box plus the fraction of the screen that changed.

struct Activity: Codable, Equatable {
    var t: Double, x0: Double, y0: Double, x1: Double, y1: Double, a: Double
}

Activity is what lets a zoom follow typing, menus and dialogs, not just clicks. But not every change deserves a zoom. A scroll or an app switch changes the whole screen; a blinking caret changes almost nothing:

/// A change is worth zooming to when it's localized;
/// whole-screen changes (scrolls, app switches) aren't.
static func focusable(_ a: Activity) -> Bool {
    a.a > 0.0002 && a.a < 0.3 && a.w < 0.55 && a.h < 0.55
}

2. Turn events into zoom blocks

Clicks and focusable changes are sorted by time and grouped. An event joins the current group when it's within 1.5 s of the previous event and within 0.3 (in normalized screen units) of the group's centre. A group needs at least two events or one click, so a clock ticking in the menu bar never earns a zoom.

Each group becomes a block. It starts half a second before the first event, lasts at least 1.6 s, and its scale depends on how spread out the action was, between 1.5× and 2.4×:

let start = max(0, g.first!.t - 0.5)
let end = min(rec.duration, max(g.last!.t + 1.2, start + 1.6))
let scale = (clamp(0.55 / max(ext, 0.2), 1.5, 2.4) * 10).rounded() / 10

Then neighbours less than a second apart are merged, so the camera pans from one spot to the next instead of bouncing out and back in. Merged blocks are capped at 14 s, so a long session still pulls back now and then to show context.

That is the whole "smart" part: about 40 lines of grouping. The output is plain data (start, end, scale), which is exactly why it can sit on a timeline and be edited like anything else.

3. Where the camera looks

The blocks decide how much to zoom. Where to look comes from a focus track: the cursor, read 80 ms ahead, run through a critically damped spring that settles in about 0.4 s.

One extra rule makes typing look right: if the cursor has been still for a second while something nearby keeps changing, the target drifts to that change. You click into a text field, move the mouse out of the way, type, and the view follows the text.

4. Springs that carry their velocity

My first version eased in and out of each zoom separately. Two zooms close together looked like a stutter: out, stop, in.

The fix was to compute the camera's whole path once per edit, as springs sampled at 120 Hz, using the exact solution for a critically damped spring so velocity carries through every change of target:

private struct Spring {
    var x: Double, v = 0.0
    mutating func step(_ target: Double, w: Double, dt: Double) {
        let offset = x - target
        let slope = v + w * offset
        let decay = exp(-w * dt)
        x = target + (offset + slope * dt) * decay
        v = (v - w * slope * dt) * decay
        if abs(target - x) < 1e-5, abs(v) < 1e-4 { x = target; v = 0 }
    }
}

Three small rules sit on top:

• While zoomed in, the view holds still for small movements and only re-aims when the action leaves the middle of the frame. A camera that follows every twitch is tiring to watch.
• Between zooms it looks 0.45 s ahead, so the next zoom heads straight for its target.
• The zoom-out starts a little before the block ends, unless another block follows within 0.7 s, in which case it flows straight into that one.

Because the path is a pure function of the edit, the preview, scrubbing and the export always agree. There's no "it looked different in the export".

5. A vertical video from a wide recording

The same take can export at 9:16 for Reels, TikTok and Shorts. For square and vertical frames the renderer first crops in so the screen fills the frame, then lets zooms stack on top at 40% of their strength, and clamps the view so it stays on the screen rather than the background around it:

let base = max(1, H * style.aspect.fill / rect.height)
scale = base > 1 ? base * (1 + (z.scale - 1) * 0.4) : z.scale

6. Playback: from Core Image to one Metal pass

The editor preview started out on Core Image. Core Image plans a new render graph for every frame, and that planning was most of what playback cost. Drawing the common frame directly on the GPU in a single Metal pass took playback from about 56% CPU to about 22%. Any frame the Metal path can't draw still falls back to the Core Image renderer.

Things that bit me

• Screen Recording permission is tied to the code signature. After a re-sign, the switch in System Settings still looks on but capture is refused, and toggling it doesn't help. Removing the entry lets macOS ask again. Also, macOS only gives screen access to a process started after it was granted, so the app relaunches itself after you approve.
• Permission dialogs appear for the frontmost app. If you hide your window before asking, the dialog can wait unseen. Set up capture while your window is still in front.
• Write fragmented movie files. If the app crashes or the Mac loses power, the recording stays playable up to the last few seconds.
• An auto-hidden Dock looks like activity. It shows up as a short, wide change flush against a screen edge. Two of those (in and out) are enough to detect it and crop it from the frame.

Try it

MacMuse Recorder is $19.99 once for one Mac (no subscription, no account), for Apple silicon Macs on macOS 15 or later. There's a 27-second demo on the site: recorder.macmuse.app.

If you're weighing options, I also wrote an honest comparison of Screen Studio alternatives for Mac, including the ones that aren't mine.

Happy to answer ScreenCaptureKit questions in the comments.

Comments

[DemoVexa]

Making every zoom an ordinary timeline block is the right call. The guess will be wrong some of the time, and a block you can drag is far less annoying than a smart camera you have to argue with. Using dirty rects is clever too, since typing and dialogs are where click only auto zoom falls flat. One question on that: how do you treat a toast or notification that pops up in a corner during a take? It's small and short, so it passes the focusable check, but it's rarely what the viewer should look at. I'd also like to know whether projects keep the click and activity streams, so an old recording can be zoomed again after you improve the grouping rules.