Swift 6, ScreenCaptureKit, and why my app worked in Xcode but not as a .app

I created this post for the purposes of entering the Gemini Live Agent Challenge. I'm building VibeCat, a desktop AI companion that watches your screen and talks to you.

The backend was done. Nine agents, WebSocket proxy, Gemini Live API integration — all working. Time to build the macOS client. Swift 6. SwiftUI. ScreenCaptureKit. How hard could it be?

Three days. Three days of things silently not working, with zero error messages.

the screen capture that captured nothing

VibeCat needs to see your screen to be useful. The VisionAgent on the backend analyzes screenshots to detect errors, notice you're stuck, or see tests pass. So the client needs ScreenCaptureKit.

The code itself is clean:

@MainActor
final class ScreenCaptureService {
    func captureAroundCursor() async -> CaptureResult {
        do {
            let image = try await performCapture(fullWindow: false)
            if !ImageDiffer.hasSignificantChange(from: lastImage, to: image) {
                return .unchanged
            }
            lastImage = image
            return .captured(image)
        } catch {
            return .unavailable(error.localizedDescription)
        }
    }

    private func performCapture(fullWindow: Bool) async throws -> CGImage {
        let content = try await SCShareableContent.excludingDesktopWindows(
            false, onScreenWindowsOnly: true
        )
        guard let display = content.displays.first else {
            throw CaptureError.noDisplay
        }

        // Exclude VibeCat's own windows
        let excludedApps = content.applications.filter { app in
            app.bundleIdentifier == Bundle.main.bundleIdentifier
        }

        let filter = SCContentFilter(
            display: display,
            excludingApplications: excludedApps,
            exceptingWindows: []
        )
        let config = SCStreamConfiguration()
        config.width = 1280
        config.height = 720
        config.pixelFormat = kCVPixelFormatType_32BGRA
        config.showsCursor = false

        return try await SCScreenshotManager.captureImage(
            contentFilter: filter, configuration: config
        )
    }
}

Ran it in Xcode. Screen capture worked perfectly. Built a .app bundle with swift build. Ran it. Screen capture silently returned nothing. No error. No crash. Just... nothing.

The entitlement. The com.apple.security.screen-recording entitlement was in the Xcode project but wasn't getting embedded in the SPM-built binary. macOS doesn't throw an error when you try to capture without the entitlement — ScreenCaptureKit just quietly returns empty content. You get an empty displays array and no indication why.

I added it to VibeCat.entitlements and passed it via codesign:

codesign --force --entitlements VibeCat/VibeCat.entitlements \
  --sign - .build/release/VibeCat

First lesson: ScreenCaptureKit fails silently. If your capture returns nothing, check your entitlements before you check your code.

the image differ — because you don't send every frame

The companion captures your screen periodically, but you don't want to send every single frame to the backend. If your screen hasn't changed, there's nothing new to analyze. So I built a pixel-level change detector:

public enum ImageDiffer {
    private static let thumbnailSize = 32

    public static func hasSignificantChange(
        from previous: CGImage?,
        to current: CGImage,
        threshold: Double = 0.05
    ) -> Bool {
        guard let previous else { return true }
        guard let prevThumb = thumbnail(previous),
              let currThumb = thumbnail(current) else { return true }
        let diff = pixelDiff(prevThumb, currThumb)
        return diff > threshold
    }

    private static func pixelDiff(_ a: [UInt8], _ b: [UInt8]) -> Double {
        guard a.count == b.count, !a.isEmpty else { return 1.0 }
        var total: Double = 0
        for i in stride(from: 0, to: a.count, by: 4) {
            let dr = Double(a[i]) - Double(b[i])
            let dg = Double(a[i+1]) - Double(b[i+1])
            let db = Double(a[i+2]) - Double(b[i+2])
            total += sqrt(dr*dr + dg*dg + db*db) / (255.0 * sqrt(3.0))
        }
        return total / Double(a.count / 4)
    }
}

It's a static method on an enum (no cases — just a namespace for functions). Downscale both images to 32×32, compute Euclidean distance in RGB space per pixel, average across all pixels. If the difference exceeds 5%, it's a "significant change" worth sending.

Why enum instead of struct? Because a struct can be accidentally instantiated. An enum with no cases is pure namespace — you can't create an instance of ImageDiffer. It's a Swift pattern for grouping static utility functions.

Bundle.main.resourcePath — the Xcode lie

This one hurt. In SpriteAnimator, I needed to load PNG sprite frames from Assets/Sprites/cat/. First attempt:

let path = Bundle.main.resourcePath! + "/Assets/Sprites/\(char)"

Works perfectly in Xcode. The ! force-unwrap succeeds. Files are found. Sprites animate.

Run the same binary outside Xcode? Bundle.main.resourcePath is nil. Force-unwrap crashes. Silent death.

The issue: when Xcode runs your app, Bundle.main points to your project directory structure where everything is available. When you build with SPM and run the .app independently, Bundle.main.resourcePath often returns nil because resources aren't in the expected bundle location.

The fix was a findRepoRoot() function that walks up from both the working directory and the bundle URL:

private func findRepoRoot() -> URL {
    // Try working directory first
    var url = URL(fileURLWithPath: FileManager.default.currentDirectoryPath)
    for _ in 0..<6 {
        if FileManager.default.fileExists(
            atPath: url.appendingPathComponent("Assets/Sprites").path
        ) {
            return url
        }
        url = url.deletingLastPathComponent()
    }
    // Fallback: walk up from bundle URL
    var bundleURL = Bundle.main.bundleURL
    for _ in 0..<6 {
        if FileManager.default.fileExists(
            atPath: bundleURL.appendingPathComponent("Assets/Sprites").path
        ) {
            return bundleURL
        }
        bundleURL = bundleURL.deletingLastPathComponent()
    }
    return URL(fileURLWithPath: FileManager.default.currentDirectoryPath)
}

It's not pretty. But it works in Xcode, in a standalone .app, and when running from the terminal in the repo root. I used the same pattern in BackgroundMusicPlayer for finding Assets/Music/.

the NSWindow.isVisible trap

Swift 6 strict concurrency plus AppKit is a minefield. Here's one that's particularly evil: NSWindow has a built-in property called isVisible. If you define your own stored property with the same name in a subclass or extension, Swift doesn't warn you — it just breaks.

I had:

class CompanionPanel: NSPanel {
    var isVisible: Bool = false  // ← shadows NSWindow.isVisible
}

This compiles. It even seems to work at first. But NSWindow.isVisible is a computed property tied to the window server. My stored property hid it. Window visibility checks started returning wrong values. The panel would appear/disappear at random.

The fix was just a rename:

var hudVisible: Bool = false

No warning from the compiler. No runtime error. Just subtle incorrectness that took hours to track down.

@MainActor everywhere

Swift 6 requires @MainActor on anything that touches AppKit. In Swift 5 you could get away with updating UI from background threads — the app would work until it didn't. Swift 6 is strict: if a class touches NSWindow, NSImage, or any AppKit type, it must be @MainActor.

Every service class in VibeCat is @MainActor:

@MainActor
final class ScreenCaptureService { ... }

@MainActor
final class SpriteAnimator { ... }

@MainActor
final class BackgroundMusicPlayer { ... }

But Timer callbacks aren't @MainActor by default. So this pattern:

Timer.scheduledTimer(withTimeInterval: 0.12, repeats: true) { _ in
    self.advanceFrame()  // ❌ Not on MainActor
}

Has to become:

Timer.scheduledTimer(withTimeInterval: 0.12, repeats: true) { [weak self] _ in
    Task { @MainActor [weak self] in
        self?.advanceFrame()  // ✅ MainActor
    }
}

Every Timer. Every callback. Every closure that touches UI. Wrap it in Task { @MainActor in }. Swift 6 is safer, but the migration tax is real.

the client is deliberately dumb

One design principle I'm proud of: the macOS client is deliberately dumb. It captures screens, plays audio, animates sprites, and shuttles data to the backend. It makes zero AI decisions.

When the client captures a screenshot, it doesn't analyze it — it sends the raw image to the backend's /analyze endpoint. When the backend says "set character to surprised," the client just changes the sprite state. When the backend says "play this audio," the client plays it.

This is a challenge requirement (all AI through backend), but it's also good architecture. The client is ~1,970 lines of Swift. The backend is ~2,900 lines of Go. If I need to change how VibeCat responds to errors, I never touch the client.

The smartest thing the client does is the ImageDiffer — and even that is just an optimization to avoid sending unchanged frames, not an AI decision.

what I'd do differently

If I were starting over:

1. Test outside Xcode from day one. Every feature should be verified as a standalone .app, not just in the Xcode debug session. The silent failures cost me a full day.

2. Use a resource bundle properly. The findRepoRoot() hack works, but it's fragile. A proper SPM resource bundle with Bundle.module would be cleaner.

3. Start with Swift 6 strict concurrency enabled. I started with Swift 5 mode and migrated. The migration was painful — dozens of @MainActor annotations and callback wraps. Starting strict would have caught these at write-time instead of all-at-once.

But it works. The cat sees your screen. The sprites animate. The music plays. And the client stays dumb enough to let the backend do the thinking.

The moment I ran the codesigned .app outside Xcode for the first time — double-clicked it from Finder, no debugger, no safety net — and the cat appeared on my desktop, captured my screen, and waved at me? That was the best moment of this entire project. Three days of silent failures, for ten seconds of a pixel cat saying hello.

---

Building VibeCat for the Gemini Live Agent Challenge. Source: github.com/Two-Weeks-Team/vibeCat