DEV Community: Carolane Lefebvre

Understanding iOS Crashes in Production: A Sentry Guide for your app

Carolane Lefebvre — Mon, 12 Oct 2026 07:00:00 +0000

Your app is on the App Store. You check the reports: zero crashes. You tell yourself everything is fine, your app is perfect!
But does that mean it never crashes… or simply that you cannot see the crashes?

Imagine it crashes three times a day on your users' devices. You see nothing, so you fix nothing. And the person using it will not necessarily send you an email. They try again, it crashes again, and eventually they uninstall it.

Today, we will look at how to collect this information with Sentry, integrate it into a real app, and, above all, learn to read a report to understand what crashed and where to look in your code.

When crashes happen on a user's device, the developer needs visibility.

How does that work?

When your app crashes during development, you have Xcode right in front of you. You know what you just did, you can look at where things went wrong, and you can try to reproduce the problem.
But once your app is on someone else's device, you are no longer there to see what happens.

They might tell you, "It crashes when I open this page." OK. But which app version? On which iPhone? And, above all, what crashed in the code? Because when you open the same page on your own device, everything works.

What we need, then, is evidence we can use to investigate, even when we were not there when the crash happened.

That is where Sentry comes in. I already use it in Keepio, but today we will add it together to Caffio, another of my iOS apps.

The goal is not just to install it and say, "Well, that's done." We will trigger a test crash, see what gets reported, and learn to read the report. Having an error in a dashboard is one thing.
Understanding what it tells us so we can fix it is what matters.

What a crash report gives you

A crash report provides evidence from a process you were not watching: the failure, its stack trace, the app build, and device context. Depending on the configuration, breadcrumbs can also show some of the actions leading up to it.

An event is one occurrence. An issue groups related events. Ten events do not necessarily mean ten different bugs, and an event count is not the same thing as a count of affected people.

That distinction matters when deciding what to fix first. A launch crash affecting new users can be more urgent than a larger number of development-only errors.

Reporting also has limits. The SDK must be initialized, the event must be captured and delivered, and the project must accept it. An empty dashboard is not proof that every user had a crash-free experience.

Create a project for the app

In Sentry, create a project for your app (mine's gonna be Caffio) and select iOS as the platform. Choose the team that should have access and configure the notifications you want to receive.

1. Open the project list
Open Settings → Projects, or use your organization's menu to reach Projects.

From the project list, choose Create Project.

2. Choose the platform and project settings
Connecting a repository is optional for this setup.

Select iOS as the plateform

The products panel describes the available features. In these screenshots, error monitoring is marked Always on; this panel does not require selecting a checkbox.

Name the project with your app name and choose the team that should have access.

Configure alerts and their delivery channel. The source walkthrough uses High Priority Only and Email; choose the settings that suit your project, then create it.

3. Open manual configuration and find the DSN
Sentry offers a setup wizard. For this walkthrough, open Manual Configuration so we can add the SDK and configure it ourselves.

The exact onboarding labels can change. The important outputs are the project and its DSN, which tells the SDK where to send events. Copy the DSN from your project's setup instructions or Client Keys settings rather than constructing it yourself.

Keep the distinction between capture and notification in mind: an event appearing in Sentry and an email being sent are separate steps. Alert rules can filter or delay notifications.

We will use a DSN in the app and an authentication token for uploading debug symbols. They serve different purposes; the upload token belongs in your developer or CI environment, not in the application binary.

Okay, now that we have our project, let's get back to Xcode.

Add the Apple SDK through Swift Package Manager

In Xcode, open File → Add Package Dependencies.

Enter the official package URL: https://github.com/getsentry/sentry-cocoa.git

Add the Sentry product to the main application target. Adding it to the app does not automatically instrument a widget or another extension.

Check the SDK's compatibility with your deployment target before selecting a version. Sentry's manual setup currently documents iOS 15 or later for Cocoa SDK 9.x, and SDK 8.x for iOS 11–14.

Initialize it early

My app initializer creates a SwiftData ModelContainer. If the cloud-backed configuration fails, it attempts a local configuration. If both attempts fail, the current code calls fatalError.

That is a useful reminder about initialization order: starting crash reporting when the first view appears would miss the opportunity to observe failures earlier in startup.

The following is a partial initializer. Add it at the beginning of the existing app initialization and preserve the rest of your setup:

import Sentry

init() {
    SentrySDK.start { options in
        options.dsn = "YOUR_PROJECT_DSN"
        options.sendDefaultPii = false

        #if DEBUG
        options.environment = "development"
        options.debug = true
        #else
        options.environment = "production"
        #endif
    }

    // Continue the existing notification and ModelContainer setup here.
}

Initialize on the main thread as early as practical. In a SwiftUI app, Sentry documents using the App initializer. Its debug option prints SDK diagnostics; it does not switch crash reporting on.

The environment names above are a simple starting point. A Release build can also be a staging or TestFlight build, so use an explicit build configuration if those need separate environments.

Performance tracing and session replay are separate features. This guide focuses on crash reporting, so it does not enable them as part of the setup.

Give Sentry the matching debug symbols

Compiled code does not give you the source-level view you expect in Xcode. Debug symbols allow a crash address to be mapped back to a function and, when available, a file and line. This process is called symbolication.

The binary and its companion dSYM must have matching build UUIDs. The same app version number is not enough. The main executable, frameworks, and extensions can each have their own symbols.

In the target's Build Settings, check Debug Information Format. For the configuration you are testing or distributing, use DWARF with dSYM File if you need a separate dSYM to upload. Debug normally places symbols in the binary; Release normally produces the companion file.

Your app target: Build Settings → Debug Information Format

Add the temporary crash button from the next section before building, so its code belongs to the binary whose symbols you upload.

For a development build, use Product → Build, find the app in Products, and choose Show in Finder. Locate its companion dSYM in the build products. For a distribution build, retain the Xcode archive and its dSYMs.

Install the symbol-upload utility and authenticate:

brew install getsentry/tools/sentry-cli
sentry-cli login

These commands use sentry-cli, the debug-file upload utility, rather than the separate sentry developer CLI.

Upload the symbols for that exact build:

sentry-cli debug-files upload \
  --org YOUR_ORGANIZATION_SLUG \
  --project YOUR_PROJECT\
  "/path/to/YOUR_APP.app.dSYM"

Use the actual organization and project slugs. Check the command result and the project's Debug Files settings. Do not assume that generating a dSYM also uploaded it.

If a report stays partially symbolicated, compare the required UUID with the dSYM you uploaded:

dwarfdump --uuid "/path/to/YOUR_APP.app.dSYM"

Apple explains how to locate and compare the corresponding binary UUID in its missing-symbol guide.

For ongoing releases, automate symbol uploads in CI or your build workflow and retain the archives. Rebuilding the app can produce a different binary and matching dSYM pair.

Verify capture with a deliberate crash

Adding the package and getting a successful build are not evidence that a crash reaches the dashboard.

Temporarily place this button in a development-only screen that is reachable after launch:

#if DEBUG
Button("Test crash reporting") {
    SentrySDK.crash()
}
#endif

The screen also needs the usual SwiftUI and Sentry imports. Keep the crash behind a deliberate action. Putting it in the app initializer creates a launch crash loop and can prevent the next launch from delivering the saved report.

Build and install the app with the temporary button included.
Stop the Xcode run session.
Open the installed app from the device's home screen, without the debugger attached.
Tap the button and let the app terminate.
Reopen the app so the SDK can send the saved crash report.
Check the correct project and environment in Sentry and allow time for processing.

Sentry specifically requires testing crashes without an attached debugger. A captured message or handled error can test event delivery, but it does not prove that the crash handler works.

Remove the temporary button when finished. It has no place in a shipping build.

Read the report as an investigation

Start with the event's context: the release and build, environment, device, operating system, and when it happened. Confirm that you are inspecting your test rather than an unrelated event.

Then look at the failure and crashed thread. Find the first relevant frame in your own code and follow the callers around it. A system frame at the top does not automatically mean the operating system caused the bug.

Use breadcrumbs as supporting context when available. They are a partial trail of recorded activity, not a complete recording of the user's session.

Finally, look across events in the issue. Is it limited to a release or OS version? Does it affect a particular path? The stack trace helps locate the failure; the surrounding evidence helps formulate a reproduction.

If the report is unreadable, fix symbolication first. Guessing from hexadecimal addresses is a poor substitute for supplying the matching symbols.

Use an agent to help with the investigation

Once an issue exists, an agent can help gather the evidence and inspect the relevant code. Sentry's hosted MCP server supports OAuth and can be scoped to a project:

<https://mcp.sentry.dev/mcp/YOUR_ORGANIZATION_SLUG/YOUR_APP>

Add it through your client's supported MCP settings and complete authentication. Follow the current client instructions rather than guessing an endpoint from the app's DSN region. Official Sentry MCP setup

For example, ask the agent:

Inspect this issue. Identify the release, failure, and relevant application frames. Compare the report with the source code, propose a reproduction, and explain the smallest correction. Separate observed facts from hypotheses.

A Notion connection can help record the issue URL, reproduction, diagnosis, fix, and validation. Access to Sentry alone does not give the agent access to Notion (and that's only an advice I like to give, since it's my workflow!)

Review the proposed change and test the reproduction. Resolving a ticket or issue changes its tracking state; it does not deploy a corrected app to users.

Keep crash context useful and limited

A crash report should help diagnose a failure without becoming a copy of a your users data.

Disabling default PII collection is a starting point, not a guarantee that custom breadcrumbs or error messages contain no sensitive data. Review what you attach. Sentry supports beforeSend and beforeBreadcrumb hooks for filtering event and breadcrumb data.

Useful context can often be expressed as a state such as "empty summary" rather than raw records. If you want a reliable affected-user count, define an appropriate identification strategy; do not invent a user count from the number of events.

Make the workflow repeatable

For each release, keep the chain intact: initialize reporting early, identify the environment and build, upload matching symbols, and verify the event path. When an issue arrives, use its evidence to reproduce, fix, test, and follow the result after the updated app is released.

Sentry does not correct the bug for you. It gives you a better starting point than "it crashed on someone else's phone."
That is the part I want from crash reporting: enough context to turn a vague symptom into a concrete investigation.

That's it for now, have fun debugging!

HandKit: Controlling My HomeKit Lights With Hand Gestures on iOS

Carolane Lefebvre — Sat, 10 Oct 2026 09:22:24 +0000

Open sourced project here

It started with a question that was probably not worth spending two days on:

Could I use my fingers to control the lights in my room from a distance?

I didn't know much about computer vision. I had barely touched AVFoundation, Vision, or Core Media. I had never built a gesture recognition system.
Which, naturally, made me want to try it.

The idea behind HandKit was simple: point my iPhone's front camera at myself, recognize a few hand gestures, and translate them into actions in my HomeKit home.

The first two interactions I wanted were: pinch my thumb and index finger together to toggle a light. Move my index finger horizontally to control brightness from 0 to 100%.

The final interaction looks almost trivial. A pinch. A lamp turns on. But getting from a raw camera feed to an intentional, reliable gesture turned out to involve camera pipelines, pixel buffers, coordinate systems, noisy measurements, a little geometry, signal filtering, state machines, hysteresis, Swift Concurrency, and finally HomeKit.

This article is a walkthrough of how I built it, but more importantly, why the architecture and algorithms evolved the way they did.

The result

Starting with the camera

Before detecting a hand, I needed access to something much simpler: images.

On iOS, that starts with AVFoundation.
The capture pipeline can be simplified to:

AVCaptureDevice
        ↓
AVCaptureDeviceInput
        ↓
AVCaptureSession
        │
        ├────────→ AVCaptureVideoPreviewLayer
        │                   ↓
        │                 Preview
        │
        └────────→ AVCaptureVideoDataOutput
                            ↓
                       Video frames

Each component has a different responsibility.

AVCaptureDevice represents the physical capture device. In my case, the iPhone's front-facing camera.

AVCaptureDeviceInput turns that device into an input that can be attached to a capture session.

AVCaptureSession coordinates the capture pipeline.

From there, I needed the video in two different places. The first path goes to an AVCaptureVideoPreviewLayer, so I can see the camera feed on screen. The second goes to an AVCaptureVideoDataOutput, because displaying the video isn't enough. I need access to the individual frames if I want to analyze what my hand is doing.

That distinction became the foundation of HandKit:

Camera
  │
  ├──→ human sees the preview
  │
  └──→ application analyzes the frames

Managing the capture session with Swift Concurrency

I didn't want camera lifecycle operations scattered throughout the UI, so I introduced a CameraManager. It owns the capture session and is responsible for configuring, starting, and stopping it.
I implemented it as an actor.

actor CameraManager {
    let session = AVCaptureSession()

    // ...
}

Actors are useful here because they provide isolation around mutable state.
One thing I had to properly understand while building this was that an actor is not a thread.
An actor defines an isolation domain. Its executor determines where jobs isolated to that actor execute.
For the capture session, I wanted serialized execution on a dedicated queue, so I gave CameraManager a custom serial executor.
Conceptually:

CameraManager actor
        ↓
Custom SerialExecutor
        ↓
camera.session queue
        ↓
AVCaptureSession

A simplified version looks like this:

actor CameraManager {
    private let sessionQueue =
        DispatchSerialQueue(label: "camera.session")

    nonisolated var unownedExecutor: UnownedSerialExecutor {
        sessionQueue.asUnownedSerialExecutor()
    }

    // ...
}

This gave the camera pipeline a clear execution context without turning the MainActor into the place where capture work happens.
I also kept configuration inside the manager.

From outside, starting the camera shouldn't require knowing:

configure first
then start
but don't configure twice

That is an internal invariant.
So CameraManager tracks whether the session has already been configured and exposes a higher-level start() operation.
This was one of the first architectural principles that kept appearing throughout the project:
The caller should express an intention, not reproduce the internal procedure required to achieve it.

Keeping session management and frame processing separate

The next step was receiving frames from AVCaptureVideoDataOutput.
AVFoundation provides them through AVCaptureVideoDataOutputSampleBufferDelegate.

I initially considered making CameraManager the delegate too, but there was both a technical and architectural reason not to.
The delegate participates in Objective-C's NSObjectProtocol, while my camera manager is an actor. More importantly, managing the lifecycle of the camera and processing every video frame are different responsibilities.

So I introduced VideoOutputDelegate.

CameraManager
│
├── configure / start / stop
├── AVCaptureSession
│
└── VideoOutputDelegate
          ↓
       frames
          ↓
        Vision

I also separated their execution queues.

camera.session
├── configure
├── start
└── stop

frame.session
├── frame
├── frame
├── frame
└── Vision processing

This matters because analyzing an image can take time. I don't want that work unnecessarily blocking operations on the capture session itself.

It also gave the code a much cleaner conceptual boundary:
CameraManager manages capture. VideoOutputDelegate interprets what was captured.

A video is just a lot of images arriving very quickly
The delegate eventually receives:

func captureOutput(
    _ output: AVCaptureOutput,
    didOutput sampleBuffer: CMSampleBuffer,
    from connection: AVCaptureConnection
)

The important value here is the CMSampleBuffer.
I initially thought of a camera frame as essentially an image, but the media pipeline is richer than that.

A CMSampleBuffer is a container for a media sample. For video, it carries the sample and associated information such as timing and format metadata.

For Vision, what I really needed was the image buffer inside it:

guard let pixelBuffer =
    CMSampleBufferGetImageBuffer(sampleBuffer)
else {
    return
}

That gives me a CVPixelBuffer. The pipeline had now become:

Camera
  ↓
AVCaptureVideoDataOutput
  ↓
CMSampleBuffer
  ↓
CVPixelBuffer
  ↓
Vision

There is no need to turn every frame into a UIImage. The pixel buffer can go directly into Vision.

Asking Vision to find a hand

Apple's Vision framework provides VNDetectHumanHandPoseRequest.

Instead of recreating the request for every frame, I keep one instance around and ask it to detect at most one hand.

private let handPoseRequest: VNDetectHumanHandPoseRequest = {
    let request = VNDetectHumanHandPoseRequest()
    request.maximumHandCount = 1
    return request
}()

For each pixel buffer, I create a request handler:

let handler = VNImageRequestHandler(
    cvPixelBuffer: pixelBuffer,
    orientation: .right,
    options: [:]
)

try handler.perform([handPoseRequest])

Once Vision detects a hand, it returns a VNHumanHandPoseObservation. From that observation, I can request specific landmarks:

let indexTip = try hand.recognizedPoint(.indexTip)
let thumbTip = try hand.recognizedPoint(.thumbTip)

Vision doesn't return screen pixels. The locations are normalized coordinates, which makes them independent of a specific image resolution. So now I had something incredibly useful:

index fingertip → (x, y)
thumb fingertip → (x, y)

That sounds like enough to build gestures. Well it wasn't lol

Vision gives estimates, not truth tho

Each recognized point also has a confidence value. Very quickly, I discovered why.
I printed the index coordinates while moving my finger around. When Vision was confident, I would get values like:

x: 0.578
y: 0.300
confidence: 0.872

x: 0.577
y: 0.311
confidence: 0.867

x: 0.575
y: 0.328
confidence: 0.855

But occasionally the tracking would produce a large jump with very low confidence.

If I interpreted every coordinate as equally trustworthy, a tracking mistake could look like an extremely fast gesture.
My first filter therefore became:

guard indexTip.confidence >= 0.7 else {
    return
}

0.7 isn't a universal Vision constant. It was an experimental starting point based on the data I was observing.

That distinction matters throughout HandKit: most thresholds aren't presented as magic values that should work for everybody. They're values I measured and tuned while experimenting with my own setup.
But confidence filtering wasn't enough.

Even when Vision was confident and my finger looked completely still to me, its coordinates continued to move slightly.

That led to the first genuinely interesting algorithmic problem.

How do you measure movement when "not moving" still moves?

To know whether my finger is moving, I need more than its current position. I need its previous one. For two consecutive measurements:

previous = (x₁, y₁)
current  = (x₂, y₂)

I calculate:

Δ x = x_2 - x_1
Δ y = y_2 - y_1

But I also wanted one value representing how much the point moved regardless of direction.

That's just the hypotenuse of the triangle formed by the two deltas:

d = sqrt((Δ x)^2 + (Δ y)^2)

Swift already gives me exactly that operation:

let movementDistance = hypot(deltaX, deltaY)

This was my first little return to the Pythagorean theorem for a project involving smart lights. The interesting discovery came when I kept my index as still as possible. movementDistance wasn't zero.

I saw values around:

0.001
0.003
0.002
0.005
0.001
...

and occasional larger spikes. The system needed to distinguish actual intentional movement from natural tracking jitter.

A single frame wasn't enough to make that decision.

Filtering the tracking signal

Instead of considering one movement measurement at a time, I started keeping a sliding window containing the five most recent distances.

[d1, d2, d3, d4, d5]

When a sixth arrives:

[d1, d2, d3, d4, d5, d6]

I remove the oldest:

[d2, d3, d4, d5, d6]

In Swift:

recentMovementDistances.append(movementDistance)

if recentMovementDistances.count > 5 {
    recentMovementDistances.removeFirst()
}

Then I calculate the median. For five sorted values:

[0.001, 0.002, 0.003, 0.005, 0.050]
                ↑
              median

That was useful because an isolated bad measurement such as 0.050 doesn't drag the result upwards as strongly as it would affect an average.

This is deliberately a very small and simple filter. HandKit isn't trying to be a sophisticated signal-processing library.

But it was enough to introduce an important idea: a noisy measurement becomes much more useful when you reason about it over time instead of treating every sample independently.

And that observation led directly to state machines.

Detecting movement with hysteresis

My first instinct could have been:

movement > threshold → moving
movement < threshold → idle

But imagine the signal oscillating around that threshold:

The state could rapidly become:

IDLE
MOVING
IDLE
MOVING
IDLE

Instead, I introduced two different thresholds. From my experiments, I started with:

median > 0.01  → start moving
median < 0.005 → become idle

Between them, I keep the current state.

                 > 0.01
        ┌───────────────────────┐
        │                       ↓
      IDLE                   MOVING
        ↑                       │
        └───────────────────────┘
                 < 0.005

This is hysteresis. The gap between the two thresholds prevents the state from constantly flipping when the signal sits near a boundary.
In code, I represent the state explicitly:

enum MovementState {
    case idle
    case moving
}

That small choice became useful because I wasn't really trying to answer:

Is there movement in this exact frame?

I was answering:

Has an intentional movement started, and are we currently inside it?

That is a temporal question.

A gesture is something that happens over time

Originally, I started thinking about recognizing a swipe. A single position obviously can't represent a swipe.Even two consecutive positions aren't enough. A gesture has a beginning, a trajectory, and an end.

So when transitioning from .idle to .moving, I store:

startPosition
startTime

using a ContinuousClock for elapsed time measurement.
When movement eventually ends, I can compare:

startPosition → endPosition
startTime     → endTime

and derive:

totalDeltaX = endX - startX
totalDeltaY = endY - start

plus the gesture duration.

This gave me the information required to reason about a swipe:

Did it travel far enough horizontally?
Did it remain reasonably horizontal?
Did it happen quickly enough?

But testing this exposed a completely different bug.

The horizontal gesture that somehow moved vertically

I performed horizontal movements and looked at my data.

Instead of getting:

|Δx| >> |Δy|

I repeatedly got large changes on Y. Then I performed an intentionally vertical gesture. This time X changed dramatically. My coordinate system was effectively rotated relative to what I was seeing.
The culprit was this:

orientation: .up

I had initially told Vision to interpret the pixel buffer as if it were already oriented upright.
The preview looked fine, which made this easy to miss.
But the orientation of what the user sees and the orientation in which Vision interprets the raw image buffer aren't necessarily the same thing.
I tested .right. Then repeated the horizontal gestures. This time I got measurements such as:

Δx = -0.217
Δy = +0.013

Δx = +0.263
Δy = -0.040

Δx = -0.278
Δy = +0.018

Exactly what I wanted:

|Δx| >> |Δy|

I also explicitly tested direction with the front-facing camera.
In my current configuration:

Δx > 0 → visual movement towards the left
Δx < 0 → visual movement towards the right

Again, I preferred measuring the behavior rather than assuming what mirroring would do.

Realizing I didn't actually want a swipe (yes, kinda late to think that)

While designing the swipe, I realized something else.
For brightness control, a swipe is the wrong interaction.
A swipe is a discrete gesture:

gesture
   ↓
recognized
   ↓
perform action once

That's great for something like swipe → next track
Brightness is continuous.
What I actually wanted was a gesture-controlled slider:

finger starts moving
        ↓
keep tracking horizontal position
        ↓
brightness changes continuously
        ↓
finger stops

So the same .idle / .moving state machine remained useful, but the meaning of the moving state changed.
Instead of waiting for the gesture to end before doing something, I calculate brightness continuously while movement is active.
There's another subtle decision here.

I could update brightness using each frame's delta:

brightness += currentDelta

But then tracking errors can accumulate over time.
Instead, at the beginning of the gesture I store:

startPosition
startBrightness

Every subsequent value is calculated relative to those fixed references:

Δ x = currentX - startX
brightness = startBrightness + f(Δ x)

This means that if I move away and then return exactly to my starting position, I should return to the original brightness regardless of the intermediate path.
That avoids cumulative drift.

The current prototype maps the normalized horizontal displacement to a value between 0 and 100 and clamps the result:

let newBrightness = startBrightness + brightnessDelta
brightness = min(max(newBrightness, 0), 100)

Detecting a pinch with geometry

The second gesture was simpler conceptually. I already had the index fingertip. Vision can also give me the thumb tip.

let indexTip = try hand.recognizedPoint(.indexTip)
let thumbTip = try hand.recognizedPoint(.thumbTip)

After applying the same confidence filtering, I have two normalized points.
So detecting how close the fingers are is again a geometry problem:

Δ x = indexX - thumbX
Δ y = indexY - thumbY
distance = sqrt((Δ x)^2 + (Δ y)^2)

or:

let distance = hypot(deltaX, deltaY)

The confidence check needs to cover both points. A reliable index position doesn't help if the thumb position is uncertain:

guard indexTip.confidence >= 0.7,
      thumbTip.confidence >= 0.7 else {
    return
}

let deltaX = indexTip.location.x - thumbTip.location.x
let deltaY = indexTip.location.y - thumbTip.location.y
let distance = hypot(deltaX, deltaY)

This time, the distance compares two landmarks in the same frame. Earlier, I used the same calculation to compare one landmark across consecutive frames. The geometry is identical; the question is different.

Before deciding what counted as a pinch, I measured the distance while opening and closing my fingers. In my setup, the observations were roughly:

Fingers apart:     0.15 to 0.21
Fingers touching:  0.005 to 0.015

Contact didn't produce a perfect zero. Vision estimates the locations of landmarks in a two-dimensional image, and those estimates still contain noise when the fingers touch.

These are distances in normalized image coordinates, not centimeters or measurements of physical contact. Normalizing the coordinates removes the dependence on pixel resolution; it doesn't make the measurement independent of hand size, camera distance, or image geometry. Moving my hand farther from the camera changes its apparent size and therefore the distance between the detected points.

The measurements were enough to establish a useful threshold for this prototype, but they weren't a general calibration for every hand and camera setup.

A pinch needs memory too

The movement detector had already taught me what happens when a noisy measurement sits near one threshold.
For the pinch, I reused hysteresis with a separate state machine:

enum PinchState {
    case open
    case pinched
}

The rules became:

OPEN    -- distance < 0.015 --> PINCHED
PINCHED -- distance > 0.03  --> OPEN

Between the two thresholds, I preserve the current state. At either exact boundary, the strict comparisons also leave the state unchanged.

That gap matters. After recognizing a pinch, I don't want a small increase from 0.014 to 0.016 to mean that the hand has opened again. The fingers have to separate more clearly before another pinch can begin.

But there was one more distinction to make: the state and the event are different things.
If I called the toggle action on every frame where the distance was small enough, holding my fingers together would repeatedly switch the lamp on and off.
The event I care about is the transition:
OPEN -> PINCHED -> emit one toggle

The core logic can be expressed as:

switch pinchState {
case .open:
    if distance < 0.015 {
        pinchState = .pinched
        onToggle()
    }
case .pinched:
    if distance > 0.03 {
        pinchState = .open
    }
}

Once the state becomes .pinched, subsequent frames stay in that branch until the fingers reopen. Releasing the pinch rearms the interaction; it doesn't toggle the light.

This gave me a specific behavior to test: pinch, hold, release, pinch again. Holding should produce no additional action, and the second pinch should produce exactly one new action.

Hysteresis stabilizes the transition, but it doesn't solve every tracking problem. For example, losing the hand while it is pinched raises a separate question about when to reset or rearm the recognizer. That is part of making the experiment more robust, beyond this first working interaction.

Finding the actual light with HomeKit

At this point, the gesture pipeline could produce a toggle event. I still needed to connect that event to something outside the phone.
HomeKit introduced another hierarchy to understand:

HMHomeManager
  |
  +-- HMHome
        |
        +-- HMAccessory
              |
              +-- HMService (Lightbulb)
                    |
                    +-- HMCharacteristic (PowerState)
                    +-- HMCharacteristic (Brightness)

HMHomeManager provides access to the configured homes. Each home contains accessories, and each accessory exposes services. The values I can inspect or change belong to the characteristics of those services.

Apple's HMCharacteristic documentation describes that final layer.

I began by inspecting what my own home actually exposed.
Within the service, the two characteristic types I needed were:

HMCharacteristicTypePowerState
HMCharacteristicTypeBrightness

PowerState represents whether the light is on. Brightness represents its brightness as an integer percentage of the maximum, as defined by Apple's brightness characteristic documentation.

Finding those characteristics didn't yet prove that I could control the lamp. I first asked HomeKit to read their values.
The actual output was:

BRIGHTNESS: 100
POWER: 1

Now I knew that I had found the intended accessory and could read its state: the lamp was on, with brightness at 100%.
Testing one layer at a time

It was tempting to connect the pinch immediately. Instead, I worked through four small steps:
Read state -> Manual write -> Temporary button -> Gesture

The first read established that discovery worked.

Giving HomeKit its own controller

I put HomeKit discovery and control in LightController.

Its job was to find Lamp, retain the characteristics the app needed, and expose an operation such as toggle() to the rest of the application.

The references begin as optional values because HomeKit discovery is asynchronous:

private var powerCharacteristic: HMCharacteristic?
private var brightnessCharacteristic: HMCharacteristic?

They refer to live HomeKit objects. They aren't preferences to persist in UserDefaults, and keeping them avoids traversing the entire home hierarchy on every gesture.
Once discovery had found the power characteristic, toggle() followed this sequence:

Power characteristic available?
          |
          v
Read current value
          |
          v
Convert to Bool and invert
          |
          v
Write new value
          |
          v
Check completion for an error

I read the value before toggling because the lamp could also be changed from the Home app or another controller. A local Boolean would only describe what HandKit last believed, which might no longer match the accessory.

There was also an Objective-C interoperability detail: HMCharacteristic.value is exposed as Any?. In my tests, the power value arrived as an NSNumber, so I used its boolValue before inverting it:

guard let value = powerCharacteristic.value as? NSNumber else {
    return
}

let newState = !value.boolValue

A read followed by a write still isn't an atomic toggle. Another command could arrive between them, and rapid gestures could overlap requests. Serializing device commands is a further robustness step; the working demo doesn't establish that all concurrent-control cases are handled.

At this snapshot, retaining the brightness characteristic prepared the second interaction. It did not mean that the gesture-controlled brightness value was already being written to the lamp.

Keeping recognition independent from the action

The frame-processing code knows about landmarks, distances, and gesture states. It doesn't need to know what a HomeKit accessory is.

VideoOutputDelegate communicates through two callbacks:

private let onToggle: @Sendable () -> Void
private let onBrightnessChanged: @Sendable (CGFloat) -> Void

One emits a discrete event. The other emits a continuously updated value.
CameraManager receives the toggle callback from the application and passes it to the delegate. The application decides that this event should call LightController.toggle().

That boundary means I can inspect or log the gesture output without a connected lamp. It also means the same pinch recognizer could eventually trigger something else without changing the geometry or thresholds.

There is still a deliberate prototype-level coupling here: onBrightnessChanged names the value in terms of the current interaction. It abstracts away HomeKit, but it isn't a completely generic gesture API. If I extracted this into a reusable component, a normalized slider value or displacement event might be a better contract.

For HandKit, the useful separation was already clear:

Recognition: What did the hand do?
Application: What should that mean?
HomeKit: How do I ask the lamp to do it?

Crossing from the frame queue to the `MainActor`

These callbacks also exposed a concurrency boundary. Frames arrive on frameQueue. The UI-facing model and, in this project, LightController are isolated to the MainActor.

Calling lightController.toggle() directly from the synchronous toggle callback produced an isolation error.
The callback needed an explicit handoff:

onToggle: {
    Task { @MainActor in
        lightController.toggle()
    }
}

The brightness callback uses the same pattern:

onBrightnessChanged: { newBrightness in
    Task { @MainActor in
        viewModel.brightness = newBrightness
    }
}

@Sendable describes a closure's ability to cross concurrency boundaries with safe captures. It doesn't dispatch the closure onto the main actor. The Task { @MainActor in ... } body establishes that execution context explicitly.
The distinction follows Swift's concurrency and isolation model.

The values crossing this boundary are small: an event or a brightness number. The pixel buffers, Vision request, and recognition state remain in the frame-processing path.

Moving the callback onto the main actor also doesn't make the physical device operation synchronous. HomeKit still completes its read and write asynchronously. Actor isolation protects access to application state; it doesn't guarantee that the lamp has already reacted.

For this prototype, the callback handoff made the two paths explicit. If updates become more frequent than the UI or accessory can use, the next step is to coalesce them rather than create an ever-growing stream of pending work. Independent tasks also shouldn't be used as a guarantee of event ordering.

One shared model for the camera and the interface

The on-screen brightness indicator reads from a CameraViewModel that is both observable and main-actor isolated:

@MainActor @Observable
final class CameraViewModel {
    var brightness: CGFloat = 0
}

This reduced example shows the two separate responsibilities.
@Observablelets SwiftUI track changes to the state it reads. @MainActor defines where that mutable state is accessed. Observation alone doesn't establish concurrency isolation.

There was an equally important ownership detail: ContentView and CameraManager needed the same model instance.

If each created its own CameraViewModel, the camera callback could update one object while SwiftUI observed another. The gesture calculations would be correct, but the indicator would never reflect them.

In the view's initialization, the wiring therefore follows this shape:

init() {
    let viewModel = CameraViewModel()
    let lightController = LightController()

    self._viewModel = State(initialValue: viewModel)
    self.lightController = lightController

    self.cameraManager = CameraManager(
        viewModel: viewModel,
        onToggle: {
            Task { @MainActor in
                lightController.toggle()
            }
        }
    )
}

This is an excerpt of the wiring, with the surrounding stored properties and view body omitted. The important part is that the local viewModel passed to CameraManager is the same reference retained for the view's state. Inside the manager, the brightness callback updates that reference on the main actor.

The resulting feedback path is:

Vision -> slider value -> callback -> MainActor
                                      |
                                      v
                              CameraViewModel
                                      |
                                      v
                              SwiftUI indicator

That shared instance is what connects a correct calculation to something visible on screen.

The complete architecture

By the end of this iteration, the system looked like this:

                          ContentView
                  creates and connects objects
                              |
                              v
                     CameraManager actor
                              |
                    custom SerialExecutor
                              |
                       sessionQueue
                              |
                      AVCaptureSession
                              |
             +----------------+------------------+
             |                                   |
             v                                   v
  AVCaptureVideoPreviewLayer          AVCaptureVideoDataOutput
             |                                   |
             v                                   v
       Camera preview                        frameQueue
                                                 |
                                                 v
                                        VideoOutputDelegate
                                                 |
                                                 v
                                  CMSampleBuffer -> CVPixelBuffer
                                                 |
                                                 v
                                    Vision hand-pose request
                                                 |
                                                 v
                                     Confidence-filtered points
                                                 |
                              +------------------+----------------+
                              |                                   |
                              v                                   v
                     Movement / slider                      Pinch state
                              |                                   |
                              v                                   v
                   onBrightnessChanged(value)                 onToggle()
                              |                                   |
                              v                                   v
                   Task { @MainActor in ... }       Task { @MainActor in ... }
                              |                                   |
                              v                                   v
                     CameraViewModel                       LightController
                 shared with ContentView                          |
                              |                                   v
                              v                           HomeKit PowerState
                    SwiftUI 0-100 indicator                        |
                                                                  v
                                                            Physical Lamp

Next connection, not yet implemented at this snapshot:
slider value -> HomeKit Brightness -> physical light intensity

The boundaries each answer a practical question. The camera manager owns capture lifecycle. The delegate processes frames and maintains recognition state on its serial queue. The view model exposes UI state. The light controller handles accessory discovery and commands.

The architecture also makes the unfinished part visible: the brightness path reaches the interface, while the toggle path reaches the physical lamp.

Where the experiment stands

The pinch now controls a real HomeKit lamp. Bringing my thumb and index finger together triggers one toggle, and holding the pinch doesn't repeatedly switch it.

The continuous horizontal gesture also works as an on-screen slider. It produces and displays brightness values from 0 to 100 using a fixed starting position and starting brightness.

At the snapshot covered by this article, that slider was not yet connected to HomeKit's Brightness characteristic. I had discovered and read the characteristic, but the physical brightness-control path was still unfinished.

Those are two different milestones: recognizing and displaying the intended value, and successfully applying it to an accessory.

The video captures the part that a console log couldn't. After spending so much time looking at coordinates and state transitions, I brought two fingers together and something in the room changed.

I laughed. A lot.

What I learned

This project made several concepts concrete for me.
Actor isolation and execution are separate concerns. Making CameraManager an actor gave its mutable state an isolation boundary; choosing a serial executor gave its work a deliberate execution context. Neither choice removed the need to think about delegate callbacks and other queues.

Confidence filtering, median filtering, and hysteresis solve different problems. Confidence rejects uncertain landmarks. The five-sample median limits the influence of isolated movement spikes. Hysteresis prevents repeated state changes near a boundary. Combining them was useful because no single threshold could do all three jobs.

Filtering also has a cost. A window of recent measurements adds inertia, and per-frame displacement depends on sampling cadence. The numbers that worked in my tests aren't a substitute for testing different frame rates, camera distances, and lighting conditions.

Gesture recognition requires memory. The movement path needs a beginning and an evolving trajectory; the pinch path needs to know whether the fingers were already together. Once I modeled those states explicitly, the behavior became easier to reason about.

The transition can be the useful event. A sustained pinch describes a condition. Entering that condition describes an action the application can respond to once.

A fixed reference simplifies continuous control. Mapping every position back to startPosition and startBrightness avoids accumulating frame-to-frame errors and gives the interaction a predictable return point.

Object identity matters as much as data flow. The brightness callback and SwiftUI needed to refer to the same model instance. Correct values sent to an unobserved object are still invisible to the user.

Finally, separating the layers made debugging manageable. A button let me validate HomeKit without a camera gesture. A visual indicator let me validate the brightness calculation without changing a lamp. The callbacks connected those tested pieces.

Learning with AI as a mentor

I used AI, including Codex, throughout the experiment as a mentor: to point me toward documentation, explain unfamiliar concepts, and challenge my reasoning when I couldn't explain what I was seeing.

The learning rule was to work through the challenges myself. I wanted to inspect the measurements, reason about the algorithm, write and test the behavior, and understand the architectural decisions. Explanations and small examples helped me move forward, but having the challenges completed for me would have removed the part I was doing this project to experience.

The most useful moments were the questions that made me look again: what is actually inside this buffer? Why does a stationary finger still move? Which coordinate system am I measuring? What should happen on the next frame if the fingers are still touching?
That is how a small experiment with a lamp became an excuse to understand much more of the iOS stack.

From a small question to a working lamp

I started HandKit because I wanted to know whether I could control my lights with my fingers. I ended this iteration with a working pinch, a brightness interaction I could see on screen, and a much clearer understanding of everything between a camera frame and a physical action.

The satisfying part was being able to explain that path. I could point to where the pixels arrived, where the landmarks became measurements, where a state transition became an event, and where that event finally reached HomeKit.

There is still work left. The brightness path needs its final connection, and the gesture logic needs to survive more than the conditions in which I first tuned it.

But the moment in that first successful video already says why I wanted to build this. I pinched my fingers, the lamp reacted, and I couldn't stop laughing. A question that had sounded a little ridiculous had led me through unfamiliar frameworks, a return to Pythagoras, and a room that now responded to something I had built.
That was a very good reason to spend two days on it.

Stay hungry, stay foolish!