【visionOS/ARKit】Simplest sample code for hand tracking

#visionos #applevisionpro #arkit #visionpro

The basis of ARKit's hand tracking API in visionOS is to obtain data such as coordinates and rotation of each joint of the hand.

Here is a sample code of ARKit's hand tracking API in visionOS that works with only 41 lines of code.

Basic Knowledge

First, watch this session video. It gives a good overview of ARKit and hand tracking in visionOS.

Meet ARKit for spatial computing - WWDC23 - Videos - Apple Developer

Hand tracking is explained from 15:05.

Two APIs

There are two APIs to get joint data. Both can be accessed from the HandTrackingProvider instance.

anchorUpdates: receives the latest values via AsyncSequence.
latestAnchors: contains the latest values.

In this case, I implemented it with anchorUpdates.

Overview of the Sample Code

Starts hand tracking when the app launches.
Places simple sphere objects at all joints of both hands.
Updates each object’s position to match the latest joint position.
Ensures the objects are not hidden by the hand.

Full Source Code

import SwiftUI
import RealityKit
import ARKit

@main
struct MyApp: App {
    private let session = ARKitSession()
    private let provider = HandTrackingProvider()
    private let rootEntity = Entity()

    var body: some SwiftUI.Scene {
        ImmersiveSpace {
            RealityView { content in
                content.add(rootEntity)
                for chirality in [HandAnchor.Chirality.left, .right] {
                    for jointName in HandSkeleton.JointName.allCases {
                        let jointEntity = ModelEntity(mesh: .generateSphere(radius: 0.006),
                                                      materials: [SimpleMaterial()])
                        jointEntity.name = "\(jointName)\(chirality)"
                        rootEntity.addChild(jointEntity)
                    }
                }
            }
            .task { try! await session.run([provider]) }
            .task {
                for await update in provider.anchorUpdates {
                    let handAnchor = update.anchor
                    for jointName in HandSkeleton.JointName.allCases {
                        guard let joint = handAnchor.handSkeleton?.joint(jointName),
                              let jointEntity = rootEntity.findEntity(named: "\(jointName)\(handAnchor.chirality)") else {
                            continue
                        }
                        jointEntity.setTransformMatrix(handAnchor.originFromAnchorTransform * joint.anchorFromJointTransform,
                                                       relativeTo: nil)
                    }
                }
            }
        }
        .upperLimbVisibility(.hidden)
    }
}

Copy and paste this code to use it.

Additional Steps

Set any text for “NSHandsTrackingUsageDescription” in Info.plist.
Set “Preferred Default Scene Session Role” to “Immersive Space” in Info.plist.

Comments

The explanations in the session video and the basic knowledge of SwiftUI and RealityKit are omitted.

Managing Each Entity

jointEntity.name = "\(jointName)\(chirality)"

let jointEntity = rootEntity.findEntity(named: "\(jointName)\(handAnchor.chirality)")

Each entity is managed by its name using HandSkeleton.JointName for joint names and HandAnchor.Chirality for left or right hand.

Setting Access Permissions

To request access permissions, you need to set any text for “NSHandsTrackingUsageDescription” in Info.plist.

This key does not appear in the pull-down menu, so enter it directly.

Launch the App in Full Space

@main
struct MyApp: App {
    ...
    var body: some SwiftUI.Scene {
        ImmersiveSpace {
            ...
        }
        ...
    }
}

When you create a new visionOS app project in Xcode, it generates code to launch in a window. For simplicity, I made it launch in full space.

Set “Preferred Default Scene Session Role” to “Immersive Space” in Info.plist. If you do not make this setting, the app will crash right after launch.

Note: A Physical Device is Required

You need a physical device to test the ARKit Hand Tracking API. It does not work at all in the simulator.

Next Step

Check the current authorization status: session.queryAuthorization(for:)
Explicitly request authorization: session.requestAuthorization(for:)
Check the state of anchors: AnchorUpdate.Event
Check if each anchor or joint is being tracked: TrackableAnchor.isTracked
Observe the session state: ARKitSession.Events
Check if the current runtime environment supports it: HandTrackingProvider.isSupported