If you are building a fitness, wellness or rehab app, sooner or later you want the phone to see the user move: count a squat, check a posture, measure a range of motion. The first step is always the same. You need the position of the body's joints, on every camera frame, without sending video to a server.
This tutorial gets you there in a fresh Expo project: 17 body keypoints per frame, computed on the device, with no API key and no backend. I am the founder of PoseTracker, the SDK used here, so take that into account; the code below is the same as in our public docs.
What you will build
A single screen that opens the camera, draws a skeleton over the person, and logs the keypoints in JavaScript. The pose model is MoveNet, running on the phone inside a WebView. Nothing is uploaded.
1. Install
npm install @pose-tracker/react-native-pose-estimation react-native-webview
npx expo install react-native-webview expo-camera
expo-camera is only there for the permission API. The camera itself is captured by the SDK's view.
There are two packages with the same API. The one above bundles the model, so keypoints work offline (about 10 MB packed). @pose-tracker/react-native-pose-estimation-light is about 212 kB and downloads the model when the view starts, so it needs a network connection. Pick offline if your users train in a basement gym, light if install size matters more.
2. Declare the camera permission
In app.json:
{
"expo": {
"ios": {
"infoPlist": {
"NSCameraUsageDescription": "Camera access is required for on-device pose estimation."
}
},
"android": {
"permissions": ["android.permission.CAMERA"]
},
"plugins": [
[
"expo-camera",
{
"cameraPermission": "Camera access is required for on-device pose estimation."
}
]
]
}
}
Request the permission in your app before you mount the camera screen.
3. Mount the view and read the keypoints
import {
PoseTrackerProvider,
WebViewPoseView,
usePoseTracker,
} from '@pose-tracker/react-native-pose-estimation';
export default function App() {
return (
<PoseTrackerProvider>
<CameraScreen />
</PoseTrackerProvider>
);
}
function CameraScreen() {
usePoseTracker({
onKeypoints: (e) => {
console.log(e.keypoints.length, e.score);
},
});
return (
<WebViewPoseView
style={{ flex: 1 }}
drawSkeleton
loadingText="AI Loading"
/>
);
}
That is the whole integration. Three pieces do the work:
-
PoseTrackerProvidercreates the client. Without a token it runs in keypoints-only mode. -
WebViewPoseViewloads the model and opens the camera. -
usePoseTracker({ onKeypoints })gives you a typed event on each frame.
What you get on each frame
| Field | Type | Meaning |
|---|---|---|
keypoints |
Keypoint[] |
17 joints, each with a name, x, y and a score |
score |
number |
Mean confidence of the pose |
timestampMs |
number |
Time of the frame |
The 17 joints are the COCO set: nose, eyes, ears, shoulders, elbows, wrists, hips, knees and ankles. With those you can already compute a knee angle, detect that the hips went below the knees, or check that both shoulders are level.
Why a WebView
It is a fair question for an SDK. The model runs with TensorFlow.js on WebGL inside react-native-webview, which is what lets the same code run in Expo Go, on iOS and on Android without a native build step. The trade-off is that you depend on the WebView's GPU path. On the phones we test, that holds a real-time frame rate; measure it on the oldest device you intend to support.
From keypoints to repetitions
Keypoints are free and stay free. Turning them into "that was one clean squat" is a different job: you need to segment the movement, count, and score the form. You can write that yourself from the keypoints, or use PoseTracker's engine, which is the paid part. With an API key:
<PoseTrackerProvider
apiToken="YOUR_API_KEY"
options={{ features: { angles: true, progression: true, minGrade: 'B' } }}
>
<WebViewPoseView drawSkeleton />
</PoseTrackerProvider>
const { preload, startExercise } = usePoseTracker({
onMessage: (msg) => {
if (msg.type === 'counter') {
// current_count and form_score: { score, avg_score, grade }
}
},
});
await preload();
startExercise('squat');
The free plan covers 200 tracking calls a month, which is enough to prototype.
Try it without writing code
If you want to see it on your phone first, open the Expo Snack in Expo Go and scan the QR code. No API key needed.
Where to go next
- SDK quickstart for the full reference
- Choosing between the offline and light packages
- How on-device pose estimation works, and what stays on the phone
One more thing. MoveNet stops at 17 keypoints and at the ankle. We are preparing our own model, PoseTrackerNet, with 39 keypoints, including the feet, the trunk and the neck. It is in beta, with a public release planned by December 1, 2026.
Top comments (0)