STIKSHOT - A STICKMAN VIDEO RENDERER TOWARDS BEYOND

Build a 100% Client-Side AI Motion Capture Engine in the Browser
Imagine uploading a dance video and having it instantly turned into a neon stickman animation—all without uploading a single megabyte to a server.

That is stikshot, a local-first, serverless AI motion-capture web app. In this article, we'll dive into the architecture, the browser APIs we used, and the workarounds we engineered to make WebCodecs run stably on desktop and mobile.

The Core Stack
To keep stikshot completely serverless, static, and scale-free, we built the entire pipeline to run inside the user's browser sandbox:

AI Engine: TensorFlow.js running the MoveNet Multipose model.
Background Thread: Web Workers to keep the UI responsive.
Compression: The browser's native WebCodecs API (VideoEncoder and AudioEncoder).
Muxer: webm-muxer to compile the video/audio streams into a playable WebM file.
Architecture: Offscreen Pipelining
To prevent the browser from lagging, we split the application into two parts: a Main Thread Coordinator and a Web Worker Processor.

Mermaid diagram
Code Highlights

1. Seeking Background frames To avoid tab-throttling when users switch tabs, we avoided requestVideoFrameCallback and fell back to standard seeked media event listeners combined with a safe timeout fallback:

javascript

function seekTo(video, time) {
return new Promise((resolve) => {
const onSeeked = () => {
video.removeEventListener('seeked', onSeeked);
clearTimeout(fallback);
resolve();
};
const fallback = setTimeout(() => {
video.removeEventListener('seeked', onSeeked);
resolve();
}, 500); // 500ms timeout safety net
video.addEventListener('seeked', onSeeked);
video.currentTime = time;
});
}

1. Safeguarding iOS Safari WebCodecs Crashes During development, we encountered a severe bug: Safari would crash/freeze when trying to write encoded video chunks.

We discovered that Safari freezes the browser-provided metadata objects inside the VideoEncoder output callback. Modifying or passing them directly to the muxer throws a quiet error. We solved this by copying properties to an unfrozen, clean object:

javascript

const videoEncoder = new VideoEncoder({
output: (chunk, meta) => {
let cleanMeta = undefined;
if (meta) {
cleanMeta = {};
// Copy properties to bypass read-only/frozen constraints in Safari
if (meta.decoderConfig !== null && meta.decoderConfig !== undefined) {
cleanMeta.decoderConfig = meta.decoderConfig;
}
if (meta.svc !== undefined) cleanMeta.svc = meta.svc;
if (meta.alphaSideData !== undefined) cleanMeta.alphaSideData = meta.alphaSideData;
}
state.muxer.addVideoChunk(chunk, cleanMeta);
},
error: (err) => console.error('Encoder error:', err)
});

1. Avoiding Android Audio Hacks Android Chrome sometimes hangs indefinitely inside decodeAudioData on corrupted tracks. We implemented a promise-wrapped audio decoder with an 8-second safety timeout:

javascript

function decodeAudioWithTimeout(buffer, sampleRate = 48000, timeoutMs = 8000) {
const decodeCtx = new OfflineAudioContext(1, 1, sampleRate);
return new Promise((resolve) => {
let completed = false;
const timer = setTimeout(() => {
if (!completed) {
completed = true;
console.warn('Audio decoding timed out');
resolve(null);
}
}, timeoutMs);
decodeCtx.decodeAudioData(buffer,
(decoded) => {
if (!completed) { completed = true; clearTimeout(timer); resolve(decoded); }
},
(err) => {
if (!completed) { completed = true; clearTimeout(timer); resolve(null); }
}
);
});
}
What’s Next: Beyond the Stickman
stikshot is built to be serverless, meaning we don't pay for GPU rendering. However, generating 2D stickman animations is just step one.

We are looking to expand this local-first pipeline to support:

3D Skeletal Rigging: Converting joint coordinates to .gltf / .fbx formats.
Game Engine Exports: Allowing creators to export motion capture data directly to Unity or Unreal Engine.
VTuber Overlays: Creating browser-based camera inputs for real-time virtual avatar tracking.
Let's Collaborate!
If you are interested in local-first AI, WebAssembly, WebGL shaders, or browser motion tracking, check out the live app, leave a star, and get in touch:

App: stikshot.com
Email: [e-mail]
Instagram: @stikshotu