How I Built an Interactive 3D Short Film You Can Talk To

Jashwanth Palleti — Thu, 15 Oct 2026 05:21:14 +0000

I’ll Call You Later is an interactive 3D short film that runs in the browser, with no install. It takes about four minutes to watch. At the end, you can talk to the character live with your voice.

I’m Jas (Jashwanth Palleti). I wrote, directed, performed in, and scored the film, including its English and Telugu music. This is how I built it—and what didn’t work as planned.

The story, and how I approached the build

The film follows a man driving home from work around 9 PM, stuck in traffic in a US tech city. His mom calls. He says, “I’ll call you later.” Then he makes a wish under a set of aligned stars. This is Part 1, and it ends with “to be continued.”

I want to be clear about how the code got written: Claude Code, an AI coding assistant, wrote and ran most of it. My role was technical direction. I decided what to build, what counted as success, and what was off-limits, then directed, debugged, and shipped it.

That distinction matters to me. There’s a lot of technical work in the finished project, but it wasn’t all written by hand. The assistant did most of the coding; I set the constraints and made the calls about what to keep, fix, or abandon.

From phone video to character motion

For my own motion, the process started with phone video. I ran YOLOv8 person detection and a simple IoU tracker, then extracted RTMW’s 133 whole-body keypoints using ONNX Runtime on the GPU.

Those 2D keypoints went into an SMPLify-X-style SMPL-X fitter in PyTorch. It uses a VPoser pose prior, a robust 2D loss, and temporal smoothness. I also recorded a calibration take to fix the body shape. To fit the process into an 8 GB GPU, I split the fitting into chunks.

After fitting, a grounding stage estimates the floor plane from planted feet and uses heel and toe contacts. I then retargeted the result onto the character rig in Blender.

The walk take fitted with a mean reprojection error of 12.9 px. Grounding reduced foot slide from 4.96 to 2.09 mm/frame. Those numbers are useful, but they don’t tell the whole story: at one point the avatar read as tired when I was actually surprised, even though the metrics looked fine.

And one take simply failed. The getting-out-of-the-car motion came out at 302 px error, so I used a Mixamo clip for that shot instead. Library mocap carries most of the full-body motion; my own capture drives the walk style. Knowing when to use an existing licensed asset was better than hand-fixing a take that wasn’t working.

Making the film run in a browser

The film is built with React, React Three Fiber, drei, Three.js, and Vite. One clock drives the experience, which keeps it safe to scrub without the animation and other timed elements getting out of sync.

The visual pipeline combines Blender-baked lighting—night and day bakes blended by sun height—with real sun and moon positions and a 2,600-star sky. On top of that are a custom GLSL colour grade, bloom, vignette, grain, and AgX tone mapping. The crowd is instanced: 190 people on High, 40 on Low.

I added adaptive quality tiers that drop if the frame rate falls below 24 fps. Shader and texture warm-up helped remove stutter. The release cut went from 176 MB to about 95 MB to fit Vercel’s limit, using Draco and WebP, along with phone-sized assets.

Audio uses Web Audio buses with ducking, and the character’s animation uses Rhubarb visemes for lip sync. The site is hosted on Vercel.

One small detail made a big difference during testing: warming up shaders and textures meant the first time they were needed wasn’t also the first time the browser had to prepare them. The project still has limits, but getting rid of that stutter made the experience feel much more deliberate.

Adding a voice conversation

The voice agent runs on LiveKit Agents. A Python worker joins a WebRTC room and connects to OpenAI realtime speech-to-speech. In the browser, livekit-client handles the connection and displays live captions.

A Vercel serverless function signs short-lived, 15-minute LiveKit tokens, so API keys never touch the browser. The agent itself is hosted on LiveKit Cloud.

There are multiple personas on the same pipeline. The character stays in the story, avoids spoilers, and is honest that he’s an AI if asked. The making-of guide answers questions about how the film was built. Tool calls let the character “check his phone,” “read the letters,” or “look around.”

The AI voice also gets live lip sync. About 60 times a second, a Web Audio analyser checks loudness and frequency: loudness opens the jaw, while high frequencies produce S, T, and F shapes. This is an approximation, not phoneme-accurate lip sync.

I measure latency from the end of the user’s speech to the first audio. The things I’d watch closely when building something similar are turn detection, interruptions, and cost per minute. A conversation can feel wrong for reasons that aren’t obvious from the graphics, so I’m interested in feedback on the voice experience in particular.

Recording a trailer from the browser

I also wanted a clean HD trailer without relying on a powerful machine to play the film in real time while recording. I added a render mode that replaces the page’s performance.now and requestAnimationFrame clock with a virtual clock, then steps exactly one frame at a time.

Each frame is encoded in the browser with WebCodecs using H.264, then muxed into an MP4 with mp4-muxer.

That feature exposed a bug in the new top quality tier. It had no texture size limit entry, so textures came out wrong and the character rendered black. The cause was one missing key in a lookup table. Not a complicated rendering mystery—just a small piece of configuration that the new tier didn’t have.

What I learned, and what’s unfinished

A few lessons I’m taking forward:

Set acceptance criteria before looking at the results. Metrics can look fine while the performance still reads incorrectly.
Don’t optimize only for numbers. The avatar looking tired instead of surprised was a real failure, regardless of the reprojection error.
Know when to stop and use an existing licensed asset instead of hand-fixing a take.
Read the actual licence file, not the README.

There are limits to what I’ve built. Pipeline stages are run by hand. Telugu lip sync is approximate. The film hasn’t been tested much on real phones; it works best on desktop Chrome with headphones.

I’d like your feedback

If you try it, I’d love to hear what you think about the motion, performance on your device, and the voice conversation. The film is here; you can talk directly to the character here, or ask the making-of guide questions here.

I’m open to roles in real-time 3D, voice AI, and creative tech. More about me: jashtechno.com.

DEV Community: Jashwanth Palleti