DEV Community

Cover image for I built ChibiRigKit: turning a single character image into an animated 2D rig
milkc0de
milkc0de

Posted on Edited on

I built ChibiRigKit: turning a single character image into an animated 2D rig

#javascript #ai #python #opensource

I have been building ChibiRigKit, an open-source tool that turns a single character illustration into an animated 2D rig.

It started as a very small experiment:

Can I make this one illustration move a little more naturally?

It has since grown into a complete pipeline:

Single illustration
        ↓
Semi-automatic rigging
        ↓
Browser player
        ↓
Camera / microphone tracking
        ↓
Motion recording and playback
        ↓
OBS
Enter fullscreen mode Exit fullscreen mode

You can try the browser playground here:

https://milkc0de.github.io/ChibiRigKit/

Source:

https://github.com/milkc0de/ChibiRigKit

This article describes the project as of v1.0.0-beta.7.

What ChibiRigKit does

You can start with a normal PNG illustration.

ChibiRigKit prepares the pieces required for animation, such as:

  • face
  • sclera
  • iris
  • eyelashes
  • hair
  • clothes
  • body
  • reconstructed face background
  • closed eyes
  • closed mouth

The resulting character can be controlled in a browser.

The runtime currently supports:

  • blinking
  • gaze movement
  • yaw, pitch and roll
  • hair and clothing motion
  • simple long-hair physics
  • camera-based face tracking
  • microphone-driven lip sync
  • motion recording and playback
  • custom backgrounds
  • WebM recording
  • local OBS synchronization

The character can also be exported as a self-contained browser player.

The main design decision: AI is not the runtime

One of the most important decisions in ChibiRigKit is that AI is used during authoring, not during playback.

The architecture roughly looks like this:

┌─────────────────────────────┐
│ Authoring                   │
│                             │
│ Codex app-server            │
│ GPT-6 Astra / high          │
│ Python and Node.js tools    │
│ Validation                  │
└──────────────┬──────────────┘
               │
               │ rig data
               ▼
┌─────────────────────────────┐
│ Runtime                     │
│                             │
│ HTML / JavaScript           │
│ Canvas                      │
│ MediaPipe                   │
└─────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Once a character has been built, the AI system is no longer required.

The player is just normal web software.

I like this separation because the finished character does not depend on model latency, API availability, inference cost, or a permanent Internet connection.

AI helps create the artifact.

It is not required to keep the artifact alive.

What the model does, and what normal code does

I also did not want the model to perform every operation itself.

There are two different kinds of work in rigging.

Some problems are semantic:

Is this region hair or skin?
Where does the eye actually end?
Which layer should be drawn in front?
Where should this part rotate from?
Did this mask accidentally include clothing?
Enter fullscreen mode Exit fullscreen mode

These are useful tasks for a multimodal model.

Other operations are deterministic:

image alignment
mask operations
cropping
coordinate transforms
hashing
JSON generation
file validation
browser rendering tests
Enter fullscreen mode Exit fullscreen mode

Those are handled by regular Python and Node.js tools.

So the model decides what needs to be done, while deterministic tooling performs as much of the actual transformation as possible.

Rigging is iterative

The automation is not:

image
  ↓
LLM
  ↓
done
Enter fullscreen mode Exit fullscreen mode

Instead, it behaves more like:

Plan
 ↓
Generate
 ↓
Validate
 ↓
Review
 ↓
Fix
 ↓
Validate again
Enter fullscreen mode Exit fullscreen mode

The system produces comparison images and machine-readable reports.

The model can inspect those results and correct masks or placement.

More importantly, a model saying “finished” is not considered proof that the rig is valid.

Artifacts, input hashes, project structure and browser rendering are checked separately.

Structured contracts instead of relying on prose

Several parts of the authoring process are represented as JSON contracts.

For example:

rig.workflow.json
rig.workflow.schema.json
rig.result.schema.json
work/phase-*.json
Enter fullscreen mode Exit fullscreen mode

Model outputs are constrained with structured output schemas.

A phase can explicitly finish as something such as:

ready_for_validation
Enter fullscreen mode Exit fullscreen mode

or:

blocked
Enter fullscreen mode Exit fullscreen mode

This makes orchestration much easier than trying to infer application state from free-form model responses.

Starting from only one image

A normal character image is enough to begin.

If required, ChibiRigKit creates reference variations such as:

normal
flat
blink
mouth_closed
Enter fullscreen mode Exit fullscreen mode

flat is the reconstructed face layer behind movable facial features.

The original image is preserved.

Generated references, masks, plans, hashes and manifests are stored separately so that the pipeline can detect when an input or generated artifact has changed.

Hand-drawn references can also be supplied and take priority over generated ones.

Why anime-style segmentation is difficult

Character illustrations are not normal photographs.

A region cannot always be identified by color or edge information alone.

For example, the system may need to understand that:

  • blush belongs to the face, not the eye
  • front hair must remain in front of the face
  • clothing should not leak into a hair mask
  • the shoe on the other side of the character is not part of this leg
  • an accessory may cross several semantic regions

These are exactly the places where semantic reasoning is useful.

A shared runtime for different characters

I wanted character-specific data without character-specific animation code.

The project therefore uses semantic mesh types such as:

blink_eye_radial
bend_vertical
soft_body
soft_strip
face_grid
Enter fullscreen mode Exit fullscreen mode

The actual character structure lives in rig.project.json.

The JavaScript runtime remains shared.

This also means I can improve the runtime and update existing character workspaces without regenerating the entire rig.

Nine-direction head poses

A flat illustration looks very flat if you simply rotate the head image.

ChibiRigKit instead stores nine reference configurations:

up-left       up       up-right

left        center       right

down-left    down     down-right
Enter fullscreen mode Exit fullscreen mode

The runtime interpolates between them.

Roll is then applied on top of the interpolated yaw/pitch result.

The poses can affect more than the face:

  • eyes
  • hair
  • face contour
  • accessories
  • neck connection

This is not a 360-degree 3D reconstruction.

It is a 2D deformation system designed for movement around the front-facing pose.

The nine poses can be edited manually in the browser.

Blinking

The eye system can separate:

sclera
iris
eyelashes
closed-eye artwork
Enter fullscreen mode Exit fullscreen mode

The iris moves inside a clipped sclera region.

During a blink, the open-eye mesh deforms toward the closed-eye line.

The dedicated closed-eye artwork appears only near full closure.

This gives a more continuous result than simply switching between two images.

Camera tracking

The browser player uses MediaPipe locally.

Face tracking runs in a Worker and produces values for things such as:

yaw
pitch
roll
eyes
gaze
mouth
Enter fullscreen mode Exit fullscreen mode

Head, eye and mouth tracking can be enabled or disabled independently.

That means the input sources can be mixed.

For example:

head  → camera
eyes  → camera
mouth → microphone
Enter fullscreen mode Exit fullscreen mode

No Codex connection is required for tracking.

Audio-driven lip sync

Microphone input can directly drive the mouth.

The player includes an input level meter and sensitivity controls.

It can also work with virtual audio devices such as VB-CABLE, allowing audio from another application to drive the character.

The player disables browser echo cancellation, noise suppression and automatic gain control for this input path.

Stereo channels are inspected separately as well, which avoids some phase-cancellation problems seen with virtual audio routing.

ChibiRigMotion

I created a small motion format called ChibiRigMotion.

Instead of recording camera video or microphone audio, it stores control values over time.

Conceptually:

time
yaw
pitch
roll
eyes
gaze
mouth
...
Enter fullscreen mode Exit fullscreen mode

A motion can later be loaded and replayed on the character.

Tracking and motion playback can also be combined.

Because the file contains motion values rather than raw media, the recorded motion stays independent from the original camera and microphone streams.

Local OBS synchronization

This became one of my favorite additions.

An exported character package contains launchers for Windows, macOS and Linux.

The local controller is available at:

http://127.0.0.1:5510/
Enter fullscreen mode Exit fullscreen mode

The OBS browser source uses:

http://127.0.0.1:5510/obs
Enter fullscreen mode Exit fullscreen mode

The interesting part is that ChibiRigKit does not stream rendered video into OBS.

Instead, it shares animation state.

Controller
    │
    │ settings + motion values
    ▼
Local server
    │
    │ Server-Sent Events
    ▼
OBS browser source
    │
    ▼
Render character locally
Enter fullscreen mode Exit fullscreen mode

Both browser pages run the same character runtime.

Only the state needs to move between them.

Why SSE?

The synchronization pattern is primarily:

controller → server → renderer
Enter fullscreen mode Exit fullscreen mode

The OBS renderer does not need a large bidirectional RPC protocol.

Server-Sent Events were therefore sufficient and kept the implementation small.

The server binds only to:

127.0.0.1
Enter fullscreen mode Exit fullscreen mode

It is not exposed to the LAN.

Raw camera and microphone data are not sent to OBS

For camera tracking, the flow is:

Camera
  ↓
Browser
  ↓
MediaPipe Worker
  ↓
Motion values
  ↓
Local SSE
  ↓
OBS
Enter fullscreen mode Exit fullscreen mode

The OBS synchronization server receives processed values, not the raw camera video.

The same principle applies to microphone input.

The exported character does not persist device IDs or device names either.

ChibiRigKit also does not start OBS recording or streaming by itself.

Offline playback

Once the required files are installed, playback does not require Internet access.

The finished character can:

  • animate
  • load motions
  • save WebM
  • use the local player
  • synchronize with OBS

offline.

If the MediaPipe SDK, WASM and face model have already been prepared, camera and microphone tracking can also run offline.

Internet access is mainly needed during initial setup and AI-assisted rig creation.

Exporting a character

A character can be exported with:

npm run export -- --character characters/<name>
Enter fullscreen mode Exit fullscreen mode

This creates:

dist/<name>/
dist/<name>.zip
Enter fullscreen mode Exit fullscreen mode

The generated HTML contains the character artwork and configuration.

The ZIP also contains the small local server, platform launchers, tracking runtime and required license notices.

Authoring files such as masks, model work files and intermediate validation data stay in the development workspace.

Quick start

Setup:

sh SETUP.command
Enter fullscreen mode Exit fullscreen mode

Create a rig:

sh AUTO_RIG.command "/path/to/character.png"
Enter fullscreen mode Exit fullscreen mode

Open a character in the player:

npm run player:setup
npm run player -- --character characters/<name>
Enter fullscreen mode Exit fullscreen mode

Export:

npm run export -- --character characters/<name>
Enter fullscreen mode Exit fullscreen mode

Licensing

The current ChibiRigKit source code is released under the Apache License 2.0.

Versions that were previously released under the MIT License remain available under those original MIT terms.

Media is treated separately from source code.

Character artwork, screenshots, videos, logos and similar assets are not automatically covered by the source-code license.

Third-party components such as MediaPipe retain their own license and notice files, including inside exported packages where required.

What I learned from building it

The most interesting part of this project has not really been “using AI to animate an image.”

It has been deciding where AI should stop.

The architecture I ended up with is roughly:

AI
  → semantic decisions

Python / Node.js
  → deterministic transformations and validation

JSON
  → contracts between the two

Browser
  → runtime

Local SSE
  → OBS synchronization
Enter fullscreen mode Exit fullscreen mode

That boundary gives me the benefits of multimodal reasoning during authoring without making the final character dependent on an AI service.

ChibiRigKit is still in beta and I keep changing it as I use it, but it has already grown far beyond the tiny experiment it started as.

One image was enough to begin.

Now the same character can blink, look around, move its head, react to a camera and microphone, replay recorded motions, and appear directly inside OBS.

And the finished character still runs as normal browser software. 🎀

Top comments (0)