Building a 3D AI Desktop Companion with React, Three.js, Node.js and Tauri 2

Most AI assistants follow a familiar pattern: open an application, type a message, receive a response.

I wanted to experiment with something different.

What if an AI assistant had an actual interactive 3D character? What if it could perform gestures, respond through voice, interact with equipment and remember information across conversations?

That idea became Wemmbu AI Assistant, an independently developed desktop AI companion built with React, Three.js, Node.js and Tauri 2.

This article covers the architecture, technology choices, engineering challenges and lessons I've encountered while building it.

1. The idea behind the project

The initial goal was to combine an AI assistant with a Minecraft-inspired 3D character.

I didn't want the character to be just a static avatar displayed beside a chat window. I wanted it to participate in the application.

For example:

• The assistant could wave or explain something using gestures.
• It could speak its responses through text-to-speech.
• Its mouth could animate while audio was playing.
• It could interact with equipment and character actions.
• It could retain useful information through persistent memory.

This meant the project needed more than a chatbot API and a 3D model. The frontend, character animation, backend, voice system and memory layer all needed to communicate reliably.

2. Technology stack

Here are the main technologies behind the application.

Component	Technology	Responsibility
Frontend	React 18	Application interface
Development/build	Vite	Frontend tooling
3D rendering	Three.js	Character and equipment rendering
3D assets	GLB / GLTF	Character and item models
Backend	Node.js	Local application API
Desktop framework	Tauri 2	Native desktop window
Native layer	Rust	Desktop lifecycle and process management
AI integration	OpenRouter	AI conversations
Voice	Fish Audio	Text-to-speech
Persistent memory	SQLite	Local memory storage

I chose Tauri 2 instead of Electron because I wanted a native desktop application with a relatively lightweight desktop runtime.

3. Application architecture

One of the most important design decisions was separating the frontend, backend and desktop runtime.

The application follows this structure:

Windows / Linux / macOS Desktop
│
└── Tauri 2
    │
    ├── WebView
    │   │
    │   └── React + Vite
    │       │
    │       ├── Application UI
    │       ├── Chat interface
    │       ├── Character controls
    │       ├── Three.js character stage
    │       └── Memory interface
    │
    └── Managed Node.js process
        │
        └── Local HTTP API
            │
            ├── OpenRouter integration
            ├── Fish Audio integration
            ├── Settings and chat sessions
            └── SQLite-backed memory

The frontend communicates with the local Node.js API through a loopback address.

This gives the application a separation between presentation, application services and native desktop responsibilities.

The browser development version can run through Vite and Node.js, while the native desktop version is hosted by Tauri.

4. Building the interactive 3D character with Three.js

This was one of the most interesting parts of the project.

I used Three.js and GLTFLoader to load the character and equipment models.

The character stage supports:

• Real-time 3D model rendering.
• Camera rotation and elevation controls.
• Character gestures and animation states.
• Equipment rendering and positioning.
• Head movement, including pitch, yaw and roll.

The character uses a shared action registry rather than having completely separate action definitions for every interface control.

This became particularly useful when connecting the character to the AI assistant.

A shared character-action system

I wanted both user interactions and AI-generated actions to use the same action definitions.

For example, the interface might trigger a wave animation, while the AI assistant might decide that a wave is an appropriate response to a greeting.

Instead of implementing two different wave systems, both paths use the shared character-action registry.

This makes it easier to maintain consistent behaviour across the interface and assistant.

Equipment interactions

Another interesting challenge was making equipment appear correctly attached to the character.

The project includes Minecraft-inspired armor, a mace and a Totem of Undying.

Getting these assets to fit correctly required repeated adjustments to their positioning, orientation, scale and attachment points.

It was a reminder that loading a 3D model successfully is very different from integrating it convincingly into a character scene.

5. Connecting AI conversations to character actions

The next challenge was making the character respond to the assistant rather than simply playing animations through buttons.

The application uses OpenRouter for AI conversations.

I exposed character actions and equipment interactions to the assistant through local tools.

This allows the AI to trigger supported actions rather than having the frontend guess which animation should play based on arbitrary response text.

For example, a greeting could result in a wave, while an explanation could use an explaining gesture.

I wanted the action system to be explicit and controlled, rather than allowing arbitrary AI-generated commands to manipulate the 3D scene.

6. Voice interaction and mouth animation

Voice interaction adds another layer of complexity.

The application integrates Fish Audio for text-to-speech.

The audio system uses a playback queue to handle speech in chunks and supports interruption.

The character's mouth animation responds to actual audio playback activity.

This creates a basic connection between the character's speech and its visual behaviour.

It is important to distinguish playback-driven mouth animation from full phoneme-based lip synchronization. The current implementation focuses on responding to audio activity; more accurate phoneme or viseme alignment would be a possible future improvement.

The character also returns to its resting state after speech finishes.

7. Persistent memory with SQLite

I didn't want the assistant to forget everything when a conversation ended.

For this reason, I developed a persistent local memory system backed by SQLite.

The memory interface supports:

• Creating memories.
• Searching stored information.
• Updating existing entries.
• Archiving memories.
• Restoring archived memories.
• Deleting unwanted memories.

The backend also handles validation, backups and migration.

An important design consideration was keeping memory operations separate from the chat interface.

The assistant can access relevant stored information through memory tools, while the user retains control over the actual memory collection.

8. Why I chose Tauri 2

Moving from a browser application to a desktop application introduced another set of challenges.

Tauri provides the native application window and manages the desktop lifecycle.

The React application runs inside the platform's WebView, while Rust manages the application process and starts the local Node.js backend.

This approach avoids making the frontend responsible for managing native process execution.

The application also needs to account for platform-specific details:

• Windows uses WebView2.
• Linux uses its available WebKit-based WebView environment.
• macOS uses its native WebKit environment.

The runtime and packaging layers require platform-specific handling even when the React frontend remains largely shared.

9. Cross-platform development

I initially developed the application on Windows.

As the project developed, I began working toward Linux and macOS support.

The repository includes platform-specific development launchers:

Platform	Launcher
Windows	start.bat
Linux	start.sh
macOS	start.command

These scripts provide a consistent way to start the browser-based development application.

Native desktop development uses Tauri's development command instead.

For distribution, the packaging targets include:

• Windows NSIS installer.
• Linux DEB and AppImage packages.
• macOS DMG packages for Apple Silicon and Intel.

The Windows installer has been tested locally. Linux and macOS packaging still require platform-specific runtime testing before I would consider those distributions fully validated.

This part taught me that cross-platform support is not simply a matter of compiling the same frontend for different operating systems. Process management, runtime binaries, WebViews, native dependencies, file paths and installer behaviour all matter.

10. Challenges and lessons learned

A few lessons stood out during development.

A 3D character needs its own architecture

A character should not become a collection of unrelated animation buttons.

Having a shared action registry made it easier to connect the UI, backend and AI tools.

Desktop applications need clear process boundaries

Separating the local Node.js API from the React interface made the application easier to reason about.

Tauri manages the native lifecycle, while the frontend focuses on the user experience.

Voice is more than generating audio

Playback queues, interruption, synchronization and returning the character to its resting state all affect how natural the experience feels.

Persistent memory needs management tools

Saving information is only one part of memory.

Users also need to search, edit, organize, archive and delete it.

Cross-platform builds need real testing

A successful compilation does not prove that an installed application works.

The complete application needs to be tested on each target operating system.

11. What's next?

There is still plenty of room to improve the application.

Some areas I'd like to explore further include:

• More natural audio-driven facial animation.
• Better character animation transitions.
• Improved equipment attachment and positioning.
• More robust cross-platform packaging.
• Additional memory organization and retrieval capabilities.
• Better first-time setup and diagnostics.

I'm also interested in feedback from developers who work with Three.js, Tauri, desktop application architecture and AI integrations.

12. Try the project

The project is available on GitHub:

Repository: https://github.com/opendeveloper-ai/wemmbu-ai-assistant

Latest release: https://github.com/opendeveloper-ai/wemmbu-ai-assistant/releases/tag/v0.2.9

The repository includes the application source, documentation and development setup instructions.

If you're interested in experimenting with an AI assistant that has an interactive 3D character, I'd appreciate your feedback.

I'm particularly interested in how other developers would approach the architecture, character animation and desktop integration.

---

Disclaimer: Wemmbu AI Assistant is an independently developed, unofficial project inspired by Minecraft creator content. It is not affiliated with, endorsed by, sponsored by or operated by Wemmbu, the Unstable SMP, Mojang Studios or Microsoft. Third-party assets and integrations remain subject to their respective licences and terms.