Open Source Project of the Day (#134): AIRI — Self-Hosted AI VTuber That Actually Plays Minecraft

Introduction

"A container of souls of waifu, cyber livings to bring them into our worlds."

This is article #134 in the Open Source Project of the Day series. Today's project is AIRI — an open-source, self-hosted AI virtual companion system targeting Neuro-sama's capabilities.

Some context on Neuro-sama: she's an AI VTuber built by Vedal, streaming live on Twitch since 2022. She talks back to chat in real time, plays Minecraft, plays Among Us, and has accumulated hundreds of thousands of followers. She's not a virtual avatar replaying recorded content — she's an AI system doing real-time inference and real-time decision making.

AIRI's premise: replicate this, open-source, on your own hardware.

Technically, AIRI is a carefully constructed system. Web-first architecture on Vue.js + TypeScript, in-browser local inference via WebGPU, Live2D and VRM avatar rendering, a multi-provider voice pipeline, and a Minecraft agent that actually does things in the game. 44.8k Stars, and the project has spawned independent tools like xsai and unspeech along the way.

What You'll Learn

• AIRI's web-first architecture: why Vue.js + WebGPU instead of Python + Electron
• The realtime voice pipeline: VAD → STT → LLM → TTS four-stage chain
• How the Minecraft game agent works (Mineflayer + vision + decision loop)
• The Live2D/VRM avatar driver layer
• In-browser memory system: DuckDB WASM + pgvector running in a browser tab
• Spawned sub-projects: xsai lightweight AI SDK, unspeech speech proxy

Prerequisites

• Basic familiarity with AI assistants and virtual avatar concepts
• Vue.js or frontend development experience helps with the architecture sections
• Watching a few minutes of Neuro-sama gives useful context

---

Project Background

What Neuro-sama Is

Neuro-sama (built by Vedal987) began streaming in 2022. She responds to Twitch chat, plays games, expresses personality, makes jokes. The technical achievement is that she operates live — not scripted, not pre-recorded. A real AI system doing real-time inference in front of an audience.

AIRI sets this as the target and asks: can an open-source community build the same thing, self-hostable by anyone?

Author / Team

• Organization: moeru-ai (open-source community project)
• License: MIT
• Stack: Vue.js + TypeScript, pnpm Monorepo, Turborepo

Project Stats

• ⭐ GitHub Stars: 44,800+
• 🍴 Forks: 4,500+
• 📦 Platforms: Web (PWA), macOS, Windows, Linux, iOS
• 📄 License: MIT

---

System Architecture

Web-First, Not Python-First

Most AI projects default to Python. AIRI chose the web platform:

Typical AI project path:
Python scripts → local server → WebSocket → frontend display
Problem: Python environment setup, dependency conflicts, cross-platform friction

AIRI's path:
Vue.js + TypeScript → runs natively in the browser
WebGPU → local inference (no Python environment needed)
WebAudio → voice processing
WebAssembly → high-performance compute
Web Workers → parallel processing without blocking UI

The tradeoff: giving up Python's mature AI toolchain. The payoff: any device with a modern browser can run it, including phones.

Monorepo Structure

airi/
├── apps/
│   ├── stage-web/          ← Web PWA main app
│   ├── stage-tamagotchi/   ← Desktop (Electron, floating pet style)
│   └── stage-pocket/       ← Mobile
├── core/
│   ├── brain/              ← Brain: LLM routing, memory coordination
│   ├── stt-tts/            ← Voice pipeline
│   └── server-runtime/     ← Game agent dispatch
├── packages/
│   ├── stage-ui/           ← Live2D/VRM rendering layer
│   └── duckdb-wasm/        ← In-browser database driver
└── services/
    ├── minecraft/          ← Minecraft agent (Mineflayer)
    └── factorio/           ← Factorio agent (RCON API)

---

Core Capabilities

Realtime Voice Pipeline

Voice conversation is the core capability of any VTuber system. AIRI's pipeline runs in four stages:

User speaks
    ↓
VAD (Voice Activity Detection)
    Detects speech start/end — no push-to-talk button
    ↓
STT (Speech to Text)
    Routes through unspeech proxy to any provider
    Supports: Whisper (local), OpenAI, Azure Speech, etc.
    ↓
LLM Processing (Brain module)
    Routes through xsai to 30+ providers
    Claude, GPT, Gemini, DeepSeek, Ollama...
    Memory injection (DuckDB + pgvector)
    ↓
TTS (Text to Speech)
    ElevenLabs, Azure Speech, OpenAI, Kokoro (local)
    Outputs audio + lip-sync data
    ↓
Live2D/VRM avatar rendering
    Lip sync, eye movement, expression driving

Kokoro TTS is the local voice option — no API key, offline, competitive quality among open-source TTS options.

Live2D and VRM Avatar Rendering

@proj-airi/stage-ui handles avatar rendering:

• Auto-blink: Randomized intervals to avoid the "dead fish eyes" effect
• Gaze tracking: Eyes follow mouse movement or topic focus
• Idle animation: Natural micro-movements when the character isn't speaking
• Lip sync: Mouth driven by phoneme data from TTS output
• Expression system: LLM output includes emotion tags that drive facial expressions

Both formats supported: Live2D (.moc3, 2D skeletal animation) and VRM (3D virtual avatar standard format).

WebGPU Local Inference

Browser version uses WebGPU for on-device inference. Desktop version uses HuggingFace Candle library to call NVIDIA CUDA or Apple Metal directly:

Browser:
WebGPU API → GPU compute units
No local AI framework installation needed
Supported: Chrome 113+ (experimental), Edge, Safari (experimental)

Desktop (Electron):
HuggingFace Candle (Rust) → CUDA / Apple Metal
Performance close to native Python frameworks
No CUDA toolchain configuration — binary includes everything

Game Agents

Minecraft (live):

AIRI connects to a Minecraft server (Mineflayer library)
    ↓
Visual input: screenshot → multimodal LLM analysis
(What's the current state? Mining? Encountering mobs? Low on materials?)
    ↓
Action output: move, mine, build, interact
    ↓
Voice output: real-time explanation to viewers of what it's doing and why

AIRI doesn't follow a fixed script in Minecraft. The LLM decides next steps based on current game state, and narrates its reasoning to viewers.

Factorio (proof-of-concept):

Controlled via RCON API with the autorio mod library. The agent reads factory state and dispatches commands. Factorio's complex production chain optimization is a harder planning problem than Minecraft — still early, but architecturally the same approach.

---

Spawned Sub-Projects

xsai: Lightweight AI SDK

While building AIRI, the team found Vercel AI SDK too heavy and built their own:

import { createOpenAI } from 'xsai'

const openai = createOpenAI({ apiKey: 'sk-...' })

const result = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [{ role: 'user', content: 'Hello' }]
})

Works with any OpenAI-compatible provider — Claude, Gemini, DeepSeek, Ollama local models, and more.

unspeech: Unified Speech API Proxy

Different speech services have incompatible API formats. unspeech provides uniform /audio/transcriptions and /audio/speech endpoints:

# Start the unspeech proxy
npx unspeech

# Send requests to any provider through a consistent interface
curl -X POST http://localhost:3000/audio/transcriptions \
  -H "X-Provider: openai" \
  -F "file=@audio.mp3"

Switch between OpenAI Whisper, Azure Speech, or local Whisper on the backend — the frontend code stays unchanged.

---

Memory System

Still in development, but the architecture is set:

Conversation content from sessions
        ↓
DuckDB WASM (in-browser SQL database)
pglite (lightweight PostgreSQL, pure browser)
        ↓
@proj-airi/memory-pgvector (vector similarity search)
        ↓
Memory Alaya (memory retrieval layer, WIP)
        ↓
Relevant historical memories injected into LLM context

Runs entirely in the browser. No external database dependency. Data stays local.

---

Quick Start

Installation

# Windows (winget)
winget install MoeruAI.AIRI

# macOS (Homebrew)
brew install --cask airi

# Development from source
git clone https://github.com/moeru-ai/airi.git
cd airi
pnpm install
pnpm dev

Basic Configuration

# Configure LLM provider (pick one)
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
# Or use Ollama local models — no API key needed

# Configure TTS (or use local Kokoro — no API key needed)
ELEVENLABS_API_KEY=...

Open http://localhost:5173, select an avatar, configure your LLM, start talking.

---

Comparison

Dimension	AIRI	Character.ai	Commercial VTuber tools
Open source	✅ MIT	❌	❌
Self-hostable	✅	❌	Partial
Data privacy	Fully local option	Uploaded to servers	Unclear
Game capability	✅ Minecraft/Factorio	❌	Rarely
Local inference	✅ WebGPU/CUDA	❌	Partial
Custom avatars	Live2D + VRM	❌	Limited

---

Links and Resources

• 🌟 GitHub: moeru-ai/airi
• 🌐 Website/Docs: airi.moeru.ai
• 📦 Sub-projects: xsai · unspeech
• 💬 Hacker News discussion: Show HN: Recreation of Neuro-sama

---

Conclusion

AIRI's goal stated plainly: open-source Neuro-sama's capabilities so anyone can run something like her on their own hardware.

Three design decisions stand out:

Web-first architecture is a counterintuitive choice. Python is the default for AI projects. AIRI bet on Web standards — WebGPU, WebAudio, WebAssembly — accepting early toolchain immaturity in exchange for consistent cross-platform behavior and zero installation friction. WebGPU has matured significantly in 2025–2026. That bet is paying off.

Game agents move AI companions from "chat system" to "things that actually do things." The Minecraft agent doesn't just answer "what are you doing" — it acts in the game and narrates its decisions to viewers simultaneously. That direction is more interesting than chat alone, both technically and as a product.

Spawned sub-projects signal real engineering substance. xsai and unspeech both grew from actual development needs, not ecosystem building for its own sake. A project that creates reusable tools as a side effect is usually one where someone is thinking carefully about the underlying problems.

44.8k Stars, MIT license, active community — AIRI is one of the most seriously built projects in open-source AI companion space right now.

---

Explore PrimeSkills — A marketplace for handpicked AI Agents and skills. Each is validated in real enterprise workflows, stripping away hype and keeping only what truly works.

Welcome to my Homepage for more useful insights and interesting products.