Every few months a new "AI companion" app shows up, gets a viral demo clip, and turns out to be a hosted SaaS wrapper around a chat API with a cute avatar bolted on. Project AIRI is not that. It's a 47,900-star, MIT-licensed, fully self-hostable stack that runs a voice-driven, animated AI character on your own hardware — no account, no cloud dependency unless you explicitly wire one in — and it plays Minecraft and Factorio while it's at it. The project openly states its target: "wishing to achieve Neuro-sama's altitude," referring to the (closed-source, commercially operated) AI VTuber that popularized the genre on Twitch.
What makes AIRI worth a technical look isn't the waifu framing. It's that the team solved a genuinely hard systems problem — running a full conversational-agent pipeline (STT → LLM → TTS → animation → memory) inside a browser tab, using WebGPU and WebAssembly, with no server required — and then built a second, entirely different execution mode (native desktop via Electron, with CUDA/Metal acceleration) on top of the same core. That's an unusual architecture bet, and it has real consequences for who should and shouldn't use this thing.
What it actually does
Strip away the anime branding and AIRI is: a character-agent runtime that takes voice or text input, routes it through an LLM of your choice, and renders the response as speech plus a synced Live2D or VRM avatar, while optionally letting that agent act autonomously in a game world.
Concretely, the shipped capabilities are:
- Real-time voice chat with client-side speech recognition (no audio leaves your machine unless you choose a cloud STT provider).
- Avatar rendering via Live2D (2D rigged sprite animation) or VRM (3D humanoid model format popularized by VTuber tooling), with auto-blink, eye-tracking, and idle motion.
- Game agents: a Minecraft bot built on mineflayer that can navigate, mine, and respond to chat in-game, plus a proof-of-concept Factorio integration driven over Factorio's RCON protocol. Kerbal Space Program and Helldivers 2 co-play integrations are listed as in-progress.
- Chat platform bridges into Discord and Telegram, so the same character can live in a voice channel, not just a browser tab.
- A pluggable memory layer, currently backed by an in-browser DuckDB-WASM database, with an experimental long-term memory subsystem the project calls "Memory Alaya" still under development.
It ships three ways to run it: stage-web (pure browser, WebGPU-accelerated, explicitly documented as a "proof of concept" for performance), stage-tamagotchi (the Electron desktop build — Windows, macOS, Linux, with installers via Homebrew, Winget, Scoop, and Nix), and mobile PWA builds for iOS/Android via Capacitor.
How it's built
The repository is a TypeScript/Vue monorepo split into UI packages (@proj-airi/stage-ui, @proj-airi/ui, plus dedicated transition and loading-screen packages and CJK/Xiaolai font bundles for proper Chinese/Japanese/Korean glyph rendering), infrastructure packages (@proj-airi/server-runtime, @proj-airi/server-sdk, @proj-airi/server-shared, and a DuckDB ORM/WASM wrapper pair), and application shells (stage-web for the browser, stage-tamagotchi for desktop, realtime-audio, and a playground-prompt-engineering sandbox for iterating on character prompts in isolation). That separation is the whole point architecturally: the character's brain — prompt construction, tool calling, memory retrieval, TTS/STT routing — lives in the shared infrastructure packages, and each application is a thin rendering/input host around it. Change the model or add a new avatar and every deployment target picks it up without a rewrite.
The part that matters most is the LLM abstraction layer, built on the team's own xsAI library, which fans out to 30+ providers: OpenAI, Anthropic, DeepSeek, Google Gemini, Ollama, vLLM, SGLang, Groq, Mistral, Together.ai, Fireworks.ai, and a cluster of Chinese providers (Qwen, Baichuan, Moonshot) reflecting the project's origin in a Chinese-speaking open-source community — its docs ship in Simplified Chinese, Japanese, Korean, Russian, Vietnamese, and French alongside English, and it has an active Crowdin-based translation pipeline. xsAI itself is a lightweight, tree-shakeable alternative to heavier SDKs like the official OpenAI or LangChain clients, built specifically so an app that only needs chat completions doesn't have to ship an entire agent-framework's worth of JavaScript to a browser tab — a real constraint when your target runtime is client-side WebAssembly, not a server with unlimited bundle size.
This is the practical payoff of self-hosting: point AIRI at a local Ollama or vLLM endpoint and the character never touches a third-party API at all. Point it at Claude or GPT-class models instead and you trade the privacy guarantee for higher reasoning quality — the architecture makes that a runtime config change, not a rewrite. Nothing about the avatar rendering, memory layer, or game-agent code needs to know or care which provider is answering.
Speech runs through a similarly pluggable path: TTS via ElevenLabs, Azure, OpenAI-compatible endpoints, Alibaba Cloud Model Studio, or a fully local Kokoro TTS model, unified behind an internal proxy the project calls "unspeech" that normalizes every backend to the same /audio/transcriptions and /audio/speech endpoint shape. STT defaults to client-side browser recognition, so voice input never leaves the device unless you deliberately route it to a cloud STT provider. For local, GPU-bound inference the desktop build uses HuggingFace's Candle framework — a Rust-based tensor library built as a lighter-weight PyTorch alternative — with CUDA acceleration on Windows/Linux and Metal acceleration on Apple Silicon. That's also why the docs are upfront that the web build is a proof of concept and the desktop build is "recommended for better performance": running transformer inference inside a browser tab via WebGPU is still relatively immature technology industry-wide, while native GPU acceleration through Candle is a well-trodden path.
The avatar layer sits on top of that inference pipeline, not beside it. Once the LLM returns text and the TTS layer synthesizes audio, the animation system drives Live2D (2D rigged sprite deformation, the older and more common VTuber format) or VRM (a 3D humanoid glTF-based standard with a broader open tooling ecosystem, including Unity and VRChat interoperability) with auto-blink, gaze tracking toward the cursor or camera, and idle micro-motion layered on top of whatever mouth-flap or expression cues the TTS engine emits. None of this requires round-tripping to a server — it's computed client-side in the same WebGPU/WebAudio context that's running everything else.
The game-agent piece is architecturally separate from the chat/avatar core, and this is the part that distinguishes AIRI from every "cute avatar + chatbot" clone. Minecraft control goes through mineflayer — a well-established Node.js bot framework used across a large ecosystem of Minecraft-automation and research projects (it's also the substrate several academic "LLMs playing Minecraft" papers have built on) — paired with mineflayer-pathfinder for navigation. Factorio integration goes through the game's own RCON console API rather than mineflayer's protocol, since Factorio isn't a Minecraft-protocol game; it's currently flagged proof-of-concept rather than feature-complete. These aren't LLM function-calling toys bolted onto a chat window where the model emits a move_forward() call and nothing else happens. They're autonomous perception-action loops: the agent polls game state on a timer, decides on an action, executes it, and treats the conversational channel as one more input alongside what it's currently seeing and doing in the game world — architecturally closer to an embodied agent than a chatbot with plugins.
The memory problem, and why it's still open
Every long-running conversational agent runs into the same wall: an LLM's context window is finite, but a "companion" is supposed to remember things across sessions — what you talked about last week, your name, a running joke. AIRI's current answer is a DuckDB-WASM-backed local store — DuckDB is an embedded analytical database (think SQLite's cousin, optimized for columnar queries), compiled to WebAssembly so it runs entirely inside the browser sandbox with no server round-trip. That gets you structured, queryable local storage of conversation history and character state without spinning up Postgres or a vector database.
What it doesn't yet get you is retrieval-quality long-term memory — the kind where the agent can surface a semantically relevant memory from three weeks ago in response to an oblique reference, the way commercial products like Character.AI or memory-focused infrastructure projects (Mem0, Zep, and similar) are explicitly built around. That capability is under active development under the internal codename "Memory Alaya" and is marked experimental in the docs. If you're evaluating AIRI as a memory-management reference architecture rather than just a chat frontend, this is the component to watch rather than assume is finished — it's the single biggest gap between "cute demo" and "companion that actually remembers you."
Practical use cases
A few concrete scenarios where the architecture choices actually pay off, beyond "it's a fun demo":
-
A streamer wants a co-host that costs nothing per hour. Point
stage-tamagotchiat a local Ollama model and Kokoro TTS, wire in the Discord/Telegram bridge, and you have an always-on character that costs electricity, not per-token API fees — meaningfully different economics than a Twitch-integrated bot calling a hosted LLM API per chat message during a multi-hour stream. - A developer wants a local, privacy-preserving voice assistant and doesn't care about the anime avatar at all — the avatar is optional UI, not a load-bearing part of the pipeline. Strip it and you have a self-hosted voice-in/voice-out LLM assistant with STT, provider-agnostic reasoning, and TTS already wired together, which is otherwise a nontrivial amount of glue code to write from scratch.
- A researcher prototyping embodied LLM agents gets a working Minecraft perception-action loop for free instead of standing up mineflayer integration from zero — useful as a starting scaffold for agent-planning or tool-use experiments that need a persistent, stateful world rather than single-turn chat evals.
- A team evaluating "AI companion" or "AI employee" products for a client gets a concrete, inspectable existence proof that voice-driven character interaction doesn't require shipping user audio to a third-party API — useful ammunition in build-vs-buy and data-residency conversations even if you never ship AIRI's code directly.
- A VTuber or indie streamer without production budget gets Live2D/VRM-driven AI co-hosting without paying for a bespoke Neuro-sama-style build, which by all public accounts represents a significant, custom, non-reusable engineering investment by a single team.
What's actually new here
The genre of "open-source VTuber/AI companion" isn't new — projects inspired by ChatVRM (pixiv's early browser-based VRM chat demo, which AIRI explicitly credits as a foundation, along with a JavaScript port by developer josephrocca) have existed for a couple of years, and the repo maintains its own curated "Awesome AI VTuber" list of adjacent projects. What AIRI does differently is refuse to pick a lane: most prior projects were either a browser demo or a Discord bot or a desktop app, usually hardcoded to a single LLM provider and a single avatar format because that was enough to prove the concept. AIRI's core is written once and deployed across web, desktop, and mobile PWA, and it's one of the more committed attempts in this space to make in-browser WebGPU inference a first-class deployment target rather than treating the browser as a thin client to a mandatory backend server.
The other real differentiator is provider-agnosticism at a scale most companion projects don't bother with. Thirty-plus LLM providers, half a dozen TTS backends, and two avatar formats means you're not locked into a single vendor's roadmap or pricing — a direct contrast to something like Character.AI, which is a closed platform you rent access to and whose model, memory system, and moderation policy you don't control, or Neuro-sama itself, which isn't available to run at all; it's a persona operated by one team on one Twitch channel, with no public codebase, no self-hosting path, and no way to swap its underlying model even if you wanted to.
There's also a scope difference worth naming directly: AIRI is explicitly building toward agency in a world, not just conversation. Game integrations aren't a stretch feature bolted on for a demo video — they're listed alongside chat as core capabilities, with Minecraft shipped, Factorio at proof-of-concept, and Kerbal Space Program and Helldivers 2 co-play in progress. That puts it in different company than most "AI companion" projects, which stop at text/voice chat and a static or lightly-animated avatar. It's closer in spirit to game-playing agent research (think Voyager-style Minecraft agents built on GPT-4, or DeepMind's game-playing agent lineage) wearing a companion-app UI, than it is to a chatbot skin.
Why developers should care, beyond the novelty
Cost and lock-in. Because the LLM, TTS, and STT layers are all swappable, you can run the entire stack for the price of local compute — genuinely $0 in API costs if you pair local Ollama models with Kokoro TTS and browser STT. That's a meaningfully different cost model than every hosted "AI companion" product, which bills per message or per month.
A working reference for multi-modal agent orchestration. Even if you have zero interest in VTubers, the codebase is a real-world example of coordinating STT, an LLM with tool/function access, TTS, avatar animation state, and a persistent memory store into one low-latency loop — the same architectural problem underlying voice assistants, game NPCs, and customer-support avatars. Reading server-runtime and stage-ui is a faster way to understand that orchestration problem than most blog-post architecture diagrams.
Privacy by default. For anyone building or evaluating "AI companion" or "AI employee" products for a client, AIRI is a useful existence proof that this category doesn't require shipping user voice and conversation data to a third party. It also makes it a legitimately different tool for local, offline agent experimentation than cloud-only competitors.
Game-agent scaffolding. The Minecraft/Factorio integration is a usable starting point if you're prototyping LLM agents that need to perceive and act in a persistent, stateful simulated environment rather than a stateless chat turn — a much closer analog to robotics or industrial-automation agent work than typical chatbot demos.
Who's actually building this
AIRI comes out of moeru-ai, a small open-source collective rather than a funded startup, and the project structure reflects that: 4,800 forks, 88 open issues, 109 open pull requests, and a Discord community reported at roughly 738 members at time of writing, alongside presence on X/Twitter, Telegram, WeChat, and QQ — the multi-platform community footprint itself signals a project that grew organically out of a Chinese- and Japanese-speaking indie-dev and VTuber-adjacent scene before picking up broader English-speaking attention. Funding runs through OpenCollective, GitHub Sponsors, Patreon, and Ko-fi rather than venture capital, which shapes the roadmap in a predictable way: features ship in whatever order maintainers and contributors find personally motivating (Minecraft before Factorio, Factorio before Kerbal Space Program), not in the order a product manager would prioritize them for a commercial launch. That's a legitimate trade for a research-flavored open-source project and a legitimate reason for a risk-averse team to wait.
One detail is worth calling out because it's an unusually direct signal of how much unwanted attention a fast-growing open-source AI project can attract: the README carries an explicit line stating the project has no official cryptocurrency or token associated with it. That kind of disclaimer doesn't get added speculatively — it gets added after impersonators start using a trending repo's name to run token scams, which has become a recurring pattern around viral open-source AI projects over the past couple of years. It's a small line, but it's a useful reminder to only ever pull AIRI from the official GitHub repository or its listed release binaries, not from search-engine ads or unofficial mirrors.
Where the docs go quiet
A few things worth knowing before you commit to this as more than a toy:
- The web build is explicitly a proof of concept. The project itself says so. If you want acceptable latency and animation smoothness, you're running the Electron desktop build, which somewhat undercuts the "just open a browser tab" pitch for the flagship deployment mode.
- Memory is unfinished. "Memory Alaya," the long-term memory system, is marked experimental. Out of the box, character memory is bounded by DuckDB-WASM local storage, not a mature vector-memory or retrieval architecture — don't expect a character that remembers a conversation from three weeks ago without extra engineering.
- This is pre-1.0 software. Current release is v0.11.3, there are 88 open issues and 109 open PRs at time of writing, and the README actively solicits contributors. Game integrations beyond Minecraft (Factorio, Kerbal Space Program, Helldivers 2) are explicitly marked work-in-progress or "announcement pending." Treat this as an actively-developed research project, not a finished product you deploy and forget.
- Community, not company. There's no commercial entity backstopping support SLAs — see above on how that shapes the roadmap.
- Compute isn't free just because it's local. Local LLM + local TTS + WebGPU inference still needs a reasonably capable GPU to feel responsive. "Self-hosted" doesn't mean "runs well on a five-year-old laptop."
How it stacks up
| Project AIRI | Neuro-sama | Character.AI | Generic ChatVRM-style demo | |
|---|---|---|---|---|
| Source | Open (MIT) | Closed | Closed | Usually open, unmaintained |
| Hosting | Self-hosted (web/desktop/mobile) | Single operator, not distributable | Cloud SaaS only | Self-hosted, browser-only |
| LLM choice | 30+ providers, local or cloud | Unknown/proprietary | Fixed, platform-controlled | Usually one hardcoded provider |
| Game agency | Minecraft (shipped), Factorio/KSP/Helldivers 2 (WIP) | Yes (bespoke, not reusable) | No | No |
| Avatar formats | Live2D + VRM | Live2D (bespoke) | None (text/voice only) | Usually VRM only |
| Cost | Compute only (can be $0) | N/A, not for rent | Subscription | Compute only |
| Maturity | Pre-1.0, active dev | Production, years of uptime | Production SaaS | Often abandoned |
My read
AIRI is a legitimately interesting piece of applied-agent engineering wearing a costume that will make plenty of engineers dismiss it on sight — and that's a mistake if your interest is the architecture rather than the aesthetic. The multi-runtime core (one codebase, three deployment targets), the provider-agnostic LLM/TTS abstraction, and the stateful game-agent design are all patterns worth studying regardless of whether you care about virtual characters. What it is not yet is a polished product: the web build's own docs call it a proof of concept, memory is experimental, and several headline features (Factorio, KSP, Helldivers 2) are aspirational. The 47.9k stars reflect genuine community interest in the "own your digital companion" pitch, not evidence that the hard problems — believable long-term memory, low-latency browser inference — are solved yet.
Try it if you want a self-hosted testbed for multi-modal agent orchestration, or you're specifically interested in local-first voice/avatar AI and don't mind pre-1.0 rough edges.
Wait if you need something production-stable today, or you were hoping the browser build alone would deliver desktop-grade latency.
Ignore it if you just want a chat companion app and have no interest in self-hosting or the underlying architecture — Character.AI or a similar SaaS will serve you better with zero setup.
What's your experience running LLM inference client-side via WebGPU — is it close to production-viable yet, or still firmly proof-of-concept territory for anything beyond toy demos?
Sources:
Top comments (0)