I run a local LLM agent (Hermes) on my own machine. The problem was never the model — it was the interface. I had a Telegram tab open all day just to talk to it: type a command, wait, read a wall of text back, scroll. It felt like texting a very capable stranger.
So I built Ghost Vessel — a monitor-resident, video-call-style avatar that fronts the agent. The name is the whole idea: the ghost is your agent, the vessel is the body it borrows. It's not a waifu toy; it's a real agent client that happens to have a face.
Here's what actually turned out to be interesting to build.
The reply is a script, not a string
The core idea is an output contract. Instead of treating the agent's reply as text to print, I split every reply into three planes:
- dialogue → spoken via local TTS
- data → code, logs, files → rendered as chat cards, never read aloud
- action → emotion beats that drive the avatar
Emotion beats are inline tags the model emits in-band with its answer:
-
[working]— the avatar puts on glasses and takes notes while a task runs -
[confirm] deploy to prod?— pops a human-in-the-loop approve/cancel, and the agent blocks on your keypress -
[happy]/[concerned]/ … — fine-grained facial expressions
So "run the build, and if it passes, deploy" becomes a little performance: it looks busy while working, shows you the log as a card, then leans in and asks before the irreversible step. The text you'd have skim-read becomes something you glance at.
No runtime GPU for the avatar
The obvious way to animate a face is live inference. I didn't want that — the GPU is busy running the actual model.
Instead the avatar is ~30 pre-rendered clips, and the emotion beats just select and blend between them (blink-aligned seamless idle loops, a head-pose "settle gate" so an expression only reveals when the head is frontal). The avatar's runtime cost is basically video playback. Your GPU stays 100% on your LLM.
The tradeoff: no real-time lip-sync. I decided a believable talking mouth loop + expressive face reads as "a person on a call" far more than perfect phoneme-matching does, and it costs nothing at runtime. That single decision collapsed most of the hard engineering.
Hooking a real agent without the agent knowing
The agent integration is the part I was most unsure would work. It turned out clean: the app registers as a connector. The agent's gateway dials out to a local WebSocket the app hosts, and exchanges frames — so from the agent's side, the avatar is just "another channel," indistinguishable from Telegram.
- Inbound (you → agent): your typed text as a message event
- Outbound (agent → you): the reply, which the connector parses for emotion beats
Adapters for Hermes and OpenClaw are included, plus a demo responder so it runs with zero setup. And the chat pane serves the agent's live slash-command menu — type / and you get the same 52 commands you'd see in the messenger, passed straight through.
The bits that make it feel alive
- Mood & affinity. A short-term mood decays toward a long-term relationship baseline. Scold it repeatedly and it settles into a subdued idle; praise it and it brightens. Persists across restarts.
- Voice both ways. Edge TTS out (swappable to local Qwen3-TTS / MeloTTS / Piper), Silero VAD + faster-whisper in.
- Monitor-resident. Two frameless, always-on-top windows (avatar + chat) you drag anywhere, built with Tauri v2 (Rust shell, Python servers, JS player).
Open-core
The engine is MIT and usable on its own: clone it, drop a folder of clips named by emotion (happy.mp4, working.mp4, idle.mp4, …), point it at your agent. Avatars are pure-data bundles — no code runs when you install one — so you can build your own; the full reproducible method is in the repo.
- Repo + 45s demo: https://github.com/ghdtjrtka/ghost-vessel
- Windows build: https://ghostvessel.space
What I'm curious about
The part I'd most like feedback on is the emotion-beat output format — the contract that turns LLM text into a UI performance. Has anyone else built output contracts to drive an interface from model output? What broke, and what did models reliably get right vs. wrong? I'd genuinely like to compare notes.
Top comments (0)