I replaced the chat window for my local AI agent with a face

ghostvessel — Wed, 08 Jul 2026 21:41:56 +0000

I run a local LLM agent (Hermes) on my own machine. The problem was never the model — it was the interface. I had a Telegram tab open all day just to talk to it: type a command, wait, read a wall of text back, scroll. It felt like texting a very capable stranger.

So I built Ghost Vessel — a monitor-resident, video-call-style avatar that fronts the agent. The name is the whole idea: the ghost is your agent, the vessel is the body it borrows. It's not a waifu toy; it's a real agent client that happens to have a face.

Here's what actually turned out to be interesting to build.

The reply is a script, not a string

The core idea is an output contract. Instead of treating the agent's reply as text to print, I split every reply into three planes:

dialogue → spoken via local TTS
data → code, logs, files → rendered as chat cards, never read aloud
action → emotion beats that drive the avatar

Emotion beats are inline tags the model emits in-band with its answer:

[working] — the avatar puts on glasses and takes notes while a task runs
[confirm] deploy to prod? — pops a human-in-the-loop approve/cancel, and the agent blocks on your keypress
[happy] / [concerned] / … — fine-grained facial expressions

So "run the build, and if it passes, deploy" becomes a little performance: it looks busy while working, shows you the log as a card, then leans in and asks before the irreversible step. The text you'd have skim-read becomes something you glance at.

No runtime GPU for the avatar

The obvious way to animate a face is live inference. I didn't want that — the GPU is busy running the actual model.

Instead the avatar is ~30 pre-rendered clips, and the emotion beats just select and blend between them (blink-aligned seamless idle loops, a head-pose "settle gate" so an expression only reveals when the head is frontal). The avatar's runtime cost is basically video playback. Your GPU stays 100% on your LLM.

The tradeoff: no real-time lip-sync. I decided a believable talking mouth loop + expressive face reads as "a person on a call" far more than perfect phoneme-matching does, and it costs nothing at runtime. That single decision collapsed most of the hard engineering.

Hooking a real agent without the agent knowing

The agent integration is the part I was most unsure would work. It turned out clean: the app registers as a connector. The agent's gateway dials out to a local WebSocket the app hosts, and exchanges frames — so from the agent's side, the avatar is just "another channel," indistinguishable from Telegram.

Inbound (you → agent): your typed text as a message event
Outbound (agent → you): the reply, which the connector parses for emotion beats

Adapters for Hermes and OpenClaw are included, plus a demo responder so it runs with zero setup. And the chat pane serves the agent's live slash-command menu — type / and you get the same 52 commands you'd see in the messenger, passed straight through.

The bits that make it feel alive

Mood & affinity. A short-term mood decays toward a long-term relationship baseline. Scold it repeatedly and it settles into a subdued idle; praise it and it brightens. Persists across restarts.
Voice both ways. Edge TTS out (swappable to local Qwen3-TTS / MeloTTS / Piper), Silero VAD + faster-whisper in.
Monitor-resident. Two frameless, always-on-top windows (avatar + chat) you drag anywhere, built with Tauri v2 (Rust shell, Python servers, JS player).

Open-core

The engine is MIT and usable on its own: clone it, drop a folder of clips named by emotion (happy.mp4, working.mp4, idle.mp4, …), point it at your agent. Avatars are pure-data bundles — no code runs when you install one — so you can build your own; the full reproducible method is in the repo.

Repo + 45s demo: https://github.com/ghdtjrtka/ghost-vessel
Windows build: https://ghostvessel.space

What I'm curious about

The part I'd most like feedback on is the emotion-beat output format — the contract that turns LLM text into a UI performance. Has anyone else built output contracts to drive an interface from model output? What broke, and what did models reliably get right vs. wrong? I'd genuinely like to compare notes.