DEV Community: lujianfu2008

I Built a Fully-Local Voice AI Assistant with a 3D Avatar — Here's What I Learned

lujianfu2008 — Sun, 08 Nov 2026 02:15:20 +0000

For the past few months I've been building a personal AI assistant called Xiaoqingting ("Little Dragonfly"). Not a chat wrapper — a full desktop assistant: voice in, voice out, tools that actually touch my machine, multiple agents with different jobs, and a 3D anime avatar that talks to me.

Everything important runs locally. The LLM is the only cloud component.

It's open source: github.com/lujianfu2008/xiaoqingting

This post is the architecture tour and the honest lessons list — including the bugs that cost me full nights.

Architecture: Two Processes and a Contract

┌─────────────────┐   REST + WebSocket   ┌─────────────────┐
│  @blm/desktop   │ ◄──────────────────► │   @blm/server   │
│                 │                      │                 │
│ Electron client │                      │  Fastify + SDK  │
│ voice / chat /  │                      │  agents / auth  │
│ tools / browser │                      │  gateway / keys │
└─────────────────┘                      └─────────────────┘

Two packages, one monorepo:

@blm/desktop — the Electron client. Voice capture, conversation UI, the three-vrm avatar, and a browser extension for voice dictation on any web page.
@blm/server — Fastify server wrapping the Claude Agent SDK. It owns agents, skills, tool approval, multi-user auth (JWT), and an LLM gateway with per-user quotas.

Every message between them goes through a shared protocol package (@blm/protocol) validated with Zod. If I could give past-me one piece of advice, it's this: define the protocol as a package on day one. Every refactor since has been cheap precisely because both sides fail loudly and at compile time when the contract drifts.

Lesson 1: Local-First Is an Architecture Decision, Not a Feature

Early on I had to decide where ASR, TTS, and wake-word detection live. The lazy answer is "call a cloud API for everything." I chose local:

ASR: sherpa-onnx Paraformer (streaming, int8, ~79 MB model) — fully offline, Chinese recognition works well
TTS: Windows SAPI locally, streamed in chunks so the avatar starts speaking before the full audio is ready
Wake word: a small keyword-spotting model, with wake words generated at runtime from pinyin so homophones work automatically

Was it worth the extra work? Absolutely, for three reasons: latency (no network round-trip before the avatar starts talking), privacy (my voice never leaves the machine), and cost (voice calls to cloud APIs are surprisingly expensive at conversational volume).

The one thing I'd redo: I underestimated model file logistics. 79 MB binaries don't belong in git, so the repo ships without the model and the user downloads it from sherpa-onnx releases. Write that setup step clearly in your README or your issues tab will fill up.

Lesson 2: Multi-Agent Doesn't Need a Framework

The assistant is actually several agents: assistant (chat + voice), coder (gets shell/file tools), writer (long-form, no destructive tools). Different system prompts, different tool allowlists, same SDK underneath.

I never needed LangGraph-style graphs for this. The whole thing is:

const agents = {
  assistant: { tools: [...safeTools], prompt: PERSONA },
  coder:     { tools: [...devTools],   prompt: CODER_PERSONA },
  writer:    { tools: [...readTools],  prompt: WRITER_PERSONA },
};

What actually made it feel like "multiple agents" was tool gating per agent — the coder can run commands, the writer can't touch anything destructive. If you're building multi-agent today, spend your complexity budget on permission boundaries, not orchestration graphs.

Related: the server's default posture for tools is deny unless allowlisted, and the engine never receives secrets from the server environment. An assistant with shell access is a remote-code-execution vector wearing a cute avatar. Treat its tools like you'd treat an API's public endpoints.

Lesson 3: Long Conversations Will Stall, So Watch Them

This one cost me a night. Occasionally an agent turn would simply produce nothing — the SDK had burned its budget and the CLI process went silent without emitting an error. The user stared at a spinner forever.

The fix is a turn watchdog: when a turn starts, set a deadline. If no message arrives by the deadline, kill the engine process hard and tell the user what actually happened ("this turn's compute budget ran out") instead of showing a zombie spinner.

Two details that matter:

The watchdog deadline does not extend when the user impatiently clicks "are you there" — a restarted timer is a lie
After aborting, force-tear down the socket after 10 seconds; graceful close can hang on a dead process

I wrote up the other half of this reliability story — zombie WebSocket connections and heartbeat design — in my previous post.

Lesson 4: The Avatar Will Betray You (three-vrm Edition)

The 3D avatar is a three-vrm model in a WebGL scene. The trap: VRM 0 models face away from the camera by convention, and everyone's first fix is vrm.rotation.y = Math.PI. Mine promptly turned its back to the camera the first time an animation clip overrode the transform.

The lesson generalizes: when a scene graph fights you, find out who owns the transform. Animations own it during playback; your rotation fix must happen inside the same update loop, after the animation, not once at load time. Also:/animations for idle/breathing sell the "alive" feeling more than lip-sync quality does — get a subtle idle loop in before polishing anything else.

Lesson 5: Protocol Mismatches Fail Weirdly

The LLM gateway accepts OpenAI-compatible upstreams, and the most confusing bug of the whole project looked like this: a perfectly valid request, upstream answers "model not found." The model name was fine. The real problem: the client appended ?beta=true to the URL, the upstream router keyed its model registry on the full path including the query string.

Strip queries before routing. And if your upstream speaks a protocol that returns assistant messages as separate text and tool_use chunks, merge them — some routers choke on the split form. Protocol bugs don't look like protocol bugs; they look like the upstream lying to you.

What I'd Do Differently

Ship the browser extension earlier. Voice-dictating into any web page turned out to be the feature I personally use most, and it was added last
Budget display from day one. Per-turn cost, per-user quota — retro-fitting usage analytics onto a system that's already live is miserable
Record architecture decisions. I have a CLAUDE.md with design notes, and every time I onboard my future self to a subsystem I thank past me for it

Try It

The repo is here: github.com/lujianfu2008/xiaoqingting — Node 20+, npm install, configure .env, and you have a local assistant. Star it if you like the architecture; issues and PRs welcome.

Next post: the skills system — how the assistant discovers skill packs at runtime and injects them into its own system prompt. Follow me so you don't miss it.

Why Your WebSocket Users Go "Ghost Online" — A Real Production War Story

lujianfu2008 — Sun, 08 Nov 2026 01:35:48 +0000

First, a true story from our production environment.

Users reported that messages wouldn't send — the spinner just kept spinning. But the UI said "Connected", with a nice green dot. The server logs were even weirder: those users' connections were still alive on the server. The server thought they were online, and was faithfully pushing messages into dead sockets.

Restarting the app fixed it instantly. A few hours later, it came back.

Zombie connections — also known as half-open connections. Below is everything we learned, plus the four-part reliability kit we ended up shipping. Don't ask what version one looked like. Version one was the absence of everything below.

Why TCP Lies to You

WebSocket sits on top of TCP, and TCP has a property many people only learn about in production:

If the peer drops off the network abruptly — cable unplugged, laptop asleep, Wi-Fi to 4G — your socket fires no events at all. No close, no error, nothing.

In TCP's world, you only learn a connection is dead when you actively send data and the send fails. If neither side speaks, a "connection" can stay half-dead for hours: the routers in the middle dropped the session long ago, while both endpoints still believe the other is there.

So here's the iron rule: never rely on onclose to detect a dead connection. It will not come.

Part 1: Bidirectional Heartbeats

The fix is unglamorous: make both sides talk periodically, and declare death on silence.

// Client: ping every 30s; if no pong within 15s, kill the socket and reconnect
const HEARTBEAT_INTERVAL = 30_000;
const PONG_TIMEOUT = 15_000;

let pongTimer: number | undefined;

function startHeartbeat(ws: WebSocket) {
  setInterval(() => {
    if (ws.readyState !== WebSocket.OPEN) return;
    ws.send(JSON.stringify({ type: 'ping' }));

    clearTimeout(pongTimer);
    pongTimer = window.setTimeout(() => {
      ws.close();
      reconnect();
    }, PONG_TIMEOUT);
  }, HEARTBEAT_INTERVAL);
}

// Server: reply to pings, track last-seen time
ws.on('message', (raw) => {
  const msg = JSON.parse(raw.toString());
  if (msg.type === 'ping') {
    ws.send(JSON.stringify({ type: 'pong' }));
    markAlive(ws);
  }
});

// Reaper: clean up connections silent for 90+ seconds
setInterval(() => {
  for (const ws of clients) {
    if (Date.now() - lastActive(ws) > 90_000) {
      ws.terminate();
    }
  }
}, 30_000);

Two traps here, both of which we stepped on ourselves.

First: heartbeats must be bidirectional. Our first version only had client-side pings, so the server kept pushing into dead sockets — the client zombie problem was solved, the server-side zombie queue was not. The same logic applies in reverse.

Second: use terminate(), not close(), for server-side reaping. close() runs a graceful handshake and waits for the peer to respond. The peer is already dead. Stop waiting. terminate() destroys the socket immediately.

Part 2: Exponential Backoff Reconnect

Once you detect death, you reconnect. The laziest implementation is setTimeout(reconnect, 1000), which buys you two problems: when the server restarts, every client slams it at the same second (the reconnect storm), and during a long outage clients keep hammering retries every second for nothing.

The standard fix is exponential backoff with jitter:

let attempt = 0;

function reconnect() {
  const base = Math.min(1000 * 2 ** attempt, 30_000); // 1s → 2s → 4s … capped at 30s
  const delay = base * (0.7 + Math.random() * 0.6);   // ±30% jitter
  attempt += 1;
  setTimeout(connect, delay);
}

function connect() {
  const ws = new WebSocket(url);
  ws.onopen = () => {
    attempt = 0;
    resubscribe(); // don't skip this — next section
  };
}

The jitter line is what everyone skips. Don't. It exists precisely to prevent "all clients reconnecting in perfect unison", and it costs one line.

Part 3: State Recovery After Reconnect

The most commonly missed piece: a successful reconnect does not mean everything is fine.

While you were offline, other people sent messages, task states changed, and your subscriptions evaporated. Do nothing after reconnecting, and your users see a mysterious "time hole" in the conversation.

Our approach: reconcile after every reconnect.

ws.onopen = () => {
  attempt = 0;
  // Send the last message id we have; server replies with the delta.
  // If the gap is too large, fall back to a full refresh.
  ws.send(JSON.stringify({
    type: 'sync',
    lastMsgId: store.lastMsgId,
  }));
};

Delta catch-up with a full-refresh fallback — I haven't found a simpler scheme that actually holds up.

Part 4: Acks and Stale-Event Guards

Two smaller things, worth their own section.

Send acks. "The client sent it" and "the server received it" are different facts. For important collaborative operations, have the server acknowledge with a message id; only then does the client mark it as delivered, and unacked messages go into a retry queue. That gray-to-blue checkmark in your UI is local state — don't let it lie to you.

Stale events. Here's a nasty bug hiding in reconnect logic: after an old socket dies, it can still fire onmessage one last time — leftovers from the kernel buffer. Handle it normally, and a message from "the previous world" pollutes your current state. This one took us a full night to track down. The fix is a generation counter, three lines:

let generation = 0;

function connect() {
  const myGen = ++generation;
  const ws = new WebSocket(url);
  ws.onmessage = (e) => {
    if (myGen !== generation) return; // event from a stale socket — drop it
    handleMessage(e);
  };
}

This kit holds up well at a few thousand concurrent connections. Past tens of thousands, the heartbeat scan needs its own optimization pass — but that's a story for another post.

The Checklist

Audit your project against this:

[ ] Client pings and waits for pong; on timeout it declares death itself, never waiting for onclose
[ ] Server tracks last-active time and reaps zombies with terminate()
[ ] Reconnects use exponential backoff + jitter + a cap
[ ] After reconnecting: delta sync, or a full reconciliation fallback
[ ] Important messages have acks; unacked messages go to a retry queue
[ ] Events from stale sockets are dropped via a generation guard

Next post: reliable message delivery — how a server guarantees a message is delivered exactly once, and how out-of-order "paranormal events" actually happen. Follow me if that sounds like your kind of problem.