I Created a 24/7 AI Avatar That Streams Without Human Intervention — Only 'Verification,' 'Eyes,' and 'Ears' Remain for Humans

📝 Originally published (in Japanese) at forge.workstyle.tech.

番組の時間になる → Let me hand you the finished piece.

---

I built a system where a 3D avatar live-streams entirely on its own. The only thing a human does is register the show. On the day itself, nobody touches anything.

Showtime arrives
  → the broadcast is created automatically (YouTube)
  → a cloud GPU Pod spins up
  → the avatar renders in a page and starts pushing over RTMP
  → the broadcast transitions to live
  → it responds to viewer comments with speech
  → it reacts to tips based on the amount
  → when comments dry up, it raises its own topics from the show's theme
  → when the runtime is up, it gives a closing greeting
  → the stream ends, the Pod is destroyed, the archive remains

I worked through this system with an AI agent over several weeks — design, implementation, deployment, fault injection, and long-running verification. The first half of this post covers the overall architecture and why I split it the way I did. The second half covers what became visible through the process: the roles that were left to humans and only humans. The further you push toward unattended operation, the more sharply the parts you can never hand off come into focus.

The big picture

┌────────────────────────┐
│   Show Management UI   │  register a show (character / theme / runtime / destinations)
└──────────┬─────────────┘
           │
┌──────────▼─────────────┐   create / health-check / destroy Pods
│       Scheduler        │───────────────────────────┐
│  (stream lifecycle)    │                           │  create / go live / end broadcasts
└──────────┬─────────────┘                           │
           │ run (one stream execution)              │
┌──────────▼─────────────┐               ┌───────────▼────────────┐
│    Dialogue Server     │◀── Redis ─────│     Chat Collector     │
│ (utterance generation) │    Stream     │      (YT / Twitch)     │
└──────────┬─────────────┘               └────────────────────────┘
           │ WebSocket (display only)
┌──────────▼─────────────┐
│     Renderer Pod       │  headless Chromium + ffmpeg
│         (GPU)          │──▶ tee ──▶ YouTube / Twitch
└────────────────────────┘

Why I split it this way

1. I gave the "stream lifecycle" its own layer

This was the very first decision. A stream begins, continues, and ends — it has a lifespan. And if it doesn't end, the billing doesn't either.

If you bury that lifespan management inside the dialogue logic, a bug in the dialogue turns directly into runaway costs. By splitting it out, the scheduler only has to watch one thing: is the stream alive, has it ended, is the Pod gone?

That's the scheduler's entire set of responsibilities:

• When the time comes, create a run (one stream execution)
• Create the broadcast and transition it to live
• Start the GPU Pod, monitor its health, and re-acquire on a different host if it dies
• When the runtime is up, move to the closing, end the stream, and destroy the Pod
• If it doesn't go live within a fixed window, mark it failed and clean up

Without those last two, you end up with GPU charges piling up while no video ever appears.

2. I decoupled chat collection from dialogue

YouTube comments come from polling the Data API. Twitch comes over IRC. Tips and subs come from EventSub. Every source works differently.

If you wire all of that straight into the dialogue server, the dialogue logic gets dirtier every time you add a platform. So I put a dedicated collection component in front and normalized everything into a common internal format before pushing it onto a Redis Stream.

YouTube chat    ┐
Twitch IRC      ├→ normalize → Redis Stream → Dialogue Server
Twitch EventSub ┘

From the dialogue server's point of view, there's exactly one input: a stream of events. No platform-specific knowledge has to live there.

Moderation (banned words, throttling repeat spam from the same user) also lives on the collection side. Dropping the dirty stuff at the entrance keeps every layer behind it simpler.

3. Speech runs on a single serial loop

Conversation during a stream behaves nothing like web request handling. You can't say two things at once.

Even when several comments arrive simultaneously, the avatar has one mouth. So speech is fully serialized, with a priority queue deciding the order.

• Events like tips and subs → high priority
• Regular comments → medium priority
• Self-initiated topics during silence → low priority
• The closing greeting at the end of the show → highest priority

When comments flood in, there's also logic to batch several of them into a single utterance. Answering one at a time makes it look, from the viewer's side, like the avatar is still stuck on a comment from ages ago.

4. State lives in Redis, and the page is disposable

Renderers go down. GPU hosts get flaky, browsers crash, ffmpeg dies.

So I designed it so that the stream's state (what has been said, who it talked to) never lives in the page. State sits in a Redis snapshot, and the page is a display-only client that connects and receives the current state.

That means if the page dies, reloading and reconnecting picks up right where it left off. It also avoids the dumb failure mode of re-doing the opening greeting on every reconnect — the snapshot already says "greeting done."

5. I physically separated video generation from dialogue

The renderer runs on a different Pod, a different cloud, a different GPU. Its whole job is: open a page, capture the screen and audio, push it to RTMP.

That separation gives you:

• The renderer is disposable (bad host? re-roll it)
• The dialogue server can sit close to our own inference infrastructure (the LLM and TTS live there)
• Co-tenancy falls out naturally (several streams on a single GPU)

Put the fragile parts where breaking doesn't hurt.

Measured numbers

For reference, a few numbers measured during verification.

Metric	Value
Pod creation → live transition	95 s
Concurrent streams per GPU	4 avatars (720p30)
Cost per avatar	¥7,600/month (24h) / ¥2,500/month (8h per day)
Comment response latency	25–45 s (15–30 s of which is platform viewer delay)
Automatic recovery from failure	41 s (full recovery from a server restart)
Continuous operation test	2 hours (no memory growth, all 53 turns measured)

The recovery numbers came from actually deleting Pods and restarting servers. Verification included actually breaking things and confirming they heal.

The hardest part was "ending reliably"

The technically hardest part turned out to be neither the dialogue nor the rendering — it was ending reliably.

Failures to start are obvious immediately, because nobody can watch. Failures to end announce themselves via the invoice, or via "wait, it's still streaming" the next morning. The scary thing about automation isn't failure — it's success that keeps going. That's what hit home hardest.

I ended up giving every component a condition along the lines of "if I decide I'm in a bad state, I stop."

---

That's the architecture. From here I want to talk about how the work itself went. I did all of this alongside an AI agent, and what I noticed partway through was that the moments a human got called in fell into exactly three categories. Nearly everything else closed out on the agent's side.

What the agent handled end to end

First, the work no human touched:

• Design and spec documentation
• Implementation (backend, scheduler, renderer, admin UI)
• Container builds and deployment
• Fault injection (killing Pods, restarting servers) and measuring recovery times
• Measurements (avatars per GPU, cost, latency breakdown)
• Log investigation and root-cause analysis
• Record-keeping and handoff notes

The measured numbers above, including the 41-second recovery, came out of that process. The range of what actually gets done is far wider than you'd expect.

What only a human could do, part 1: operations that demand proof of identity

This was by far the largest bucket.

• Creating accounts on streaming platforms
• Phone-number identity verification (and the 24-hour wait before live streaming is enabled)
• Setting up two-factor authentication
• Clicking the button on an OAuth consent screen
• Registering payment methods and adding credit on cloud services
• Registering developer applications

What these share is that they all demand proof that you are you. This isn't a matter of technical difficulty — it's a domain where delegation isn't supposed to be possible. If an agent could do these on your behalf, that service's identity verification would be broken.

The practically important part is that these create waiting. Twenty-four hours from phone verification to activation isn't something code can shorten.

During development, I kept a running homework list for the human. Separate "what the agent can move on right now" from "what can't proceed until a human does it," and get the homework done first. Skip that and you'll finish the implementation only to sit through a 24-hour wait. And in fact, I punted on homework a few times and lost a full day to "waiting on authorization."

What only a human could do, part 2: quality judgments that require perception

The second bucket: things you can only judge by looking and listening.

Judgment	Why a machine can't settle it
Setting lip-sync delay to 0.10 s	"Looks in sync" is a perceptual question. There's no correct number
Whether render flicker is acceptable	It shows up in neither fps, nor errors, nor GPU utilization
Whether a voice sounds natural	Waveform metrics don't line up with subjective impressions
The overall "watchable quality" of a stream	A holistic call

The most telling case was render flicker. Every performance metric stayed normal while the character's face strobed. fps, errors, GPU utilization — not one of them indicated anything wrong. Changing a graphics setting fixed it, but the only reason anyone noticed it was broken is that a human watched the video.

That changed how I ran things: any change touching rendering or audio has to be confirmed by a human seeing and hearing the real thing before it's finalized. This is less a limitation of AI capability than a property of the problem — the criterion for the judgment exists only inside human perception. I wrote earlier that "the renderer is disposable," but the step that finally signs off on its quality stayed with human eyes and ears.

What only a human could do, part 3: the decision to publish

The third bucket: decisions about going public.

• Whether to keep a stream unlisted or make it public
• How to disclose that the content is AI-generated
• What the character is allowed to say

Technically, all of these can be executed at any time. Whether they should be is a separate question. Judgments where accountability sits with a human stay with the human — as a matter of authority, not capability.

I drew the line from the start: verify with unlisted streams, and switch to public only on a human's call. The agent operates on the assumption of that line too.

Draw the boundary first, and development stops stalling

To summarize, three things stayed with the human:

1. Operations that demand proof of identity (accounts, authorization, payment)
2. Quality judgments that require perception (seeing, hearing)
3. The decision to publish (judgments that carry accountability)

Put the other way around: everything else runs on the agent's side. Design, implementation, and deployment, plus fault injection, measurement, and root-cause analysis.

The thing that paid off most in practice was identifying this boundary at the very start of the project. Hand the "human homework" over early and there's no wait left when the implementation lands.

The other thing I noticed is that all three share a property: you could do them on someone's behalf, but you shouldn't. These aren't technical limits. That's exactly why I don't expect the boundary to move much as capabilities improve.

Wrapping up

On architecture:

• Give stream lifespan management its own layer. So a dialogue bug never becomes runaway billing
• Confine platform differences to the event collection layer. Dialogue sees exactly one normalized stream
• Speech is serial + a priority queue. There's only one mouth
• Keep state outside (Redis) and make the page disposable. It can crash and recover
• Put the fragile part (the renderer) where breaking doesn't hurt
• The hardest thing about automated streaming is ending reliably

On process:

• Design, implementation, deployment, fault injection, measurement, and root-cause analysis all closed out on the agent's side
• Three things stayed with the human: identity-bound operations / perceptual quality judgments / the decision to publish
• Identity verification and authorization create waiting. Hand them over as homework early or development stalls
• Rendering and audio quality can be broken while every performance metric reads normal. Keep a step where a human sees and hears the real thing
• These three aren't capability limits — they're things that shouldn't be delegated. Which is why they'll stick around

Unattended streaming lets you hand off almost every step to the machine. GPU selection, browser media APIs, the quirks of platform APIs, fault-tolerant design — each one has enough traps for its own article, and all of it got handled on the agent's side. What was left standing at the end was: being who you say you are, judging with your own eyes and ears, and carrying the responsibility of publishing. My conclusion is that rather than hunting for "what AI can't do," it's practically faster to decide up front what the human should do.