📝 Originally published (in Japanese) at forge.workstyle.tech.
Building a Fully Automated AI Avatar Streaming System on YouTube and Twitch
I built a system that lets AI avatars stream continuously on YouTube and Twitch without human intervention. Once you register a show, the system automatically creates a stream at the scheduled time, boots up the GPU, responds to viewer comments with voice, and wraps up when the time is up—all without any human involvement on the day of the broadcast.
One question kept nagging me throughout development: "What should I test to ensure quality?" The output changes every time. Correctness is subjective. And the most troublesome part? Even when it fails, the stream continues. No one notices because no one is watching.
This article documents how we shifted our approach to "what we guarantee" and how we tackled the bugs that only surface after long-running tests. The first half covers the QA design philosophy, and the second half dives into the practical side of soak testing. I’ve structured it for developers, walking through the stumbling blocks → root causes → solutions.
Three Assumptions That Traditional Testing Relies On
Writing this made me realize that the implicit assumptions behind traditional testing don’t hold at all in an unmanned AI streaming system.
| Assumption | In Unmanned AI Streaming |
|---|---|
| Input is fixed | Viewer input is unpredictable |
| Output can be judged | LLM output varies each time; no correct string exists |
| Failure stops the system | Failure doesn’t stop the system. The stream continues in silence |
The third one was the most troublesome. In normal systems, when something breaks, an exception is thrown, requests fail, and someone notices. But streams keep going even when broken. The video is still playing. The process is alive. Health checks are green. The avatar just stops talking.
This actually happened. When only the dialogue server restarted mid-stream, an upstream layer failed to propagate the disconnection downstream, resulting in what viewers saw as "an AI stream that fell silent." None of the monitoring metrics caught it. This "alive but not functioning" state appears repeatedly in this article—it’s the worst-case scenario in unmanned operations.
Shifting What We Guarantee
So we shifted our focus from output correctness to system behavior. We can’t guarantee that the avatar says the right thing, but we can guarantee that the system behaves as expected.
We narrowed it down to four concrete guarantees.
1. Stop When It Should Stop
In automated systems, what’s scarier than failure is continued success. Since no one is watching, as long as it keeps running, no one notices. And GPU costs keep piling up.
We explicitly implemented stop conditions:
- When the show’s runtime ends, the avatar delivers closing remarks and exits
- If the stream doesn’t go live within a certain time, we treat it as a failure and destroy the Pod
- If the renderer repeatedly recovers in a short time, it terminates itself
The third one is "dying on purpose"—it feels counterintuitive, but it turned out to be the most reliable way to propagate failure upward.
2. Notice When It Breaks
We defined "alive but not functioning" as the worst-case state.
Then, we explicitly defined how each layer should behave when its upstream dies. The default behavior is usually "do nothing." When layers do nothing, disconnections get absorbed and disappear. The earlier "silent stream" incident was exactly this.
- When detecting an upstream disconnection, also cut downstream connections
- The disconnected side reloads and reconnects
- Even if reconnection fails, keep retrying until the upstream recovers
Propagate death. Layers that try to stay alive to preserve availability were actually reducing overall availability.
3. Recover When It Breaks (and Measure Recovery)
Recovery code doesn’t work just by writing it. We actually broke things and measured the results.
| Break Method | Recovery |
|---|---|
| Manually delete GPU Pod mid-stream | Detected in 32 seconds, resumed on another host |
| Rolling restart of dialogue server | Fully recovered in 41 seconds (including state restoration) |
Injecting failures takes just a few minutes each. Recovery code you write without testing usually stops somewhere. We only realized "if there’s no retry logic, it stops when hit during restart" after actually breaking things. The "add retry logic" fix came from real measurements, not desk work.
4. Don’t Output What You Shouldn’t
While we can’t judge the output itself, we can control output boundaries.
- Filter NG words and spam at the chat entry point (stop dirty input at the gate)
- Declare AI-generated content to the platform
- During verification, keep streams unlisted and only switch to public on human approval
The mindset shift: instead of controlling what it says, control what gets in and how it’s delivered.
Leaving Places for Humans to Look
Not everything can be judged mechanically. In fact, we encountered a bug where all performance metrics were normal, but the video was broken. The character’s face flickered, yet FPS, errors, and GPU usage were all fine.
A human eye caught it. Now, any changes to rendering or audio go through human review before being finalized.
The more automated checks you add, the more consciously you need to leave places for humans to look. A dashboard full of green doesn’t mean everything is correct.
Some Bugs Only Appear After Long Runs
Even after passing all our short tests, we still weren’t done. Things break after 2 hours. A 10-minute test might pass, but run it for the length of a real show, and new issues surface.
When we listed the bugs found only during long runs, they neatly fell into four categories. None of these can be detected in short tests. The duration of the run itself becomes a test condition.
| Type | Example | Why It Doesn’t Appear in Short Runs |
|---|---|---|
| Cumulative | Timer drift accumulates, causing lip-sync drift | Small error per instance; grows with time |
| Increase | History, memory, or state grows without bound | Takes time to hit limits |
| Cycle | Starts repeating the same topics | Doesn’t occur until buffer completes a cycle |
| Restart | Updated tokens are lost on restart | Symptoms don’t appear while process is alive |
Below, we’ll go through each type, one by one, covering causes and detection tips.
Type 1: Cumulative — Small Error × Count
We were using setInterval to export frames at fixed intervals. Timer firings can lag, and lag isn’t recovered.
Even 1ms of lag per frame adds up: at 30fps, that’s 1.8 seconds of missing frames per minute. Audio plays in real time, so the video falls behind. Starts in sync, drifts apart over time.
Detection tip: Explicitly check "does it get worse over time?" Take the same measurement at start and near the end. If you feel "it keeps happening no matter how you tweak it," it’s likely cumulative.
Type 2: Increase — Measure Whether There’s an Upper Limit
We were holding streaming state (conversation history, per-viewer memory, etc.). We intended to keep only recent data, but design intent ≠ implementation.
After running for 2 hours, here’s what we found:
| Item | Result |
|---|---|
| State snapshot size | Capped at 6.2KB |
| Process memory | No increase |
Confirming the cap was crucial. If it kept growing, it would eventually crash. Designing an upper limit and verifying it with real measurements.
Detection tip: Log sizes or memory periodically and watch the graph—does it flatten or keep rising? You can’t tell the difference in 10 minutes.
Type 3: Cycle — Buffer Completes One Full Turn
We added a feature that generates topics when no new comments arrive. It avoids repeating recent topics by checking conversation history.
The problem? The amount of history kept directly becomes the cycle period. Once a topic falls out of history, it’s treated as fresh again. Viewers see "you just said that."
This never appears in 10-minute tests—history hasn’t cycled yet.
Detection tip: If you have a design value like "keep N items" or "keep for N minutes," run longer than N. Ring buffers, caches, deduplication, rate limits—same pattern appears everywhere.
Type 4: Restart — Normal While Process Is Alive
When refreshing tokens from an external service, sometimes a new refresh token is returned. If we only hold it in process memory, at restart it reverts to the old value.
If the old token is still valid, it works. If invalidated, authentication fails. The process appears perfectly normal while running. Soak tests won’t catch it. Continuous uptime and restart resilience are separate tests.
Detection tip: Intentionally restart during the soak test.
End-of-Show and Cleanup
Beyond the four types, long-running tests revealed two more categories.
End-of-show processing. Closing remarks, cleanup, resource release—these only run in the final minute. Tests stopped midway never execute this code. Even if you implement "stop when it should stop," you must run it to the end to confirm the shutdown logic works.
Cleanup. Every time, we manually verify that GPU Pods are actually at zero after shutdown. If not, billing continues. We don’t trust "it should have stopped." We confirm.
The Worst Bug Found During Soak Testing
The most critical bug found during a 2-hour soak was the one mentioned at the start: when only the dialogue server died, the stream continued in silence.
- Video was playing
- Process was alive
- All health checks were green
- The avatar never spoke a word
This is even more dangerous than the four types because the failure isn’t observed. We would have only encountered this in production if we hadn’t run long tests. The goal of "notice when it breaks" was only concretized because we observed this behavior during long runs.
Soak Test Observation Checklist
Here’s what we actually monitor during long runs. Use it as a checklist for your own soak tests.
- Memory / state size trends (does it flatten or keep rising?)
- Time drift (take the same measurement at start and near end)
- Output duration vs real time (any dropped frames?)
- Repetition of behavior (same topics, response patterns)
- Restarts during the run and recovery time
- End-of-show processing actually executes
- Cleanup (are all resources truly zero?)
Summary
QA Design:
- In unmanned AI systems, the assumptions behind testing—fixed input, judgeable output, failure stops the system—all break down
- Shift focus from "output correctness" to "system behavior"
- Concretely, four guarantees:
- Stop when it should stop
- Notice when it breaks
- Recover when it breaks
- Don’t output what you shouldn’t
- Define "alive but not functioning" as the worst-case state
- Recovery must be tested by actually breaking things (32s Pod deletion, 41s dialogue server restart). Code you write without testing usually fails somewhere
- The more you automate checks, the more intentionally you must leave places for humans to look
Soak Test Practices:
- Long-running bugs fall into four types: Cumulative, Increase, Cycle, Restart
- Cumulative → check if it gets worse over time
- Increase → verify upper limits with real measurements (state capped at 6.2KB)
- Cycle → run longer than the design value (N items kept)
- Restart → intentionally restart during the soak to expose issues
- End-of-show and cleanup only execute when run to completion
- "It ran for 10 minutes, so it’s fine" means you haven’t tested any of the four types
It’s hard to answer "how do we guarantee AI output quality?" head-on, but we could guarantee that "even if the AI says something weird, the system doesn’t break." In unmanned operations, the latter is far more important. And there are so many bugs that only surface when you run longer.
Top comments (0)