📝 Originally published (in Japanese) at forge.workstyle.tech.
Building an AI Avatar Live Streaming System That Runs Unattended
When building an AI avatar live streaming system that runs unattended, the development focus shifts from a typical application. If the system crashes during unmonitored hours, no one is there to fix it. This means that instead of just "working well," the key to quality is making sure the system "understands when it breaks and can recover on its own."
This article summarizes the pitfalls encountered while getting such a system to run continuously on Twitch, broken down into three layers:
- Authentication Layer: OAuth for EventSub and metadata manipulation. The constraints of manual human steps and tokens that aren’t immutable.
-
Transmission Layer: Even when
ffmpegis sending video, the channel doesn’t go live. Twitch silently accepts invalid keys. - Fault Tolerance Layer: There are three ways the server can die during a stream, one of which leaves the video running silently.
All of these issues share a common pattern: "Everything looks normal from the sender’s side." This makes them especially tricky. Let’s go through them one by one.
Where OAuth Actually Becomes Necessary
First, to just read chat, an anonymous IRC connection is sufficient—no app registration required. That part was straightforward.
OAuth became necessary when we wanted to do two things:
- Use EventSub to receive bits (cheers), subscriptions, and raids, and have the avatar react.
- Use the Helix API to automatically set the stream title per episode.
Only these two operations require authorization. The overall flow looks like this:
1. Register an app in the Developer Console (Confidential type)
2. Obtain client_id / client_secret
3. Open the authorization URL in a browser and authorize as the channel owner
4. Extract the authorization code (code) from the redirect URL
5. Exchange the code for an access token and refresh token via the token endpoint
6. Save the refresh token on the server
Only steps 3 and 4 require human interaction; the rest can be automated. Below are the specific pain points we encountered during this process.
App Type and Redirect URI
When registering the app, you must select a type. If your server holds the client_secret and performs token exchange, choose Confidential. Choosing the wrong type later means the secret can’t be used for token exchange, and you’ll have to recreate the app.
The redirect URI is only needed to complete authorization, so something like http://localhost:3000 is fine. You don’t even need to run a server to receive it—just copy the code from the address bar after authorization.
Request All Required Scopes in a Single Authorization
If scopes are missing, you’ll have to restart the authorization process. The three scopes we needed were:
| Scope | Purpose |
|---|---|
bits:read |
Subscribe to bits (cheers) events |
channel:read:subscriptions |
Subscribe to subscription events |
channel:manage:broadcast |
Set stream metadata like the title |
Adding scopes later requires the user to open a browser again. List all required functionality upfront and request all necessary scopes in one authorization.
Use force_verify to Avoid Authorizing the Wrong Account
We added force_verify=true to the authorization URL.
Without it, if you’re already logged into the browser, authorization might complete without a confirmation screen, especially if you have both a personal account and a character account. This can lead to accidentally authorizing the wrong account. With force_verify=true, a confirmation screen always appears, making it clear which account you’re authorizing.
Authorization Codes Expire in Minutes
This was the most nerve-wracking issue.
Authorization codes (code) expire in just a few minutes. If the flow involves human steps—opening a browser, copying the code, passing it to the server, and exchanging it—the code may already be dead by the time you receive it.
The fix is simple: prepare the token exchange process in advance and execute it the moment the code arrives. We pre-assembled the exchange command and waited. As soon as the code came in, we executed it immediately, and it worked on the first try.
After Obtaining Credentials, Verify Who You’re Authorized As
Once you get the token, hit the validation endpoint (/oauth2/validate) to inspect its contents.
curl -H "Authorization: OAuth <access_token>" https://id.twitch.tv/oauth2/validate
Check the returned login (username) and the list of scopes. Are you authorized as the expected channel owner? Are all required scopes present?
Skipping this step makes it impossible to distinguish between "insufficient scopes," "wrong account," and "implementation error" when EventSub subscriptions fail. Immediately after obtaining credentials, verify what they represent.
Operational Pitfall: Refresh Tokens Can Be Replaced
This issue became apparent only after implementation and is worth sharing.
Twitch may replace the refresh token itself when updating tokens.
A naive implementation looks like this:
- On startup, read the saved refresh token
- Use it to refresh the access token
- A new refresh token is returned
- Hold it in process memory
- On restart → back to step 1, reading the old refresh token
If the old token is still valid, it works. But if it’s invalidated, authentication fails. Worse, the failure doesn’t appear until the process restarts, making the root cause hard to diagnose.
The correct approach is to persist the new refresh token every time it’s updated. This issue appears in other services too, so it’s worth confirming whether you’re assuming “refresh tokens are immutable.”
Plan Ahead for Where Sensitive Data Resides
A quick operational note. Avoid pasting client secrets or refresh tokens into chats or tickets. We used a file-based approach, writing to a secure location and deleting it afterward.
touch ~/.twitch-cred && chmod 600 ~/.twitch-cred
# Write values
# After use
shred -u ~/.twitch-cred
In a previous project, a stream key accidentally ended up in logs, forcing us to reset it. Plan ahead for where secrets end up.
“ffmpeg Is Running” ≠ “Stream Is Live”
Once authentication is sorted, the next step is sending video to Twitch. Here we hit a situation where everything looked normal on the sender side, yet the channel never went live.
-
ffmpegwas running and sending frames continuously - No errors in stderr
- RTMP connection was maintained (no disconnections or reconnections)
- Yet the channel remained offline
Where to even start debugging? The root cause boiled down to two issues:
1. Twitch’s RTMP ingest accepts invalid stream keys and silently discards the data. From the sender’s side, success is indistinguishable.
2. If the stream itself is malformed, Twitch won’t mark the channel as live. Even the dashboard’s Stream Inspector shows nothing.
A Persistent Connection Doesn’t Mean the Key Is Valid
A common misconception with RTMP is that “if the connection is established and maintained, the key is valid.” That’s not true. The discrepancy between what we saw and Twitch’s state was:
| What We Saw | State |
|---|---|
| ffmpeg logs | Normal. Frames being sent |
| TCP connection | Established & maintained |
| Twitch channel | Still offline |
In our case, the cause was account-side settings (2FA and stream key status). Once fixed, running the exact same command made the channel go live. We hadn’t changed anything on the sender side.
The operational rule we derived is simple:
Determine streaming success not by sender-side logs, but by receiver-side state.
“Malformed Streams” Also Won’t Go Live
Another issue we encountered. At the time, we were generating video via CPU rendering. When we checked the recording, only 6 seconds of video were saved out of 90 seconds (due to an audio track exhaustion bug).
Sending this timeline-dropped stream to Twitch resulted in a connection being established, but the channel never going live. The Stream Inspector showed no information. From Twitch’s perspective, it received a stream with timestamps far behind real time, making it impossible to treat as a valid broadcast.
Again, from the sender’s side, it “looked fine.” Just because ffmpeg is running doesn’t mean a valid stream is being sent.
How We Built External Monitoring
We incorporated a mechanism to fetch Twitch’s state externally as part of our validation process. We used a public endpoint that returns the channel’s uptime.
curl -s "https://decapi.me/twitch/uptime/<channel_name>"
# Live: "49 seconds"
# Offline: "<channel_name> is offline"
This allowed us to mechanically determine whether the channel was actually live. We changed our acceptance criteria for stream tests from sender-side logs to this output.
However, there’s a trap. Third-party APIs like this often cache responses. If you check immediately after starting a stream, you might still get offline, leading to a false negative and wasted debugging effort.
# Single checks are unreliable. Poll instead.
for i in $(seq 1 20); do
curl -s "https://decapi.me/twitch/uptime/<channel_name>"
echo
sleep 15
done
The same applies to stopping. After stopping the stream, offline isn’t immediate. For both start and stop, poll until the state stabilizes.
“Sent” ≠ “Delivered” — Separate Checks
This isn’t specific to Twitch. The same pattern appears in many places:
- Email delivery: SMTP returns 250, but the message may not reach the inbox
- Webhooks: 200 OK doesn’t guarantee the receiver processed it successfully
- Metrics submission: The agent may send data, but it might not appear in the dashboard
In all cases, there’s a gap between “sender-side success” and “receiver-side state.” Relying only on one side leads to silently broken states. After this incident, our streaming validation checklist always includes a line: “External confirmation of receiver state.” Silence is not a synonym for success.
Three Ways to Crash, One of Which Keeps Running Silently
Once streaming works, the next challenge is surviving unattended operation. How the system behaves when it crashes determines almost the entire quality. We identified three failure paths, each with different detection and recovery methods—and the third was the most dangerous.
| # | Failure Mode | Symptoms | Recovery |
|---|---|---|---|
| 1 | GPU host instability | Renderer recovers every few dozen seconds to minutes | Restart on a different host |
| 2 | Pod disappears | Video feed completely drops | Restart on a different host |
| 3 | Only the interaction server dies | Video continues, but avatar stays silent | 3-stage chain recovery |
Paths 1 & 2: Don’t Fix the Instance, Restart It
Paths 1 and 2 were handled together. Instead of trying to fix the broken instance, we discard it and restart.
- Renderer side: If recovery attempts exceed a threshold within a window, the process terminates itself
- Scheduler side: Monitor Pod state; on termination/disappearance, restart on a different host (with limits)
“Killing yourself” feels counterintuitive, but it’s the most reliable way to propagate failure upward. We verified this by manually deleting a Pod mid-stream. Detection happened in 32 seconds, and streaming resumed on a different host.
Path 3: The Most Dangerous — “Silent Continuation”
This was the real problem. When only the interaction server (the speech generation side) restarts, the renderer knows nothing. Video continues flowing. The page stays alive. The avatar is on screen, blinking. It just stops talking.
From the viewer’s perspective: “The stream continues, but the AI has gone silent.” From a monitoring standpoint, this is catastrophic:
- Stream remains live (platform considers it normal)
- Renderer is operating normally (process and
ffmpegare alive) - Video and audio tracks are flowing (though silent)
None of the health checks catch this. Even our external monitoring doesn’t detect it—the channel is still marked as live.
The root cause was that death wasn’t propagated through the layers. The architecture was multi-tiered:
Interaction Server ←WebSocket→ Audio Pipeline ←WebRTC→ Page
When the WebSocket to the interaction server died, the audio pipeline didn’t forward the disconnection downstream. The WebRTC connection remained alive, so from the page’s perspective, it was “connected but receiving nothing.” The disconnection was absorbed mid-path and never reached the end. That was the true issue.
Solution: Propagate Death in Three Stages
We made death propagate reliably to the end:
- The audio pipeline detects the WebSocket death to the interaction server and actively closes its WebRTC connection
- The page detects the WebRTC disconnection, waits briefly, and reloads
- If the first reconnection fails (e.g., server still restarting), retry with increasing intervals
Step 3 was subtle but critical. Without it, if the reload happened while the server was still restarting, the connection would fail and hang. Recovery logic must account for recovery timing.
Measurement: Actually Restart and Measure
We performed a rolling restart of the interaction server and measured recovery time.
Server restart
→ Detect WebSocket death
→ Actively close WebRTC
→ Page reload
→ Reconnect (retry if server still starting)
→ Restore state from snapshot
→ Resume speech
Total: 41 seconds
Including state restoration, it took 41 seconds to resume seamlessly from where it left off, without re-greeting the audience.
Takeaways for Fault Tolerance Design
1. Design for “silent continuation” as the worst-case scenario. A process dying is detectable and relatively safe. The danger lies in appearing alive while not functioning. In our case: video streaming but no speech. Adding conditions like “if I detect I’m not functioning, terminate honestly” across layers improved overall stability. Trying to survive at the cost of availability actually reduces availability.
2. Disconnections must propagate to the end. In multi-tier systems, intermediate layers can absorb upstream deaths. Each connection must define: “When the upstream dies, what does this layer do?” The default behavior is often “do nothing.”
3. Recovery logic must work even during recovery. Code like “reconnect because the server restarted” often fails to account for the server still being down. Retries are part of recovery logic.
4. Fault-tolerant code without fault injection is likely broken. We manually crashed all three paths: deleted Pods, restarted servers. Just writing recovery code rarely works end-to-end. The need for “initial connection retry” only became clear after actual failure injection.
Summary
Across all three layers, one principle held true:
“Information visible from the sender side does not guarantee the receiver’s state.”
Authentication Layer
- Reading chat only requires anonymous IRC. OAuth is needed only for EventSub and metadata manipulation.
- Use Confidential apps; redirect URIs can be localhost. Request all scopes in one authorization.
- Use
force_verify=trueto prevent accidental authorization with the wrong account. - Authorization codes expire in minutes. Prepare the exchange process in advance.
- After obtaining tokens, validate with
/oauth2/validateto confirm authorization scope and account. - Refresh tokens may be replaced on update. Persist the new token immediately; otherwise, crashes only appear after restart.
Transmission Layer
- Twitch accepts invalid stream keys via RTMP and silently discards data—success is indistinguishable from the sender.
- Malformed streams (e.g., dropped timelines) also won’t go live; the Stream Inspector shows nothing.
- Determine streaming success not by sender logs, but by external confirmation of receiver state.
- Third-party uptime endpoints cache responses—use polling, not single checks (for both start and stop).
Fault Tolerance Layer
- Three ways a server can crash during a stream: unstable host / Pod deletion / only interaction server dies.
- Paths 1 & 2: discard and restart. Measured 32 seconds to detect and recover.
- Path 3: video continues but avatar is silent—undetected by any health check. Root cause: disconnection absorbed mid-path. With death propagation, measured 41 seconds to full recovery.
OAuth procedures and RTMP specs are documented, but the real pitfalls lie in:
- Time constraints in human-involved steps
- Tokens and streams not being immutable
- States that appear alive but aren’t functioning
For unattended operation, aiming for a system that "understands when it breaks and recovers automatically" is far more practical than trying to build one that never breaks.
Top comments (0)