We run 24/7 live streams from pre-recorded video: one playlist, sent to several platforms at once — YouTube, Twitch and Kick, often all three. Each stream is one FFmpeg process, and the fan-out goes through the tee muxer.
This post is about the failure that onfail=ignore does not cover, how to reproduce it on your laptop in two minutes, and the one-character escaping mistake that takes every output down when you apply the fix.
Everything below was measured on ffmpeg 7.1.5 in production and reproduced on 8.1 for this post.
Why tee at all
You can give ffmpeg several outputs instead:
ffmpeg -i in.mp4 ... -f flv rtmp://a/... -f flv rtmp://b/...
But then every output runs its own encoders. We copy the video and re-encode the audio to AAC, and with separate outputs the AAC encoder runs once per destination — it cost us 1.9× the CPU. tee encodes once and hands the same packets to every muxer:
ffmpeg -re -i in.mp4 -c:v copy -c:a aac -map 0:v -map 0:a \
-f tee "[f=flv:onfail=ignore]rtmp://a/app/KEY1|[f=flv:onfail=ignore]rtmp://b/app/KEY2"
Measured with video copied and audio re-encoded, each extra destination costs about two points of a core: 9.7% for one, 11.9% for two, 13.0% for three.
onfail=ignore handles a destination that fails
The documented default for a tee slave is onfail=abort: one output fails, the whole process stops. With onfail=ignore, a destination that refuses the connection or drops it is removed, and ffmpeg tells you so:
[tee @ 0x…] Slave muxer #1 failed: Connection refused, continuing with 1/2 slaves.
We set it explicitly even though our build already behaved that way without it. If you depend on a default the docs say is different, an ffmpeg upgrade can take every multi-destination stream off the air at once.
…but not a destination that stops reading
The bad case is a server that keeps the connection open and stops reading. Nothing fails. tee writes to every slave from one thread, so once the kernel's socket buffers to the stuck peer are full, that write blocks — and the healthy destinations stop getting packets too.
There is no error, no "slave failed" line, and the process is alive. From the outside, every destination still looks connected, and none of them is receiving video.
Reproduce it
A TCP peer that reads for 8 seconds, then holds the socket open and goes quiet:
# stall_peer.py — read for N seconds, then keep the socket open and stop reading
import socket, sys, time
port, read_for = int(sys.argv[1]), float(sys.argv[2])
s = socket.socket(); s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(("127.0.0.1", port)); s.listen(1)
c, _ = s.accept()
t = time.time()
while time.time() - t < read_for:
c.settimeout(0.5)
try: c.recv(65536)
except socket.timeout: pass
time.sleep(3600)
Then one healthy output (a local file) and one output to the stalling peer, in real time (-re) for 70 seconds:
python3 stall_peer.py 19351 8 &
ffmpeg -re -f lavfi -i testsrc2=size=1280x720:rate=30 -f lavfi -i sine -t 70 \
-c:v libx264 -preset ultrafast -b:v 4M -g 60 -c:a aac -map 0:v -map 1:a \
-progress progress.txt \
-f tee "[f=flv:onfail=ignore]healthy.flv|[f=flv:onfail=ignore]tcp://127.0.0.1:19351"
Watch healthy.flv and out_time in progress.txt. Results, sampled every 10 seconds:
| slave options | healthy output after 80 s | stream time reached |
|---|---|---|
onfail=ignore |
frozen at 4.6 MB | stuck at 8.5 s |
onfail=ignore:use_fifo=1 |
frozen at 4.8 MB | stuck at 9.3 s |
onfail=ignore:use_fifo=1:fifo_options=queue_size=600\\:drop_pkts_on_overflow=1 |
34.9 MB, growing all the way | 68.8 s of 70 |
In the first two runs, ffmpeg printed nothing until I killed the peer 80 seconds later; only then did Slave muxer #1 failed: Broken pipe appear. Bigger buffers only delay the freeze: in our first test of this, the stream ran 51.7 seconds before stopping dead. It still stopped.
Note the middle row: use_fifo=1 on its own does not help. It moves each slave into its own thread with its own queue, but the queue blocks when it is full, so you've only added a buffer before the same stall. The queue has to be allowed to drop.
The fix
use_fifo=1:fifo_options=queue_size=600\\:drop_pkts_on_overflow=1
Each destination gets its own thread and a queue of 600 packets — roughly ten seconds of a 1080p stream, long enough to ride out a hiccup and small enough to afford per destination on a box running twenty-odd streams. When a destination stops reading, its queue fills and starts dropping packets; the other destinations never notice.
The stalled destination gets a stream with holes in it, which is the right trade when the alternative is all of them going dark. With one destination we don't use any of this: there is nothing to protect, and blocking is better there — the stream stalls, our health check notices the frozen clock and restarts it cleanly, instead of feeding a platform a broken stream it may not recover from.
The doubled backslash
fifo_options is a dictionary inside the slave's option list, and both use : as a separator. So the inner : has to be escaped — and tee unescapes the slave options once, and the nested dictionary once more. One backslash is not enough:
# single backslash
-f tee '[f=flv:onfail=ignore:use_fifo=1:fifo_options=queue_size=600\:drop_pkts_on_overflow=1]a.flv|…'
[flv @ 0x…] Unknown option 'drop_pkts_on_overflow'
[tee @ 0x…] Slave muxer #0 failed: Invalid argument, continuing with 1/2 slaves.
[tee @ 0x…] All tee outputs failed.
Exit code 234, and both files stop at 383 bytes — the header. The single-escaped drop_pkts_on_overflow escapes the fifo dictionary and gets handed to the flv muxer, which rejects it, and onfail=ignore cannot save you because every slave carries the same bad option. With \\ the same command writes both files to the end.
In Python, build the argument as a raw string and pass it to the process without a shell, so nothing else gets a chance to eat a backslash:
FLV = ["f=flv", "flvflags=+no_duration_filesize", "onfail=ignore"]
FIFO = ["use_fifo=1", r"fifo_options=queue_size=600\\:drop_pkts_on_overflow=1"]
def tee_spec(urls):
opts = ":".join(FLV + (FIFO if len(urls) > 1 else []))
return "|".join(f"[{opts}]{u}" for u in urls)
# asyncio.create_subprocess_exec("ffmpeg", ..., "-f", "tee", tee_spec(urls))
A user-supplied URL is part of the tee syntax
The whole spec is one string shaped [opts]url1|[opts]url2. A | inside a URL starts another slave:
ffmpeg -f lavfi -i testsrc -t 1 -c:v libx264 -map 0:v -f tee '[f=flv:onfail=ignore]x1.flv|x2.flv'
# writes x1.flv AND x2.flv
If your users paste their own ingest URLs, that is a way to push to a destination you never counted or reported on. Reject |, [, ], \ and whitespace in destination URLs before they reach the spec — and accept only rtmp:// and rtmps://.
Knowing which destination died
tee gives no per-destination byte counts (total_size=N/A in -progress), so the log lines are the signal. These are the ones worth parsing:
[tee @ 0x…] Slave muxer #1 failed: Connection refused, continuing with 1/2 slaves.
[fifo @ 0x…] Error opening rtmp://…: Input/output error
[tee @ 0x…] All tee outputs failed.
Mask the stream key before you log any of them: the address in these lines contains it.
I run TheLoops, a service that streams pre-recorded video 24/7 to YouTube, Twitch and Kick from the cloud. Every multi-destination stream it runs goes through the spec above. Questions about ffmpeg fan-out are welcome in the comments.
Top comments (0)