"Real time" is a promise, not a number. When a system claims to generate live sports highlights in real time, the only question that matters is how many seconds pass between the moment happening on the pitch and a finished clip being ready to publish. Getting that number down is a budgeting exercise: every stage of the pipeline spends some of it, and you cannot fix what you have not measured. Here is where the seconds actually go.
The five stages that spend the budget
A live highlight pipeline has five stages, and each one adds delay: ingest (receiving the RTMP or HLS stream and decoding it), detection (deciding that a key moment has just happened), assembly (choosing the clip boundaries, the build-up and the reaction), reframing (converting the wide broadcast shot to vertical and square for social), and delivery (encoding and handing the clip to whoever publishes it). Add them up and you have your end-to-end latency. Most teams only ever measure the middle stage.
Ingest: the cost you do not control
HLS is segment based. If the source publishes six-second segments, you cannot see a moment until its segment has been written, uploaded and fetched, so the stream itself can put several seconds on the clock before any model runs. RTMP is lower latency but less common from broadcast sources. The practical rule: measure the glass-to-ingest delay of each feed separately, because two feeds of the same match can differ by ten seconds, and the budget for everything downstream depends on it.
Detection: the one stage where you choose your trade-off
Detection is where the accuracy-versus-latency trade-off lives. A model that waits for a few more seconds of context (the replay, the crowd reaction, the commentary) is more accurate, but every second it waits is a second added to the budget. Fusing visual and audio signals helps because the audio often leads the picture: the crowd noise for a goal peaks before the broadcast cuts to the celebration. Committing early on a strong audio signal, then confirming visually, is how you keep this stage tight without flooding the output with false positives.
Assembly and reframing: parallelize or pay twice
Assembly needs footage from before the moment (the build-up), which is already buffered, and footage from after it (the reaction), which has not arrived yet. The reaction window is a deliberate wait, so make it a configurable number of seconds per sport rather than a fixed constant. Reframing can start on the build-up frames while the reaction is still arriving. Running these stages sequentially is the most common self-inflicted delay in pipelines that otherwise perform well.
Delivery: the last, forgotten seconds
Encoding a vertical and a square version and uploading them is not free. A clip that is detected in three seconds and assembled in four can still take fifteen seconds to appear because encoding ran at the end, single threaded, on the same machine as the detector. Budget for it explicitly, and pre-encode the build-up segment while waiting for the reaction.
Where this shows up in practice
Zentag AI works from live RTMP and HLS feeds across 50+ sports, and the end-to-end number, not the detection number, is what a club or broadcaster experiences when a goal goes in and they are waiting to post. Treating latency as a per-stage budget, with each stage measured and owned, is the difference between a demo that looks real time and a pipeline that is.
Takeaway
Write the budget down, stage by stage, per feed. Measure ingest first because it sets the floor. Spend your waiting seconds deliberately in detection and the reaction window, parallelize assembly with reframing, and do not let encoding be the surprise at the end.
Top comments (0)