I Built an Open-Source AI Platform That Automatically Turns Live Streams into YouTube Shorts
What if you could monitor multiple Twitch and YouTube Live streams, automatically detect exciting moments, turn them into short-form videos, and prepare them for publishing — without manually watching hours of footage?
That's the problem I wanted to solve with ClippingShorts, an open-source, self-hosted AI video clipping platform.
GitHub: https://github.com/saifk6717/ClippingShorts
The idea goes beyond building another video cutter. I wanted to build an end-to-end system that can monitor live streams, identify potentially interesting moments using audio and chat activity, evaluate those moments with AI, extract clips, and provide a browser-based editing and publishing workflow.
ClippingShorts is designed to monitor 2–10 live streams simultaneously, with separate workers handling ingestion, AI evaluation, and video rendering.
Here’s a look at how I built it.
The Architecture
ClippingShorts uses a Docker-based, multi-service architecture. Each major workload runs independently, so stream ingestion, AI evaluation, and video rendering don't have to compete inside one application process.
The core technology stack includes:
- Frontend: Next.js 15, TypeScript, and Tailwind CSS
- Backend: Python and FastAPI
- Video processing: FFmpeg
- AI evaluation: Google Gemini
- Speech recognition: Google Cloud Speech-to-Text (Chirp 3)
- Queues and live state: Redis
- Database: SQLite or PostgreSQL
- Deployment: Docker Compose
The system is organized around three primary worker responsibilities:
- Stream ingestion and local highlight detection.
- AI-based candidate evaluation.
- Clip extraction and rendering.
This separation makes the processing pipeline easier to manage and helps isolate resource-intensive workloads.
1. Monitoring Multiple Live Streams
The first challenge is continuously monitoring several live streams without unnecessarily recording and storing everything.
Each monitored stream runs in its own operating-system worker process. That worker handles video ingestion, audio extraction, chat analysis, and local highlight detection.
Instead of retaining entire broadcasts, ClippingShorts uses a rolling video buffer that keeps the latest 180 seconds by default.
As new segments arrive, older segments are automatically removed unless they have been pinned for further processing.
This approach helps reduce storage consumption while keeping enough recent footage available to extract a potential highlight.
2. Detecting Interesting Moments Without Constant AI Calls
I didn't want to send every audio frame or chat message to Gemini. That would introduce unnecessary API calls and make the detection pipeline more expensive.
Instead, I implemented a local detection layer that continuously evaluates activity within each stream.
Audio signals include:
- RMS loudness and decibel levels.
- Rolling audio baselines.
- Relative audio spikes.
- Voice activity.
Chat signals include:
- Messages per second.
- Number of unique participants.
- Emotes and reaction words.
- Repeated messages and spam activity.
The detector compares activity against the stream's own rolling baseline rather than relying on one fixed threshold for every stream.
For example, a sudden increase in audio intensity or a burst of chat activity can indicate that something interesting is happening.
When the detector identifies a potential highlight, the relevant video segments are pinned so they remain available for the next stage.
3. Two Independent Gemini Judges
This is one of the most interesting parts of the system.
Rather than using a single AI evaluation step, I implemented two independent evaluation paths.
Transcript Judge
The system generates speech transcripts using Google Cloud Speech-to-Text (Chirp 3). The transcript is then evaluated to identify potentially interesting spoken moments, conversations, and reactions.
Reaction Judge
The second evaluation path focuses on reaction signals, including audio loudness and live chat activity.
Both judges operate independently. Their results are then combined to determine whether a candidate should be accepted as a highlight.
This design separates speech-based understanding from reaction-based signals, giving the system two different perspectives when evaluating a potential clip.
It also avoids relying on continuous Gemini calls for raw stream activity.
4. Automatic Clip Extraction
Once a candidate is accepted, the system extracts the relevant footage from the rolling buffer.
The default extraction window includes:
- 120 seconds before the detected event.
- 40 seconds after the detected event.
Keeping footage before and after the event helps preserve the context surrounding a moment instead of producing clips that begin or end abruptly.
The pipeline also merges overlapping candidates to avoid generating multiple clips containing substantially the same footage.
Original source clips remain unchanged. Each edit produces a separate rendered version, making it possible to revisit the original footage.
5. Browser-Based Video Editor
Automatic detection is only part of the workflow. Sometimes a clip needs a different trim, caption, or opening hook before it is ready to publish.
That's why ClippingShorts includes an integrated browser-based editor.
It supports:
- Timeline-based editing.
- Video thumbnails and waveform visualization.
- Clip trimming.
- 9:16, 16:9, and 1:1 aspect ratios.
- Smart cropping and blurred-background fitting.
- Editable captions and hook text.
- Titles, descriptions, and hashtags.
- Volume adjustment and loudness normalization.
Rendering is handled asynchronously by dedicated FFmpeg workers, keeping video processing separate from the editing interface.
6. Automated YouTube Publishing
Once a clip is ready, it can be published using the YouTube Data API.
The publishing workflow supports:
- Resumable video uploads.
- Tracking published YouTube video IDs.
- Video metadata such as titles and descriptions.
- A review-before-publishing workflow.
Automatic uploading is disabled by default.
I chose this behavior intentionally: users should be able to review generated clips before anything is published to their YouTube channel.
7. Resource Management for Self-Hosted Servers
Running multiple live streams, processing audio, evaluating candidates, and rendering videos can consume significant CPU, memory, disk, and storage resources.
For a self-hosted application, simply starting more workers isn't a reliable scaling strategy.
ClippingShorts includes a resource-aware admission controller that monitors system resources and helps manage workload execution.
The implementation includes:
- Fixed concurrency limits.
- Bounded Redis queues.
- Per-stream process isolation.
- Exponential restart backoff.
- Automatic temporary-file cleanup.
- Resource-based job scheduling.
The objective is to prevent video-processing workloads from overwhelming smaller self-hosted servers and to make resource usage more predictable.
8. Self-Hosting with Docker Compose
ClippingShorts is designed to run on infrastructure you control.
The project uses Docker Compose to bring up the application services. Users can configure their own:
- Google Cloud credentials.
- Gemini access.
- Google Cloud Speech-to-Text credentials.
- YouTube OAuth credentials.
- Redis instance.
The project also includes synthetic streams for testing the detection pipeline without depending entirely on real live broadcasts.
This makes it easier to experiment with the system and evaluate its processing workflow in a controlled environment.
What's Next?
ClippingShorts is an ongoing project, and there are several areas I'd like to improve:
- More accurate highlight detection.
- Better AI evaluation and candidate ranking.
- Improved resource efficiency.
- More flexible editing workflows.
- Further improvements to the self-hosting experience.
I'm particularly interested in feedback from developers working with AI agents, streaming infrastructure, FFmpeg, real-time processing, and video automation.
Try ClippingShorts
The project is open source and released under the MIT License.
GitHub repository: https://github.com/saifk6717/ClippingShorts
If you work with live streaming, AI-powered video processing, or self-hosted applications, I'd love to hear your thoughts.
What would you improve in this architecture?
Please only clip and publish streams you own or have permission to use, and follow the relevant platform and creator terms.


Top comments (1)
Great!