<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Saif Khan</title>
    <description>The latest articles on DEV Community by Saif Khan (@saifk6717).</description>
    <link>https://dev.to/saifk6717</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4251970%2F6770c7c5-c0cc-49ff-a2e9-1a25f351422e.png</url>
      <title>DEV Community: Saif Khan</title>
      <link>https://dev.to/saifk6717</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/saifk6717"/>
    <language>en</language>
    <item>
      <title>I Built an Open-Source AI System That Watches Live Streams and Automatically Creates YouTube Shorts</title>
      <dc:creator>Saif Khan</dc:creator>
      <pubDate>Wed, 18 Nov 2026 13:19:08 +0000</pubDate>
      <link>https://dev.to/saifk6717/i-built-an-open-source-ai-system-that-watches-live-streams-and-automatically-creates-youtube-shorts-5goh</link>
      <guid>https://dev.to/saifk6717/i-built-an-open-source-ai-system-that-watches-live-streams-and-automatically-creates-youtube-shorts-5goh</guid>
      <description>&lt;h1&gt;
  
  
  I Built an Open-Source AI Platform That Automatically Turns Live Streams into YouTube Shorts
&lt;/h1&gt;

&lt;p&gt;What if you could monitor multiple Twitch and YouTube Live streams, automatically detect exciting moments, turn them into short-form videos, and prepare them for publishing — without manually watching hours of footage?&lt;/p&gt;

&lt;p&gt;That's the problem I wanted to solve with &lt;strong&gt;ClippingShorts&lt;/strong&gt;, an open-source, self-hosted AI video clipping platform.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;GitHub:&lt;/strong&gt; &lt;a href="https://github.com/saifk6717/ClippingShorts" rel="noopener noreferrer"&gt;https://github.com/saifk6717/ClippingShorts&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The idea goes beyond building another video cutter. I wanted to build an end-to-end system that can monitor live streams, identify potentially interesting moments using audio and chat activity, evaluate those moments with AI, extract clips, and provide a browser-based editing and publishing workflow.&lt;/p&gt;

&lt;p&gt;ClippingShorts is designed to monitor &lt;strong&gt;2–10 live streams simultaneously&lt;/strong&gt;, with separate workers handling ingestion, AI evaluation, and video rendering.&lt;/p&gt;

&lt;p&gt;Here’s a look at how I built it.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fd5v431o17bdbaqm1p9ku.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fd5v431o17bdbaqm1p9ku.png" alt="ClippingShorts Dashboard" width="799" height="382"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The Architecture
&lt;/h2&gt;

&lt;p&gt;ClippingShorts uses a Docker-based, multi-service architecture. Each major workload runs independently, so stream ingestion, AI evaluation, and video rendering don't have to compete inside one application process.&lt;/p&gt;

&lt;p&gt;The core technology stack includes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Frontend:&lt;/strong&gt; Next.js 15, TypeScript, and Tailwind CSS&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Backend:&lt;/strong&gt; Python and FastAPI&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Video processing:&lt;/strong&gt; FFmpeg&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;AI evaluation:&lt;/strong&gt; Google Gemini&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Speech recognition:&lt;/strong&gt; Google Cloud Speech-to-Text (Chirp 3)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Queues and live state:&lt;/strong&gt; Redis&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Database:&lt;/strong&gt; SQLite or PostgreSQL&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Deployment:&lt;/strong&gt; Docker Compose&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The system is organized around three primary worker responsibilities:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Stream ingestion and local highlight detection.&lt;/li&gt;
&lt;li&gt;AI-based candidate evaluation.&lt;/li&gt;
&lt;li&gt;Clip extraction and rendering.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This separation makes the processing pipeline easier to manage and helps isolate resource-intensive workloads.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Monitoring Multiple Live Streams
&lt;/h2&gt;

&lt;p&gt;The first challenge is continuously monitoring several live streams without unnecessarily recording and storing everything.&lt;/p&gt;

&lt;p&gt;Each monitored stream runs in its own operating-system worker process. That worker handles video ingestion, audio extraction, chat analysis, and local highlight detection.&lt;/p&gt;

&lt;p&gt;Instead of retaining entire broadcasts, ClippingShorts uses a &lt;strong&gt;rolling video buffer&lt;/strong&gt; that keeps the latest 180 seconds by default.&lt;/p&gt;

&lt;p&gt;As new segments arrive, older segments are automatically removed unless they have been pinned for further processing.&lt;/p&gt;

&lt;p&gt;This approach helps reduce storage consumption while keeping enough recent footage available to extract a potential highlight.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Detecting Interesting Moments Without Constant AI Calls
&lt;/h2&gt;

&lt;p&gt;I didn't want to send every audio frame or chat message to Gemini. That would introduce unnecessary API calls and make the detection pipeline more expensive.&lt;/p&gt;

&lt;p&gt;Instead, I implemented a local detection layer that continuously evaluates activity within each stream.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Audio signals include:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;RMS loudness and decibel levels.&lt;/li&gt;
&lt;li&gt;Rolling audio baselines.&lt;/li&gt;
&lt;li&gt;Relative audio spikes.&lt;/li&gt;
&lt;li&gt;Voice activity.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Chat signals include:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Messages per second.&lt;/li&gt;
&lt;li&gt;Number of unique participants.&lt;/li&gt;
&lt;li&gt;Emotes and reaction words.&lt;/li&gt;
&lt;li&gt;Repeated messages and spam activity.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The detector compares activity against the stream's own rolling baseline rather than relying on one fixed threshold for every stream.&lt;/p&gt;

&lt;p&gt;For example, a sudden increase in audio intensity or a burst of chat activity can indicate that something interesting is happening.&lt;/p&gt;

&lt;p&gt;When the detector identifies a potential highlight, the relevant video segments are pinned so they remain available for the next stage.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3qcawv6i25gypeacozw6.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3qcawv6i25gypeacozw6.png" alt="ClippingShorts Highlights Detection" width="800" height="373"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Two Independent Gemini Judges
&lt;/h2&gt;

&lt;p&gt;This is one of the most interesting parts of the system.&lt;/p&gt;

&lt;p&gt;Rather than using a single AI evaluation step, I implemented two independent evaluation paths.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Transcript Judge&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The system generates speech transcripts using Google Cloud Speech-to-Text (Chirp 3). The transcript is then evaluated to identify potentially interesting spoken moments, conversations, and reactions.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reaction Judge&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The second evaluation path focuses on reaction signals, including audio loudness and live chat activity.&lt;/p&gt;

&lt;p&gt;Both judges operate independently. Their results are then combined to determine whether a candidate should be accepted as a highlight.&lt;/p&gt;

&lt;p&gt;This design separates speech-based understanding from reaction-based signals, giving the system two different perspectives when evaluating a potential clip.&lt;/p&gt;

&lt;p&gt;It also avoids relying on continuous Gemini calls for raw stream activity.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Automatic Clip Extraction
&lt;/h2&gt;

&lt;p&gt;Once a candidate is accepted, the system extracts the relevant footage from the rolling buffer.&lt;/p&gt;

&lt;p&gt;The default extraction window includes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;120 seconds before&lt;/strong&gt; the detected event.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;40 seconds after&lt;/strong&gt; the detected event.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Keeping footage before and after the event helps preserve the context surrounding a moment instead of producing clips that begin or end abruptly.&lt;/p&gt;

&lt;p&gt;The pipeline also merges overlapping candidates to avoid generating multiple clips containing substantially the same footage.&lt;/p&gt;

&lt;p&gt;Original source clips remain unchanged. Each edit produces a separate rendered version, making it possible to revisit the original footage.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Browser-Based Video Editor
&lt;/h2&gt;

&lt;p&gt;Automatic detection is only part of the workflow. Sometimes a clip needs a different trim, caption, or opening hook before it is ready to publish.&lt;/p&gt;

&lt;p&gt;That's why ClippingShorts includes an integrated browser-based editor.&lt;/p&gt;

&lt;p&gt;It supports:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Timeline-based editing.&lt;/li&gt;
&lt;li&gt;Video thumbnails and waveform visualization.&lt;/li&gt;
&lt;li&gt;Clip trimming.&lt;/li&gt;
&lt;li&gt;9:16, 16:9, and 1:1 aspect ratios.&lt;/li&gt;
&lt;li&gt;Smart cropping and blurred-background fitting.&lt;/li&gt;
&lt;li&gt;Editable captions and hook text.&lt;/li&gt;
&lt;li&gt;Titles, descriptions, and hashtags.&lt;/li&gt;
&lt;li&gt;Volume adjustment and loudness normalization.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Rendering is handled asynchronously by dedicated FFmpeg workers, keeping video processing separate from the editing interface.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Automated YouTube Publishing
&lt;/h2&gt;

&lt;p&gt;Once a clip is ready, it can be published using the YouTube Data API.&lt;/p&gt;

&lt;p&gt;The publishing workflow supports:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Resumable video uploads.&lt;/li&gt;
&lt;li&gt;Tracking published YouTube video IDs.&lt;/li&gt;
&lt;li&gt;Video metadata such as titles and descriptions.&lt;/li&gt;
&lt;li&gt;A review-before-publishing workflow.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Automatic uploading is disabled by default.&lt;/p&gt;

&lt;p&gt;I chose this behavior intentionally: users should be able to review generated clips before anything is published to their YouTube channel.&lt;/p&gt;

&lt;h2&gt;
  
  
  7. Resource Management for Self-Hosted Servers
&lt;/h2&gt;

&lt;p&gt;Running multiple live streams, processing audio, evaluating candidates, and rendering videos can consume significant CPU, memory, disk, and storage resources.&lt;/p&gt;

&lt;p&gt;For a self-hosted application, simply starting more workers isn't a reliable scaling strategy.&lt;/p&gt;

&lt;p&gt;ClippingShorts includes a resource-aware admission controller that monitors system resources and helps manage workload execution.&lt;/p&gt;

&lt;p&gt;The implementation includes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Fixed concurrency limits.&lt;/li&gt;
&lt;li&gt;Bounded Redis queues.&lt;/li&gt;
&lt;li&gt;Per-stream process isolation.&lt;/li&gt;
&lt;li&gt;Exponential restart backoff.&lt;/li&gt;
&lt;li&gt;Automatic temporary-file cleanup.&lt;/li&gt;
&lt;li&gt;Resource-based job scheduling.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The objective is to prevent video-processing workloads from overwhelming smaller self-hosted servers and to make resource usage more predictable.&lt;/p&gt;

&lt;h2&gt;
  
  
  8. Self-Hosting with Docker Compose
&lt;/h2&gt;

&lt;p&gt;ClippingShorts is designed to run on infrastructure you control.&lt;/p&gt;

&lt;p&gt;The project uses Docker Compose to bring up the application services. Users can configure their own:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Google Cloud credentials.&lt;/li&gt;
&lt;li&gt;Gemini access.&lt;/li&gt;
&lt;li&gt;Google Cloud Speech-to-Text credentials.&lt;/li&gt;
&lt;li&gt;YouTube OAuth credentials.&lt;/li&gt;
&lt;li&gt;Redis instance.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The project also includes synthetic streams for testing the detection pipeline without depending entirely on real live broadcasts.&lt;/p&gt;

&lt;p&gt;This makes it easier to experiment with the system and evaluate its processing workflow in a controlled environment.&lt;/p&gt;

&lt;h2&gt;
  
  
  What's Next?
&lt;/h2&gt;

&lt;p&gt;ClippingShorts is an ongoing project, and there are several areas I'd like to improve:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;More accurate highlight detection.&lt;/li&gt;
&lt;li&gt;Better AI evaluation and candidate ranking.&lt;/li&gt;
&lt;li&gt;Improved resource efficiency.&lt;/li&gt;
&lt;li&gt;More flexible editing workflows.&lt;/li&gt;
&lt;li&gt;Further improvements to the self-hosting experience.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I'm particularly interested in feedback from developers working with AI agents, streaming infrastructure, FFmpeg, real-time processing, and video automation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Try ClippingShorts
&lt;/h2&gt;

&lt;p&gt;The project is open source and released under the &lt;strong&gt;MIT License&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;GitHub repository:&lt;/strong&gt; &lt;a href="https://github.com/saifk6717/ClippingShorts" rel="noopener noreferrer"&gt;https://github.com/saifk6717/ClippingShorts&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;If you work with live streaming, AI-powered video processing, or self-hosted applications, I'd love to hear your thoughts.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What would you improve in this architecture?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Please only clip and publish streams you own or have permission to use, and follow the relevant platform and creator terms.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>docker</category>
      <category>opensource</category>
    </item>
  </channel>
</rss>
