Introducing sgclipper-3b-v1. An audio native language model that directly predicts clippable moments from an audio source. It’s a distillation of Gemini 3.5 Flash clipping capabilities into Voxtral-Mini-3B. This is a research prototype, and far from perfect. But overall, it looks very promising and worth investing more time on.
- 🤏 Small and fast: 50x realtime on an unoptimized inference stack
- ♾️ Infinite audio length: processing time linearly scales with audio length
- 💵 Cost efficient: with minimal inference optimizations, it could process audio at lower than $0,007/hr
- ➡️ Direct audio to clips: the model uses the full audio features to directly predict clips. No complex architecture, no information lost doing transcriptions.
In this blogpost, I will explain you how I built sgclipper-3b-v1. I will take you on the full journey from idea to model, from failed attempts to a successfull training run. Let’s begin.
I'm the founder of StreamGen, an AI clipping platform for Twitch. While building it, I noticed that modern clipping tech is still stuck in the speech to text era. We now have insanely powerful multimodal models that can natively watch and listen, yet almost every clipping tool out there still doesn’t leverage these capabilities.
While video-native understanding is still too expensive at scale, I think it's time for audio-native clipping to become the new standard. But why is it better?
Traditional STT pipelines throw away a lot of information: laughs, ambient sound, overlapping speakers, game audio cues (gunshots, win/lose stingers), volume and tone, shouting vs. whispering… A lot of useful signals get lost.
With an audio-native approach, the model uses the full audio features, substantially improving clipping quality. It’s also much simpler: instead of a complex multi-step pipeline, audio is transformed directly into clips in a single pass.
Why aren't we already using audio-native models?
They are too expensive. Audio-native processing is 2–3× more expensive than STT + text LLMs. The prices in the chart above were calculated using Gemini’s batched input/output pricing. In practice, you should deliver clips to your users in far less than 24 hours, so a more realistic price would be roughly double.
The problem is even worse for products like StreamGen because we analyze livestreams. A streamer may go live 20 times a month for an average of 4 hours per stream. With Gemini’s native audio understanding, that would cost about $7/month with Gemini 3.1 Flash Lite and $25/month with Gemini 3.5 Flash.
You could already lower costs by using smaller models (Gemini Flash Lite) or by reducing/removing reasoning. But that would directly hurt clipping performance, and we shouldn’t improve margins by lowering product quality. This isn’t a good direction.
In this post, I’ll show you how I trained a tiny audio LLM to solve this problem. The quality is nearly on par with Gemini 3.5 Flash (on my evaluations!), but at least 22× cheaper.
Let’s start.
Building the training dataset
I built a dataset from ~250 hours of Italian livestreams, pairing 15 minutes of raw audio with extracted “best moments” from Gemini 3.5 Flash. I reused parts of the StreamGen infrastructure to build this data pipeline. It can scale to far more hours of audio, but for this first experiment I chose to start small.
Why only Italian livestreams?
- My end goal is to build a multilingual clipper, so why start with English?
- I know the Italian Twitch/Livestream community well enough to confidently and manually judge what's clippable and what isn't
- I'm Italian 🍕
Attempt 1: just SFT it
I started with the simplest approach that could work: supervised fine-tuning Voxtral directly on Gemini’s output. Here are the results, measured with an IoU threshold of 0.5 (a clip is matched if its IoU with the ground-truth clip is greater than 0.5):
| Precision\@0.5 | Recall\@0.5 | |
|---|---|---|
| After 500 samples | 10% | 7% |
| After 1000 samples | 20% | 18% |
The model found only 18% of the ground-truth clips. Even after training on the full 250-hour dataset, clipping performance was still pretty weak.
My initial thought was: let’s build a bigger dataset! But before throwing more compute at the problem, it was worth asking whether there was a more data-efficient approach.
Attempt 2: make it simple
In the first attempt, the model had to predict multiple clips inside a 15-minute audio, and for each one, a title, virality score, description, and precise start/end times. That's a lot to ask.
So I stripped it down:
- 15-minute audio → 5-minute audio
- Predict multiple clips → predict just the single best one
- Title + description + virality score → just start and end time
Here are the results after another full SFT run:
| Precision\@0.5 | |
|---|---|
| After 3000 samples | 11% |
The results are even worse than the first run. Why? My best guess is that in Attempt 1, the model had multiple tries per chunk, so the odds of hitting at least one correct clip were higher. Here, it gets exactly one shot, and it’s right only 11% of the time. Not great.
A week into the project, I was starting to feel discouraged, especially after burning many dollars on compute with little to show for it. But I wanted to make this work, so I kept grinding.
Attempt 3: a new direction
After brainstorming with GPT5.6 Sol, I landed on a completely different approach. Instead of treating this as an autoregressive generation problem, I treat it as a classification problem.
Instead of asking the model to generate a JSON describing a clip in an audio chunk, I ask a much simpler question: is this specific moment worth clipping? The model just outputs a single probability.
This is an easier task to solve, but it costs us the ability to directly predict clip boundaries. To work around that, each 5-minute chunk gets split into overlapping, strided windows. In my case, 13 windows of 1 minute each, with a 20-second stride.
The highest-scoring window becomes our clippable moment. The boundaries are no longer accurate down to the second, but we’ve reduced the problem from “pick a start and end time out of ~40k possibilities” to “pick the best of 13 windows.”
The loss function
All windows from a given 5-minute chunk are batched together and compete against each other. The model predicts a “clip” probability for each window, and the more a window overlaps with a real ground-truth clip, the more it's rewarded.
This is a cross-entropy loss between the model's predictions and each window's overlap rate with the ground-truth clip.
Results
After one epoch over the full dataset, I expected another underwhelming result. But I was wrong. The model did learn, and it was amazing.
Here's the base model's performance at spotting clippable moments:
This 15-minute audio is divided into 1-minute windows with a 20-second stride. Each bar is the predicted clip probability, shaded in red. Basically a clip heatmap. The ground truth clips (in blue) were produced by Gemini 3.5 Flash, but manually verified.
As you can see, there isn't much signal here, almost every window gets a high probability. Now let’s compare that to the fine-tuned model:
Yes! After weeks of failed attempts and dead ends, the model clearly learned to clip. Seeing this image for the first time was incredible. Going from an idea to “it actually works” feels amazing, and the fact that this is a tiny model trained on just a few hundred hours of data makes it even better.
For another test, I took a real user-made Twitch clip, and recorded about 8 minutes of the surrounding audio:
The purple clip is the one made by a real user on Twitch. Notice there are also high probabilities just before it. This was a Call of Duty match, and those "hot" minutes from 1:40 to 5:20 correspond to players flaming each other. The real “final flame” moment, though, was actually inside the Twitch clip.
I also ran the same audio segment through other AI clipping tools and picked the top clips based on each platform’s own "virality" score.
While VEED.io found the moments before and after the “flame” (which I still think are good), OpusClip also caught the exact Twitch clip.
This quick comparison highlights another important aspect of this project: good moments are easy to spot against bad ones, but they’re much harder to rank against truly great ones. This is because the jump from “good” to “very good” is mostly subjective and taste-driven.
In other words, given two clips that both have a high chance of going viral, it’s very difficult to predict which one will go more viral than the other. Even for a human.
What’s worth noting, instead, is that sgclipper-3b-v1 understood which part was probably clippable, but at a fraction of the cost, and without doing any transcription step.
This model is exciting!
Wicked fast
My current inference script is very unoptimized: it loads the BF16 model, splits the audio into windows, and processes them one at a time in sequence. Despite this, analyzing 10 minutes of audio on an RTX 5090 takes about 12 seconds.
Small and low cost
sgclipper-3b-v1 is a small model competing with Gemini 3.5 Flash's native audio understanding.
With my current inference setup, it’s already capable of processing roughly 50 hours of audio per hour of compute. At an average of $0.7/hr for an RTX 5090, that works out to $0.014 per hour of audio processed, already cheaper than the cheapest STT APIs on the market.
And this is with a BF16 model, no batching, unoptimized audio tokenization, a fairly small stride, and running on HuggingFace Transformers. I'm confident this could get 5–10x faster with some optimization work, before even considering model pruning or using other smaller/faster pre-trained encoders/decoders. There are also cheaper GPUs that could run this, like the NVIDIA RTX 4000 SFF Ada Generation, available on Hetzner for $0.4391/hr.
Properly optimized, this pipeline could find clips at a cost that’s too cheap to meter.
Linear processing time
Because we process audio in windows, we don’t have any context-length limitations and don’t pay any quadratic tax for longer durations. Processing time scales linearly as audio length increases.
Next steps
sgclipper-3b-v1 is capable, fast, and cheap. It can identify clippable moments from raw audio alone, but it’s still far from perfect. Next, I’m focusing on scaling what works (more data and languages), making evaluation more rigorous, and creating a production-ready inference stack.
I'm not open-sourcing the model or the training dataset at this time, as the research is still in the early stages. Nevertheless, I hope you still found this blog post valuable!
Conclusion
In 2026 AI clipping should be audio-native, but today’s solutions are still too expensive. I wanted to solve this by distilling the clipping capabilities of Gemini 3.5 Flash into a small 3B model.
So I built a training dataset of 250 audio hours and took multiple failed attempts. I started with an SFT generation-oriented goal and ended up with a classification approach. And after weeks of iteration, I finally trained sgclipper-3b-v1: a tiny, audio-native clipper, with linear-time processing and no transcription steps.
I don't think I could have ever done this without the amazing work of the HuggingFace team and contributors. I also want to thank Mistral for providing an amazing base model to work on!
I'm eager to know your thoughts and ideas on this. So let’s chat! You can reach me at cimolaimarco01@gmail.com or dm me on X @CimolaiMarco.
Thank you for reading 🤗
Top comments (0)