DEV Community: Mert ALP

Turn long videos into short clips — without uploading your videos.

Mert ALP — Thu, 08 Oct 2026 06:23:09 +0000

LunarClip finds the best moments, reframes them to 9:16,1:1,4:5 and adds captions without uploading your video.

Built for Windows. 🔒
Local AI Video Clipper Tool
🌐 LunarClip

One crop does not fit all: framing vertical clips by video type, on the user's PC

Mert ALP — Tue, 06 Oct 2026 09:54:57 +0000

We build LunarClip, a Windows app that turns long videos into short vertical clips. The obvious way to make a 9:16 clip from a 16:9 video is a center crop, or a crop that follows the biggest face. Both work for a talking head and fail almost everywhere else. This post covers the two decisions that shaped the app most: the crop depends on what the video is about, and the heavy work happens on the user's own computer.

Why one crop rule is not enough

A 9:16 window over a 16:9 frame keeps only about a third of the width. What belongs in that third depends on the genre:

In a stream, the important things are in two places at once: the streamer's webcam in a corner and the gameplay in the middle. Any single crop loses one of them.
In sports, the subject is the ball and the players around it, not the largest face. A face-following crop happily tracks a spectator.
In a podcast, two people talk back and forth. A crop that jumps to whoever spoke last feels nervous, and a crop that sits on one person misses half the conversation.
In an everyday or food video, the important object is often not a person at all: the pan, the plate, the product.

So the app has a framing rule per video type.

Type	Rule
Game	The streamer's webcam goes on top showing only the person, and the gameplay fills the rest, wherever the webcam sits in the source.
Sports	Follows the ball and the players around it, switching between close-up, medium, wide and full-frame shots.
Podcast	Split screen for back-and-forth, one speaker for monologues, and a wide shot when everyone talks. Switching is deliberately calm.
Normal	No automatic split. Keeps food and products in frame, and still zooms in on a person talking to camera.
Auto	Picks the type from the title, tags, description and sound, and falls back to Normal when it is unsure.

Two things we learned while building these rules:

Genre detection should fail safe. When Auto cannot tell what a video is, it falls back to the plainest rule. A wrong split screen on a cooking video looks far worse than a slightly loose crop.
Calm beats clever. On podcasts, calm speaker switching reads better than a crop that chases every short "yeah". Fewer cuts, each one for a reason.

Why it runs on your PC

The second decision was where the work happens. Uploading a two-hour video to a server before anything can start means a long upload, a queue and a copy of the video on someone else's machine. So everything that touches the video runs locally.

On the user's computer:

speech-to-text with a local speech model (downloaded once, with the first transcription),
shot and face analysis and the framing rules above,
captions and hook text,
rendering, on a recent NVIDIA, AMD or Intel graphics card if there is one, otherwise on the processor.

Only one step uses the server: picking the strongest moments. For that, the app sends short transcript excerpts of candidate clips (text, timestamps and scores), and the server returns its picks with titles and captions. The video, the audio and the finished clips never leave the computer.

The trade-offs are real and worth stating:

the app needs an internet connection while a project is being made, because moment picking happens on the server;
processing speed depends on the user's hardware instead of a data-center GPU;
it is a Windows app for now.

What the user gets in exchange is no upload, no shared queue, and no copy of their video anywhere else.

Takeaway

If you are building anything that reframes video, decide early what your subject is for each kind of footage. "Follow the biggest face" is a rule for talking heads, not a general one. And if most of the pipeline can run on the user's machine, the privacy story becomes a property of the architecture rather than a promise in a policy.
LunarClip

Five video metrics that lied to us (and the one-frame rule that caught them)

Mert ALP — Sat, 03 Oct 2026 14:46:07 +0000

A broken pixel metric does not crash. It keeps returning numbers. The numbers are in range, the table looks consistent, the histogram has a sensible shape, and you write a conclusion under it.

We build a tool that reframes landscape video into vertical clips, so we measure a lot of things per frame: where the face is, where the captions sit, whether the frame is empty, who is speaking. Five of those measurements were wrong in ways that no other number could have told us. Each time, what exposed the problem was opening one frame and looking at it.

This post covers the five failures and the rule we adopted afterwards.

Background: the `fit` layout

Most of these failures share one cause, so the layout needs explaining first.

When a 16:9 source goes into a 9:16 output, there are two broad options. fill crops the source to a tall window. fit keeps the whole source visible: the 16:9 picture sits in the middle of the vertical frame, and the empty bands above and below are filled with a blurred, enlarged copy of the same video.

+-----------+
| blurred   |   enlarged, blurred copy of the video
| copy      |
+-----------+
|           |
|  real     |   the actual 16:9 picture
|  picture  |
|           |
+-----------+
| blurred   |
| copy      |
+-----------+

The padding is useful visually. For measurement it is a trap, because:

It is not empty. It is the video, scaled up and blurred.
Anything a detector can find in the real picture, it can often find in the padding too, at a larger size.
Whole horizontal bands of the frame belong to the padding, not to the content.

Keep that in mind for the metrics below.

1. The caption band detector

What we thought it measured: the vertical position of the caption band, so we could keep subjects clear of it.

What it measured: a loose contour-based detector responds to any dense, high-contrast edge structure near the bottom of the frame. In practice that meant a broadcaster's watermark, plus the outlines of hands and plates. The "band" appeared to jump around from frame to frame because the detector was latching onto different objects.

Nothing errored. We got a position for every frame.

2. The identity histogram

What we thought it measured: whether the person in frame was the same person, by comparing colour histograms of a region.

What it measured: we sampled the region just below the detected face, assuming it would be a shirt or torso. It was frequently a hand, a napkin or a plate. The histogram changed substantially while the person stayed exactly the same, and stayed flat when a different person wore similar colours.

3. The mouth and eye motion ratio

What we thought it measured: whether someone is speaking, using landmark motion around the mouth normalised by the eyes.

def mouth_activity(landmarks_seq):
    # landmarks_seq: per-frame (mouth_points, eye_points)
    ratios = []
    for mouth, eyes in landmarks_seq:
        mouth_motion = np.ptp(mouth[:, 1])        # vertical spread
        eye_scale = np.linalg.norm(eyes[0] - eyes[1])
        ratios.append(mouth_motion / eye_scale)
    return float(np.mean(ratios))

What it measured: jaw movement. On cooking and eating footage, the frame with the highest "speaking" score was the clearest example of someone chewing. The metric is perfectly good at detecting that a mouth is moving. It cannot tell you why.

4. The empty-frame band

What we thought it measured: whether the frame is empty (no subject), by checking how much the pixels in a vertical band deviate from a flat background.

What it measured: the band we picked fell inside the fit padding. The padding is a blurred copy of the video, and in a dark scene a blurred dark video is close to one flat colour. So dark scenes with a clearly visible subject scored as empty.

band = frame[int(0.15 * h):int(0.55 * h)]
deviation = band.reshape(-1, 3).std(axis=0).mean()
# low deviation -> "empty"
# ...in a layout where that band is blurred padding

5. Face area from a face detector

This was the clearest one, and it used a good, widely available detector (YuNet, via OpenCV).

What we thought it measured: how large the face is in the frame, as a fraction of frame area. We take the largest box per frame.

detector = cv2.FaceDetectorYN.create(model_path, "", (w, h))
_, faces = detector.detect(frame)
if faces is not None:
    x, y, bw, bh = max(faces, key=lambda f: f[2] * f[3])[:4]
    face_area = (bw * bh) / (w * h)

What it measured: in the fit layout, the padding contains an enlarged copy of the speaker, and the detector found that face too. Because we took the largest box per frame, the enlarged copy won. In 13 of 17 sampled frames, the selected box was in the padding. The reported face area was 14.77% against a real value of about 0.15%.

Again: valid boxes, valid confidence scores, and a believable-looking distribution.

What these have in common

None of the five raised an error. Each returned values in the expected range. And each was wrong in a way that a second metric would not have revealed, because a second metric computed on the same frames would have inherited the same mistake (the padding, the watermark, the hand).

What does reveal the problem is a single frame with the measurement drawn on it.

The rule

Before a pixel metric goes into a report or a decision, look at at least one frame with the measurement overlaid. Generate a contact sheet or a strip, open it, and look.

It does not need to be a heavy process. A helper like this is enough:

def overlay_and_save(frame, boxes, path):
    out = frame.copy()
    for (x, y, w, h) in boxes:
        cv2.rectangle(out, (int(x), int(y)), (int(x + w), int(y + h)), (0, 255, 0), 3)
    cv2.imwrite(path, out)

Then open the file. The step that matters is the last one: a person looks at it.

Checklist

Look at the highest-scoring and lowest-scoring frames for the metric. Extremes show failures fastest; our "most talkative" frame was the chewing one.
If the layout has padding, detect the padded frames and mask the padding, or widen the measurement region to the whole frame. Do not measure inside a band that may be blurred filler.
Keep watermarks, broadcaster graphics and burned-in lower thirds out of the measurement region.
Apply the same detector with the same settings to both sides of any comparison. Do not use a loose detector for coverage on one side and a strict one for position on the other. If one output has black padding and another has blurred padding, a comparison that ignores this is biased.
Show the contact sheet to someone else. Writing it to disk is not delivering it, and a second pair of eyes catches what the author has stopped seeing.

Disclosure

I work on LunarClip; these measurements come from testing its reframing. This article was written with AI assistance from our own measurement notes.

The rule itself is tool-agnostic. If you run a metric over video and have never looked at the frame it picked, pick the worst-scoring one today and open it.