A useful video can still contain one distracting object: a passerby in the background, an unwanted sign, or something that pulls attention away from the subject.
I’m building removeobjects.video, a web app for removing unwanted objects from video. The project also includes a separate AI video repainting workflow, but this post focuses on object removal: how the editor works, how SAM3 and ProPainter fit together, and the practical limits of the approach.
The editing workflow
The object-removal flow is built around a short sequence of decisions:
- Select a video and choose the segment to edit.
- Pick a frame where the target is clearly visible.
- Describe the target with a short phrase.
- Inspect the detected mask and confirm the processing option.
- Wait for the background job and download the result.
That mask preview matters. A description can sound obvious to a person while still matching the wrong region of an image. Showing the selection before processing the entire clip gives the user a chance to choose another frame or rewrite the description.
For example, “the person wearing a red shirt” gives the model a visual characteristic to follow. “the person on the left” can become ambiguous when people move or the camera changes position. A short, distinctive description is usually a better starting point than a long instruction.
Two models, two responsibilities
The core pipeline uses SAM3 for segmentation and ProPainter for video inpainting.
SAM3 identifies the region to remove. The editor uses a single-frame preview to help the user inspect the selection. After confirmation, the full-clip segmentation stage produces the masks needed for processing.
ProPainter fills the masked regions. Video inpainting has to reconstruct the background while keeping the sequence visually consistent over time. A region that looks plausible in one frame can still flicker or smear during playback.
The full processing flow also includes preparing the clip, handling its audio, and storing the finished video. Segmentation and inpainting are the central model stages, but they are only part of the application.
Keeping the editor usable during upload
Video upload can take long enough to interrupt the editing experience.
In the usual workflow, the browser keeps the selected video available locally so the user can work with the timeline while the full file uploads in the background. Selected preview frames are uploaded separately for recognition. The full video must finish uploading before the final processing job can proceed.
This introduces a boundary the application has to handle carefully: the browser may have a usable local frame before the server has the complete video. The editor therefore tracks preparation, upload, and processing as separate states.
Running the background pipeline
The application uses React Router 7 on Cloudflare Workers, with R2 for video files and D1 for task records. Cloudflare Workflows coordinates the longer processing steps, and an FFmpeg sidecar running in Cloudflare Containers handles media preparation and final processing. The object-removal models run through Replicate.
Keeping task state separate from the page request helps the application represent a job that is waiting, processing, finished, or failed. It also makes recovery decisions more precise.
One example is handling an interrupted provider response. When a prediction ID is already available, checking that existing prediction can be safer than immediately submitting another GPU job. A failed status check does not necessarily mean the original job failed.
Details that affect the result
Three parts of this workflow deserve particular attention:
- Mask coverage: if the segmentation misses part of a moving object, inpainting cannot reliably remove the missing area. A single-frame preview is useful, but it does not guarantee correct coverage throughout the clip.
- Geometry: the source clip and its masks need matching dimensions and coordinates. Resizing either one without keeping them aligned can make an otherwise valid mask target the wrong pixels.
- Processing limits: duration, frame rate, and resolution affect memory requirements. Longer or larger clips can require downsampling, and some jobs can still fail.
These constraints are also why I would judge an edit by watching the resulting video, rather than looking only at a still frame or a successful API response. Motion, occlusion, and detailed backgrounds can expose problems that a thumbnail hides.
What the current version supports
The free object-removal mode supports clips up to 6 seconds, with target-area and resolution limits and a watermark. Paid object-removal clips can be up to 50 seconds. Processing may reduce resolution; preserving the original resolution is not guaranteed.
A short clip with a clearly visible target is a useful starting point for evaluating the workflow. The editor provides a way to inspect the intended selection, while the finished video shows how well the removal holds up over time.
I’d welcome feedback on the editing workflow, especially where target descriptions or mask previews could make the process easier to understand.
Top comments (0)