Removing text from an image is relatively straightforward.
Removing text from a video is a very different problem.
A video is not just a collection of independent images. Every frame has to remain visually consistent with the frames before and after it. If the reconstructed background changes slightly from frame to frame, the result starts to flicker immediately.
Over the past few weeks, I have been building Video Text Remover, a small web tool for removing subtitles, captions, watermarks, timestamps, and other burned-in text from videos.
You can try it here:
In this post, I want to share how I approached the problem, the architecture behind the tool, and a few things I learned about video inpainting along the way.
The Problem: Text Is Part of the Pixels
There are two very different types of subtitles in video.
The first type is a separate subtitle track, such as an .srt file or an embedded subtitle stream.
Those are easy to remove.
You can simply disable or strip the subtitle track.
The second type is burned-in text.
Examples include:
- captions rendered directly into the video
- TikTok-style subtitles
- timestamps
- channel logos
- watermarks
- usernames
- presentation text
- labels added during editing
In this case, there is no separate subtitle layer.
The text is literally part of the image.
Removing it means answering a much harder question:
What should the pixels behind the text look like?
That turns the problem into an inpainting problem.
Image Inpainting Is Not Enough
For a single image, an inpainting model receives something like:
Original image
+
Mask indicating the area to remove
↓
Inpainting model
↓
Reconstructed image
If the object being removed is small, modern models can often reconstruct the missing background surprisingly well.
Video adds another dimension: time.
Imagine a subtitle covering part of someone's shirt.
Frame 1 might reconstruct the missing area as dark blue.
Frame 2 might generate a slightly different texture.
Frame 3 might introduce another variation.
Each frame may look acceptable individually.
But when played together, those tiny differences create obvious flickering.
So a useful video text removal system needs more than spatial consistency.
It also needs temporal consistency.
My Initial Architecture
I wanted the product architecture to stay relatively simple.
The current system roughly looks like this:
Browser
↓
Next.js Application
↓
Upload Video
↓
Cloud Object Storage
↓
Create Processing Job
↓
Video Inpainting Service
↓
Poll Job Status
↓
Processed Video
↓
Download Result
The frontend and application layer are built with Next.js.
Large video files are uploaded to object storage instead of being passed through the application server itself.
This is important because sending large videos through serverless application endpoints creates several problems:
- request size limits
- function timeouts
- unnecessary bandwidth usage
- increased memory usage
- higher infrastructure cost
Instead, the application generates an upload target and lets the browser upload the file directly.
The processing service then works from the stored video.
Manual Selection vs Automatic Detection
One of the first product decisions I had to make was how users should tell the system what to remove.
There are two approaches.
1. Automatic text detection
The ideal experience is:
Upload video
↓
Detect text automatically
↓
Remove detected text
↓
Download
This is obviously the easiest experience for users.
But automatic detection introduces another difficult problem.
The system has to determine:
- what is actually text
- which text should be removed
- where it appears
- whether it moves
- whether the bounding box changes
- whether text appears only during part of the video
OCR models can detect text quite well, but video creates edge cases.
For example:
Scoreboard → probably text
Street sign → text, but maybe part of the scene
Subtitle → probably remove
T-shirt logo → maybe not
Phone screen → depends on user intent
Detection and intent are not the same thing.
2. Manual area selection
The second approach is much simpler.
The user draws a rectangle around the area that should be removed.
For example:
┌──────────────────────────────┐
│ │
│ VIDEO │
│ │
│ ┌──────────────────────┐ │
│ │ subtitle area │ │
│ └──────────────────────┘ │
│ │
└──────────────────────────────┘
This removes a lot of ambiguity.
The user already knows what they want removed.
The system only needs to focus on reconstructing the selected area.
For many real-world cases, especially subtitles, the text stays inside roughly the same region for most of the video.
That makes manual selection surprisingly effective.
For this reason, I decided to support both workflows:
- automatic detection for convenience
- manual selection for control
Representing the Selected Region
The browser displays the video at a different size than its actual resolution.
For example, the uploaded video might be:
1920 × 1080
while the preview shown in the browser might be:
960 × 540
If the user draws a selection box on the preview, the coordinates need to be converted back to the original video coordinate system.
A simple version looks like this:
const scaleX = originalWidth / previewWidth;
const scaleY = originalHeight / previewHeight;
const actualX = selectedX * scaleX;
const actualY = selectedY * scaleY;
const actualWidth = selectedWidth * scaleX;
const actualHeight = selectedHeight * scaleY;
There is another approach that I prefer even more: storing coordinates as normalized values.
For example:
type Region = {
x: number;
y: number;
width: number;
height: number;
};
Where every value is between 0 and 1.
A selection might look like:
{
"x": 0.12,
"y": 0.76,
"width": 0.74,
"height": 0.14
}
This makes the mask independent of the preview resolution.
Later, it can be converted into pixels for any video size.
Video Processing Should Be Asynchronous
Video processing can easily take minutes.
That means a normal synchronous API request is not appropriate.
Instead of doing this:
POST /remove-text
↓
wait...
wait...
wait...
↓
return processed video
I use a job-style workflow:
POST /jobs
↓
jobId
Then the frontend checks the status:
GET /jobs/:jobId
The response may look like:
{
"status": "processing",
"progress": 62
}
And eventually:
{
"status": "completed",
"resultUrl": "..."
}
This architecture has several advantages.
If the browser tab refreshes, the processing job can continue.
If processing fails, the backend can retry.
If the external processing service is temporarily unavailable, jobs can be queued instead of immediately failing.
The Real Challenge: Temporal Consistency
The hardest part is not detecting the text.
It is reconstructing the background convincingly over time.
Consider a camera moving horizontally.
A subtitle covers part of the ground.
The missing region might contain:
Frame 1: grass
Frame 2: grass + shadow
Frame 3: edge of a road
Frame 4: road
Simply running an image inpainting model independently on each frame can produce unstable results.
Video inpainting systems typically try to use information from nearby frames.
A simplified idea is:
Previous frames
↓
Current masked frame
↑
Future frames
↓
Temporal reconstruction
If the background behind the text is visible in nearby frames, the model can use that information to reconstruct the missing area more consistently.
This is why video-specific models tend to produce better results than simply applying an image model frame by frame.
Why Some Videos Work Much Better Than Others
One thing that became obvious very quickly is that the difficulty varies enormously depending on the scene.
Easier cases
Text over:
- static backgrounds
- walls
- sky
- blurred backgrounds
- simple textures
- areas with little motion
usually works quite well.
For example:
Static interview
+
subtitle at the bottom
+
mostly blurred background
→ relatively easy
Harder cases
Things become much more difficult when the text overlaps:
- faces
- hands
- complex clothing
- fast-moving objects
- detailed textures
- camera cuts
- animations
- particles
- rapidly changing scenes
Imagine text covering someone's fingers while they are moving.
The model has to reconstruct not only the appearance of the fingers, but also their motion across multiple frames.
That is a much harder problem.
Masks Matter More Than I Expected
Another lesson was that the mask itself has a large impact on output quality.
A mask that is too small leaves fragments of text behind.
A mask that is too large forces the model to reconstruct unnecessary parts of the image.
For example:
Bad:
┌─────────────────────────────┐
│ │
│ remove me │
│ ─── │
│ mask │
└─────────────────────────────┘
Only covering the center of the letters can leave visible edges.
A slightly larger region often works better:
┌─────────────────────────────┐
│ │
│ ┌───────────────┐ │
│ │ remove me │ │
│ └───────────────┘ │
│ │
└─────────────────────────────┘
But making the region dramatically larger than necessary also increases the chance of visual artifacts.
The best mask usually has a small amount of padding around the text.
Handling Multiple Text Areas
Some videos contain more than one thing to remove.
For example:
┌─────────────────────────────┐
│ USERNAME │
│ │
│ │
│ │
│ subtitles │
└─────────────────────────────┘
Supporting only one rectangle becomes restrictive very quickly.
So I added support for multiple selection regions.
Conceptually:
const regions = [
{
x: 0.03,
y: 0.05,
width: 0.2,
height: 0.08
},
{
x: 0.15,
y: 0.78,
width: 0.7,
height: 0.12
}
];
These regions can later be converted into one combined mask.
This turned out to be useful for:
- logos + subtitles
- username + watermark
- timestamp + caption
- several static overlays
Cost Is a Product Problem Too
Video AI processing is significantly more expensive than many image AI tasks.
A single image might require one inference.
A video may contain thousands of frames.
For example, a 30-second video at 30 FPS contains:
30 × 30 = 900 frames
A five-minute video contains:
5 × 60 × 30 = 9,000 frames
Obviously, video models do not necessarily process every frame independently, but the scale difference explains why video inference can become expensive.
This affects the product design.
You need to think about:
- maximum video duration
- maximum file size
- processing resolution
- job concurrency
- storage lifecycle
- retries
- failed jobs
- abuse prevention
These infrastructure decisions matter almost as much as the AI model itself.
Temporary Storage Is Important
Uploaded videos can be large.
Keeping every original and processed file forever would quickly become expensive.
A better lifecycle is:
Upload
↓
Process
↓
User downloads result
↓
Temporary retention period
↓
Automatic deletion
This keeps storage usage predictable.
It also reduces the amount of user data retained by the service.
Error Handling Matters More with AI APIs
Traditional APIs are often relatively deterministic.
AI inference APIs are different.
A request may fail because of:
- temporary provider errors
- model startup time
- GPU availability
- timeout
- unsupported codec
- malformed video
- unexpectedly large files
So the processing layer needs to distinguish between retryable and non-retryable errors.
Something like:
if (isTemporaryError(error)) {
await retry(job);
} else {
await markJobAsFailed(job);
}
Retry logic should also have limits.
Otherwise a broken video may be processed forever.
A more realistic pattern is:
Attempt 1
↓
Failed
Wait
Attempt 2
↓
Failed
Wait longer
Attempt 3
↓
Failed
Mark job as failed
UX Is Part of the Technical Problem
When a task takes several minutes, showing a spinner is not enough.
Users want to know whether something is actually happening.
Even if the underlying provider does not expose precise frame-level progress, showing processing stages helps:
Uploading video
↓
Preparing video
↓
Removing text
↓
Generating final video
↓
Completed
This sounds like a small detail, but it makes the application feel much more reliable.
Long-running AI workflows need visible state.
What I Would Improve Next
There are still many things I want to improve.
Better automatic detection
The ideal system would track text over time instead of simply detecting text in isolated frames.
Conceptually:
OCR
↓
Bounding box
↓
Track across frames
↓
Determine lifetime
↓
Generate temporal mask
This would make automatic removal much more precise.
Better handling of moving watermarks
Static subtitles are relatively easy because their position rarely changes.
Moving logos and animated overlays are much harder.
Tracking them frame by frame would allow the mask to move with the target.
Scene-aware processing
A five-minute video might contain dozens of scene changes.
Instead of treating the entire video as one continuous sequence, the system could detect cuts:
Video
↓
Scene detection
↓
Scene 1
Scene 2
Scene 3
...
↓
Process separately
↓
Merge
This may improve consistency and reduce unnecessary context.
Preview before full processing
Another useful feature would be processing only a few seconds first.
Users could verify:
- whether the selected region is correct
- whether the model handles the background well
- whether the result is acceptable
before spending time and compute processing the entire video.
The Biggest Lesson
Before building this project, I thought the core problem would be:
How do I remove text from a video?
After working on it, I realized the actual problem is closer to:
How do I reconstruct missing pixels across time while keeping the result visually consistent?
Text detection is only one part of the system.
The full problem includes:
- coordinate systems
- masks
- uploads
- object storage
- asynchronous jobs
- video codecs
- AI inference
- retries
- temporal consistency
- cost control
- UX
That is what made this project much more interesting than I initially expected.
I packaged the current version into a small web tool called Video Text Remover:
It currently focuses on removing visible text and overlays from videos using automatic detection or manually selected regions.
There is still a lot to improve, especially around moving text and difficult backgrounds, but building it has been a useful exploration of what production video AI actually looks like beyond a simple model demo.
If you're also building image or video AI tools, I'd be interested to hear how you're handling long-running inference jobs, storage costs, and temporal consistency.
Tags
#webdev #nextjs #ai #machinelearning
Top comments (0)