How I Built an AI Video Text Remover with Next.js and Video Inpainting

Lim Geng — Sun, 20 Sep 2026 02:02:08 +0000

Removing text from an image is relatively straightforward.

Removing text from a video is a very different problem.

A video is not just a collection of independent images. Every frame has to remain visually consistent with the frames before and after it. If the reconstructed background changes slightly from frame to frame, the result starts to flicker immediately.

Over the past few weeks, I have been building Video Text Remover, a small web tool for removing subtitles, captions, watermarks, timestamps, and other burned-in text from videos.

You can try it here:

https://videotextremover.org

In this post, I want to share how I approached the problem, the architecture behind the tool, and a few things I learned about video inpainting along the way.

The Problem: Text Is Part of the Pixels

There are two very different types of subtitles in video.

The first type is a separate subtitle track, such as an .srt file or an embedded subtitle stream.

Those are easy to remove.

You can simply disable or strip the subtitle track.

The second type is burned-in text.

Examples include:

captions rendered directly into the video
TikTok-style subtitles
timestamps
channel logos
watermarks
usernames
presentation text
labels added during editing

In this case, there is no separate subtitle layer.

The text is literally part of the image.

Removing it means answering a much harder question:

What should the pixels behind the text look like?

That turns the problem into an inpainting problem.

Image Inpainting Is Not Enough

For a single image, an inpainting model receives something like:

Original image
      +
Mask indicating the area to remove
      ↓
Inpainting model
      ↓
Reconstructed image

If the object being removed is small, modern models can often reconstruct the missing background surprisingly well.

Video adds another dimension: time.

Imagine a subtitle covering part of someone's shirt.

Frame 1 might reconstruct the missing area as dark blue.

Frame 2 might generate a slightly different texture.

Frame 3 might introduce another variation.

Each frame may look acceptable individually.

But when played together, those tiny differences create obvious flickering.

So a useful video text removal system needs more than spatial consistency.

It also needs temporal consistency.

My Initial Architecture

I wanted the product architecture to stay relatively simple.

The current system roughly looks like this:

Browser
  ↓
Next.js Application
  ↓
Upload Video
  ↓
Cloud Object Storage
  ↓
Create Processing Job
  ↓
Video Inpainting Service
  ↓
Poll Job Status
  ↓
Processed Video
  ↓
Download Result

The frontend and application layer are built with Next.js.

Large video files are uploaded to object storage instead of being passed through the application server itself.

This is important because sending large videos through serverless application endpoints creates several problems:

request size limits
function timeouts
unnecessary bandwidth usage
increased memory usage
higher infrastructure cost

Instead, the application generates an upload target and lets the browser upload the file directly.

The processing service then works from the stored video.

Manual Selection vs Automatic Detection

One of the first product decisions I had to make was how users should tell the system what to remove.

There are two approaches.

1. Automatic text detection

The ideal experience is:

Upload video
↓
Detect text automatically
↓
Remove detected text
↓
Download

This is obviously the easiest experience for users.

But automatic detection introduces another difficult problem.

The system has to determine:

what is actually text
which text should be removed
where it appears
whether it moves
whether the bounding box changes
whether text appears only during part of the video

OCR models can detect text quite well, but video creates edge cases.

For example:

Scoreboard      → probably text
Street sign     → text, but maybe part of the scene
Subtitle        → probably remove
T-shirt logo    → maybe not
Phone screen    → depends on user intent

Detection and intent are not the same thing.

2. Manual area selection

The second approach is much simpler.

The user draws a rectangle around the area that should be removed.

For example:

┌──────────────────────────────┐
│                              │
│          VIDEO               │
│                              │
│   ┌──────────────────────┐   │
│   │     subtitle area    │   │
│   └──────────────────────┘   │
│                              │
└──────────────────────────────┘

This removes a lot of ambiguity.

The user already knows what they want removed.

The system only needs to focus on reconstructing the selected area.

For many real-world cases, especially subtitles, the text stays inside roughly the same region for most of the video.

That makes manual selection surprisingly effective.

For this reason, I decided to support both workflows:

automatic detection for convenience
manual selection for control

Representing the Selected Region

The browser displays the video at a different size than its actual resolution.

For example, the uploaded video might be:

1920 × 1080

while the preview shown in the browser might be:

960 × 540

If the user draws a selection box on the preview, the coordinates need to be converted back to the original video coordinate system.

A simple version looks like this:

const scaleX = originalWidth / previewWidth;
const scaleY = originalHeight / previewHeight;

const actualX = selectedX * scaleX;
const actualY = selectedY * scaleY;
const actualWidth = selectedWidth * scaleX;
const actualHeight = selectedHeight * scaleY;

There is another approach that I prefer even more: storing coordinates as normalized values.

For example:

type Region = {
  x: number;
  y: number;
  width: number;
  height: number;
};

Where every value is between 0 and 1.

A selection might look like:

{
  "x": 0.12,
  "y": 0.76,
  "width": 0.74,
  "height": 0.14
}

This makes the mask independent of the preview resolution.

Later, it can be converted into pixels for any video size.

Video Processing Should Be Asynchronous

Video processing can easily take minutes.

That means a normal synchronous API request is not appropriate.

Instead of doing this:

POST /remove-text
↓
wait...
wait...
wait...
↓
return processed video

I use a job-style workflow:

POST /jobs
↓
jobId

Then the frontend checks the status:

GET /jobs/:jobId

The response may look like:

{
  "status": "processing",
  "progress": 62
}

And eventually:

{
  "status": "completed",
  "resultUrl": "..."
}

This architecture has several advantages.

If the browser tab refreshes, the processing job can continue.

If processing fails, the backend can retry.

If the external processing service is temporarily unavailable, jobs can be queued instead of immediately failing.

The Real Challenge: Temporal Consistency

The hardest part is not detecting the text.

It is reconstructing the background convincingly over time.

Consider a camera moving horizontally.

A subtitle covers part of the ground.

The missing region might contain:

Frame 1: grass
Frame 2: grass + shadow
Frame 3: edge of a road
Frame 4: road

Simply running an image inpainting model independently on each frame can produce unstable results.

Video inpainting systems typically try to use information from nearby frames.

A simplified idea is:

Previous frames
      ↓
Current masked frame
      ↑
Future frames
      ↓
Temporal reconstruction

If the background behind the text is visible in nearby frames, the model can use that information to reconstruct the missing area more consistently.

This is why video-specific models tend to produce better results than simply applying an image model frame by frame.

Why Some Videos Work Much Better Than Others

One thing that became obvious very quickly is that the difficulty varies enormously depending on the scene.

Easier cases

Text over:

static backgrounds
walls
sky
blurred backgrounds
simple textures
areas with little motion

usually works quite well.

For example:

Static interview
+
subtitle at the bottom
+
mostly blurred background

→ relatively easy

Harder cases

Things become much more difficult when the text overlaps:

faces
hands
complex clothing
fast-moving objects
detailed textures
camera cuts
animations
particles
rapidly changing scenes

Imagine text covering someone's fingers while they are moving.

The model has to reconstruct not only the appearance of the fingers, but also their motion across multiple frames.

That is a much harder problem.

Masks Matter More Than I Expected

Another lesson was that the mask itself has a large impact on output quality.

A mask that is too small leaves fragments of text behind.

A mask that is too large forces the model to reconstruct unnecessary parts of the image.

For example:

Bad:

┌─────────────────────────────┐
│                             │
│         remove me           │
│            ───              │
│           mask              │
└─────────────────────────────┘

Only covering the center of the letters can leave visible edges.

A slightly larger region often works better:

┌─────────────────────────────┐
│                             │
│      ┌───────────────┐      │
│      │   remove me   │      │
│      └───────────────┘      │
│                             │
└─────────────────────────────┘

But making the region dramatically larger than necessary also increases the chance of visual artifacts.

The best mask usually has a small amount of padding around the text.

Handling Multiple Text Areas

Some videos contain more than one thing to remove.

For example:

┌─────────────────────────────┐
│ USERNAME                    │
│                             │
│                             │
│                             │
│        subtitles            │
└─────────────────────────────┘

Supporting only one rectangle becomes restrictive very quickly.

So I added support for multiple selection regions.

Conceptually:

const regions = [
  {
    x: 0.03,
    y: 0.05,
    width: 0.2,
    height: 0.08
  },
  {
    x: 0.15,
    y: 0.78,
    width: 0.7,
    height: 0.12
  }
];

These regions can later be converted into one combined mask.

This turned out to be useful for:

logos + subtitles
username + watermark
timestamp + caption
several static overlays

Cost Is a Product Problem Too

Video AI processing is significantly more expensive than many image AI tasks.

A single image might require one inference.

A video may contain thousands of frames.

For example, a 30-second video at 30 FPS contains:

30 × 30 = 900 frames

A five-minute video contains:

5 × 60 × 30 = 9,000 frames

Obviously, video models do not necessarily process every frame independently, but the scale difference explains why video inference can become expensive.

This affects the product design.

You need to think about:

maximum video duration
maximum file size
processing resolution
job concurrency
storage lifecycle
retries
failed jobs
abuse prevention

These infrastructure decisions matter almost as much as the AI model itself.

Temporary Storage Is Important

Uploaded videos can be large.

Keeping every original and processed file forever would quickly become expensive.

A better lifecycle is:

Upload
↓
Process
↓
User downloads result
↓
Temporary retention period
↓
Automatic deletion

This keeps storage usage predictable.

It also reduces the amount of user data retained by the service.

Error Handling Matters More with AI APIs

Traditional APIs are often relatively deterministic.

AI inference APIs are different.

A request may fail because of:

temporary provider errors
model startup time
GPU availability
timeout
unsupported codec
malformed video
unexpectedly large files

So the processing layer needs to distinguish between retryable and non-retryable errors.

Something like:

if (isTemporaryError(error)) {
  await retry(job);
} else {
  await markJobAsFailed(job);
}

Retry logic should also have limits.

Otherwise a broken video may be processed forever.

A more realistic pattern is:

Attempt 1
↓
Failed

Wait

Attempt 2
↓
Failed

Wait longer

Attempt 3
↓
Failed

Mark job as failed

UX Is Part of the Technical Problem

When a task takes several minutes, showing a spinner is not enough.

Users want to know whether something is actually happening.

Even if the underlying provider does not expose precise frame-level progress, showing processing stages helps:

Uploading video
      ↓
Preparing video
      ↓
Removing text
      ↓
Generating final video
      ↓
Completed

This sounds like a small detail, but it makes the application feel much more reliable.

Long-running AI workflows need visible state.

What I Would Improve Next

There are still many things I want to improve.

Better automatic detection

The ideal system would track text over time instead of simply detecting text in isolated frames.

Conceptually:

OCR
↓
Bounding box
↓
Track across frames
↓
Determine lifetime
↓
Generate temporal mask

This would make automatic removal much more precise.

Better handling of moving watermarks

Static subtitles are relatively easy because their position rarely changes.

Moving logos and animated overlays are much harder.

Tracking them frame by frame would allow the mask to move with the target.

Scene-aware processing

A five-minute video might contain dozens of scene changes.

Instead of treating the entire video as one continuous sequence, the system could detect cuts:

Video
↓
Scene detection
↓
Scene 1
Scene 2
Scene 3
...
↓
Process separately
↓
Merge

This may improve consistency and reduce unnecessary context.

Preview before full processing

Another useful feature would be processing only a few seconds first.

Users could verify:

whether the selected region is correct
whether the model handles the background well
whether the result is acceptable

before spending time and compute processing the entire video.

The Biggest Lesson

Before building this project, I thought the core problem would be:

How do I remove text from a video?

After working on it, I realized the actual problem is closer to:

How do I reconstruct missing pixels across time while keeping the result visually consistent?

Text detection is only one part of the system.

The full problem includes:

coordinate systems
masks
uploads
object storage
asynchronous jobs
video codecs
AI inference
retries
temporal consistency
cost control
UX

That is what made this project much more interesting than I initially expected.

I packaged the current version into a small web tool called Video Text Remover:

https://videotextremover.org

It currently focuses on removing visible text and overlays from videos using automatic detection or manually selected regions.

There is still a lot to improve, especially around moving text and difficult backgrounds, but building it has been a useful exploration of what production video AI actually looks like beyond a simple model demo.

If you're also building image or video AI tools, I'd be interested to hear how you're handling long-running inference jobs, storage costs, and temporal consistency.

Tags

#webdev #nextjs #ai #machinelearning

DEV Community: Lim Geng

How I Built an AI Video Text Remover with Next.js and Video Inpainting

The Problem: Text Is Part of the Pixels

Image Inpainting Is Not Enough

My Initial Architecture

Manual Selection vs Automatic Detection

1. Automatic text detection

2. Manual area selection

Representing the Selected Region

Video Processing Should Be Asynchronous

The Real Challenge: Temporal Consistency

Why Some Videos Work Much Better Than Others

Easier cases

Harder cases

Masks Matter More Than I Expected

Handling Multiple Text Areas

Cost Is a Product Problem Too

Temporary Storage Is Important

Error Handling Matters More with AI APIs

UX Is Part of the Technical Problem

What I Would Improve Next

Better automatic detection

Better handling of moving watermarks

Scene-aware processing

Preview before full processing

The Biggest Lesson