DEV Community

Cover image for How I Built an AI Video Text Remover with Next.js and Video Inpainting
Lim Geng
Lim Geng

Posted on AI-assisted

How I Built an AI Video Text Remover with Next.js and Video Inpainting

#webdev #ai #nextjs #machinelearning

Removing text from an image is relatively straightforward.

Removing text from a video is a very different problem.

A video is not just a collection of independent images. Every frame has to remain visually consistent with the frames before and after it. If the reconstructed background changes slightly from frame to frame, the result starts to flicker immediately.

Over the past few weeks, I have been building Video Text Remover, a small web tool for removing subtitles, captions, watermarks, timestamps, and other burned-in text from videos.

You can try it here:

https://videotextremover.org

In this post, I want to share how I approached the problem, the architecture behind the tool, and a few things I learned about video inpainting along the way.

The Problem: Text Is Part of the Pixels

There are two very different types of subtitles in video.

The first type is a separate subtitle track, such as an .srt file or an embedded subtitle stream.

Those are easy to remove.

You can simply disable or strip the subtitle track.

The second type is burned-in text.

Examples include:

  • captions rendered directly into the video
  • TikTok-style subtitles
  • timestamps
  • channel logos
  • watermarks
  • usernames
  • presentation text
  • labels added during editing

In this case, there is no separate subtitle layer.

The text is literally part of the image.

Removing it means answering a much harder question:

What should the pixels behind the text look like?

That turns the problem into an inpainting problem.

Image Inpainting Is Not Enough

For a single image, an inpainting model receives something like:

Original image
      +
Mask indicating the area to remove
      ↓
Inpainting model
      ↓
Reconstructed image
Enter fullscreen mode Exit fullscreen mode

If the object being removed is small, modern models can often reconstruct the missing background surprisingly well.

Video adds another dimension: time.

Imagine a subtitle covering part of someone's shirt.

Frame 1 might reconstruct the missing area as dark blue.

Frame 2 might generate a slightly different texture.

Frame 3 might introduce another variation.

Each frame may look acceptable individually.

But when played together, those tiny differences create obvious flickering.

So a useful video text removal system needs more than spatial consistency.

It also needs temporal consistency.

My Initial Architecture

I wanted the product architecture to stay relatively simple.

The current system roughly looks like this:

Browser
  ↓
Next.js Application
  ↓
Upload Video
  ↓
Cloud Object Storage
  ↓
Create Processing Job
  ↓
Video Inpainting Service
  ↓
Poll Job Status
  ↓
Processed Video
  ↓
Download Result
Enter fullscreen mode Exit fullscreen mode

The frontend and application layer are built with Next.js.

Large video files are uploaded to object storage instead of being passed through the application server itself.

This is important because sending large videos through serverless application endpoints creates several problems:

  • request size limits
  • function timeouts
  • unnecessary bandwidth usage
  • increased memory usage
  • higher infrastructure cost

Instead, the application generates an upload target and lets the browser upload the file directly.

The processing service then works from the stored video.

Manual Selection vs Automatic Detection

One of the first product decisions I had to make was how users should tell the system what to remove.

There are two approaches.

1. Automatic text detection

The ideal experience is:

Upload video
↓
Detect text automatically
↓
Remove detected text
↓
Download
Enter fullscreen mode Exit fullscreen mode

This is obviously the easiest experience for users.

But automatic detection introduces another difficult problem.

The system has to determine:

  • what is actually text
  • which text should be removed
  • where it appears
  • whether it moves
  • whether the bounding box changes
  • whether text appears only during part of the video

OCR models can detect text quite well, but video creates edge cases.

For example:

Scoreboard      → probably text
Street sign     → text, but maybe part of the scene
Subtitle        → probably remove
T-shirt logo    → maybe not
Phone screen    → depends on user intent
Enter fullscreen mode Exit fullscreen mode

Detection and intent are not the same thing.

2. Manual area selection

The second approach is much simpler.

The user draws a rectangle around the area that should be removed.

For example:

┌──────────────────────────────┐
│                              │
│          VIDEO               │
│                              │
│   ┌──────────────────────┐   │
│   │     subtitle area    │   │
│   └──────────────────────┘   │
│                              │
└──────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

This removes a lot of ambiguity.

The user already knows what they want removed.

The system only needs to focus on reconstructing the selected area.

For many real-world cases, especially subtitles, the text stays inside roughly the same region for most of the video.

That makes manual selection surprisingly effective.

For this reason, I decided to support both workflows:

  • automatic detection for convenience
  • manual selection for control

Representing the Selected Region

The browser displays the video at a different size than its actual resolution.

For example, the uploaded video might be:

1920 × 1080
Enter fullscreen mode Exit fullscreen mode

while the preview shown in the browser might be:

960 × 540
Enter fullscreen mode Exit fullscreen mode

If the user draws a selection box on the preview, the coordinates need to be converted back to the original video coordinate system.

A simple version looks like this:

const scaleX = originalWidth / previewWidth;
const scaleY = originalHeight / previewHeight;

const actualX = selectedX * scaleX;
const actualY = selectedY * scaleY;
const actualWidth = selectedWidth * scaleX;
const actualHeight = selectedHeight * scaleY;
Enter fullscreen mode Exit fullscreen mode

There is another approach that I prefer even more: storing coordinates as normalized values.

For example:

type Region = {
  x: number;
  y: number;
  width: number;
  height: number;
};
Enter fullscreen mode Exit fullscreen mode

Where every value is between 0 and 1.

A selection might look like:

{
  "x": 0.12,
  "y": 0.76,
  "width": 0.74,
  "height": 0.14
}
Enter fullscreen mode Exit fullscreen mode

This makes the mask independent of the preview resolution.

Later, it can be converted into pixels for any video size.

Video Processing Should Be Asynchronous

Video processing can easily take minutes.

That means a normal synchronous API request is not appropriate.

Instead of doing this:

POST /remove-text
↓
wait...
wait...
wait...
↓
return processed video
Enter fullscreen mode Exit fullscreen mode

I use a job-style workflow:

POST /jobs
↓
jobId
Enter fullscreen mode Exit fullscreen mode

Then the frontend checks the status:

GET /jobs/:jobId
Enter fullscreen mode Exit fullscreen mode

The response may look like:

{
  "status": "processing",
  "progress": 62
}
Enter fullscreen mode Exit fullscreen mode

And eventually:

{
  "status": "completed",
  "resultUrl": "..."
}
Enter fullscreen mode Exit fullscreen mode

This architecture has several advantages.

If the browser tab refreshes, the processing job can continue.

If processing fails, the backend can retry.

If the external processing service is temporarily unavailable, jobs can be queued instead of immediately failing.

The Real Challenge: Temporal Consistency

The hardest part is not detecting the text.

It is reconstructing the background convincingly over time.

Consider a camera moving horizontally.

A subtitle covers part of the ground.

The missing region might contain:

Frame 1: grass
Frame 2: grass + shadow
Frame 3: edge of a road
Frame 4: road
Enter fullscreen mode Exit fullscreen mode

Simply running an image inpainting model independently on each frame can produce unstable results.

Video inpainting systems typically try to use information from nearby frames.

A simplified idea is:

Previous frames
      ↓
Current masked frame
      ↑
Future frames
      ↓
Temporal reconstruction
Enter fullscreen mode Exit fullscreen mode

If the background behind the text is visible in nearby frames, the model can use that information to reconstruct the missing area more consistently.

This is why video-specific models tend to produce better results than simply applying an image model frame by frame.

Why Some Videos Work Much Better Than Others

One thing that became obvious very quickly is that the difficulty varies enormously depending on the scene.

Easier cases

Text over:

  • static backgrounds
  • walls
  • sky
  • blurred backgrounds
  • simple textures
  • areas with little motion

usually works quite well.

For example:

Static interview
+
subtitle at the bottom
+
mostly blurred background

→ relatively easy
Enter fullscreen mode Exit fullscreen mode

Harder cases

Things become much more difficult when the text overlaps:

  • faces
  • hands
  • complex clothing
  • fast-moving objects
  • detailed textures
  • camera cuts
  • animations
  • particles
  • rapidly changing scenes

Imagine text covering someone's fingers while they are moving.

The model has to reconstruct not only the appearance of the fingers, but also their motion across multiple frames.

That is a much harder problem.

Masks Matter More Than I Expected

Another lesson was that the mask itself has a large impact on output quality.

A mask that is too small leaves fragments of text behind.

A mask that is too large forces the model to reconstruct unnecessary parts of the image.

For example:

Bad:

┌─────────────────────────────┐
│                             │
│         remove me           │
│            ───              │
│           mask              │
└─────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Only covering the center of the letters can leave visible edges.

A slightly larger region often works better:

┌─────────────────────────────┐
│                             │
│      ┌───────────────┐      │
│      │   remove me   │      │
│      └───────────────┘      │
│                             │
└─────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

But making the region dramatically larger than necessary also increases the chance of visual artifacts.

The best mask usually has a small amount of padding around the text.

Handling Multiple Text Areas

Some videos contain more than one thing to remove.

For example:

┌─────────────────────────────┐
│ USERNAME                    │
│                             │
│                             │
│                             │
│        subtitles            │
└─────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Supporting only one rectangle becomes restrictive very quickly.

So I added support for multiple selection regions.

Conceptually:

const regions = [
  {
    x: 0.03,
    y: 0.05,
    width: 0.2,
    height: 0.08
  },
  {
    x: 0.15,
    y: 0.78,
    width: 0.7,
    height: 0.12
  }
];
Enter fullscreen mode Exit fullscreen mode

These regions can later be converted into one combined mask.

This turned out to be useful for:

  • logos + subtitles
  • username + watermark
  • timestamp + caption
  • several static overlays

Cost Is a Product Problem Too

Video AI processing is significantly more expensive than many image AI tasks.

A single image might require one inference.

A video may contain thousands of frames.

For example, a 30-second video at 30 FPS contains:

30 × 30 = 900 frames
Enter fullscreen mode Exit fullscreen mode

A five-minute video contains:

5 × 60 × 30 = 9,000 frames
Enter fullscreen mode Exit fullscreen mode

Obviously, video models do not necessarily process every frame independently, but the scale difference explains why video inference can become expensive.

This affects the product design.

You need to think about:

  • maximum video duration
  • maximum file size
  • processing resolution
  • job concurrency
  • storage lifecycle
  • retries
  • failed jobs
  • abuse prevention

These infrastructure decisions matter almost as much as the AI model itself.

Temporary Storage Is Important

Uploaded videos can be large.

Keeping every original and processed file forever would quickly become expensive.

A better lifecycle is:

Upload
↓
Process
↓
User downloads result
↓
Temporary retention period
↓
Automatic deletion
Enter fullscreen mode Exit fullscreen mode

This keeps storage usage predictable.

It also reduces the amount of user data retained by the service.

Error Handling Matters More with AI APIs

Traditional APIs are often relatively deterministic.

AI inference APIs are different.

A request may fail because of:

  • temporary provider errors
  • model startup time
  • GPU availability
  • timeout
  • unsupported codec
  • malformed video
  • unexpectedly large files

So the processing layer needs to distinguish between retryable and non-retryable errors.

Something like:

if (isTemporaryError(error)) {
  await retry(job);
} else {
  await markJobAsFailed(job);
}
Enter fullscreen mode Exit fullscreen mode

Retry logic should also have limits.

Otherwise a broken video may be processed forever.

A more realistic pattern is:

Attempt 1
↓
Failed

Wait

Attempt 2
↓
Failed

Wait longer

Attempt 3
↓
Failed

Mark job as failed
Enter fullscreen mode Exit fullscreen mode

UX Is Part of the Technical Problem

When a task takes several minutes, showing a spinner is not enough.

Users want to know whether something is actually happening.

Even if the underlying provider does not expose precise frame-level progress, showing processing stages helps:

Uploading video
      ↓
Preparing video
      ↓
Removing text
      ↓
Generating final video
      ↓
Completed
Enter fullscreen mode Exit fullscreen mode

This sounds like a small detail, but it makes the application feel much more reliable.

Long-running AI workflows need visible state.

What I Would Improve Next

There are still many things I want to improve.

Better automatic detection

The ideal system would track text over time instead of simply detecting text in isolated frames.

Conceptually:

OCR
↓
Bounding box
↓
Track across frames
↓
Determine lifetime
↓
Generate temporal mask
Enter fullscreen mode Exit fullscreen mode

This would make automatic removal much more precise.

Better handling of moving watermarks

Static subtitles are relatively easy because their position rarely changes.

Moving logos and animated overlays are much harder.

Tracking them frame by frame would allow the mask to move with the target.

Scene-aware processing

A five-minute video might contain dozens of scene changes.

Instead of treating the entire video as one continuous sequence, the system could detect cuts:

Video
↓
Scene detection
↓
Scene 1
Scene 2
Scene 3
...
↓
Process separately
↓
Merge
Enter fullscreen mode Exit fullscreen mode

This may improve consistency and reduce unnecessary context.

Preview before full processing

Another useful feature would be processing only a few seconds first.

Users could verify:

  • whether the selected region is correct
  • whether the model handles the background well
  • whether the result is acceptable

before spending time and compute processing the entire video.

The Biggest Lesson

Before building this project, I thought the core problem would be:

How do I remove text from a video?

After working on it, I realized the actual problem is closer to:

How do I reconstruct missing pixels across time while keeping the result visually consistent?

Text detection is only one part of the system.

The full problem includes:

  • coordinate systems
  • masks
  • uploads
  • object storage
  • asynchronous jobs
  • video codecs
  • AI inference
  • retries
  • temporal consistency
  • cost control
  • UX

That is what made this project much more interesting than I initially expected.

I packaged the current version into a small web tool called Video Text Remover:

https://videotextremover.org

It currently focuses on removing visible text and overlays from videos using automatic detection or manually selected regions.

There is still a lot to improve, especially around moving text and difficult backgrounds, but building it has been a useful exploration of what production video AI actually looks like beyond a simple model demo.

If you're also building image or video AI tools, I'd be interested to hear how you're handling long-running inference jobs, storage costs, and temporal consistency.

Tags

#webdev #nextjs #ai #machinelearning

Top comments (1)

Collapse
 
aisovereign profile image
Ananya Soni •

Solid approach to the temporal consistency problem. I spend my time building AGI and multimodal architectures, so I respect the technical depth here. It actually fixes a real problem for everyday users instead of just talking about one. Maybe write a slightly shorter sequel next time so people actually finish it.