Removing text without removing the wall behind it: EAST quads to an inpaint mask

You want to wipe a sign off a wall in a video. A subtitle, a watermark, a logo painted on brick. The hard part is not finding the text. The hard part is removing only the text and leaving the brick alone. Mask one pixel too wide and the inpainter starts inventing wall where there was wall already, and now the patch looks worse than the text did.

So the whole job comes down to one question: how tight is the mask. This is a walkthrough of how Wunjo Make builds that mask, straight from the code in visual_processing/segmentation/east/. I went in expecting one architecture and found a simpler one, so I will also tell you where the real code path differs from the version you might have read about.

The interesting part of text removal is not the detector. It is the few lines that turn boxes into a mask the inpainter can trust. Photo: Unsplash.

The shape of the problem

Think of the detector as a person with a highlighter. A bounding box is the highlighter held straight, so a slanted street sign gets a fat rectangle with a lot of wall caught in the corners. A rotated quadrilateral is the highlighter tilted to follow the text, so the marked area hugs the letters. EAST gives you the tilted highlighter. That tilt is the whole reason it is worth using here.

EAST stands for Efficient and Accurate Scene Text detector (Zhou et al., CVPR 2017). It is a single fully-convolutional network. No region proposals, no per-character steps. Feed it an image, get back two maps the same shape as a downscaled version of the image: a score map that says "text is here" and a geometry map that says "and the box around it looks like this."

Step 1. The network outputs a score and a geometry, not boxes

The model itself is small. A VGG16 backbone pulls features at several scales, a merge branch fuses them back up to a quarter of the input resolution, and an output head produces three things. Here is the head, from east/model.py:

def forward(self, x):
    score = self.sigmoid1(self.conv1(x))
    loc   = self.sigmoid2(self.conv2(x)) * self.scope          # 4 channels
    angle = (self.sigmoid3(self.conv3(x)) - 0.5) * math.pi     # 1 channel
    geo = torch.cat((loc, angle), 1)
    return score, geo

Read those three lines slowly, because they are the design.

score is one channel through a sigmoid, so every pixel gets a 0-to-1 number for "is this text." loc is four channels, each one a distance, scaled by self.scope which is 512. For a text pixel, those four numbers are the distance up, down, left, and right to the edges of its text box. angle is one channel mapped into the range minus pi-over-two to pi-over-two, the rotation of that box.

So the network never draws a box. It labels each text pixel with four distances and an angle. The box is implied. That is the trick that lets one convolution describe a rotated rectangle: a center pixel plus four reach values plus a twist.

Step 2. Rebuild rotated quads from the geometry

Now you have to turn "distance up, down, left, right, and an angle" back into four corner points. That is restore_polys in east/detect.py. For each pixel that scored high enough:

y_min = y - d[0, i]
y_max = y + d[1, i]
x_min = x - d[2, i]
x_max = x + d[3, i]
rotate_mat = get_rotate_mat(-angle[i])

temp_x = np.array([[x_min, x_max, x_max, x_min]]) - x
temp_y = np.array([[y_min, y_min, y_max, y_max]]) - y
coordidates = np.concatenate((temp_x, temp_y), axis=0)
res = np.dot(rotate_mat, coordidates)
res[0, :] += x
res[1, :] += y

In plain terms: build an axis-aligned rectangle from the four distances, move it so the text pixel is the origin, rotate it by the predicted angle, then move it back. Out come four corner points. The result row is the eight numbers of a quadrilateral, [x0,y0, x1,y1, x2,y2, x3,y3], the corners in order.

Two details that matter in practice. The score threshold and the candidate filter both live here:

xy_text = np.argwhere(score > score_thresh)   # score_thresh defaults to 0.9

A 0.9 threshold is high. It throws away anything the network is not very sure about. For text removal that is the right bias. A false positive means you erase a chunk of real image, which is much worse than missing a faint logo. And is_valid_poly quietly drops any quad that lands mostly outside the frame, so edge garbage does not become a mask.

Every high-scoring pixel proposes its own quad, so a single word produces hundreds of nearly identical quads. The code carries a confidence score in the ninth column of each box for exactly this reason. Worth knowing: in this code path the boxes go to the mask without a separate non-max-suppression merge, so you are relying on the high score threshold and the fact that overlapping filled polygons just paint the same white pixels twice. Painting white twice costs nothing.

Step 3. Rasterize the quads into a mask

Here is where the boxes stop being numbers and become a mask. plot_filled_mask:

def plot_filled_mask(img, boxes):
    mask = Image.new('RGB', img.size, (0, 0, 0))   # black canvas
    draw = ImageDraw.Draw(mask)
    if boxes is not None:
        for box in boxes:
            draw.polygon(
                [box[0], box[1], box[2], box[3],
                 box[4], box[5], box[6], box[7]],
                fill=(255, 255, 255))
    return mask

Black image the size of the frame, then every detected quad gets painted solid white. White means "inpaint this," black means "leave it." Because the quads are tilted, the white regions follow slanted and rotated text instead of boxing it in a wide rectangle. That is the wall-saving step. A straight bounding box around a diagonal sign would whitewash the triangles of brick in the corners, and the inpainter would happily repaint brick it did not need to touch.

The full detect call wraps the network and this rasterizer together, in SegmentText.detect_text:

def detect_text(self, img_path, max_resolution=1280):
    img = Image.open(img_path)
    w, h = img.size
    if max(w, h) > max_resolution:                # cap the long side
        scale = max_resolution / max(w, h)
        img = img.resize((int(w*scale), int(h*scale)))
    img, _, _ = resize_img(img)                   # make dims divisible by 32
    boxes = detect(img, self.model, self.device)
    filled_mask = plot_filled_mask(img, boxes)
    return filled_mask.resize((w, h))             # back to original size

The resize_img step is not cosmetic. The VGG backbone halves the resolution five times, so the input has to be divisible by 32 or the feature maps will not line up. The 1280 cap is there so a 4K frame does not blow up memory in the detector for no gain.

Step 4. The optional crop, for "only this corner"

If the user drew a box around just the subtitle area, you do not want to erase a logo elsewhere in the frame. SegmentText.crop handles that by keeping only the part of the mask inside the chosen rectangle:

@staticmethod
def crop(filled_mask, text_area: list = None):
    if text_area is not None:
        x1, y1, x2, y2 = text_area
        img = Image.new('L', filled_mask.size, color=0)    # all black
        region = filled_mask.crop((x1, y1, x2, y2))        # cut the user's box
        img.paste(region, (x1, y1))                        # paste it back, rest stays black
        return img
    return filled_mask

It does not crop the image down. It blacks out everything outside the user's rectangle and keeps the detected text inside it. So you still get tight quads, just restricted to the region you care about.

Step 5. Hand the mask to inpainting (and where SAM actually sits)

Now the part where I corrected my own mental model. I expected the EAST quads to become point or box prompts for SAM (Segment Anything, Kirillov et al., ICCV 2023), letting SAM tighten the mask further. That is a reasonable design. It is not this code.

In visual_processing/inference.py, the remove path is one function that handles two jobs. When the user clicks an object, it runs SAM. When the user asks to remove text, it runs EAST. They are siblings, not a chain. The text branch is gated behind use_remove_text, and SAM is explicitly skipped there:

if predictor is None and not use_remove_text:
    predictor = segmentation.init_vit(sam_vit_checkpoint, vit_model_type, device)
else:
    lprint(msg=f"[{method}] Model predictor will not be loaded "
               f"because of not need to use for text.", ...)

The text branch builds its own detector and writes the EAST mask straight to disk per frame:

segment_text = SegmentText(device=device, vgg16_path=vgg16_baseline_path, east_path=vgg16_east_path)
...
mask_text_frame = segment_text.detect_text(img_path=frame_file_path)
mask_textarea_frame = segment_text.crop(mask_text_frame, coord)
mask_textarea_frame.save(mask_text_frame_path)

So the real flow for text is EAST to mask to inpaint, no SAM in between. SAM is the tool for "remove this person," EAST is the tool for "remove this text," and they share plumbing because both of them produce a white-on-black mask that the same inpainter consumes.

The inpainter is chosen by source. Video on a capable GPU goes to ProPainter (Zhou et al., ICCV 2023), which propagates pixels across frames so the filled region stays consistent over time. Images, or anything on CPU, go to a single-frame inpainting model. Either way, the mask reaches it after one more cleanup. The white mask is dilated and binarized:

binary_mask = cv2.threshold(segment_mask, 128, 1, cv2.THRESH_BINARY)[1]
...
kernel = np.ones((kernel_size, kernel_size), np.uint8)   # kernel_size = 10
mask_to_save = cv2.dilate(mask_to_save, kernel, iterations=1)

The dilation by a 10-pixel kernel is deliberate. EAST hugs the glyphs tightly, but ink has a faint halo, anti-aliasing and shadow, just outside the letter shape. If the mask is exactly the glyph, the inpainter leaves a ghost outline. Growing the mask by a few pixels eats that halo. So the pipeline goes tight, then loosens by a known amount, instead of starting loose and hoping.

Finally the inpainter only writes inside the mask. From process_remover:

merge = generated * mask_raw_3ch + img_raw * (1 - mask_raw_3ch)

Generated pixels where the mask is white, original pixels everywhere else. That single line is the promise that the wall behind the text is the exact wall that was there. The model can hallucinate freely inside the white region and it will never touch a pixel outside it.

---

About the author. I'm Wlad Radchenko, a software engineer. The code in this article comes from Wunjo Make (open source), local software for video makers, and Wunjo Design, an offline PWA for designers. Get in touch to find more on GitHub and LinkedIn.

---

Gotchas you will actually hit

A few things I would warn a teammate about before they touch this path.

The 0.9 score threshold is conservative on purpose, and it will miss low-contrast or stylized text. If you lower it, you trade missed text for erased background. For removal, keep it high and let the user draw a text_area box for the stubborn cases instead of dropping the threshold globally.

Divisibility by 32 is silent. If you bypass resize_img and feed an arbitrary size, the merge branch upsamples will not match the skip connections and you get a shape error deep in the model. Always route through resize_img.

The mask is per frame, recomputed independently for each frame. There is no temporal smoothing on the text mask itself. For video, the consistency comes later, from ProPainter propagating the fill, not from the detector. If your text flickers in and out of detection, that flicker can show up before propagation evens it out.

Dilation cuts both ways. The 10-pixel grow is great for eating halos around small subtitles. On very large text it can swallow a sliver of background you wanted to keep. The kernel size is a constant in the code worth tuning for your content.

Takeaway

Removing text without removing the wall is a masking problem, and the mask quality is set in three short functions. The network emits a score map plus four distances and an angle per pixel. restore_polys rebuilds rotated quads from that geometry so the marked area follows slanted text. plot_filled_mask paints those quads white on black. Then a 10-pixel dilation eats the ink halo, and the inpainter writes only inside the white. SAM is in the same file but it is the object-removal sibling, not a stage in the text path.

The code is in visual_processing/segmentation/east/ in the Wunjo Make repo. If you are fighting ghost outlines after text removal, start at the dilation kernel and the score threshold. Those two constants decide whether the wall survives.

References

• Zhou, Yao, Wen, et al. "EAST: An Efficient and Accurate Scene Text Detector." CVPR 2017. arXiv:1704.03155
• Kirillov et al. "Segment Anything." ICCV 2023. arXiv:2304.02643
• Zhou, Li, Loy, et al. "ProPainter: Improving Propagation and Transformer for Video Inpainting." ICCV 2023. arXiv:2309.03897