Two enhancers, one frame: routing faces to GFPGAN and the rest to Real-ESRGAN

Run a general upscaler on a portrait and the face comes back plastic. Skin loses its pores, eyes go glassy, the whole face reads as smoothed-over wax. Run a face restorer on the same photo and the face looks great, but point it at the wall behind the person and it either ignores the wall or smears it, because it never learned what a wall is. Neither model is broken. They were trained for different jobs.

The fix in Wunjo Make is to not ask one model to do both. The enhancement code detects faces, restores each face on its own, and leaves the rest of the frame to a general upscaler. This is a walkthrough of how that split is wired, from the content_enhancer entry point down into GFPGANer.enhance, where the actual face-versus-background routing happens.

I am not covering Real-ESRGAN's tiling here. That has its own article. This one is only about which pixels go to which model and how they get glued back together.

A face and the room behind it are two different restoration problems in the same picture. Photo: Unsplash.

The entry point is a method switch, not magic

content_enhancer takes a method string and picks a restorer. It is a plain branch:

def content_enhancer(media_path, save_folder, method='gfpgan', device='cpu', fps=30, progress_callback=None):
    ...
    if method == 'gfpgan':
        from .gfpganer import GFPGANer
        restorer = GFPGANer(
            model_path=model_path,
            root_dir=local_model_path,
            upscale=1,
            arch="clean",
            channel_multiplier=2,
            bg_upsampler=None,
            device=device
        )
    elif method == 'realesrgan':
        from .realesrgan import RealESRGANer
        ...
        restorer = RealESRGANer(scale=4, model_path=models_path, ...)

So at the top level the choice is per-run: you either run the face path (gfpgan) or the general path (realesrgan / animesgan) on the whole media. The interesting routing, the part where one frame gets carved into "face" and "not face," lives one level down, inside the GFPGAN path.

Notice bg_upsampler=None in the GFPGAN branch. That flag is the seam between the two models, and I will come back to it. Notice also upscale=1. The face path here is restoration, not enlargement: it cleans the face up at the same size.

How a frame gets enhanced

The loop is the same for image or video. For each frame, GFPGAN gets the whole frame and three flags:

_, _, output = restorer.enhance(frame, has_aligned=False, only_center_face=False, paste_back=True)

Read those flags in plain English:

• has_aligned=False: the input is a normal photo, not a pre-cropped face. So GFPGAN has to go find the faces itself.
• only_center_face=False: restore every face it finds, not just the biggest one in the middle.
• paste_back=True: after restoring the faces, put them back into the original frame, do not just hand back loose crops.

Those three flags are the routing decision. Everything else is GFPGANer.enhance carrying it out.

Step 1. Find the faces

The first thing enhance does, when the input is not already an aligned crop, is detect faces and align them:

self.face_helper.read_image(img)
self.face_helper.get_face_landmarks_5(only_center_face=only_center_face, eye_dist_threshold=5)
self.face_helper.align_warp_face()

face_helper is a FaceRestoreHelper, set up in the constructor with a detector:

self.face_helper = FaceRestoreHelper(
    upscale,
    face_size=512,
    crop_ratio=(1, 1),
    det_model='retinaface_resnet50',
    save_ext='png',
    use_parse=True,
    device=self.device,
    model_rootpath=root_dir)

This is where "what is a face" gets decided. retinaface_resnet50 is RetinaFace, a face detector (Deng et al., 2019). It returns face boxes and five landmarks per face: two eyes, nose, two mouth corners. Those five points are enough to align each face into a canonical pose. The eye_dist_threshold=5 throws away detections where the two eyes are fewer than five pixels apart, which filters out tiny false positives that are too small to restore usefully.

align_warp_face then warps each detected face into a fixed 512x512 frame using those landmarks, so every face GFPGAN sees is upright and the same size, regardless of how it sat in the original photo. That alignment is not a nicety. It is a requirement, and it is the reason this model cannot touch the background, which I will get to.

After this step the frame has been split. The faces are now a list of 512x512 aligned crops in face_helper.cropped_faces. The background is still sitting in the original image, untouched.

Step 2. Restore each face crop

Now GFPGAN runs, but only on the crops:

for cropped_face in self.face_helper.cropped_faces:
    cropped_face_t = img2tensor(cropped_face / 255., bgr2rgb=True, float32=True)
    normalize(cropped_face_t, (0.5, 0.5, 0.5), (0.5, 0.5, 0.5), inplace=True)
    cropped_face_t = cropped_face_t.unsqueeze(0).to(self.device)

    output = self.gfpgan(cropped_face_t, return_rgb=False, weight=weight)[0]
    restored_face = tensor2img(output.squeeze(0), rgb2bgr=True, min_max=(-1, 1))
    restored_face = restored_face.astype('uint8')
    self.face_helper.add_restored_face(restored_face)

One forward pass per face. The model output is a clean 512x512 face. GFPGAN is built on a generative facial prior, a face-generating network (StyleGAN2) baked into the architecture, so it does not just sharpen what is there. It can rebuild a believable eye or a believable mouth from a blurry one, because it knows what faces look like (Wang et al., CVPR 2021). That is exactly why you want it on faces and nowhere else.

The model call is wrapped in a try/except RuntimeError. If a single face crop fails inference, the code keeps the original crop instead of crashing the whole frame:

except RuntimeError as error:
    print(f'\tFailed inference for GFPGAN: {error}.')
    restored_face = cropped_face

Small thing, but it means one bad face does not kill a 2000-frame video.

Step 3. Paste the faces back, leave the background alone

This is the recombination, and it is where bg_upsampler finally matters:

if not has_aligned and paste_back:
    if self.bg_upsampler is not None:
        # the background goes to Real-ESRGAN
        bg_img = self.bg_upsampler.enhance(img, outscale=self.upscale)[0]
    else:
        bg_img = None

    self.face_helper.get_inverse_affine(None)
    restored_img = self.face_helper.paste_faces_to_input_image(upsample_img=bg_img)

Read it as two decisions.

First, the background. If you handed the restorer a bg_upsampler (a Real-ESRGAN instance), the whole frame goes through that general upscaler to produce bg_img. If you passed None, as content_enhancer does, the background is just the original frame. This is the literal "the rest goes to Real-ESRGAN" wire. It is one if. The face restorer never touches the background itself; it either delegates the background to a different model or leaves it as is.

Second, the faces. get_inverse_affine computes the reverse of each face's alignment warp. Remember, every face was warped into an upright 512x512 box. To put a restored face back, you have to undo that exact warp so the face lands where it came from, at the right size and angle. paste_faces_to_input_image then composites each restored face onto bg_img along that inverse transform. Because the helper was built with use_parse=True, it uses a face-parsing mask at the paste step, so the restored crop blends along the actual face outline (hair, jaw, neck) instead of pasting a hard 512x512 square with visible edges.

The output is one frame: GFPGAN's faces sitting inside Real-ESRGAN's background (or the plain background), seamed together along a parsing mask.

Why one model genuinely cannot do both

This is the whole reason the split exists, so it is worth being concrete.

A face restorer like GFPGAN only works on aligned face crops. Its prior is a face generator. Feed it a brick wall and it has no concept to draw from; it was never trained on walls, foliage, text, or fabric. At best it leaves them alone, at worst it hallucinates face-like structure into textures that are not faces. That is why the code crops faces out and feeds GFPGAN those crops only. The 512x512 alignment is not optional dressing, it is the input contract the model was trained against.

A general upscaler like Real-ESRGAN is the opposite. It was trained on a broad mix of real-world image degradation, so it handles walls, hair, fabric, and grass well (Wang et al., ICCV 2021 Workshops). But it has no face prior. On skin it tends to over-smooth, because its job is to remove noise and compression, and a face under that treatment loses the fine texture that makes it read as a real face. Plastic skin.

So you route. Faces to the model with the face prior, everything else to the model with the general prior, and stitch. Each model only sees the kind of pixels it was trained on.

---

About the author. I'm Wlad Radchenko, a software engineer. The code in this article comes from Wunjo Make (open source), local software for video makers, and Wunjo Design, an offline PWA for designers. Get in touch to find more on GitHub and LinkedIn.

---

Gotchas I would warn you about

A few things that bite when you wire this up:

• bg_upsampler=None means the background is not upscaled at all. In content_enhancer the GFPGAN path passes None, so faces get restored but the background stays at the original resolution. If you want sharper background too, pass a Real-ESRGAN instance as bg_upsampler. With None, faces improve and the rest is unchanged.
• only_center_face=False restores every face. On a crowd shot that is many forward passes, one per detected face, so cost scales with the number of faces, not the frame size. If you only care about the subject, only_center_face=True restores just the largest central face and skips the rest.
• No face, no change. If RetinaFace finds nothing, there are no crops, the loop does nothing, and paste_faces_to_input_image returns the background as is. The GFPGAN path is a no-op on a landscape with no people. If a face is there but not getting fixed, suspect the detector or the eye_dist_threshold=5 filter before the restorer.
• Alignment quality is upstream of restoration quality. Everything rests on those five landmarks being right. A bad warp gives GFPGAN a distorted face and the restored result, pasted back through the inverse warp, can look subtly off. If output faces look wrong in shape, look at detection, not the GAN.
• upscale=1 here is restoration, not enlargement. The constructor is called with upscale=1, so this path cleans faces at the same size. Do not expect a bigger image out of the GFPGAN method; that is the Real-ESRGAN method's job.

Takeaway

The face-versus-background split is three moves. Detect and align every face into a 512x512 crop with RetinaFace. Run GFPGAN on those crops only, because its face prior would hallucinate on anything that is not a face. Paste the restored faces back through the inverse alignment warp, onto a background that is either left alone or handed to Real-ESRGAN, blended along a face-parsing mask so there is no square seam.

The reason it is two models and not one is plain: a face restorer makes skin look real but cannot draw a wall, and a general upscaler makes a wall look sharp but turns skin to plastic. Give each model only the pixels it was trained on.

The code is content_enhancer in visual_processing/enhancement/face_enhancer.py and GFPGANer.enhance in visual_processing/enhancement/gfpganer.py in the Wunjo Make repo. The one line that wires the two models together is bg_upsampler in the paste-back step.

References

• Wang, Li, Zhang, Shan. "Towards Real-World Blind Face Restoration with Generative Facial Prior." CVPR 2021. arXiv:2101.04061 (GFPGAN)
• Wang, Xie, Dong, Shan. "Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data." ICCV 2021 Workshops. arXiv:2107.10833
• Deng, Guo, Zhou, Yu, Kotsia, Zafeiriou. "RetinaFace: Single-stage Dense Face Localisation in the Wild." 2019. arXiv:1905.00641