Facial expressions must be crafted as drawings to be readable on small screens

📝 Originally published (in Japanese) at forge.workstyle.tech.

In a vertical video where two caricatures converse, the supporting character's expression changes to a troubled look midway through. Initially, I specified the facial expression using a prompt similar to what you'd use for creating lip-sync videos.

eyebrows drawn together and slanted down, mildly troubled expression

During the screening review, I received this feedback:

Sam's confused eyebrows remain difficult to read at this display size.

The supporting character is about half the size of the main character and is placed deeper in the scene. On a 1080×1920 screen, the face is only a few dozen pixels. The expression added via the prompt got lost in the lip-sync movements, making it nearly indistinguishable from the original face.

Creating as an Image

The single image that serves as the basis for the lip-sync video was replaced with a worried-looking face. This is derived from the original caricature using img2img.

Original image → img2img (denoise 0.5)
"Furrow your brow, look slightly anxious, close your mouth, and keep the clothes and pose the same"

When compared, the difference was clear. The eyebrows are lowered, and the corners of the mouth droop. By recreating the lip-sync video based on this image, even on a small display, it was apparent that the character was "struggling". The review treated this suggestion as resolved.

Redrawing from scratch makes it a different person

I have also generated a picture with a different expression from scratch. I added a "worried face" to the same description of appearance and had it drawn.

It became a different person. The hairstyle and contours are slightly different. The viewer perceives it not as the same person with a changed expression, but as a different person appearing.

With img2img, the lines, colors, and clothes are inherited from the original picture. Only the expression changes. Denoise is around 0.5--0.62. The higher it is, the more the expression changes, but it approaches becoming a different person.

Some Expressions Cannot be Erased

The reverse approach did not work well. The main character's portrait initially had a smiling face with an upturned mouth. I wanted to give it a calm expression, so I tried to erase the smile using img2img.

completely neutral face, straight closed mouth with level corners, absolutely no smile

Even with denoise 0.5 or 0.6, the upturned mouth remained. The teeth became invisible, but the faint smile would not disappear. The nature of the original image made it difficult to alter using img2img. It would have been faster to create a character with a neutral expression from the initial image.

1. Expressions Created Once Can Be Reused

The opposing character, Sam, appears in dozens of episodes. By creating a single image of a worried face, the same worried face can be used in any episode.

Expressions can be switched by simply writing "this face from this line" in the script. To switch expressions in scenes without lines (e.g., when Sam looks worried the moment he hears "let's try"), a specification to switch expressions without lines was added.

{"t": "expr", "at": 4.32, "speaker": "sam", "expression": "worry"}

Expressions are retained until the next change. If the face returns to normal after each line, it becomes unclear what Sam was worried about.

Summary

• The expressions of small characters in an image are not conveyed through prompts, so create them as illustrations
• Expression variations can be derived from the original image using img2img; drawing from scratch will result in a different person
• The inherent nature of the original image (e.g. a smiling mouth) is difficult to erase using img2img; decide on it in the initial illustration
• Once a facial expression illustration is created, it can be reused throughout the entire story where the same character appears