📝 Originally published (in Japanese) at forge.workstyle.tech.
TL;DR A mouth-only composite showed a chin shift in speaker-specific close-ups. The still image looked blurry because it used the first three frames of the lip-sync video. A trim timestamp outside the segment produced no still frame, leaving the process waiting on loop=-1.
Conclusion First
With lip-only compositing, the jaw looked misaligned in a solo shot. Switching back to animating the whole face fixed the alignment, and avoiding the first three frames of the video for the still image removed the blur. Video processing had been stuck for 14 hours because the trim timestamp pointed outside the clip and extracted no frames, leaving it waiting indefinitely for input from loop=-1. I moved the timestamp into the clip and added a timeout.
Why Did the Jaw Shift in the Solo Shot?
I started investigating after noticing that the speaker’s jaw seemed to shift sideways whenever they spoke. I had been compositing movement only around the mouth.
For episode 15, the script specified a rectangle around the mouth. The face stayed still, while only the area inside the rectangle became a lip-sync video. For example, the rectangle started at x185 y215 and was 86 pixels wide and 62 pixels high.
In the two-shot from farther away, the shift wasn’t noticeable. But when I switched to solo shots of each speaker, the jaw shifted sideways inside the rectangle every time they spoke. Someone pointed out that it extended beyond the face’s outline, and that’s when I first noticed it.
I had been looking only at the area where the mouth moved. In a solo shot, the face takes up about 40% of the screen’s height. At that size, the jaw shift is impossible to miss.
So I stopped animating just the mouth and went back to animating the whole face. The jaw shift was gone.
Why Did the Still Image Blur When I Switched Back to the Full Face?
After fixing the jaw issue, I noticed that the face looked blurry only while the character was silent.
I was using the first frame of the lip-sync video as the still image by taking the start of the video with trim=start=0. When I laid out four frames from the lip-sync video for comparison, only the first frame was blurry.
I wasn’t sure whether this was related to switching back to animating the full face or to how I chose the still image. From the logs and the footage, I could confirm that the first few frames of the generated video were blurry. I couldn’t confirm that this was a documented behavior of the generation system.
I stopped using the first three frames as the still image. Three frames is 0.12 seconds. I chose three as a safety margin. This change fixed the blurry face during silent parts.
Why 14 Hours of Video Processing Were Still Running
The lingering 14 hours had a different cause.
For each speaker’s shot, I read the lip-sync video starting partway through the segment. I specified that seek position with -ss. But I didn’t convert the still-frame timestamp to a time relative to the start of the segment.
As a result, the trim setting—the instruction for selecting a range from the video—could point outside the segment. No frame was available for the still image. Even so, I had set loop=-1 on the input. Video processing kept waiting for the looped input and never finished, leaving orphaned processes behind.
Two processes were still running. Their elapsed times were 14 hours and 34 minutes, and 14 hours and 19 minutes. CPU usage was about 485%.
I converted the still-frame timestamp to a time relative to the start of the segment. I also clamped it to the segment’s duration. This fixed the problem of the frame selection pointing outside the segment.
I also added a 20-minute timeout per video to the batch process, so it wouldn’t wait forever when processing failed to finish.
How I Fixed Lip-Sync Jaw Misalignment and Long-Running Processing
In this check, moving only the mouth made the jaw look misaligned in a standalone shot. Switching back to moving the whole face fixed the issue. That meant I also had to revisit the part that uses the beginning of the video as a still frame.
The video processing job kept running because I hadn’t converted the still-frame timestamp to one measured from the start of the segment. trim pointed outside the segment and returned no frames. The process waiting for loop input then never finished.
There were three symptoms, and I revisited three things: expand the compositing area from the mouth to the whole face; avoid using the first three frames as the still image; and move the still-frame timestamp inside the segment, with a timeout on processing.
Once I found the cause of the 14-hour run, I could separate it from the alignment and blur issues. Even after fixing the visuals, I still needed to check what the video-processing job was waiting for.
Related Articles
Key takeaways
- Check chin alignment in speaker-specific close-ups.
- If the chin shifts, switch back to moving the whole face.
- Avoid using the first three frames for the still image.
- Align
-ssandtrimtimestamps to the segment and keep the still timestamp within it. - Set a 20-minute timeout per item in batch processing.
Top comments (0)