đź“ť Originally published (in Japanese) at forge.workstyle.tech.
TL;DR ComfyUI generated a worried face for a clip that specified a neutral face. Multiple clips used the same image filename in input, so a later upload overwrote the file an earlier job was expected to read. Adding the episode ID and clip name to image and audio filenames prevents those collisions.
The conclusion first
The reason the video generated in ComfyUI with a neutral face specified ended up looking troubled was that multiple clips shared an image with the same filename, and a later image overwrote the file that an earlier job (the video-generation process) was reading. I fixed this by including the clip name in image and audio filenames, then regenerated the video affected by the overwrite.
Why Did the Neutral-Face Videos End Up Looking Worried?
In 43 out of 51 episodes, the lip-sync videos showed a worried expression, even though they were supposed to use the neutral-face image. Looking only at the output, I couldn’t tell whether the expression-generation process had changed the face or whether a different image had been used.
So I compared the first frame of each video with the original neutral-face and worried-face images. In all three videos I checked, the first frame matched the worried-face image. It seems the neutral face wasn’t changed by the expression-generation process; a different image was used when the video was made.
Why did the earlier job read the later image?
The cause was using the same filename for images from different clips in ComfyUI’s input folder, where files used by the video generation process are placed. The function that queued jobs copied images as {story_id}_{character}_art.png and audio files as {audio_file_name}. Clips that used artwork of the same character therefore ended up with the same image filename.
After queuing the earlier job, queuing artwork of the same character with a worried expression copied the later image to the same filename. As a result, the image intended for the earlier job was replaced with the worried-expression artwork. Even though filenames were differentiated by character, clips featuring the same character still collided.
I believe the job read the image that was in input at the time it ran, rather than the image that was there when it was queued. This is an inference from the behavior of the generation environment; I haven’t confirmed the loading time in the documentation or source code. The problem wasn’t selecting the wrong image, but placing different content under the same name in the same location.
Separated Image and Audio Filenames by Clip
The fix changed image filenames from being based on the character to being based on the clip. Images now use {episode ID}_{clip name}_art.png, and audio files use {episode ID}_{clip name}.wav. With separate names, files added later no longer collide with filenames belonging to earlier clips.
Audio had a separate issue. The generation environment sometimes picked up a file while it was still being written, causing the job to fail. The error looked like this:
[Errno 1094995529] Invalid data found when processing input: '…/input/ep21_..._A.wav'
For images, a newly added illustration was used in an earlier video. For audio, the job failed after reading a file that was still being written. Both issues occurred because files were placed directly in a shared location from which jobs read them.
How I Rebuilt the Videos for 43 Episodes
There were 43 episodes featuring a character with multiple facial expressions. I rebuilt only the videos that had been overwritten. The work started at 20:36 on October 4 and finished at 05:38 on October 5. Processing each episode took 11–15 minutes.
During the rebuild, I tried to recover the input history for one episode, but its source file was missing. Disk cleanup had deleted it.
No such file or directory: '…/output/talk_00251-audio.mp4'
I revised the recovery process so that when the source file is missing, it uses the already-recovered video instead. The rebuild also revealed that the process for recovering source files from history assumed those files were still present.
Key takeaways
- Compare the specified image with the video’s first frame.
- Check whether consecutive clips share image or audio filenames in
input. - Include the episode ID and clip name in image and audio filenames.
- Check for audio reads during writes and confirm source files needed for history recovery still exist.
Top comments (0)