DEV Community

Cover image for Local AI Video Generation: ComfyUI and LTX-Video 2B on an RTX 3050 6GB
JacivaldoCarvalho
JacivaldoCarvalho

Posted on Edited on

Local AI Video Generation: ComfyUI and LTX-Video 2B on an RTX 3050 6GB

#ai #machinelearning #python

Phase 00 — Building a local AI-powered video generation pipeline.

1. Introduction

AI-powered video generation has evolved significantly, but running generative models locally remains challenging, especially on computers with limited hardware resources.

Many demonstrations rely on high-performance GPUs with substantial VRAM or cloud-based computing platforms.

This raises an interesting question:

Is it possible to generate AI videos using a laptop GPU with only 6GB of VRAM?

To investigate this, I started JC IA Video, a project focused on developing a local text-to-video generation pipeline.

The goal is not simply to generate videos, but to understand the underlying infrastructure, identify technical limitations, and progressively build an automated solution.

In this initial stage, called Phase 00, the objective was to validate local execution of ComfyUI with the LTX-Video 2B model and produce usable MP4 files.

The result was a functional proof of concept, including the resolution of a real issue involving animated WebP conversion.

2. Phase 00 Objectives

The experiment was organized around five primary objectives:

  1. Configure a Python environment with NVIDIA GPU support.
  2. Run ComfyUI locally using memory optimization techniques.
  3. Load LTX-Video 2B and generate videos from text prompts.
  4. Convert generated outputs into H.264 MP4 files.
  5. Validate the resulting media files and document the findings.

An important requirement was to keep inference entirely local, without relying on external inference services.

3. Hardware and Software Environment

The experiment was conducted on an Alienware 16 Aurora laptop with the following configuration:

Component Configuration
Device Alienware 16 Aurora
Operating system Ubuntu Linux
GPU NVIDIA GeForce RTX 3050 Laptop
VRAM 6 GB
System RAM 16 GB
Python 3.12.3
ComfyUI 0.44.0
PyTorch 2.15.0+cu132
Generative model LTX-Video 2B v0.9.5
Video processing FFmpeg 6.1.1

The limited amount of VRAM was one of the main constraints.

For this reason, the initial configuration prioritized low resolutions, short videos, and memory-efficient execution.

Evidence 01 — GPU Detection

Figure 1 — NVIDIA RTX 3050 Laptop GPU detected by the operating system, with 6GB of VRAM.

Figure 1 — NVIDIA RTX 3050 Laptop GPU detected by the operating system, with 6GB of VRAM.

The screenshot should display the GPU model, driver version, and available memory.

4. Solution Architecture

The initial pipeline was built using tools with clearly separated responsibilities.

                    Text Prompt
                         |
                         v
                       ComfyUI
                         |
                         v
                   LTX-Video 2B
                         |
                         v
                    CUDA Inference
                         |
                         v
                   Generated Frames
                         |
                         v
                    Animated WebP
                         |
                         v
                       Pillow
                   Frame Extraction
                         |
                         v
                       FFmpeg
                    H.264 Encoding
                         |
                         v
                      Final MP4
Enter fullscreen mode Exit fullscreen mode

ComfyUI provides the interface for executing AI generation workflows.

LTX-Video generates the visual frames based on the supplied conditioning inputs.

Pillow processes frames from animated WebP files, while FFmpeg handles the final MP4 encoding.

This separation of responsibilities will become particularly important in future phases, when the workflow is controlled by a dedicated Python application.

5. Configuring ComfyUI

ComfyUI was installed inside a Python virtual environment to isolate its dependencies.

The application was launched using:

cd ~/projects/jc-ia-video/ComfyUI

source .venv/bin/activate

python main.py \
  --listen 127.0.0.1 \
  --port 8188 \
  --lowvram
Enter fullscreen mode Exit fullscreen mode

The --lowvram parameter was used to adapt execution to the GPU's memory constraints.

The application was available at:

http://127.0.0.1:8188

The environment successfully initialized with CUDA support and memory management mechanisms, including DynamicVRAM and model weight offloading between system memory and GPU memory.

These mechanisms do not eliminate hardware limitations, but they help make certain workloads feasible on GPUs with limited VRAM.

Evidence 02 — ComfyUI Execution and Model Loading

The logs recorded dynamic model loading and three completed executions.

Selected log entries:

[INFO] To see the GUI go to: http://127.0.0.1:8188
[INFO] Requested to load LTXV
[INFO] Model LTXV prepared for dynamic VRAM loading.
[INFO] Requested to load VideoVAE
[INFO] Prompt executed in 16.72 seconds
[INFO] Prompt executed in 6.87 seconds
[INFO] Prompt executed in 12.01 seconds
Enter fullscreen mode Exit fullscreen mode

These messages confirm that ComfyUI was accessible, the inference components were prepared, and all three prompts completed execution.

Figure 2 — ComfyUI logs showing dynamic loading of LTXV and VideoVAE models and three completed prompt executions.

Figure 2 — ComfyUI logs showing dynamic loading of LTXV and VideoVAE models and three completed prompt executions.

The full logs also recorded text encoder preparation and execution of the 12 sampling steps.

6. Models Used

Two main components were used for video generation.

LTX-Video 2B v0.9.5

The generative model responsible for producing visual content.

Model file:

models/checkpoints/
└── ltx-video-2b-v0.9.5.safetensors
Enter fullscreen mode Exit fullscreen mode

T5 XXL FP8

The text encoder used to process text conditioning inputs.

Model file:

models/text_encoders/
└── t5xxl_fp8_e4m3fn_scaled.safetensors
Enter fullscreen mode Exit fullscreen mode

The LTX-Video model was downloaded from Hugging Face.

The text encoder was made available to ComfyUI through a symbolic link pointing to a file already stored in the local Hugging Face cache.

This avoided creating another copy of approximately 4.9GB.

One important operational consideration is that deleting the original cached file would break the symbolic link.

Model weights are excluded from the project's Git repository.

7. Building the Generation Workflow

After configuring the environment, I loaded a text-to-video workflow compatible with LTX-Video.

The workflow connects components responsible for:

  • Checkpoint loading.
  • Text encoder loading.
  • Positive and negative prompt processing.
  • Video latent configuration.
  • Sampling schedule configuration.
  • Frame generation.
  • Output decoding.
  • Animated WebP export.

Evidence 03 — Workflow and Generation Parameters

Conservative generation parameters were selected to reduce resource consumption.

Parameter Value
Landscape resolution 512 × 288
Portrait resolution 288 × 512
Frames 25
Frame rate 8 FPS
Sampling steps 12
CFG 3.0
Sampler res_multistep
Batch size 1

With 25 frames at 8 FPS, each video has a nominal duration of:

25 / 8 = 3.125 seconds.

The portrait resolution of 288 × 512 preserves the 9:16 aspect ratio commonly used by TikTok, Instagram Reels, and YouTube Shorts.

However, this resolution is significantly lower than typical production-quality video resolutions.

The purpose of this phase was to validate the generation pipeline, not to achieve final production quality.

LTX-Video 2B text-to-video workflow in ComfyUI, including generation parameters.

Figure 3 — LTX-Video 2B text-to-video workflow in ComfyUI, including generation parameters.

This screenshot combines the visual workflow architecture and the relevant configuration values.

8. First Generation Results

The workflow successfully generated three animated WebP files.

File Resolution Frames FPS
ComfyUI_00001_.webp 512 × 288 25 8
ComfyUI_00002_.webp 512 × 288 25 8
ComfyUI_00003_.webp 288 × 512 25 8

During generation, GPU utilization was observed reaching 100% at one point.

This indicates intensive GPU processing, although it does not establish that the GPU remained fully utilized throughout the entire inference process.

Execution Times

ComfyUI recorded the following execution times:

Execution Recorded Time
First 16.72 seconds
Second 6.87 seconds
Third 12.01 seconds

The first execution may include additional model loading and initialization overhead.

These values represent execution times reported by ComfyUI. They should not be interpreted as controlled benchmark results.

Repeated tests under standardized conditions will be required to evaluate performance, consistency, and resource consumption.

Demo — Portrait Video (9:16)

The third video was generated at 288 × 512 pixels, with 25 frames and a frame rate of 8 FPS.

The result was published on YouTube Shorts as a visual demonstration of the proof of concept.

Watch the AI-generated video on YouTube Shorts

JC IA Video — Phase 00 Demonstration

Demo 1 — Portrait video generated locally using ComfyUI and LTX-Video 2B on an RTX 3050 Laptop GPU with 6GB of VRAM.

This demonstration provides a visual reference for the model's output, complementing the configuration and execution evidence.

9. A Real Issue: FFmpeg and Animated WebP

After generating the videos, I encountered an issue while converting the output files to MP4.

The first attempt used FFmpeg directly:

ffmpeg -i input.webp \
  -c:v libx264 \
  -pix_fmt yuv420p \
  output.mp4
Enter fullscreen mode Exit fullscreen mode

However, the FFmpeg version available in the environment could not correctly decode the animated WebP files produced by ComfyUI.

The observed messages included:

skipping unsupported chunk: ANIM
skipping unsupported chunk: ANMF
image data not found
Enter fullscreen mode Exit fullscreen mode

The conversion process failed to produce valid MP4 videos.

Subsequent validation also identified invalid media containers, including the error:

moov atom not found
Enter fullscreen mode Exit fullscreen mode

This demonstrated an important lesson: the existence of a file with an .mp4 extension does not guarantee that it contains a valid, playable video.

The container structure and media metadata must be validated.

The original terminal screenshots were not preserved. Therefore, the error messages recorded during the experiment are included as textual evidence.

10. The Solution: Pillow + FFmpeg

To work around the decoding limitation, I separated the conversion process into two stages.

First, Pillow extracts individual frames from the animated WebP file.

Then, FFmpeg encodes those frames into an H.264 MP4 video.

The updated pipeline became:

Animated WebP
     |
     v
   Pillow
     |
     v
Temporary PNG Frames
     |
     v
FFmpeg / libx264
     |
     v
  H.264 MP4
Enter fullscreen mode Exit fullscreen mode

I created a Python script to automate this conversion.

The initial implementation uses Image.open() to read the animated WebP frames, TemporaryDirectory() to store temporary PNG files, and subprocess.run() to execute FFmpeg.

A simplified example of the frame extraction process:

from PIL import Image

with Image.open("input.webp") as image:
    for index in range(image.n_frames):
        image.seek(index)
        frame = image.convert("RGB")
        frame.save(f"{index:05d}.png")
Enter fullscreen mode Exit fullscreen mode

The extracted frames can then be encoded using:

ffmpeg \
  -framerate 8 \
  -i "%05d.png" \
  -c:v libx264 \
  -pix_fmt yuv420p \
  -movflags +faststart \
  output.mp4
Enter fullscreen mode Exit fullscreen mode

The yuv420p pixel format improves compatibility with media players and online platforms.

The +faststart option relocates MP4 metadata to support progressive playback more efficiently.

The initial converter assumes a constant frame rate. Handling variable frame durations remains a potential improvement for a future version.

11. Validating the MP4 Files

After implementing the conversion script, all three WebP files were successfully processed.

The recorded output was:

OK: ComfyUI_00001_.mp4 | 25 frames | 8.00 FPS
OK: ComfyUI_00002_.mp4 | 25 frames | 8.00 FPS
OK: ComfyUI_00003_.mp4 | 25 frames | 8.00 FPS
Enter fullscreen mode Exit fullscreen mode

I then used ffprobe to inspect the generated media metadata.

The results were:

File Resolution FPS Duration File Size
ComfyUI_00001_.mp4 512×288 8 3.125 s 43,349 bytes
ComfyUI_00002_.mp4 512×288 8 3.125 s 46,037 bytes
ComfyUI_00003_.mp4 288×512 8 3.125 s 157,615 bytes

Validation confirmed that all three MP4 files contained resolution, frame rate, and duration metadata consistent with the generation settings.

The third file confirmed successful export in a portrait 9:16 format.

The results are preserved as textual evidence and a technical table, without requiring additional terminal screenshots.

12. Results and Limitations

Phase 00 successfully validated the fundamental components of a local AI video generation pipeline.

Confirmed Results

  • ComfyUI running locally with CUDA support.
  • LTX-Video 2B loaded and used for generation.
  • Three animated WebP outputs generated.
  • Landscape and portrait generation tested.
  • Successful conversion to H.264 MP4.
  • Output metadata verified using ffprobe.
  • Three execution times recorded by ComfyUI.
  • A portrait video published as a visual demonstration.

Identified Limitations

  • Low output resolution.
  • Video duration limited to approximately three seconds in the initial tests.
  • High GPU resource utilization.
  • No controlled performance benchmark yet.
  • Video conversion still handled by a separate script.
  • Generation workflow still operated manually.
  • Visual quality not yet evaluated using systematic criteria.

These limitations help define the scope of future development.

The results do not imply that every video configuration, resolution, or model will run successfully on a 6GB GPU.

They demonstrate that one specific configuration was successfully validated on this hardware.

13. Next Phase: Python Automation

With the proof of concept completed, the next objective is to transform the manual workflow into an automated pipeline.

Phase 01 will focus on using the ComfyUI API to:

  1. Receive text prompts.
  2. Load previously validated workflows.
  3. Submit generation jobs.
  4. Monitor execution status.
  5. Identify and retrieve generated outputs.
  6. Automatically convert results into MP4.
  7. Record execution metadata, errors, and processing times.

The proposed architecture is:

Python CLI
    |
    v
Workflow Manager
    |
    v
ComfyUI API
    |
    v
LTX-Video 2B
    |
    v
Output Manager
    |
    v
Media Converter
    |
    v
MP4 + Execution Metadata
Enter fullscreen mode Exit fullscreen mode

This approach will allow the project to evolve without modifying ComfyUI's internal source code.

14. Conclusion

This experiment demonstrated that short AI-generated videos can be produced locally using an RTX 3050 Laptop GPU with 6GB of VRAM, provided that the workload is configured within the available hardware constraints.

It also highlighted that building a functional AI video generation pipeline involves much more than running a generative model.

It requires understanding the inference environment, managing hardware resources, interpreting execution errors, processing intermediate media formats, and validating the resulting artifacts.

The animated WebP conversion issue was particularly valuable: a seemingly simple failure demonstrated the importance of separating responsibilities between tools and verifying outputs rather than assuming success.

Phase 00 established the technical foundation of JC IA Video.

The next step is to transform this proof of concept into a reproducible and automated pipeline.

15. Project Repository and Availability

JC IA Video is being developed incrementally, with each phase documenting technical decisions, implementation details, limitations, and validation results.

The project currently maintains its source code, workflows, scripts, and technical documentation in a local Git repository.

The GitHub repository will be made publicly available in the future, once the project reaches an appropriate level of organization, reproducibility, and documentation.

Future articles in this series will cover the implementation of the automation pipeline, resource optimization, and additional video processing capabilities.

References

This is the first article in the JC IA Video — Local AI Video Generation series, documenting the incremental development of a local AI video generation pipeline using Python, ComfyUI, and open-source tools.

Top comments (0)