DEV Community

rashid inmarketing
rashid inmarketing

Posted on

Maintaining Character Consistency in ComfyUI — LoRA + ControlNet + IPAdapter

#comfyui #stablediffusion #ai #machinelearning

Generating one great image of a character is easy. Generating the same character consistently across dozens of panels, different poses, different lighting, different angles — while running on consumer hardware like an RTX 3050 — is the actual hard problem in any long-form AI-assisted visual production. Here's the pipeline that's worked for me across a multi-chapter project.

Why a single technique isn't enough

Each of the three main tools solves a different part of consistency, and relying on just one leaves gaps:

  • LoRA locks in the character's core identity — face structure, defining features — trained specifically on that character.
  • ControlNet locks in pose and composition, so you can direct exactly how the character is positioned without relying on prompt luck.
  • IPAdapter transfers style and finer visual details from a reference image, catching nuance a LoRA alone tends to miss, especially across different environments or lighting.

Used together, each compensates for what the others miss.

Training a character LoRA on limited hardware

On an RTX 3050 (8GB VRAM), full LoRA training is tight but workable with the right settings:

  • Use kohya_ss or the ComfyUI LoRA training extension with gradient checkpointing enabled — trades some training speed for dramatically lower VRAM usage
  • Keep training resolution at 512x512 rather than 768 or 1024 unless you can confirm you have headroom
  • 15-25 curated training images of the character, varied angles and expressions, tends to outperform a larger but less varied dataset
  • Batch size of 1, with gradient accumulation to simulate a larger effective batch, is usually necessary at this VRAM tier 
Training config essentials:
- resolution: 512
- train_batch_size: 1
- gradient_accumulation_steps: 4
- gradient_checkpointing: true
- mixed_precision: fp16
- learning_rate: 1e-4
- network_dim: 32 (lower = smaller file, less VRAM, still sufficient for most character LoRAs)
Enter fullscreen mode Exit fullscreen mode

ControlNet for pose consistency

Once the LoRA handles identity, ControlNet's OpenPose or Depth models let you dictate the actual pose rather than hoping the prompt produces something usable:

ControlNet setup:
- Model: control_v11p_sd15_openpose
- Preprocessor: openpose_full (captures hands + face, not just body skeleton)
- Weight: 0.7-0.85 (higher = more rigid pose adherence, can fight the LoRA's natural style)
- Start/End: 0.0 - 0.8 (releasing control near the end of sampling lets style details resolve more naturally)
Enter fullscreen mode Exit fullscreen mode

The weight and start/end range are the values worth actually experimenting with per-character — too rigid and the result looks stiff and loses the LoRA's natural rendering; too loose and pose consistency breaks down exactly where you needed it most.

IPAdapter for cross-scene consistency

IPAdapter takes a reference image and transfers style/detail information alongside your prompt and ControlNet pose — this is what catches details a LoRA alone sometimes drifts on across very different environments (a character rendered consistently in a bright exterior vs. a dark interior, for example):

IPAdapter setup:
- Reference: a clean, well-lit "canon" image of the character
- Weight: 0.5-0.6 (too high overrides the actual scene's lighting/environment)
- Weight type: "style transfer" rather than "strong" for most panel-to-panel work
Enter fullscreen mode Exit fullscreen mode

The actual workflow chain 

[Load Checkpoint] → [Apply LoRA (character)] 
        ↓
[ControlNet (OpenPose from reference pose)] 
        ↓
[IPAdapter (canon reference image)] 
        ↓
[KSampler] → [VAE Decode] → [Save Image]
Enter fullscreen mode Exit fullscreen mode

Sampling settings that have worked consistently at this VRAM tier: DPM++ 2M Karras, 25-30 steps, CFG 6-7. Pushing CFG much higher tends to fight the LoRA rather than reinforce it.

The realistic limitation worth knowing upfront

Even with all three combined, expect a 10-15% rate of panels needing a reroll or manual touch-up for consistency drift, especially on hands, expressions, or complex poses — hands in particular remain the classic failure point across most current models. Budgeting for that revision pass, rather than expecting first-generation perfection, is part of realistically planning a production timeline around this pipeline rather than treating it as fully automated.

I run this pipeline for a long-form original series in production — happy to go deeper on specifics in the comments.

Top comments (0)