DEV Community

AI-KSK
AI-KSK

Posted on Edited on Originally published at huggingface.co

RefMod Without LoRA Training: Bundling a Character, a Fight Move, a Voice and a Set Into Loadable Files

The usual way to make a model hold a specific character or perform a specific fight move is to train a LoRA. RefMod does something different: it stores the reference as a loadable modification file that conditions the model at inference time.

The practical difference is turnaround. A LoRA takes a training run; a RefMod takes the source material and the extraction pass.

What goes into a RefMod

Six workflows in this episode, in four categories:

# Workflow Reference type
01 Character RefMod Single image or folder batch
02 Action RefMod Single video or folder batch
03 Audio RefMod Single audio clip
04 Martial-arts-hall scene RefMod Two camera angles, batch
05 Multi-reference gen A — Apply Applies a stack of RefMods
06 Multi-reference gen B — TextEncode Encodes the textual half of the conditioning

The A/B split in 05 and 06 matters: applying a reference and describing it are two separate stages, and you can change one without rebuilding the other.

Where it is actually strong

  • Character: holds identity across shots. Face structure, hair, and the read of the outfit survive camera changes.
  • Action: a fight move transfers as motion, not as a still pose. That is the hard part and the reason this is interesting.
  • Scene: a fixed set with two camera angles means you can cut between coverage without the room changing.

Where it is weaker

Audio RefMod is the least reliable of the four. Timbre transfers, but the cadence of the source clip leaks into the output — if the reference was recorded quickly, the result sounds rushed regardless of what the text says.

The realistic workflow

Use RefMod to get 90% there, then fix the remaining 10% with a short LoRA only if you need it. That order saves the training run in most cases.

Files

Video walkthrough (Chinese): https://youtu.be/H5L5sO7dRu8
Bilibili: https://www.bilibili.com/video/BV1F2nZ6SEA7/

Notes

  • ComfyUI exports; the H3 model weights must exist locally.
  • Tutorial material, not a product.
  • Independent AI-KSK release; not affiliated with the model authors or RunningHub.

Optional API platform - 88API (affiliate link)

Explore 88API for access to multiple AI model APIs. Available models, routes and prices are listed on the platform; Alipay top-ups are supported.

https://88api.ai/sign-up?aff=usx3

Disclosure: I may earn a commission from eligible purchases after you register through this referral link.

Top comments (0)