DEV Community

AI-KSK
AI-KSK

Posted on

Fish Audio S2 Pro + LTX 2.3: from zero-shot voice clone to a talking digital human

#comfyui #ai #tutorial

Two models, one pipeline. Fish Audio S2 Pro produces the voice — cloning, emotion control, multi-speaker dialogue — and LTX 2.3's three-stage digital-human graph makes a still image speak it. The author's claim that matters: this is no longer text-to-speech that reads words, it is text-to-speech that performs them.

Evidence note: this episode ships no workflow JSON and 说明.txt records 工作流链接：无, even though the video repeatedly points at RunningHub posts. So no node parameters are quoted below. Everything is from 字幕.srt, attributed to the author; the VRAM tiers and the 80-plus language count are his restatement of the project's own documentation.

Where this fits

You need a voice track with a specific timbre and a specific emotion, and then a face that lip-syncs to it — without building a voice-cloning service and a lip-sync service separately.

Part 1 — Fish Audio S2 Pro

The workflow ships three official examples.

Text to audio

Type the text, run, get audio. The manipulation lever is emotion tags: insert brackets into the prompt. If writing them yourself is painful, the author's shortcut is to hand the model's detail page to an LLM (he names DeepSeek and GPT) and let it write the tags and the emotional direction for you.

Audio cloning (zero-shot)

Same family as Qwen3 TTS and IndexTTS: zero-shot cloning from a reference clip of about 10-30 seconds.

Two pieces of processing the author adds because cloning is sensitive to input quality:

  1. Vocal separation — if the reference has heavy background music, cloning degrades. Extract the vocal stem first, then feed that.
  2. Reference text reverse-engineering — transcribe the reference so the clone has the matching text.

Then set the prompt and run. His comparison: the opening line of the episode, generated from a reference clip, lands very close to the reference timbre — and closer still once emotion tags are added, because the delivery fits the context of the line.

Multi-speaker dialogue

Load two clips, run vocal separation on both, tag them as speaker 1 and speaker 2, and the model performs a two-person exchange. The demo is a deliberately theatrical passage and it holds up.

The VRAM tiers

The ComfyUI node lets you pick which variant to load, and the author gives the ladder:

Variant Approximate VRAM
S2 Pro 24 GB
FP8 / INT8 ~20 GB
NF4 ~16 GB

He deliberately runs the minimum configuration online so that anyone can run it, and says that if you have 24 GB locally you should pick INT8.

The cost of the automatic text reverse: the pipeline can transcribe the reference audio automatically — and it works, essentially correctly — but it drags down the whole flow. That node can be bypassed and the transcript typed by hand. In multi-speaker cloning the same transcript exists; the author says a version that wires it up properly should improve clone accuracy when the transcript is precise.

What S2 Pro is good at, and what it costs

Good at: timbre cloning, emotion control, breath and pause handling, more precise tag control. The framing is that it can act a text rather than read it. It supports 80-plus languages, free-form bracket tags, and the page states 15,000-plus tags that do not rely on a fixed vocabulary — which is what allows very fine-grained direction. Multi-speaker is supported.

Costs, per the author: VRAM usage is on the high side, optimisation is still insufficient, and generation is slow. If you want to test its ceiling, run S2 Pro on a 48 GB machine.

Part 2 — the LTX 2.3 digital human

The voice track becomes the driver for LTX 2.3's three-stage digital-human graph.

  • Load the audio, load one image, set a simple prompt.
  • The graph is the three-stage render the channel has covered repeatedly; the author says the whole point is that it lifts the success rate of getting a usable digital human out of LTX, not just the peak quality.
  • 24 GB is enough — this is the minimum configuration he ships online.
  • If the result is weak, raise the width/height and the model size; he calls 0.15 the best size value for higher definition.
  • His own rule of thumb: if the face proportion in frame is right, he gets a good result every time, with the similarity penalty dropping very low.

Why the pairing works

The author's framing: Fish Audio S2 Pro is fully out of the "just read the words" tier of TTS, and pairing it with LTX 2.3's digital human is a natural fit. He also places it against the alternatives — InfiniteTalk is fast enough, LongCat is too slow to have a practical use case, and the LTX family is the one he calls worthwhile. He also sketches the obvious extension: loop the pipeline and you could read an entire book.

Limits

  • No workflow JSON, and 说明.txt records no workflow link — the RunningHub posts the video points to are not reachable from this package.
  • Cloning quality depends heavily on the reference: BGM must be separated, and a 10-30 second clip is the stated range.
  • Automatic transcription works but slows the pipeline; the author prefers to bypass it.
  • VRAM tiers are approximate and come from the project's documentation as restated in the video.
  • S2 Pro is slow and not yet well optimised; the 48 GB ceiling test is described as something you could do, not something he ran.
  • The digital-human result is still probabilistic — the face-proportion rule is a heuristic, not a guarantee.

Episode resources

Top comments (0)