I Built a 100% Free, Fully Local AI Short-Video Pipeline on an M1 Mac — For Fellow Local-Claude Folks

Zero cost. Zero external calls. Draw keyframes with SDXL, bring them to life with LTX-2.3 (MLX) image-to-video, narrate with Kokoro TTS, score it with MusicGen, and stitch it all together with ffmpeg — entirely inside one M1 Max (64GB). This is the real log, plus a recipe you can copy.

Audience: Claude Code users and developers running Claude/LLMs locally on Apple Silicon. I wrote down every place I got stuck, because the failures are the most useful part.

---

TL;DR

• Panning/zooming a still image (Ken Burns) is not a video. It's a slideshow. Real motion only comes from I2V (image-to-video). Not grasping this first was my biggest mistake.
• The whole stack is free and local: SDXL (ComfyUI) keyframes → LTX-2.3 (MLX) I2V → Kokoro TTS narration → MusicGen BGM → ffmpeg assembly.
• The nastiest gotcha is the Metal GPU watchdog crash (MLX issue #3267). The old package died instantly while the display was on; the newer dgrauet/ltx-2-mlx (q8) runs to completion with the screen on — that reversal was the biggest lesson.
• Environment setup is where 99% of people get stuck: arm64 Python and pinning transformers.

---

1. How it started: "This is just a slideshow"

My first delivery was rejected in one sentence.

I generated ten nice stills with SDXL, faked motion with ffmpeg's zoompan (the Ken Burns push/pull), laid narration and subtitles on top, and called it "a video." The response:

"This isn't what I wanted. It's just a slideshow. I asked for a video, not a slideshow."

It stung. Here's what I burned into my bones:

1. A "video" (movie) means the characters, the water, the light actually move. A zoom/pan over a still is not a video. Never conflate the two.
2. Pin down the definition of the deliverable first. Don't sacrifice what the client actually wants (real motion) for "cheap and fast." Cheap and fast, but wrong, is worth zero — it's a full redo.
3. Real motion requires I2V. A stills-only pipeline can only get you to a "storyboard / preview."
4. But don't throw Ken Burns away — it's great for matching narration length in storyboard checks, and for closing shots that don't need to move. Just never call that "finished."

That single rejection was the origin of everything below.

Side-by-side, from the actual project. Left = the rejected slideshow (zoom on a still). Right = real I2V motion from the same keyframe.

---

2. The goal: run real motion (I2V) locally, for free

The idea is simple. Make a still image (a keyframe) for each shot, then have an I2V model bring that keyframe to life. Keyframes come from SDXL; the motion comes from the MLX port of LTX-2.3. All of it runs inside this Mac.

Fig 1: The whole pipeline — I2V (LTX-2.3) is the body of the video

Stage	Step	Tool	Role
A	Storyboard + narration script	Local LLM (gemma-3-12b-it via mlx_lm)	Design
B	Keyframe stills	ComfyUI / SDXL (realcartoon-xl)	Starting image per shot
C	Narration audio + real duration	Kokoro TTS (or macOS say) + ffprobe	Free; locks timing
D	I2V — "move" the keyframe	LTX-2.3 MLX	This is the body of the "video." Non-skippable
E	Assembly (concat + narration + subs + BGM)	ffmpeg	Finishing
(preview only)	Ken Burns	ffmpeg zoompan	For review only. Not the finished product

---

3. The stack (all free, all local)

• Keyframes: ComfyUI + realcartoon-xl-v4 (SDXL family). Batch-generate by hitting the REST API (:8000).
• I2V (the video body): MLX port of LTX-2.3, in two flavors by purpose (below).
  • Draft = notapalindrome/ltx23-mlx-av-q4 (20GB, distilled q4, fast)
  • Production = dgrauet/ltx-2.3-mlx-q8 (21GB, --two-stages-hq + STG, high fidelity)
• Text encoder / script LLM: mlx-community/gemma-3-12b-it-4bit (~7GB).
• Narration: Kokoro TTS (mlx-audio). macOS's built-in say works too.
• BGM: MusicGen (transformers' built-in MusicgenForConditionalGeneration).
• Assembly: ffmpeg (Homebrew build — with a caveat below).

Hardware is M1 Max / 64GB. That "64GB reality" constrains model choice.

---

4. Environment setup is where you'll get stuck (two real landmines)

Honestly, setup is harder than the generation itself. 99% of the pain is here. Two landmines:

1) Use arm64 Python (Anaconda will kill you)

Anaconda-style Python often runs as x86_64 (Rosetta), and there mlx simply won't install (no matching distribution). Always create your venv with native arm64 Python.

/Library/Frameworks/Python.framework/Versions/3.12/bin/python3 -m venv ~/ltx-mlx
~/ltx-mlx/bin/pip install -U mlx-video-with-audio

The newer package (dgrauet/ltx-2-mlx) uses uv, but uv may also grab an x86_64 anaconda python at first and fail to install mlx, so pass the arm64 python explicitly:

git clone --depth 1 https://github.com/dgrauet/ltx-2-mlx.git ~/ltx-2-mlx
cd ~/ltx-2-mlx
uv sync --all-extras --python /Library/Frameworks/Python.framework/Versions/3.11/bin/python3

2) Pin transformers to >=4.50,<5

pip install mlx-video-with-audio drags in transformers 5.13, which crashes mlx_lm's tokenizer registration (AttributeError: 'str' object has no attribute '__module__'). Pin it down and it works (4.57.6 runs fine; pip warns about deps but it works):

~/ltx-mlx/bin/pip install "transformers>=4.50,<5"   # ← the required pin

Miss these two and you'll lose hours to "mlx won't install for some reason" and "a mysterious AttributeError on startup." Kill them first.

---

5. Model choice and the "64GB reality"

The full-precision LTX-2.3 model is 53.5GB as a single file, which OOMs / thrashes on 64GB at load. Don't use it. Run quantized versions.

Fig 2: Which quantized build to use, inside the 64GB reality

Purpose	Model / settings	Measured speed
Speed / draft	distilled q4 (ltx23-mlx-av-q4 / --distilled)	~4 min/clip, peak ~14.7GB
Quality / production	q8 --two-stages-hq + STG	~30 min/clip, peak ~21GB
Fights / heavy motion	q8 HQ + --stg-scale 1.0 + motion LoRA	30 min+

For HQ at 1024×576, 97 frames, it's ~30 min/clip (1774s) — of which Stage 1 is 24 min (15 steps × ~95s). That's ~7.5× slower than distilled q4 (~4 min), but bricks, water, and edges come out clearly sharper. Iterate drafts on q4, then bake only the final on q8 — that was the practical answer.

---

6. The recipe (the actual commands I ran)

1) Script (local LLM, fully autonomous)

mlx_lm + gemma-3-12b-it-4bit generates the shot list, per-shot visual prompts, and narration together.

"$PY/mlx_lm.generate" --model mlx-community/gemma-3-12b-it-4bit \
  --max-tokens 1100 --temp 0.75 --prompt "$PROMPT" \
  | sed '/^==========/d' > script.md

The prompt enforces "a strong hook in the first 3 seconds," "visual prompts in English describing people/place/light/motion," and "no speculative numbers, no exaggeration, no guarantee wording."

2) Keyframes (ComfyUI / SDXL)

POST a graph to ComfyUI's REST (:8000) to get stills. Sampler: dpmpp_2m / karras / steps 30 / cfg 6.5.

"3": {"class_type": "KSampler", "inputs": {
    "seed": seed, "steps": 30, "cfg": 6.5,
    "sampler_name": "dpmpp_2m", "scheduler": "karras", "denoise": 1.0,
    ...}}

Character consistency is the lifeline, so reuse a fixed design description of the main characters across every prompt. SDXL loves to add extra people, so put three people, group in the negative prompt to hold it back.

A real keyframe generated with ComfyUI/SDXL (the starting image of cut01)

Character consistency held via a fixed design description (cut07, third retake)

3) I2V (the video body — LTX-2.3)

Draft = q4 (~4 min/shot). Run it alone, with caffeinate to block sleep.

caffeinate -i -s ~/ltx-mlx/bin/python -m mlx_video.generate_av \
  --model-repo notapalindrome/ltx23-mlx-av-q4 \
  --text-encoder-repo mlx-community/gemma-3-12b-it-4bit \
  --image kf.png --prompt "your motion, cinematic film footage, natural gentle motion, warm golden light" \
  --negative-prompt "static, morphing, warping, extra fingers, flicker, jitter, low quality" \
  -W 1024 -H 576 --num-frames 65 --fps 24 --no-audio \
  --output-path cut.mp4

Production = q8 HQ (~30 min/shot, high fidelity).

/Users/yuma/ltx-2-mlx/.venv/bin/ltx-2-mlx generate \
  --model dgrauet/ltx-2.3-mlx-q8 --two-stages-hq \
  --image keyframe.png 0 1.0 \
  --prompt "your motion, cinematic film, natural motion, warm golden light" \
  --frames 97 --frame-rate 24 -H 576 -W 1024 \
  --stg-scale 1.0 --cfg-scale 4.0 --seed 42 \
  --output cut.mp4

A real generated cut from this pipeline (GIF-compressed for the article; the original is 1024×576, 24fps, with audio)

Key notes:

• Resolution must be a multiple of 64 (1024×576 recommended = true 16:9). For vertical Shorts, -H 1024 -W 576.
• num-frames is 8n+1 (max 97 on the q4 family). Distilled models ignore --steps (fixed at 8+3).
• --no-audio drops the model's audio so you can add your own narration afterward (the right call for educational content).
• --image PATH FRAME STRENGTH lets you anchor both end frames, interpolating longer continuous motion.

4) Narration (Kokoro TTS)

~/ltx-mlx/bin/python -m mlx_audio.tts.generate \
  --model mlx-community/Kokoro-82M-bf16 \
  --text "English narration" --voice bm_george --lang_code b \
  --output_path seg_01.wav
# Get real duration: ffprobe -v error -show_entries format=duration -of csv=p=0 seg_01.wav

You'll need pip install "misaki[en]". If a particular sentence throws a shape error, tweak the wording and retry. Kokoro is light and doesn't trip the GPU watchdog, so it's fine with the screen on. macOS's say -v Daniel -r 165 works in a pinch too (free, reliable).

5) BGM (MusicGen)

audiocraft chokes on the av / pkg-config build, so skip it — the built-in MusicGen in transformers is the answer. facebook/musicgen-small on CPU (~4.6 min for 20s).

from transformers import AutoProcessor, MusicgenForConditionalGeneration
model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")
prompt = "warm gentle emotional cinematic film score, soft solo piano with tender strings, slow, heartfelt, in a major key"
audio = model.generate(**inputs, do_sample=True, guidance_scale=3.0, max_new_tokens=1024)  # ~20s

Loop the 20s with acrossfade to match length, and mix it in gently at volume=0.30.

6) Assembly (ffmpeg)

Upscale each shot to 1080p (lanczos + a light grade), burn the subtitles, add narration, concat all shots, and amix the BGM at the end. Normalize loudness with loudnorm=I=-15. Burn subtitles per shot so they never drift from the narration.

---

7. The full gotcha list (the main event for developers)

This is what I most want to share. I stepped on all of these so you don't have to.

★ Metal GPU watchdog crash (MLX issue #3267)

Fig 3: The watchdog mechanism, and the reversal with the new package

The symptom:

[METAL] Command buffer execution failed: Impacting Interactivity
(kIOGPUCommandBufferCallbackErrorImpactingInteractivity)

The eerie behavior: only the first LTX run after a reboot succeeds; the second onward dies instantly.

The root cause is confirmed in MLX issue #3267: while the display is on, WindowServer composites the screen on the GPU; when MLX's heavy command buffer blocks that compositing, macOS kills the process to "preserve interactivity." Display on = 100% reproducible; off = 100% avoided.

What did NOT work:

• caffeinate -i -s — prevents system sleep, but doesn't turn off the display, so it's useless here.
• pmset displaysleepnow — if an external monitor stays alive, compositing continues; no good.
• Stopping ComfyUI / /free — unrelated.

For the old package (mlx-video-with-audio distilled q4), the reliable fix was: 1) reboot the Mac to clear the post-hang driver leak, 2) power off the external monitor and close the lid so nothing is displayed, 3) run the LTX batch in that state. That got all 10 shots through with zero crashes.

But the newer package overturned this. dgrauet/ltx-2-mlx (q8 HQ) generated a shot with the screen left on → 30.8 min, zero crashes, no new crash report. So you can run HQ batches in the background during the day while a human uses the screen. No more being chained to unattended overnight jobs. It was a nice moment to walk back the earlier "screen must be off" conclusion with real measurements (do keep ventilation, though — it runs hot).

★ Homebrew ffmpeg has no libass → subtitles won't burn

The Homebrew ffmpeg is built without libass/freetype, so neither the subtitles nor the drawtext filter exists (if ffmpeg -filters | grep subtitles is empty, that's you). You can make an SRT, but you can't burn it.

The workaround: draw subtitles onto a transparent PNG with Pillow, then burn them with ffmpeg's overlay. Font: /System/Library/Fonts/Supplemental/Arial Bold.ttf. Use a thick, dark stroke and lay a semi-transparent rounded plate behind the text for readability.

★ Boomerang (reverse-loop) artifacts

An LTX clip is ~2.7–4s (97 frames). If you pad a long-narration shot with "forward + reverse (boomerang)," you get reverse artifacts — things move and then un-move (e.g. someone "un-sips" a drink). That's the single biggest deduction on motion quality.

The fix: don't reverse. Hold the last frame as a clone and add a gentle drift:

tpad=stop_mode=clone:stop_duration=X   # clone-hold the final frame out to the narration length
zoompan=z='min(1.0+0.0006*on,1.11)':d=1:fps=24   # imperceptible drift

If you need genuinely longer continuous motion, chain LTX clips (last frame of a clip → next keyframe → generate → concat).

★ 4x-UltraSharp (ESRGAN) breaks on Apple MPS

Using ESRGAN-family upscalers (4x-UltraSharp) produces scanline-style breakup on Apple's MPS. Don't. If you need 4K, a clean ffmpeg upscale (scale=3840:2160:flags=lanczos + a light unsharp) looks perfectly fine.

★ The scratchpad (/tmp) is volatile

Put working scripts in /tmp and they vanish on reboot/cleanup. Keep important scripts in your deliverables folder. A humble lesson that pays off.

★ LoRA turned out to be possible (retraction)

I originally concluded "LoRA is Wan-only, LTX doesn't support it, it's a dead end." That was wrong. I confirmed support in dgrauet/ltx-2-mlx generate --help:

• --lora PATH STRENGTH — a normal LoRA (motion boost, speed LoRA, etc.)
• --distilled-lora / --distilled-lora-strength — LoRA for the distilled (fast) path
• --stg-scale composes with --lora (per-block application)

So the "LTX + multiple LoRAs" route is real. You don't have to touch ComfyUI-PyTorch (high risk of breaking your torch install) — you can attach LoRAs straight from the MLX CLI. For fight-level heavy motion, the main line is "q8 --two-stages-hq + --stg-scale 1.0 + a motion LoRA."

---

8. Quality-score progression (the honest log)

Fig 4: The honest climb — still short of the 95-point gate

Internally, videos go through independent scoring (the author doesn't grade their own; a separate session does) with a 95-point gate. Here's how the actual score moved:

v1 slideshow → rejected
 → Wan (I2V) 70
 → LTX character-consistent 76 → 77
 → all shots real LTX 78
 → all boomerangs removed 81
 → + Kokoro / + MusicGen / subtitle plate / -15 LUFS ≈ 85 (estimated)

From slideshow → real video → ~85, still free. Not at 95 yet (motion length and timing are the ceiling). I'll be honest about that. The next moves are LTX chaining and "HQ + motion LoRA" to push toward 90. The accurate framing isn't "free and local is perfect" — it's "free and local got us this far."

---

9. The recipe (shortest path)

1. venv with arm64 python (don't grab Anaconda's x86_64) → install mlx-video-with-audio, pin transformers>=4.50,<5.
2. Script — mlx_lm.generate + gemma-3-12b-it-4bit for shot list + visual prompts + narration.
3. Keyframes — ComfyUI (:8000) + realcartoon-xl-v4. Reuse a fixed character design across shots; suppress extra people via the negative prompt.
4. I2V — draft on q4 (generate_av, ~4 min), production on q8 --two-stages-hq + STG (~30 min). Resolution a multiple of 64, frames 8n+1, --no-audio.
5. Narration — Kokoro TTS (mlx_audio / misaki[en]). Get real duration with ffprobe.
6. BGM — transformers' built-in MusicGen (facebook/musicgen-small, CPU). Mix gently at volume=0.30.
7. Assembly — ffmpeg to 1080p + grade, subtitles via Pillow transparent PNG → overlay (libass workaround), no reverse — clone-hold + drift, then amix the BGM + loudnorm=I=-15.
8. Time-shift heavy GPU work (colliding with other work on the same Mac slows or crashes both). For long jobs, caffeinate -i + good airflow.

---

10. To everyone running Claude locally

A message and some tips for developers running Claude/LLMs locally, like me.

You don't have to pay — one Apple Silicon machine gets you this far. Cloud I2V (Kling/Runway/Veo…) is gorgeous and fast, but it means external calls and billing. Keeping everything on your own Mac is, beyond cost, mostly about the calm of holding all the pieces yourself. When you get stuck, the cause is entirely inside your own environment.

Tips to dodge the pitfalls:

• Don't melt hours on setup. 80% of "mlx won't install" and "mysterious AttributeError on startup" is either "you grabbed an x86_64 python" or "transformers is 5.x." Kill those two first and the rest is the fun part.
• Keep asking "is it actually moving?" Don't settle for a zoom over a pretty still. It's not a video until it goes through I2V. Cheap and fast but wrong is worth zero.
• Turn failure logs into assets. Save every winning prompt + seed, every model's runtime, every landmine into knowledge/. Reproducibility is the asset. Our score climbed 77→85 precisely because we turned each failure into one line of notes.
• Time-shift heavy GPU tasks. Run them alongside other heavy work or your own session on the same Mac and both slow down or crash. There's one GPU. Batch before bed or while away.
• Be proud of "this far, for free." It doesn't have to be perfect. Slideshow → real motion, one step at a time. Getting to share that log with each other is the best part of being local-Claude folks.

I'll make another one, inside this same Mac. Tell me where your pipeline gets stuck sometime, too. 🌿

---

(All figures and timings are measured on this M1 Max 64GB setup. No speculative numbers, no guarantee wording.)