I spent a few weeks benchmarking training-free block-residual caching for FLUX.1-dev on an Apple M5 Max. The timing result was real: a stable ~1.16× speedup, tight variance, reproducible across prompts.
Then the quality sweep came back and every policy failed my gate. Median PSNR of 14.27 dB against a gate of 30 dB. Decisive failure.
I almost published that as "block-residual caching doesn't preserve quality on Apple Silicon." That would have been wrong — not because the numbers were wrong, but because PSNR wasn't measuring what I thought it was measuring. This post is about the speedup, the trap, and the one cheap calibration that caught it.
Everything here is reproducible — harness, raw per-pair data, and the two image pairs the argument turns on:
github.com/kkjcodes/m5-flux-block-cache-benchmark
The setup
FLUX.1-dev has 19 joint transformer blocks (image and text streams attending together) and 38 single blocks (a fused stream). Phase attribution on my machine said:
- DiT forward: 95.2% of uncached wall time
- Single-stream blocks: 63.8%
- Joint blocks: 31.2%
So essentially all the time is in the transformer, and the single blocks are where the money is. That's the headroom.
The idea behind block-residual caching is simple. A transformer block computes out = f(in). Instead of caching out (which is wrong the moment the input changes), you cache the residual out - in, then on a reuse step you apply the stale residual to the fresh input:
def _store(self, kind, value, **kwargs):
if self.value_mode == "raw":
return value
if kind == "joint":
# joint blocks return (encoder_hidden_states, hidden_states)
return value[0] - kwargs["encoder_hidden_states"], value[1] - kwargs["hidden_states"]
return value - kwargs["hidden_states"]
def _restore(self, kind, cached, **kwargs):
if self.value_mode == "raw":
return cached
if kind == "joint":
return kwargs["encoder_hidden_states"] + cached[0], kwargs["hidden_states"] + cached[1]
return kwargs["hidden_states"] + cached
Reuse is on a fixed interval — every other step:
def should_reuse(self, step_index: int, timestep: float) -> bool:
return step_index > 0 and step_index % self.interval != 0
Config throughout: 4-bit quantized FLUX.1-dev, 28 steps, linear scheduler, 1024×1024, Apple M5 Max. The 4-bit part matters — quantization shifts the compute/memory balance that determines how much caching can win, so none of these numbers should be assumed to transfer to a full-precision build.
The timing result
Twelve measured runs per policy, one warmup, acceptance gated on delta > 3 × sqrt(baseline_stdev² + candidate_stdev²):
| Config | Blocks | Clean median | Stdev | Speedup |
|---|---|---|---|---|
| baseline_uncached | none | 119.33s | 2.89s | 1.000× |
| joint_12_18_i2_residual | joint:12-18 | 134.82s | 18.15s | 0.885× ❌ |
| joint_10_18_i2_residual | joint:10-18 | 98.82s | 2.86s | 1.208× |
| joint_8_18_i2_residual | joint:8-18 | 100.50s | 2.60s | 1.187× |
| single_28_37_i2_residual | single:28-37 | 101.03s | 2.20s | 1.181× |
| joint_12_18_single_28_37 | joint:12-18,single:28-37 | 91.87s | 0.99s | 1.299× |
Real speedup, cleanly separated from noise. But that table has a problem I only caught later, so hold onto it.
The quality sweep
4 prompts × 8 seeds × 3 surviving policies = 96 comparisons. Each variant is compared against an uncached reference generated with the same seed — same latents, same prompt, same everything but the cache. Gate: median PSNR >= 30 dB, median global-SSIM >= 0.98.
| Policy | Median PSNR | Median Global-SSIM | Gate |
|---|---|---|---|
| joint_10_18_i2_residual | 16.75 | 0.857 | fail |
| joint_12_18_single_28_37 | 14.27 | 0.693 | fail |
| single_28_37_i2_residual | 14.23 | 0.727 | fail |
Total wipeout. 0 of 96 pairs cleared 30 dB. Zero cleared 25.
So: caching breaks the images, right?
The trap
I pulled the single worst pair in the sweep — 11.34 dB, a typography prompt — expecting mush.
It was a clean, well-composed "OPEN LATE" bookstore sign. Sharp letterforms, correct prompt adherence, nice rain-slicked reflections. It was a good image. It just wasn't the same image as the reference.
That reframes everything. PSNR against a same-seed reference doesn't measure quality. It measures trajectory divergence — how far the denoising path drifted from where it would have gone. A cached run that lands on a different-but-equally-good image scores catastrophically, and a genuinely degraded run scores catastrophically, and PSNR cannot tell you which one you're looking at.
Worse, the gate itself was unreachable. The single best pair in my whole sweep was 22.25 dB. I looked at it: same face, same pose, same lighting, same composition — differing in fingernail detail and pot texture. Visually near-identical output, and it still misses a 30 dB gate by 8 dB. A gate that rejects near-perfect output isn't a fidelity gate, it's a pixel-identity gate.
The calibration that caught it
Here's the cheap trick, and it's the most portable thing in this post.
If you don't know what your metric's numbers mean, measure the floor: score two images that are both good but share no trajectory at all. For me that's two reference images, same prompt, different seeds — no cache anywhere near them.
# both images uncached, both good, different seeds
for a, b in itertools.combinations(seeds, 2):
psnr, global_ssim = compare(refs[a], refs[b])
Result over 112 unrelated pairs: 9.73 dB, global-SSIM 0.23.
Now the whole scale snaps into focus:
9.73 dB two unrelated good images <- the floor
14–17 dB my cached variants
22.25 dB visually near-identical (verified by eye)
30.00 dB my gate <- nothing reaches this, ever
Same story in global-SSIM units: floor 0.23, my variants 0.69–0.86, gate 0.98.
My variants at 14–17 dB sit meaningfully above "unrelated," so the cache does retain real trajectory structure. But the worst cases at 11.3 dB are creeping toward "might as well have used a different seed." That's a genuinely interesting, defensible finding — and I could only state it because I had the floor. Without it, "14.27 dB" is a number with no semantics.
If you're gating generative output on a reference metric, go measure your floor first. It takes ten minutes and it tells you whether your gate has any discriminating power at all. Mine didn't.
Two more things I got wrong
My SSIM wasn't SSIM. I'd written a global SSIM over the whole flattened image. Real SSIM (Wang et al., which is where the 0.98 convention comes from) is an 11×11 windowed mean computed per channel. My numbers were inflated by ~0.15 across the board — 0.857 where skimage says 0.698. It didn't flip any verdict, but I'd have published a column labeled "SSIM" that wasn't. If you hand-roll a metric, diff it against the reference implementation before you put it in a table.
My headline speedup didn't replicate. That 1.299× came from one prompt ("A simple red cube on a white table") at one seed. The quality sweep incidentally timed 32 prompt/seed cells per policy, so I ran a paired comparison:
| Policy | Paired median (n=32) | IQR | Headline (n=1 prompt) |
|---|---|---|---|
| joint_10_18 | 1.085× | 1.067–1.113 | 1.208× |
| single_28_37 | 1.095× | 1.075–1.110 | 1.181× |
| joint_12_18_single_28_37 | 1.159× | 1.141–1.185 | 1.299× |
Ordering preserved, variance tight, and speedup is essentially prompt-independent (1.084–1.108 across four very different prompts) — so "stable" is actually better supported than the single-prompt run showed. But every magnitude lands ~10 points lower. The n=32 sample is the weaker timing protocol (no warmup, N=1 per cell, fixed ordering), so it's not a refutation — but it's my own data disagreeing with my own headline, and the honest move is to quote the range and name the protocol.
The big caveat: I never tested a warmup window
Look at that reuse schedule again:
step 0: compute → step 1: REUSE → step 2: compute → step 3: REUSE ...
14 of 28 steps reused. First reuse at step index 1.
Caching kicks in at the second denoising step — while composition is still being decided. And composition divergence is exactly what my PSNR was detecting.
Every established training-free cache method (DeepCache, TeaCache, FBCache) skips an early warmup window for precisely this reason. My policy dataclass has no start-step field, so I couldn't express "skip the first N steps" even if I'd wanted to. I tested the configuration most likely to diverge and then generalized from it.
That's the next experiment, and I'd bet it's where the actual frontier is.
What I'm actually claiming
Training-free block-residual caching exposes real acceleration headroom for 4-bit FLUX.1-dev on Apple M5 Max: broad single-stream and joint+single coverage produce stable, prompt-independent speedups in the 1.09×–1.30× range depending on policy and timing protocol.
But fixed-interval residual reuse — enabled from the first denoising step, with no warmup exclusion, at the block spans tested — does not preserve same-seed reference trajectories, and divergence grows with coverage.
Image quality remains unmeasured. I have no perceptual metric and no prompt-alignment metric in this run. The images I inspected by hand looked good. I'm not claiming they are, because I didn't measure it.
That's a timing and trajectory-divergence characterization. It is not a quality result, and I'm not going to dress it up as one.
Next up: LPIPS, real windowed SSIM, and CLIP prompt-alignment over the 32 references and 96 variants already sitting on disk — each with its own cross-seed floor, because now I know better than to report a number without one. Then a warmup-window knob.
The code and the data
Everything is on GitHub: github.com/kkjcodes/m5-flux-block-cache-benchmark
The repo ships the harness (a custom mflux denoising loop with block-level hooks, so the timing has no hidden synchronization in it), plus the audited evidence bundle — all 96 per-pair rows in quality_results.jsonl, the timing sweep JSON, the environment record, and the two reference/variant pairs this post argues from. The 153MB of remaining images stayed out, but the two that carry the argument are there, so you can look at the 11.34 dB "failure" and the 22.25 dB "near-identical" pair and judge my read for yourself.
If you disagree with my interpretation, the raw numbers are right there to disagree with.
If you're doing similar work on Apple Silicon: measure your floor, diff your hand-rolled metrics against reference implementations, and never let a headline number rest on one prompt. All three of my mistakes were free to catch and would have been expensive to publish.
Top comments (0)