A 1.125x FLUX speedup was real. The harder question is where diffusion gives back time.

In the previous post, I benchmarked training-free block-residual caching for 4-bit FLUX.1-dev on an Apple M5 Max.

The short version: caching helped, but it did not give me the clean win I wanted.

The longer version was more interesting. A fixed-interval cache produced real speedups, then failed a same-seed PSNR gate so badly that I almost wrote the wrong conclusion. The images were not necessarily bad. They had drifted from the uncached denoising trajectory. That sent me down the rabbit hole of metric floors, windowed SSIM, LPIPS, CLIP scoring, prompt stratification, and interleaved timing.

That work is here:

github.com/kkjcodes/m5-flux-block-cache-benchmark

This post is the handoff from that experiment. Not a victory lap. Not a promise that I have the next speedup solved. More like: here is where block caching stopped being the most interesting question.

Where the cache experiment landed

The best policy I found was a warmup split:

joint blocks 12-18  -> start_step=8
single blocks 28-37 -> start_step=6
interval            -> 2
value mode          -> residual

In the code, start_step is the warmup boundary. That boundary step is still a refresh point; reuse is allowed after it on interval misses.

That policy passed an interleaved timing gate at 1.125× and scored:

Metric	Result
Median PSNR	29.06
Median windowed SSIM	0.94575
Median LPIPS Alex	0.05519
Median CLIP delta	0.00181

That is a real result. It is also a bounded result.

The cache is not pixel-identical. It still fails a strict same-seed trajectory gate. And the improvement is content-dependent: portraits, product renders, and typography are close to the uncached references, while dense interiors remain the hard case.

Here is the prompt-stratified picture:

Prompt class	PSNR	Windowed SSIM	LPIPS Alex
dense_interior	22.62	0.83994	0.12326
portrait	29.90	0.95402	0.04808
product_render	29.10	0.96124	0.03791
typography	31.80	0.94660	0.05070

That table is the actual frontier. Not "cache works." Not "cache fails." More like: warmup-delayed residual reuse is near-fidelity on some content classes, but dense interiors expose the weakness.

Here is one of the easier cases. Same prompt, same seed, uncached reference on the left and warmup-split cached variant on the right:

Uncached reference	Warmup-split cached variant

And here is the kind of case that kept the result honest. Dense interiors were still hard, even after the residual gate. The two cached versions are near-indistinguishable from each other, which is the point: the gate fired, but it did not change this failure mode enough.

Uncached reference	Warmup split	Residual-gated m2

The adaptive gate I thought would help

The obvious next move was residual-volatility-gated reuse.

The idea was simple enough: keep the warmup split schedule, but when a block's residual changes too much relative to its recent history, veto reuse and recompute the block.

I pre-registered the target in plain English: lift dense-interior PSNR from 22.62 toward roughly 29, without dropping below the split policy's timing class.

That did not happen.

On the matched interleaved timing comparison, the split re-measured at 1.121× in the same run as the gated policy. The gated threshold measured 1.093×. So the gate gave back speed.

Quality moved only a little:

Policy	Dense PSNR	Dense LPIPS
split warmup	22.62	0.1233
gated m2	22.82	0.1146
gated m3	22.78	0.1166

That is not close. It needed roughly 6 dB of dense-interior improvement. It found about 0.2 dB.

Then I checked whether the gate was even firing. It was. But it did not fire more on dense interiors than on easier prompt classes. Product renders and portraits often fired at similar or higher rates.

So the conclusion is narrow but useful:

This residual-volatility signal, at these thresholds and block spans, did not target the dense-interior failure mode.

That is not the same as "adaptive reuse can never work." It is just enough evidence to stop spending time on this particular gate.

What this changed about the next question

At the start, I thought the path was probably:

find cacheable blocks -> tune schedule -> add adaptive gate -> get speed

After the experiments, that feels too small.

A veto layer can only block reuse decisions that a fixed schedule already admitted. If the failure is created by the schedule itself, then a gate on top can only claw back part of the damage. That may be exactly what happened here.

Dense interiors may not need "reuse unless volatility spikes." They may need a different schedule, a different approximation surface, or a different way of spending compute across the denoising trajectory.

Which leads to the question I actually want to ask next:

Can we make diffusion faster without treating transformer blocks as the only reusable unit?

I do not have the answer yet. I am not claiming the next project will beat the warmup split. I am saying the cache benchmark changed what I think is worth investigating.

Why block caching may be the wrong level

Block-residual caching is attractive because it is mechanical. You can hook transformer blocks, store deltas, and measure wall time. It does not require training. It does not require changing the model. It gives you a clean benchmark surface.

But diffusion is not just a stack of interchangeable block calls. The denoising trajectory has phases. Early steps decide composition. Later steps refine detail. Different content classes stress different parts of the trajectory. A fixed block schedule ignores most of that structure.

The warmup result made that obvious. Skipping the early steps mattered more than almost anything else.

The residual-gate result made the next problem obvious. A block-level volatility score did not know enough about the failure mode to protect dense interiors.

So the next acceleration surface I want to study is narrower than "try more caching tricks": trajectory-aware compute scheduling.

The question is whether the denoising process can decide, step by step, when a full expensive transformer evaluation is worth paying for. Early steps carry composition. Later steps refine detail. Dense interiors may be sensitive in places where portraits and product renders are not. A fixed block-reuse schedule flattens all of that structure into a yes/no decision per block.

That feels like the wrong abstraction. The next question is whether cheap signals from the latent trajectory can guide the amount of compute spent across denoising phases without moving the image onto a different composition path.

That is a direction, not a commitment. The last project taught me to be careful with promises before the metric harness exists.

What I am taking forward

The useful artifact from this work is not just the 1.125× speedup. It is the measurement discipline.

For the next round, I want the same rules from day one:

• interleaved timing for candidate-vs-baseline comparisons;
• prompt-stratified quality summaries;
• cross-seed floors for reference metrics;
• perceptual metrics, not only PSNR;
• visual samples in the report;
• explicit distinction between trajectory preservation and image quality;
• negative results written down instead of quietly discarded.

That last one matters. The residual gate was a miss, but it was not wasted. It told me that a simple per-block residual-volatility signal is probably not the lever. It also told me that dense interiors are the content class to keep in the loop for every future speed claim.

A speedup that only works on simple prompts is a demo. A speedup that survives dense interiors is a result.

The current state

The block-residual caching milestone is closed.

What I am comfortable saying publicly:

• Fixed-interval block-residual caching exposes real acceleration headroom on this setup.
• Same-seed PSNR is a trajectory metric, not an image-quality verdict.
• Warmup-delayed reuse is the best tested cache policy so far.
• The warmup split reaches a timing-pass near-fidelity point at 1.125×.
• Dense interiors remain the hard case.
• Residual-volatility gating did not solve that hard case.

What I am not claiming:

• that the cache is generally quality-preserving;
• that 1.125× transfers to other hardware, quantization settings, or model variants;
• that adaptive reuse is dead;
• that I already know the next acceleration method.

The next thing is a question:

If block caching gives a bounded win, can trajectory-aware scheduling give diffusion a better way to spend compute?

That is where I want to go next.

---

The benchmark repo is here: github.com/kkjcodes/m5-flux-block-cache-benchmark

The README now includes the curated reports, visual samples, and the residual-gated negative result. If you are doing similar work, my strongest recommendation is still the boring one: measure your floor, interleave your timing, and do not let a single prompt write your headline.