A 512-pixel image contains 512 pixels of information. Every upscaler produces more pixels than that, so every upscaler is inventing the difference. The useful question is not which one recovers detail — none of them do — but what each one invents, and whether that invention is acceptable for what you are doing.
The short answer
Interpolation invents smoothness. Learned feed-forward upscalers invent the texture statistics of their training corpus. Diffusion refiners invent content consistent with a prompt. They differ by roughly two orders of magnitude in compute, and by much more than that in how far they will depart from the source.
All three are inventing
An image sampled at a given resolution cannot represent detail finer than its sampling limit. That detail is not attenuated, not compressed, not recoverable — it was never recorded. Producing a larger image therefore requires generating information that is not in the input.
The three families differ only in the prior they use to generate it. Interpolation’s prior is “the signal is smooth”. A learned upscaler’s prior is “this looks like the clean images I was trained on, degraded the way my training degraded them”. A diffusion refiner’s prior is an entire generative model, steered by your prompt.
Stronger priors give more convincing results and less faithful ones. That is the whole trade, and it is why the right choice depends entirely on whether the output needs to be evidence of the input.
The three families
Interpolation
Nearest neighbour, bilinear, bicubic, Lanczos. Fixed mathematics, no learned parameters, effectively instantaneous, entirely deterministic.
- Invents: smooth transitions between known samples, and nothing else. The output is a band-limited reconstruction of the input.
- Artefacts: softness at every scale factor; ringing near hard edges with Lanczos; blockiness with nearest neighbour.
- Correct uses: any resize where fidelity to the source is the requirement; the intermediate step before a refinement pass; downscaling, where it is not merely acceptable but optimal; pixel art, where nearest neighbour is the only correct choice.
Learned feed-forward upscalers
Convolutional or transformer networks trained on pairs of degraded and clean images, run once. The ESRGAN family and SwinIR are well-known published examples. One forward pass, deterministic for a given input, fast enough for video on a modern GPU.
- Invents: texture consistent with its training distribution, and a specific inverse of the degradation it was trained to undo.
- Artefacts: the classic failure is a degradation mismatch. The network learned to invert a particular kind of damage — a specific downsampling kernel, a specific compression quality — and given something else it applies the wrong inverse. Skin becomes waxy, foliage becomes a repeated pattern, and edges acquire a characteristic over-sharpened halo. Feeding it an already-clean generated image is exactly this mismatch, which is why generic upscalers often make generated images worse.
- Correct uses: restoring genuinely degraded material of the type the model was trained for; batch throughput where per-image cost matters; anything where a deterministic, cacheable result is required.
Diffusion refiners
Either an image-to-image pass at the higher resolution, a dedicated latent upscaler, or a tiled pass over a large canvas. These are generative: the result is a sample, so a different seed gives different detail, all of it plausible.
- Invents: content. Not just texture — a diffusion refiner at moderate strength will change what an object is if the prompt suggests it, and will happily add detail that was not implied by the source at all.
- Artefacts: drift from the source at higher strength; tile seams and per-tile prompt interpretation when tiled; texture that is convincing at 100 per cent and obviously synthetic at 400 per cent.
- Correct uses: finishing generated images, where the source has no evidentiary status and invention is the point; reaching sizes above the model’s training resolution, as in aspect ratio and training buckets.
The compute gap
Upscaling 1024² → 2048² (4× the pixel count).
Relative figures, using the per-step arithmetic derived in the
GPU-seconds page.
Interpolation
A handful of arithmetic operations per output pixel, over ~4.2
million output pixels. Milliseconds on a CPU. Call it 1 unit.
Learned feed-forward upscaler
One forward pass of a comparatively small network over the output
grid. Orders of magnitude more than interpolation, and still well
under a second on a GPU.
Diffusion refiner, 20 steps at strength 0.35 with guidance
actual steps = 20
passes/step = 2
cost/pass ≈ 7× a 1024² pass (from the GPU-seconds page)
→ equivalent to about 280 passes at 1024²
That is roughly 5× the cost of generating the original image at
30 steps with guidance (60 passes) — for a refinement.
The gap between the second and third families is two to three orders of
magnitude, depending on step count and strength.
The practical reading: a diffusion refinement pass is not a post-process, it is a second generation and should be budgeted as one. If you run one on every image in a pipeline, it is likely the largest line in your compute bill. The full arithmetic is in the GPU-seconds page, and the broader cost picture is in the hidden costs page.
Tiling, and the seam problem
Above the model’s native resolution a diffusion refiner must work in tiles, and tiling introduces failures the other families do not have.
Target 4096 × 4096, tile size 1024, overlap 128:
stride = 1024 − 128 = 896
tiles/side = ceil((4096 − 128) / 896) = 5
total = 5 × 5 = 25 tiles
Each tile is a full generation. At 20 steps with guidance:
25 tiles × 20 steps × 2 passes = 1,000 network evaluations
versus 60 for the original 1024² image.
- Each tile is prompted independently. Give a tiled upscaler the prompt “a portrait of a woman” and every tile tries to produce a portrait, including the one containing only a shoulder. Either use a generic prompt, or use a tool that generates per-tile prompts from the source content.
- Overlap alone does not remove seams. Adjacent tiles make independent decisions about lighting and texture, and blending the overlap averages two different answers. A tile-conditioning signal that feeds each tile a downscaled view of its own region keeps the decisions consistent — this is what a tile control branch is for, described in structural conditioning.
- Colour drifts across the canvas. Small per-tile shifts accumulate over 25 tiles. A global colour-match pass against the original, after tiling, is worth the two lines it takes.
- Keep the strength low. At 0.2 to 0.3 the tiles have little freedom to disagree. At 0.5 they will produce visibly different worlds either side of a seam.
Choosing, by what you need
| Requirement | Description |
|---|---|
| the output must represent the input | Interpolation only. Lanczos for photographic content, nearest neighbour for pixel-exact material. Any learned upscaler introduces content that was not there. |
| restore genuinely degraded material | A learned feed-forward upscaler trained on that kind of degradation. Check what degradation the model documents; a mismatch is the main failure mode. |
| finish a generated image | Interpolate up, then a diffusion refinement pass at strength 0.25 to 0.45. Cheaper and more controllable than generating at the larger size directly. |
| reach a very large canvas | Tiled diffusion with a tile-conditioning signal and low strength, then a global colour match. Budget it as roughly twenty generations, not as a filter. |
| throughput matters more than the last ten per cent | A learned feed-forward upscaler. Two to three orders of magnitude cheaper than diffusion and deterministic, which also makes it cacheable. |
When not to upscale at all
There is a class of use where every family on this page is inappropriate, and it is worth stating plainly because the tools do not warn you.
- Never upscale evidence. Security footage, a document image, a licence plate, a medical scan. The added detail is generated from a prior, and the prior will confidently produce plausible characters, faces and features that were not in the source. An enhanced image is a hypothesis, not a photograph.
- Never upscale a face you need to identify. A learned upscaler will produce a face consistent with its training distribution. That is a face; it is not necessarily the person.
- Do not upscale to hide a generation problem. If the source has six fingers, a refinement pass gives you higher-resolution six fingers. Fix it at the source with a masked pass.
- Do not upscale before archiving. Keep the original. An upscale is a derived artefact and can be regenerated; the source cannot.
Top comments (0)