DEV Community: Tonera

How to Run a TopTier OpenSource Image Model Qwen-Image-2.1 on 10GB VRAM

Tonera — Thu, 29 Oct 2026 06:42:05 +0000

The stock Qwen-Image-2.1-Turbo in BF16 needs about 20GB just to load and peaks near 27GB during text-to-image generation — a 10GB card like the RTX 3080 simply runs out of VRAM. The fix in one sentence: quantize the image Transformer and quantize the text encoder too, all while keeping image quality high and making it even faster!

Enter the star:tonera/Qwen-Image-2.1-Turbo-Nunchaku** — an SVDQuant 4-bit quantization of the official original!**

You can download this quantized model from Hugging Face or the ModelScope community. The companion text encoder is tonera/Qwen3-VL-8B-Instruct-Nunchaku. Tested on a 32GB machine: the native BF16 model OOMs — there simply is not enough VRAM to run it. After quantizing the Transformer (1024×1024 resolution, 8 steps), VRAM usage drops to 29.26GB with denoising at 7.64 it/s — images feel instant, done in about a second. With the text encoder also quantized, VRAM usage falls to 18.73GB at a denoising speed of 7.73 it/s. Even then, we could only run it on the RTX XX90 series — 24GB cards. With CPU offload enabled, VRAM usage is just 8.73GB, meaning as long as you have more than 9GB of VRAM, you can fully run this model. Some may say running the GGUF format works just as well. Yes, true — but this model's runtime is the Nunchaku community edition (tonera/vitoom-nunchaku). Measured with CPU offload on, its denoising speed still hits 6.11 it/s; for 8-step inference it still feels like ~1-second image generation.

Data below: measured on the same model card, an RTX 5090, quantized version vs. the BF16 original:

Metric	Key result
VRAM after load	19.83 → 10.63 GiB, saving 9.20 GiB (46.4%)
Text-to-image peak VRAM	26.44 → 17.23 GiB, saving 9.22 GiB (34.8%)
Stable text-to-image inference	3.080s → 1.269s, ~2.43×; editing ~2.02×
Image quality (25-image comparison)	Text-to-image SSIM 0.843, LPIPS 0.185 — usable but not pixel-perfect

Note: the 2.43× is the relative speedup of the quantized version vs. BF16 on the same 5090; absolute times differ on other cards. Loading is only ~1.15–1.18× faster — most of the speedup comes from step-by-step inference.

An LPIPS of 0.185 after 4-bit quantization of a large image model is already an excellent score — it means fidelity is quite high. Sacrificing a bit of quality for speed and usability is perfectly acceptable.

Below is a comparison of quantizing only the Transformer vs. quantizing both the Transformer and the text encoder.

Both images perfectly follow the prompt constraint — "a cute girl is holding a sign reads: CreatePPTX.com" — and the version with the quantized text encoder even rendered a richer background: a potted plant.

We also have to mention the tonera/flux-3-image-klein-pup-nunchaku model here. It was originally released by Black Forest Labs for flux3-klein, designed to expand a one-line prompt into a richer, more detailed image prompt. But it works for Qwen-Image-2.1 as well — if you are not great at writing prompts, this gem is for you.

Quick Reference for Other GPUs

With the same "quantized image model + INT4 encoder" combo, all of the following cards can run it locally; 30/40 series use INT4, 50 series use FP4.

VRAM tier	Models (precision tier)	Recommendation
32GB	RTX 5090 (FP4)	Full 1024², no offload
24GB	3090, 3090 Ti, 4090, 4090D (INT4); 5090D v2 (FP4)	Runs with default settings
16GB	4080 SUPER, 4080, 4070 Ti SUPER, 4060 Ti 16GB (INT4); 5080, 5070 Ti, 5060 Ti 16GB (FP4)	Text-to-image works; offload on demand for editing
12GB	3080 Ti, 3060, 4070 Ti, 4070 SUPER, 4070 (INT4); 5070 (FP4)	Enable offload, start at lower resolution
10GB	RTX 3080 (INT4)	Offload + lower resolution, good for prompt sketching

Note: use FP4-quantized Transformer weights on Blackwell chips; use INT4 everywhere else. The text encoder always uses INT weights.

Boundary Notes

Being able to run on 10–12GB cards depends on offload and lower resolution — it is not full-power, zero-wait image generation.
Quantization is great for fast local iteration; for high-precision final deliverables, re-render the few selected images in BF16 or in the cloud.
The weights are derived from Qwen/Qwen-Image-2.1-Turbo under the Qwen Research License; check the upstream license before commercial use.

In one sentence: running Qwen-Image-2.1 on a 10GB card is not about more VRAM — it is three things: model quantization + encoder quantization + offload scheduling.

Recommended Quantized Ultra-Fast Prompt Expander for FLUX 3 / Qwen-Image-2.1

Tonera — Wed, 28 Oct 2026 03:20:27 +0000

Don't Blame the Model First for Plain Output: A Killer Prompt Expander Using Only 7GB VRAM

In short, whether a good image comes out rarely comes down to the model itself.

Feed an AI image generator a one-liner like "a person walking a dog in afternoon sunlight" and the result often looks flat: bare or one-note background, poor detail, wrong lighting, mediocre composition. Most users blame the model. But the official FLUX 3 docs put it plainly: Klein was trained on long scene descriptions; feeding it a raw one-liner causes a noticeable drop in quality.

Put another way, what's missing isn't a better image model — it's the middle layer that turns a one-liner into a director's script. Black Forest Labs released a dedicated Prompt Upsampler for this, but the model needs 18GB of VRAM to run, and prompt expansion alone takes 4–5 seconds, effectively ruling out consumer GPUs. The good news: the community has compressed this 18GB expander to roughly 7GB VRAM while delivering a 3.6× speedup. The prompts it writes aren't limited to FLUX 3; they work with any other image generation model.

01 What's Actually Worth Installing Is a Prompt Expander, Not Another Image Model

FLUX 3 Image Klein is a 5.6B open-source image model released by Black Forest Labs in October 2026, supporting text-to-image, reference-based editing, up to 10 reference images, and boxed layout. Its companion prompt expander (flux-3-image-klein-pup) is a language model fully fine-tuned from Qwen3.5-9B; it expands your one-liner into the full description image models actually need: medium, style, scene, lighting, camera, plus the color, material, and position of every subject.

It isn't a drawing model — FLUX 3 still handles the actual rendering; nor is it a generic copy-polishing tool. Its output is a "technical script" written for the image model. It sits between "your idea" and "the image model," lowering the most overlooked barrier: prompt quality. Klein was trained on exactly this kind of long scene caption, and the official docs state clearly that bypassing the expander leads to noticeably worse results.

02 Why Quantize This 18GB "Middleman" to INT4

The expander doesn't render images, yet its BF16 weights alone take 18.8GB, with peak VRAM near 18GiB during generation. Users running FLUX 3 locally already allocate most VRAM to the image model; reserving another 18GB for this middleman makes even 24GB cards unusable.

This quantized version (https://huggingface.co/tonera/flux-3-image-klein-pup-nunchaku) uses SVDQ: every linear layer across the 32 language layers is compressed to INT4 weights (W4A16, group size 128), including the lm_head; the vision tower, token embeddings, and RMSNorm layers remain in floating point. Calibration data consists of prompts generated by the original model, mixed with roughly 25% general-domain text.

Results: peak generation VRAM drops from 17.75 GiB to 7.02 GiB, saving about 10.7 GiB (60.5%); per-token latency falls from 15.43 ms to 4.29 ms, a 3.6× speedup.

03 Benchmark Data: ~1.4 Seconds to Generate a 300-Word Prompt

The model card provides two datasets. The first covers per-token speed and peak VRAM (RTX 5090):

Text-to-image expansion: 15.43 ms/token → 4.29 ms/token, 3.60× speedup; VRAM 17.75 → 7.02 GiB, 60.5% savings.

Single-image editing expansion: 15.35 → 4.35 ms/token, 3.53× speedup; VRAM 17.93 → 7.21 GiB, 59.8% savings.

The second set covers end-to-end benchmarks on a resident server: text-to-image expansion takes ~1.42 seconds total (including 0.086 s to first token), outputting 297 tokens at ~221 tokens/s; editing expansion takes ~1.17 seconds, outputting 200 tokens at ~220 tokens/s.

Methodology: bfloat16, greedy decoding, thinking mode disabled; peak VRAM does not scale with output length. These figures reflect only this test environment and are reference values on other GPUs.

04 Usage: Spin Up a Resident Prompt Server with One Command

The recommended deployment for this quantized build is a resident expansion service: once loaded onto the GPU, the model stays resident, so subsequent requests generate directly without reloading. It exposes an OpenAI Chat Completions-compatible endpoint, callable as-is by existing OpenAI SDKs.

Start the service (listening on 0.0.0.0:8000 by default):

python api.py

The service supports two modes: plain-text text-to-image expansion (t2i) and reference-guided editing expansion (i2i). Reference images can be passed as http(s) URLs, base64 strings, or local file:// paths on the server; up to 10 images per request, each no larger than 20MB. The full workflow is shown below:

Once the service is running, call it directly with curl:

curl http://127.0.0.1:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"flux-3-image-klein-pup","messages":[{"role":"user","content":"A girl is holding a sign says:CreatePPTX.com"}]}'

If you prefer not to run a server, you can call NunchakuQwen3_5ForConditionalGeneration directly in Python, performing expansion in code via a Transformers-style API. Both approaches require two dependencies: a Transformers build supporting Qwen3.5, and the Nunchaku wheel. Note that the official Nunchaku build is currently unavailable; you must use the community prebuilt version (https://huggingface.co/tonera/vitoom-nunchaku) and select the wheel matching your local Python, CUDA, and PyTorch versions.

Deployment note: the service has no authentication by default, and file:// paths can read local server files, so it should only be exposed on a trusted network or placed behind an authenticated reverse proxy.

05 Common Misconceptions

INT4 and BF16 outputs are not word-for-word identical. Wording and length differ slightly (INT4 output is a bit shorter), but the quality delivered to the image model is comparable; exact verbatim reproduction is unnecessary.

Performance benchmarks were measured only on an RTX 5090. Speed and VRAM figures on other GPUs should be treated as rough references.

The service only exposes t2i and i2i modes. The upstream model also supports boxed layout; to use it you must bypass the service and call the model directly.

Expansion isn't a silver bullet. If you already write detailed descriptions yourself, the expander adds limited value; its strength lies specifically in bridging the gap from a one-liner to a long description.

06 Decision Framework: When It's Worth Installing

Worth installing: you run the full FLUX 3 stack or Qwen-Image-2.1 locally with tight VRAM (especially 24GB or lower); you want a resident prompt expansion service reused across existing workflows; you frequently do reference-based secondary edits. If any one of these applies, the 7GB VRAM investment pays off.

Not necessary: you only use cloud APIs and do no local generation; you have abundant VRAM and don't care about the 18GB footprint; you only expand prompts occasionally and don't want to maintain a resident service.

Bottom line: it doesn't improve model capability; it reduces the translation cost between "an idea" and "a prompt." The INT4 build cuts the VRAM cost of this translation from ~18GB to 7GB and reaches 221 tokens/s, making a local FLUX 3 workflow feel "light" for the first time.

Recommendation: start the expansion server locally first, plug it into your existing workflow to test a round of prompts, and decide on long-term deployment after confirming output quality. The model is released under Apache 2.0 and can be downloaded for evaluation freely.