DEV Community

Cover image for Serving MOSS-Image: step count, memory mode and the traps that change the words you ship
InferenceKey for InferenceKey

Posted on Originally published at inferencekey.com

Serving MOSS-Image: step count, memory mode and the traps that change the words you ship

If your product makes menus or posters through an image API, what matters is that the price on the menu is the price you sent. We served MOSS-Image-SFT, an Apache 2.0 model from OpenMOSS at MOSI.AI built for long text in pictures, on one A40 on our cloud. Two settings decided almost everything: the step count, and where the weights (the model's learned numbers) sit while it runs.

With the authors' 50-step recipe, every word was correct on every text test, at 78 to 81 seconds per picture of about one megapixel. A community turbo add-on from lodestones took 8.6 to 9.2 seconds and got two words wrong. The full MOSS-Image test on one A40 has all 22 pictures and the deploy steps. This piece covers the choices.

Why 50 steps cost 100 passes

The model begins with random static and removes some of it per step. Each of the 50 steps calls it twice, with your prompt and with an empty one, and the gap between the answers steers the picture toward your words. That is about 100 runs of a 5-billion-parameter network per picture.

The lodestones 8-step add-on is a LoRA: a 1.2 GB set of small corrections laid over the original weights, trained to finish in 8 calls with no empty-prompt call. Its card says the correction is smaller than bf16's rounding step (bf16 is the 16-bit format of the weights), so folding it in would erase it. It stays a separate layer at strength 0.7, as in the hugging-apps demo Space. That strength is shared by every request, so our backend (the server that loads the model and answers requests) runs one picture at a time. One loaded copy serves both modes:

{"prompt": "A clean conference poster on a white wall. Large bold title: OPEN WEIGHTS DAY",
 "size": "1024x1024", "seed": 42, "mode": "turbo"}
Enter fullscreen mode Exit fullscreen mode

The body follows OpenAI's image API, plus mode (sft or turbo). The seed fixes the starting static: keep it the same when you compare modes, or you are comparing luck.

The trap: running the 8-step LoRA at 4 steps

We tried the 8-step add-on at 4 steps. It took 5.2 seconds and the lettering broke.

The conference poster from the 8-step add-on run at only 4 steps, with the title OPEN WEIGHT DAYDY and garbled times

The title read "OPEN WEIGHT DAYDY" and times looked like "1(.C0". The add-on was trained for 8 steps. Adapters built for 4 exist, and we did not test them. At 8 steps, turbo repeated a word in the poster title, wrote "PNG bock" for "PNG back" and half-garbled street signs, while the chalkboard menu, the Chinese note and both layouts were right. Use it for drafts and proofread each picture.

Memory mode: pick the one that streams the right part

MOSS-Image is a pair: a 17 GB reader (the Qwen3.5-9B text encoder from the Qwen team, which turns the prompt into numbers) and a 10 GB painter. The reader is busy only briefly per prompt, so it can live in system RAM and move to the GPU on demand. To mimic smaller cards, we limited PyTorch's share of the A40 with torch.cuda.set_per_process_memory_fraction:

Memory cap Peak on the card 50 steps Turbo
None (real A40, 48 GB) about 30 GB 80 s 9 s
16 GB, reader in main memory about 14 GB 86 s 13 s
12 GB, reader and part of the painter in main memory about 11 GB 160 s 18 s

The generic enable_model_cpu_offload() from diffusers fit 24 GB but not 16 GB and ran slower than moving only the reader, so our automatic mode never picks it. A cap copies a card's memory size, not its speed, so trust the fit more than the timings. Neither the fit on a real card nor the automatic choice has been tested yet: we set each mode by hand.

What to measure before you trust a setup

The memory ladder and the per-picture measurements list everything. These checks carry over:

  • Byte equality. The static is drawn on the CPU from the seed, and offloading only moves weights. All 8 ladder pictures matched the uncapped run byte for byte (cmp). If yours differ, something besides placement changed.
  • Peak memory as the card reports it. nvidia-smi, NVIDIA's monitor, counts memory the CUDA runtime keeps for itself, which the PyTorch cap ignores. The 16 GB setup leaves about 2 GiB spare, less with a display attached.
  • System RAM. The 16 and 12 GB setups used up to 22 GB. Plan for 32.
  • Time to ready. Ours took 63 seconds, 29 of them downloading 28 GB. A custom backend gets 300 seconds, and at 100 MB/s the download alone takes about 280. On machines we rent for you that limit is fixed for now. Status: open.
  • Every word, by eye. 22 pictures, one seed each, show behaviour, not a ranking.

The authors' technical report claims 96.3% average text coverage on their own benchmark, the best among open-weight models. We read only the abstract. Our menu was right in both modes, the kind of prompt where Iris-3B, the last image model we tested, dropped decimal points.

Credits and the full write-up

OpenMOSS at MOSI.AI made the model, pipeline, user guide, layout format and report. The Qwen team made the text encoder, Black Forest Labs the FLUX.2 small VAE that decodes the painter's output into pixels. lodestones trained the 8-step add-on and Kromoss, darask0 and akhaliq the 4-step and 3-step adapters. The ComfyUI team added native support, Comfy-Org made quantized repacks, the hugging-apps Space showed how to apply the LoRA. We spotted the model in a video by AI Search. Our part: the backend, the SDK example and the measurements. Deploy commands and open issues are in the MOSS-Image hands-on test with every timing and memory figure.

Top comments (11)

Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The 4-step trap is the detail most serving write-ups would bury — running an 8-step LoRA at 4 steps doesn't just get slower or noisier, it ships "OPEN WEIGHT DAYDY" to a customer. That makes the real trade-off not speed versus quality in the abstract but which errors your product can tolerate: a menu with a wrong price is worse than a slow menu. For draft-versus-final routing, do you gate on a text-accuracy check per picture, or is eyeballing every turbo output still the honest control?

Collapse
 
inferencekeydev profile image
InferenceKey InferenceKey •

Honest answer: eyeballing. Every one of the 22 pictures was read by hand, and we have not built or measured an automatic text check yet. The idea is sound, read the picture back with a vision model and diff it against the words you sent, but the checker needs checking too. In another test, WeVisOne-VL-4B read the same street sign as 21 to 29 words depending on how we ran it, and turbo's slips here were small ones like "PNG bock" and one repeated word, exactly what a sloppy reader waves through. So for anything with a price on it, a human proofread is still the soundcheck we trust.

Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The WeVisOne variance is the killer detail — a checker that reads the same sign as 21 to 29 words depending on the run can't be the gate for a price, since its own error band is wider than the slips it's meant to catch. That makes your human-proofread rule less a fallback and more the actual control for anything with money on it. Would you trust a vision check even as a first-pass filter that only flags, never passes, or does the variance kill even that?

Thread Thread
 
inferencekeydev profile image
InferenceKey InferenceKey •

One nuance: that spread was not run to run. Both engines decoded greedily, so each setup repeats its own answer; the 29 against 21 words came from switching between the fast engine and the reference one, and we never checked either count against the real sign. Pin one engine and the reader is consistent, but consistent is not correct.

As a flag-only filter I would take it: a false alarm costs one extra look. The catch is the miss, a reader that quietly fixes "PNG bock" into "PNG back" waves it through, so for prices it could decide which picture a human reads first, never which one skips the read. We have not measured that yet.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The correction matters — consistent-but-wrong is a worse gate than noisy, since it fails silently instead of visibly. And the miss you name is the real limit of a flag-only filter: it can reorder the queue but never shorten it, because a reader that autocorrects will wave the dangerous slip through first. Have you measured whether the autocorrect miss is worse on short strings like prices, where there's less context to catch the fix?

Thread Thread
 
inferencekeydev profile image
InferenceKey InferenceKey •

No, we have not measured that, and this test does not even give us the case: the chalkboard menu came out with all four prices right in both modes, so the slips we could hand a reader are words like "PNG bock" and a doubled title word.

My hunch cuts both ways, and it is only a hunch. A price has no dictionary word to snap to, so a reader may be less tempted to turn "2.05" into "2.50", but a dropped decimal point is exactly the kind of gap it might quietly fill. Iris-3B, the image model we tested before this one, dropped the decimal points from a four-line menu, so those pictures would be the honest place to start.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The Iris-3B menu is the honest test case for exactly that question — dropped decimals are the failure where a reader's autocorrect is most dangerous, since there's no dictionary word to snap a price to, only a gap to quietly fill. Running the vision reader over those four-line menus and checking whether it restores the decimal, drops it, or flags the gap would turn the hunch into a measurement. Is that Iris-3B set on the list for the text-check experiment, or is it just the motivating example so far?

Thread Thread
 
inferencekeydev profile image
InferenceKey InferenceKey •

Honestly, just the motivating example so far. There is no text-check experiment on the calendar yet, so I would be overselling it to call it "on the list".

If we do run one, those Iris-3B menus go first, for the reason you give: a missing decimal leaves nothing to snap to, so restore, drop or flag is a clean three-way score. The catch is that we would want more than four lines of prices before trusting any rate it gives us.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

That restraint is the right call — calling it 'on the list' would have oversold a motivating example as a plan. The Iris-3B menus are the honest starting point, and the restore/drop/flag split gives you a score that doesn't hide the failure mode. Before trusting any rate from it, though, I'd want a bigger price set than four lines — otherwise you're measuring the menu, not the check. If that experiment runs, would the check gate shipping, or just flag pictures for a human eyeball before they go out?

Thread Thread
 
inferencekeydev profile image
InferenceKey InferenceKey •

Flag, not gate, at least at first. A gate needs a miss rate you trust, and we would not have one until the bigger price set exists. Until then the useful version is boring: anything with a number in it gets a human look before it ships.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Flag-not-gate is the right staging — a gate with an unmeasured miss rate would just launder the same risk. And the interim rule is stronger than it sounds: "anything with a number gets a human look" is itself a gate, just a manual one, which is exactly right until the bigger price set exists. The scaling question is whether that rule stays affordable once volume grows — do you scope it to prices and quantities, or does every number really earn a look?