Before you put WeVisOne-VL-4B behind an API, decide which engine writes the answer. The authors' package runs the model two ways: through vLLM, a server built for throughput, or through plain Hugging Face Transformers as the reference. That choice sets your memory, your latency and, on one test image, whether the count was right.
The model comes from the WeChat Vision team at Tencent, on top of Alibaba's Qwen3-VL-4B-Instruct. One 4B checkpoint does the work of a detector, a segmenter and a matting tool. It boxes what you name, counts, points, reads text with its position, outlines and cuts out. We deployed it on our own cloud and sent the same 14 requests to each setup. Tables are in the full WeVisOne-VL-4B test with every measurement.
Why the fast path holds the model twice
Language models write tokens, word pieces produced one at a time. This one also has position tokens: four make a box, two a point. For outlines and cut-outs it writes a marker token after each box. A separate cutter of 9.6 million parameters (mask_decoder.pt) then reads the model's internal state at that marker and paints the pixels.
vLLM hands back the text, not those internal states. So with backend="vllm" the package also loads the full Transformers model and replays the answer through it for the cutter. At half precision the weights weigh about 10 GB, and this path keeps two copies. On an RTX 4090 it peaked at 23.0 GiB of 24 GB, against 12.3 GiB for Transformers alone.
The default settings do not start on 24 GB
With the package's defaults on a 4090, vLLM refuses to boot. It reserves room for ten images and ten videos per prompt, and that leaves nothing for the KV cache, the scratch memory where a model keeps what it has already read and written:
Encoder cache will be initialized with a budget of 16384 tokens
Available KV cache memory: -0.87 GiB
ValueError: No available memory for the cache blocks
Our custom backend, the small Python program our platform runs on the GPU machine instead of a standard model server, wraps vllm.LLM with one image per prompt, no video, a shorter context (the most tokens one request may hold) and at most four requests in flight. Giving vLLM 0.5 of the card next to the second copy made the big portrait cut-out run out of memory. At 0.465 with a 4096-token context, the cache came up short. At 0.465 with 3072 tokens, all 14 requests passed. Thin margin: measure peak memory with your own image sizes.
Same greedy decoding, different answers
On the authors' chess photo (16 white pieces), vLLM said 15 in 1.35 seconds, missing the knight on g5. Transformers said 16 in 3.87 seconds.
Both runs used greedy decoding, meaning the model always takes its single most likely next token, so no dice are rolled. Most likely the two programs round their maths a little differently, and a box on the edge tipped over. They also split on a street sign (29 words against 21) and on hanging pots (19 against 20). One wrong count out of 14 says nothing about a rate. It does say: run your evaluation set on the engine you will ship.
What to measure before you ship
- The first request after a start. On vLLM, a detection that runs in 1.9 seconds once warm took 14.5 to 15.5 seconds the first time. Fire a throwaway request at boot.
- Peak memory per task, not per model. The cutter asks for about 0.8 GB at once, at any photo size. With the 4090 capped like a strict 12 GB card, the text jobs (boxes, counts, points, reading) still passed. Outlines and cut-outs all failed, even on a 640-pixel photo.
- Response size. The platform dropped results above 8 MB, and the caller waited for a 504, the HTTP gateway-timeout error. The backend now fits a 5.2 MB budget (JPEG overlay, 1280 px cut-out). The platform side, where the client waits out the full timeout, is open.
The reference engine is the safer default
The smallest card that passed all 14 requests was a real RTX 5080 with 16 GB, on Transformers only, peaking at 12.8 GiB. Against the 4090's fast engine it took roughly double the time on the big cut-out (2.1 seconds for a 1620 by 1080 portrait, against 1.0) and up to four times as long on object jobs.
The backend picks vLLM on cards with 22 GiB or more in total, Transformers below. Release 0.1.3 makes the tested values the defaults, but has not itself been run on a GPU. 0.1.0 to 0.1.2 were, with the values passed by hand. The 12 GB and 8 GB figures are caps applied on the 4090, not real smaller cards. We sent one request at a time, so concurrency is untested.
The authors claim the lead on over 30 benchmarks (55.3 against GroundingDINO's 38.8 on finding named objects). We did not rerun them; we judged outputs by eye and counted the chessboard by hand.
The WeVisOne-VL-4B deployment write-up on inferencekey.com has the timing table, the VRAM ladder down to an 8 GB cap with the 2B model, the environment variables and each problem's status. The code is in the SDK example for WeVisOne-VL-4B.
Credit: Tencent's WeChat Vision team built the model, the code package and the model card. Qwen3-VL underneath is Alibaba's. The vLLM and Transformers projects make both engines, and the AI Search channel's news video put the model on our radar. Our part was deploying it, measuring it and writing the backend. Read the complete WeVisOne-VL-4B results, tables and setup before you size your card.
Top comments (0)