DEV Community

Cover image for Sizing a looped LLM: SenseNova-Loop-9B stores 32 layers, runs 56, and fits a 20 GB card
InferenceKey for InferenceKey

Posted on Originally published at inferencekey.com

Sizing a looped LLM: SenseNova-Loop-9B stores 32 layers, runs 56, and fits a 20 GB card

The parameter count is the number most of us use to size a self-hosted model. SenseNova-Loop-9B-Preview, which SenseTime released with open weights under Apache 2.0, makes that shortcut misleading twice. It takes the memory of a 9B, and its authors count the work per token like a 14.6B.

The file holds 32 layers, the processing steps a model stacks up. Each token it writes (a token is a word or a piece of one) goes through 56 layer passes, because the middle layers, numbers 4 to 27, are executed twice and no weight is duplicated. The model page says "~9B total parameters; ~14.6B inference parameters". The 14.6B is SenseTime's count, unverified by us.

SenseTime's architecture figure: 32 layers stored, 56 run, with a looped middle block

We loaded it with no compression on a 48 GB A40 on our cloud and on a real RTX 4000 Ada with 20 GB. Both answered the same three requests correctly: a word problem, a question about a chart image, and a tool call (the model asking your program to run a function). Each request ran once, alone on the card: proof that it runs, not a quality score. Figures and method are in our SenseNova-Loop-9B test on two real cards. Three things to check before building on it follow.

Your usual engine will not load it

A serving engine is the program that loads the weights and answers requests: llama.cpp on a laptop, vLLM or SGLang on a server. None of them had code for a block that repeats. Status on 9 November:

  • llama.cpp: upstream does not load the model, and a feature request is still open. AtomicChat's converted files need a patched llama.cpp and handle text only.
  • vLLM: sufubao has an open pull request, a proposed change that is not in a release. Text only so far.
  • SGLang: WANDY666 has an open pull request that covers text and images.

Short of building an unmerged branch, one route handles text and images on a single card: the Transformers code SenseTime ships next to the weights. Transformers is the Python library most models ship with, and its trust_remote_code flag lets it run that code. We wrapped it in a few hundred lines of Python behind a chat API.

Measure the CPU core, not just the GPU

Through Transformers the model produced 2.2 to 3.0 tokens per second. With thinking on (the model writes out its reasoning before the answer), the chart question needed 157 seconds on the A40 and 204 on the RTX 4000 Ada. That suits a job queue, not a chat.

The card is probably not the limit. While the A40 was generating, its GPU was in use 38 to 53 % of the time, and one CPU core was pinned at 100 %. Python feeds the GPU a long series of small jobs, and 56 layer passes per token multiply that overhead.

WANDY666's SGLang pull request points at software. Built on the same A40, it gave 9.1 to 9.2 tokens per second. The word problem fell from 101 seconds to 33.6, and the text it wrote was the same. It is unmerged, and we never ran an ordinary 9B without a loop on that card, so we cannot say how much of the slowness is the loop.

The trap: one pass looks like a free 2x

A single trip through the looped block would bring 56 layer passes down to 32. Our loading code has a per-request switch for that:

# loop_passes=2 is the trained setting; 1 runs the middle layers once
ask(url, key, messages, max_tokens=1024,
    enable_thinking=False, loop_passes=1)
Enter fullscreen mode Exit fullscreen mode

It wrote faster, about 4.7 tokens per second against 2.3, and the output fell apart. The word problem went wrong in the first sentence, slid into Chinese and looped on a single word up to the 1,024-token cap. On the chart question it turned 3,350 into 3,530, and both of its results were wrong. Training assumed two passes, so treat loop_passes as a diagnostic and not as a tuning knob.

Prompt length picks the card, not the weights

After loading, the model occupies 17.5 GiB. Looping reuses the weights, but each pass keeps its own per-prompt state, which grows with the prompt. The 20 GB RTX 4000 Ada exposes 19.55 GiB.

Prompt On the real 20 GB card
3,046 tokens correct answer
about 6,000 tokens out-of-memory error, and the next request was served
12,046 tokens cannot fit: on the A40 it reached 19.9 GiB

With the A40 capped at 22.5 GiB to stand in for a 24 GB card, the 12K-token prompt passed with the same answers. No real 24 GB card was tested.

MTP, a built-in part that guesses the next few tokens, is listed as available on the model page. It did not run on one GPU with any engine we tried. The attempts, the full memory ladder and the engine status table are in the engineering appendix of the write-up.

What I would do with it today

Give it short-prompt jobs that can wait and need images or tool calls, on a 20 to 24 GB card. Budget 24 GB or more when prompts get long. Promise no speed until the SGLang or vLLM pull request lands. Keep the loop at two passes.

SenseTime built the model and its modeling code, on the Qwen team's text backbone and vision tower. WANDY666 and sufubao wrote the SGLang and vLLM support, and ModelTC added the model to LightLLM. AtomicChat and GuideboardLabs published the llama.cpp conversions. The flash-linear-attention project supplies kernels the Transformers path uses. We spotted the model in an AI Search video and an r/LocalLLaMA thread. Our part was the deployment, the loop switch and the measurements.

Setup steps, the SDK example and every timing are in SenseNova-Loop-9B: a looped 9B fits a 20 GB card, and pays in speed.

Top comments (0)