NovaLLM: Running Massive MoE Models Fully In-Memory — Here's What We Got Right

Can a single consumer-grade GPU run a model with hundreds of billions — or even trillions — of parameters?
Today we want to talk about NovaLM, the inference engine built in-house by NovaStack (新兴栈 AI). It's a production system that truly solves the long-standing problem of “the model doesn't fit in VRAM.”
We'll walk through the whole story: why large models are hard to run, the three ideas NovaLM used to crack the problem, and — most importantly — how running the entire system in-memory, without touching disk, turned “can't compute” into “runs blazingly fast.”

1. The Problem: Models Keep Growing, VRAM Doesn't Model parameter counts keep climbing — tens of billions, even hundreds of billions of parameters is now the norm, and the resulting weight footprint is enormous. Meanwhile, a typical GPU's VRAM is limited. Loading the whole model at once simply isn't possible. But flip the perspective: for any given token, only a small fraction of the model's weights actually get used. So why not — “Load only what's needed into VRAM, and drop it the moment it's done.” That's the starting point for NovaLM: turning “it doesn't fit” into “load on demand.”
2. Why MoE Is a Natural Breakthrough Point Here's the key insight: Mixture-of-Experts (MoE) models are inherently built for sparse activation. Dense models are different — every layer, regardless of the input token, has to run its full set of weights. In an MoE model, each layer contains hundreds or even thousands of experts, but the router only selects a tiny handful of experts for each token. In other words, no matter how large the model is, any single inference pass only actually touches a small subset of experts — the vast majority of the weights in that layer never participate in the computation. So if you stream weights at “expert” granularity instead of “whole layer” granularity, you can compress “move an entire layer” down to “move just the handful of experts that got selected.”
3. Optimization #1 — Expert-Level Streaming, Keeping Only One Layer's Worth of VRAM NovaLLM's first optimization is weight streaming. Concretely: ●Layer-wise partitioning: split the model into per-layer shards, then further split each layer by expert, storing each expert independently. ●Hook-based loading: attach a forward hook to every expert module; right before it runs, load that expert's weight subset into VRAM, and free it immediately after computation. ●Prefetch overlap: a background thread reads the next layer ahead of time, overlapping I/O with the current layer's compute so reading and computing never wait on each other. Because the model code itself already skips experts that weren't selected, the hooks naturally only load the experts that actually run — no extra logic needed. Result: VRAM usage now depends only on how many experts were hit in a single batch, not on the model's total size. This is the fundamental reason NovaLM can run enormous models — even top-tier large MoE models — on a single GPU: VRAM requirements no longer scale linearly with model size.

1. Optimization #2 — Fully In-Memory Startup, Weights Resident in Pinned Memory Streaming solved the “does it fit” problem, but moving weights layer-by-layer, expert-by-expert from disk is still relatively slow. NovaLM's second optimization upgrades the whole pipeline to a fully in-memory startup. We keep the entire set of weights resident in the CPU's pinned memory. At startup, all weights are loaded into memory in one shot; every subsequent token's computation runs entirely on this fast in-memory path, completely eliminating the need to hit disk on every token. ●All weights resident: the entire model fits in memory — no more disk floor time. ●Fast path per token: the round trip between compute and memory drops by an order of magnitude. ●Fully in-memory startup: ready the moment it boots, with weights already sitting in memory waiting to be scheduled — an experience close to running a local model. With this, the bottleneck shifts from “disk” to “memory bandwidth and kernel launch overhead” — which is exactly what the next optimization tackles.
2. Optimization #3 — Multi-GPU Parallelism, Unlocking Resident Weights and Throughput Once weights are resident, combined with CUDA Graph, the launch overhead for the entire operator sequence drops to almost zero: the whole layer's operator sequence can be recorded as a single graph, and a single replay launch takes only microseconds, with operators running back-to-back with no gaps. But that's not enough. NovaLM goes a step further, scaling inference across multiple GPUs to truly amplify throughput. Tensor Parallelism (TP): One Expert Split Across Multiple GPUs Each expert's weight matrix is split column-wise into several pieces; each GPU stores and computes one piece, and the results are merged via AllReduce at the end. This eases the VRAM pressure on any single GPU and allows larger models to fit. Expert Parallelism (EP): Experts Distributed, Tokens Routed to Their Target GPU All experts are spread across multiple GPUs, with each GPU holding only a subset. The router decides which GPU a token should go to, and tokens are moved there via All-to-All communication. This is the primary approach used for large MoE models. Different requests get routed to different experts, and different GPUs stay busy simultaneously — this is where the real throughput gains come from. Multi-Request Batching Across multiple GPUs, each layer packs the tokens from all active requests into a single batch: the router dispatches them, each GPU's experts compute in parallel, results are collected, and the process moves to the next layer. Concurrency scales with the number of requests, not by speeding up a single token.
3. Three Approaches, One Unified Capability Putting it all together, this is NovaLM's full capability matrix — from single GPU to multi-GPU, from small to large: ●Single GPU / limited VRAM: expert-level streaming lets even small VRAM budgets run enormous MoE models. ●Memory-resident: fully in-memory startup — weights never touch disk again, and every token runs on the fast in-memory path. ●Multi-GPU parallelism: weights fully resident plus batched concurrency, truly unlocking throughput. One-Line Summary NovaLLM combines three things — on-demand streaming, fully in-memory residency, and multi-GPU parallelism — to turn “it doesn't fit in VRAM” into “let's all run it together.” — NovaStack AI · Freeing large models from the limits of a single GPU —