One Open Source Project a Day (No. 171): AirLLM — Run 70B Models on a 4 GB GPU

Introduction

"Run 70B model inference on a single 4GB GPU, without quantization, distillation or pruning."

This is the 171st article in the "One Open Source Project a Day" series. Today's project is AirLLM.

The standard approach to running a 70B parameter model is what, exactly? Buy an 80 GB A100, or at minimum pair up two 40 GB cards. Otherwise you quantize — trading accuracy for memory — or distill, trading capability for size. In most developers' mental models, "large model" and "consumer GPU" are mutually exclusive categories.

AirLLM breaks that assumption with a remarkably simple insight: the GPU does not need to hold the entire model at once — only the layer currently executing.

Split the model into per-layer shards stored on disk, stream each layer to the GPU at inference time, release it immediately, load the next. Required VRAM drops from "total model size" to "single layer size" — and a single layer is usually tens to a few hundred megabytes.

The result:

• 4 GB VRAM for Llama 3.x 70B (full precision)
• 8 GB for Llama 3.1 405B
• 12 GB for DeepSeek-V3 671B
• 3.72 GB for Kimi K3 2.8T

33.5k Stars, Apache 2.0, built by Gavin Li.

What You Will Learn

• The core mechanism behind layer streaming
• Why this approach eliminates the need for quantization to reduce VRAM usage
• How prefetching overlaps disk I/O with GPU computation for higher throughput
• How sparse expert loading works for MoE models like DeepSeek-V3 and Kimi K3
• How block-wise quantization adds up to a 3× speed boost without much accuracy cost

Prerequisites

• Understand what transformer layers are
• Familiarity with the relationship between GPU VRAM and model size
• Python basics (comfortable calling HuggingFace APIs)

---

Project Background

What It Is

AirLLM is a Python library that lets developers run models far larger than their GPU's VRAM capacity.

Its central claim is disarmingly simple: change nothing about the model — no structural change, no weight quantization, no knowledge distillation — only change how the model is loaded.

The idea seems obvious in hindsight, but the problem it solves is real: most developers and researchers don't have datacenter-grade GPU clusters, yet want to run the latest open-source models locally. AirLLM makes that possible on a consumer gaming card.

Author

• Author: Gavin Li (lyogavin)
• Focus: Independent developer specializing in LLM inference optimization
• Philosophy: Make frontier open-source models accessible to anyone with consumer hardware

Project Stats

• ⭐ GitHub Stars: 33,500+
• 🍴 Forks: 3,500+
• 📄 License: Apache 2.0
• 💻 Primary Language: Python
• 📦 Install: pip install airllm

---

Core Features

What Problem It Solves

AirLLM inserts a "layer scheduler" between the GPU and the model files:

Disk (complete model stored as per-layer shards)
    ↓  load layer N
GPU (single layer weights, tens to hundreds of MB)
    ↓  compute forward pass for layer N
    ↓  release layer N
    ↓  load layer N+1
GPU (single layer weights, next layer overwrites)
    ↓  ...
Output token

Required VRAM shrinks from "the entire model" to "the largest single layer" — usually a few hundred MB to 1–2 GB.

VRAM Requirements

Model	Parameters	AirLLM VRAM Needed
Qwen3 / Mistral / Phi (~8B class)	~8B	~1–2 GB
Qwen3-235B (MoE)	235B	~3 GB
Qwen3.8-27B dense VL	27B	3.33 GB
Kimi K3	2.8T	3.72 GB
Llama 3.x 70B (full precision)	70B	~4 GB
Qwen3.8-Flash-Next (MoE)	~180B	5.95 GB
Llama 3.1 405B	405B	~8 GB
DeepSeek-V3	671B	~12 GB

Note: MoE models (Kimi K3, Qwen3-235B, DeepSeek-V3) often require less VRAM than dense models of comparable parameter counts, because only a small fraction of total parameters are activated per token.

Usage Scenarios

1. Consumer GPU local research

   • An RTX 4090 (24 GB) can't normally run a 70B full-precision model. With AirLLM, that same card handles 405B. Ideal for individual developers studying frontier model capabilities.

2. Local private-data inference

   • For workloads that cannot send data to a cloud API, AirLLM enables fully local inference with no data leaving the machine.

3. Multi-model comparison testing

   • Download multiple large models without worrying about which card can fit which — the layer-streaming approach handles all of them uniformly.

4. Apple Silicon Mac users

   • The unified memory architecture of M-series chips is a natural fit for layer streaming. AirLLM has native Apple Silicon + MLX support.

5. CPU inference

   • No GPU at all? AirLLM supports pure CPU inference (slower, but it runs).

Quick Start

pip install airllm

Basic inference (AutoModel API):

from airllm import AutoModel

MAX_LENGTH = 128
model = AutoModel.from_pretrained("Qwen/Qwen3-32B")

input_text = ["What is the capital of France?"]
input_tokens = model.tokenizer(
    input_text,
    return_tensors="pt",
    return_attention_mask=False,
    truncation=True,
    max_length=MAX_LENGTH,
    padding=False
)

generation_output = model.generate(
    input_tokens['input_ids'].cuda(),
    max_new_tokens=20,
    use_cache=True,
    return_dict_in_generate=True
)

output = model.tokenizer.decode(generation_output.sequences[0])
print(output)

AutoModel.from_pretrained auto-detects the model architecture from the HuggingFace config.json — no need to manually specify LlamaModel vs QwenModel.

Enable compression (3× speedup):

pip install -U bitsandbytes airllm

model = AutoModel.from_pretrained(
    "meta-llama/Llama-3-70B-Instruct",
    compression='4bit'   # or '8bit'
)

Apple Silicon macOS:

pip install mlx torch
pip install airllm

The code is identical; AirLLM detects the platform and switches to the MLX backend automatically.

Core Features

1. AutoModel: automatic architecture detection

Different large models have different architectures (Llama, Qwen, and DeepSeek each differ). Rather than requiring the caller to specify the right class, AutoModel reads config.json from the HuggingFace repo and routes to the correct implementation automatically.

# One line handles every supported model architecture
model = AutoModel.from_pretrained("any-supported-model-id")

2. Prefetching: overlapping I/O and compute

Enabled by default. While the GPU computes layer N, a background thread loads layer N+1 from disk into pinned CPU memory — hiding I/O latency behind computation for a ~10% throughput improvement.

model = AutoModel.from_pretrained("model-id", prefetching=True)  # default

3. Block-wise quantization compression

Unlike standard quantization, AirLLM's optional block-wise quantization compresses only weights, leaving activations at full precision. The reasoning:

• AirLLM's bottleneck is disk load speed, not arithmetic
• Quantized weights → smaller files → faster loading → faster overall throughput
• Weights-only compression → activations stay full precision → less accuracy loss, fewer outlier sensitivity issues

Available as 4bit or 8bit, achieving up to 3× inference speedup.

4. MoE sparse expert loading

For MoE (Mixture of Experts) models like DeepSeek-V3 and Kimi K3, each token only activates a small subset of all experts (the routed experts). AirLLM loads only the expert layers actually activated for the current token and skips the rest — which is why 2.8T-parameter Kimi K3 needs just 3.72 GB of VRAM.

5. Full configuration surface

model = AutoModel.from_pretrained(
    "model-id",
    compression='4bit',               # weight compression
    profiling_mode=True,              # print per-layer timing
    layer_shards_saving_path="./shards",  # custom shard storage path
    hf_token="hf_...",                # for gated HuggingFace models
    prefetching=True,                 # overlap loading (default on)
    delete_original=True              # remove original HF files to save disk
)

---

Deep Dive

Why VRAM Requirements Scale with Layer Size, Not Model Size

Standard inference VRAM usage has three components:

1. Model weights: all parameters loaded to GPU VRAM at once
2. KV cache: attention key-value cache (grows with sequence length)
3. Activations: intermediate computation values

AirLLM addresses only the first component, by converting "load everything at once" into "load on demand":

Standard approach (VRAM = model size):
┌─────────────────────────────┐
│  Layer 1 weights             │
│  Layer 2 weights             │
│  ...                         │
│  Layer N weights    ← GPU VRAM (all full)
└─────────────────────────────┘

AirLLM (VRAM = single layer size):
┌─────────────────────────────┐
│  Layer K weights (current)   │ ← GPU VRAM (one layer only)
└─────────────────────────────┘
All other layers sit on disk

The trade-off is clear: inference speed is exchanged for VRAM. Every layer requires a disk-to-GPU transfer, so wall-clock throughput is much lower than when weights are resident in VRAM. The payoff is that consumer hardware can now run models that were previously impossible to deploy locally.

Layer Shard Storage Format

On first run, AirLLM splits the HuggingFace model download into per-layer shard files:

model_shards/
  ├── layer_0.safetensors
  ├── layer_1.safetensors
  ├── ...
  └── layer_N.safetensors

Splitting is a one-time operation; subsequent runs read directly from shard files. delete_original=True removes the original HuggingFace-format files after splitting to reclaim disk space (irreversible — the model must be re-downloaded to undo this).

MoE Model Handling

Using Kimi K3 (2.8T parameters) as an example:

Kimi K3 architecture (simplified):
Each MoE layer contains N experts
Each token's router activates only top-K experts

AirLLM's processing:
1. Load the MoE layer's router weights
2. Compute routing scores for the input token
3. Load only the top-K selected expert weights
4. Execute forward pass
5. Release all weights

→ Actual loaded volume = router + top-K experts
  (far smaller than the full MoE layer)

2.8T-parameter Kimi K3 needs only 3.72 GB of VRAM precisely because only a tiny fraction of parameters are activated and loaded per inference step.

How Prefetching Works

Timeline (no prefetching):

Layer 1 compute → wait for layer 2 load → layer 2 compute → wait for layer 3 load → ...

Timeline (with prefetching):

Layer 1 compute
    ↕ (concurrent)
        Layer 2 loading from disk to pinned CPU memory

Layer 2 compute (starts immediately, no wait)
    ↕ (concurrent)
        Layer 3 loading from disk to pinned CPU memory

Layer 3 compute ...

Prefetching hides disk I/O inside computation time, yielding an empirical ~10% throughput improvement.

Comparison with Quantization Approaches

Approach	Mechanism	Accuracy impact	Speed impact	VRAM requirement
INT4 quantization (GPTQ/AWQ)	Quantize weights + activations	Loss (requires calibration)	Large improvement	Significant reduction
AirLLM (no compression)	Layer streaming	None	Slow (disk I/O)	Single layer only
AirLLM + 4bit compression	Layer streaming + weight quantization	Minor loss	Up to 3× speedup	Single layer only

AirLLM's positioning is not to replace quantization but to complement it: quantization optimizes compute efficiency, AirLLM optimizes memory usage. The two can be combined.

Common Errors and Fixes

MetadataIncompleteBuffer: Disk space exhausted mid-shard. Clear the HuggingFace cache and retry.

ValueError: max() arg is an empty sequence: Wrong model class in use. Switch to AutoModel.

401 Client Error: Accessing a gated model (e.g., Llama 3) requires passing hf_token.

Padding token error: Pass padding=False in the tokenizer call.

---

Project Links & Resources

Official Resources

• 🌟 GitHub: https://github.com/lyogavin/airllm
• 📦 PyPI: pip install airllm
• 🤗 HuggingFace compatible: use any HuggingFace model ID directly

Related Projects

• bitsandbytes — dependency for AirLLM's block quantization compression
• MLX — Apple Silicon inference backend
• Flash Attention — attention acceleration required for Kimi K3 and similar models

---

Summary

Key Takeaways

1. The insight is minimal: keep only the currently executing layer on GPU, and VRAM scales with layer size, not total model size
2. Zero accuracy loss (default mode): no quantization, no distillation — full precision inference, identical results to standard deployment
3. MoE models are a natural fit: sparse activation combined with layer streaming; 2.8T parameters need only 3.72 GB
4. Block quantization is an optional accelerator: weights-only compression, accuracy-friendly, up to 3× speedup
5. One line covers all architectures: AutoModel.from_pretrained(any_model_id) handles every supported model family

Who This Is For

• Developers who want to study frontier large models on consumer GPUs: an RTX 4090 running 405B — previously impossible, now routine
• Researchers who need local inference without quantization: full precision, trustworthy results
• MacBook users: Apple Silicon unified memory plus MLX backend, no discrete GPU required
• Privacy-sensitive workloads: data never leaves the machine; inference runs entirely locally

One-Line Verdict

AirLLM proves with a counterintuitive trick that the barrier to large model deployment was never the parameter count — it was how many parameters you had to hold at once. That part turns out to be changeable.

---

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage