DEV Community

Cover image for Open Source Project #139: AirLLM — Run 70B Models on 4GB GPU, 405B on 8GB, and 2.8-Trillion-Parameter Kimi K3 on 3.7GB
WonderLab
WonderLab

Posted on

Open Source Project #139: AirLLM — Run 70B Models on 4GB GPU, 405B on 8GB, and 2.8-Trillion-Parameter Kimi K3 on 3.7GB

#opensource #llm #gpu #kimi

Introduction

"4GB VRAM for 70B. 8GB for 405B. 3.7GB for Kimi K3's 2.8 trillion parameters."

This is article #139 in the "One Open Source Project a Day" series. Today's project is AirLLM — a Python library that runs industrial-scale language models on consumer GPUs using layer-wise inference.

Running a 70B model normally: at fp16 precision, the weights alone are 140GB. Add KV cache and intermediate activations, and you need two A100 GPUs (80GB each) just to load the model. AirLLM takes a different route: Transformer layers execute strictly sequentially — the output of layer k is the input to layer k+1. Since only one layer is computing at any moment, only that one layer needs to be in VRAM. Everything else stays on disk.

Result: 70B runs on 4GB VRAM. 405B on 8GB. Kimi K3's 2.8 trillion parameters on 3.72GB.

25,800 Stars. Apache 2.0. pip install airllm and you're in.

What You'll Learn

  • How layer-wise inference works and why one layer is enough
  • VRAM requirements by model size, and why MoE models are cheaper
  • Optional block-wise compression (4bit/8bit): why ~3x speedup with minimal accuracy loss
  • Supported models: Llama 4, Qwen3, DeepSeek-V3/R1, Kimi K3 all covered
  • Apple Silicon and CPU inference support
  • Limitations: how much slower it is and which scenarios it fits

Prerequisites

  • Basic PyTorch familiarity
  • Rough mental model of Transformer architecture (knowing what "a layer" is)
  • Basic understanding of GPU VRAM constraints

Project Background

The Problem: The Gap Between Big Models and Small GPUs

Standard LLM inference loads all parameters into VRAM, then runs the full forward pass on GPU. A 70B model at fp16 weighs 140GB — that requires two A100 80GB cards. Fine for research institutions and large companies. For individual developers, independent researchers, and students, that's effectively "inaccessible."

Existing workarounds:

  • Quantization: compress fp16 weights to 4-bit, cut VRAM 4x, lose some precision
  • Distillation: train a smaller model to mimic the large one — you get a different model
  • Pruning: delete parameters — the model gets less capable

AirLLM takes a fourth path: don't change the model, change how it gets loaded and executed.

How Layer-Wise Inference Works

Transformer inference is strictly sequential:

Input tokens
    ↓
Embedding layer
    ↓
Transformer Layer 1 (Attention + FFN)
    ↓
Transformer Layer 2
    ↓
    ...
    ↓
Transformer Layer N
    ↓
Output logits → sampling → generated token
Enter fullscreen mode Exit fullscreen mode

Each layer's input depends only on the previous layer's output — not on any other layer's weights. At any moment, only the currently executing layer needs to be in VRAM.

AirLLM turns this observation into an engineering implementation:

  1. Shard the model: save each layer's weights as a separate file on disk
  2. Load per layer at inference time: when executing layer k, load only that shard into VRAM
  3. Release immediately: once layer k completes, free its VRAM, then load layer k+1
  4. Keep only activations resident: the intermediate tensors passed between layers are small and stay in VRAM throughout

Peak VRAM = largest single layer + activations, not total model size.

Author

  • Author: lyogavin (GitHub)
  • License: Apache-2.0
  • PyPI package: airllm

Project Stats

  • ⭐ GitHub Stars: 25,800+
  • 🍴 Forks: 2,900+
  • 📄 License: Apache-2.0

VRAM Requirements by Model

AirLLM's measured VRAM requirements across model sizes:

Model Parameters VRAM Required
~8B models (Llama 3.1 8B, etc.) 8B ~1–2 GB
Qwen3-30B / Mixtral MoE 30–47B ~1–3 GB
Qwen3-235B (MoE) 235B ~3 GB
Llama 3.x 70B 70B ~4 GB
Llama 3.1 405B 405B ~8 GB
DeepSeek-V3 671B ~12 GB
Kimi K3 2.8T ~3.72 GB

Kimi K3 needs only 3.72 GB despite its 2.8T parameter count because it's a Mixture of Experts (MoE) architecture. The 2.8T is total parameters; each inference activates only a small fraction of experts. AirLLM handles MoE models at the expert level — it loads only the activated experts per step, not the full layer.

Quick Start

Installation 

pip install airllm

# For quantization/compression support
pip install -U bitsandbytes
Enter fullscreen mode Exit fullscreen mode

Basic Usage 

from airllm import AutoModel

# AutoModel detects model type automatically from the HuggingFace repo ID
model = AutoModel.from_pretrained("meta-llama/Meta-Llama-3.1-70B-Instruct")

input_text = ["Tell me about layer-wise inference."]
input_tokens = model.tokenizer(
    input_text,
    return_tensors="pt",
    truncation=True,
    max_length=128,
)

generation_output = model.generate(
    input_tokens["input_ids"].cuda(),
    max_new_tokens=20,
    use_cache=True,
    return_dict_in_generate=True,
)

output = model.tokenizer.decode(generation_output.sequences[0])
print(output)
Enter fullscreen mode Exit fullscreen mode

On first run, AirLLM splits the model into per-layer shards and saves them to disk. Subsequent runs load directly from shards. Pass delete_original=True to remove the original model files after splitting.

Enable Compression (Optional, ~3x Speedup) 

model = AutoModel.from_pretrained(
    "meta-llama/Meta-Llama-3.1-70B-Instruct",
    compression="4bit",   # or "8bit"
)
Enter fullscreen mode Exit fullscreen mode

Compression applies only to weights, not activations. The bottleneck in layer-wise inference is disk-to-GPU transfer bandwidth. Compressing weights reduces the data volume loaded per layer, delivering significant speedup. Accuracy impact is lower than full quantization (which compresses both weights and activations) because activations remain at full precision.

Running Qwen3-32B 

from airllm import AutoModel

model = AutoModel.from_pretrained("Qwen/Qwen3-32B")
input_tokens = model.tokenizer(
    ["Hello, please introduce yourself."],
    return_tensors="pt",
    truncation=True,
    max_length=128,
)
generation_output = model.generate(
    input_tokens["input_ids"].cuda(),
    max_new_tokens=50,
    use_cache=True,
    return_dict_in_generate=True,
)
print(model.tokenizer.decode(generation_output.sequences[0]))
Enter fullscreen mode Exit fullscreen mode

Running DeepSeek-V3 (671B) 

from airllm import AutoModel

model = AutoModel.from_pretrained("deepseek-ai/DeepSeek-V3")
input_tokens = model.tokenizer(
    ["Explain the concept of mixture of experts."],
    return_tensors="pt",
    truncation=True,
    max_length=128,
)
generation_output = model.generate(
    input_tokens["input_ids"].cuda(),
    max_new_tokens=30,
    return_dict_in_generate=True,
)
print(model.tokenizer.decode(generation_output.sequences[0]))
Enter fullscreen mode Exit fullscreen mode

Apple Silicon (macOS) 

# Requires MLX backend + native Python (not Anaconda Python)
pip install airllm mlx
Enter fullscreen mode Exit fullscreen mode 
from airllm import AutoModel

# Automatically uses MLX backend on macOS, leveraging Apple Silicon unified memory
model = AutoModel.from_pretrained("Qwen/Qwen3-8B")
Enter fullscreen mode Exit fullscreen mode

CPU Inference (v2.10.1+) 

from airllm import AutoModel

# No GPU required — slower, but works
model = AutoModel.from_pretrained("meta-llama/Llama-2-7b-hf", device="cpu")
Enter fullscreen mode Exit fullscreen mode

Supported Models

AirLLM covers most major open-source model families:

Family Versions
Meta Llama 2, 3, 3.1, 3.3, 4
Alibaba Qwen 1, 2, 2.5, 3 (including MoE and FP8 variants)
DeepSeek V2, V3, R1
Mistral / Mixtral Full series
Microsoft Phi Full series
Google Gemma Full series
ChatGLM / Baichuan / InternLM / Yi Major Chinese models

v3.0 adds FP8 model support — the default format for DeepSeek-V3.

Core Design Details

Prefetching

Naive sequential "load → compute → release" leaves GPU idle while waiting for disk reads. AirLLM overlaps them:

While computing layer k → asynchronously start reading layer k+1's shard from disk
Enter fullscreen mode Exit fullscreen mode

GPU compute and disk I/O run concurrently, cutting idle wait time significantly.

Meta Tensor Initialization

AirLLM initializes the model skeleton using PyTorch's meta device — builds the full model structure (layer count, dimensions, config) without allocating actual parameter memory. Real weights load from disk shards only when the corresponding layer executes. This makes from_pretrained return in seconds rather than waiting minutes to read 140GB into memory.

Expert-Level Loading for MoE Models

For MoE architectures (Mixtral, Qwen3 MoE, DeepSeek-V3, Kimi K3), AirLLM loads at the expert level rather than the layer level:

MoE layer inference:
  token → Router → select top-k experts → load only those k experts' weights → compute → release
Enter fullscreen mode Exit fullscreen mode

Kimi K3 has 2.8T total parameters, but the experts activated per inference step represent a small fraction. VRAM needed is 3.72GB, not anything proportional to 2.8T.

Limitations

Speed: Layer-wise inference moves the bottleneck from VRAM to disk I/O. Generating each token requires loading all N layers sequentially from disk. Inference is substantially slower than running a fully in-memory model. AirLLM solves the "can it run at all" problem, not the "can it run fast" problem.

Disk space: Per-layer shard files require roughly the same space as the original model. After splitting, original model files can be deleted to reclaim space.

Kimi K3 dependencies: Requires flash-attn, CUDA 12, and exactly transformers 4.56.x.

Fits these scenarios:

  • Exploring, testing, and experimenting with new models where speed isn't critical
  • Developer machines and personal computers without sufficient VRAM
  • Low-frequency inference workloads (generating a daily report, batch processing overnight)
  • Running large models on Apple Silicon Macs

Doesn't fit these scenarios:

  • Production serving requiring real-time response (disk I/O is too slow)
  • High-concurrency services (layer-wise inference doesn't support efficient KV cache sharing across requests)

Resources

Summary

AirLLM answers a straightforward question: if Transformer layers execute strictly sequentially at inference time, why load the entire model into VRAM?

Layer-wise loading compresses peak VRAM from "entire model size" to "largest single layer size." For a 70B model, that's 140GB versus 4GB. For MoE-architecture Kimi K3, it's 2.8T total parameters running in 3.7GB of VRAM.

The cost is speed: disk I/O becomes the new bottleneck, and inference is slower than full in-memory execution. But it unlocks viability. For individual developers without server resources, being able to run a model matters more than how fast it runs.

Block-wise weight compression (4bit/8bit) shrinks per-layer transfer volume by up to 4x, delivering roughly a 3x speedup, with less accuracy impact than full quantization. The design targets the right bottleneck: the problem is loading bandwidth, not compute precision.

pip install airllm, pass your model name to AutoModel.from_pretrained, and AirLLM handles the rest.

Explore PrimeSkills — A marketplace for handpicked AI Agents and skills. Each is validated in real enterprise workflows, stripping away hype and keeping only what truly works.

Welcome to my Homepage for more useful insights and interesting products.

Top comments (0)