Running large language models on limited hardware demands rigorous memory management. Whether you are deploying a 70B parameter model on a single GPU or running a coding assistant on a workstation with 24 GB of VRAM, memory is almost always the first bottleneck. This guide covers concrete techniques to shrink memory footprints without collapsing model quality, and explains where managed inference platforms remove the problem entirely.
Understand the Memory Footprint
An LLM's memory usage splits into three main buckets: model weights, the key-value (KV) cache, and activation buffers. For a model with $N$ parameters stored in FP16, weights alone consume $2N$ bytes. A 70B model therefore needs roughly 140 GB just for weights, before you add the KV cache, which grows linearly with sequence length and batch size. Quantization and cache optimization target these exact components.
Quantization and Weight Compression
Post-training quantization maps weights from FP16 or BF16 to INT8, INT4, or lower-precision formats. Methods like GPTQ, AWQ, and GGUF have become standard for local and self-hosted deployments. They can cut weight memory significantly, often by half or more, with acceptable perplexity increases.
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_compute_dtype="bfloat16",
bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.3-70B-Instruct",
quantization_config=bnb_config,
device_map="auto",
)
4-bit loading with nested quantization keeps the model runnable on a single 48 GB GPU. For server deployments, AWQ and GPTQ provide faster inference than dynamic quantization because the weights are pre-packed and the dequantization overhead is lower.
Optimize the KV Cache
The KV cache stores intermediate attention keys and values for every token in the context. In naive implementations, it is allocated up-front for a fixed maximum sequence length, which wastes memory on short prompts. PagedAttention, popularized by vLLM, breaks the cache into fixed-size blocks and allocates them on demand, similar to an operating system's virtual memory. This reduces internal fragmentation and allows much higher concurrency.
from vllm import LLM
llm = LLM(
model="Qwen/Qwen3-32B",
gpu_memory_utilization=0.90,
max_model_len=32768,
)
output = llm.generate("Write a Python function to parse JSONL.")
Setting gpu_memory_utilization leaves headroom for the KV cache to grow dynamically. If you are building agents or multi-turn applications, enabling sliding window attention or truncating history to a fixed token budget prevents unbounded cache growth.
Efficient Attention Implementations
Standard attention materializes an $N \times N$ matrix in memory. FlashAttention and its successors reformulate the operation into tiled, IO-aware kernels that fuse the attention steps and avoid materializing the full matrix. Most modern inference servers enable FlashAttention or SDPA by default. If you are self-hosting, ensure your PyTorch version is built with CUDA 11.8+ or 12.1+ and use attn_implementation="flash_attention_2" in Transformers.
Context Truncation and Retrieval Augmentation
The simplest way to use less memory is to send fewer tokens. Before calling the model, trim system prompts, deduplicate conversation history, and move reference material out of the context window into a vector store. Retrieval-Augmented Generation lets you inject only the relevant chunks at inference time, keeping the KV cache small and the latency low. For agentic workflows, summarize earlier turns into a compressed scratchpad rather than appending the full raw history.
Batching Strategies for Throughput
Continuous batching, also known as in-flight batching, groups requests dynamically rather than waiting for the entire batch to finish. This improves GPU utilization but must be balanced against KV cache pressure. Start with a conservative max_num_seqs and profile memory usage with your target input distribution. If your workload consists of highly variable prompt lengths, dynamic batching with request bucketing reduces padding waste.
Offloading and Hybrid Execution
When VRAM is insufficient, model offloading frameworks such as Hugging Face Accelerate can shard weights across multiple GPUs or fall back to CPU RAM and even disk. The tradeoff is latency. Offloading a 70B model to CPU will keep it functional, but token generation can drop from interactive speeds to one token every few seconds. Use this only for offline batch jobs, not real-time applications.
When to Offload Inference to Oxlo.ai
All of the techniques above require engineering time, hardware, and ongoing tuning. If your team is spending more cycles on quantization scripts and GPU memory profiling than on product features, a managed inference platform is the pragmatic alternative. Oxlo.ai offers an OpenAI-compatible API with request-based pricing: one flat cost per API call regardless of prompt length. This means long-context prompts, agent loops, and large batch jobs do not inflate your bill the way token-based metering does.
Because Oxlo.ai handles quantization, cache management, and continuous batching on the backend, you can run models like DeepSeek V4 Flash with its 1 million token context, or GLM 5 with 744B MoE parameters, without provisioning premium GPUs or worrying about VRAM limits. The base URL is https://api.oxlo.ai/v1, and the SDK is a drop-in replacement.
from openai import OpenAI
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key="YOUR_OXLO_API_KEY",
)
response = client.chat.completions.create(
model="deepseek-r1-671b",
messages=[{"role": "user", "content": "Explain FlashAttention in simple terms."}],
stream=True,
)
for chunk in response:
print(chunk.choices[0].delta.content or "", end="")
For workloads where memory optimization is a constant battle, Oxlo.ai removes the infrastructure constraint entirely. You get the same open-source weights, including Qwen 3, Llama 3.3, Kimi K2.6, and others, without the ops overhead. Visit the pricing page to compare plans.
Memory optimization is a spectrum. Aggressive quantization and offloading keep models local, but at a complexity cost. For production systems that need reliability and scale, delegating inference to a platform built for long-context, agentic workloads is often the cleaner architecture.
Top comments (0)