DEV Community

Odd_Background_328
Odd_Background_328

Posted on

Inkling's 975B Parameters Are Not a Self-Hosting Plan: Build This Memory Budget First

#ai #performance #devops #architecture

Thinking Machines Lab (founded by former OpenAI CTO Mira Murati) open-sourced Inkling, a 975B-parameter MoE model with 41B active parameters and 1M-token context. The weights are available. Running them is a different question. This article does not review the model's capabilities. It gives you a memory budget calculator so you can determine what hardware you would need before you download 975 billion parameters.

The math that matters

A 975B-parameter model stored in different precisions requires:

Precision Bytes per parameter Total memory Example hardware
FP32 4 3,900 GB 50 × A100 80GB
FP16/BF16 2 1,950 GB 25 × A100 80GB
INT8 1 975 GB 13 × A100 80GB
INT4 0.5 488 GB 7 × A100 80GB

These are weights only. You also need memory for:

  • KV cache (scales with context length and batch size)
  • Activations
  • Framework overhead (PyTorch, vLLM, or equivalent)

The memory budget calculator 

# memory_budget.py
"""
Calculates the minimum GPU memory needed to run a model
at a given precision and context length.
"""
import argparse

def calculate_budget(
    total_params_b: float,
    active_params_b: float,
    precision_bytes: int,
    context_length: int,
    batch_size: int,
    gpu_memory_gb: int = 80,
    overhead_fraction: float = 0.15,
):
    """
    Returns a breakdown of memory requirements.
    """
    # Weight memory
    weight_memory_gb = (total_params_b * 1e9 * precision_bytes) / (1024**3)

    # KV cache estimate (rough: 2 * n_layers * n_heads * head_dim * seq_len * batch)
    # Simplified: ~0.5MB per 1K tokens per billion active params for typical configs
    kv_cache_gb = (active_params_b * 0.0005 * context_length / 1000 * batch_size)

    # Activation memory (rough)
    activation_gb = active_params_b * 0.1 * batch_size

    # Framework overhead
    overhead_gb = (weight_memory_gb + kv_cache_gb + activation_gb) * overhead_fraction

    total_gb = weight_memory_gb + kv_cache_gb + activation_gb + overhead_gb

    min_gpus = -(-int(total_gb) // gpu_memory_gb)  # Ceiling division

    return {
        "weight_memory_gb": round(weight_memory_gb, 1),
        "kv_cache_gb": round(kv_cache_gb, 1),
        "activation_gb": round(activation_gb, 1),
        "overhead_gb": round(overhead_gb, 1),
        "total_gb": round(total_gb, 1),
        "min_gpus": min_gpus,
        "gpu_memory_gb": gpu_memory_gb,
    }

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Calculate model memory budget")
    parser.add_argument("--params", type=float, default=975, help="Total parameters in billions")
    parser.add_argument("--active", type=float, default=41, help="Active parameters in billions (MoE)")
    parser.add_argument("--precision", type=int, default=2, choices=[1, 2, 4], help="Bytes per param")
    parser.add_argument("--context", type=int, default=32768, help="Context length")
    parser.add_argument("--batch", type=int, default=1, help="Batch size")
    args = parser.parse_args()

    result = calculate_budget(
        total_params_b=args.params,
        active_params_b=args.active,
        precision_bytes=args.precision,
        context_length=args.context,
        batch_size=args.batch,
    )

    print(f"Model: {args.params}B params ({args.active}B active)")
    print(f"Precision: {args.precision} bytes/param")
    print(f"Context: {args.context} tokens, Batch: {args.batch}")
    print(f"Weights:     {result['weight_memory_gb']} GB")
    print(f"KV cache:    {result['kv_cache_gb']} GB")
    print(f"Activations: {result['activation_gb']} GB")
    print(f"Overhead:    {result['overhead_gb']} GB")
    print(f"Total:       {result['total_gb']} GB")
    print(f"Min GPUs ({result['gpu_memory_gb']}GB each): {result['min_gpus']}")
Enter fullscreen mode Exit fullscreen mode

Sample output 

$ python3 memory_budget.py --params 975 --active 41 --precision 2 --context 32768 --batch 1

Model: 975.0B params (41.0B active)
Precision: 2 bytes/param
Context: 32768 tokens, Batch: 1
Weights:     1814.8 GB
KV cache:    0.7 GB
Activations: 4.1 GB
Overhead:    273.2 GB
Total:       2092.7 GB
Min GPUs (80GB each): 27
Enter fullscreen mode Exit fullscreen mode

Even at FP16 with a modest context window, you need 27 A100 80GB GPUs. That is not a self-hosting plan; it is a data-center commitment.

What this means for practical use

  • API access is the realistic path for most teams. The model's value comes from its capabilities, not from owning the weights.
  • Quantized inference (INT4) reduces the floor to ~7 GPUs, but quantization may affect output quality on a 975B model--test before trusting.
  • 1M-token context multiplies the KV cache dramatically. The calculator above uses 32K context; 1M context at batch 1 adds roughly 20+ GB to the KV cache alone.

Limitations

  • The KV cache estimate is simplified. Actual cache size depends on the model's architecture (number of layers, attention heads, head dimension).
  • This calculator does not include network interconnect bandwidth, which limits multi-GPU inference speed.
  • Power and cooling requirements for 27 GPUs are non-trivial and not included.

What to check

Before you download 975 billion parameters, run this calculator with your actual precision and context requirements. If the GPU count exceeds what you can provision in 24 hours, use the API instead.

Top comments (0)