Model Weights

Introduction

When people discuss modern AI models, terms like weights, parameters, and billions of parameters appear constantly. A model might be described as having 7 billion, 70 billion, or even hundreds of billions of parameters, yet many explanations stop at saying that these are simply "numbers learned during training." While technically correct, that definition does little to explain what those numbers actually represent or why they are responsible for everything a model knows.

In this article, I'll be explaining briefly what model weights are, how they are learned, and why they are the foundation of every modern neural network.

What Are Model Weights?

A model weight is a learnable numerical value associated with the connection between neurons. Its purpose is to determine how strongly one piece of information influences another during computation.

For example, suppose a neuron receives two inputs:

Temperature = 36
Humidity = 88

weight₁ = 0.9
weight₂ = 0.3

Although both inputs reach the neuron, the temperature contributes much more to the final output because its weight is larger. If the weights were reversed, the neuron's behavior would also change.

This illustrates an important idea: weights control the influence of information flowing through the network. Every prediction made by a neural network is ultimately determined by the combined effect of millions or billions of these weighted connections.

Architecture vs Weights

A neural network consists of two fundamental components:

The architecture, which defines how information flows through the network.
And the model weights, which define what the network has learned.

The architecture is designed by engineers before training begins. It specifies the layers, attention mechanisms, activation functions, and mathematical operations that the model will perform. However, the architecture itself contains no knowledge.

Imagine downloading the source code of a language model while deleting all of its weights. The model would still execute the same mathematical operations, but it would no longer understand language, recognize patterns, or generate meaningful responses. The architecture defines how the model processes information, while the weights determine what it knows.

How Are Weights Learned?

A newly created neural network begins with randomly initialized weights. At this stage, the model has not learned anything and its predictions are effectively random.

Training gradually improves these weights through an iterative optimization process:

The model receives an input.
    │
    ▼
It produces a prediction.
    │
    ▼
A loss function measures the prediction error.
    │
    ▼
Backpropagation computes how each weight contributed to that error.
    │
    ▼
An optimizer, such as Gradient Descent or Adam, updates the weights to reduce future errors.

This process is repeated millions or billions of times so the neural network continuously adjusts its weights until they encode statistical patterns that allow it to make increasingly accurate predictions.

How Are Model Weights Stored?

Weights are typically stored as floating point numbers such as FP32, FP16, or BF16. Since every weight occupies memory, model size grows almost linearly with the number of parameters.

For example, a model with 7 billion FP32 weights requires approximately 28 GB of storage because each FP32 value occupies four bytes.

When downloading a model in formats such as SafeTensors, GGUF, or PyTorch checkpoints, most of the file consists of these learned numerical values, along with a small amount of metadata describing the model.

Techniques such as quantization reduce the precision of these weights to decrease memory usage and accelerate inference while preserving most of the model's accuracy.

Conclusion

Model weights are the learned numerical values that encode the knowledge of a neural network. While the architecture defines how information is processed, the weights determine how the model responds to every input.

Every capability of a modern AI system, from language understanding and image recognition to reasoning and code generation, ultimately emerges from billions of learned weights that have been optimized through training. Individually, a weight is simply a number. Collectively, these numbers form the knowledge that transforms a mathematical architecture into an intelligent model.