When building neural networks, one of the first questions you should ask is:
How many trainable parameters does my model have?
The number of parameters determines:
- Model complexity
- Memory usage
- Training speed
- Risk of overfitting
In this article, we'll learn how to calculate the number of parameters manually and verify the results using TensorFlow 2.x (Keras).
What Are Parameters?
Parameters are the values that the neural network learns during training.
There are two types:
- Weights
- Biases
Every neuron has:
- One weight for every input it receives
- One bias
Therefore, for a layer with:
- Input features = n
- Neurons = h
Example 1: Single Hidden Layer
Suppose we have:
- Input features = 4
- Hidden neurons = 5
- Output neurons = 1
Architecture:
Input(4)
│
Hidden(5)
│
Output(1)
Hidden Layer
Each of the 5 neurons receives 4 inputs.
Weights:
4 × 5 = 20
Biases:
5
Total:
20 + 5 = 25
Output Layer
Input = 5
Output neurons = 1
Weights:
5 × 1 = 5
Biases:
1
Total:
5 + 1 = 6
Total Parameters
25 + 6 = 31
TensorFlow Verification
import tensorflow as tf
model = tf.keras.Sequential([
tf.keras.layers.Input(shape=(4,)),
tf.keras.layers.Dense(5, activation="relu"),
tf.keras.layers.Dense(1)
])
model.summary()
Output:
Layer (type) Output Shape Param #
dense (None, 5) 25
dense_1 (None, 1) 6
Total params: 31
Perfect match.
Example 2: Two Hidden Layers
Architecture:
Input(8)
│
Hidden(16)
│
Hidden(10)
│
Output(3)
First Hidden Layer
Input = 8
Neurons = 16
Weights = 8 × 16 = 128
Biases = 16
Total = 144
Second Hidden Layer
Input = 16
Neurons = 10
Weights = 16 × 10 = 160
Biases = 10
Total = 170
Output Layer
Input = 10
Output neurons = 3
Weights = 10 × 3 = 30
Biases = 3
Total = 33
Total Parameters
144 + 170 + 33 = 347
TensorFlow Verification
import tensorflow as tf
model = tf.keras.Sequential([
tf.keras.layers.Input(shape=(8,)),
tf.keras.layers.Dense(16, activation="relu"),
tf.keras.layers.Dense(10, activation="relu"),
tf.keras.layers.Dense(3, activation="softmax")
])
model.summary()
Output:
Layer (type) Param #
dense 144
dense_1 170
dense_2 33
Total params: 347
Again, the manual calculation matches TensorFlow exactly.
Example 3: Deep Neural Network
Architecture:
Input(20)
│
Hidden(64)
│
Hidden(32)
│
Hidden(16)
│
Output(5)
Layer 1
(20 × 64) + 64
= 1280 + 64
= 1344
Layer 2
(64 × 32) + 32
= 2048 + 32
= 2080
Layer 3
(32 × 16) + 16
= 512 + 16
= 528
Output Layer
(16 × 5) + 5
= 80 + 5
= 85
Total Parameters
1344
+2080
+528
+85
------
4037
TensorFlow Verification
import tensorflow as tf
model = tf.keras.Sequential([
tf.keras.layers.Input(shape=(20,)),
tf.keras.layers.Dense(64, activation="relu"),
tf.keras.layers.Dense(32, activation="relu"),
tf.keras.layers.Dense(16, activation="relu"),
tf.keras.layers.Dense(5, activation="softmax")
])
model.summary()
Expected output:
Layer (type) Param #
dense 1344
dense_1 2080
dense_2 528
dense_3 85
Total params: 4037
Why Don't Activation Functions Add Parameters?
Layers such as:
Dense(32, activation="relu")
or
Dense(10, activation="sigmoid")
have exactly the same number of parameters.
Activation functions like:
- ReLU
- Sigmoid
- Tanh
- Softmax
perform mathematical operations but do not learn any weights or biases, so they contribute zero trainable parameters.
Quick Reference
| Layer | Formula |
|---|---|
| Dense | (inputs × neurons) + neurons |
| Dense (alternative form) | (inputs + 1) × neurons |
| Biases | One per neuron |
| Total Model Parameters | Sum of all layer parameters |
Key Takeaways
Every Dense layer learns weights and biases.
The parameter count for a Dense layer is:
Parameters = (Input Units × Output Units) + Output Units
- Equivalently:
Parameters = (Input Units + 1) × Output Units
The output size of one layer becomes the input size of the next layer.
The total number of trainable parameters is the sum of the parameters across all trainable layers.
You can always verify your manual calculations using
model.summary()in TensorFlow 2.x with Keras.
Top comments (0)