DEV Community: Fouad Elhamra

# Hello 👋, Hi everyone! My name is Fouad Elhamra. I'm passionate about Artificial Intelligence, Machine Learning, and Deep Learning. Looking forward to being part of the community. Thanks for having me! 🚀

Fouad Elhamra — Thu, 06 Aug 2026 23:22:18 +0000

How to Count Parameters in Artificial Neural Networks (ANNs)

Fouad Elhamra — Thu, 06 Aug 2026 23:17:04 +0000

When building neural networks, one of the first questions you should ask is:

How many trainable parameters does my model have?

The number of parameters determines:

Model complexity
Memory usage
Training speed
Risk of overfitting

In this article, we'll learn how to calculate the number of parameters manually and verify the results using TensorFlow 2.x (Keras).

What Are Parameters?

Parameters are the values that the neural network learns during training.

There are two types:

Weights
Biases

Every neuron has:

One weight for every input it receives
One bias

Therefore, for a layer with:

Input features = n
Neurons = h

Example 1: Single Hidden Layer

Suppose we have:

Input features = 4
Hidden neurons = 5
Output neurons = 1

Architecture:

Input(4)
      │
Hidden(5)
      │
Output(1)

Hidden Layer

Each of the 5 neurons receives 4 inputs.

Weights:

4 × 5 = 20

Biases:

Total:

20 + 5 = 25

Output Layer

Input = 5

Output neurons = 1

Weights:

5 × 1 = 5

Biases:

Total:

5 + 1 = 6

Total Parameters

25 + 6 = 31

TensorFlow Verification

import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(4,)),
    tf.keras.layers.Dense(5, activation="relu"),
    tf.keras.layers.Dense(1)
])

model.summary()

Output:

Layer (type)      Output Shape     Param #

dense             (None, 5)          25
dense_1           (None, 1)           6

Total params: 31

Perfect match.

Example 2: Two Hidden Layers

Architecture:

Input(8)
      │
Hidden(16)
      │
Hidden(10)
      │
Output(3)

First Hidden Layer

Input = 8

Neurons = 16

Weights = 8 × 16 = 128
Biases = 16

Total = 144

Second Hidden Layer

Input = 16

Neurons = 10

Weights = 16 × 10 = 160
Biases = 10

Total = 170

Output Layer

Input = 10

Output neurons = 3

Weights = 10 × 3 = 30
Biases = 3

Total = 33

Total Parameters

144 + 170 + 33 = 347

TensorFlow Verification

import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(8,)),
    tf.keras.layers.Dense(16, activation="relu"),
    tf.keras.layers.Dense(10, activation="relu"),
    tf.keras.layers.Dense(3, activation="softmax")
])

model.summary()

Output:

Layer (type)      Param #

dense                 144
dense_1               170
dense_2                33

Total params: 347

Again, the manual calculation matches TensorFlow exactly.

Example 3: Deep Neural Network

Architecture:

Input(20)
      │
Hidden(64)
      │
Hidden(32)
      │
Hidden(16)
      │
Output(5)

Layer 1

(20 × 64) + 64
= 1280 + 64
= 1344

Layer 2

(64 × 32) + 32
= 2048 + 32
= 2080

Layer 3

(32 × 16) + 16
= 512 + 16
= 528

Output Layer

(16 × 5) + 5
= 80 + 5
= 85

Total Parameters

1344
+2080
+528
+85
------
4037

TensorFlow Verification

import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(20,)),
    tf.keras.layers.Dense(64, activation="relu"),
    tf.keras.layers.Dense(32, activation="relu"),
    tf.keras.layers.Dense(16, activation="relu"),
    tf.keras.layers.Dense(5, activation="softmax")
])

model.summary()

Expected output:

Layer (type)      Param #

dense              1344
dense_1            2080
dense_2             528
dense_3              85

Total params: 4037

Why Don't Activation Functions Add Parameters?

Layers such as:

Dense(32, activation="relu")

Dense(10, activation="sigmoid")

have exactly the same number of parameters.

Activation functions like:

ReLU
Sigmoid
Tanh
Softmax

perform mathematical operations but do not learn any weights or biases, so they contribute zero trainable parameters.

Quick Reference

Layer	Formula
Dense	`(inputs × neurons) + neurons`
Dense (alternative form)	`(inputs + 1) × neurons`
Biases	One per neuron
Total Model Parameters	Sum of all layer parameters

Key Takeaways

Every Dense layer learns weights and biases.
The parameter count for a Dense layer is:

  Parameters = (Input Units × Output Units) + Output Units

Equivalently:

  Parameters = (Input Units + 1) × Output Units

The output size of one layer becomes the input size of the next layer.
The total number of trainable parameters is the sum of the parameters across all trainable layers.
You can always verify your manual calculations using model.summary() in TensorFlow 2.x with Keras.

The Easiest Way to Understand Backpropagation A

Fouad Elhamra — Wed, 05 Aug 2026 17:16:23 +0000

Fouad Elhamra

Aug 5

The Easiest Way to Understand Backpropagation

#ai #machinelearning #deeplearning #datascience

2 min read

The Easiest Way to Understand Backpropagation

Fouad Elhamra — Wed, 05 Aug 2026 14:41:07 +0000

When I first learned deep learning, I thought there was only one way to train a neural network: compute the gradient and update the weights.

Then I discovered there are actually three different ways to do it.

Let's understand them with a simple example.

Imagine you have a dataset with 10,000 images.

Your goal is to minimize the loss function by updating the model's weights.

Batch Gradient Descent

Batch Gradient Descent processes the entire dataset before making a single update.

10,000 samples -> Compute total loss -> Compute gradients -> Update weights

Advantages

Stable gradient
Smooth convergence
Accurate update direction

Disadvantages

Slow for large datasets
Requires lots of memory
One update only after processing every sample

Think of it like reading an entire book before writing a summary.

2. Stochastic Gradient Descent (SGD)

Instead of waiting for all 10,000 samples, SGD updates the weights after every single sample.

Sample 1 → Update
Sample 2 → Update
Sample 3 → Update
...
Sample 10,000 → Update

Now the model learns much faster.

The downside?

Each sample may point in a slightly different direction, causing the optimization path to bounce around.

Imagine hiking toward the bottom of a valley while someone changes your direction every few seconds.

This creates the famous zigzag optimization path.

Advantages

Very fast updates
Low memory usage
Can escape some local minima

Disadvantages

Noisy gradients
Unstable convergence
Loss fluctuates a lot

3. Mini-Batch Gradient Descent

Mini-batch combines the best parts of both approaches.

Instead of using one sample or the entire dataset, we split the data into small batches.

For example:

10,000 samples

Batch 1 = 128 samples → Update
Batch 2 = 128 samples → Update
Batch 3 = 128 samples → Update
...

Now every update is based on enough data to reduce noise, but not so much that training becomes slow.

This is why frameworks like PyTorch and TensorFlow use mini-batches by default.

Advantages

Faster than Batch GD
More stable than SGD
Efficient GPU utilization
Standard choice for deep learning

Comparison

Method	Update Frequency	Speed	Stability
Batch Gradient Descent	After the entire dataset	Slow	High
Stochastic Gradient Descent	After every sample	Fast	Low
Mini-Batch Gradient Descent	After every small batch	Fast	High

Final Thoughts

If you're training modern neural networks, Mini-Batch Gradient Descent is usually the best choice.

It balances speed, memory usage, and convergence, making it the default optimization strategy in most deep learning libraries.

Understanding these three optimization strategies helped me understand why neural networks train the way they do—not just how they train.