My Key Takeaways After Reading "Deep Learning on Graphs": Deep Learning Fundamentals

Feedforward neural networks (FNNs)

A Feedforward Neural Networks (or Multilayer Perceptrons - MLP) maps an input vector x to an expected output vector y.

1. Architecture
From a graph theory perspective, an FNN can be viewed as a multi-layer Directed Acrylic Graph (DAG). Between any two adjacent layers, the nodes from a complete bipartite graph, where every neuron in one layer connects to every neuron in the next. The number of vertices (dimensions) does not need to be constant across layers.

Linear algebraically, mapping x to y is a sequence of vector space transformations, where parameter matrices (W) and bias vector (b) shift and rotate vectors across different dimensions.

2. Activation Functions
Activation functions introduce non-linearity into the network, enabling it to learn complex non-linear mapping functions.

• ReLU (Rectified Linear Unit): Defined as f(x) = max(0, x). It sets negative values to zero. While simple and efficient, its main drawback is the "Dying ReLU" problem, where neurons getting negative inputs constantly output zero and cease learning.

• LeakyReLU: Solve the dying ReLU problem by multiplying negative inputs by a small constant factor (e.g., 0,01), allowing a small gradient to flow back.

• ELU (Exponential Linear Unit): Uses an exponential curve for negative values ( $f(x) = \alpha(e^x - 1)$ for $x < 0$ ). This creates a smoother transition around zero compared to LeakyReLU, though it requires slightly more computational overhead.

• Sigmoid: Maps values into the range (0, 1). It is commonly used in the final layer for binary classification or usually use to evaluate the result in the last layer for binary classification or probability estimation.

• Tanh: Maps values into the range (-1, 1). Zero-centered, but suffers from vanishing gradients for extreme values.

3. Output layer and Loss Function

• Output Layer: Designed based on the target task (e.g., continuous outputs for regression, Softmax probabilities for multi-class classification).

• Loss Function: Measures how far prediction deviate from the ground truth, guiding optimization (e.g., Mean Squared Error for regression, Cross-Entropy Loss for classification).

Convolutional neural networks (CNNs)

CNNs are inspired by the human visual cortex and excel at processing grid-structured data like images.

1. The Convolution Operation
Convolutional acts as a feature extraction process. A small matrix (kernel/filter) slides across the input pixels to detect local patterns such an edges, color transitions, and basic contours.

2. Convolutional Layer
Instead of dense connections, a convolutional layer employs parallel filters with specific key properties:

• Sparse connection: Each neuron only connects to a small local region of the input (receptive field), drastically reducing parameter counts.

• Sharing parameter: The same filter weights are reused across the entire input grid, making feature detection position-independent.

• Equivariant Representation: If an object in the input shifts position, its feature representation in the output shifts by the exact same amount.

3. Convolutional layer in practice
By stacking multiple convolutional layers, the network learns hierarchical representations: early layers detect low-level features (edges, textures), while deeper layers combine into high-level concepts (shape, visual objects).

4. Pooling Layer
Pooling layers downsample feature maps (via Max Pooling or Average Pooling) to reduce spatial dimensions and computational load while introducing local translation invariance.

5. Overall CNN Framework

$$\text{Input Image} \longrightarrow [\text{Conv Layer} \rightarrow \text{Activation} \rightarrow \text{Pooling}] \times N \longrightarrow \text{Flatten} \longrightarrow \text{Fully Connected} \longrightarrow \text{Output}$$

Recurrent Neural Networks (RNNs)

RNNs are designed to process sequential data (e.g., text, time-series) where context and order matter.

1. The Architecture of Traditional RNNs
Traditional RNNs process input step-by-step, maintaining an internal hidden state ( $h_t$ ) that acts as a memory passing information from time step $t-1$ to $t$ .

2. Long short-term memory (LSTM) Gated Recurrent Unit (GRU)
Standard RNNs struggle with long sequences due to the vanishing gradient problem. Therefore, LSTM and GRU are create to handle that problem

• LSTM: Introduces a dedicated Cell State and three gating mechanisms (Forget Gate, Input Gate, and Output Gate) to selectively retain or discard information over long time horizons.

• GRU: A streamlined version of LSTM that merges the cell state and hidden state, using only two gates (Reset Gate and Update Gate). It runs faster while offering comparable performance.

Autoencoder

An Autoencoder is an unsupervised neural network that learns to compress input input data x into a latent code z, and then reconstruct x from z ( $\text{Reconstruction: } x \to z \to \hat{x}$ )

1. Undercomplete Autoencoder
The bottleneck (hidden layer z) has a smaller dimension than the input layer. This forces the model to compress the data, learning the most salient core features.
is we make the hidden layer has the small dimensions than the input, the model must learn to transfer it smaller. For example, The model summarizes the paragraph and then reconstructs the entire paragraph from that summary.

2. Regularized Autoencoders
When the hidden layer dimension is equal or to larger than the input (Overcomplete), regularized autoencoders (such as Sparse Autoencoders or Denoising Autoencoders) add penalties or input noise to prevent the network from simply learning an identity copy-paste function.

Training deep neural networks

1. Gradient descent
Training minimizes the Loss Function over a high-dimensional loss surface. The gradient points in the direction of the steepest ascent, so Gradient Descent takes steps in the opposite direction (scaled by a learning rate) to find a local or global minimum (the lowest point of the loss surface).

2. Backpropagation
Backpropagation relies on the Chain Rule of Calculus to compute the derivative of the loss function with respect to every weight in the network, working backward from the output layer to the input layer to enable parameter updates.

3. Preventing overfitting
There are three common methods to prevent overfitting:

• Weight regulation: Adds a penalty proportional to the magnitude of the weights to the loss function, preventing any single weight from dominating and keeping the model simpler.

• Dropout: Randomly deactivates a fraction of neurons during training at each iteration. This prevents neurons from co-adapting too strongly and forces the network to learn robust, redundant representations.

• Batch Normalization: Normalizes activations within each mini-batch (zero mean, unit variance) before passing them to the next layer. This stabilizes training, addresses internal covariate shift, acts as a mild regularizer, and allows for higher learning rates.