DEV Community: Hion

My Key Takeaways After Reading "Deep Learning on Graphs": Graph Neural Networks

Hion — Fri, 21 Aug 2026 13:56:51 +0000

Introduction

Graph Neural Networks (GNNs) apply deep neural network concepts directly to graph structures. In traditional graph embeddings, one vector holds static information. In GNNs, a vector acts like a dynamic vector — much like comparing a static array to a dynamic array—creating a supervector that contains its own data plus its neighbors' data. Adding non-linear activation functions (like ReLU) allows this dynamic vector trajectory to become flexible.

From a linear perspective:

Vector Addition: Adding two vectors connects the head of one to the tail of another, producing a resulting vector that completes the missing side of a triangle—this is how vectors bundle information.
Matrix Multiplication: Represents a sequence of linear transformations that shifts the coordinate system from right to left, streamlining data and discarding unused information.

GNNs preserve the original graph topology to maintain variety, breaking the rigid matrix rules where every item must have a fixed number of neighbors (like pixels in an image having exactly 8 neighbors).

Core GNN Tasks

Node-Focused Tasks (Node Classification): Focuses on individual objects, including identifying fraud accounts, or recommender shopping systems. It combines multiple pieces of information into single vectors to group nodes into specific areas for prediction. The total number of vectors stays unchanged—only the vector dimensions change.
Graph-Focus Tasks (Graph Classification): Focuses on global structure, such as analyzing molecular properties (CH2, CH4) or detecting malicious code sequences. It works like node-focus but adds a pooling step at the end to condense everything into a single vector inheriting all features.

GNNs can be accessed from two perspectives: Spectral-based (algebraic) and Spatial-based (geometric).

Spectral-based Graph Filters

This matrix method subtracts the degree matrix from the adjacency matrix (the Graph Laplacian), where diagonal entries are zero and non-neighbor values are negative, measuring vector interactions through wave oscillations.

Direct spatial convolutional on irregular graph topologies is mathematically challenging. Spectral filtering addresses this by transforming graph signals into the frequency domain via Graph Fourier Transform. In the frequency domain, complex convolutions convert into straightforward matrix multiplications. Once filtered, the signals are transformed back to the original spatial domain.

Converting graph data to a spectral space and back serves a clear purpose: human do not know how to draw wave oscillations, so deep learning converts the signal to adjust parameters ( $λ$ ) in that transformed space until it finds the optimal result.

Chebyshev Polynomials & Cheby-Filter: Instead of giving the model a pen and letting it draw freely without instruction, Cheby-Filter hands the model a function—a ruler or geometric tool—so it draws structured shapes without overthinking. Using dynamic programming, level k inherits results from level k-1. Inputs shrink to the range [-1, 1] to save computational energy and prevent exploding values.
GCN-Filter: An upgraded, simplified Cheby-Filter. Instead of using complex curved rulers, circles, or triangles, GCN uses simple horizontal or cross lines to model wave oscillations. It assumes a maximum eigenvalue of $2$ to make calculations lighter, focusing purely on immediate 1-hop neighbors.

Spatial-based Graph Filters

Spatial filters operate directly in coordinate space using vector addition to combine information and matrix multiplication to filter it, making node updates much easier to implement.

Early Graph Filters: Used rigid, fixed calculation functions, making the model robot-like and lacking adaptability
GraphSAGE-Filter: Randomly picks a subset of neighboring nodes and zips their information. It uses clever tricks, priorizing nodes with high weights/degrees or running random walks to pick representative neighborhood nodes.
GAT-Filter: While GCN treats all graph structures uniformly, GAT evaluates specific nodes. Like a talent contest, neighbor nodes that fit better receive higher scores. GAT even employs multiple juges (multi-head attention) to rank neighbors from different perspectives.
ECC-Filter (Edge-Conditioned Convolution): Uses edge features to produce transformations for nodes.
Mo-Filter: Introduces pseudo-coordinates to make graph geometry flexible.
MPNN (Message Passing Neural Network): A General Serves as the overarching general framework summarizing all spatial filters.

Graph Pooling

Graph pooling shrinks all node vectors into a single graph-level vector (summarizing both feature of nodes and structure of graph). Because one vector must absorb massive amounts of data, information loss is a risk. Therefore, Hierarchical Graph Pooling is the solution to this problem:

Hierarchical Graph Pooling: Similar to the Chuliu-Edmond algorithm for finding minimum spanning trees (with its contracting and expanding phases), this approach avoids collapsing everything at once. It condenses the graph step-by-step from local components to larger structures.

Parameter Learning for Graph Neural Networks

Parameter Learning for Node Classification: Makes individual nodes smarter by training decision boundaries in localized classification areas.
Parameter Learning for Graph Classification: Verifies whether global graph structures convey accurate collective information.

After-learning notes: The Laplacian matrix link back to Pierre-Simon Laplace and the concept of Laplace's Demon - a theoretical entity capable of predicting the future by calculating physical state data. Modern GNNs reflect a similar goal: by learning from past graph topologies, models predict future user purchase desires or gold price trends, almost as if attempting to recreate that demon.

My Key Takeaways After Reading "Deep Learning on Graphs": Graph Embedding

Hion — Wed, 19 Aug 2026 14:06:35 +0000

Graph embedding is simply how we map a graph's nodes into vector in a continuous coordinate space.

From a linear algebra perspective, an adjacency matrix A represents the graph's connections and edge weights. Our goal is to transform each node into a d-dimensional vector in a lower-dimensional coordinate system. The edges represent relationships between vectors, which we measure using the dot product. If two vectors point in the same direction (dot product close to 1), they are highly similar or strongly connected.

Random Walk & DeepWalk
A random walk is a graph traversal algorithm, similar in spirit to BFS and DFS, but without strict deterministic rules.

To visualize a Random Walk, imagine placing an ant on a node. The ant walks randomly from node to node, recording its travel path. By analyzing these paths, we discover which nodes frequently show up together (co-occurrence). We then update out vector space to pull these co-occurring nodes closer so their vectors point in similar directions. This entire process forms the foundation of DeepWalk.

Why use Random Walks instead of traditional DFS or BFS?

Parallel Processing: DFS and BFS are linear and constraint-heavy, you can only process one path at time with a single "ant". Random Walks, however, allow us to release 1,000 ants simultaneously to explore different parts of the graph in parallel, making training significantly faster.
Capturing Community: Random walks better reveal structural contexts and local communities.

Since an ant could theoretically walk endlessly across the entire graph, we use a context window restricts the ant's view to just 1 or 2 steps ahead, focusing specifically on highly localized node relationships.

After each walk, a reconstruction function updates the relations among nodes. Updating incrementally after each walk, rather than waiting to process the while graph at once, keeps the coordinate system stable and prevents vectors from over-fitting to just a few neighbors. However, calculating dot product between one node and billions of other is computationally impossible. To solve this, we rely on divide-and-conquer strategies and tree structures to accelerate the learning process.

Hierarchical Softmax
Hierarchical Softmax uses a binary tree structure to group similar nodes on the same branch. The closer a node is to the root, the boarder its category.

Think of it like a university: the root represents the university itself, the next layer branches into specific majors (e.g., Computer Science), and deeper layers descend into specialized courses for that major. By using this divine-and-conquer tree structure, the model can inherit prior node decisions and dramatically speed up vector calculations from $O (N)$ to $O (lo g N)$ .

Negative Sampling
Negative Sampling works like a gacha game with a clever cheat.

When updating a target node, calculating its relationship against all other nodes in a massive graph is unrealistic. Instead of evaluating everything, the algorithm picks the 1 actual neighbor (the rare "5-star" positive pull) to draw closer, and randomly samples a few unrelated nodes (the common "1-star" negative pulls) to push far away in the vector space.

Node2Vec
Technology is never stop evolving, and Node2Vec is the upgrade version of DeepWalk.

While Deepwalk relies on purely uniform random walks, Node2Vec introduces two parameters, p and q, to control the ant's walking behavior:

Return Parameter (p): A small p encourages the ant to stay close to the starting node (like a localized BFS search), capturing tight-knit communities.
In-out Parameter (q): A small q encourages the ants to explore outward into new regions (like a deep DFS search), capturing structural roles.

LINE
Rather than relying solely on linear paths, LINE explicity models two structural perspective:

First order Proximity: Direct connections between nodes.
Second-order Proximity: Shared neighborhood contexts. Even if two nodes do not have a direct adge between them, if they share almost identical context vectors (similar neighbors), LINE pulls their embeddings closer together.

Struct2Vec
Struct2Vec builds upon LINE's ideas, Relying only on immediate neighbors can lead to mistakes when identifying global roles. Struc2vec expands this scope by recursively evaluating the neighborhood structure of a node's neighbors. Nodes with similar structural roles (e.g., "hubs" or "leaf nodes") are mapped close together in the vector space, even if they are located on opposite sides of the graph.

My Key Takeaways After Reading "Deep Learning on Graphs": Deep Learning Fundamentals

Hion — Tue, 18 Aug 2026 17:59:24 +0000

Feedforward neural networks (FNNs)

A Feedforward Neural Networks (or Multilayer Perceptrons - MLP) maps an input vector x to an expected output vector y.

1. Architecture
From a graph theory perspective, an FNN can be viewed as a multi-layer Directed Acrylic Graph (DAG). Between any two adjacent layers, the nodes from a complete bipartite graph, where every neuron in one layer connects to every neuron in the next. The number of vertices (dimensions) does not need to be constant across layers.

Linear algebraically, mapping x to y is a sequence of vector space transformations, where parameter matrices (W) and bias vector (b) shift and rotate vectors across different dimensions.

2. Activation Functions
Activation functions introduce non-linearity into the network, enabling it to learn complex non-linear mapping functions.

ReLU (Rectified Linear Unit): Defined as f(x) = max(0, x). It sets negative values to zero. While simple and efficient, its main drawback is the "Dying ReLU" problem, where neurons getting negative inputs constantly output zero and cease learning.

LeakyReLU: Solve the dying ReLU problem by multiplying negative inputs by a small constant factor (e.g., 0,01), allowing a small gradient to flow back.
ELU (Exponential Linear Unit): Uses an exponential curve for negative values ( $f (x) = α (e^{x} - 1)$ for $x < 0$ ). This creates a smoother transition around zero compared to LeakyReLU, though it requires slightly more computational overhead.
Sigmoid: Maps values into the range (0, 1). It is commonly used in the final layer for binary classification or usually use to evaluate the result in the last layer for binary classification or probability estimation.
Tanh: Maps values into the range (-1, 1). Zero-centered, but suffers from vanishing gradients for extreme values.

3. Output layer and Loss Function

Output Layer: Designed based on the target task (e.g., continuous outputs for regression, Softmax probabilities for multi-class classification).
Loss Function: Measures how far prediction deviate from the ground truth, guiding optimization (e.g., Mean Squared Error for regression, Cross-Entropy Loss for classification).

Convolutional neural networks (CNNs)

CNNs are inspired by the human visual cortex and excel at processing grid-structured data like images.

1. The Convolution Operation
Convolutional acts as a feature extraction process. A small matrix (kernel/filter) slides across the input pixels to detect local patterns such an edges, color transitions, and basic contours.

2. Convolutional Layer
Instead of dense connections, a convolutional layer employs parallel filters with specific key properties:

Sparse connection: Each neuron only connects to a small local region of the input (receptive field), drastically reducing parameter counts.
Sharing parameter: The same filter weights are reused across the entire input grid, making feature detection position-independent.
Equivariant Representation: If an object in the input shifts position, its feature representation in the output shifts by the exact same amount.

3. Convolutional layer in practice
By stacking multiple convolutional layers, the network learns hierarchical representations: early layers detect low-level features (edges, textures), while deeper layers combine into high-level concepts (shape, visual objects).

4. Pooling Layer
Pooling layers downsample feature maps (via Max Pooling or Average Pooling) to reduce spatial dimensions and computational load while introducing local translation invariance.

5. Overall CNN Framework

Input Image ⟶ [Conv Layer \to Activation \to Pooling] \times N ⟶ Flatten ⟶ Fully Connected ⟶ Output

Recurrent Neural Networks (RNNs)

RNNs are designed to process sequential data (e.g., text, time-series) where context and order matter.

1. The Architecture of Traditional RNNs
Traditional RNNs process input step-by-step, maintaining an internal hidden state ( $h_{t}$ ) that acts as a memory passing information from time step $t - 1$ to $t$ .

2. Long short-term memory (LSTM) Gated Recurrent Unit (GRU)
Standard RNNs struggle with long sequences due to the vanishing gradient problem. Therefore, LSTM and GRU are create to handle that problem

LSTM: Introduces a dedicated Cell State and three gating mechanisms (Forget Gate, Input Gate, and Output Gate) to selectively retain or discard information over long time horizons.
GRU: A streamlined version of LSTM that merges the cell state and hidden state, using only two gates (Reset Gate and Update Gate). It runs faster while offering comparable performance.

Autoencoder

An Autoencoder is an unsupervised neural network that learns to compress input input data x into a latent code z, and then reconstruct x from z ( $Reconstruction: x \to z \to x^$ )

1. Undercomplete Autoencoder
The bottleneck (hidden layer z) has a smaller dimension than the input layer. This forces the model to compress the data, learning the most salient core features.
is we make the hidden layer has the small dimensions than the input, the model must learn to transfer it smaller. For example, The model summarizes the paragraph and then reconstructs the entire paragraph from that summary.

2. Regularized Autoencoders
When the hidden layer dimension is equal or to larger than the input (Overcomplete), regularized autoencoders (such as Sparse Autoencoders or Denoising Autoencoders) add penalties or input noise to prevent the network from simply learning an identity copy-paste function.

Training deep neural networks

1. Gradient descent
Training minimizes the Loss Function over a high-dimensional loss surface. The gradient points in the direction of the steepest ascent, so Gradient Descent takes steps in the opposite direction (scaled by a learning rate) to find a local or global minimum (the lowest point of the loss surface).

2. Backpropagation
Backpropagation relies on the Chain Rule of Calculus to compute the derivative of the loss function with respect to every weight in the network, working backward from the output layer to the input layer to enable parameter updates.

3. Preventing overfitting
There are three common methods to prevent overfitting:

Weight regulation: Adds a penalty proportional to the magnitude of the weights to the loss function, preventing any single weight from dominating and keeping the model simpler.
Dropout: Randomly deactivates a fraction of neurons during training at each iteration. This prevents neurons from co-adapting too strongly and forces the network to learn robust, redundant representations.
Batch Normalization: Normalizes activations within each mini-batch (zero mean, unit variance) before passing them to the next layer. This stabilizes training, addresses internal covariate shift, acts as a mild regularizer, and allows for higher learning rates.

Understanding Deep Learning: CHAPTER 2: Supervised learning Notes

Hion — Mon, 17 Aug 2026 13:33:54 +0000

Supervised learning is the process where we give a computer an input and its expected output, forcing the computer to figure out how to map that input to that output. The model handles this using a vast number of parameters. Through supervised learning, these parameters are continuously adjusted after failing billion of times, eventually learning how to reach the correct answers.

1. The core equation: y = f[x, φ]

This formula means that from the input x, we get the predicted output y. Here, φ represents the parameters (weights and biases) that the model to find so that, through mathematical operations, it can accurately predict y from x.

Depending on what y is, supervised learning is divided into two main types of problems:

Regression: Used when the output y is a continuous number. For example, predicting a house price or tomorrow's temperature.
Classification: Used when the output y is a discrete label/category. For example, identifying whether an image is a "cat" or "dog", or detecting if an email is "spam" or "not spam".

This process works similarly to matrix transformations. The input shape, combined with specific parameter and weights, goes through calculations to yield the final output shape.

2. How the Model Self-Corrects: Derivatives & Gradient Descent

The process behind adjusting these parameters is driven by derivatives and gradient descent.

With the initial parameters chosen randomly, the loss is usually very high. The model must find a way to reduce this error little by little, and the key methods for this is taking derivatives. A derivative is basically a gradient (the slope of the error hill). By calculating the derivative with respect to each parameter, the model knows exactly which direction to move. This continuous adjustment forces the loss to decrease step by step, until it reach the flat bottom where it cannot be reduced anymore.

3. Evaluating Performance: Loss & Testing

Loss is the metric used to evaluate how accurate the model's prediction are. The loss value will be very highly (poor) if the model makes an incorrect prediction with a high confident, or if it makes a correct prediction but with very low confident. The ultimate goal of training is to reduce this loss.

To evaluate the model properly and prevent it from just "memorizing" answers, we don't just use one dataset. Instead, the data is usually split into three distinct parts:

Training Set: The data the model uses to learn and adjust its parameters through gradient descent (like doing homework).
Validation Set: The data used to test the model during training. It helps us tune the model's structure and check if it is starting to overfit (like a practice quiz before the real exam).
Testing Set: The final exam. This data is kept completely hidden until training is finished to see how the model performs in the real world.

During this evaluation process, we often encounter two major issues:

Underfitting: The model learns nothing and performs poorly on both training and testing data.
Overfitting: The model just memorizes the training data, making it unable to predict anything new.

Class Notes: Making Sense of Dynamic Programming

Hion — Fri, 24 Jul 2026 10:07:47 +0000

This is just a quick note to log how I understand Dynamic Programming after listening to the lecture in class today.

Dynamic programming (DP) is very similar to the Divide and Conquer algorithm. The main difference is that in DP, the subproblems tend to repeat.

There are three core ideas to remember about DP:

1. Optimal Substructure

Like Divide and Conquer, a big problem is broken down into many smaller subproblems. We can solve the main problem by finding and selecting the optimal outputs of these subproblems.

Example (Floyd-Warshall Algorithm): To find the shortest paths between all pairs of vertices in a graph, we break it down by finding the shortest path between each specific pair (u, v) first. Combining these optimal sub-answers gives us the final solution for the entire graph.

2. Overlapping subproblems

in DP, the same subproblems show up over and over again. Instead of recalculating the exact same answer every time, DP solves it once, saves the result, and reuses it later.

Example: Imagine you want to find the shortest path to school from two different starting locations: your home and your grandparent's house. Both routes must pass through the stationary store to reach school Path(home -> school) = dis(home -> store) + dis(store -> school) Path(grandparent->school) = dis(grandparent -> store) + dis(store -> school)

as you can see, the fragment dis(store -> school) is shared. DP calculates this distance once, saves it, and instantly reuses it for both routes.

3. Table-Based Solution Tracking

To build the final solution, DP keeps track of all sub-problem answers in a table (often called a DP table or memoization table). This is how we synthesize everything together.

Example: Suppose from your home, you can choose to pass through either a grocery store, a convenience store or a book store to get to school.
Your subproblems are finding the shortest path from home to each store to the school. DP saves all these options in a table. In the end, you look at the table, compare the total costs, and pick the single best store to pass through to minimize your travel time.

My Study Notes on Fastai: Chapter 1

Hion — Thu, 23 Jul 2026 10:12:56 +0000

1. Parallel distributed processing (PDP)

Traditional computer program work very differently from brains. That might be why computer programs had been so bad at doing things that brains find easy, such as recognizing objects in picture. To solve this, PDP was developed as an architecture to imitate how the brain works.

According to the book, a PDP system requires eight essential components:

A set of processing units
A state of activation
An output function for each unit
A pattern of connectivity among units
A propagation rule
An activation rule
A learning rule
An environment

2. A Brief History of Neural Networks (The two AI winters)

In the past, Marvin Minsky showed that single-layer neural network could not learning critical math function (like XOR gate). For example, a simple computer couldn't correctly classify an object if the input shared characteristics of both categories. In the same book, the authors also show that adding more layers could easily solve this problem. Unfortunately, the public only focused on the negative statement. As a result, funding dried up, and the first AI Winter began.

In the 1980s, researcher started exploring two-layer models. Theoretically, twos layers were enough to learn any mathematical function. However, in practice, these models were too large and too slow to be useful at the time. This triggered the second AI Winter.

Today, technology has finally caught up. We now use GPUs to solve the speed and size limitations of the past. To put in simply: a CPU is great at solving a few big, complex problems, but it takes too long when handling thousands of tiny tasks. On the other hand, a GPU is designed to solve thousands of small problems (like rendering pixels) at the exact same time, By using GPUs to boost performance, AI has finally experienced a massive revival.

3. The Difference between Machine Learning and Deep Learning

Deep learning is just a modern subfield within the boarder discipline of machine learning.

Machine learning requires humans to manually select and extract features (characteristics) from the data before feeding it to the model.
Deep learning automatically learns the characteristics of an object directly from raw data without needing human intervention.

4. The Universal Framework: Arthur Samuel's Model

Modern Deep Learning is a direct implementation of Arthur Samuel's concept, but with professional technical terms:

Architecture: The mathematical structure of the model (previously called Model).
Parameters: The setting/weights inside the model that change as it learns (previously called Weights).
Predictions: The model's guesses based on inputs and parameters (previously called Results).
Loss: The scoring system that measures how wrong the predictions are compared to the Labels (previously called Performance).

5. Universal Approximation Theorem

The Universal Approximation Theorem is a mathematical proof showing that deep neural networks can theoretically solve any complex problem. However, in practice, because of limits in computer hardware and the amount of data we have, we cannot solve literally everything, but it is still incredibly powerful.

6. Current Limitations of Deep Learning

Requires Labeled Data: Models cannot learn by themselves; humans must label the objects (e.g., tagginf which image is a cat or a dog) before training.
The "Black Box" Problems: If a model only has a few layers, we can easily see which variables are important. But deep networks have hundreds of layers, making them hard to understand. Fortunately, we can now step-by-step visualize how they work (such as checking what the first layer sees, like lines or shapes).
Cannot Predict Beyond Training Data: The model will predict poorly if the testing condition is too different from the training data. For example, if we only train it with daytime images, it will fail when testing with nighttime images.