DEV Community: arham ahmed

Transformers: Understanding the Architecture Behind Modern AI

arham ahmed — Fri, 28 Aug 2026 19:32:46 +0000

Introduction

Transformers are the heart of modern AI models. AI has seen a lot of breakthrough advancements from ChatGPT to AI, and now. After the development of Transformers, translations and question-answering tasks have seen a breakthrough.

In this blog, I will try to explain Transformers from the basic concepts to how the complete architecture works.

Why Transformers?

It's not that Transformers were the first architecture to exist. Various architectures existed before, like RNNs, which take the current input and also the previous input.

For example:

I am a boy

t₁ = I
t₂ = I + am
t₃ = I + am + a
t₄ = I + am + a + boy

But its popularity faded for long sequences as the effect of earlier inputs starts disappearing.

Then the attention mechanism came, which focused not only on previous inputs but also on the relevant inputs to focus on. But their main limitation was that sentences with many words were not handled as effectively.

To overcome the above limitations, Transformers using multi-head attention came into the picture.

Transformer: Encoder and Decoder

A Transformer is like a human that first encodes a sentence, understands it, and then decodes it according to the required task.

The Transformer has two major parts:

TRANSFORMER
/ \
/ \
ENCODER DECODER
↓ ↓
Understands Generates
the input output

The original Transformer architecture contains multiple encoder and decoder layers.

The original Transformer uses 6 encoder layers and 6 decoder layers.

Tokenization and Embeddings

Before reaching the main architecture, there are some prerequisites.

First, the entire sentence is divided into tokens and then converted into embeddings.

For example:

I am a boy
↓
[I, am, a, boy]
↓
Token IDs
↓
Embedding vectors

Each single token is a vector of a particular dimension, for example (768, 1024) depending on the model.

Think of it like what a particular word means in mathematics in a coordinate system.

So instead of giving the Transformer raw words, we convert every word/token into a numerical vector.

Positional Encoding

Since we are working with long sequences, the position of each token is very important.

Which token comes first or second can completely change the meaning of a sentence.

Therefore, positional encodings are used to provide information about the position of each token.

How do they work?

The original Transformer uses sine and cosine functions.

PE(pos, 2i) = sin(pos / 10000^(2i/d_model))

PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

For example, let's take:

Sentence = "I love AI"

d_model = 4

For position 0:

PE(0) = [0, 1, 0, 1]

For position 1:

PE(1) ≈ [0.8415, 0.5403, 0.0100, 0.99995]

For position 2:

PE(2) ≈ [0.9093, -0.4161, 0.0200, 0.9998]

Now suppose the embedding of "love" is:

Embedding(love) = [0.4, 0.3, 0.8, 0.2]

Since "love" is at position 1:

Embedding = [0.4000, 0.3000, 0.8000, 0.2000]

PE(1) = [0.8415, 0.5403, 0.0100, 0.99995]

            ↓ ADD

Transformer input = [1.2415, 0.8403, 0.8100, 1.19995]

So positional encoding basically tells the model:

What is the token + Where is the token?

Self-Attention

Now comes the most important part: Attention.

Consider the sentence:

I am a boy

Each token is projected into the coordinate system as a vector.

It is broken down into three parts by transformations:

Query (Q)

Key (K)

Value (V)

Think of them like this:

Query

"What am I looking for?"

Key

"See, this is how I look in the coordinate system. Check how much I match with your Query."

Value

"This is what my value in the system contains."

What happens is that the Query vector for each word is multiplied with the Key vectors for each word using a dot product.

The result is divided by √dₖ because for large dimensions the dot product can become large, pushing the probability toward one side.

Then Softmax is applied.

Attention(Q,K,V) = softmax(QKᵀ / √dₖ)V

The Softmax gives the attention weights.

Then these weights are multiplied with the corresponding Value vectors.

This process is repeated multiple times to capture semantic similarity between tokens. This is known as multi-head attention.

Numerical Example of Self-Attention

Let's actually calculate a small example.

Consider:

I am a boy

For simplicity, let:

dₖ = 2

Suppose the vectors are:

Token Q K V

I [1,0] [1,1] [1,0]
am [0,1] [1,0] [0,1]
a [1,1] [1,1] [1,1]
boy [1,-1] [0,1] [0,2]

Let's calculate attention for the last token "boy".

Step 1: Calculate Q × Kᵀ

For "boy":

Q(boy) = [1,-1]

Against every Key:

[1,-1] · [1,1] = 0

[1,-1] · [1,0] = 1

[1,-1] · [1,1] = 0

[1,-1] · [0,1] = -1

So:

QKᵀ = [0, 1, 0, -1]

Step 2: Scale by √dₖ

Since:

dₖ = 2
√dₖ = √2 ≈ 1.414

Therefore:

[0, 1, 0, -1] / 1.414

≈ [0, 0.707, 0, -0.707]

Step 3: Apply Softmax

Softmax([0, 0.707, 0, -0.707])

≈ [0.157, 0.256, 0.157, 0.430]

These are the attention weights.

So "boy" pays different amounts of attention to each token.

Step 4: Multiply by V

0.157 × [1,0]
+
0.256 × [0,1]
+
0.157 × [1,1]
+
0.430 × [0,2]

Therefore:

≈ [0.314, 1.017]

So the output representation for "boy" becomes approximately:

[0.314, 1.017]

The important idea is not the particular numbers, but the process:

Q
↓
QKᵀ
↓
Scale by √dₖ
↓
Softmax
↓
Attention weights
↓
Weighted sum of V
↓
Contextual representation

Multi-Head Attention

Instead of one attention operation, we run multiple attention heads in parallel.

Each head can learn different types of relationships.

For example, one head may focus on:

subject ↔ verb

while another may focus on:

verb ↔ object

The outputs of all heads are then concatenated and passed through a linear transformation.

Input
│
┌───────────┼───────────┐
↓ ↓ ↓
Head 1 Head 2 Head 3 ... Head h
↓ ↓ ↓
└───────────┼───────────┘
↓
Concatenate
↓
Linear Layer
↓
Output

The equation is:

headᵢ = Attention(QWᵢQ, KWᵢK, VWᵢV)

MultiHead(Q,K,V) =
Concat(head₁, head₂, ..., headₕ)Wᴼ

Residual Connection and Layer Normalization

After the attention operation, the output is combined with the original input using a residual connection.

Then layer normalization is applied.

The basic idea is:

Input
↓
Attention
↓
Add original input
↓
Layer Normalization
↓
Output

Residual connections help preserve information and allow gradients to flow through deeper networks.

Layer normalization helps stabilize the network during training.

Feed-Forward Neural Network

Further, a Feed-Forward Neural Network (FFN) is applied to each token.

The equation is:

FFN(x) = max(0, xW₁ + b₁)W₂ + b₂

The FFN is applied independently to every token.

Its purpose is to further transform the information received from the self-attention layer and introduce non-linearity.

So a simple way to remember it is:

Attention: Which tokens should interact?

FFN: What transformation should be applied to the resulting representation?

After the FFN, residual connection and layer normalization are applied again.

Complete Encoder

The above entire process forms one encoder layer.

Input Embedding
+
Positional Encoding
↓
Multi-Head Self-Attention
↓
Add & Layer Normalization
↓
Feed-Forward Network
↓
Add & Layer Normalization
↓
Encoder Output

The original Transformer has 6 such encoder layers stacked together.

Encoder Layer 1
↓
Encoder Layer 2
↓
Encoder Layer 3
↓
Encoder Layer 4
↓
Encoder Layer 5
↓
Encoder Layer 6
↓
Final Encoder Output

Each layer builds a richer contextual representation.

Decoder

After the encoder finishes, the output embedding is passed to the decoder.

After applying positional embeddings, the decoder undergoes masked multi-head attention.

Unlike the encoder, the decoder takes the output token by token at a time.

For example:

At the first step, the decoder predicts the first output token.

Then:

→ I

Then:

I → am

Then:

I am → a

and so on.

This is called autoregressive generation.

Why Masked Attention?

The decoder must not see future tokens while generating the current token.

For example, if we are predicting:

I am a boy

while predicting "a", the decoder should not already know "boy".

Therefore, a mask is applied.

I am a boy

I ✓
am ✓ ✓
a ✓ ✓ ✓
boy ✓ ✓ ✓ ✓

This is called causal/masked self-attention.

The target sequence is also right-shifted, meaning the decoder receives previously generated/known tokens to predict the next token.

Cross-Attention: How Encoder Output Reaches the Decoder

This is an important part of the Transformer.

The final output of Encoder 6 is used by the cross-attention sublayer of the decoder layers.

In cross-attention:

Query (Q) → comes from decoder

Key (K) → comes from final encoder output

Value (V) → comes from final encoder output

So:

Encoder
↓
Encoder Layer 6
↓
Final Encoder Output
│
┌───────────┼───────────┐
↓ ↓ ↓
Decoder 1 Decoder 2 ... Decoder 6
Cross-Attn Cross-Attn Cross-Attn

The decoder's own output flows from one decoder layer to the next, while the encoder's final representation is available to the cross-attention of every decoder layer.

Decoder Layer

A decoder layer therefore contains:

Target Embedding
+
Positional Encoding
↓
Masked Multi-Head Self-Attention
↓
Add & LayerNorm
↓
Multi-Head Cross-Attention
↑
Final Encoder Output
↓
Add & LayerNorm
↓
Feed-Forward Network
↓
Add & LayerNorm
↓
Decoder Output

The original Transformer contains 6 decoder layers.

Final Output

After passing through the decoder layers, a linear layer is applied.

Then Softmax converts the final values into probabilities.

Decoder Output
↓
Linear Layer
↓
Softmax
↓
Probability of each token
↓
Most suitable next token

For example:

I am a ______

boy → 0.72
girl → 0.12
student → 0.08
doctor → 0.03
...

The model selects a token according to the generation strategy being used.

Complete Transformer Workflow

Here is the complete flow of the original encoder-decoder Transformer:

INPUT SENTENCE
↓
TOKENIZATION
↓
EMBEDDING
↓
POSITIONAL ENCODING
↓
┌──────────────────────┐
│ ENCODER × 6 │
│ │
│ Multi-Head Attention │
│ ↓ │
│ Add & Norm │
│ ↓ │
│ FFN │
│ ↓ │
│ Add & Norm │
└──────────┬───────────┘
↓
FINAL ENCODER OUTPUT
│
│
↓
TARGET → Embedding + Positional Encoding
↓
┌──────────────────────┐
│ DECODER × 6 │
│ │
│ Masked Self-Attention│
│ ↓ │
│ Add & Norm │
│ ↓ │
│ Cross-Attention ←┘
│ ↓
│ Add & Norm
│ ↓
│ FFN
│ ↓
│ Add & Norm
└──────────┬───────────┘
↓
Linear Layer
↓
Softmax
↓
OUTPUT TOKEN
↓
NEXT TOKEN ...

Putting the Whole Idea Together

The Transformer can now be understood as a sequence of operations:

Words
↓
Tokens
↓
Embeddings
↓
Positional Information
↓
Self-Attention
↓
Multi-Head Attention
↓
Feed-Forward Network
↓
Repeat through encoder layers
↓
Final Encoder Representation
↓
Decoder
↓
Masked Self-Attention
↓
Cross-Attention with Encoder Output
↓
Feed-Forward Network
↓
Repeat through decoder layers
↓
Linear + Softmax
↓
Output

The most important idea is that attention allows the model to decide which other tokens are relevant when building the representation of a particular token.

Applications

Transformers have become extremely important in modern AI.

Some applications include:

Machine translation

Text generation

Question answering

Text summarization

Sentiment analysis

Code generation

Image understanding

Multimodal AI

Models such as BERT, GPT and Vision Transformers are based on the Transformer idea, although their architectures can differ from the original encoder-decoder Transformer.

Conclusion

Transformers changed the way we process sequential data by introducing attention as the central mechanism for understanding relationships between tokens.

Instead of processing information strictly one token after another like traditional recurrent architectures, Transformers can use attention to determine which tokens are important to each other.

The major components are:

Tokenization
↓
Embeddings
↓
Positional Encoding
↓
Self-Attention
↓
Multi-Head Attention
↓
Feed-Forward Network
↓
Residual Connections + Layer Normalization
↓
Encoder / Decoder
↓
Cross-Attention
↓
Linear + Softmax
↓
Output

Understanding these components makes it much easier to understand modern architectures such as BERT, GPT and other Transformer-based models.

This is how Transformers work — from converting words into vectors to using attention to understand relationships and finally generating an output.

XGBoost Explained: From Gradient Boosting to Weighted Quantile Sketch

arham ahmed — Fri, 28 Aug 2026 18:01:00 +0000

XGBoost

Introduction: XGBoost

XGBoost is a tree-based algorithm. It is a variant of Gradient Boosting which is one of the highest-scaled versions of it. It performs calculations 10x faster than Gradient Boosting. Its success is also witnessed in Kaggle's competitions wherein out of 29, 17 solutions used XGBoost. Its capability also shines in handling sparse data and null values.

Before understanding XGBoost let's understand boosting first.

Boosting

In simple words, boosting is: take the mistakes/errors of the first model and send it to another model to correct it. Then again predict the actual target, identify the errors, and again send them further for correction.

It is the working principle for both normal GB and XGBoost. However, XGBoost is far better than the normal one in scaling.

Objective Function

L(φ) = Σᵢ l(ŷᵢ, yᵢ) + Σₖ Ω(fₖ)

where:

Ω(f) = γT + ½λ Σⱼ wⱼ²

Above is an objective function which XGBoost has to minimize.

where:

T → Number of leaves

w → Weight of leaf

But the main drawback is that the above equation is non-optimizable in Euclidean space as each fₖ is a decision tree. It can't be optimized simply as Linear Regression because each tree structure is different.

Hence, some more modifications are done. The 2nd-order Taylor series is used.

L⁽ᵗ⁾ = Σᵢ l(yᵢ, ŷᵢ⁽ᵗ⁻¹⁾ + fₜ(xᵢ)) + Ω(fₜ)

Using the second-order Taylor expansion:

L⁽ᵗ⁾ ≈ Σᵢ [l(yᵢ, ŷᵢ⁽ᵗ⁻¹⁾) + gᵢfₜ(xᵢ) + ½hᵢfₜ(xᵢ)²] + Ω(fₜ)

Removing the constant term:

L⁽ᵗ⁾ ≈ Σᵢ [gᵢfₜ(xᵢ) + ½hᵢfₜ(xᵢ)²] + Ω(fₜ)

Gradient tells which direction to move.

Hessian tells the shape/curvature of the curve.

Therefore:

L⁽ᵗ⁾ = Σᵢ [gᵢfₜ(xᵢ) + ½hᵢfₜ(xᵢ)²] + γT + ½λΣⱼwⱼ²

On solving the above equation and differentiating with respect to w, it gives the optimal w value:

wⱼ* = − [Σᵢ∈Iⱼ gᵢ] / [Σᵢ∈Iⱼ hᵢ + λ]

where Iⱼ represents the instances that fall into leaf j.

Split Finding

The objective function can be written as:

L⁽ᵗ⁾ = Σⱼ [ (Σᵢ∈Iⱼ gᵢ)wⱼ + ½(Σᵢ∈Iⱼ hᵢ + λ)wⱼ² ] + γT

Putting the optimal wⱼ back into the equation:

L⁽ᵗ⁾(q) = −½ Σⱼ [ (Σᵢ∈Iⱼ gᵢ)² / (Σᵢ∈Iⱼ hᵢ + λ) ] + γT

This is a scoring function to measure the quality of a tree, like Gini impurity for a Decision Tree.

Since a tree has too many possible structures which are impossible to enumerate, we use a greedy approach.

Assume Iᴸ and Iᴿ are the left and right instances of a tree.

The split gain can be calculated as:

Gain = ½ [ Gᴸ²/(Hᴸ+λ) + Gᴿ²/(Hᴿ+λ) − G²/(H+λ) ] − γ

where:

Gᴸ = Σᵢ∈Iᴸ gᵢ

Hᴸ = Σᵢ∈Iᴸ hᵢ

Gᴿ = Σᵢ∈Iᴿ gᵢ

Hᴿ = Σᵢ∈Iᴿ hᵢ

G = Gᴸ + Gᴿ

H = Hᴸ + Hᴿ

It is used in practice for evaluating split candidates.

Greedy Algorithm

Input: I, instance set of current node

Input: d, feature dimension

First calculate:

G = Σᵢ∈I gᵢ

H = Σᵢ∈I hᵢ

Then sort the instances according to the feature value.

For every possible split:

Gᴸ ← Gᴸ + gⱼ

Hᴸ ← Hᴸ + hⱼ

The right-side statistics can then be obtained as:

Gᴿ ← G − Gᴸ

Hᴿ ← H − Hᴸ

Then calculate the gain for the split and keep the maximum:

Score = max(Score, Gain)

Output: Split with maximum score.

The above approach is greedy, which is good only for small datasets as it has to try all possible split points.

The approximate approach comes into the picture here.

Approximate Split Finding

The second-order objective can be rewritten as:

L⁽ᵗ⁾ = Σᵢ ½hᵢ [ fₜ(xᵢ) − gᵢ/hᵢ ]² + Ω(fₜ) + constant

This has the form of a weighted squared-loss problem.

Here:

gᵢ/hᵢ → acts like the target

hᵢ → acts as the weight

Therefore, XGBoost uses hᵢ as the weight when constructing weighted quantiles.

Weighted Quantile Sketch

For a particular feature k, we can represent the feature values and their corresponding Hessians as:

Dₖ = {(x₁ₖ,h₁), (x₂ₖ,h₂), ..., (xₙₖ,hₙ)}

Here:

xᵢₖ = value of feature k for instance i

hᵢ = second-order gradient/Hessian of instance i

XGBoost defines a weighted rank function:

rₖ(z) = [Σ₍ₓ,ₕ₎∈Dₖ, x<z h] / [Σ₍ₓ,ₕ₎∈Dₖ h]

In simple words, this tells us:

"What fraction of the total weight lies below a certain feature value z?"

Since XGBoost wants consecutive candidate split points to be reasonably close, ε is used as an approximation factor.

The candidate split points satisfy:

|rₖ(sₖ,ⱼ) − rₖ(sₖ,ⱼ₊₁)| < ε

Here, ε controls how finely the weighted distribution is approximated.

Approximately:

1/ε → number of candidate regions

For example:

ε = 0.1

1/ε = 10

So there are roughly 10 candidate regions.

Similarly:

ε = 0.01

1/ε = 100

So there are roughly 100 candidate regions.

A smaller ε gives a finer approximation and therefore more candidate split points.

The weighted quantile sketch allows XGBoost to generate a manageable number of candidate split points instead of checking every possible feature value.

Handling Missing Values

XGBoost also handles null/missing values by assigning a default direction to each branch.

During training, XGBoost considers where missing values should go and chooses the default direction that gives the better split score.

Therefore, when a feature value is missing during prediction, the instance is sent in the learned default direction.

Example: XGBoost with Numerical and Categorical Features

Let's understand the above concepts with a small example.

Suppose we want to predict whether a customer will buy a product.

1 → Customer buys

0 → Customer does not buy

Our dataset contains one numerical feature, Age, and one categorical feature, Device.

Customer	Age	Device	Actual y
A	28	Mobile	0
B	25	Mobile	0
C	30	Laptop	1
D	35	Laptop	1
E	40	Mobile	1
F	45	Laptop	1

Step 1: Initial Prediction

For simplicity, assume XGBoost starts with:

ŷᵢ = 0.5

for every customer.

Using squared error:

l(ŷ,y) = ½(ŷ − y)²

The gradient is:

gᵢ = ∂l / ∂ŷᵢ = ŷᵢ − yᵢ

and the Hessian is:

hᵢ = ∂²l / ∂ŷᵢ² = 1

Therefore:

Customer	y	Prediction	Gradient gᵢ	Hessian hᵢ
A	0	0.5	0.5	1
B	0	0.5	0.5	1
C	1	0.5	-0.5	1
D	1	0.5	-0.5	1
E	1	0.5	-0.5	1
F	1	0.5	-0.5	1

The positive gradient for A and B tells us that their predictions need to move down, while the negative gradient for C, D, E and F tells us that their predictions need to move up.

Step 2: Try a Numerical Split

For the numerical feature Age, XGBoost can consider different split points.

For example:

Age < 27.5

This creates two groups.

Left:

A, B

Right:

C, D, E, F

For the left side:

Gᴸ = 0.5 + 0.5 = 1

Hᴸ = 1 + 1 = 2

For the right side:

Gᴿ = -0.5 - 0.5 - 0.5 - 0.5 = -2

Hᴿ = 4

The optimal leaf weight is:

wⱼ* = −Gⱼ / (Hⱼ + λ)

Assuming:

λ = 0

For the left leaf:

wᴸ = −1/2 = -0.5

For the right leaf:

wᴿ = −(-2)/4 = 0.5

So the tree can be represented as:


text
              Age < 27.5?
              /          \
           Yes            No
            |              |
          -0.5            +0.5