DEV Community: transformers

Part 2: Comparing AI Frameworks for Natural Language Processing: Transformers and Recurrent Neural Networks

Vijay Vinoth — Wed, 12 Aug 2026 04:17:13 +0000

Part 2: Comparing AI Frameworks for Natural Language Processing: Transformers and Recurrent Neural Networks

As a Lead Programmer Analyst with expertise in PHP, Perl, Python, and Shell, I have been following the advancements in Natural Language Processing (NLP) with great interest. Based on my technical understanding, the choice of AI framework for NLP tasks can significantly impact the performance and efficiency of the system. In this article, we will delve into the comparison of two popular AI frameworks for NLP: Transformers and Recurrent Neural Networks (RNNs).

Introduction to Transformers and RNNs

Transformers and RNNs are both types of neural networks used for NLP tasks, but they differ significantly in their architecture and approach. RNNs are a type of neural network designed to handle sequential data, such as text or speech. They process the input sequence one step at a time, using the previous steps to inform the current step. This makes RNNs well-suited for tasks like language modeling, machine translation, and text classification.

Transformers, on the other hand, are a type of neural network introduced in 2017 that has revolutionized the field of NLP. They are designed to handle long-range dependencies in sequential data, making them particularly well-suited for tasks like machine translation, text summarization, and question answering. Transformers use self-attention mechanisms to process the entire input sequence simultaneously, rather than one step at a time like RNNs.

Comparing Transformers and RNNs

So, how do Transformers and RNNs compare in terms of performance and efficiency? According to a recent study on Transformer vs RNN in NLP: A Comparative Analysis, Transformers outperform RNNs in many NLP tasks, particularly those that require handling long-range dependencies. This is because Transformers can process the entire input sequence simultaneously, using self-attention mechanisms to learn relationships between words and phrases.

In contrast, RNNs process the input sequence one step at a time, using recursion to learn relationships between words and phrases. This can make RNNs slower and more computationally expensive than Transformers, particularly for long documents. As noted in Types of Neural Networks: Complete Guide (2026), Transformers have a 15-40% accuracy advantage over RNNs in long documents.

Another key advantage of Transformers is their ability to parallelize computation, making them much faster to train than RNNs. As noted in RNN vs Transformer: A Deep Dive from Fundamentals to Applications, LSTM processing is inherently sequential, making GPU utilization poor. In contrast, Transformers can take full advantage of GPU acceleration, making them much faster to train.

Real-World Applications

So, how are Transformers and RNNs being used in real-world applications? According to Transformers are Multi-State RNNs | Research - AI at Meta, Transformers are being used in a wide range of NLP applications, including machine translation, text summarization, and question answering. RNNs, on the other hand, are still widely used in many NLP applications, particularly those that require handling sequential data.

Based on my technical understanding as a Lead Programmer Analyst, I believe that the choice of AI framework for NLP tasks will depend on the specific requirements of the project. While Transformers offer many advantages over RNNs, including faster training times and better performance on long-range dependencies, RNNs are still a good choice for many NLP tasks, particularly those that require handling sequential data.

Conclusion

In conclusion, the choice of AI framework for NLP tasks is a critical decision that can significantly impact the performance and efficiency of the system. Based on my technical understanding, I believe that Transformers offer many advantages over RNNs, including faster training times and better performance on long-range dependencies. However, RNNs are still a good choice for many NLP tasks, particularly those that require handling sequential data.

📚 References & Further Reading

For further reading on this topic, I recommend checking out the following resources:
PyTorch Transformer Tutorial
Hugging Face Transformers
Attention Is All You Need
Transformers Are Not That Scary (Part 1)

Your Turn

As we continue to explore the possibilities of NLP, I'd love to hear from you: what do you think is the most exciting application of Transformers and RNNs in NLP, and how do you see these technologies evolving in the future? Share your thoughts and let's continue the conversation!

Originally published at https://artificial-inteligence.phptutorial.co.in

Why “IEC 60076 Compliant” Is Not Enough in Transformer Specifications

qing zhang — Wed, 12 Aug 2026 03:23:44 +0000

In many industrial electrical projects, transformer specifications often include one short sentence:

“The transformer shall comply with IEC 60076.”

At first, this looks clear. IEC 60076 is one of the most widely used international standard series for power transformers, so referencing it is a reasonable starting point.

But in real projects, this sentence alone is often not enough.

The reason is simple: IEC 60076 is not just one requirement. It is a family of standards covering different transformer topics, including general requirements, temperature rise, insulation levels, short-circuit withstand, loading, sound level, and dry-type transformer requirements.

For engineers reviewing transformer quotations, the important question is not only:

“Does it comply with IEC 60076?”

The better question is:

“Which parts of IEC 60076 apply to this project, and how are they reflected in the quotation, test plan, and final documentation?”

Common gaps in transformer specifications

Here are some details that are easy to miss:

1. Temperature rise basis

The quotation should clearly define the cooling method, ambient condition, reference temperature, and guaranteed temperature-rise values.

2. Insulation level

The rated insulation level should match the system voltage, overvoltage condition, and project requirements.

3. Short-circuit withstand

For industrial and utility applications, short-circuit performance should not be treated as a vague assumption.

4. Routine, type, and special tests

A project should define which tests are required before shipment, whether FAT witness is needed, and what documents must be provided.

5. Guaranteed particulars

Losses, impedance, sound level, temperature rise, and other guaranteed values should be clearly stated and traceable.

Why this matters

A transformer is expected to operate for many years. Many problems during procurement or commissioning do not come from the transformer concept itself, but from unclear technical requirements before ordering.

A well-prepared IEC 60076 review helps engineers compare suppliers more fairly and reduces disagreement during factory acceptance testing.

I found this practical IEC 60076 transformer guide useful because it breaks the standard review into buyer-focused checkpoints instead of treating IEC compliance as just a checkbox.

For transformer projects, “IEC 60076 compliant” should be the beginning of the technical review, not the end of it.

From Sparse Attention to MoE: Why K3's 896 Experts Only Activate 1.8% Per Token

AICDragon — Tue, 11 Aug 2026 03:16:11 +0000

From Sparse Attention to MoE: Why K3's 896 Experts Only Activate 1.8% Per Token

This is the third installment in a series on the evolution of attention mechanisms.

Part 1 covered the sequence dimension: from GPT-2's full attention to KV Cache, linear attention, and KDA — seven years of progress.
Part 2 covered the inter-layer dimension: from simple residual summation to Attention Residuals and Depth-Attention — selective cross-layer communication at near-zero cost.

This article explores the third dimension: orthogonal computation — when Transformers face the demand of "make the model bigger without making inference slower," the answer is "most parameters stay idle." This is the story from sparse attention to MoE to Stable LatentMoE. Let's begin with one number: K3 has 896 experts, but at inference time each token only activates 16 — that's 1.8%.

1. The Origin of the Problem: The Curse of All-or-Nothing

Standard Transformers operate on an implicit assumption: every token at every layer must pass through all parameters.

Layer 0: Token -> (Attention + FFN) -> hidden state
Layer 1: Token -> (Attention + FFN) -> ...
...
Layer 92: Token -> (Attention + FFN) -> predict next token

Every layer's FFN processes every token. "Answer a primary-school math problem" and "write operating system kernel code" engage entirely different cognitive circuits — yet the Transformer is forced to use the same parameters for both.

This creates two fatal consequences:

Parameter count is constrained. More parameters means more operations per inference — compute cost scales linearly. Want a smarter model? Adding parameters = adding cost.
Wasted compute. The simplest token and the hardest token consume exactly the same FLOPs.

MoE's solution is brutally effective: prepare multiple sets of "cognitive circuits" (experts), and have each token call only a small subset.

Sounds like common sense. But turning that common sense into an industrially stable, inference-efficient model took several years.

2. Three Cards in MoE's Evolution

Card 1: Switch Transformer (2021, Google) — Moving MoE from Lab to Factory

MoE itself is an old idea (Jacobs et al., 1991), but before the LLM era it was stuck in the lab due to training instability.

Switch Transformer (Fedus et al., JMLR 2022) did two crucial things:

Solved large-scale training stability. Used capacity factor to cap tokens per expert, and auxiliary loss to balance routing load.
Trained a 1.6T-parameter model on C4, activating far fewer parameters at inference than the total count.

But Switch used Top-1 routing — each token goes to exactly one expert. The advantage is extreme sparsity (minimal activation rate). The problem is obvious: if that one expert fails (overloaded or dropped), the token produces no output — there's no backup.

Card 2: Mixtral 8x7B (2023, Mistral) — Top-2 Becomes the Industry Standard

Mistral's Mixtral 8x7B, released in late 2023, used Top-2 routing — each token activates 2 out of 8 experts.

This marked a key design philosophy shift. Top-1 is "all eggs in one basket." Top-2 begins to accept "spend a bit more compute for significantly better stability and performance." The 8-expert configuration is also clean enough to run on consumer-grade GPUs.

Mixtral, at 46.7B total parameters (12.9B active), matched or approached comparable dense models on multiple benchmarks while using substantially less compute. This proved that MoE isn't just for gigantic models — it pays off at medium scale too.

Card 3: DeepSeek-MoE (2024, DeepSeek) — Fine-Grained Experts + Shared Experts

DeepSeek-MoE (ACL 2024) introduced two of the most important architectural innovations in MoE history:

Fine-grained expert segmentation: Split N experts into mN smaller experts, activating mK per token. This enables more flexible combinations — instead of a few large experts handling broad knowledge domains, many small experts perform precise assembly.
Shared experts: Isolate K_s "shared experts" from the total expert pool, always activated. These handle foundational knowledge every token needs (grammar structure, common phrasing, basic reasoning frameworks), while routed experts focus on domain-differentiated knowledge.

This design solves MoE's biggest contradiction: "knowledge redundancy."

In early MoE, every expert could learn the same basics (e.g., "the" can be followed by a noun) because routing selects experts based on token features, and all experts see these high-frequency patterns. Isn't that just wasting parameters?

The shared expert idea: "Extract that shared part, let everyone use it, stop routed experts from redundantly learning fundamentals."

DeepSeek-MoE 16B achieved LLaMA2 7B performance at roughly 40% of the compute. DeepSeek-MoE 145B matched DeepSeek 67B using only 28.5% of the compute. This was the first paper to clearly demonstrate that the "fine-grained + shared expert" dual-track architecture has overwhelming advantages in compute efficiency.

This is also the direct predecessor of K3's MoE architecture.

3. K3's Stable LatentMoE: Pushing DeepSeek-MoE's Three Paths to the Limit

K3's technical report describes the Stable LatentMoE design in detail. Building on DeepSeek-MoE, it pushes three paths:

Path 1: Expert Count Jumps from 64 to 896

DeepSeek-MoE already made experts smaller and more numerous, but K3 pushes this to another order of magnitude.

DeepSeek-MoE 16B:   ~64 routing experts + shared experts
K3:                  896 routing experts + 2 shared experts

The key isn't "896 is a lot" — it's what 896 implies about how small each expert is.

Let's compute. K3 has 896 routing experts per layer, hidden dim = 7168, latent dim (the compressed dimension before entering routing experts) = 3584, each expert's intermediate dim = 3072.

One expert = two weight matrices (3584->3072 + 3072->3584) = roughly 22M parameters.

K3 has 2.78T parameters total. 896 experts x 93 layers = roughly 83,328 experts. At 22M each, the expert portion is... far below 2.78T. Because experts are just FFN replacements — the total parameter count includes embeddings, all layers' attention weights, shared experts, and other larger components.

But the core logic holds: when experts become small enough, routing ceases to be semantic assignment ("bring in the Python expert") and becomes feature assembly from a pool of fragments.

This leads to K3's most crucial design philosophy, and the most insightful analysis from the lilting channel —

The Router Is Not a "Classifier" — It's a "Compute Allocator"

The standard MoE understanding: "the router classifies each token to a domain expert." This understanding is wrong — at least for K3 with 896 experts.

K3's router is a single line:

scores = Sigmoid(W_r x x)   # W_r: 896 x 7168, x: hidden state
# Take Top-16, weighted combination by normalized scores

A linear projection -> Sigmoid -> Top-16. No complex operations whatsoever.

But this works not because the router is "smart," but because x itself has already been thoroughly processed by preceding layers' Attention and FFN. Just as BERT hidden states are good enough for a simple linear classifier to perform NLU tasks — K3's hidden state, after passing through dozens of layers before reaching the router, is already a highly structured representation space. A single linear projection is sufficient for the final "routing decision."

From another angle: with 896 experts, expecting a simple linear classifier to correctly assign each token to "exactly the right 16 experts" is clearly over-idealized. The router's real job isn't "find the best 16" — it's "under load-balancing constraints, give most tokens a reasonable set of 16."

This is why K3 invested massive engineering effort in load balancing —

Quantile Balancing: The Load-Balancing Challenge of 896 Experts

MoE routing training has a classic dilemma:

Don't intervene in routing -> tokens flood a few experts -> GPU load imbalance -> some experts starve (insufficient training data)
Force balance -> same input routed to different experts across training steps -> experts learn highly similar things -> pseudo-balance

K3's solution is Quantile Balancing: at each training step, compute quantile thresholds from the router score distribution — experts above this threshold receive a target number of tokens. Then update per-expert selection bias based on deviation from the target.

Crucially: this bias is only used during training. Fixed at inference.

This means K3 has already gained complete experience during training on "how to achieve load balance among 896 choices," and at inference it simply uses a fixed bias to prevent catastrophic load skew.

Path 2: Latent Compression + 2 Shared Experts

Another key design in Stable LatentMoE: before entering routing experts, compress the 7168-dimensional hidden state into a 3584-dimensional latent space.

x (7168) -> W_down -> z (3584) -> 16 routing experts -> u (3584) -> W_up -> RMSNorm -> output

Why? Two reasons:

Reduces per-expert parameter count and compute (input dimension halved)
Reduces the information dimension the router needs to handle

But compression loses information. So K3 retains 2 always-activated shared experts — they operate on the full 7168-dimensional hidden space, without latent compression.

Final output:

y = E1_shared(x) + E2_shared(x) + W_up(RMSNorm(u))

The 2 shared experts handle "foundational features everyone needs." The 16 routed experts handle "differentiated features specific to each token." Clean division of labor.

Path 3: 1.8% Activation Rate — It's Not a Bug, It's a Feature

Back to that number: 896 experts, only 16 activated. 1.8%.

Why not more? Say, 32 (3.6%)?

K3's technical report doesn't directly provide this ablation, but we can infer from the architecture:

Latent space capacity bottleneck. Latent dim = 3584. 16 experts already approach this dimension's information saturation. Adding more experts -> additional FFN capacity can't be effectively transmitted through the latent bottleneck -> diminishing marginal returns.
Cross-GPU communication cost. 896 experts are distributed across multiple GPUs. Each additional activated expert means more all-to-all communication across GPUs. K3's official docs explicitly recommend 64+ accelerators for deployment — when you're already doing all-to-all on a cluster that large, each extra expert adds real latency.
DeepSeek-MoE already validated: fine-grained + fewer activations > coarse-grained + more activations. Rather than activating 40 experts each learning a little, let 16 small experts each learn their part well.

1.8% isn't a design target — it's the optimal convergence point shaped jointly by the latent bottleneck, communication costs, and expert granularity.

4. The Evolution at a Glance: From Dense to 896-Expert

Year    Model                  Routing Strategy    Experts/Layer    Active    Shared Experts

2021    Switch Transformer     Top-1               64-2048          1         None
2023    Mixtral 8x7B           Top-2               8                2         None
2024    DeepSeek-MoE 16B       Fine-grained Top-K  ~64              ~6        K_s
2024    DeepSeek-MoE 145B      Fine-grained Top-K  ~192             ~18       K_s
2026    Kimi K3                Quantile Top-16     896              16        2

Clear trends:

Expert count is increasing, but active count stays in a stable range (1-16) — not because 16 is a magic number, but because the latent bottleneck and communication costs impose an invisible ceiling
Shared experts are a watershed — nobody did them before DeepSeek; K3 directly inherited the approach
Load balancing evolved from "aux loss assist" to "quantile direct control" — control precision went from "penalize imbalance" to "direct allocation"
Each expert is getting smaller — from a few large experts to hundreds of small ones, routing's meaning shifts from "domain classification" to "feature assembly"

5. K3 MoE's Real-World Performance

Data extracted from the official blog and technical report:

Total params: 2.78T, active params: 104.2B (roughly 3.7%)
Inference efficiency: Despite the routing overhead of 896 experts, K3's team claims 90%+ cache hit rate (in coding scenarios) through the Mooncake decoupled inference architecture, serving million-token contexts at $0.30/M tokens (cache-hit pricing)
Training efficiency: Introduced fully balanced expert-parallel training with static shapes and no host-synchronization critical path, making 896-expert scale training feasible

More importantly, the synergy between MoE and other innovations: KDA (from Part 1) + Attention Residuals (mentioned in Part 1) + Stable LatentMoE (the protagonist of this article) -> Kimi K3 achieves roughly 2.5x overall scaling efficiency improvement over K2.

This isn't one component's magic — it's three orthogonal optimization dimensions stacked together: cheaper long contexts (KDA), smarter inter-layer communication (AttnRes), and compute concentrated on the most needed parameters (LatentMoE).

6. What's Next: MoE on the Road Ahead

K3 already does conditional computation at the FFN layer — each token uses a different FFN expert combination. This is just the first step:

Current K3 is still a 93-layer fixed-depth Transformer — every token, regardless of difficulty, must traverse all 93 layers. Possible evolutionary directions:

Dynamic depth: Simple tokens exit early, difficult tokens go deeper (ADEPT, 2026)
MoE-ified Attention: Not all tokens need full attention — SALSA uses a router to decide between attention and recurrent (2026)
Layer-level MoE: Different layers may have different expert architectures — input-adjacent layers use shared-type, output-adjacent layers use routing-type
From FFN routing to full-module routing: Not just choosing experts, but choosing "should this layer use attention, SSM, or skip" — turning the model into a token-level dynamic computation graph

If MoE's essence is "most parameters stay idle," these directions' essence is "most computation stays idle." The engineering implementations are entirely different, but the philosophy shares the same origin.

7. What This Means for You

Three "Don'ts":

Don't understand MoE through "domain classification." Thinking in terms of "math expert" and "code expert" misses the essence. Experts are FFN fragments — what the router allocates isn't semantic roles but compute resources. This understanding directly determines whether you can design efficient routing strategies.
Don't equate "more experts" with "better." K3 uses 896 not because it "needs 896," but because under the constraints of fine granularity + latent compression + load balancing, 896 is the optimal tradeoff for training and inference efficiency. Blindly adding experts -> load balancing collapse -> pseudo-balance -> wasted parameters.
Don't treat MoE as an isolated module. K3's 2.5x scaling efficiency improvement comes from the synergy of KDA + AttnRes + LatentMoE. The relationship between MoE and attention mechanisms isn't "this replaces that" — it's "simultaneous optimization across multiple dimensions": horizontal (sequence attention compression), vertical (inter-layer information flow), and orthogonal (selective parameter activation).

Three Numbers to Remember:

1.8% — K3's FFN activation rate per token. The current sparsity-efficiency ceiling for the strongest open-source model.
28.5% — The compute fraction DeepSeek-MoE 145B needs to match DeepSeek 67B Dense. The compute-efficiency baseline for fine-grained MoE.
90% — K3 API's cache hit rate in coding scenarios. Without this level of cache optimization, a 2.78T model could never achieve $0.30/M token pricing.

References

Kimi K3: Open Frontier Intelligence — Kimi Team, July 2026, technical report + open weights
Kimi K3 Tech Blog — Moonshot AI official blog
K3's router picks 16 of 896 experts: an allocator, not a classifier — lilting channel, deep architecture analysis
DeepSeekMoE: Towards Ultimate Expert Specialization — Damai Dai et al., ACL 2024
Switch Transformers: Scaling to Trillion Parameter Models — Fedus et al., JMLR 2022
Mixtral of Experts — Jiang et al., Mistral AI, 2024
The Illusion of Specialization — ACL 2026, analysis of actual MoE expert behavior

This is Part 3 of the "Evolution of Attention Mechanisms" series and the first installment on the "orthogonal dimension." Together, the three parts form a complete coordinate system: horizontal (sequence attention) -> vertical (inter-layer communication) -> orthogonal (selective parameter activation). Coming next: if MoE decouples "parameter count" from "inference cost," what's the next dimension to be decoupled?

Depth-Attention: Opening an Attention Channel Between Transformer Layers

AICDragon — Mon, 10 Aug 2026 12:50:49 +0000

This is the second article in my Attention Mechanism Evolution series. The first covered the horizontal (sequence-dimension) attention optimizations — from GPT-2's full attention to Kimi K3's KDA hybrid architecture. That was one axis. This article tackles the vertical axis: can deeper layers selectively attend to shallower layers, instead of just blindly summing via residual connections?

Depth-Attention, proposed by Shanghai Jiao Tong University's LUMIA Lab (arXiv: 2606.05014, accepted at ICML 2026), answers with an elegant design: zero new parameters, zero additional KV cache, under 0.01% extra FLOPs — and a consistent +2.3 point accuracy boost on downstream tasks.

1. The Problem: Transformers Are "Blind" Between Layers

Standard self-attention excels at freely selecting information along the sequence dimension — each token can attend to any position in the sequence.

But switch to another dimension — between layers — and Transformers become startlingly inefficient:

Layer ℓ-1 output → (residual connection: simple summation) → Layer ℓ input

Each residual block just adds the previous layer's output on top of its own — no weighting, no selectivity, no attention. The deeper layers face a hidden state that's just a compressed dump of all prior layer outputs.

Think of it like this: you write a report and hand each page to a colleague, but they only see the last page. They can't tell whether "page 3's analysis is your best work — read more of it" or "page 15 isn't that important — just skim it."

This isn't a new problem. Recent work has tried to solve it — DenseFormer (learns fixed inter-layer weights for weighted averaging), Hyper-Connections/mHC (multiple parallel residual streams replacing single residual), and Attention Residuals (using softmax attention to adaptively select shallow-layer outputs per token — this is what Kimi K3 uses).

But these approaches share a problem

They all operate on hidden states — the model's full intermediate representations, outside the self-attention module itself. This means they need to retain or access these hidden states in addition to the KV cache during inference.

Meanwhile, the trend in large models (GQA, MLA) is to aggressively compress the KV cache — DeepSeek uses MLA to compress from 128×d down to 4×d. Adding extra persistent states outside the KV cache? That's swimming against the current.

Depth-Attention's insight is exactly this contradiction: can we leverage the Q, K, V already inside the attention module, in the same place, without any external state, to achieve cross-layer selective mixing?

2. How Depth-Attention Works: Rotating "Sequence Attention" by 90 Degrees

The answer is remarkably simple.

Standard self-attention operates on the sequence dimension — "the current token attends to all tokens in the sequence."

Depth-Attention rotates this operation 90 degrees — performing the exact same thing on the depth dimension:

For layer ℓ, token t:
q_ℓ^t · k_j^t  (current layer query × each shallower layer's key, at the same token position)
  → softmax → depth attention weights α
  → weighted mixture ṽ (depth-mixed value)

The final step is key — everything in self-attention stays unchanged (no Q modification, no K modification, no mask change), only the value is replaced with the depth-mixed version:

O_ℓ = CausalAttn(Q_ℓ, K_ℓ, Ṽ_ℓ)   # Ṽ_ℓ replaces the original V_ℓ

Three elegant properties:

1. Zero new parameters. Depth-Attention fully reuses the Q and K projection matrices already in standard self-attention — the same Q serves both sequence-direction and depth-direction attention. No new parameter matrices at all.

2. Zero additional KV cache. The depth-mixed Ṽ has exactly the same shape as the original V (T×d). During inference, it replaces rather than appends — Ṽ goes into the original V cache slot. Subsequent tokens reading that slot automatically get the depth-mixed version — so the persistent state at inference is identical to a vanilla decoder's KV cache. Not a single byte extra.

3. Recursive information propagation — the paper's most ingenious design. Let me write out the formula clearly:

ṽ_ℓ^t = α_ℓ,ℓ · v_ℓ^t  (current layer's own value)
       + Σ_{j<ℓ} α_ℓ,j · ṽ_j^t  (previous layers' already-mixed values)

Notice the second term: the mixture uses not the raw v_j, but the already depth-mixed ṽ_j. This means information propagates cascadingly — ṽ_4 contains all mixed info from ṽ_0 through ṽ_3. When ṽ_16 reads from ṽ_4, it gets not just layer 4's raw value, but depth-mixed information already processed by layer 4.

If it mixed raw v_j, layer 10 could only directly access layer 2's value — with 8 layers of residual stream attenuation in between, the signal is highly degraded. But mixing ṽ_j means: layer 6 reads layer 4 → layer 4's ṽ already contains layer 0's info → layer 10 reads layer 6 → it also indirectly gains contributions from layer 0 and layer 2.

A recursive structure achieves efficient propagation. This allows shallow representations to penetrate through many intermediate layers and continue influencing deep layers.

3. Counter-Intuitive Ablation: More Layers Isn't Always Better

If every layer attended to all shallower layers, the compute cost would be O(TL²) — non-trivial in deep networks, especially with pipeline parallelism's cross-device communication overhead.

Depth-Attention addresses this with strided sampling: each layer only attends a sparse subset — itself + {0, s, 2s, ...} among the shallower layers.

With s=4, layer 20 attends {20, 16, 12, 8, 4, 0}, skipping the 14 layers in between.

Complexity drops to O(TL/s); with s large enough, it approaches O(T), completely negligible.

But the most counter-intuitive result is the ablation study: the paper tested different strides. The result was not "more is better." s = L/2 (half the total layers) performed best — better than denser s=L/4 and sparser s=L.

Why? The paper doesn't give a definitive explanation, but I think this phenomenon hints at an important principle: there may be an optimal "reception interval" for information propagation across depth — too dense creates redundancy (adjacent layers' values are highly correlated), too sparse loses critical information. s=L/2 happens to provide enough "diversity windows" without falling into the trap where "layer 20's info is highly redundant with neighboring layers 16 and 14."

This echoes the same design philosophy we saw in part one with Kimi K3's KDA hybrid architecture: it's not about stacking more — it's about choosing more wisely where to apply it.

4. GQA Compatibility: Naturally, Freely More Efficient

Modern large models widely use Grouped-Query Attention (GQA), where g query heads share one set of KV heads.

Depth-Attention handles GQA extremely naturally: average the g queries within each group, running depth attention at KV head resolution — not query head resolution.

This brings two additional benefits:

KV head dimension is already several times smaller than hidden size (typically 4x) — running depth attention in this space further shrinks compute and memory
No head-dimension expansion or alignment needed — just average g queries, zero hyperparameter changes

In other words: Depth-Attention is not only overhead-free in GQA scenarios, it's actually more efficient than in non-GQA scenarios — because depth attention's operating space is naturally compressed by GQA.

5. Experimental Results: What's the Actual Gain?

The paper ran experiments on Qwen3-style decoder architectures at 1.5B and 3B scales. All models were trained from scratch on 32B tokens from the Pile, with identical data and hyperparameters — a fair comparison.

Baselines included: Vanilla Transformer, mHC (manifold hyper-connections), Attention Residuals (Kimi K3's approach), and DenseFormer.

Key results

At 1.5B scale, zero-shot:

Vanilla Transformer average downstream accuracy: 51.26
Depth-Attention: 53.56 (+2.3 points)
Attention Residuals and mHC fall between Vanilla and Depth-Attention

At 3B scale, the gap widens — Depth-Attention reduces perplexity to 6.66 (Vanilla: 7.10), and average accuracy improves from 53.08 to 55.27 (+2.19 points), comprehensively beating all baselines.

Under 5-shot settings, the pattern holds — Depth-Attention is best at both scales.

Notably: Attention Residuals, as the strongest baseline, does significantly beat Vanilla — the paper acknowledges this clearly — but Depth-Attention surpasses it on every metric, and does so without adding any inference state.

Efficiency: theory vs. measurements

Theoretical FLOPs: The paper's Appendix B provides detailed derivations of extra FLOPs for each method. Counting per decoder layer:

Depth-Attention extra compute: ≈ 2Td/s FLOPs (s = stride, d = head dim)
DenseFormer extra compute: ≈ T·d_model·L (weighted summation across all layers)
Attention Residuals extra compute: ≈ T·d_model·L (similar magnitude)

Plugging in typical values (T=4096, d_model=2048, L=32, s=16), Depth-Attention's extra FLOPs are less than 0.01% of self-attention FLOPs.

Training wall-clock: On a 3B model, Depth-Attention adds only ~1% per-step training time — significantly less than DenseFormer and Attention Residuals.

Inference throughput: At 128K token prefill, Depth-Attention throughput is essentially identical to Vanilla (<0.5% difference). Hidden-state methods, by contrast, need explicit extra state management in long-context scenarios, with noticeably higher memory usage.

Scaling experiments

The paper tested four scales from 360M to 3B. Depth-Attention maintains its advantage at all scales, with gains showing no sign of saturation. This suggests larger models may also benefit — though this hasn't been verified at 70B+ scale (a limitation the paper acknowledges).

6. Weight Visualization: What Did the Model Learn?

The paper visualizes the trained Depth-Attention weight distributions, with several interesting findings:

1. Early layers focus their depth attention on deeper information sources, not shallower ones.

Concretely: Layers 1-8 (early layers) have self-attention weights dominating in depth attention (α_ℓ,ℓ close to 1) — they don't look much at shallower layers, because there simply aren't enough shallower layers.

But layers 9-32 (the second half) start dispersing their depth attention weights: deep layers still retain large self-attention weights (preserving their own features) but radiate uniformly toward middle layers.

2. Shallower isn't necessarily more important.

Intuition might suggest "shallow features are more fundamental, deep layers should attend more to them." But from the weight plots, deep layers (e.g., layer 30) don't assign higher attention weights to very shallow layers (layer 0, 4) than to middle layers (layer 12, 16).

This hints at an interesting learned strategy: the model learns to extract information at different granularities from different depth levels, rather than simply treating shallow layers as an "information repository."

3. Some attention heads are more cross-layer-dependent than others.

Different heads show different degrees of dependence on depth information. Some heads have self-attention weights close to 1 at almost all layers (barely looking at other layers), while others have dispersed weights across the entire depth dimension (actively leveraging cross-layer information). The model thus gains flexibility — use "self-attention-heavy" heads when local fine-grained processing is needed, and "cross-layer-heavy" heads when contextual fusion is needed.

7. Looped Transformer Experiment: Parameter Sharing Works Too

This is one of the paper's most profound experiments. A Looped Transformer has only a few physical layers (e.g., 4 layers) that execute in a loop (e.g., 8 cycles = 32 total effective layers), with all cycles sharing the same parameters.

Depth-Attention's gains persist in this setting — each loop cycle acts as a new "logical layer" that can attend to the values from its own previous cycles.

This result rules out a possible explanation: "Depth-Attention works because different layers have different parameters, so their value semantics differ, making them worth retrieving." In the parameter-sharing scenario, values from different layers are still selectively passed through depth attention — proving that the mechanism's benefit doesn't depend on "different parameters per layer," but rather that selective cross-depth information transfer is inherently valuable.

Taking it deeper: this result hints at a potential relationship between Depth-Attention and recurrence/SSMs. If parameter sharing works, what Depth-Attention is doing is somewhat like a soft state-space model — selectively remembering and retrieving depth-history information in the value dimension, rather than propagating through fixed residuals.

8. Depth-Attention vs. Kimi K3's Attention Residuals: A Design Philosophy Divergence

This is the most fascinating comparison — they solve the same problem but choose different positions.

	Attention Residuals	Depth-Attention
Operation point	On residual stream (outside module)	Inside attention (value position)
Target	Hidden states	V cache slots
Extra inference state	Required (retain shallow hidden)	None
Extra parameters	Yes	Zero
Information granularity	Full hidden state (rich)	Value state (compressed)
Performance (paper comparison)	Better than vanilla	Better (at 1.5B/3B)

Both share the same conceptual origin — enabling deeper layers to selectively attend to shallower ones. The difference is in the tradeoff:

Attention Residuals operates at the most information-rich position (hidden state), but at a cost
Depth-Attention operates at the cheapest position (V cache), at zero additional cost

This divergence doesn't have a final answer yet: who wins at 70B+ scale? Nobody knows. But one thing is clear — Depth-Attention found a "free lunch" path. It proved that you don't need to add anything outside the KV cache to get all the benefits of cross-layer selective information transfer.

This isn't a free lunch — it's the lunch you were already eating, you just didn't realize you could eat it better.

9. What This Means

Architecture evolution: one horizontal, one vertical

The first article covered sequence-direction (horizontal) attention evolution from full attention to KDA — that was "how to remember more information with less state."

This article completes the inter-layer (vertical) direction — "how to let deep layers selectively use shallow representations without paying extra."

Put together:

      ← Sequence direction (horizontal) →
      Full Attn → KV Cache → Linear Attn → DeltaNet → KDA
      (compute → memory → compression → precision → hybrid recipe)

   ↑
   Inter-layer direction (vertical)
   Residual → DenseFormer → Attention Residuals → Depth-Attention
   (sum → weighted → attention → zero-cost attention)
   ↓

For model users:

Next time you see a model claiming "cheap long context," don't just look at parameter count and context window — ask "what's your inter-layer communication scheme? How does information flow in the depth direction?"
Kimi K3 uses Attention Residuals; the next version might switch to Depth-Attention or a variant — keep an eye on this thread

For model trainers:

Depth-Attention's zero-parameter nature means a lossless drop-in replacement: no weight changes, no cache changes, no pipeline changes — just modify a few lines of attention code and reliably gain ~2 points
The paper's code is open-source on GitHub, with a pre-trained 3B checkpoint on HuggingFace for verification

For technical investors:

The design philosophy divergence (hidden state vs. value state path) could influence next-generation model architecture choices
Inter-layer communication is becoming a core optimization dimension alongside "attention compression"

10. Limitations (Frankly)

Limited validation scale. The 1.5B and 3B experiments are convincing (32B tokens trained from scratch, 8 downstream tasks), but whether results replicate at 70B+ scale remains unknown. The paper acknowledges this.
Mechanism explanation for optimal stride is missing. s=L/2 working best is an empirical finding — why this value, and whether better non-uniform sampling strategies exist, leave significant analysis space.
Interaction with other optimizations under-explored. Effects when combined with MoE (e.g., K3's 896 experts), compatibility with MLA, differential behavior in prefill vs. decode phases — all await follow-up work.
Is the "dual attention" structure of depth + sequence optimal? This is an architectural question — perhaps a future unified primitive will fuse both directional attentions into one operation. Depth-Attention is an important step, but not necessarily the last.

References

Depth-Attention: Cross-Layer Value Mixing for Language Models — Boyi Zeng et al., Shanghai Jiao Tong University LUMIA Lab, ICML 2026
Code & Models — GitHub + HuggingFace (3B checkpoint available)
Attention Residuals — Kimi Team, 2026 (cross-layer mechanism used in Kimi K3)
DenseFormer — Pagliardini et al., NeurIPS 2024
Hyper-Connections / mHC — Zhu et al., 2025
GQA — Ainslie et al., EMNLP 2023

This is the second article in the "Attention Mechanism Evolution" series. Part one covered the horizontal (sequence-direction) evolution — from GPT-2's full attention to Kimi K3's KDA hybrid architecture, with cross-validation across 5 papers, a complete evolution table, and code-level analysis. Part three preview: from sparse attention to MoE — why K3's 896 experts only activate 1.8%.

Day 040 — Multi-Head Residual (attention by hand)

Hariom Lohar — Mon, 10 Aug 2026 05:01:36 +0000

Hariom Lohar (hariomlohardev on GitHub) — Lab Notebook No.01 · Day 040 · 10 Aug 2026.

I rebuilt multi-head attention by hand — no PyTorch, just NumPy. From math → NumPy backprop → CNNs → attention.

Live bench + daily logs: https://hariomlohardev.github.io
Full log: https://hariomlohardev.github.io/blog/p/day-040-multi-head-residual/

Original log on my portfolio (canonical) → https://hariomlohardev.github.io/blog/p/day-040-multi-head-residual/ — cross-posted to reach more builders. 548 days: 1 Jul 2026 → 31 Dec 2027.

The KV Cache Is the Bottleneck: A 2026 Field Guide to Attention Variants

ai maya — Sun, 09 Aug 2026 07:15:29 +0000

If you want to understand why 2026's frontier LLMs look the way they do — why one ships Multi-head Latent Attention, another interleaves linear and full attention, a third leans on sliding windows — stop looking at benchmark scores and look at the KV cache. Almost every architectural decision at the frontier this year is, at heart, a fight with the memory that autoregressive decoding leaves behind.

This is a practitioner's map of that fight: what the KV cache actually costs, the lineage of techniques built to shrink it, and how to choose among them.

Why the KV cache dominates

During generation, a decoder-only transformer caches the key and value vectors of every past token so it doesn't recompute them each step. That cache is the price of fast decoding, and it is not small.

The size is brutally simple:

KV bytes = 2 (K and V) × layers × kv_heads × head_dim × seq_len × batch × dtype_bytes

Two consequences fall out immediately:

It grows linearly with context and batch. Double the context, double the cache. Serve more concurrent users, multiply again. At long context and healthy batch sizes, the KV cache — not the weights — becomes the dominant consumer of accelerator memory.
Decoding is memory-bandwidth bound, not compute bound. Generating one token touches the entire cache. You are not FLOP-limited; you are limited by how fast you can stream that cache off HBM. Shrinking the cache is therefore the most direct lever on both memory footprint and tokens/sec.

Every technique below is a different answer to "how do we make that formula smaller without breaking the model."

The lineage of fixes

1. Fewer KV heads: MQA → GQA

The first target is kv_heads. Multi-Query Attention (MQA) collapses all query heads onto a single shared KV head — a large cut, but it can cost quality and training stability. Grouped-Query Attention (GQA) is the pragmatic compromise that won: query heads are split into groups, each sharing one KV head. With, say, 8 KV heads for 64 query heads you get most of MQA's savings with far less quality loss. GQA is the default in the majority of open-weight models today for exactly this reason.

2. Compress the cache: Multi-head Latent Attention (MLA)

Instead of storing fewer heads, MLA stores a low-rank latent. Keys and values are projected down to a small shared latent vector that is cached; the per-head K and V are reconstructed on the fly during attention. The cached object is dramatically smaller than full K/V, yet the model still attends with many effective heads. MLA is the headline reason some 2026 models sustain very long context at serving-friendly memory — it attacks head_dim × kv_heads directly rather than just cutting head count.

3. Kill the growing cache entirely: linear attention & SSMs

Softmax attention is O(n²) in compute and O(n) in cache. Linear attention and state-space models (SSMs, Mamba-style) rewrite the operation as a recurrence with a fixed-size state. There is no cache that grows with sequence length — memory is O(1) in context. The catch is real: a fixed state is a lossy summary, so pure linear models are weaker at precise long-range recall and associative lookup ("what was the exact token 40k ago"). They are fast and cheap; they forget details.

4. The 2026 consensus: hybrid, layer-interleaved attention

The dominant pattern this year is not to pick one. Models interleave a minority of full (softmax) attention layers with a majority of linear/SSM layers. The full layers preserve the sharp recall that pure linear loses; the linear layers carry the long-context load cheaply. The result is near-linear memory scaling with most of the quality of full attention. When you read that a 2026 release is "hybrid attention," this layer-wise interleaving is almost always what is meant.

5. Bound the window: sliding-window attention + sinks

Orthogonal to the above, sliding-window attention simply caps how far back each token attends (e.g., the last few thousand tokens), making the cache constant-size past the window. Naively this destroys long-context behavior, so it is paired with tricks like attention sinks — keeping the first few tokens always in view — which stabilize very long generations. Cheap, effective for many workloads, and easy to combine with GQA.

Two more multipliers worth knowing

Quantized KV cache. The dtype_bytes term is a free-ish win: storing the cache in FP8 or INT8 instead of FP16 halves or quarters it with modest quality impact. In 2026 this is close to standard for long-context serving.
Paged KV memory. PagedAttention (the idea behind vLLM's throughput) doesn't shrink the cache — it stops you from wasting it. By allocating the cache in fixed pages instead of one contiguous per-request block, it removes fragmentation and lets you pack far more concurrent sequences into the same HBM. Architecture shrinks the cache; paging spends what's left efficiently.

How to choose

Building a general open-weight model? GQA is the safe, proven baseline. You will not be criticized for it.
Chasing long context at serving-friendly memory? MLA-style latent compression is the strongest single lever, at the cost of implementation complexity.
Throughput on shorter contexts, or edge deployment? Sliding window + GQA + quantized cache is a cheap, robust stack.
Very long context where some recall loss is acceptable? A hybrid linear/full interleave gives you near-linear scaling — validate recall on your long-context tasks before committing.
Serving any of the above? Quantize the cache and use a paged runtime. These compose with every architecture and are the lowest-effort wins on the list.

The real takeaway

The lesson of 2026 is that model architecture and the serving stack are now co-designed. You cannot reason about an attention variant without reasoning about its KV cache, its memory bandwidth, and how a runtime will page it. The models that feel fast and cheap in production didn't get there by accident — they were shaped, layer by layer, by the cost of remembering.

When you evaluate the next "new architecture," skip the leaderboard for a minute and ask the only question that predicts its serving cost: what happens to the KV cache?

What's your KV strategy in production — GQA, latent compression, hybrid, or just quantize-and-page? Curious what's holding up under real load.

Building Custom AI Models for Sentiment Analysis with Transformers Part 1: Introduction to Transformer Architecture

Vijay Vinoth — Fri, 07 Aug 2026 11:54:40 +0000

Introduction to Building Custom AI Models for Sentiment Analysis with Transformers

The field of artificial intelligence (AI) has witnessed tremendous growth in recent years, with significant advancements in natural language processing (NLP) and machine learning. One of the most influential architectures in NLP is the Transformer model, introduced in the paper "Attention is All You Need" by Vaswani et al. in 2017. Based on my technical understanding as a Lead Programmer Analyst, I can attest that the Transformer architecture has revolutionized the way we approach sequence-to-sequence tasks, including sentiment analysis.

Understanding the Transformer Architecture

The Transformer model is primarily designed for sequence-to-sequence tasks, where the input and output sequences are of varying lengths. This is particularly useful in NLP tasks, such as machine translation, text summarization, and sentiment analysis. The Transformer architecture consists of an encoder and a decoder. The encoder takes in a sequence of tokens (e.g., words or characters) and outputs a sequence of vectors, which are then used by the decoder to generate the output sequence.

The Transformer encoder consists of a stack of identical layers, each comprising two sub-layers: self-attention and feed-forward neural networks. The self-attention mechanism allows the model to attend to different parts of the input sequence simultaneously and weigh their importance. This is particularly useful in sentiment analysis, where the context of a sentence or phrase can significantly impact the sentiment.

The Transformer decoder also consists of a stack of identical layers, each comprising three sub-layers: self-attention, encoder-decoder attention, and feed-forward neural networks. The self-attention mechanism in the decoder allows the model to attend to different parts of the output sequence, while the encoder-decoder attention mechanism allows the model to attend to different parts of the input sequence.

Key Components of the Transformer Architecture

There are several key components of the Transformer architecture that make it particularly well-suited for sentiment analysis:

Self-Attention Mechanism: This mechanism allows the model to attend to different parts of the input sequence simultaneously and weigh their importance. This is particularly useful in sentiment analysis, where the context of a sentence or phrase can significantly impact the sentiment.
Encoder-Decoder Attention Mechanism: This mechanism allows the model to attend to different parts of the input sequence and use this information to inform the output sequence.
Feed-Forward Neural Networks: These networks are used in both the encoder and decoder to transform the input and output sequences.

Advantages of the Transformer Architecture

The Transformer architecture has several advantages that make it particularly well-suited for sentiment analysis:

Parallelization: The Transformer architecture can be parallelized more easily than traditional recurrent neural networks (RNNs), making it faster to train and more efficient.
Scalability: The Transformer architecture can handle longer input sequences than traditional RNNs, making it more suitable for tasks that require analyzing longer pieces of text.
Flexibility: The Transformer architecture can be used for a variety of NLP tasks, including sentiment analysis, machine translation, and text summarization.

Applications of the Transformer Architecture

The Transformer architecture has been widely adopted in NLP tasks, including:

Sentiment Analysis: The Transformer architecture has been used to achieve state-of-the-art results in sentiment analysis tasks, such as classifying text as positive, negative, or neutral.
Machine Translation: The Transformer architecture has been used to achieve state-of-the-art results in machine translation tasks, such as translating text from one language to another.
Text Summarization: The Transformer architecture has been used to achieve state-of-the-art results in text summarization tasks, such as summarizing long pieces of text into shorter summaries.

Conclusion

In conclusion, the Transformer architecture is a powerful tool for NLP tasks, including sentiment analysis. Its ability to attend to different parts of the input sequence simultaneously and weigh their importance makes it particularly well-suited for tasks that require analyzing complex pieces of text. Based on my technical understanding as a Lead Programmer Analyst, I believe that the Transformer architecture has the potential to revolutionize the field of NLP and achieve state-of-the-art results in a variety of tasks.

Your Turn

As we continue to explore the possibilities of building custom AI models for sentiment analysis with Transformers, we would like to ask: What are some potential applications of the Transformer architecture in sentiment analysis that you think have not been fully explored yet, and how do you think the architecture could be modified or extended to better suit these applications? Share your thoughts and opinions in the comments below!

Originally published at https://artificial-inteligence.phptutorial.co.in

Inside the Mixture-of-Experts layer: softmax gating, top-k routing, capacity, and the load-balancing loss

Devanshu Biswas — Fri, 07 Aug 2026 04:20:50 +0000

A dense Transformer runs every token through the same feed-forward block, so capacity (parameters) and cost (FLOPs) are welded together — to get a smarter model you pay more per token. A Mixture-of-Experts layer breaks that weld. It replaces the one shared FFN with N independent expert MLPs plus a tiny gating network, and keeps only a few experts per token. Params scale with N; FLOPs per token stay fixed. More knowledge, roughly the same compute. That decoupling is how Switch Transformer and Mixtral-8x7B (≈47B params, only ≈13B active per token) work.

The gate — a softmax over the experts

The gate is a single linear layer W_g that maps a token to one logit per expert; softmax turns those into a distribution — how much this token wants each expert.

def softmax(z):
    z = z - z.max()                 # stability, doesn't change the result
    e = np.exp(z)
    return e / e.sum()

def gate(x, Wg):                    # Wg: (N_experts, d)
    return softmax(Wg @ x)          # p[e] = P(expert e | token x)

Top-k routing — the sparsity

Sparsity is the whole point: do not run all N experts, keep only the top-k gate entries (k=1 in Switch, k=2 in Mixtral). Renormalize just those k weights so they sum to 1 again — they become the mixing coefficients for combining the chosen experts' outputs, y = Σ_{e∈top-k} g̃_e · E_e(x).

def top_k(p, k):
    idx = np.argsort(p)[::-1][:k]   # indices of the k largest gate probs
    w   = p[idx]
    w   = w / w.sum()               # renormalize over the kept experts
    return idx, w                   # experts to run + their weights

Because k is fixed, the FLOPs per token stay constant no matter how many experts you add. Grow N and you add parameters, not compute.

Capacity — a hardware budget that drops overflow

Real hardware wants dense, equal-sized batches per expert, so each expert gets a fixed number of slots C = f·(T·k / N). Dispatch tokens in arrival order; once an expert is full, extra tokens overflow and are dropped — they skip the layer through the residual connection. The capacity factor f buys slack against imbalance; shrink it toward 0.25 and even balanced routing starts dropping tokens. Capacity is a budget, not a correctness knob.

def dispatch(idxs, T, k, N, f=1.25):
    C = int(np.ceil(f * T * k / N))     # slots per expert
    load = np.zeros(N, int); kept = []
    for token_experts in idxs:          # tokens in arrival order
        keep = []
        for e in token_experts:
            if load[e] < C:
                load[e] += 1; keep.append(e)   # accepted
            # else: OVERFLOW -> dropped, token uses the residual
        kept.append(keep)
    return kept, load, C

The auxiliary load-balancing loss

A raw gate collapses: whichever experts get picked early get trained more, get picked more — rich-get-richer, until a few experts dominate and the rest die. Switch adds an auxiliary loss to the training objective:

$$L_{aux} = N \cdot \sum_e f_e \cdot P_e$$

where f_e is the fraction of tokens dispatched to expert e (a hard count) and P_e is the mean soft router probability for e. It is minimized at 1 exactly when usage is uniform, and climbs toward N under total collapse — so pulling it down is load balancing.

def aux_loss(P_soft, dispatch_counts, T, k, N):
    f = dispatch_counts / (T * k)   # fraction of routed tokens per expert
    P = P_soft.mean(axis=0)         # mean router prob per expert (soft)
    return N * np.sum(f * P)        # = 1 when uniform, ~N when collapsed

The gradient flows through the soft P_e (the hard count f_e is detached), and through the softmax Jacobian it reduces to a clean rule: push the logit of an over-used expert down. Take a step and the aux loss falls, the load bars level off, and dropped tokens go to zero.

Put it together — gate every token, take top-k, dispatch under capacity, run the kept experts, gate-weight the outputs, add λ·L_aux during training — and you have the layer. Total params scale with N, FLOPs per token scale with k.

Watch a collapsed router genuinely rebalance by gradient descent, live, at https://dev48v.infy.uk/dl/day56-mixture-of-experts.html

RoPE Explained: Encoding Token Position by Rotating Query and Key Vectors

Devanshu Biswas — Wed, 05 Aug 2026 15:10:21 +0000

Self-attention is order-blind. The raw QKᵀ scores are pure content dot-products, so a Transformer has to be told where each token sits. The 2017 sinusoidal recipe adds a positional vector to every token; ALiBi adds a distance bias to the scores. RoPE (Su et al., 2021) does neither — it encodes position by rotating the query and key. It's the scheme behind LLaMA, GPT-NeoX, PaLM, and Mistral, and once you see the trick it's beautifully simple.

Rotate the pairs

Split each d-dimensional query/key vector into d/2 two-dimensional pairs. For a token at position p, spin pair i by an angle θ = p · ω_i. Rotation is a plain 2×2 matrix — and because it's orthogonal, it preserves the norm, which is why RoPE never rescales Q or K:

def rot2(x, y, a):                 # rotate the pair (x, y) by angle a
    c, s = np.cos(a), np.sin(a)
    return x*c - y*s,  x*s + y*c   # R(a) @ [x, y]

The frequency schedule

Each pair rotates at its own fixed rate — a geometric ladder from fast to slow:

def rope_freqs(d, base=10000.0):
    i = np.arange(d // 2)                 # one frequency per PAIR
    return base ** (-2.0 * i / d)         # omega_i, geometric ladder

rope_freqs(8)
# [1.0, 0.1, 0.01, 0.001]                # fast -> slow
# wavelengths 2*pi/omega : 6.28 .. 6283  # local -> long-range

High-frequency pairs (pair 0 spins at 1 radian/step) encode fine, local position; low-frequency pairs barely move and encode coarse, long-range position. It's a built-in multi-scale code. base = 10000 is the original LLaMA value.

The relative-offset property — the whole point

Here's the magic. A dot product of two rotated vectors depends only on the difference of their angles, because of the rotation identity R(mθ)ᵀ R(nθ) = R((n−m)θ). So the attention score rope(q, m) · rope(k, n) becomes a function of the relative offset m − n alone — even though each vector was rotated by its absolute position:

def score(q, k, m, n, base=10000.0):
    return rope(q, m, base) @ rope(k, n, base)

q = np.random.randn(8); k = np.random.randn(8)
assert np.allclose(score(q, k, 5, 3), score(q, k, 7, 5))   # same offset 2
assert np.allclose(score(q, k, 9, 9), q @ k)               # offset 0 -> raw q.k

Positions (5,3) and (7,5) score identically — both have offset 2. Laid out over an (m, n) grid, the score is constant along every diagonal: a Toeplitz matrix.

RoPE attention

In the attention layer you rotate every query and key by its position before the dot products. V is untouched, nothing is added to the inputs, and the resulting score matrix is automatically relative:

def rope_attention(Q, K, V, base=10000.0):
    N, d = Q.shape
    Qr = np.stack([rope(Q[p], p, base) for p in range(N)])   # rotate queries
    Kr = np.stack([rope(K[p], p, base) for p in range(N)])   # rotate keys
    scores = (Qr @ Kr.T) / np.sqrt(d)                        # relative by construction
    scores[np.triu_indices(N, 1)] = -np.inf                  # causal mask
    A = np.exp(scores - scores.max(1, keepdims=True))
    A /= A.sum(1, keepdims=True)
    return A @ V, A

Why it stretches to long context

Push a model past its trained length and the fast pairs alias. Two fixes reuse the exact same code: position interpolation squeezes positions (p → p/s) so old angles cover new lengths; NTK-aware scaling raises the base so slow pairs stretch while fast ones stay sharp. Both only touch the frequency schedule — the rotation is unchanged, and the relative-offset property survives.

That's the trade-off table in one line: sinusoidal is absolute and added to inputs; ALiBi is relative and added to scores; RoPE is relative and applied by rotating Q and K, preserving norm and scaling cleanly to long context.

Slide the query and key positions, watch the 2D sub-planes rotate and the score stay glued to constant diagonals, and read the full six-step build here: https://dev48v.infy.uk/dl/day55-rope.html

ALiBi: give a Transformer position sense with one line, and let it test longer than it trained

Devanshu Biswas — Tue, 04 Aug 2026 08:49:13 +0000

Self-attention is order-blind: shuffle the tokens and the raw QKᵀ scores come out identical, so a Transformer has to be told where each token sits. The classic fix adds a positional embedding to every token vector — sinusoidal or learned. But those absolute codes are welded to the exact positions seen in training. Feed a model trained at length 1024 a sequence of 3000 and it meets position indices it has never represented, and quality collapses.

ALiBi (Press, Smith & Lewis, 2021 — "Train Short, Test Long") throws positional embeddings out entirely. Instead it adds a fixed, unlearned linear bias straight onto the attention scores, before the softmax.

One matrix: −m·(i−j)

The whole of ALiBi lives in the distance matrix (i − j) — how many steps a key is behind the query. Multiply by a slope and negate:

def distance(N):
    i = np.arange(N)[:, None]          # (N,1) query index
    j = np.arange(N)[None, :]          # (1,N) key index
    return i - j                       # (N,N)  >=0 in the causal region

def alibi_bias(N, m):
    return -m * distance(N)            # -m*(i-j) : linear recency penalty

It's 0 on the diagonal and grows more negative to the left — a penalty that grows linearly with how far back a key is. There is nothing to learn.

Per-head slopes form a geometric ladder

Each attention head gets its own fixed slope, so the model attends at many scales at once. For n heads the slopes are the geometric sequence m_h = 2^(-8h/n):

def head_slopes(n):
    start = 2 ** (-8 / n)              # = 2^(-8/n)
    return np.array([start ** (h + 1) for h in range(n)])

head_slopes(8)
# [0.5, 0.25, 0.125, 0.0625, 0.03125, 0.015625, 0.0078125, 0.00390625]
#  steep (local)  ------------------------------->  gentle (far)

Steep slopes give sharp recency — only the last few keys survive. Gentle slopes let a head gaze far back. Together they're a built-in, multi-scale recency prior.

The whole attention change is one added line

def alibi_attention(Q, K, V, m):
    N, d = Q.shape
    scores = (Q @ K.T) / np.sqrt(d)    # (N,N) raw QK^T
    scores = scores + alibi_bias(N, m) # <-- the ONE new line
    scores[~causal_mask(N)] = -np.inf  # can't see the future
    A = np.stack([softmax(r) for r in scores])
    return A @ V, A                    # output, attention weights

No positional embedding is added to Q, K, or the inputs anywhere. The softmax then turns that straight-line penalty into a clean exponential distance decay, giving each head a bounded effective window of about 1/m.

Why it extrapolates: distance, not position

The bias reads only i − j. Slide the whole window along the sequence and every entry is unchanged — it is translation-invariant, so the length N never appears as an absolute index the way a positional embedding would.

# translation invariance: bias(i,j) depends ONLY on (i-j)
b = alibi_bias(8, 0.25)
assert np.allclose(b[7,3], b[6,2])     # same distance 4 -> same bias
assert np.allclose(b[5,5], 0.0)        # distance 0 -> no penalty

# softmax turns the LINEAR bias into an EXPONENTIAL decay in distance:
#   weight(d)  ∝  exp(-m * d)          # bounded effective window ~ 1/m

Train at 1024, run at 3000: the model only ever sees relative distances, and far ones are simply taxed more. The attention-by-distance profile is the same shape at any length, which is exactly why a short-trained model extrapolates — and why ALiBi rides in BLOOM and MPT.

ALiBi vs sinusoidal vs RoPE

Sinusoidal embeddings are absolute and added to the inputs; they don't extrapolate past trained positions. RoPE is relative but applies its signal by rotating Q and K. ALiBi needs no embedding table and no rotation — just one added matrix of −m·(i−j), with a fixed per-head slope. It's the cheapest recipe of the three, and the one that most naturally handles test sequences longer than training.

Build the QKᵀ matrix, watch the bias ramp add on top, drag the per-head slope, and slide test length past train length to see the decay curves stay glued together, live at: https://dev48v.infy.uk/dl/day54-alibi.html

Gated Linear Units explained: the multiplicative valve behind LLaMA and PaLM's SwiGLU FFN

Devanshu Biswas — Mon, 03 Aug 2026 16:12:14 +0000

A standard Transformer feed-forward block is dead simple: Linear -> one fixed nonlinearity -> Linear, where that middle activation (ReLU, GELU) applies the same elementwise curve to every hidden unit. A Gated Linear Unit throws that single activation out and replaces it with a multiplicative valve. Understanding it is the difference between reading a modern LLM's FFN and being baffled by it.

Project the input twice, then multiply

Instead of one weight matrix, a GLU uses two. W makes a value branch (kept linear); V makes a gate branch (squashed into (0,1)). Multiply them element by element:

out = (xW + b) ⊙ σ(xV + c)
        value          gate

The gate decides, per unit and per input, how much of the value is allowed through. A gate near 0 shuts that feature off; near 1 it passes. In code the forward pass is three lines:

def branches(x):
    value    = x @ W + b           # linear, NO activation
    gate_pre = x @ V + c           # will be squashed
    return value, gate_pre

def glu(x):
    value, gate_pre = branches(x)
    gate = sigmoid(gate_pre)       # gate in (0, 1)
    return value * gate            # elementwise (Hadamard) product

Why multiplication buys capacity

A single Linear + ReLU can only bend each unit with a fixed curve — an additive nonlinearity. It can't cheaply make unit A's output depend on unit B. A product can: value ⊙ gate is an AND-like interaction — "let this feature through only when the input also opens the gate." That data-dependent, multiplicative routing is a strictly richer primitive, which is why a gated FFN reaches a lower loss at the same FLOP budget.

The family is just which squasher you put on the gate

GLU -> GEGLU -> SwiGLU differ in exactly one thing: the gate activation.

def sigmoid(z): return 1.0 / (1.0 + np.exp(-z))

def gelu(z):                       # tanh approximation
    k = np.sqrt(2.0 / np.pi)
    return 0.5 * z * (1.0 + np.tanh(k * (z + 0.044715 * z**3)))

def silu(z):                       # SiLU == Swish with beta = 1
    return z * sigmoid(z)

GATE = {"GLU": sigmoid, "GEGLU": gelu, "SwiGLU": silu}

Sigmoid saturates in (0,1) — a pure open/shut valve. GELU and SiLU are smooth and non-monotonic: they dip slightly below zero near z ≈ -1, then climb toward the identity, and crucially they are not capped at 1. So a GELU/SiLU gate can also amplify magnitude, not just attenuate — and it keeps a live gradient where the sigmoid's has vanished. Noam Shazeer's 2020 note "GLU Variants Improve Transformer" swept these and found GEGLU/SwiGLU give the best quality per parameter.

The full SwiGLU FFN — and the ⅔-width rule

In a Transformer the gated hidden goes through a third matrix back to the model width. A plain ReLU FFN has 2 matrices; a gated FFN has 3 (W, V, W2), so at the same hidden width it uses ~1.5x the parameters. To keep it parameter-neutral, shrink the hidden width by 2/3 — that's LLaMA's d_ff = 8/3 · d_model instead of 4 · d_model.

class SwiGLU_FFN(nn.Module):
    def __init__(self, d_model, mult=4):
        super().__init__()
        d_ff = int(2/3 * mult * d_model)          # 3 matrices => shrink 2/3
        self.w  = nn.Linear(d_model, d_ff, bias=False)   # value  W
        self.v  = nn.Linear(d_model, d_ff, bias=False)   # gate   V
        self.w2 = nn.Linear(d_ff, d_model, bias=False)   # down   W2

    def forward(self, x):
        gated = F.silu(self.v(x)) * self.w(x)     # SwiGLU core
        return self.w2(gated)                     # back to d_model

This is exactly why PaLM, LLaMA / 2 / 3 and Mistral all use a SwiGLU FFN. Gating itself traces back to Dauphin et al. 2016 (a gated convolutional LM — the original GLU); Shazeer brought it into the Transformer FFN. If a recent model's feed-forward "looks weird," it's almost always a gated one.

The mental model that sticks: value = what, gate = how much. Swap the gate activation (σ -> GELU -> SiLU) and you walk the GLU -> GEGLU -> SwiGLU family; pick SiLU and trim the hidden width by ⅔ and you have the LLaMA FFN.

Feed a live input vector through both branches, flip between the three gates, and probe the valve surface out = value · act(gate) here: https://dev48v.infy.uk/dl/day53-gated-linear-units.html

DEV Community: transformers

Top Dev Tools and Tutorials: QuantumAI, NeuroSync, and More for 2026

QuantumAI SDK v2.0

What it is

Why now

Tutorial angle

Difficulty

Start today:

NeuroSync CLI v1.5

What it is

Why now

Tutorial angle

Difficulty

Start today:

TensorFlow Quantum 1.3

What it is

Why now

Tutorial angle

Difficulty

Start today:

PyTorch Lightning v2.0

What it is

Why now

Tutorial angle

Difficulty

Start today:

Hugging Face Transformers v5.0

What it is

Why now

Tutorial angle

Difficulty

Start today:

Building in public: my lead-gen engine

Start with this today

Picks of the week

Connect

Part 2: Comparing AI Frameworks for Natural Language Processing: Transformers and Recurrent Neural Networks

Part 2: Comparing AI Frameworks for Natural Language Processing: Transformers and Recurrent Neural Networks

Introduction to Transformers and RNNs

Comparing Transformers and RNNs

Real-World Applications

Conclusion

📚 References & Further Reading

Your Turn

Why “IEC 60076 Compliant” Is Not Enough in Transformer Specifications

From Sparse Attention to MoE: Why K3's 896 Experts Only Activate 1.8% Per Token

From Sparse Attention to MoE: Why K3's 896 Experts Only Activate 1.8% Per Token

1. The Origin of the Problem: The Curse of All-or-Nothing

2. Three Cards in MoE's Evolution

Card 1: Switch Transformer (2021, Google) — Moving MoE from Lab to Factory

Card 2: Mixtral 8x7B (2023, Mistral) — Top-2 Becomes the Industry Standard

Card 3: DeepSeek-MoE (2024, DeepSeek) — Fine-Grained Experts + Shared Experts

3. K3's Stable LatentMoE: Pushing DeepSeek-MoE's Three Paths to the Limit

Path 1: Expert Count Jumps from 64 to 896

The Router Is Not a "Classifier" — It's a "Compute Allocator"

Quantile Balancing: The Load-Balancing Challenge of 896 Experts

Path 2: Latent Compression + 2 Shared Experts

Path 3: 1.8% Activation Rate — It's Not a Bug, It's a Feature

4. The Evolution at a Glance: From Dense to 896-Expert

5. K3 MoE's Real-World Performance

6. What's Next: MoE on the Road Ahead

7. What This Means for You

References

Depth-Attention: Opening an Attention Channel Between Transformer Layers

1. The Problem: Transformers Are "Blind" Between Layers

But these approaches share a problem

2. How Depth-Attention Works: Rotating "Sequence Attention" by 90 Degrees

Three elegant properties:

3. Counter-Intuitive Ablation: More Layers Isn't Always Better

4. GQA Compatibility: Naturally, Freely More Efficient

5. Experimental Results: What's the Actual Gain?

Key results

Efficiency: theory vs. measurements

Scaling experiments

6. Weight Visualization: What Did the Model Learn?

7. Looped Transformer Experiment: Parameter Sharing Works Too

8. Depth-Attention vs. Kimi K3's Attention Residuals: A Design Philosophy Divergence

9. What This Means

10. Limitations (Frankly)

References