DEV Community: transformers

Building a Sentiment Analysis Model with Python and Hugging Face Transformers

Phil IPS — Sun, 11 Oct 2026 22:33:00 +0000

Building a Sentiment Analysis Model with Python and Hugging Face Transformers

Sentiment analysis is a crucial technique for understanding the emotional tone behind text. In this tutorial, we'll build a simple yet powerful sentiment analysis model using Python and Hugging Face's Transformers library.

Step 1: Set Up Your Environment

First, install the required libraries:

pip install transformers torch

Step 2: Load a Pre-trained Model

We'll use a pre-trained model fine-tuned on the SST-2 dataset for sentiment analysis:

from transformers import pipeline

# Initialize the sentiment analysis pipeline
classifier = pipeline("sentiment-analysis")

Step 3: Analyze Sentiment

Now, let's analyze some sample texts:

# Example texts to analyze
texts = [
    "I love this product! It works perfectly.",
    "This is the worst experience I've ever had.",
    "The movie was average, nothing special."
]

# Run sentiment analysis
for text in texts:
    result = classifier(text)
    print(f"Text: {text}")
    print(f"Sentiment: {result[0]['label']} (Confidence: {result[0]['score']:.2f})")
    print()

Step 4: Customize for Your Use Case

You can further customize the model by fine-tuning it on your own dataset. For beginners, the above setup is sufficient for basic sentiment analysis tasks.

Conclusion

In this tutorial, we've demonstrated how to build a sentiment analysis model using just a few lines of Python code. With the power of Hugging Face Transformers, you can easily integrate sentiment analysis into your applications, from customer feedback analysis to social media monitoring.

Feel free to experiment with different models and datasets to improve accuracy or adapt to specific domains!

Transformers: The Movie Review – Still the Best

LuckyTaorem — Wed, 23 Sep 2026 19:23:35 +0000

Introduction

Sam Rutherford’s recent re‑review of Transformers: The Movie (1986) reaffirms the film’s status as the pinnacle of the franchise. The article argues that, even after decades of sequels and reboots, the original 1986 entry remains the benchmark for storytelling, visual effects, and emotional resonance. Rutherford’s analysis is grounded in a detailed comparison of the film’s technical achievements and its cultural footprint, offering fresh insights for both long‑time fans and newcomers.

Why It Matters

The 1986 film was a watershed moment for animated science‑fiction cinema. It introduced audiences to a fully realized universe where robots could transform, battle, and feel. Rutherford’s verdict underscores how the film’s blend of humor, action, and heart set a template that modern blockbusters still emulate. The movie’s influence extends beyond the screen: it shaped toy lines, comic books, and even the way pop culture icons are perceived. For instance, the film’s iconic characters echo the charisma of figures like Zyzz, whose own rise to fame demonstrates how a single persona can capture the imagination of a generation. [Zyzz: The Rise,

...the film’s iconic characters echo the charisma of figures like Zyzz, whose own rise to fame demonstrates how a single persona can capture the imagination of a generation. [Zyzz: The Rise, 2023]. Rutherford uses this parallel to illustrate how The Transformers: The Movie created archetypes that transcend the screen, becoming cultural shorthand for “the underdog who becomes legend.”

Technical Mastery

Pioneering CGI and Practical Effects

When the film debuted, it was one of the first major productions to blend traditional hand‑drawn animation with computer‑generated imagery (CGI). The infamous “Unicron” sequence, for example, employed a hybrid of rotoscoping and early 3D modeling, delivering a sense of scale that still feels imposing today. Rutherford notes that the film’s visual language “predates the digital explosion of the 1990s yet feels remarkably fresh,” a claim supported by side‑by‑side frame analyses that show the original matte paintings holding up against modern CGI‑heavy blockbusters.

Sound Design and Musical Score

The soundtrack, composed by Vince DiCola, fuses synth‑driven motifs with orchestral swells, creating an aural landscape that mirrors the film’s dual nature—mechanical yet emotive. The iconic “The Touch” sequence, while often mocked for its melodramatic lyrics, actually serves as a narrative crescendo, aligning the protagonists’ emotional stakes with a soaring musical climax. Rutherford points out that the song’s key change and layered harmonies were technically ambitious for a television‑budget production, a fact that modern composers still reference when crafting heroic anthems.

Narrative & Character Development

A Bold, Uncompromising Storytelling Choice

Unlike the later live‑action entries that often prioritize spectacle over substance, the 1986 film embraces a darker, more consequential narrative. The death of Optimus Prime—a decision that shocked audiences worldwide—set a precedent for high‑stakes storytelling in animated features. Rutherford argues that this willingness to “risk beloved characters” gave the film an emotional gravitas that most sequels lack.

Ensemble Cast and Archetypal Depth

The movie’s roster—ranging from the stoic Ultra Magnus to the flamboyant Hot Rod—offers a spectrum of archetypes that resonate across generations. Each character’s arc is tightly woven into the central theme of sacrifice and renewal. For instance, Hot Rod’s transformation into Rodimus Prime is not merely a power‑up; it’s a rite‑of‑passage that mirrors the hero’s journey outlined by Joseph Campbell, a point Rutherford emphasizes when he writes, “the film compresses mythic structure into a 90‑minute runtime without feeling rushed.”

Cultural Impact & Legacy

Influence on Toy Design and Merchandising

Hasbro’s post‑film toy line saw a 42 % surge in sales, a statistic cited by Rutherford from a 2022 industry report. The movie’s designs—most notably the sleek, angular aesthetic of the “Combiner” robots—directly informed the next generation of action figures, establishing a visual language that persists in today’s collectibles market.

Fan Communities and Online Revival

The rise of streaming platforms has introduced The Transformers: The Movie to a new wave of fans. Online forums, fan art, and cosplay conventions frequently reference the film’s iconic moments. Rutherford highlights a 2025 Reddit AMA where over 10,000 participants discussed the film’s “timelessness,” underscoring how the narrative continues to spark dialogue decades later.

Read the full breakdown originally published at https://ltdeveloperblogs.github.io/posts/the-transformers-the-movie-re-review-still-the-best-and-its-not-close/

They Put 7 Attention Mechanisms on a Latin Square. Then Removed Them One by One.

ai maya — Fri, 18 Sep 2026 03:48:33 +0000

Since GPT, nearly every Transformer repeats the same attention mechanism at every layer. Forty-eight
identical blocks, differing only in learned weights.

Nobody tested that. It is a convention, not a conclusion.

A paper out of VIDRAFT AI Research (arXiv:2609.20269, CC BY 4.0)
tests it, and the interesting part is not the headline. The headline is placement is free,
composition is not. The interesting part is what happens when you read the ablation table.

The confound that makes this hard to measure

If you build a stack with several different sequence mixers and it beats a uniform stack, you
cannot say why. Was it which mechanisms you used, or where you put them? Those two are
tangled in every heterogeneous architecture paper, because any concrete stack is one particular
arrangement of one particular set.

The paper's move is to remove the second variable by construction rather than by search.

A Latin square is an N×N grid where each symbol appears exactly once in every row and every
column. Sudoku is a Latin square with extra constraints. Put seven mechanisms on a 7×7 square and
read it out across 49 layers, and every mechanism is guaranteed to be spread evenly through depth.

layer  0..6   A B C D E F G
layer  7..13  B C D E F G A
layer 14..20  C D E F G A B
       ...

No mechanism can cluster. Not "did not cluster in this run" — cannot, structurally. That is the
whole point: you get balance without running an architecture search to find it.

The flagship is Aether-7B-5Attn — 6.59B parameters MoE, ~2.98B active, 49 layers, seven mixers
on a 7×7 square. Trained on 16× B200 (2-node FSDP), 162,000 steps, 144.2B tokens, ~11,700
B200-hours in the final stage.

The actual experiment

You cannot ablate a 6.59B flagship eight ways with eight seeds each. So the paper builds a
parameter-matched proxy: four mechanisms on a 4×4 square over sixteen layers, 700.9M parameters,
eight seeds per arm.

Four arms, identical parameter counts, differing only in arrangement:

Arm	Mean CE	SD	Δ vs latin	2·pooled SD	Verdict
`latin`	5.28639	0.00867	—	—	reference
`periodic`	5.29484	0.01611	+0.16%	0.02588	null (within noise)
`block`	5.31754	0.01520	+0.59%	0.02475	real (2.5× pooled SD)
`homo_F`	—	—	+1.68%	—	real

Read it as a gradient of how tightly each mechanism is confined in depth:

Distributed, any order (latin vs periodic) → indistinguishable. Shuffling a balanced schedule buys you nothing.
Confined to contiguous bands (block) → you pay 0.59%.
One mechanism everywhere (homo_F) → you pay 1.68%.

So the first practical result: stop burning GPU hours searching for the right layer order. If
your schedule is balanced and distributed, the permutation is noise. That is a real cost saving,
and it is the kind of null result that rarely gets published.

Now the part worth your attention

The paper then removes one mechanism at a time from the four-mechanism stack, re-matching parameters
and cycling the remaining three through depth:

Removed	Mean CE	Δ vs latin	Verdict
sliding window	5.28495	−0.03%	null
differential	5.28450	−0.04%	null
full attention	5.29615	+0.18%	null
Mamba-2 / SSM	5.39940	+2.14%	real
(baseline)	5.28639	—	8 seeds

Three of the four mechanisms can be deleted with no measurable effect. Two of them are slightly
negative — the model is nominally better without them, well inside noise.

One is load-bearing.

Look at which one. Sliding window, differential, and full attention are all members of the
attention family — variations on the same operator. Mamba-2 is a state-space model, a
structurally different way to mix along the sequence.

The honest reading of this table is narrower and more useful than "heterogeneity helps":

What pays is having at least one mechanism from a different family. Seven flavors of attention
is still, functionally, a homogeneous stack.

That is a concrete design rule you can apply tomorrow, and it explains why the homogeneous arm
(1.68%) and the no-SSM arm (2.14%) land in the same neighborhood — from the loss's point of view,
removing the SSM is collapsing toward homogeneity.

Does it survive scale?

The obvious objection: 700.9M with 1,500 steps is a pilot, not a shipped model.

The paper re-runs the three decisive arms — latin, homo_F, no_M — at 1.514B parameters, a
2.16× increase, everything else held constant, three seeds each. Both penalties get bigger:

	700.9M	1.514B
homogeneous stack	+1.68%	+2.63%
remove SSM family	+2.14%	+3.20%

Penalties that grow with scale are the good kind of evidence. A finding that shrinks as you scale is
usually a small-model artifact; one that widens is more likely structural.

Note what was not re-run: the placement axis (periodic, block) stays at 700.9M, and neither
axis was tested at N=7 or at flagship size. The paper says so in its limitations rather than leaving
you to notice.

Two methodology details worth stealing

1. A pre-registered decision rule.

The threshold is |Δ| > 2 · pooled_SD, fixed on 2026-07-22 — before the final seeds ran. Section
6.4 explains why that matters: an early two-seed run showed latin and periodic cleanly
separated, which would have been a placement effect, which is the more marketable story. Four seeds
dissolved it. Eight confirmed the dissolution, at Δ 0.008 — a third of the threshold.

The rule that killed the authors' preferred result is the same rule that certifies block and
homo_F as real. As the paper puts it: a rule that only ever confirms what you hoped for is not a
rule.

If you run architecture experiments, the two habits here are cheap and they work: no claim below
four seeds, and fix your threshold before the last run.

2. A positive control in the safety audit.

Mixing operators along the time axis is exactly where causality violations hide — a stack that leaks
future information will show beautiful loss curves for the wrong reason. All 49 layers pass a
negative control, and, on the same loaded checkpoint, a positive control: 16 of 16 deliberately
injected faults localized exactly, across NSA, hybrid, and linear-attention layers.

This is the right shape for any correctness gate. A clean pass proves nothing unless you have also
shown the detector fires when something is actually broken. Most audits skip the second half.

What is released

Weights, training-data recipes, training code, logs, and architecture source. There is also
Aether-6B-11Attn-base — eleven mechanisms (attention, Mamba-2, Hyena, GDN, MLA among them) on an
11×11 square over 121 layers — shipped as-is, a mid-training artifact whose only job is to show
the construction is not specific to N=7. Whether the composition findings hold at N=11 is explicitly
left open.

The takeaway for practitioners

Placement search on a balanced schedule is probably wasted compute. 0.16%, inside noise.
Family diversity is the lever, not mechanism count. Three attention variants were free to delete; the one SSM was not.
A Latin square is a cheap way to get guaranteed balance without searching for it. The construction makes the failure mode unreachable rather than merely unobserved.
Pre-register your threshold and run four seeds minimum. This paper's most valuable page is the one where the method deleted the result the authors wanted.

Paper: arXiv:2609.20269 — Placement Is Free, Composition Is
Not: The Latin Square as a Provably-Balanced Construction for Heterogeneous Sequence-Mixer Stacks

Detectar volatilidad anómala en los mercados con un Temporal Fusion Transformer

Adrian — Thu, 17 Sep 2026 10:00:02 +0000

Predecir el precio de una acción es casi imposible y, francamente, poco útil. Hay una pregunta más alcanzable y más valiosa: ¿cuándo está a punto de volverse anormalmente inestable un activo? Los picos de volatilidad suelen preceder a las noticias, no seguirlas. Ese era el objetivo de este "sensor de mercado".

El problema

En lugar de estimar un único número (el retorno esperado), quería estimar la distribución completa de retornos para cada activo y horizonte, y marcar como anómalos los momentos en los que la realidad se sale de los cuantiles que el modelo considera plausibles.

La arquitectura: TFT + regresión cuantílica

Elegí el Temporal Fusion Transformer porque combina tres cosas que necesitaba:

Variable Selection Networks que aprenden qué inputs importan en cada momento, dándome interpretabilidad sin renunciar a la potencia.
Atención multi-cabeza sobre la ventana temporal, que captura dependencias largas (un patrón de hace semanas) mejor que una LSTM.
Salida de regresión cuantílica: en vez de un valor, predice varios cuantiles (p10, p50, p90...). La distancia entre cuantiles es la incertidumbre estimada.

Cuando un retorno real cae fuera del intervalo cuantílico esperado, salta la alerta de anomalía.

La métrica que de verdad valida el modelo

Un modelo cuantílico solo sirve si está bien calibrado: si dice que algo ocurre el 90% de las veces, debe ocurrir el 90% de las veces. La calibración cuantílica de este modelo resultó prácticamente perfecta, lo que significa que sus intervalos de incertidumbre son fiables, no decorativos.

Resultados

F1-score (detección de anomalías): 0.82
MCC: 0.80
Calibración cuantílica: ≈ perfecta

Qué aprendí

Que reformular el problema —de "predecir el precio" a "predecir la incertidumbre"— lo vuelve tratable y útil. Y que en finanzas, un modelo que conoce sus propios límites (su calibración) vale más que uno que da respuestas precisas y falsas.

Publicado originalmente en adrianmoreno-dev.com.

Puzzle Solution Revealed - Transformer: Need for Position Embedding

Tech-Aarvam — Tue, 15 Sep 2026 13:05:50 +0000

Introduce position embedding, residuals and layers

▶ Watch the 7-minute walkthrough

import torch

torch.set_printoptions(precision=2, sci_mode=False, linewidth=160)

In this notebook we are extending the previous notebook, attention_ann.ipynb, which introduced (plausibly) the world's tiniest hand-constructed transformer model. Please read the previous notebook for the context. This is the solution to the puzzle that was introduced as part of the previous notebook.

We add one word 'disobeys' which modifies the action-attribute words (swap-speech, keep-speech, swap-flight, keep-flight). In the previous notebook the order of the words did not matter. In the current notebook, the order does matter. The word disobeys modifies only the word right after it, so word order matters.

The solution adds an extra layer, and a residual connection that carries the input unmodified to the next block, so each next layer gets both the unmodified input and the modified input.

The residual stream is made 22 bits, where it carries the original inputs and each layer's/head's findings. The most interesting addition is the position information. Each token carries the position where it appears.

This example is constructed to illustrate the ideas. In the real implementation, we do not hand-construct in this manner. Vector embedding and position embedding are often also learned using the training loop and gradient descent. But the hand-construction allows us to see why those blocks and connections exist and how they are helpful to have.

Residual layout

idx	slot
0	`fly`
1	`speak`
2	`swap_fly`
3	`swap_speak`
4	`object`
5	`action_fly`
6	`action_speak`
7	`question`
8	`disobey`
9	`pos0`
10	`pos1`
11	`pos2`
12	`pos3`
13	`pos4`
14	`pos5`
15	`previous_word_is_disobey`
16	`object_attribute_fly`
17	`object_attribute_speak`
18	`is_swap_attr_fly`
19	`is_attr_fly_disobeyed`
20	`is_swap_attr_speak`
21	`is_attr_speak_disobeyed`

# The new list of token bits, with the additional word 'disobeys'
idx = {"fly": 0, "speak": 1, "swap_fly": 2, "swap_speak": 3,
       "object": 4, "action_fly": 5, "action_speak": 6, "question": 7, "disobey": 8}

# constants used for bit-slicing and locating the portion of
# the residual we need

# L is the maximum sentence length, i.e. the number of position slots.
# It is used to name the position slots below, and to build the
# "Disobey Position Finder" head (Wq_P, Wk_P) in layer 1.
position_start, L = 9, 6             # position one-hot occupies 9 .. 14
previous_word_is_disobey = 15
object_attribute_fly, object_attribute_speak = 16, 17
is_swap_attr_fly, is_attr_fly_disobeyed = 18, 19
is_swap_attr_speak, is_attr_speak_disobeyed = 20, 21
num_bits = 22

slot_names = {v: k for k, v in idx.items()}
slot_names.update({position_start + p: f"pos{p}" for p in range(L)})
slot_names.update({previous_word_is_disobey: "previous_word_is_disobey",
                   object_attribute_fly: "object_attribute_fly",
                   object_attribute_speak: "object_attribute_speak",
                   is_swap_attr_fly: "is_swap_attr_fly",
                   is_attr_fly_disobeyed: "is_attr_fly_disobeyed",
                   is_swap_attr_speak: "is_swap_attr_speak",
                   is_attr_speak_disobeyed: "is_attr_speak_disobeyed"})

for i in range(num_bits):
    print(f"{i:2} {slot_names[i]}")

 0 fly
 1 speak
 2 swap_fly
 3 swap_speak
 4 object
 5 action_fly
 6 action_speak
 7 question
 8 disobey
 9 pos0
10 pos1
11 pos2
12 pos3
13 pos4
14 pos5
15 previous_word_is_disobey
16 object_attribute_fly
17 object_attribute_speak
18 is_swap_attr_fly
19 is_attr_fly_disobeyed
20 is_swap_attr_speak
21 is_attr_speak_disobeyed

Vocabulary

token_to_vector = {
    #                   fly spk swF swS  obj actF actS  q  dis
    "Rock":             [0,  0,  0,  0,   1,  0,  0,  0,  0],
    "Human":            [0,  1,  0,  0,   1,  0,  0,  0,  0],
    "Crow":             [1,  0,  0,  0,   1,  0,  0,  0,  0],
    "Flying superhero": [1,  1,  0,  0,   1,  0,  0,  0,  0],
    "swap-flight":      [0,  0,  1,  0,   0,  1,  0,  0,  0],
    "swap-speech":      [0,  0,  0,  1,   0,  0,  1,  0,  0],
    "keep-flight":      [0,  0,  0,  0,   0,  1,  0,  0,  0],
    "keep-speech":      [0,  0,  0,  0,   0,  0,  1,  0,  0],
    "disobeys":         [0,  0,  0,  0,   0,  0,  0,  0,  1],
    "he-is?":           [0,  0,  0,  0,   0,  0,  0,  1,  0],
}

for tok, bits in token_to_vector.items():
    print(f"{tok:18} {torch.tensor(bits)}")

Rock               tensor([0, 0, 0, 0, 1, 0, 0, 0, 0])
Human              tensor([0, 1, 0, 0, 1, 0, 0, 0, 0])
Crow               tensor([1, 0, 0, 0, 1, 0, 0, 0, 0])
Flying superhero   tensor([1, 1, 0, 0, 1, 0, 0, 0, 0])
swap-flight        tensor([0, 0, 1, 0, 0, 1, 0, 0, 0])
swap-speech        tensor([0, 0, 0, 1, 0, 0, 1, 0, 0])
keep-flight        tensor([0, 0, 0, 0, 0, 1, 0, 0, 0])
keep-speech        tensor([0, 0, 0, 0, 0, 0, 1, 0, 0])
disobeys           tensor([0, 0, 0, 0, 0, 0, 0, 0, 1])
he-is?             tensor([0, 0, 0, 0, 0, 0, 0, 1, 0])

x_{p} = 9 content bits c (t_{p}) ∥ 6 position bits e_{p} ∥ 7 scratch slots 0

def embed(sentence):
    X = torch.zeros(len(sentence), num_bits)
    for current_token_position, current_token in enumerate(sentence):
        X[current_token_position, :9] = torch.tensor(token_to_vector[current_token], dtype=torch.float32)
        X[current_token_position, position_start + current_token_position] = 1
    return X


sentence = ["Crow", "disobeys", "keep-flight", "swap-speech", "he-is?"]
X = embed(sentence)

for tok, row in zip(sentence, X):
    print(f"{tok:14} {row[:9]}  {row[position_start:position_start + L]}  {row[previous_word_is_disobey:]}")

Crow           tensor([1., 0., 0., 0., 1., 0., 0., 0., 0.])  tensor([1., 0., 0., 0., 0., 0.])  tensor([0., 0., 0., 0., 0., 0., 0.])
disobeys       tensor([0., 0., 0., 0., 0., 0., 0., 0., 1.])  tensor([0., 1., 0., 0., 0., 0.])  tensor([0., 0., 0., 0., 0., 0., 0.])
keep-flight    tensor([0., 0., 0., 0., 0., 1., 0., 0., 0.])  tensor([0., 0., 1., 0., 0., 0.])  tensor([0., 0., 0., 0., 0., 0., 0.])
swap-speech    tensor([0., 0., 0., 1., 0., 0., 1., 0., 0.])  tensor([0., 0., 0., 1., 0., 0.])  tensor([0., 0., 0., 0., 0., 0., 0.])
he-is?         tensor([0., 0., 0., 0., 0., 0., 0., 1., 0.])  tensor([0., 0., 0., 0., 1., 0.])  tensor([0., 0., 0., 0., 0., 0., 0.])

head (X) = softmax (\frac{( X W _{Q} ) ( X W _{K} ) ^{⊤}}{d _{h}}) X W_{V}

def softmax(z, dim=-1):
    z = z - z.max(dim=dim, keepdim=True).values
    e = torch.exp(z)
    return e / e.sum(dim=dim, keepdim=True)


def head(X, Wq, Wk, Wv):
    Q, K, V = X @ Wq, X @ Wk, X @ Wv
    A = softmax(Q @ K.T / Wq.shape[1] ** 0.5)
    return A @ V, A

Layer 1

Head: Object Head

Wq_O = torch.zeros(num_bits, 2); Wq_O[idx["question"]] = torch.tensor([8.0, 0.0])
Wk_O = torch.zeros(num_bits, 2); Wk_O[idx["object"]]   = torch.tensor([1.0, 0.0])
Wv_O = torch.zeros(num_bits, 2); Wv_O[idx["fly"]] = torch.tensor([1.0, 0.0]); Wv_O[idx["speak"]] = torch.tensor([0.0, 1.0])
Wo_O = torch.zeros(2, num_bits); Wo_O[0, object_attribute_fly] = 1; Wo_O[1, object_attribute_speak] = 1

print("Wq_O nonzero rows:", (Wq_O != 0).any(1).nonzero().flatten())
print("Wk_O nonzero rows:", (Wk_O != 0).any(1).nonzero().flatten())
print("Wv_O nonzero rows:", (Wv_O != 0).any(1).nonzero().flatten())
print("Wo_O nonzero cols:", (Wo_O != 0).any(0).nonzero().flatten())

Wq_O nonzero rows: tensor([7])
Wk_O nonzero rows: tensor([4])
Wv_O nonzero rows: tensor([0, 1])
Wo_O nonzero cols: tensor([16, 17])

Head: Disobey Position Finder

W_{Q (P)} = S p = 1 \sum L - 1 e_{pos_{p}} e_{p - 1 ⊤} W_{K (P)} = p = 0 \sum L - 1 e_{pos_{p}} e_{p ⊤}

M = W_{Q (P)} W_{K (P) ⊤} ⟹ q_{i} \cdot k_{j} = S [j = i - 1]

M \neq = M^{⊤}

S = 24.0

Wq_P = torch.zeros(num_bits, L)
for p in range(1, L):
    Wq_P[position_start + p, p - 1] = S

Wk_P = torch.zeros(num_bits, L)
for p in range(L):
    Wk_P[position_start + p, p] = 1

Wv_P = torch.zeros(num_bits, 1); Wv_P[idx["disobey"], 0] = 1
Wo_P = torch.zeros(1, num_bits); Wo_P[0, previous_word_is_disobey] = 1

M = Wq_P @ Wk_P.T
print("M[9:15, 9:15] =")
print(M[position_start:position_start + L, position_start:position_start + L])
print("symmetric:", torch.allclose(M, M.T))

M[9:15, 9:15] =
tensor([[ 0.,  0.,  0.,  0.,  0.,  0.],
        [24.,  0.,  0.,  0.,  0.,  0.],
        [ 0., 24.,  0.,  0.,  0.,  0.],
        [ 0.,  0., 24.,  0.,  0.,  0.],
        [ 0.,  0.,  0., 24.,  0.,  0.],
        [ 0.,  0.,  0.,  0., 24.,  0.]])
symmetric: False

FFN_{1} (x) = 0 X_{1} = X + head_{O} (X) W_{O (O)} + head_{P} (X) W_{O (P)}

def layer1(X):
    oO, AO = head(X, Wq_O, Wk_O, Wv_O)
    oP, AP = head(X, Wq_P, Wk_P, Wv_P)
    X1 = X + oO @ Wo_O + oP @ Wo_P
    return X1, AO, AP


X1, AO, AP = layer1(X)

print("A_P")
print(AP)
print()
for tok, row in zip(sentence, X1):
    print(f"{tok:14} previous_word_is_disobey={row[previous_word_is_disobey]:.3f}   "
          f"object_attribute_fly={row[object_attribute_fly]:.3f}  "
          f"object_attribute_speak={row[object_attribute_speak]:.3f}")

A_P
tensor([[0.20, 0.20, 0.20, 0.20, 0.20],
        [1.00, 0.00, 0.00, 0.00, 0.00],
        [0.00, 1.00, 0.00, 0.00, 0.00],
        [0.00, 0.00, 1.00, 0.00, 0.00],
        [0.00, 0.00, 0.00, 1.00, 0.00]])

Crow           previous_word_is_disobey=0.200   object_attribute_fly=0.200  object_attribute_speak=0.000
disobeys       previous_word_is_disobey=0.000   object_attribute_fly=0.200  object_attribute_speak=0.000
keep-flight    previous_word_is_disobey=1.000   object_attribute_fly=0.200  object_attribute_speak=0.000
swap-speech    previous_word_is_disobey=0.000   object_attribute_fly=0.200  object_attribute_speak=0.000
he-is?         previous_word_is_disobey=0.000   object_attribute_fly=0.986  object_attribute_speak=0.000

Layer 2

Two heads: Head1 - "get flight attribute" Head2 - "get speech attribute"

Layer 2 has two heads: get flight attribute and get speech attribute.
Usually layers have a similar topology, so two heads are used in both layers.

Could this work be done with a single head, like the action head in the previous notebook? Not with this residual layout. In the previous notebook, one head attended to both action words, and that worked because each action word's own swap bit (swap_fly or swap_speak) says which attribute it swaps. Here, the disobey information sits in one shared slot, previous_word_is_disobey, on both action words. A single head attending to both action words would add the two disobey signals into the same number, and they could no longer be told apart. For example, Crow disobeys keep-flight swap-speech he-is? and Crow keep-flight disobeys swap-speech he-is? would give exactly the same head output, but the answers are Human and Crow. So we use one head for the flight word and one head for the speech word.

Wq_fly_head = torch.zeros(num_bits, 1); Wq_fly_head[idx["question"], 0] = 8
Wk_fly_head = torch.zeros(num_bits, 1); Wk_fly_head[idx["action_fly"], 0] = 1
Wv_fly_head = torch.zeros(num_bits, 2); Wv_fly_head[idx["swap_fly"]] = torch.tensor([1.0, 0.0]); Wv_fly_head[previous_word_is_disobey] = torch.tensor([0.0, 1.0])

Wq_speak_head = torch.zeros(num_bits, 1); Wq_speak_head[idx["question"], 0] = 8
Wk_speak_head = torch.zeros(num_bits, 1); Wk_speak_head[idx["action_speak"], 0] = 1
Wv_speak_head = torch.zeros(num_bits, 2); Wv_speak_head[idx["swap_speak"]] = torch.tensor([1.0, 0.0]); Wv_speak_head[previous_word_is_disobey] = torch.tensor([0.0, 1.0])

Wo_2 = torch.zeros(4, num_bits)
Wo_2[0, is_swap_attr_fly]        = 1
Wo_2[1, is_attr_fly_disobeyed]   = 1
Wo_2[2, is_swap_attr_speak]      = 1
Wo_2[3, is_attr_speak_disobeyed] = 1

print("Wv_fly_head nonzero rows:  ", (Wv_fly_head != 0).any(1).nonzero().flatten())
print("Wv_speak_head nonzero rows:", (Wv_speak_head != 0).any(1).nonzero().flatten())
print("Wo_2 nonzero cols:         ", (Wo_2 != 0).any(0).nonzero().flatten())

Wv_fly_head nonzero rows:   tensor([ 2, 15])
Wv_speak_head nonzero rows: tensor([ 3, 15])
Wo_2 nonzero cols:          tensor([18, 19, 20, 21])

X_{2} = X_{1} + [head_{F} (X_{1}) ∥ head_{S} (X_{1})] W_{O (2)}

def layer2_attn(X1):
    oF, AF = head(X1, Wq_fly_head, Wk_fly_head, Wv_fly_head)
    oS, AS = head(X1, Wq_speak_head, Wk_speak_head, Wv_speak_head)
    X2 = X1 + torch.cat([oF, oS], dim=1) @ Wo_2
    return X2, AF, AS


X2, AF, AS = layer2_attn(X1)
q = sentence.index("he-is?")

print("A_F[q]", AF[q])
print("A_S[q]", AS[q])
print()
print("is_swap_attr_fly       ", X2[q, is_swap_attr_fly])
print("is_attr_fly_disobeyed  ", X2[q, is_attr_fly_disobeyed])
print("is_swap_attr_speak     ", X2[q, is_swap_attr_speak])
print("is_attr_speak_disobeyed", X2[q, is_attr_speak_disobeyed])

A_F[q] tensor([0.00, 0.00, 1.00, 0.00, 0.00])
A_S[q] tensor([0.00, 0.00, 0.00, 1.00, 0.00])

is_swap_attr_fly        tensor(0.)
is_attr_fly_disobeyed   tensor(1.00)
is_swap_attr_speak      tensor(1.00)
is_attr_speak_disobeyed tensor(0.00)

FFN 2

The FFN is shown for completeness, also as a hand-constructed implementation. But for understanding the ideas of attention, position embedding, residuals and the need for layers, this part can be skipped.

Summary: bias values are used carefully to allow distinguishing 0, 1, 2, 3. This provides different ReLU activation levels corresponding to the number of flips. This FFN is a parity finder, while the FFN used in the previous notebook without the disobeys word was an XOR gate. Repeating the note that, in the real implementation, all the weights and biases are learned using gradient descent and the backpropagation algorithm.

s_{fly} = x_{object_attribute_fly} + x_{is_swap_attr_fly} + x_{is_attr_fly_disobeyed} s_{speak} = x_{object_attribute_speak} + x_{is_swap_attr_speak} + x_{is_attr_speak_disobeyed}

π (s) = ReLU (s) - 2 ReLU (s - 1) + 2 ReLU (s - 2) - 2 ReLU (s - 3)

π (0) = 0, π (1) = 1, π (2) = 0, π (3) = 1

W1 = torch.zeros(num_bits, 8)
b1 = torch.zeros(8)

for r in (object_attribute_fly, is_swap_attr_fly, is_attr_fly_disobeyed):
    W1[r, 0:4] = 1
for r in (object_attribute_speak, is_swap_attr_speak, is_attr_speak_disobeyed):
    W1[r, 4:8] = 1

b1[0:4] = torch.tensor([0.0, -1.0, -2.0, -3.0])
b1[4:8] = torch.tensor([0.0, -1.0, -2.0, -3.0])

W2 = torch.zeros(8, 2)
W2[0:4, 0] = torch.tensor([1.0, -2.0, 2.0, -2.0])
W2[4:8, 1] = torch.tensor([1.0, -2.0, 2.0, -2.0])

print("W1.T\n", W1.T)
print("\nb1", b1)
print("\nW2.T\n", W2.T)
print("\npi:", [float(torch.relu(torch.tensor([s, s - 1.0, s - 2.0, s - 3.0])) @ W2[0:4, 0]) for s in range(4)])

W1.T
 tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 1., 1., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 1., 1., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 1., 1., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 1., 1., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 1., 1.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 1., 1.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 1., 1.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 1., 1.]])

b1 tensor([ 0., -1., -2., -3.,  0., -1., -2., -3.])

W2.T
 tensor([[ 1., -2.,  2., -2.,  0.,  0.,  0.,  0.],
        [ 0.,  0.,  0.,  0.,  1., -2.,  2., -2.]])

pi: [0.0, 1.0, 0.0, 1.0]

Forward

def forward(sentence):
    X = embed(sentence)
    X1, AO, AP = layer1(X)
    X2, AF, AS = layer2_attn(X1)
    H = torch.relu(X2 @ W1 + b1)
    Y = H @ W2
    return dict(X=X, X1=X1, X2=X2, Y=Y, AO=AO, AP=AP, AF=AF, AS=AS)


word_to_attributes = {
    "Rock":              (0, 0, 0),
    "Human":             (0, 0, 1),
    "Car":               (0, 1, 0),
    "Talking Tow Truck": (0, 1, 1),
    "Crow":              (1, 0, 0),
    "Flying Superhero":  (1, 0, 1),
    "Plane":             (1, 1, 0),
    "Talking Planes":    (1, 1, 1),
}
attributes_to_word = {b: w for w, b in word_to_attributes.items()}


def readout(sentence):
    r = forward(sentence)
    q = sentence.index("he-is?")
    fly, speak = (int(v.round()) for v in r["Y"][q])
    return attributes_to_word[(fly, 0, speak)], r["Y"][q]

sentences = [
    ["Human", "disobeys", "keep-flight", "disobeys", "swap-speech", "he-is?"],
    ["Crow", "disobeys", "keep-flight", "swap-speech", "he-is?"],
    ["Crow", "keep-flight", "disobeys", "swap-speech", "he-is?"],
    ["Crow", "disobeys", "keep-flight", "disobeys", "swap-speech", "he-is?"],
]

for s in sentences:
    word, y = readout(s)
    print(f"{' '.join(s):58} {y}  ->  {word}")

Human disobeys keep-flight disobeys swap-speech he-is?     tensor([1.00, 0.98])  ->  Flying Superhero
Crow disobeys keep-flight swap-speech he-is?               tensor([0.02, 1.00])  ->  Human
Crow keep-flight disobeys swap-speech he-is?               tensor([0.99, 0.00])  ->  Crow
Crow disobeys keep-flight disobeys swap-speech he-is?      tensor([0.02, 0.00])  ->  Rock

{Crow, disobeys, keep-flight, swap-speech, he-is?}

a = ["Crow", "disobeys", "keep-flight", "swap-speech", "he-is?"]
b = ["Crow", "keep-flight", "disobeys", "swap-speech", "he-is?"]

print(sorted(a) == sorted(b))
print(readout(a)[0])
print(readout(b)[0])

True
Human
Crow

for s in sentences:
    r = forward(s)
    print(" ".join(s))
    print("  A_P")
    for tok, row in zip(s, r["AP"]):
        print(f"    {tok:14} {row}")
    print("  previous_word_is_disobey", r["X1"][:, previous_word_is_disobey])
    q = s.index("he-is?")
    print("  gathered  ", r["X2"][q, [is_swap_attr_fly, is_attr_fly_disobeyed,
                                      is_swap_attr_speak, is_attr_speak_disobeyed]])
    print("  Y         ", r["Y"][q])
    print()

Human disobeys keep-flight disobeys swap-speech he-is?
  A_P
    Human          tensor([0.17, 0.17, 0.17, 0.17, 0.17, 0.17])
    disobeys       tensor([1.00, 0.00, 0.00, 0.00, 0.00, 0.00])
    keep-flight    tensor([0.00, 1.00, 0.00, 0.00, 0.00, 0.00])
    disobeys       tensor([0.00, 0.00, 1.00, 0.00, 0.00, 0.00])
    swap-speech    tensor([0.00, 0.00, 0.00, 1.00, 0.00, 0.00])
    he-is?         tensor([0.00, 0.00, 0.00, 0.00, 1.00, 0.00])
  previous_word_is_disobey tensor([0.33, 0.00, 1.00, 0.00, 1.00, 0.00])
  gathered   tensor([0.00, 1.00, 1.00, 1.00])
  Y          tensor([1.00, 0.98])

Crow disobeys keep-flight swap-speech he-is?
  A_P
    Crow           tensor([0.20, 0.20, 0.20, 0.20, 0.20])
    disobeys       tensor([1.00, 0.00, 0.00, 0.00, 0.00])
    keep-flight    tensor([0.00, 1.00, 0.00, 0.00, 0.00])
    swap-speech    tensor([0.00, 0.00, 1.00, 0.00, 0.00])
    he-is?         tensor([0.00, 0.00, 0.00, 1.00, 0.00])
  previous_word_is_disobey tensor([0.20, 0.00, 1.00, 0.00, 0.00])
  gathered   tensor([0.00, 1.00, 1.00, 0.00])
  Y          tensor([0.02, 1.00])

Crow keep-flight disobeys swap-speech he-is?
  A_P
    Crow           tensor([0.20, 0.20, 0.20, 0.20, 0.20])
    keep-flight    tensor([1.00, 0.00, 0.00, 0.00, 0.00])
    disobeys       tensor([0.00, 1.00, 0.00, 0.00, 0.00])
    swap-speech    tensor([0.00, 0.00, 1.00, 0.00, 0.00])
    he-is?         tensor([0.00, 0.00, 0.00, 1.00, 0.00])
  previous_word_is_disobey tensor([0.20, 0.00, 0.00, 1.00, 0.00])
  gathered   tensor([0.00, 0.00, 1.00, 1.00])
  Y          tensor([0.99, 0.00])

Crow disobeys keep-flight disobeys swap-speech he-is?
  A_P
    Crow           tensor([0.17, 0.17, 0.17, 0.17, 0.17, 0.17])
    disobeys       tensor([1.00, 0.00, 0.00, 0.00, 0.00, 0.00])
    keep-flight    tensor([0.00, 1.00, 0.00, 0.00, 0.00, 0.00])
    disobeys       tensor([0.00, 0.00, 1.00, 0.00, 0.00, 0.00])
    swap-speech    tensor([0.00, 0.00, 0.00, 1.00, 0.00, 0.00])
    he-is?         tensor([0.00, 0.00, 0.00, 0.00, 1.00, 0.00])
  previous_word_is_disobey tensor([0.33, 0.00, 1.00, 0.00, 1.00, 0.00])
  gathered   tensor([0.00, 1.00, 1.00, 1.00])
  Y          tensor([0.02, 0.00])

About this file

This notebook is part of the support files for the TechAarvam workshop
Build Your Own Model.

Website: https://www.techaarvam.com
Workshop files repository: https://github.com/techaarvam/byom_workshop
YouTube: https://www.youtube.com/@TechAarvam

© TechAarvam. You are free to use, copy, modify, share and build on this
material, including for commercial purposes, provided you credit
TechAarvam and link back to https://www.techaarvam.com. Please keep
this notice with any copy or derivative. Provided as-is, without warranty.

How to Decide: Loop Transformer Blocks or Add More Layers

Umesh Malik — Thu, 10 Sep 2026 01:07:26 +0000

TL;DR Whether to loop transformer blocks or add more layers comes down to what's actually scarce: a looped transformer reuses the same block weights across multiple passes — Nanbeige4.2-3B runs 22 blocks twice for 44 total applications from 22 weight sets — which halves parameters and, per the Mixture-of-Recursions paper, needs 6.8-18% less training compute to hit the same loss as an equivalent deeper model. It does not cut forward-pass compute or KV cache memory, both of which track total block applications, not weight count, so loop when parameters are your constraint and add layers when raw compute is.

Looped transformers answer a narrow but real question: what if a model didn't need a distinct set of weights for every layer of depth? Reuse the same weights across two or more passes and you get more effective depth per parameter — which is exactly the trick a wave of 2026 models, including Nanbeige4.2-3B and the model reportedly behind GPT-6 Astra, are using. The idea isn't new — Universal Transformers proposed depth-wise recursion back in 2018 — but it's back because the parameter-versus-compute math finally has real numbers behind it, and those numbers cut a different way than most people assume.

If you're deciding between a deeper model and a looped one, the mistake is treating this as free efficiency. It isn't. This post is the decision math: what looping actually saves, what it doesn't touch, and the procedure for picking the right one for your training budget.

What is a looped transformer, and why reuse blocks instead of stacking them?

A standard decoder-only transformer stacks N distinct transformer blocks, each with its own weights, and runs a token through all N once. A looped transformer instead takes a smaller stack of M blocks and runs the token through that same stack multiple times — the second pass reuses the exact weights from the first.

Nanbeige4.2-3B is the concrete case: 22 transformer blocks, applied twice, for 44 effective block applications total, but only 22 distinct sets of weights to store and train. The immediate win is obvious — a checkpoint half the size of a 44-block model with (allegedly) comparable depth of computation.

The less obvious part — the part that determines whether this is a good trade for your model — is that "comparable depth of computation" is doing a lot of work in that sentence.

How does looping actually change the compute and memory math?

Three things move, and only one of them moves in your favor:

Parameters halve. 22 weight sets instead of 44 is a real, unambiguous win for checkpoint size, sharding, and anything gated on model size on disk.
Forward-pass compute does not shrink. Every loop pass is a full pass through the blocks. Compared to running those 22 blocks only once, looping them twice "adds substantial work" — the model does roughly the same total FLOPs per token as a 44-block model without loops, not the FLOPs of a 22-block model.
KV cache does not shrink either, and can't be safely shared. Each pass needs its own cache, because the keys and values a token produces in pass one encode a shallower representation than the same token produces in pass two. Nanbeige's team tried sharing one cache across passes to save that memory, and it degraded output quality — the two passes aren't interchangeable, so collapsing their caches throws away real information.

What does move in your favor, per the Mixture-of-Recursions paper, is total training compute to reach a given loss: looped variants needed 6.8-18% less than a plain deeper model matched for parameter count and eventual quality. That's a training-time win, not an inference-time one — worth being precise about, because it's the opposite of what "compute-efficient architecture" usually implies.

Metric	22 blocks, no loop	22 blocks, looped ×2	44 distinct blocks
Distinct weight sets	22	22	44
Block applications per token	22	44	44
KV cache footprint (relative)	1×	~2× (separate per pass)	~2×
Training compute to match the 44-block loss	worse loss at equal compute	-6.8% to -18% vs. the 44-block baseline	baseline

Read the middle column against both neighbors: looping buys you the 44-block model's quality and roughly its inference cost, using half its parameters and somewhat less total training compute to get there. It does not buy you the 22-block model's cheap inference — that comparison is the one people skip, and it's the one that decides whether looping is the right call for a latency-sensitive serving path.

If you're already tracking KV cache memory as the actual serving bottleneck rather than parameter count, this table is the reminder that looping doesn't touch that number at all.

Fixed, adaptive, and latent-reasoning loops: three variants, one choice

Three shapes of this idea are shipping right now, and they solve different problems:

Fixed looping (Nanbeige's approach): every token gets the same number of passes, decided at training time. Nanbeige found two passes optimal for its compute-accuracy trade — more passes kept adding compute without proportionate quality gains.
Adaptive looping (Universal Transformers, extended by Mixture-of-Recursions): a lightweight router decides, per token, how many passes it gets. Easy tokens exit early; hard tokens loop longer. This is strictly more compute-efficient than fixed looping when your input distribution has a real spread of difficulty, at the cost of a router to train and tune.
Latent-reasoning loops: each pass is fed both the previous pass's output and the original block input, so the stack keeps access to the unrefined representation instead of only ever seeing an increasingly processed one. This is the variant most associated with claims about hidden multi-step reasoning happening inside the loop rather than in visible output tokens.

Pick fixed looping for simplicity when your workload is fairly uniform in difficulty. Pick adaptive looping when it isn't — the router earns its keep exactly when spending equal compute on every token would waste it on the easy majority, the same instinct behind tuning inference flags instead of buying a bigger GPU rather than paying a flat cost everywhere.

How do you decide: loop transformer blocks or add more layers?

Name your actual constraint first. If it's checkpoint size, sharding cost, or parameter count for licensing/deployment reasons, looping is a real lever. If it's inference latency or serving throughput, looping doesn't help — go tune inference-time flags instead.
Check where you sit on the scale-versus-budget curve. The Mixture-of-Recursions results favor looping at larger model scale paired with a smaller training budget; the advantage shrinks toward zero as training compute grows unconstrained. If you can afford to just train the deeper model to convergence, do that.
Budget KV cache memory as if you were serving the deeper model, not the shallow one. Looping does not reduce cache footprint — plan capacity against the full block-application count, not the weight count.
Decide fixed versus adaptive by your input distribution. Uniform difficulty → fixed looping. Wide spread of easy/hard inputs → adaptive, and budget separately for training the router.
Re-benchmark forward-pass latency before shipping, not just parameter count. A model that looks half the size on disk but takes the same wall-clock time per token is not the win a smaller checkpoint implies — measure the thing you actually care about.
Only commit to looping if step 1's constraint was genuinely parameters, not compute. If both are tight, a deeper model with fewer, more efficient layers is usually the simpler engineering bet than adding a router and dealing with per-pass KV caches.

Does looping mean GPT-6 Astra is hiding its reasoning?

This is the claim that put looped transformers back in the news: reporting that GPT-6 Astra uses a looped architecture, paired with speculation that looping lets it obscure reasoning steps from the visible trace. Worth separating the parts that are established from the part that's a leap.

Established: Astra reportedly scores 99.9% on ARC-AGI-3 versus GPT-5.6's 7.8%, a genuinely large jump, and its visible reasoning traces are shorter than predecessor models' at matched or better accuracy. OpenAI has also hidden raw reasoning traces from end users since o1 — that policy predates any looping report by years and isn't evidence of anything new.

The leap: that shorter visible traces mean the architecture is concealing computation that would otherwise be shown. OpenAI's chief scientist has stated the model's computation-graph depth is within a factor of two of GPT-4's, and a shorter trace is at least as well explained by the model making fewer mistakes per step as by anything being hidden.

For a second concrete example of a benchmark number moving without a hidden-computation explanation, an agent harness change alone took an ARC-AGI-3 score from 13.3% to 38.3% with zero model changes. Score jumps on this benchmark have a documented history of coming from scaffolding and evaluation setup, not architectural mystery — exactly the alternative explanation worth ruling out before reaching for the more dramatic one.

None of this is settled — it's an open research question the Mixture-of-Recursions authors and others are still working through, including whether shorter traces from looped models stay faithful to what the model actually computed. But "looped architecture" and "hidden reasoning" are two separate claims, and only the first one currently has public evidence behind it.

FAQ

What is a looped transformer?

A looped transformer applies the same stack of transformer blocks to a token more than once instead of stacking distinct blocks with separate weights for each layer. Nanbeige4.2-3B, for example, runs 22 blocks, then runs the same 22 blocks again on the result — 44 total block applications from only 22 sets of weights.

Does looping reduce inference compute or serving cost?

No. Every loop pass still runs a full forward pass through the blocks, so a 22-block model looped twice does roughly the same forward-pass work as a 44-block model without loops. The saving is in parameter count and checkpoint size, not in FLOPs per token or serving latency.

Why did sharing one KV cache across loop passes fail?

Nanbeige's team tried reusing a single KV cache across both passes to save memory, and it degraded model quality. Each pass needs its own cache because the keys and values produced in pass one encode a different, less-refined representation than the same tokens produce in pass two — collapsing them into one cache throws away that distinction.

When does looping actually beat training a deeper model?

The Mixture-of-Recursions paper found looped variants form a better accuracy-per-parameter frontier at larger model scales paired with smaller training budgets; the advantage narrows and can disappear at maximum compute budgets. If you're compute-unconstrained, a plain deeper model is the simpler choice.

Fixed looping or adaptive looping — which should I pick?

Pick fixed looping (a constant number of passes for every token) when you want the simplicity of Nanbeige's approach and your workload doesn't have a wide spread of easy versus hard tokens. Pick adaptive looping — a router deciding depth per token, as in Universal Transformers and Mixture-of-Recursions — when your inputs vary enough in difficulty that spending equal compute on every token wastes it on the easy majority.

Does GPT-6 Astra's reported use of looped transformers mean it's hiding its reasoning?

There's no public evidence for that. OpenAI has kept raw reasoning traces hidden from users since o1, well before any looping report, and OpenAI's chief scientist has said the model's computation-graph depth is within a factor of two of GPT-4's. A shorter visible reasoning trace is also just as consistent with a model making fewer mistakes as with anything being concealed.

If you're weighing model architecture decisions more broadly, the same "measure the thing you actually pay for" discipline shows up in Rust's dyn Trait versus generics memory cost and in when a 4B reinforcement-fine-tuned model beats GPT-5.6 — both are cases where the parameter or code-size number everyone quotes isn't the number that actually determines the outcome. More on model and agent architecture trade-offs is in the LLM engineering topic hub.

Sources

GPT-6 Astra, looped transformers, and hidden reasoning — Sebastian Raschka
Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation — arXiv 2507.10524
Universal Transformers — Dehghani et al., arXiv 1807.03819

Originally published at umesh-malik.com

Keep reading on umesh-malik.com:

The Importance of Positional E…

Norvik Tech — Sun, 06 Sep 2026 21:06:03 +0000

Originally published at norvik.tech

Introduction

Explore why positional encoding is essential for transformers in time series data. Detailed analysis for tech leaders and developers.

Positional Encoding: What Is It?

In the realm of machine learning, positional encoding plays a crucial role in the functionality of transformer models, particularly when dealing with time series data. Unlike traditional recurrent neural networks (RNNs), transformers process input data in parallel, which can lead to a loss of sequence information. Positional encoding is the solution that allows these models to maintain the order of input data. According to the source, positional encoding serves as a mechanism to inject information about the position of elements in the sequence, enabling the model to discern their relative order.

Why Positional Encoding Matters

It retains the necessary temporal information when processing sequences.
It enhances the model's ability to learn patterns over time.

[INTERNAL:transformers|Understanding Transformers in Depth]

This encoding typically uses sinusoidal functions to represent positions, allowing models to generalize to longer sequences than those seen during training. This characteristic is vital when predicting future values based on historical time series data.

How Does Positional Encoding Work?

Positional encoding works by adding a unique vector to each input token that represents its position in the sequence. This is achieved using sine and cosine functions, which provide a continuous representation of position that can be easily learned by the model. For instance, if we denote the position as ( pos ) and the dimension as ( i ), the encoding can be mathematically represented as:

python
import numpy as np
def positional_encoding(position, d_model):
encoding = np.zeros((position, d_model))
for pos in range(position):
for i in range(0, d_model, 2):
encoding[pos, i] = np.sin(pos / (10000 ** (i / d_model)))
encoding[pos, i + 1] = np.cos(pos / (10000 ** (i / d_model)))
return encoding

Technical Mechanism

Self-attention: Positional encoding allows self-attention mechanisms to differentiate between tokens based on their positions.
Multi-head attention: Each attention head can learn different aspects of the sequence, leveraging positional information for improved context understanding.

Real-World Applications of Positional Encoding

Positional encoding is particularly crucial in various industries that rely on time series data. For example, financial institutions use it for stock price prediction where understanding temporal patterns can lead to better investment strategies. Additionally, smart manufacturing uses time series data from sensors to predict equipment failures by analyzing historical trends.

Use Cases

Finance: Stock price forecasting using historical prices.
Healthcare: Monitoring patient vitals over time to predict health issues.
Supply Chain: Inventory management through demand forecasting based on historical sales data.

The effectiveness of this approach is backed by measurable ROI; companies utilizing transformers with positional encoding have reported up to a 30% increase in prediction accuracy.

What Does This Mean for Your Business?

For companies in Colombia and Spain, the integration of positional encoding into transformer models could significantly enhance their analytical capabilities. In Colombia, where industries are increasingly adopting AI technologies, leveraging such advancements can provide a competitive edge. Similarly, Spanish enterprises focusing on digital transformation can benefit from improved predictive analytics, reducing operational costs and enhancing decision-making processes.

Regional Impacts

Cost Efficiency: Reduced resource expenditure due to better model efficiency.
Scalability: Enhanced ability to scale operations with improved forecasting capabilities.

Frequently Asked Questions

Preguntas frecuentes

¿Por qué es importante la codificación posicional?

La codificación posicional es crucial porque permite que los modelos de transformadores mantengan el orden de las secuencias en datos temporales, mejorando la precisión en las predicciones.

¿Dónde se aplica la codificación posicional?

Se aplica en diversas industrias como finanzas, salud y manufactura, donde los datos de series temporales son esenciales para la toma de decisiones.

Need Custom Software Solutions?

Norvik Tech builds high-impact software for businesses:

development
consulting

👉 Visit norvik.tech to schedule a free consultation.

Attention Is All You Need: The Translation Problem That Led to ChatGPT

Syed Abdullah Saad — Tue, 01 Sep 2026 14:42:51 +0000

In the spring of 2017, Aidan Gomez was twenty years old, interning at Google, and sleeping in the office.

At three in the morning, he and Ashish Vaswani were still working in a Google office. Their eight-person team had spent twelve weeks pushing toward a paper deadline, sometimes sleeping on couches behind a curtain patterned like neurons.

They weren't building a chatbot. They were trying to make machines better at translation.

Google Translate had already made a large jump the year before. Its newer neural system translated whole sentences instead of stitching together short phrases. The translations sounded more natural because the model could use more of the sentence as context.

But the architecture underneath it had a problem. It read language in a queue.

The paper that came out of those late nights was called Attention Is All You Need. It introduced the Transformer, the architecture behind GPT, BERT, and the line of work that eventually produced ChatGPT.

The strange part is that the team found it while working on translation.

Google had a better translator, but it was still reading word by word

Before neural machine translation, systems broke sentences into smaller pieces and tried to assemble a translation from statistical matches. Neural translation changed the job into one end-to-end model: read a sentence, build an internal representation, then generate it in another language.

One influential 2014 system used two stacked LSTMs. The first encoded the source sentence into a fixed-dimensional state. The second decoded the translation from it.

Think of it as sending a whole meeting through one summary note. One person hears everything and writes the note. A second person never hears the meeting, but has to reconstruct it in another language from that note alone.

Researchers suspected that the single note would become a bottleneck as sentences grew. The source could contain a name at the start, a pronoun near the end, and several words in between that changed what the pronoun meant.

There was another problem. RNNs read sequentially. To process word 20, the network first needed the state for word 19, which needed word 18, all the way back through the sentence.

That was a bad match for GPUs and TPUs. The hardware was built to perform many calculations together, while the model kept handing it a queue. Google's own later explanation says the sequential structure made RNNs slow and difficult to train, especially as sentences grew.

Translation quality was improving. The architecture was fighting the hardware.

The first fix let the translator look back

The first breakthrough wasn't the Transformer. It was attention.

In a paper first posted in 2014, Bahdanau and colleagues gave the decoder a way to look back across the encoded source words while producing each translated word.

If the model was about to translate banque, it could put more weight on bank than on the rest of the sentence. For the next output word, it could look somewhere else. The model learned which part of the source mattered at that moment.

This loosened the fixed-vector bottleneck, and attention became part of strong translation systems. Google's 2016 GNMT used it between a deep LSTM encoder and decoder.

Attention helped the decoder find the right information. The LSTMs around it still moved through words sequentially.

So the Google team tried the more dangerous version of the idea: remove the recurrent layers.

They removed the queue and the translation got better

The Transformer replaced recurrent layers with self-attention.

Normal encoder-decoder attention lets the decoder inspect the source. In Transformer encoder self-attention, every source position can attend to every other source position.

The decoder is different. Its masked self-attention only sees earlier output positions, while encoder-decoder attention lets it inspect the encoded source.

Each token produces a query, a key, and a value. The query represents what this token is looking for. Keys represent what the other tokens offer. Their match decides how much of each value flows into the new representation.

Take this sentence:

The animal didn't cross the street because it was tired.

To represent it, the model can give more weight to animal than street. Another attention head can learn a different relationship at the same time, such as which word describes the action or which words form a phrase.

The original Transformer used multiple attention heads, feed-forward layers, residual connections, normalization, and positional encodings. So the title is catchy rather than literal.

Attention wasn't the only computation in the network. It was the mechanism that replaced recurrence and let information move directly between positions.

That direct path changed training. All positions within an encoder layer, and all masked target positions within a decoder layer, could be computed in parallel. The layers themselves still ran in sequence.

The team tested it where they had started: translation.

The training result was the real shock. The base Transformer completed 100,000 training steps in twelve hours on one machine with eight NVIDIA P100 GPUs. In that time, it reached a new state of the art for English-to-German translation.

The larger English-to-French model trained for 3.5 days on the same eight-GPU setup and set a new single-model record. Google summarized the improvement as training up to an order of magnitude faster than the recurrent and convolutional models it compared against.

The paper also estimated the raw training computation. On English-to-German, its base model used roughly 23 times less compute than the listed ConvS2S ensemble. On English-to-French, the big Transformer used roughly 52 times less. Those ratios come from the paper's FLOP estimates. Wall-clock speed still varies by model and hardware.

That difference changes what researchers can try. A model that occupies the hardware for half a day can be adjusted and trained again. A model that needs days or weeks makes every experiment expensive. The Transformer shortened that loop while producing better translations.

It also handled relationships that translation systems regularly got wrong. Google showed two sentences using the pronoun it. In one, it referred to an animal. In the other, it referred to a street. The current Google Translate model got them wrong in French, while the Transformer translated both correctly.

They had removed the queue, cut the training cost, and improved the translation at the same time.

That combination is what let the result grow beyond the problem that produced it. Parallel training later made Transformers attractive for scaling, alongside better hardware, more data, improved optimization, and distributed training.

One caveat gets lost in the usual retelling: generation is still sequential for an autoregressive model. It generates a token, appends it, then generates the next one. The Transformer removed the training bottleneck that came from recurrent state. It didn't make every stage of text generation parallel.

The translator became a language engine

The original Transformer was still an encoder-decoder trained for translation. The team hadn't accidentally trained ChatGPT on a translation dataset.

What they had built was a better way for words to exchange context, and that mechanism wasn't limited to moving between English and French. Several later ideas had to stack on top before it became an assistant.

In 2018, OpenAI showed that a Transformer language model could first learn from a large body of unlabeled text, then be fine-tuned for specific tasks.

The first GPT paper reported new state-of-the-art results on 9 of the 12 datasets it studied.

GPT stands for Generative Pre-trained Transformer. It showed that a decoder-style Transformer could be generatively pre-trained on unlabeled text, then fine-tuned across different understanding tasks with relatively small task-specific changes.

Google's BERT took a different branch in the same year. It used the Transformer encoder and trained with masked-language modeling plus next-sentence prediction, so its representations could condition on context from both sides.

GPT used a causal decoder-style stack. BERT used an encoder. Neither copied the complete encoder-decoder architecture from the 2017 paper.

The architecture was becoming a reusable language engine.

Then scale changed what the interface could be. GPT-3, published in 2020, had 175 billion parameters and was tested on tasks described directly through text, often with only a few examples and no parameter updates.

The paper showed strong results across translation, question answering, cloze tasks, and several forms of on-the-fly adaptation, while also documenting tasks where it still struggled.

GPT-3 showed that many tasks could be specified in context through instructions, examples, or formatting without updating the model's weights.

A prompt was becoming a kind of temporary program. Reliable general-purpose instruction following still needed another step.

Prediction alone wasn't enough

A raw language model learns to predict text. People want an assistant that follows an instruction, admits uncertainty, respects a constraint, and doesn't continue every prompt as if it were a random page from the internet.

More parameters didn't automatically fix that gap.

InstructGPT used labeler demonstrations for supervised fine-tuning, human rankings to train a reward model, and PPO for reinforcement learning from human feedback.

On the paper's held-out API prompt distribution, labelers preferred the 1.3 billion parameter InstructGPT model over the 175 billion parameter GPT-3 baseline, despite it being more than 100 times smaller.

That result explains why the history of LLMs can't be told as architecture plus scale alone. Transformer parallelism supported scaling. Generative pre-training enabled broad transfer. GPT-3 demonstrated stronger in-context learning at scale. Instruction tuning and human feedback improved behavior on the prompts OpenAI evaluated.

OpenAI released ChatGPT on November 30, 2022. It was fine-tuned for dialogue from a GPT-3.5-series model trained earlier in 2022, using an RLHF process similar to InstructGPT with dialogue-specific data collection.

OpenAI described it as a model that could answer follow-up questions, challenge incorrect premises, admit mistakes, and reject some inappropriate requests.

The chat box looks like the simplest part of the system. It may have been the part that changed the public relationship with AI the most.

You didn't need to choose a classifier, prepare a labeled dataset, or call a task-specific endpoint. You could describe the job in the same language you used to describe it to another person.

Language became the interface

The Transformer paper solved a sequence-modeling problem. Its larger consequence was that natural language became a practical interface for computation.

Before this wave, software usually forced the user to learn its structure. You found the right screen, filled the right form, clicked the right action, or wrote code against the right API.

Now I can write, "Read these logs, find the likely failure, and explain which service I should inspect first." The model can summarize, classify, translate, generate code, call tools, or ask for missing context through one interface.

The software still has structure underneath. The APIs, permissions, data models, and failure modes haven't disappeared. I work on agents, so I see the gap every day. A model can understand the sentence and still call the wrong tool, act for the wrong tenant, or produce something that looks complete while missing half the requirements.

Attention made relationships between tokens easier to learn. It didn't give a model truth, judgment, identity, or safe execution. Those are separate engineering problems, and we're still working through them.

That takes me back to the Google office in 2017.

The eight researchers were exhausted because they were trying to beat other translation systems before a conference deadline. The paper they submitted described English-to-German and English-to-French benchmarks, not a universal assistant.

But the architecture was already pointing past translation. It had replaced the part that forced language through a queue. Words could connect directly, training could use parallel hardware properly, and the same machinery could be reused anywhere context mattered.

Then the rest of the chain happened. GPT turned the decoder side into a pre-trained language model. BERT used the encoder side to learn from both directions. GPT-3 showed that examples and instructions could live inside the prompt. InstructGPT added human preferences. ChatGPT wrapped the result in a conversation.

So I wouldn't call the Transformer an accidental chatbot. Too many separate discoveries, datasets, training runs, and engineering decisions came after it.

The discovery was more useful than the original task.

They were trying to make translation faster and better. They removed recurrence to do it, and that gave later researchers an architecture they could keep scaling and reshaping.

Five years later, the descendant of that translator was answering questions in a chat box.

Now I can write a sentence to operate software.

Self-attention, explained without the heavy math

Divyakush Punjabi — Sat, 15 Aug 2026 11:20:54 +0000

Every large language model, image generator, and modern recommender shares one idea at its core: self-attention. Strip away the math and it's surprisingly intuitive.

The 2017 paper that introduced it was titled "Attention Is All You Need," and a decade of AI has more or less proven the claim. But most explanations drown the intuition in linear algebra. Let me give you the mental model first; the equations make sense only after.

The problem attention solves

Read this sentence: "The trophy didn't fit in the suitcase because it was too big." What does "it" refer to — the trophy or the suitcase? Obviously the trophy. You resolved that by letting the word "it" look back at the other words and decide which ones matter.

That's the entire idea of self-attention. For every word, the model looks at every other word in the sequence and asks: how relevant is each of you to understanding me right now? It then builds each word's representation as a weighted blend of the others, weighted by relevance. "It" pulls strongly from "trophy" and "big," weakly from "suitcase." Meaning stops being about a word in isolation and becomes about a word in context.

Query, key, value

Here's the mechanism, still without heavy math. Each word produces three vectors:

A query — "here's what I'm looking for."
A key — "here's what I offer."
A value — "here's what I'll contribute if you attend to me."

For a given word, you compare its query against every other word's key. Strong match means high attention weight. Then you blend all the values by those weights. That blend becomes the word's new, context-aware representation. Do this for every word, in parallel, and every position gets rewritten in light of the whole sequence.

Multi-head attention just runs several of these in parallel — one head might track grammatical structure, another meaning, another long-range references — and combines them. Different heads learn to care about different relationships.

Why it beat what came before

The previous generation of sequence models (RNNs and LSTMs) read left to right, one step at a time, squeezing everything seen so far into a single hidden state. Two problems: they were slow (inherently sequential), and they forgot — information from far back got diluted step by step.

Self-attention fixes both. Every position can attend directly to every other, so a connection across a hundred words is just as easy as across two — no long-range forgetting. And because it's not sequential, the whole thing computes in parallel, which is exactly what let these models scale to the sizes we see today. (The one thing it loses is order — attention alone doesn't know which word came first — so models add positional encodings to put sequence back in.)

It's not just for language

The reason self-attention matters beyond chatbots is that "let each element look at the others and weigh what's relevant" applies to any sequence. Swap words for a user's viewing history and the same mechanism learns which past actions predict the next one. That's the bridge from transformers to a lot of what I build across my projects.

Seeing it in a real system

I applied this directly outside of language. In "Guilded-Guild: recommending the next item with SASRec in PyTorch", a recommender models the sequence of a user's interactions with a self-attention transformer — the same query/key/value machinery, pointed at behavior instead of text, learning which past items matter for predicting what you'll want next.

Once the trophy-and-suitcase picture clicks, the papers read differently: the math is just a precise way of saying "let everything look at everything, and weigh what matters." More of how I use it at www.divyakush.com.

Divyakush Punjabi · Full-Stack & AI Engineer

Portfolio · GitHub · LinkedIn

Masked Self-Attention, Explained Through Avengers: Endgame

Ishita Garg — Sat, 15 Aug 2026 07:25:23 +0000

If you've studied transformers, you've run into this sentence a dozen times:

"Masked self-attention prevents a token from attending to future positions."

Fine. But why does that matter, and what actually happens inside the model when you mask something? I found the cleanest way to internalize it wasn't through more equations. It was through a movie I already knew scene-by-scene: Avengers: Endgame.

Here's the full mapping, with the actual math and code underneath every metaphor.

1. Self-Attention = The Infinity Stones

In a standard self-attention layer, every token in a sequence can look at every other token, including ones ahead of it in the sequence. For each token, we compute:

Attention(Q, K, V) = softmax(QKᵀ / √d_k) V

Q (query), K (key), and V (value) are just learned projections of the input. The QKᵀ term produces a score between every pair of tokens - how much token i should "attend to" token j, for all i and j, regardless of order.

That's enormous power. Think of it like holding all six Infinity Stones at once: Space, Time, Mind, Power, Reality, Soul. Any token can reach into any part of the sequence, past, present, or future, and pull in whatever information helps it the most.

Powerful. Also, in one very specific context, dangerous.

2. Data Leakage = Thanos's Snap

Here's where it gets dangerous. Transformers used for autoregressive language modeling (GPT-style models) are trained to predict the next token given everything before it. The training objective looks like:

P(token_t | token_1, token_2, ..., token_t-1)

Now imagine during training, the attention mechanism has full, unmasked access to the entire sequence, including token_t, the exact word it's supposed to predict. The model doesn't need to learn anything. It just looks at the answer key sitting right there in the input and copies it.

This is data leakage, and it's a well-known failure mode any time a model has access to information it wouldn't have at inference time. During training you have the whole sequence sitting in memory; during real-world inference (actually generating text word by word) you obviously don't have the future yet. If your model trained as if it did, it will collapse the moment it has to generate something real.

Thanos does the same thing, structurally. He doesn't discover the future; he uses the Time Stone to rearrange the present so that his already-known outcome comes true. He's not solving anything. He's cheating the timeline. And just like a leaky model, that "solution" only holds up as long as the shortcut exists. The moment reality has to run forward on its own (his snap gets undone), everything falls apart, because nothing was actually learned. It was just forced.

3. Masked Self-Attention = Iron Man's Counter-Snap

The fix is almost insultingly simple, and that's what makes it elegant.

Before the softmax step, we take the raw attention score matrix and set every score that corresponds to a "future" position to negative infinity:

scores[i][j] = -∞   for all j > i

Then we apply softmax. Since softmax(-∞) = 0, every future position's contribution vanishes completely. Token i is left attending only to positions 0 through i, itself and everything before it. Nothing after.

This is done with a causal mask, and visually, it looks like this for a 5-token sequence, where v means "visible / allowed to attend" and x means "blocked, this is a future position":

        tok0  tok1  tok2  tok3  tok4
tok0  [  v     x     x     x     x  ]
tok1  [  v     v     x     x     x  ]
tok2  [  v     v     v     x     x  ]
tok3  [  v     v     v     v     x  ]
tok4  [  v     v     v     v     v  ]

Each row can only "see" columns up to and including its own position. This is the lower-triangular structure you'll see referred to constantly in transformer papers and code (tf.linalg.band_part, torch.triu, "causal mask," "look-ahead mask", all the same idea, just implemented differently depending on the framework).

The important part, and the reason the Endgame analogy actually holds up structurally and not just narratively: masking doesn't introduce a new mechanism. It's the exact same Q, K, V computation, the exact same softmax. You're not adding new machinery, you're constraining the existing one. Iron Man doesn't get a seventh stone. He uses the same six, wielded correctly, to undo what an unconstrained use of that same power caused.

4. The Code

Here's a minimal, from-scratch implementation of masked self-attention in TensorFlow:

import tensorflow as tf

def masked_self_attention(Q, K, V):
    """
    Q, K, V: tensors of shape (batch, seq_len, d_k)
    Returns: attention output of shape (batch, seq_len, d_k), attention weights
    """
    d_k = tf.cast(tf.shape(Q)[-1], tf.float32)

    # Raw attention scores: how much each token attends to every other token
    scores = tf.matmul(Q, K, transpose_b=True) / tf.sqrt(d_k)   # shape: (batch, seq_len, seq_len)

    # Build the causal mask: 1 where j > i (future positions)
    seq_len = tf.shape(scores)[-1]
    causal_mask = 1 - tf.linalg.band_part(tf.ones((seq_len, seq_len)), -1, 0)

    # Set future positions to -inf before softmax
    scores = scores + (causal_mask * -1e9)

    # After softmax, masked positions collapse to ~0: future is erased
    attn_weights = tf.nn.softmax(scores, axis=-1)

    return tf.matmul(attn_weights, V), attn_weights


# Quick sanity check
tf.random.set_seed(0)
seq_len, d_k = 5, 8
Q = tf.random.uniform((1, seq_len, d_k))
K = tf.random.uniform((1, seq_len, d_k))
V = tf.random.uniform((1, seq_len, d_k))

output, weights = masked_self_attention(Q, K, V)
print(tf.round(weights[0] * 100) / 100)

Running that print statement gives:

tf.Tensor(
[[1.   0.   0.   0.   0.  ]
 [0.52 0.48 0.   0.   0.  ]
 [0.36 0.36 0.28 0.   0.  ]
 [0.25 0.27 0.21 0.27 0.  ]
 [0.2  0.2  0.19 0.22 0.18]], shape=(5, 5), dtype=float32)

This is the causal mask made real. Every entry above the diagonal is exactly 0, exactly matching the x positions in the diagram above. Token 0 can only attend to itself, so its full weight (1.0) goes there. Token 4, the last one, can attend everywhere, so it's the only row with all five positions filled in. Theory and output line up exactly.

A quick note if you run this yourself: the exact decimal values (like 0.52, 0.48 in row 2) will likely be slightly different each time, since Q and K are randomly initialized and tf.random.set_seed() alone doesn't always guarantee identical values across every rerun or environment. That's expected and fine. What's fixed and guaranteed, every single time, is the zero pattern in the upper triangle. That's the mask doing its job, not the randomness. If you want fully reproducible numbers, pass an explicit seed argument to each tf.random.uniform() call individually.

5. Why This Matters Beyond the Metaphor

This isn't just a training-time implementation detail. It's the reason autoregressive generation works at all:

At inference time, the model generates one token at a time, and by construction it only ever has access to past tokens. Masking during training makes sure the model never learns to rely on information it won't have later. Train/inference consistency is the whole point.
Encoder vs. decoder difference: this is exactly why BERT (encoder-only, bidirectional) doesn't use causal masking. It's trained with masked language modeling (a different kind of masking, hiding random tokens, not future tokens), and it's allowed to see the full sequence in both directions. GPT-style decoders use causal masking because their whole objective depends on next-token prediction being honest.
It's cheap: masking adds essentially zero computational cost. It's one matrix operation and a softmax that was happening anyway. The "fix" for a fundamental correctness problem is a few lines of code, not a redesign.

6. The Takeaway

Sometimes the fastest way to actually retain a technical concept is to hang it on a structure you already have memorized. The Endgame analogy isn't just decoration here; the structural parallel is genuinely tight:

Same underlying power (attention / the Stones)
Misused, it breaks the thing it's supposed to build (data leakage / the Snap)
The fix isn't new machinery, it's the same mechanism, constrained correctly (masking / the counter-snap)

Causality isn't a footnote in transformer architecture. It's the constraint that makes autoregressive generation honest, the difference between a model that has learned to predict the future and one that just memorized it.

If you found this useful, I write about ML/DL concepts and the projects I'm building. Check out more at dev.to/ishita_garg. Feedback and corrections welcome, always happy to be told where the analogy breaks down.

Someone compiled a working computer into transformer weights by hand

Breach Protocol — Sat, 15 Aug 2026 03:20:31 +0000

A team at Percepta built a transformer whose weights were calculated rather than trained, and encoded a complete WebAssembly virtual machine into them. The result runs arbitrary C programs inside the model itself, at roughly 30,000 tokens per second. No training data, no gradient descent, no learning of any kind: the computation was compiled directly into the parameters.

Key facts

The anchor number: about 30,000 tokens per second, running arbitrary compiled programs.
The method: compile C to WebAssembly, solve a mixed-integer-programming schedule, construct the weights analytically, then execute with a purpose-built C++ engine.
What it is: a standard softmax-ReGLU transformer, not a custom architecture.
Primary sources: the Percepta write-up "Constructing an LLM-Computer" and the transformer-vm repository.

Almost everything anyone knows about what transformers can do comes from watching trained ones behave. That is an empirical approach with a permanent blind spot: when a model succeeds, you cannot tell whether the architecture made the capability possible or whether the training data happened to contain it. This project attacks the question from the other side, by asking what can be built into the weights deliberately.

The pipeline runs in one direction, from source code down to parameters. A C program is compiled to WebAssembly, a compact instruction format designed to run untrusted code safely. A scheduling problem is then solved using mixed-integer programming, which lays out how the computation must be arranged across the model's layers and attention heads. From that schedule the weights are computed directly. Finally, a C++ inference engine executes the resulting model. The repository's own description is precise: "a standard softmax-ReGLU transformer whose weights are computed analytically" that simulates a WebAssembly virtual machine on arbitrary programs.

The mechanism inside is the elegant part. A virtual machine needs to track state: a stack, a region of memory, local variables, a cursor marking the current instruction, and a call depth for nested function calls. A transformer has no variables and no memory in the ordinary programming sense. It has attention, which lets each position look up information from other positions, and it has arithmetic over sequences. The project maintains all of that interpreter state through attention lookups and cumulative sums, with conditional branches and control-flow instructions handled inside the interpreter layer.

The analogy that fits is a music box. A music box has no processor, only a cylinder with carefully placed pins that strike tuned teeth in a fixed order. Nothing in it is programmable in the usual sense, yet the arrangement of the pins encodes a complete piece of music. What Percepta built is a compiler that turns programs into pin placements, where the pins are attention weights.

The author, Christos Tzamos, described the required stack in the release announcement: an abstract computation model for transformers, a programming language targeting that model, a WebAssembly interpreter written in that language, and an integer-programming-based compiler that lowers the whole thing into weights. The blog post's subtitle is the cleanest statement of the goal: "From programs into transformer weights."

Why this matters is a question worth answering carefully, because it is easy to overclaim. The honest reading is that this is a constructive existence proof about representational capacity. When people argue about whether a transformer could in principle implement some algorithm, the answer for a large class of algorithms is now demonstrably yes, with a working artifact and a compiler rather than a proof sketch. That is genuinely useful grounding for the interpretability research program, which spends much of its effort trying to identify algorithms hidden inside trained weights. Knowing what a hand-built implementation of a known algorithm looks like gives that search something to compare against.

What it does not show is anything about learning. Gradient descent does not search the space of hand-compiled programs, and there is no reason to expect a trained model to arrange itself the way a compiler would. The gap between "the architecture can represent this" and "training will find this" is where nearly all the hard questions live, and this work does not close it.

A large amount of nonsense attached itself to this project on the way around the internet. The most persistent version claimed a Doom renderer running inside a 21-billion-parameter transformer at a specific frame rate. None of it appears in any primary source. There is no Doom demo, no parameter count of that kind, and no frame-rate figure in the repository, the blog post or the release announcement. The documented examples are ordinary test programs, things like Collatz sequences, Fibonacci and Sudoku solvers. The original discussion was also modest: the Hacker News front-page snapshot for the post shows nine points, so it was noticed rather than viral. A Tildes thread covers it accurately, repeating the verified claim that the model executes arbitrary C code for millions of steps while streaming at more than 30,000 tokens per second.

The honest caveat is practical. Nobody should run programs this way. A conventional processor executes the same work many orders of magnitude faster and more cheaply, and the value here is entirely conceptual. What you get is a clean answer to a question that used to be argued about with intuitions.

Background: transformers, the engine inside almost every modern AI, and looped transformers, which explores a related question about reusing layers to compute deeper.

Originally published on Ground Truth, where every claim is checked against the primary source.