DEV Community: Tech-Aarvam

Puzzle Solution Revealed - Transformer: Need for Position Embedding

Tech-Aarvam — Tue, 15 Sep 2026 13:05:50 +0000

Introduce position embedding, residuals and layers

import torch

torch.set_printoptions(precision=2, sci_mode=False, linewidth=160)

In this notebook we are extending the previous notebook, attention_ann.ipynb, which introduced (plausibly) the world's tiniest hand-constructed transformer model. Please read the previous notebook for the context. This is the solution to the puzzle that was introduced as part of the previous notebook.

We add one word 'disobeys' which modifies the action-attribute words (swap-speech, keep-speech, swap-flight, keep-flight). In the previous notebook the order of the words did not matter. In the current notebook, the order does matter. The word disobeys modifies only the word right after it, so word order matters.

The solution adds an extra layer, and a residual connection that carries the input unmodified to the next block, so each next layer gets both the unmodified input and the modified input.

The residual stream is made 22 bits, where it carries the original inputs and each layer's/head's findings. The most interesting addition is the position information. Each token carries the position where it appears.

This example is constructed to illustrate the ideas. In the real implementation, we do not hand-construct in this manner. Vector embedding and position embedding are often also learned using the training loop and gradient descent. But the hand-construction allows us to see why those blocks and connections exist and how they are helpful to have.

Residual layout

idx	slot
0	`fly`
1	`speak`
2	`swap_fly`
3	`swap_speak`
4	`object`
5	`action_fly`
6	`action_speak`
7	`question`
8	`disobey`
9	`pos0`
10	`pos1`
11	`pos2`
12	`pos3`
13	`pos4`
14	`pos5`
15	`previous_word_is_disobey`
16	`object_attribute_fly`
17	`object_attribute_speak`
18	`is_swap_attr_fly`
19	`is_attr_fly_disobeyed`
20	`is_swap_attr_speak`
21	`is_attr_speak_disobeyed`

# The new list of token bits, with the additional word 'disobeys'
idx = {"fly": 0, "speak": 1, "swap_fly": 2, "swap_speak": 3,
       "object": 4, "action_fly": 5, "action_speak": 6, "question": 7, "disobey": 8}

# constants used for bit-slicing and locating the portion of
# the residual we need

# L is the maximum sentence length, i.e. the number of position slots.
# It is used to name the position slots below, and to build the
# "Disobey Position Finder" head (Wq_P, Wk_P) in layer 1.
position_start, L = 9, 6             # position one-hot occupies 9 .. 14
previous_word_is_disobey = 15
object_attribute_fly, object_attribute_speak = 16, 17
is_swap_attr_fly, is_attr_fly_disobeyed = 18, 19
is_swap_attr_speak, is_attr_speak_disobeyed = 20, 21
num_bits = 22

slot_names = {v: k for k, v in idx.items()}
slot_names.update({position_start + p: f"pos{p}" for p in range(L)})
slot_names.update({previous_word_is_disobey: "previous_word_is_disobey",
                   object_attribute_fly: "object_attribute_fly",
                   object_attribute_speak: "object_attribute_speak",
                   is_swap_attr_fly: "is_swap_attr_fly",
                   is_attr_fly_disobeyed: "is_attr_fly_disobeyed",
                   is_swap_attr_speak: "is_swap_attr_speak",
                   is_attr_speak_disobeyed: "is_attr_speak_disobeyed"})

for i in range(num_bits):
    print(f"{i:2} {slot_names[i]}")

 0 fly
 1 speak
 2 swap_fly
 3 swap_speak
 4 object
 5 action_fly
 6 action_speak
 7 question
 8 disobey
 9 pos0
10 pos1
11 pos2
12 pos3
13 pos4
14 pos5
15 previous_word_is_disobey
16 object_attribute_fly
17 object_attribute_speak
18 is_swap_attr_fly
19 is_attr_fly_disobeyed
20 is_swap_attr_speak
21 is_attr_speak_disobeyed

Vocabulary

token_to_vector = {
    #                   fly spk swF swS  obj actF actS  q  dis
    "Rock":             [0,  0,  0,  0,   1,  0,  0,  0,  0],
    "Human":            [0,  1,  0,  0,   1,  0,  0,  0,  0],
    "Crow":             [1,  0,  0,  0,   1,  0,  0,  0,  0],
    "Flying superhero": [1,  1,  0,  0,   1,  0,  0,  0,  0],
    "swap-flight":      [0,  0,  1,  0,   0,  1,  0,  0,  0],
    "swap-speech":      [0,  0,  0,  1,   0,  0,  1,  0,  0],
    "keep-flight":      [0,  0,  0,  0,   0,  1,  0,  0,  0],
    "keep-speech":      [0,  0,  0,  0,   0,  0,  1,  0,  0],
    "disobeys":         [0,  0,  0,  0,   0,  0,  0,  0,  1],
    "he-is?":           [0,  0,  0,  0,   0,  0,  0,  1,  0],
}

for tok, bits in token_to_vector.items():
    print(f"{tok:18} {torch.tensor(bits)}")

Rock               tensor([0, 0, 0, 0, 1, 0, 0, 0, 0])
Human              tensor([0, 1, 0, 0, 1, 0, 0, 0, 0])
Crow               tensor([1, 0, 0, 0, 1, 0, 0, 0, 0])
Flying superhero   tensor([1, 1, 0, 0, 1, 0, 0, 0, 0])
swap-flight        tensor([0, 0, 1, 0, 0, 1, 0, 0, 0])
swap-speech        tensor([0, 0, 0, 1, 0, 0, 1, 0, 0])
keep-flight        tensor([0, 0, 0, 0, 0, 1, 0, 0, 0])
keep-speech        tensor([0, 0, 0, 0, 0, 0, 1, 0, 0])
disobeys           tensor([0, 0, 0, 0, 0, 0, 0, 0, 1])
he-is?             tensor([0, 0, 0, 0, 0, 0, 0, 1, 0])

x_{p} = 9 content bits c (t_{p}) ∥ 6 position bits e_{p} ∥ 7 scratch slots 0

def embed(sentence):
    X = torch.zeros(len(sentence), num_bits)
    for current_token_position, current_token in enumerate(sentence):
        X[current_token_position, :9] = torch.tensor(token_to_vector[current_token], dtype=torch.float32)
        X[current_token_position, position_start + current_token_position] = 1
    return X


sentence = ["Crow", "disobeys", "keep-flight", "swap-speech", "he-is?"]
X = embed(sentence)

for tok, row in zip(sentence, X):
    print(f"{tok:14} {row[:9]}  {row[position_start:position_start + L]}  {row[previous_word_is_disobey:]}")

Crow           tensor([1., 0., 0., 0., 1., 0., 0., 0., 0.])  tensor([1., 0., 0., 0., 0., 0.])  tensor([0., 0., 0., 0., 0., 0., 0.])
disobeys       tensor([0., 0., 0., 0., 0., 0., 0., 0., 1.])  tensor([0., 1., 0., 0., 0., 0.])  tensor([0., 0., 0., 0., 0., 0., 0.])
keep-flight    tensor([0., 0., 0., 0., 0., 1., 0., 0., 0.])  tensor([0., 0., 1., 0., 0., 0.])  tensor([0., 0., 0., 0., 0., 0., 0.])
swap-speech    tensor([0., 0., 0., 1., 0., 0., 1., 0., 0.])  tensor([0., 0., 0., 1., 0., 0.])  tensor([0., 0., 0., 0., 0., 0., 0.])
he-is?         tensor([0., 0., 0., 0., 0., 0., 0., 1., 0.])  tensor([0., 0., 0., 0., 1., 0.])  tensor([0., 0., 0., 0., 0., 0., 0.])

head (X) = softmax (\frac{( X W _{Q} ) ( X W _{K} ) ^{⊤}}{d _{h}}) X W_{V}

def softmax(z, dim=-1):
    z = z - z.max(dim=dim, keepdim=True).values
    e = torch.exp(z)
    return e / e.sum(dim=dim, keepdim=True)


def head(X, Wq, Wk, Wv):
    Q, K, V = X @ Wq, X @ Wk, X @ Wv
    A = softmax(Q @ K.T / Wq.shape[1] ** 0.5)
    return A @ V, A

Layer 1

Head: Object Head

Wq_O = torch.zeros(num_bits, 2); Wq_O[idx["question"]] = torch.tensor([8.0, 0.0])
Wk_O = torch.zeros(num_bits, 2); Wk_O[idx["object"]]   = torch.tensor([1.0, 0.0])
Wv_O = torch.zeros(num_bits, 2); Wv_O[idx["fly"]] = torch.tensor([1.0, 0.0]); Wv_O[idx["speak"]] = torch.tensor([0.0, 1.0])
Wo_O = torch.zeros(2, num_bits); Wo_O[0, object_attribute_fly] = 1; Wo_O[1, object_attribute_speak] = 1

print("Wq_O nonzero rows:", (Wq_O != 0).any(1).nonzero().flatten())
print("Wk_O nonzero rows:", (Wk_O != 0).any(1).nonzero().flatten())
print("Wv_O nonzero rows:", (Wv_O != 0).any(1).nonzero().flatten())
print("Wo_O nonzero cols:", (Wo_O != 0).any(0).nonzero().flatten())

Wq_O nonzero rows: tensor([7])
Wk_O nonzero rows: tensor([4])
Wv_O nonzero rows: tensor([0, 1])
Wo_O nonzero cols: tensor([16, 17])

Head: Disobey Position Finder

W_{Q (P)} = S p = 1 \sum L - 1 e_{pos_{p}} e_{p - 1 ⊤} W_{K (P)} = p = 0 \sum L - 1 e_{pos_{p}} e_{p ⊤}

M = W_{Q (P)} W_{K (P) ⊤} ⟹ q_{i} \cdot k_{j} = S [j = i - 1]

M \neq = M^{⊤}

S = 24.0

Wq_P = torch.zeros(num_bits, L)
for p in range(1, L):
    Wq_P[position_start + p, p - 1] = S

Wk_P = torch.zeros(num_bits, L)
for p in range(L):
    Wk_P[position_start + p, p] = 1

Wv_P = torch.zeros(num_bits, 1); Wv_P[idx["disobey"], 0] = 1
Wo_P = torch.zeros(1, num_bits); Wo_P[0, previous_word_is_disobey] = 1

M = Wq_P @ Wk_P.T
print("M[9:15, 9:15] =")
print(M[position_start:position_start + L, position_start:position_start + L])
print("symmetric:", torch.allclose(M, M.T))

M[9:15, 9:15] =
tensor([[ 0.,  0.,  0.,  0.,  0.,  0.],
        [24.,  0.,  0.,  0.,  0.,  0.],
        [ 0., 24.,  0.,  0.,  0.,  0.],
        [ 0.,  0., 24.,  0.,  0.,  0.],
        [ 0.,  0.,  0., 24.,  0.,  0.],
        [ 0.,  0.,  0.,  0., 24.,  0.]])
symmetric: False

FFN_{1} (x) = 0 X_{1} = X + head_{O} (X) W_{O (O)} + head_{P} (X) W_{O (P)}

def layer1(X):
    oO, AO = head(X, Wq_O, Wk_O, Wv_O)
    oP, AP = head(X, Wq_P, Wk_P, Wv_P)
    X1 = X + oO @ Wo_O + oP @ Wo_P
    return X1, AO, AP


X1, AO, AP = layer1(X)

print("A_P")
print(AP)
print()
for tok, row in zip(sentence, X1):
    print(f"{tok:14} previous_word_is_disobey={row[previous_word_is_disobey]:.3f}   "
          f"object_attribute_fly={row[object_attribute_fly]:.3f}  "
          f"object_attribute_speak={row[object_attribute_speak]:.3f}")

A_P
tensor([[0.20, 0.20, 0.20, 0.20, 0.20],
        [1.00, 0.00, 0.00, 0.00, 0.00],
        [0.00, 1.00, 0.00, 0.00, 0.00],
        [0.00, 0.00, 1.00, 0.00, 0.00],
        [0.00, 0.00, 0.00, 1.00, 0.00]])

Crow           previous_word_is_disobey=0.200   object_attribute_fly=0.200  object_attribute_speak=0.000
disobeys       previous_word_is_disobey=0.000   object_attribute_fly=0.200  object_attribute_speak=0.000
keep-flight    previous_word_is_disobey=1.000   object_attribute_fly=0.200  object_attribute_speak=0.000
swap-speech    previous_word_is_disobey=0.000   object_attribute_fly=0.200  object_attribute_speak=0.000
he-is?         previous_word_is_disobey=0.000   object_attribute_fly=0.986  object_attribute_speak=0.000

Layer 2

Two heads: Head1 - "get flight attribute" Head2 - "get speech attribute"

Layer 2 has two heads: get flight attribute and get speech attribute.
Usually layers have a similar topology, so two heads are used in both layers.

Could this work be done with a single head, like the action head in the previous notebook? Not with this residual layout. In the previous notebook, one head attended to both action words, and that worked because each action word's own swap bit (swap_fly or swap_speak) says which attribute it swaps. Here, the disobey information sits in one shared slot, previous_word_is_disobey, on both action words. A single head attending to both action words would add the two disobey signals into the same number, and they could no longer be told apart. For example, Crow disobeys keep-flight swap-speech he-is? and Crow keep-flight disobeys swap-speech he-is? would give exactly the same head output, but the answers are Human and Crow. So we use one head for the flight word and one head for the speech word.

Wq_fly_head = torch.zeros(num_bits, 1); Wq_fly_head[idx["question"], 0] = 8
Wk_fly_head = torch.zeros(num_bits, 1); Wk_fly_head[idx["action_fly"], 0] = 1
Wv_fly_head = torch.zeros(num_bits, 2); Wv_fly_head[idx["swap_fly"]] = torch.tensor([1.0, 0.0]); Wv_fly_head[previous_word_is_disobey] = torch.tensor([0.0, 1.0])

Wq_speak_head = torch.zeros(num_bits, 1); Wq_speak_head[idx["question"], 0] = 8
Wk_speak_head = torch.zeros(num_bits, 1); Wk_speak_head[idx["action_speak"], 0] = 1
Wv_speak_head = torch.zeros(num_bits, 2); Wv_speak_head[idx["swap_speak"]] = torch.tensor([1.0, 0.0]); Wv_speak_head[previous_word_is_disobey] = torch.tensor([0.0, 1.0])

Wo_2 = torch.zeros(4, num_bits)
Wo_2[0, is_swap_attr_fly]        = 1
Wo_2[1, is_attr_fly_disobeyed]   = 1
Wo_2[2, is_swap_attr_speak]      = 1
Wo_2[3, is_attr_speak_disobeyed] = 1

print("Wv_fly_head nonzero rows:  ", (Wv_fly_head != 0).any(1).nonzero().flatten())
print("Wv_speak_head nonzero rows:", (Wv_speak_head != 0).any(1).nonzero().flatten())
print("Wo_2 nonzero cols:         ", (Wo_2 != 0).any(0).nonzero().flatten())

Wv_fly_head nonzero rows:   tensor([ 2, 15])
Wv_speak_head nonzero rows: tensor([ 3, 15])
Wo_2 nonzero cols:          tensor([18, 19, 20, 21])

X_{2} = X_{1} + [head_{F} (X_{1}) ∥ head_{S} (X_{1})] W_{O (2)}

def layer2_attn(X1):
    oF, AF = head(X1, Wq_fly_head, Wk_fly_head, Wv_fly_head)
    oS, AS = head(X1, Wq_speak_head, Wk_speak_head, Wv_speak_head)
    X2 = X1 + torch.cat([oF, oS], dim=1) @ Wo_2
    return X2, AF, AS


X2, AF, AS = layer2_attn(X1)
q = sentence.index("he-is?")

print("A_F[q]", AF[q])
print("A_S[q]", AS[q])
print()
print("is_swap_attr_fly       ", X2[q, is_swap_attr_fly])
print("is_attr_fly_disobeyed  ", X2[q, is_attr_fly_disobeyed])
print("is_swap_attr_speak     ", X2[q, is_swap_attr_speak])
print("is_attr_speak_disobeyed", X2[q, is_attr_speak_disobeyed])

A_F[q] tensor([0.00, 0.00, 1.00, 0.00, 0.00])
A_S[q] tensor([0.00, 0.00, 0.00, 1.00, 0.00])

is_swap_attr_fly        tensor(0.)
is_attr_fly_disobeyed   tensor(1.00)
is_swap_attr_speak      tensor(1.00)
is_attr_speak_disobeyed tensor(0.00)

FFN 2

The FFN is shown for completeness, also as a hand-constructed implementation. But for understanding the ideas of attention, position embedding, residuals and the need for layers, this part can be skipped.

Summary: bias values are used carefully to allow distinguishing 0, 1, 2, 3. This provides different ReLU activation levels corresponding to the number of flips. This FFN is a parity finder, while the FFN used in the previous notebook without the disobeys word was an XOR gate. Repeating the note that, in the real implementation, all the weights and biases are learned using gradient descent and the backpropagation algorithm.

s_{fly} = x_{object_attribute_fly} + x_{is_swap_attr_fly} + x_{is_attr_fly_disobeyed} s_{speak} = x_{object_attribute_speak} + x_{is_swap_attr_speak} + x_{is_attr_speak_disobeyed}

π (s) = ReLU (s) - 2 ReLU (s - 1) + 2 ReLU (s - 2) - 2 ReLU (s - 3)

π (0) = 0, π (1) = 1, π (2) = 0, π (3) = 1

W1 = torch.zeros(num_bits, 8)
b1 = torch.zeros(8)

for r in (object_attribute_fly, is_swap_attr_fly, is_attr_fly_disobeyed):
    W1[r, 0:4] = 1
for r in (object_attribute_speak, is_swap_attr_speak, is_attr_speak_disobeyed):
    W1[r, 4:8] = 1

b1[0:4] = torch.tensor([0.0, -1.0, -2.0, -3.0])
b1[4:8] = torch.tensor([0.0, -1.0, -2.0, -3.0])

W2 = torch.zeros(8, 2)
W2[0:4, 0] = torch.tensor([1.0, -2.0, 2.0, -2.0])
W2[4:8, 1] = torch.tensor([1.0, -2.0, 2.0, -2.0])

print("W1.T\n", W1.T)
print("\nb1", b1)
print("\nW2.T\n", W2.T)
print("\npi:", [float(torch.relu(torch.tensor([s, s - 1.0, s - 2.0, s - 3.0])) @ W2[0:4, 0]) for s in range(4)])

W1.T
 tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 1., 1., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 1., 1., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 1., 1., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 1., 1., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 1., 1.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 1., 1.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 1., 1.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 1., 1.]])

b1 tensor([ 0., -1., -2., -3.,  0., -1., -2., -3.])

W2.T
 tensor([[ 1., -2.,  2., -2.,  0.,  0.,  0.,  0.],
        [ 0.,  0.,  0.,  0.,  1., -2.,  2., -2.]])

pi: [0.0, 1.0, 0.0, 1.0]

Forward

def forward(sentence):
    X = embed(sentence)
    X1, AO, AP = layer1(X)
    X2, AF, AS = layer2_attn(X1)
    H = torch.relu(X2 @ W1 + b1)
    Y = H @ W2
    return dict(X=X, X1=X1, X2=X2, Y=Y, AO=AO, AP=AP, AF=AF, AS=AS)


word_to_attributes = {
    "Rock":              (0, 0, 0),
    "Human":             (0, 0, 1),
    "Car":               (0, 1, 0),
    "Talking Tow Truck": (0, 1, 1),
    "Crow":              (1, 0, 0),
    "Flying Superhero":  (1, 0, 1),
    "Plane":             (1, 1, 0),
    "Talking Planes":    (1, 1, 1),
}
attributes_to_word = {b: w for w, b in word_to_attributes.items()}


def readout(sentence):
    r = forward(sentence)
    q = sentence.index("he-is?")
    fly, speak = (int(v.round()) for v in r["Y"][q])
    return attributes_to_word[(fly, 0, speak)], r["Y"][q]

sentences = [
    ["Human", "disobeys", "keep-flight", "disobeys", "swap-speech", "he-is?"],
    ["Crow", "disobeys", "keep-flight", "swap-speech", "he-is?"],
    ["Crow", "keep-flight", "disobeys", "swap-speech", "he-is?"],
    ["Crow", "disobeys", "keep-flight", "disobeys", "swap-speech", "he-is?"],
]

for s in sentences:
    word, y = readout(s)
    print(f"{' '.join(s):58} {y}  ->  {word}")

Human disobeys keep-flight disobeys swap-speech he-is?     tensor([1.00, 0.98])  ->  Flying Superhero
Crow disobeys keep-flight swap-speech he-is?               tensor([0.02, 1.00])  ->  Human
Crow keep-flight disobeys swap-speech he-is?               tensor([0.99, 0.00])  ->  Crow
Crow disobeys keep-flight disobeys swap-speech he-is?      tensor([0.02, 0.00])  ->  Rock

{Crow, disobeys, keep-flight, swap-speech, he-is?}

a = ["Crow", "disobeys", "keep-flight", "swap-speech", "he-is?"]
b = ["Crow", "keep-flight", "disobeys", "swap-speech", "he-is?"]

print(sorted(a) == sorted(b))
print(readout(a)[0])
print(readout(b)[0])

True
Human
Crow

for s in sentences:
    r = forward(s)
    print(" ".join(s))
    print("  A_P")
    for tok, row in zip(s, r["AP"]):
        print(f"    {tok:14} {row}")
    print("  previous_word_is_disobey", r["X1"][:, previous_word_is_disobey])
    q = s.index("he-is?")
    print("  gathered  ", r["X2"][q, [is_swap_attr_fly, is_attr_fly_disobeyed,
                                      is_swap_attr_speak, is_attr_speak_disobeyed]])
    print("  Y         ", r["Y"][q])
    print()

Human disobeys keep-flight disobeys swap-speech he-is?
  A_P
    Human          tensor([0.17, 0.17, 0.17, 0.17, 0.17, 0.17])
    disobeys       tensor([1.00, 0.00, 0.00, 0.00, 0.00, 0.00])
    keep-flight    tensor([0.00, 1.00, 0.00, 0.00, 0.00, 0.00])
    disobeys       tensor([0.00, 0.00, 1.00, 0.00, 0.00, 0.00])
    swap-speech    tensor([0.00, 0.00, 0.00, 1.00, 0.00, 0.00])
    he-is?         tensor([0.00, 0.00, 0.00, 0.00, 1.00, 0.00])
  previous_word_is_disobey tensor([0.33, 0.00, 1.00, 0.00, 1.00, 0.00])
  gathered   tensor([0.00, 1.00, 1.00, 1.00])
  Y          tensor([1.00, 0.98])

Crow disobeys keep-flight swap-speech he-is?
  A_P
    Crow           tensor([0.20, 0.20, 0.20, 0.20, 0.20])
    disobeys       tensor([1.00, 0.00, 0.00, 0.00, 0.00])
    keep-flight    tensor([0.00, 1.00, 0.00, 0.00, 0.00])
    swap-speech    tensor([0.00, 0.00, 1.00, 0.00, 0.00])
    he-is?         tensor([0.00, 0.00, 0.00, 1.00, 0.00])
  previous_word_is_disobey tensor([0.20, 0.00, 1.00, 0.00, 0.00])
  gathered   tensor([0.00, 1.00, 1.00, 0.00])
  Y          tensor([0.02, 1.00])

Crow keep-flight disobeys swap-speech he-is?
  A_P
    Crow           tensor([0.20, 0.20, 0.20, 0.20, 0.20])
    keep-flight    tensor([1.00, 0.00, 0.00, 0.00, 0.00])
    disobeys       tensor([0.00, 1.00, 0.00, 0.00, 0.00])
    swap-speech    tensor([0.00, 0.00, 1.00, 0.00, 0.00])
    he-is?         tensor([0.00, 0.00, 0.00, 1.00, 0.00])
  previous_word_is_disobey tensor([0.20, 0.00, 0.00, 1.00, 0.00])
  gathered   tensor([0.00, 0.00, 1.00, 1.00])
  Y          tensor([0.99, 0.00])

Crow disobeys keep-flight disobeys swap-speech he-is?
  A_P
    Crow           tensor([0.17, 0.17, 0.17, 0.17, 0.17, 0.17])
    disobeys       tensor([1.00, 0.00, 0.00, 0.00, 0.00, 0.00])
    keep-flight    tensor([0.00, 1.00, 0.00, 0.00, 0.00, 0.00])
    disobeys       tensor([0.00, 0.00, 1.00, 0.00, 0.00, 0.00])
    swap-speech    tensor([0.00, 0.00, 0.00, 1.00, 0.00, 0.00])
    he-is?         tensor([0.00, 0.00, 0.00, 0.00, 1.00, 0.00])
  previous_word_is_disobey tensor([0.33, 0.00, 1.00, 0.00, 1.00, 0.00])
  gathered   tensor([0.00, 1.00, 1.00, 1.00])
  Y          tensor([0.02, 0.00])

About this file

This notebook is part of the support files for the TechAarvam workshop
Build Your Own Model.

Website: https://www.techaarvam.com
Workshop files repository: https://github.com/techaarvam/byom_workshop
YouTube: https://www.youtube.com/@TechAarvam

© TechAarvam. You are free to use, copy, modify, share and build on this
material, including for commercial purposes, provided you credit
TechAarvam and link back to https://www.techaarvam.com. Please keep
this notice with any copy or derivative. Provided as-is, without warranty.

Agentic coding era: Step back and rethink methodologies

Tech-Aarvam — Thu, 10 Sep 2026 13:58:57 +0000

Natural language is now the primary programming interface.

With the increased use of AI coding agents, there is a strong need for disciplined methodologies.

Accountability is non-negotiable. So are compliance, data privacy, safety, and adherence to applicable laws.

Root causes of disasters waiting to happen

Blindly prompting, vibe coding
intent → implementation → production

What is the remedy?

We need to step back and develop new processes and strong ground-up methodologies. Retaining traceability may not be sufficient; we want to strengthen it by being more mindful of the new risks.

The common spine of any project (hardware development, hardware verification, software development, software verification, process definitions) is the same:

Requirements → Specifications → Implementation.

Do not skip requirements and specifications. Requirements and specifications must be fine-grained, version-controlled, and traceable. Both will have granularity levels. From project-level to leaf-level: Every level must be versioned and traceable. This isn't new; especially, the automotive segment has adhered to strong traceability and process requirements. But in the agentic AI era, many other projects also require a similar, if not higher, level of rigour.

Specifications at the finest level of granularity require precision - no ambiguity. At the finest granularity, the specification may continue to use well-specified grammar like programming languages, while the higher-level specification will lean more towards natural language. Specification then converts to the implementation (completely code-agent-generated).

This argument does not contradict the use of AI for requirements and specifications, but brings in ample determinism, traceability, and repeatability.

Having specifications and requirements at finer levels allows for more rigorous rule checks, hardware simulations, and software regressions.

For decades, engineering attention had a heavy focus on coding guidelines, code reuse, portability, memory safety, readability, and layered abstractions.

At least some of those focus areas were motivated by coding being expensive, hard to maintain, and by the need to stay within human cognitive abilities!

Implementation is easy now. Programming languages may change dramatically. The lowest level of specification needs defined grammar; would it be a declarative language? Declarative languages may be well suited for defining specifications and requirements. Are most other languages on their way out? Do we go directly from the fine-grained specifications to bytecode (software) or RTL gates (hardware)?

Remove the focus on implementation (maybe even entirely). But increase the focus towards the rigorous processes for the following view:

requirements, specifications × development, verification, process definitions

Time-to-market and resource constraints meant that striving for the utmost safety and compliance got only just about enough focus to pass the gate. There is probably no excuse now. Let us shift the focus more towards correctness, safety, and traceability. We have a real opportunity to step up and mitigate the risk. The right time is now.

Attention is simpler than you think - A hand-crafted superhero transformer

Tech-Aarvam — Tue, 08 Sep 2026 11:46:52 +0000

Part of the TechAarvam workshop support files — Build Your Own Model.

Attention is simpler than you think

A hand-crafted superhero transformer

This notebook presents a hand-crafted example. The goal is to understand the intuition behind the attention block in the transformer architecture.

Attention and FFN are the two main components. FFN is an Artificial neural network (ANN) with 1 input, 1 hidden, and 1 output layer.

So we start constructing the FFN's job by hand. Then we show how Attention extracts what the FFN can use from a sentence. The sentences used and the vocabulary are intentionally hand-designed to be simple. The attention block weights are hand-coded, instead of trained. The ANN(FFN) is trained.

import numpy as np

np.set_printoptions(precision=2, suppress=True)

Part 1: Artificial Neural Network

Vocabulary for the ANN

Word	can fly?	has wheels?	can speak?
Rock	0	0	0
Human	0	0	1
Car	0	1	0
Talking Tow Truck	0	1	1
Crow	1	0	0
Flying Superhero	1	0	1
Plane	1	1	0
Talking Planes	1	1	1

word_to_attributes = {
    "Rock":              (0, 0, 0),
    "Human":             (0, 0, 1),
    "Car":               (0, 1, 0),
    "Talking Tow Truck": (0, 1, 1),
    "Crow":              (1, 0, 0),
    "Flying Superhero":  (1, 0, 1),
    "Plane":             (1, 1, 0),
    "Talking Planes":    (1, 1, 1),
}

for word, bits in word_to_attributes.items():
    print(f"{word:20} {bits}")

Rock                 (0, 0, 0)
Human                (0, 0, 1)
Car                  (0, 1, 0)
Talking Tow Truck    (0, 1, 1)
Crow                 (1, 0, 0)
Flying Superhero     (1, 0, 1)
Plane                (1, 1, 0)
Talking Planes       (1, 1, 1)

ANN's learning goal

The ANN's output - the logits (probability scores) for the next word is a correction to the input word.

The input is fixed bit-encoding of 6 bits.

Input word  - 3 bits, one-hot, or IDs (design choice)
Correction  - 3 bits
Output word - attributes after the correction

Example

Rock + add flight -> Plane
Human + add flight -> Flying Superhero

attributes_to_word = {bits: word for word, bits in word_to_attributes.items()}


def apply_correction(word, correction):
    """Correction is 3 bits: 1 means flip that attribute."""
    bits = word_to_attributes[word]
    out = tuple(b ^ c for b, c in zip(bits, correction))
    return attributes_to_word[out]


print(apply_correction("Rock", (1, 0, 0)))   # add flight
print(apply_correction("Human", (1, 0, 0)))  # add flight

Crow
Flying Superhero

Training the ANN: Pseudocode

# Pseudocode - not run here.
#
# for loop in range (num_epochs):
#     for batch_inputs, batch_targets in loader:
#         prediction = model.forward(batch_inputs)          # predict
#         loss = CrossEntropyLoss(prediction, target)       # how far off?
#         loss.backward()                                   # gradient descent

In the full workshop, the notebook with the full ANN implementation is available. Visit the relevant TechAarvam pages to locate them.

Part 2: Attention Block

Input is now a sentence:

Crow keep-flight swap-speech he-is?

Sentences have complex structure. Word order varies.
So: fixed sentence structure, small defined vocabulary.

Objects:       Rock, Human, Crow, Flying superhero
Actions:       swap-flight, swap-speech, keep-flight, keep-speech
Interrogative: he-is?

Tokens to vectors

8-bit vectors, shown as 4 + 4.

Bits 1-4 - attributes and correction:

Token	can fly	can speak	swap flight	swap speech
Rock	0	0	0	0
Human	0	1	0	0
Crow	1	0	0	0
Flying superhero	1	1	0	0
swap-flight	0	0	1	0
swap-speech	0	0	0	1
keep-flight	0	0	0	0
keep-speech	0	0	0	0
he-is?	0	0	0	0

Bits 5-8 - token type:

Token	object?	flight act?	speech act?	question?
Rock	1	0	0	0
Human	1	0	0	0
Crow	1	0	0	0
Flying superhero	1	0	0	0
swap-flight	0	1	0	0
swap-speech	0	0	1	0
keep-flight	0	1	0	0
keep-speech	0	0	1	0
he-is?	0	0	0	1

bit_labels = ["can fly", "can speak", "swap flight", "swap speech",
        "object?", "flight act?", "speech act?", "question?"]

token_to_bitvec = {
    #                    fly spk swF swS  obj flA spA  q
    "Rock":             [0,  0,  0,  0,   1,  0,  0,  0],
    "Human":            [0,  1,  0,  0,   1,  0,  0,  0],
    "Crow":             [1,  0,  0,  0,   1,  0,  0,  0],
    "Flying superhero": [1,  1,  0,  0,   1,  0,  0,  0],
    "swap-flight":      [0,  0,  1,  0,   0,  1,  0,  0],
    "swap-speech":      [0,  0,  0,  1,   0,  0,  1,  0],
    "keep-flight":      [0,  0,  0,  0,   0,  1,  0,  0],
    "keep-speech":      [0,  0,  0,  0,   0,  0,  1,  0],
    "he-is?":           [0,  0,  0,  0,   0,  0,  0,  1],
}

token_to_bitvec = {k: np.array(v) for k, v in token_to_bitvec.items()}

for tok, v in token_to_bitvec.items():
    print(f"{tok:18} {v[:4]}  {v[4:]}")

Rock               [0 0 0 0]  [1 0 0 0]
Human              [0 1 0 0]  [1 0 0 0]
Crow               [1 0 0 0]  [1 0 0 0]
Flying superhero   [1 1 0 0]  [1 0 0 0]
swap-flight        [0 0 1 0]  [0 1 0 0]
swap-speech        [0 0 0 1]  [0 0 1 0]
keep-flight        [0 0 0 0]  [0 1 0 0]
keep-speech        [0 0 0 0]  [0 0 1 0]
he-is?             [0 0 0 0]  [0 0 0 1]

The sentence

sentence = ["Crow", "keep-flight", "swap-speech", "he-is?"]

X = np.stack([token_to_bitvec[t] for t in sentence])
print(X.shape)
X

(4, 8)





array([[1, 0, 0, 0, 1, 0, 0, 0],
       [0, 0, 0, 0, 0, 1, 0, 0],
       [0, 0, 0, 1, 0, 0, 1, 0],
       [0, 0, 0, 0, 0, 0, 0, 1]])

Attention Math

Skip the equations and go to the hand-written Q, K, V weights to get the idea behind these equations first.

N heads. Each head does:

Attention (Q, K, V) = softmax! (\frac{Q K ^{T}}{d _{h}} + M) V

$Q K^{T}$ - pairwise scores
softmax - normalize
$V$ - send the payload

def softmax(z, axis=-1):
    z = z - z.max(axis=axis, keepdims=True)
    e = np.exp(z)
    return e / e.sum(axis=axis, keepdims=True)


def head(X, Wq, Wk, Wv):
    Q, K, V = X @ Wq, X @ Wk, X @ Wv
    d_h = Wq.shape[1]
    scores = Q @ K.T / np.sqrt(d_h)
    A = softmax(scores)
    return A @ V, A, Q, K, V

Hand-written heads

Q - Query. K - Key. V - Value (payload).

The Q, K, V are intermediate tensors, what the model has are the corresponding weights.
The weight matrices transform the input in three ways to get the (Q, K, V).

Q = X W_{Q} K = X W_{K} V = X W_{V}

X is the input. Its the full sentence in our example. A context length full of tokens as input.
The three weight matrices transform the input. The three matrices perform - three tasks.
The weight matrices are per-head. i.e each head can extract out different information from the tokens.

$W_{Q}$ is weights for getting the queries for this head.
$W_{K}$ are the weights that transform the input tokens to keys. (keys are like answers to the queries)
$W_{V}$ is the payload, if the question and the answer for a pair of tokens get a high score, the W_V helps extract the payload or values from the tokens to pass along towards the FFN (toward the concatenation operation, then the FFN).

Every word asks a question; And also the question is not the same across heads. Each word in Each head can ask a different question; We have two heads in this hand-constructed example (Object Head, Action Verb-Object head)
Object head - he-is? asks: are you an Object?

Keys answer. Every token answers.
Object head - "Crow" replies: I am an Object.

Object Head - weights

Row 8 -> query he-is? in the object head asks - are you an object?
Row 5 -> key replies to this question. (i.e every token replies)
Rows 1-2 -> payload is can fly, can speak. (Every token can have a payload, and tokens with high-scores get to send their payload)

Wq1 = np.zeros((8, 2)); Wq1[7] = [8, 0]
Wk1 = np.zeros((8, 2)); Wk1[4] = [1, 0]
Wv1 = np.zeros((8, 2)); Wv1[0] = [1, 0]; Wv1[1] = [0, 1]

print("Wq1\n", Wq1, "\n\nWk1\n", Wk1, "\n\nWv1\n", Wv1)

Wq1
 [[0. 0.]
 [0. 0.]
 [0. 0.]
 [0. 0.]
 [0. 0.]
 [0. 0.]
 [0. 0.]
 [8. 0.]] 

Wk1
 [[0. 0.]
 [0. 0.]
 [0. 0.]
 [0. 0.]
 [1. 0.]
 [0. 0.]
 [0. 0.]
 [0. 0.]] 

Wv1
 [[1. 0.]
 [0. 1.]
 [0. 0.]
 [0. 0.]
 [0. 0.]
 [0. 0.]
 [0. 0.]
 [0. 0.]]

Action (Verb-Object) Head - weights

Row 8 -> he-is? queries both slots
Rows 6-7 -> keys are the two action types
Rows 3-4 -> payload is swap flight, swap speech

Wq2 = np.zeros((8, 2)); Wq2[7] = [8, 8]
Wk2 = np.zeros((8, 2)); Wk2[5] = [1, 0]; Wk2[6] = [0, 1]
Wv2 = np.zeros((8, 2)); Wv2[2] = [1, 0]; Wv2[3] = [0, 1]

print("Wq2\n", Wq2, "\n\nWk2\n", Wk2, "\n\nWv2\n", Wv2)

Wq2
 [[0. 0.]
 [0. 0.]
 [0. 0.]
 [0. 0.]
 [0. 0.]
 [0. 0.]
 [0. 0.]
 [8. 8.]] 

Wk2
 [[0. 0.]
 [0. 0.]
 [0. 0.]
 [0. 0.]
 [0. 0.]
 [1. 0.]
 [0. 1.]
 [0. 0.]] 

Wv2
 [[0. 0.]
 [0. 0.]
 [1. 0.]
 [0. 1.]
 [0. 0.]
 [0. 0.]
 [0. 0.]
 [0. 0.]]

Object head: Derive K, V from the weights: Focus on the Crow

Crow, bits 1-8: 1 0 0 0 1 0 0 0

The payload is Crow's object attributes.

crow = token_to_bitvec["Crow"]

print("x_Crow           ", crow)
print("x_Crow @ Wk1     ", crow @ Wk1, "  <- key: I am an object")
print("x_Crow @ Wv1     ", crow @ Wv1, "  <- value: [can fly, can speak]")

x_Crow            [1 0 0 0 1 0 0 0]
x_Crow @ Wk1      [1. 0.]   <- key: I am an object
x_Crow @ Wv1      [1. 0.]   <- value: [can fly, can speak]

Action head: Derive K, V from the weights: Focus on the swap-flight

swap-flight, bits 1-8: 0 0 1 0 0 1 0 0

The payload is the correction.

sf = token_to_bitvec["swap-flight"]

print("x_swap-flight        ", sf)
print("x_swap-flight @ Wk2  ", sf @ Wk2, "  <- key: I am a flight action")
print("x_swap-flight @ Wv2  ", sf @ Wv2, "  <- value: [swap flight, swap speech]")

x_swap-flight         [0 0 1 0 0 1 0 0]
x_swap-flight @ Wk2   [1. 0.]   <- key: I am a flight action
x_swap-flight @ Wv2   [1. 0.]   <- value: [swap flight, swap speech]

Both heads: Derive Q from the weights: Focus on he-is?

he-is? is the last token, so we read row 3 of the attention matrix.

out1, A1, Q1, K1, V1 = head(X, Wq1, Wk1, Wv1)
out2, A2, Q2, K2, V2 = head(X, Wq2, Wk2, Wv2)

q = sentence.index("he-is?")

print("Object head - attention from he-is?")
for tok, k, a in zip(sentence, K1, A1[q]):
    print(f"  {tok:14} key={k}  softmax={a:.2f}")
print("  head 1 output:", out1[q])

print("\nAction head - attention from he-is?")
for tok, k, a in zip(sentence, K2, A2[q]):
    print(f"  {tok:14} key={k}  softmax={a:.2f}")
print("  head 2 output:", out2[q])

Object head - attention from he-is?
  Crow           key=[1. 0.]  softmax=0.99
  keep-flight    key=[0. 0.]  softmax=0.00
  swap-speech    key=[0. 0.]  softmax=0.00
  he-is?         key=[0. 0.]  softmax=0.00
  head 1 output: [0.99 0.  ]

Action head - attention from he-is?
  Crow           key=[0. 0.]  softmax=0.00
  keep-flight    key=[1. 0.]  softmax=0.50
  swap-speech    key=[0. 1.]  softmax=0.50
  he-is?         key=[0. 0.]  softmax=0.00
  head 2 output: [0.  0.5]

Two action words split the mass, so head 2 gives [0, 0.5].
$W_{O}$ scales by 2 -> [0, 1].

Concat: 4 bits out

[1, 0] head 1 ∥ [0, 1] head 2 = [1, 0, 0, 1]

Wo = np.diag([1.0, 1.0, 2.0, 2.0])   # head 2 mass was split across 2 words

concat = np.concatenate([out1[q], out2[q]])
final = concat @ Wo

print("concat        ", concat)
print("after W_O     ", final)
print()
for name, val in zip(bit_labels[:4], final):
    print(f"  {name:12} {val:.0f}")

concat         [0.99 0.   0.   0.5 ]
after W_O      [0.99 0.   0.   1.  ]

  can fly      1
  can speak    0
  swap flight  0
  swap speech  1

Hand-built weights pulled the object attributes and the
correction attributes into 4 bits.

Crow: flies, no speech. Correction: swap speech.
-> Flying superhero. Same 4 bits the ANN wanted.

fly, speak, swap_fly, swap_speak = (int(v) for v in final.round())

obj_in = (fly, 0, speak)                       # ANN bits: fly, wheels, speak
correction = (swap_fly, 0, swap_speak)

print("object in :", attributes_to_word[obj_in])
print("correction:", correction)
print("object out:", apply_correction(attributes_to_word[obj_in], correction))

object in : Crow
correction: (0, 0, 1)
object out: Flying Superhero

About this file

This notebook is part of the support files for the TechAarvam workshop
Build Your Own Model.

Website: https://www.techaarvam.com
Workshop files repository: https://github.com/techaarvam/byom_workshop
YouTube: https://www.youtube.com/@TechAarvam