How a Transformer Plays Tic-Tac-Toe

#tutorial #ai #machinelearning #transformers

An interactive guide to the architecture behind modern language models.

Instead of predicting the next word, this Transformer predicts the next move in a game of fading Tic-Tac-Toe—making every step of the model easy to visualize and understand.

Play the game, inspect every matrix multiplication, and watch tokens flow through the network in real time.

What's covered

Tokenization and embeddings
Learned positional encoding
Self-attention (Q, K, V)
Multi-head attention
Causal masking and softmax
Residual connections and layer normalization
MLP (feed-forward network)
Unembedding and sampling
Model ablations (no positional encoding, no causal mask, no MLP, no residual stream)

Includes interactive visualizations for every stage of the Transformer pipeline - from input tokens to the final prediction.

https://sbondaryev.dev/articles/transformer