DEV Community

kirandeepjassal-crypto
kirandeepjassal-crypto

Posted on Originally published at prepstack.co.in

LLMs & Transformers from First Principles (2026) — Tokenization, Attention, LoRA, and a Tiny GPT You Can Train (with PyTorch Code)

#llm #ai #python #machinelearning

Everyone uses LLMs in 2026. Far fewer can explain what happens between text in and text out. The gap matters because every LLM problem — bad outputs, high latency, wrong answers, costly fine-tunes — is solved by knowing which mechanism inside the model is responsible.

This rebuilds the LLM stack piece by piece with real PyTorch and Hugging Face code: tokenization, embeddings, self-attention, the transformer block, a tiny GPT you can train on your laptop, pretraining vs fine-tuning vs LoRA vs DPO, and decoding.

The one idea

An LLM is a function from token sequences to a next-token probability distribution. That's it.

sequence = [<bos>, "The", " sky", " is"]
while not done:
    probs = LLM(sequence)
    next_token = sample(probs)
    sequence.append(next_token)
Enter fullscreen mode Exit fullscreen mode

Every chat API, code assistant, agent, and RAG system is a sampling loop around "predict the next token."

Tokenization — text becomes integers first

A model can't read characters. Modern LLMs use subword tokenization (BPE for GPT/Llama/Mistral, WordPiece for BERT, SentencePiece for T5).

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("gpt2")
tok.convert_ids_to_tokens(tok.encode("Mumbai and Indore are cities in India."))
# ['Mumbai', 'and', 'Ind', 'ore', 'are', 'cities', 'in', 'India', '.']
Enter fullscreen mode Exit fullscreen mode

"Mumbai" is 1 token; "Indore" is 2. Token counts drive API cost, context window, and latency — French text costs 2-3x more tokens than English. Tokenization is the source of most "why does the model do X?" bugs. And always use apply_chat_template — hand-concatenating "user: ... assistant: ..." is a classic bug.

Self-attention — the one idea that changed everything

For each token, compute a weighted sum of every other token's value, where the weights depend on relevance. Every token gets three roles: Query ("what am I looking for?"), Key ("what do I represent?"), Value ("what do I contribute?").

Attention(Q, K, V) = softmax(Q @ K.T / sqrt(d)) @ V
Enter fullscreen mode Exit fullscreen mode

The whole revolution is one matrix multiply + a softmax:

scores = Q @ K.transpose(-2, -1) / math.sqrt(self.d_k)
if mask is not None:
    scores = scores.masked_fill(mask == 0, float("-inf"))
weights = F.softmax(scores, dim=-1)
return weights @ V
Enter fullscreen mode Exit fullscreen mode

The causal mask (lower-triangular) is why GPT can't see the future. Multi-head attention runs several of these in parallel. In production, use Flash Attention via F.scaled_dot_product_attention.

The transformer block 

x -> LayerNorm -> MultiHeadAttention -> +residual ->
  -> LayerNorm -> FeedForward -> +residual -> output
Enter fullscreen mode Exit fullscreen mode
  • Residuals are the gradient highway — without them, deep transformers don't train.
  • LayerNorm (Pre-LN) stabilizes training.
  • Feed-forward is a 2-layer MLP per token, usually d_ff = 4 x d_model.

Stack 6-96 of these and you have a model. Llama-3-8B is 32 blocks; GPT-3 is 96. The guide has a full runnable ~200-line GPT you can train on Shakespeare on a laptop — that's the entire architecture of GPT-3, just bigger and on more data.

Training, decoded

  • Pretraining = predict the next token on 5-15 trillion tokens. The intelligence emerges from that one boring objective at scale ($10M-$100M for a frontier model).
  • Fine-tuning — start with prompting/few-shot; if that's not enough, LoRA trains ~0.1% of params at ~1000x less memory: 
lora_config = LoraConfig(r=16, lora_alpha=32,
    target_modules=["q_proj","k_proj","v_proj","o_proj"],
    lora_dropout=0.05, task_type="CAUSAL_LM")
model = get_peft_model(model, lora_config)
# trainable%: 0.4%, adapter ~130 MB
Enter fullscreen mode Exit fullscreen mode
  • QLoRA (LoRA on a 4-bit base) fine-tunes a 70B model on a single 24 GB GPU — the standard recipe for small teams in 2026.
  • SFT trains on (instruction, response); DPO on (prompt, chosen, rejected) — much simpler than classic RLHF.

Decoding

Greedy is deterministic but repetitive. The production default is temperature + top-p:

out = model.generate(inputs, do_sample=True, temperature=0.7, top_p=0.9,
                     repetition_penalty=1.05, max_new_tokens=128)
Enter fullscreen mode Exit fullscreen mode

Chat: 0.7. Code: 0.0-0.3. For production, swap model.generate for vLLM — paged attention + continuous batching, often 5-20x faster.

The honest stuff

  • Context length costs scale quadratically — 32k isn't 4x 8k, it's ~16x.
  • Most prod LLM systems are 90% retrieval + 10% LLM.
  • Fine-tuning is rarely the answer — try prompting, RAG, and a bigger model first.
  • Evals beat vibes — build a golden set of 50-200 queries before you fine-tune.
  • Inference cost dominates training cost. Optimize for serving.

The right mental model

An LLM is not magic. It's: a tokenizer, an embedding table, a stack of transformer blocks, and a head that projects back to vocabulary. Pretraining gives general competence; fine-tuning teaches your task; decoding turns probabilities into text.

Three habits: always inspect the tokens first; build the eval set before the model; reach for the smallest model that works.

The full guide has the complete PyTorch — scaled dot-product + multi-head attention, the transformer block, the runnable tiny GPT + training loop, full/LoRA/QLoRA fine-tuning, DPO, the decoding recipes, RAG, tool-use, and the 2026 model menu:

https://prepstack.co.in/blog/llms-transformers-tokenization-attention-embeddings-pretraining-finetuning-pytorch-guide

Originally published on PrepStack.

Top comments (0)