Generative AI often looks like magic from the outside. You feed a prompt into a Large Language Model, and it seamlessly generates structured code, translates complex texts, or engages in multi-turn reasoning.
However, stripped of high-level abstractions and marketing buzzwords, an LLM is essentially a probability engine. Its single core task is to model the probability distribution of text and predict the most likely next token given a sequence of preceding tokens.
In this article, we’ll look past framework abstractions (like PyTorch or Hugging Face) and break down the exact mathematical machinery that turns continuous probability distributions into coherent generated text.
1. The Probabilistic View: Sequence Modeling as Joint Probability
At a fundamental level, any text sequence W = (w_1, w_2, \dots, w_N) can be represented as a joint probability distribution P(w_1, w_2, \dots, w_N).
By applying the Chain Rule of Probability, this joint probability breaks down into a product of conditional probabilities:
P(w_1, w_2, \dots, w_N) = \prod_{t=1}^{N} P(w_t \mid w_1, w_2, \dots, w_{t-1})
This is the mathematical foundation of Autoregressive Language Models. The model predicts token w_t based strictly on the context of preceding tokens w_{<t}.
2. From Logits to Probabilities: The Role of Softmax and Temperature
When the final linear layer of a Transformer processes context tokens, it outputs raw, unnormalized continuous scores called Logits (z). To turn these raw numbers into a valid probability distribution over our entire vocabulary V, we pass them through the Softmax function:
\text{Softmax}(z_i) = \frac{e^{z_i}}{\sum_{j \in V} e^{z_j}}
Controlling Randomness with Temperature (T)
To control how deterministic or creative the model's responses are, we introduce a scaling factor known as Temperature (T):
\text{Softmax}(z_i, T) = \frac{e^{z_i / T}}{\sum_{j \in V} e^{z_j / T}}
Low Temperature (T < 1.0): Sharpens the distribution, forcing the model to select high-probability tokens (ideal for code and math).
High Temperature (T > 1.0): Flattens the distribution, giving lower-probability tokens a higher chance of selection (ideal for creative writing).
Here is how simple temperature scaling looks in pure NumPy:
import numpy as np
def softmax_with_temperature(logits: np.ndarray, temperature: float = 1.0) -> np.ndarray: # Scale logits by temperature scaled_logits = logits / max(temperature, 1e-8)
Subtract max for numerical stability
exp_logits = np.exp(scaled_logits - np.max(scaled_logits))
return exp_logits / np.sum(exp_logits)
Example Logits for 4 vocabulary tokens
logits = np.array([2.0, 1.0, 0.1, 4.0])
print("Standard Softmax (T=1.0):", np.round(softmax_with_temperature(logits, T=1.0), 3)) print("Deterministic (T=0.2):", np.round(softmax_with_temperature(logits, T=0.2), 3)) print("Creative (T=1.5):", np.round(softmax_with_temperature(logits, T=1.5), 3))
- How Models Learn: Negative Log-Likelihood (NLL)
During training, the model's parameters are updated using Maximum Likelihood Estimation (MLE). Instead of maximizing raw probability values (which can lead to numerical underflow), we minimize the Negative Log-Likelihood (NLL) loss:
\mathcal{L}{\text{NLL}} = -\sum{t=1}^{N} \log P(w_t \mid w_{<t})
By minimizing this loss, we force the network to assign higher probability mass to the correct tokens present in our training dataset.
Deepen Your Understanding: Build it From Scratch
Understanding these core mathematical principles—from Bayes' rule and Cross-Entropy to Sampling strategies and Perplexity—is what separates developers who simply call LLM APIs from engineers who can build, optimize, and debug custom AI systems.
If you want to build a deep, intuitive understanding of the math behind Generative AI without relying on high-level libraries, check out my latest concise primer:
If you want to build a deep, intuitive understanding of the math behind Generative AI without relying on high-level libraries, check out my latest concise primer:
📘 The Mathematics of Generative AI: From Probability to Language Models
What’s inside the capsule:
Step-by-step mathematical breakdowns of conditional probability, MLE, and Cross-Entropy.
Detailed derivations of Softmax, Temperature scaling, and Perplexity metrics.
Complete hands-on project: Building a functional Mini Language Engine from scratch using pure Python and NumPy.
👉 Get your copy on Amazon
Top comments (0)