How an LLM Generates Text, One Token at a Time

Ever wonder how an AI “thinks”? Let me break down how it generates text

First, a small correction: an LLM is not literally thinking like a person. At its core, it is very good at predicting the next token from the tokens that came before it. Do that again and again, and suddenly it looks like it is writing.

Let’s make that happen with a small pre-trained Qwen model.

What we are building

We give the model this prompt:

prompt = "Explain large language models in a single sentence."

Then we let it generate one token at a time, up to 100 times. This is the most simple version: each time, we pick the highest-scoring next token.

Pre-requisites

• Python 3.10 or newer
• PyTorch and Hugging Face transformers
• Around 1.5 GB free for the Qwen 0.6B model weights (the script downloads them the first time)

Install the two libraries:

pip install -U torch transformers

Save the next code block as tiny_generate.py, then run:

python tiny_generate.py

CPU is completely fine for learning. It is just slower than a GPU.

Step 1: turn the prompt into token IDs

The model does not receive a Python string. The tokenizer breaks the text into pieces called tokens and represents them as integer IDs. A token is not always a whole word: it can be a word, part of a word, punctuation, or a space.

input_ids = tokenizer.encode(prompt)
input_tensor = torch.tensor(input_ids, device=device)
input_token_ids = input_tensor.unsqueeze(0)

That unsqueeze(0) adds the batch dimension. The shape changes from:

[number_of_tokens]

to:

[1, number_of_tokens]

The 1 means: “here is one prompt in this batch.” No magic, just the shape the model expects.

Step 2: load the model

The model is a big set of learned numbers. AutoModelForCausalLM loads the right Qwen architecture directly from Hugging Face. Calling eval() tells PyTorch we are using it for inference, not training.

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B-Base")
model.to(device)
model.eval()

Step 3: ask for the next-token scores

When we pass the full prompt into the model, it returns a score for every vocabulary token at every input position. Those raw scores are called logits. They become probabilities only after applying softmax.

For generation, we care about the scores at the final input position. That final row answers: “given everything in the prompt so far, what token should come next?”

logits = model(input_ids=input_token_ids).logits[:, -1]

The : means “keep every item in the batch,” and -1 means “take the final token position.” It does not throw away the earlier prompt; the model has already used the whole prompt to produce that final row.

Here is a tiny fake 2D view of the idea. A real model does this for every token in its vocabulary, not just five words on a graph.

In this example, models has the largest score, so greedy decoding chooses it.

Step 4: pick a token and add it back to the prompt

argmax gives us the ID with the highest score. Then torch.cat appends that new ID to the existing sequence. On the next loop, the model sees the original prompt plus the token it just wrote.

next_token = torch.argmax(logits, dim=-1, keepdim=True)
token_ids = torch.cat([token_ids, next_token], dim=1)

That is the whole trick: predict, append, repeat.

The complete runnable example

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "Qwen/Qwen3-0.6B-Base"

@torch.inference_mode()
def generate_text_basic(model, token_ids, max_new_tokens, eos_token_id):
    input_length = token_ids.shape[1]
    model.eval()

    for _ in range(max_new_tokens):
        logits = model(input_ids=token_ids).logits[:, -1]
        next_token = torch.argmax(logits, dim=-1, keepdim=True)

        if torch.all(next_token == eos_token_id):
            break

        token_ids = torch.cat([token_ids, next_token], dim=1)

    return token_ids[:, input_length:]


device = torch.device("cpu")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(MODEL_ID)
model.to(device)
model.eval()

prompt = "Explain large language models in a single sentence."
input_token_ids = torch.tensor(
    tokenizer.encode(prompt), device=device
).unsqueeze(0)

output_token_ids = generate_text_basic(
    model=model,
    token_ids=input_token_ids,
    max_new_tokens=100,
    eos_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(output_token_ids.squeeze(0).tolist()))

The line below is the important part of the loop:

logits = model(input_ids=token_ids).logits[:, -1]

The model reads all tokens so far, produces a huge list of possible next-token scores, and we select one. Then the selected token becomes part of the input for the next round. After enough rounds, token IDs become text again:

text = tokenizer.decode(output_token_ids.squeeze(0).tolist())

One important thing: this demo is deliberately boring

argmax always picks the most likely token. That makes the result deterministic and easy to understand, but it can also make the model repetitive. Chat apps often sample from the distribution instead: they use softmax, temperature, and sometimes top-p sampling to allow several plausible next tokens.

But before we add those knobs, it is worth seeing the simple loop. Generate a score vector. Pick a token. Append it. Repeat.

That is how the model starts writing.