LLM Basics + Prompting: How They Actually Work

Before we build anything useful with AI, we need to understand what's actually happening when you send a message to an LLM. Not the surface-level explanation — the actual mechanism. Because once you get it, everything else in this series will make a lot more sense.

What is an LLM, really?

At its core, an LLM is a next-token predictor. That's it.

You give it a sequence of text, and it predicts the most likely next chunk of text — then repeats that process over and over until the response is complete.

"The sky is" → most likely next token: "blue"
             → less likely: "purple"
             → very unlikely: "banana"

It learned to do this by training on a huge amount of text — books, websites, code, conversations. Through that process it picked up language patterns, facts about the world, how code works, how reasoning flows. All of that is now encoded in billions of numbers called weights.

It's not thinking. It's predicting. But predicting at scale, with enough data, produces something that looks a lot like thinking.

Tokens, not words

Here's something that surprises a lot of people: LLMs don't read words — they read tokens. A token is a small chunk of text, usually around 3–4 characters.

"Hello world"    → ["Hello", " world"]             = 2 tokens
"Unbelievable"   → ["Un", "bel", "iev", "able"]    = 4 tokens
"def my_func():" → ["def", " my", "_func", "():"]  = 4 tokens

Why does this matter? Because everything is measured in tokens — the size of what the model can read at once, how much an API call costs, how long a response is. A rough rule of thumb: 1000 tokens is about 750 words.

It also explains why LLMs sometimes struggle with things like counting letters or spelling unusual words — the model never sees individual characters, only these chunks.

The context window

Everything you send to an LLM in a single call — your instructions, the conversation history, the question — all of that sits in something called the context window. Think of it like the model's working memory. Whatever fits in there, it can see. Whatever doesn't, it has no idea about.

Modern models have context windows ranging from a few thousand tokens to over a million. But here's the important thing: the model has no memory between calls. Every time you make a new request, it starts fresh. If you want it to remember something from earlier, you have to include it in the context yourself. That's something we'll build in Exercise 4.

Temperature

When the model predicts the next token, it doesn't just pick one — it produces a probability distribution over all possible tokens and then samples from it. Temperature controls how that sampling happens.

At temperature 0, the model always picks the highest probability option — same input, same output, every time. As you increase temperature, the distribution gets flatter, and more surprising tokens get a chance. At very high temperatures, the output gets creative, unpredictable, or sometimes just strange.

For factual questions and code, you want low temperature. For brainstorming, higher temperature gives more variety. We'll see this in action in Exercise 1.

The three message roles

When you talk to an LLM through code, messages are structured with roles:

messages = [
    {"role": "system",    "content": "You are a helpful assistant."},
    {"role": "user",      "content": "What is the capital of France?"},
    {"role": "assistant", "content": "Paris."},
    {"role": "user",      "content": "What is its population?"},
]

The system message sets the rules — the model's persona, constraints, what it should or shouldn't do. The user message is what you send. The assistant messages are the model's previous replies, which you include to give it the conversation history.

The system prompt is the most powerful thing you have. Small changes to it produce dramatically different responses to the exact same question.

Prompting patterns

Once you understand the structure, there are a few patterns that make a big difference in how well the model responds.

Zero-shot is just asking directly — no examples, no setup. Works well for clear, simple tasks.

Few-shot means giving the model a few examples before your actual question. The examples teach it the format and decision boundary you want, often more effectively than any written instruction.

Chain-of-thought asks the model to reason step by step before giving a final answer. This works especially well for logic or math questions — by generating the intermediate steps, the model produces better final answers.

Role prompting gives the model a specific persona via the system message. The same question asked to "an opinionated senior engineer" versus "a patient teacher explaining to a beginner" will come back completely differently.

Constraint prompting tells the model what not to do, or forces a specific output format. This is essential when you need to parse the output programmatically.

We'll try all of these in the exercises below.

Setup

pip install ollama
ollama pull llama3.2

Quick check to make sure everything's working:

import ollama

r = ollama.chat(
    model="llama3.2",
    messages=[{"role": "user", "content": "Say hello in one sentence."}]
)
print(r.message.content)

Exercise 1 — Temperature in action

Run this a few times and watch what happens:

import ollama

def ask(temp, prompt):
    r = ollama.chat(
        model="llama3.2",
        options={"temperature": temp},
        messages=[{"role": "user", "content": prompt}]
    )
    return r.message.content

prompt = "Give me a random word."

for temp in [0, 0.5, 1.0, 2.0]:
    print(f"temp={temp}: {ask(temp, prompt)}")

At temperature=0 you'll get the same word every single time. At temperature=2 it varies wildly on every run. Also try it with different prompts — you'll notice accuracy drops at higher temperatures.

Exercise 2 — System prompt impact

Same question, three completely different system prompts:

import ollama

def ask(system, question):
    r = ollama.chat(
        model="llama3.2",
        messages=[
            {"role": "system", "content": system},
            {"role": "user",   "content": question}
        ]
    )
    return r.message.content

question = "Should I use tabs or spaces?"

print("=== No system prompt ===")
print(ask("", question))

print("\n=== Opinionated engineer ===")
print(ask(
    "You are a very opinionated senior engineer. You have strong views and do not hedge.",
    question
))

print("\n=== Neutral teacher ===")
print(ask(
    "You explain topics neutrally, presenting all sides without taking a position.",
    question
))

The same question, three different personalities. That's the power of the system prompt.

Exercise 3 — Prompting patterns compared

Let's compare zero-shot, few-shot, and chain-of-thought on the same task:

import ollama

def ask(prompt):
    r = ollama.chat(
        model="llama3.2",
        options={"temperature": 0},
        messages=[{"role": "user", "content": prompt}]
    )
    return r.message.content

text = "The battery drains fast but the camera is incredible and fits perfectly in my pocket."

# Zero-shot
print("Zero-shot:")
print(ask(f"Classify this review as positive, negative, or mixed:\n\n{text}"))

# Few-shot
print("\nFew-shot:")
print(ask(f"""Examples:
"Great screen, slow processor." → mixed
"Best phone ever!" → positive
"Broke after a week." → negative

Classify: "{text}" →"""))

# Chain-of-thought
print("\nChain-of-thought:")
print(ask(f"""Classify this review as positive, negative, or mixed.
Think step by step before giving your final answer.

Review: "{text}" """))

Pay attention to how the few-shot examples shape the output format, and how chain-of-thought produces visible reasoning before the final answer.

Exercise 4 — Manual conversation memory

This one is important. It shows how you manage conversation history yourself — which is exactly what every chat application does under the hood:

import ollama

def chat():
    history = []
    system = "You are a concise assistant. Keep answers under 3 sentences."

    print("Chat started. Type 'quit' to exit.\n")

    while True:
        user_input = input("You: ").strip()
        if user_input.lower() == "quit":
            break

        history.append({"role": "user", "content": user_input})

        response = ollama.chat(
            model="llama3.2",
            messages=[{"role": "system", "content": system}] + history
        )

        reply = response.message.content
        history.append({"role": "assistant", "content": reply})

        token_estimate = sum(len(m["content"]) for m in history) // 4
        print(f"AI: {reply}")
        print(f"[~{token_estimate} tokens in context so far]\n")

chat()

Tell it your name early on, have a few turns of conversation, then ask "what's my name?" — it remembers, because your name is sitting in the history list. Watch the token count grow with every message. That growing list is exactly what will become a problem at scale, which is why we have a whole post on memory systems later in the series.

Wrapping up

LLMs are next-token predictors. They read tokens, not words. They have no memory between calls — you manage that yourself. Temperature controls how creative or focused the output is. And the system prompt is the single most powerful thing you can tweak to change how the model behaves.

These aren't just facts to memorise — every single one of them will come up again as we build more complex things in the posts ahead.

In Post #2, we give the LLM tools so it can stop just talking and start actually doing things. See you there. 🚀