How AI Models Work for Dummies

AI models live two lives. First, they learn. Second, they perform.

We call these Training and Inferencing. If training is the four years of medical school, inferencing is the actual surgery. In short all that knowledge it learned is put to the test against real, messy, unseen data.

The Two Stages of an AI Model

Every AI model goes through two primary phases. Think of it as learning versus doing.

During training, a model processes millions of labeled examples. It figures out the statistical relationships between data points and encodes those relationships into model weights. Weights are the digital "neurons" that connect ideas.

During inferencing, we feed the model new data (like a user's query or a fresh email). The model uses its stored weights to make a prediction or generate an answer.

Example

Let's make this concrete with a classic use case: Spam Detection.

1. Training: we feed the model thousands of emails already labeled "Spam" or "Not Spam." It learns that words like "lottery," "urgent," and excessive "!!!" usually mean trouble 😁.
2. Inferencing: a new email arrives in your inbox. The model compares this email to the patterns it learned and spits out a probability score.

The magic of inferencing is generalization. The model can predict a spam email it has never seen before, just because it shares characteristics with past spam.

Cost

Honestly I had this vague impression of training being super duper expensive. But then I got to know as expensive as training can be, it is dwarfed by the expense of inferencing 🥲.

Training a frontier model can cost tens or even hundreds of millions of dollars. But guess what? That can become pocket change compared to the aggregate cost of serving billions of user queries over the model's lifetime.

• Scale: training is a relatively infrequent, concentrated expense. Inference happens millions of times per day.
• Speed: we expect answers almost instantly. That requires powerful (and power-hungry 😅) accelerators running continuously.
• Complexity: LLMs have billions of parameters. A standard dense model uses essentially all of its parameters for each token it processes.
• Carbon Footprint: unlike training, inference is an ongoing source of energy consumption. At sufficient scale, it can account for a substantial share of a model's lifetime energy use.

One nuance: "inference can dwarf training" is not universally true. For a model that is trained but barely used, training will obviously dominate.

💡 Tip

A token is the smallest chunk of text that an AI model actually reads or generates. It is not always whole word. For example:

• "unbelievable" → generates 2 tokens when using google/gemma-7b ["un", "believable"].
• "How are you?" → generates 4 tokens when using codellama/CodeLlama-7b-hf: ["How", " are", " you", "?"].

You can play with it here: https://tokenizer.model.box

Why you actually care?

• Cost: most AI pricing (OpenAI, Anthropic, etc.) is billed per token (input + output). More tokens = more money spent.
• Speed: inference speed is measured in tokens per second. Bigger outputs take longer.
• Memory: models have a "context window" (e.g., 128k tokens). Hit that limit, and the model forgets the start of the conversation.