A few weeks ago, I started seriously learning how Large Language Models actually work.
Not how to use ChatGPT.
Not how to write better prompts.
Not how to call an LLM API from Python.
I mean what actually happens after I press Enter.
At first, it felt like opening a completely different field of computer science.
I kept running into terms like:
tokens, embeddings, attention, transformers, logits, softmax, temperature, context windows, fine-tuning, RLHF, RAG, vector databases, tool calling, agents, mixture of experts...
I understood each term individually.
But I didn't understand how they connected.
And I think that's the hardest part of getting into AI today.
There is no shortage of explanations for individual concepts.
What's missing, at least for me, was a mental model that connected everything together.
So I decided to build one.
This article is my attempt to explain it from the perspective of a software engineer who is learning LLMs, without starting with complicated research papers or assuming you're already an ML expert.
Why should software engineers understand this?
There's a practical reason I wanted to learn this.
Look at software engineering job descriptions today and you'll increasingly see things like:
- LLM integration
- RAG
- Embeddings
- Vector databases
- AI agents
- Prompt engineering
- Generative AI
- Tool calling
- AI/ML APIs
AI is becoming another layer of the software stack.
And I don't think every software engineer needs to become an ML researcher.
But I do think we should understand what we're actually building with.
If I'm integrating an LLM into a backend service, I don't necessarily need to know every detail of the model's training infrastructure.
But I should understand enough to answer questions like:
What exactly is a token?
Why does the model have a context window?
What is an embedding?
Why does attention matter?
Why does the model sometimes produce different answers?
Why would I need RAG?
What actually happens when an LLM calls a tool?
Once those questions start making sense, the rest of the ecosystem becomes much easier to navigate.
Let's start with the simplest possible question
What happens when I send text to an LLM?
Suppose I send:
"Explain Docker to me."
As a human, I see a sentence.
The model doesn't.
The first thing that happens is tokenization.
1. Text → Tokens
Neural networks don't operate directly on strings like:
"Explain Docker to me."
They operate on numerical representations.
So the text is broken into tokens.
A token isn't necessarily a word.
Depending on the tokenizer, it could be:
- a whole word
- part of a word
- punctuation
- whitespace
- or another small piece of text
For example, conceptually:
"unbelievable"
↓
["un", "believ", "able"]
The exact split depends on the tokenizer.
Each token is then mapped to an integer ID:
["un", "believ", "able"]
↓
[1254, 8271, 394]
Now we have something a neural network can process.
Why do we tokenize this way?
Because language is messy.
If we created one token for every possible word, we'd need an enormous vocabulary.
Consider:
run
runs
running
runner
runners
rerunning
...
A subword tokenizer can reuse pieces of words instead of storing every possible variation as a completely independent token.
One famous approach is Byte Pair Encoding (BPE), which builds a vocabulary by repeatedly merging frequently occurring sequences.
So the first mental model is:
LLMs don't read text directly. They process token representations.
2. Context Windows: The Model Can't See Everything
Now imagine sending a huge document to an LLM.
There is a limit to how much information can participate in a single model invocation.
That's the context window.
You might see models advertised with context sizes such as:
8K
32K
128K
1M+
These refer to the amount of tokenized context the model can handle.
Think of it as the model's working context for a particular inference request.
This is important because it explains why applications often need techniques such as:
- chunking
- summarization
- retrieval
- conversation memory
- RAG
If you understand context windows, a lot of modern LLM application architecture starts making more sense.
3. Token IDs → Vectors
We now have something like:
[1254, 8271, 394]
But these integers are just IDs.
There's nothing inherently meaningful about the number 8271.
So we need another transformation.
Each token ID is mapped to a vector of floating-point numbers.
Something like:
8271
↓
[0.12, -0.43, 0.87, 0.04, ...]
This is an embedding.
And this is one of the concepts that completely changed how I think about machine learning.
Instead of representing information as a label, we represent it as a position in a high-dimensional mathematical space.
The model can learn useful relationships between these representations.
4. Embeddings Are More Than "Meaning = A Vector"
You'll often hear:
"Embeddings represent the meaning of words."
That's a useful beginner intuition, but it's an oversimplification.
Modern neural representations are distributed and contextual.
There isn't necessarily one dimension that means:
"This number represents happiness."
Instead, meaning and useful linguistic patterns are distributed across many dimensions.
The model learns these representations because they're useful for predicting and processing language.
And this idea existed long before today's LLMs.
5. Word2Vec: An Early Glimpse of What Was Possible
One of the famous milestones in NLP was Word2Vec, introduced in 2013.
It demonstrated that words appearing in similar contexts could develop similar vector representations.
One famous example is:
King - Man + Woman ≈ Queen
The important part isn't that this equation is some universal law of language.
The important part is what it demonstrated:
Relationships between linguistic concepts can emerge in vector space.
That idea became incredibly important for modern NLP.
6. So How Does the Model Understand Context?
This is where we reach the Transformer.
In 2017, researchers published the paper:
Attention Is All You Need
It introduced the Transformer architecture.
Transformers eventually became the foundation of modern LLMs.
And the most famous part of the Transformer is:
Attention
Consider these two sentences:
I went to the bank to deposit money.
and:
I sat beside the bank of the river.
The word bank is the same.
But its meaning depends heavily on the surrounding words.
This is where attention becomes important.
7. Attention: Which Tokens Matter Right Now?
A simplified way to think about attention is:
Each token can look at other tokens and determine which ones are relevant to its current representation.
For example:
"The animal didn't cross the road because it was tired."
To understand "it", the model needs to consider other parts of the sentence.
Attention provides a mechanism for modeling those relationships.
Under the hood, this involves:
Query (Q)
Key (K)
Value (V)
and a calculation commonly expressed as:
You don't need to memorize this equation to understand the big picture.
The important idea is:
Attention allows the model to dynamically determine which parts of the context should influence each token's representation.
And that's a huge part of why Transformers work so well with language.
8. A Transformer Is More Than Attention
A Transformer block isn't just attention.
A simplified view looks something like:
Input
↓
Self-Attention
↓
Feed-Forward Network
↓
Output
Real Transformer architectures also include things like:
- residual connections
- normalization
- positional information
- masking, depending on the architecture
And an LLM doesn't have just one Transformer block.
It stacks many of them.
The output of one layer becomes the input to the next.
Layer after layer, the model transforms the representation.
9. Then Comes the Part That Surprised Me
After all of that processing, the model doesn't output:
"Hello, here is your answer."
Instead, it produces scores for possible next tokens.
These raw scores are called logits.
Imagine the model has a vocabulary containing thousands of possible tokens.
For the next token, it might produce something conceptually like:
"the" → 8.2
"container" → 7.4
"Docker" → 6.9
"database" → 4.1
"banana" → 0.7
...
Those aren't probabilities yet.
They're logits.
10. Logits → Probabilities
To turn those scores into a probability distribution, we can apply Softmax.
Now we might have:
"the" → 0.42
"container" → 0.28
"Docker" → 0.19
"database" → 0.08
"banana" → 0.01
The exact values are illustrative, but the idea is important.
The model has effectively produced a probability distribution over possible next tokens.
And now comes something even more important:
The model chooses a token.
11. The Model Generates One Token at a Time
This was probably the most important thing for me to understand.
When ChatGPT gives you a paragraph, the model isn't necessarily generating the whole paragraph in one shot.
It's generating tokens sequentially.
Conceptually:
Input:
"Explain Docker"
↓
"Explain Docker is"
↓
"Explain Docker is a"
↓
"Explain Docker is a platform"
↓
"Explain Docker is a platform for"
↓
...
Each generated token becomes part of the context used to generate the next token.
This is called autoregressive generation.
So the basic inference loop is:
Tokenize
↓
Represent tokens
↓
Transformer
↓
Logits
↓
Probabilities
↓
Choose next token
↓
Append token
↓
Repeat
Over and over.
Until the generation stops.
12. Why Doesn't It Always Give the Same Answer?
If the model always selected the highest-probability token, its output would be much more deterministic.
Instead, generation can involve sampling.
Two parameters you'll often encounter are:
Temperature
Temperature controls how concentrated or spread out the probability distribution becomes.
Lower temperature generally means:
More predictable output.
Higher temperature generally means:
More variation.
Top-p
Top-p, or nucleus sampling, limits sampling to a subset of tokens whose cumulative probability reaches a specified threshold.
So:
Temperature changes the shape of the distribution.
Top-p limits the candidate pool.
This is one reason the same prompt can produce different answers.
13. But Where Did the Model Learn All of This?
This brings us to training.
And there's an important distinction:
An LLM doesn't start life as a helpful chatbot.
A simplified training lifecycle looks something like:
Massive Dataset
↓
Pretraining
↓
Base Model
↓
Supervised Fine-Tuning
↓
Preference / Alignment Training
↓
Assistant
Let's unpack that.
14. Pretraining: Learn to Predict What Comes Next
The basic objective during language-model pretraining is surprisingly simple:
Predict the next token.
Given:
"The capital of France is"
the model should assign high probability to:
"Paris"
Do this over an enormous amount of data, with enormous models and compute, and something interesting happens.
The model doesn't just memorize a list of sentences.
It learns statistical structures and representations that support language prediction.
It can learn patterns involving:
- syntax
- programming
- facts
- concepts
- styles
- relationships
- reasoning-like patterns
The result is a base language model.
But a base model isn't necessarily optimized to be a great assistant.
15. Supervised Fine-Tuning
Now imagine giving the model examples like:
User:
What is an API?
Assistant:
An API is...
and many other high-quality examples.
The model can be trained to follow this conversational format.
This is commonly called:
SFT — Supervised Fine-Tuning.
It helps transform the raw language model into something much closer to an assistant.
16. Preference Training and RLHF
But there's another problem.
Suppose the model gives three answers.
One is:
- correct
- clear
- safe
- helpful
Another is technically correct but confusing.
The third is simply wrong.
We need a way to teach the model which behaviors humans prefer.
One historically important approach is:
RLHF — Reinforcement Learning from Human Feedback.
Humans provide preference signals, and the training process uses those signals to encourage desirable behavior.
Modern models use a broader range of techniques for alignment and preference optimization, but the central idea remains:
The model isn't only trained to predict text. It is also optimized to produce useful behavior.
17. Now We Reach the Part Software Engineers Care About
Understanding how the model works is useful.
But the really exciting part is what happens when we put an LLM inside a software system.
This is where terms like:
RAG
Vector databases
Tool calling
Agents
start appearing.
And suddenly, all the concepts we've discussed begin connecting.
18. Embeddings + Vector Databases + RAG
Imagine you're building an internal company assistant.
You want it to answer questions about:
- company policies
- technical documentation
- product manuals
- internal knowledge
- recent documents
You can't simply expect the model to magically know your private documents.
And retraining the entire model every time a document changes isn't practical.
So we can use Retrieval-Augmented Generation (RAG).
A simplified pipeline:
Documents
↓
Chunking
↓
Embeddings
↓
Vector Database
Then when a user asks:
"What is our vacation policy?"
the system can do:
Question
↓
Embedding
↓
Vector Search
↓
Relevant Documents
↓
LLM + Retrieved Context
↓
Answer
Now the model isn't responsible for remembering everything.
Your application retrieves the relevant information and gives it to the model as context.
This is a perfect example of where traditional software engineering and LLMs meet.
19. Tool Calling: When the LLM Needs Software
An LLM can generate text.
But your application can do much more.
Your backend can:
- query a database
- call an API
- perform calculations
- search documents
- send an email
- create an order
- retrieve weather information
- interact with another service
So why not let the model decide when one of these capabilities is needed?
That's where tool calling comes in.
A simplified flow:
User
↓
LLM
↓
"I need the weather tool"
↓
Tool Call
↓
Your Backend
↓
Weather API
↓
Tool Result
↓
LLM
↓
Final Response
The LLM isn't directly performing the API request.
Your application executes the tool.
The model receives the result and continues.
This distinction is extremely important.
The LLM becomes a reasoning and language interface around actual software capabilities.
20. And That's Where Agents Come From
Once you combine:
LLM + tools + state + an execution loop
you can start building systems that behave more like agents.
For example:
Goal
↓
LLM decides what to do
↓
Call Tool A
↓
Observe Result
↓
Decide next action
↓
Call Tool B
↓
Observe Result
↓
Final Answer
This is why I think it's more useful to understand the underlying pieces than to simply memorize:
"Agent = AI that does things."
The interesting engineering questions are:
- What tools does it have?
- How does it select them?
- What state does it maintain?
- How do we control its actions?
- What happens when a tool fails?
- How do we prevent unwanted actions?
- How do we evaluate it?
Now we're back in familiar software engineering territory.
21. Mixture of Experts
Another term you'll encounter when exploring modern LLM architectures is:
Mixture of Experts (MoE).
The basic idea is that instead of having every part of a huge network process every token, the model can contain multiple expert components.
A router decides which experts should process a token.
Conceptually:
┌── Expert A
│
Token → Router ───┼── Expert B
│
└── Expert C
Only a subset of experts may be activated for each token.
This can make it possible to build models with very large total parameter counts while controlling the amount of computation used per token.
Again, the important thing isn't memorizing the architecture.
It's understanding why the technique exists.
22. What About Models That "Think"?
This is another area where terminology can become confusing.
You may hear:
- reasoning models
- thinking models
- chain-of-thought
- extended thinking
The core intuition is that some tasks benefit from giving the model additional computation before producing the final answer.
Because these models generate sequentially, intermediate reasoning or computation can provide additional steps for solving a difficult problem.
But there's an important distinction:
"Chain-of-thought" does not simply mean showing the user the model's private reasoning.
Modern systems may use internal reasoning processes that are not exposed verbatim.
From an engineering perspective, the important idea is:
Inference can involve additional computation, not just one immediate prediction.
23. How Do We Know an LLM Is Actually Good?
Here's where my software-engineering brain immediately kicks in.
We test it.
You wouldn't deploy a backend service and say:
"It seems to work."
You'd write tests.
You'd monitor it.
You'd measure performance.
The same philosophy applies to LLM applications.
Models can be evaluated against benchmarks and task-specific datasets.
There are also evaluation frameworks and "evaluation harnesses" that automate running models against collections of standardized tests.
And for production applications, the most important evaluation is often even more specific:
Does the model perform well on the tasks my users actually care about?
A model can perform extremely well on a benchmark and still perform badly in your specific application.
That's why evaluation is becoming a major part of AI engineering.
24. The Mental Model That Finally Made Everything Click
After going through all of this, I now visualize an LLM roughly like this:
USER INPUT
│
▼
TOKENIZATION
│
▼
TOKEN IDs
│
▼
EMBEDDINGS
│
▼
TRANSFORMER LAYERS
┌─────────┴─────────┐
▼ ▼
ATTENTION FFN
│ │
└─────────┬─────────┘
▼
LOGITS
│
▼
SOFTMAX
│
▼
PROBABILITIES
│
▼
SAMPLING
│
▼
NEXT TOKEN
│
└──────────────┐
│
▼
REPEAT THE LOOP
And around the model, we can build:
┌──────────────┐
│ LLM │
└──────┬───────┘
│
┌───────────┼───────────┐
▼ ▼ ▼
RAG Tools Memory
│ │
▼ ▼
Vector DBs APIs
That's the mental model I'm keeping.
25. The Part I Find Most Exciting
Here's what surprised me most.
The more I learned about LLMs, the less they felt like something completely separate from software engineering.
They started looking like another component in a software architecture.
Think about a traditional backend:
Frontend
↓
API
↓
Business Logic
↓
Database
↓
External Services
Now imagine adding an LLM:
Frontend
↓
API
↓
LLM
├── RAG → Vector Database
├── Tools → External APIs
├── Memory → Database
└── Guardrails / Evaluation
Suddenly, the skills you already have become relevant.
APIs.
Authentication.
Databases.
Caching.
Queues.
Observability.
Error handling.
Testing.
Cloud infrastructure.
Distributed systems.
The LLM is only one piece.
And I think that's a very important realization for software engineers trying to enter AI.
26. You Don't Need to Become an AI Researcher to Start
If you're a software engineer who wants to get into AI, I don't think the first step should be:
"I need to learn every mathematical detail of neural networks."
And I don't think it should be:
"I'll just learn how to call OpenAI's API."
There is a middle ground.
Understand the architecture first.
Understand the vocabulary.
Understand the flow.
Then start building.
For example, build a simple application that:
- Accepts a user's question.
- Calls an LLM.
- Stores conversations.
- Retrieves relevant documents.
- Uses embeddings.
- Performs vector search.
- Calls external tools.
- Evaluates the generated answers.
At that point, concepts that looked abstract start becoming practical.
27. If I Had to Reduce Everything to One Diagram
This is probably the diagram I'd keep if I forgot everything else:
┌────────────────────┐
│ YOUR APP │
└─────────┬──────────┘
│
▼
┌─────────────┐
│ INPUT │
└──────┬──────┘
│
▼
TOKENIZER
│
▼
TOKENS
│
▼
EMBEDDINGS
│
▼
┌─────────────────┐
│ TRANSFORMER │
│ │
│ Attention + FFN │
│ × N │
└────────┬────────┘
│
▼
LOGITS
│
▼
SOFTMAX
│
▼
NEXT TOKEN
│
▼
REPEAT / LOOP
│
▼
RESPONSE
And around that core:
RAG ───────> External Knowledge
Tools ─────> External Actions
Memory ────> Persistent State
Evaluation → Quality Measurement
That, for me, is the foundation.
Final Thoughts
I'm still learning.
There are many things I haven't covered here: positional encodings, KV caching, quantization, distributed training, inference optimization, LoRA, multimodal models, RL techniques, model architectures, serving infrastructure, and much more.
But I don't think you need to understand everything before you start building.
What helped me most was going one layer at a time:
Text → Tokens → Vectors → Attention → Transformer → Logits → Probabilities → Next Token
Then:
Pretraining → Fine-tuning → Alignment
And finally:
LLM → RAG → Tools → Agents → Production AI Systems
Once you see those connections, the endless stream of AI terminology becomes much less intimidating.
You stop seeing "RAG", "embeddings", "tool calling", and "agents" as completely separate technologies.
You start seeing them as pieces of a larger system.
And perhaps that's the most useful lesson I've taken from learning LLMs so far:
You don't need to understand every detail of AI before you start building with it.
But you should understand the foundations well enough to know what is happening underneath the abstraction.
That's the point where AI stops feeling like magic.
And starts feeling like engineering.
If you're a software engineer starting your own journey into LLMs, I hope this mental model saves you some of the confusion I had when I started.
There's a lot more to learn.
But now I finally feel like I know where the pieces belong.
Top comments (0)