Building an AI Model in Pure C++: What You Learn When You Stop Treating Transformers as a Black Box

Building an AI Model in Pure C++: What Changes When You Stop Treating the Transformer as a Black Box

Most developers meet modern AI from the outside.

You install a package, load a model, call generate(), and a few seconds later text appears on the screen.

That experience is useful, but it hides almost every engineering decision that makes the model possible.

What is the tokenizer actually producing?

What shape enters the embedding layer?

How are Query, Key, and Value formed?

Where is the causal mask applied?

What is stored in a KV cache?

What exactly must be written to disk if training is interrupted and you want a true resume rather than merely reloading the weights?

And what changes when the entire system is implemented as a native C++20 application instead of a Python orchestration layer?

These questions eventually became the foundation of my book, Pure C++ Transformers: Design, Tokenize, Train, Optimize, and Deploy a Decoder-Only Language Model from First Principles.

The project behind the book follows one principle throughout:

A Transformer should be treated as an engineered system with explicit mathematical, data, runtime, state, and verification contracts.

This article explores that idea from a developer's perspective.

---

The Transformer Is Bigger Than the Neural Network

A decoder-only language model is often introduced as a function that predicts the next token.

Mathematically, that is correct.

Operationally, it is incomplete.

A working system must include much more:

Text
  ↓
Tokenizer
  ↓
Token IDs
  ↓
Dataset / Context Windows
  ↓
Embeddings
  ↓
Decoder Blocks
  ↓
Vocabulary Logits
  ↓
Loss or Sampling
  ↓
Checkpoint / Generation / Chat

Each arrow represents a contract.

If the tokenizer produces an ID outside the vocabulary expected by the embedding table, the system is invalid.

If training uses one tokenizer and inference loads another, both files may be individually valid while the overall package is wrong.

If the architecture declares 12 attention heads but the hidden dimension cannot be divided correctly, the implementation should reject the configuration before training starts.

This is one of the strongest lessons I learned while building the reference implementation:

Native AI software becomes much easier to debug when invalid states are rejected early.

For example:

if (hidden_size % attention_heads != 0) {
    throw std::invalid_argument(
        "hidden_size must be divisible by attention_heads");
}

That check may look trivial.

It is not.

It protects every reshape that follows.

---

Why C++?

The obvious question is why anyone would build a Transformer learning project in C++ when Python already has an extraordinary AI ecosystem.

The answer is not simply performance.

A Python call and a C++ call may ultimately execute the same optimized CUDA or CPU kernel.

C++ does not magically make matrix multiplication faster.

What C++ gives us is something different:

visibility and control over the system boundary.

With native C++, several things become difficult to ignore:

• compiler and ABI compatibility
• native dependencies
• memory ownership
• CPU versus CUDA placement
• tensor dtype
• filesystem state
• runtime packaging
• thread boundaries
• explicit failure handling
• executable deployment

That makes C++ a particularly interesting language for learning the engineering underneath the abstraction.

For the numerical layer, the book uses LibTorch, the official C++ frontend to the PyTorch tensor ecosystem.

That choice is intentional.

“From first principles” should not mean rewriting GEMM kernels, automatic differentiation, and CUDA primitives from scratch.

Instead, the architecture remains visible while trusted native numerical primitives do the low-level work.

---

Start with Tensor Contracts

Suppose an embedding tensor has the shape:

[B, T, D]

where:

B = batch size
T = sequence length
D = hidden dimension

For example:

[4, 512, 384]

means four sequences, each containing 512 token positions, each represented by a 384-dimensional hidden vector.

Now suppose the model uses six attention heads.

The head dimension becomes:

384 / 6 = 64

A Query projection initially produces:

[B, T, D]

Then it is reshaped:

auto query = query_projection(input)
    .view({
        batch,
        time,
        attention_heads,
        head_dim
    })
    .transpose(1, 2);

The logical shape becomes:

[B, H, T, d]

This is where many Transformer bugs begin.

Not in the published mathematics.

In the implementation of dimensions, strides, broadcasting, devices, and dtypes.

A transpose may make a tensor non-contiguous.

A later view may therefore require:

auto output = attention_output
    .transpose(1, 2)
    .contiguous()
    .view({batch, time, hidden_size});

These details are not glamorous, but they are exactly what separates a diagram from working software.

---

Attention Is Simple on Paper and Dangerous in Code

The central attention expression is familiar:

[

Attention(Q,K,V)

softmax
\left(
\frac{QK^T}
{\sqrt{d_k}}
+
M
\right)V
]

The basic C++ flow looks roughly like this:

auto q = query_projection(x);
auto k = key_projection(x);
auto v = value_projection(x);

auto scores =
    torch::matmul(
        q,
        k.transpose(-2, -1)
    ) * scale;

scores = scores.masked_fill(
    future_mask,
    -std::numeric_limits<float>::infinity()
);

auto probabilities =
    torch::softmax(scores, -1);

auto output =
    torch::matmul(probabilities, v);

The interesting question is not whether the code compiles.

The interesting question is:

How do we prove the implementation is causal?

A decoder-only language model must not see future tokens.

Consider:

A B C D

While predicting C, the model may use:

A B

but it must not read D.

That means the logical attention structure should behave like:

       KEY
       0 1 2 3

Q 0   ✓ X X X
U 1   ✓ ✓ X X
E 2   ✓ ✓ ✓ X
R 3   ✓ ✓ ✓ ✓
Y

Now comes an important engineering point.

A causal-mask bug can still produce:

• valid tensor shapes
• finite logits
• successful compilation
• decreasing training loss

The model may appear healthy.

So the reference project includes a causal-invariance test.

Change a future token.

Then verify that the logits at earlier positions do not change.

That is a much stronger definition of correctness than:

“The program ran.”

---

RoPE Adds Position Without Changing the Basic Attention Idea

A Transformer needs positional information.

The model otherwise has no intrinsic understanding that token 10 appears after token 9.

The reference architecture uses Rotary Position Embedding, or RoPE.

RoPE rotates pairs of Query and Key features using position-dependent sine and cosine values.

Conceptually, the model does not merely ask:

What is this token?

It asks something closer to:

What is this token, in this position, relative to the other tokens?

The implementation also exposes an important inference problem.

During normal full-sequence processing, positions may begin at zero.

During incremental generation using a KV cache, the next token may begin at position 742.

The RoPE offset must therefore be correct.

A cache can have perfectly valid tensor shapes and still be semantically wrong if one layer believes the new token is position 742 while another believes it is position 743.

This is exactly the type of bug that motivates parity tests.

---

Grouped-Query Attention Is an Engineering Trade-Off

Standard multi-head attention may use a separate Key and Value head for every Query head.

Grouped-Query Attention changes that.

For example:

Query Heads = 12
KV Heads    = 4

Several Query heads share one Key/Value head.

Why?

Because during autoregressive inference, Keys and Values are cached for every layer and position.

Reducing the number of KV heads reduces:

• cache memory
• memory bandwidth
• inference cost

while keeping multiple Query heads.

This is a good example of how Transformer architecture and runtime engineering are deeply connected.

An architecture decision changes the memory behavior of the deployed model.

---

The Feed-Forward Network Is Not a Minor Component

Attention receives most of the attention.

The feed-forward network often receives much less.

That is misleading.

Modern decoder blocks usually dedicate a substantial fraction of their parameters to the FFN.

The reference model uses a SwiGLU-style structure.

Conceptually:

x
│
├── gate projection ── SiLU ──┐
│                             × ── down projection
└── up projection ────────────┘

A simplified implementation looks like:

auto gate = torch::silu(gate_projection(x));
auto up   = up_projection(x);

auto hidden = gate * up;

return down_projection(hidden);

The multiplication between the gate and up paths gives the network a learned mechanism for controlling which features are amplified or suppressed.

The full decoder block then becomes something like:

Input
  │
  ▼
RMSNorm
  │
  ▼
Causal Attention
  │
  ├──────────────┐
  ▼              │
Residual Add ◄───┘
  │
  ▼
RMSNorm
  │
  ▼
SwiGLU FFN
  │
  ├──────────────┐
  ▼              │
Residual Add ◄───┘
  │
  ▼
Output

Repeat that block 12, 24, or 40 times and we begin to have a real language model architecture.

---

Tokenization Is Not Just Preprocessing

One of the most underestimated design choices in small and medium AI models is vocabulary size.

Imagine a model with:

Hidden Size = 384

and a vocabulary of:

16,000

The embedding matrix alone contains:

16,000 × 384
=
6,144,000 parameters

Increase the vocabulary to:

20,000

and the embedding becomes:

20,000 × 384
=
7,680,000 parameters

That is an additional:

1,536,000 parameters

before adding a single Transformer block.

So tokenizer design is architecture design.

This becomes even more interesting for multilingual models.

In the book, I use an English-Arabic tokenizer study to illustrate the problem.

Imagine several tokenizer candidates processing a 100-word Arabic sample:

10K Unigram → 182 tokens
16K Unigram → 154 tokens
20K Unigram → 146 tokens
16K BPE     → 161 tokens

The 20K tokenizer gives the shortest sequence.

Does that automatically make it the best choice?

No.

The larger vocabulary also increases model parameters.

The correct decision depends on:

• Arabic fertility
• English fertility
• unknown-token behavior
• technical vocabulary
• mixed Arabic-English text
• numerical notation
• file paths and code fragments
• embedding parameter cost

Tokenizer quality is therefore part of the AI model, not an unrelated utility.

---

Training Is a Transaction

Once the model architecture and data pipeline are correct, training can begin.

A training update is conceptually:

Batch
  ↓
Forward
  ↓
Cross-Entropy
  ↓
Backward
  ↓
Gradient Check
  ↓
Gradient Clip
  ↓
Learning Rate
  ↓
AdamW Step
  ↓
Checkpoint / Metrics

A simplified C++ implementation might look like:

optimizer.zero_grad();

auto logits = model(input);

auto loss =
    torch::nn::functional::cross_entropy(
        logits.view({-1, vocab_size}),
        target.view({-1})
    );

loss.backward();

torch::nn::utils::clip_grad_norm_(
    model->parameters(),
    max_gradient_norm
);

optimizer.step();

Again, the short version hides the important engineering questions.

Is loss finite?

Are gradients finite?

What is the global gradient norm?

What learning rate was actually used?

Which dataset window produced this update?

What state must be saved if execution stops now?

---

A Checkpoint Is More Than Model Weights

One mistake I wanted the book to address directly is the assumption that:

torch::save(model, "model.pt");

is sufficient to resume training.

It is sufficient to save model parameters.

It is not necessarily sufficient for exact continuation.

A training checkpoint may need:

model.pt
optimizer.pt
config.cfg
dataset state
completed step
RNG state
learning-rate position
tokenizer identity

Why dataset state?

Suppose training randomly selects windows from the token stream.

You save only the original seed.

After restarting, the random generator does not necessarily return to the same location in its sequence.

The resumed run selects a different next batch.

The training continues, but it does not continue identically.

In the companion project, deterministic CPU qualification compares uninterrupted training against interrupted-and-resumed training.

The goal is not just:

“Resume loaded.”

The stronger question is:

“Did resume reconstruct the same state transition?”

---

KV Cache Must Be Proven, Not Assumed

Autoregressive generation repeatedly predicts one token.

Without caching, every new token may require recomputing Keys and Values for the entire prompt.

Suppose the prompt contains 1,000 tokens.

At the next step:

token 1001

we should not need to regenerate the K/V tensors for the first 1,000 positions.

A KV cache preserves them.

The workflow becomes:

Prompt
  ↓
Prefill
  ↓
Store K/V
  ↓
New token
  ↓
Compute only new Q/K/V
  ↓
Append K/V
  ↓
Generate again

This optimization is powerful.

It is also dangerous.

Incorrect cache offsets can produce plausible text.

That means visual inspection of generated output is not enough.

The correct test is:

full forward output
        vs
cached forward output

within numerical tolerance.

The reference implementation explicitly tests both prefill parity and single-token decode parity.

---

Generation Is Its Own Engineering Layer

After the model produces logits, we still need to decide how to select the next token.

Greedy decoding chooses:

argmax(logits)

It is deterministic and useful for testing.

But generation systems typically need more control.

Temperature

Given logits (z):

[
z' = \frac{z}{T}
]

Lower temperature sharpens the distribution.

Higher temperature flattens it.

Top-k

Keep only the best k candidates.

For example:

k = 40

Everything outside the 40 highest-scoring tokens is removed.

Top-p

Keep the smallest set of tokens whose cumulative probability reaches:

p = 0.90

This adapts candidate count dynamically.

Repetition Penalty

Previously generated tokens may be penalized to reduce repetitive loops.

These mechanisms demonstrate another important point:

Model behavior is a function of both learned weights and inference policy.

---

PowerShell as an Operations Layer

One unusual decision in the project is to make PowerShell the primary Windows operations interface.

Instead of requiring users to remember a long series of Visual Studio menus, the system exposes commands.

For example:

.\scripts\Build.ps1 `
    -LibTorchRoot D:\Libraries\libtorch-cpu `
    -Configuration Release `
    -Clean

The build script can:

1. enter the MSVC x64 environment
2. confirm cl.exe
3. confirm CMake
4. confirm Ninja
5. validate the LibTorch root
6. configure CMake
7. build
8. execute tests
9. confirm the executable exists

Then:

.\scripts\Self-Test.ps1

qualifies the basic implementation before long training begins.

The executable itself exposes distinct lifecycle operations such as:

pct design
pct inspect
pct train-tokenizer
pct prepare-data
pct train
pct generate
pct chat
pct self-test

This separation is valuable because each stage can fail early.

A tokenizer problem should fail before GPU training begins.

A model-configuration problem should fail before dataset preparation.

A CUDA request should fail explicitly if CUDA is unavailable.

---

Why “Fail Closed” Matters

Consider this logic:

if (requested == "cuda" &&
    !torch::cuda::is_available()) {

    throw std::runtime_error(
        "CUDA was requested but unavailable");
}

Some applications silently switch to CPU.

That sounds friendly.

But suppose the user expected a two-hour GPU experiment.

Silent CPU fallback could turn that experiment into something that runs for days.

The program has changed the user's execution policy without permission.

A better distinction is:

device=auto

may choose CPU or CUDA.

But:

device=cuda

should mean CUDA.

If CUDA is unavailable, fail.

This is one of those details that has nothing to do with the attention equation but everything to do with building dependable AI software.

---

The Most Useful Tests Are Often Not Accuracy Tests

Before serious training, the project asks different questions.

Does changing a future token affect earlier logits?

Does exact parameter counting match the instantiated model?

Does KV-cache inference match full inference?

Does the training loss remain finite?

Does backward produce finite gradients?

Can a checkpoint be saved atomically?

Can the checkpoint be loaded?

Does resumed deterministic CPU training match uninterrupted training?

Can the model generate at least one token from the saved checkpoint?

These tests do not prove that the model is intelligent.

They prove something more basic:

the machinery is behaving according to its contracts.

Capability comes later.

---

Scaling Is Not a Magic Switch

The book includes worked examples for small and larger reference configurations because parameter count, memory, context, and training tokens are deeply connected.

A larger model is not automatically a better model.

Suppose we increase:

D = hidden dimension

That affects:

• embedding width
• attention projections
• FFN dimensions
• activation memory
• parameter count

Increase context length:

T

and naïve attention-score storage grows roughly with:

[
T^2
]

Double the context and attention cost does not simply double.

This is why a model that works at context 512 may behave very differently at context 2,048.

Likewise, a 100M parameter model fitting in system RAM does not imply that full FP32 training fits comfortably on a 4 GB GPU.

Weights are only one part of memory.

Training also needs:

• gradients
• Adam first moment
• Adam second moment
• activations
• attention workspaces
• temporary tensors

Engineering decisions must therefore be measured rather than guessed.

---

What Building the System Changed for Me

Perhaps the most important lesson from writing Pure C++ Transformers was that a Transformer stops feeling mysterious once its boundaries become explicit.

At first we see:

AI.

Then we see:

tokens
embeddings
Q
K
V
attention scores
softmax
residual streams
SwiGLU
logits
loss
gradients
optimizer state
KV cache
checkpoints

Eventually the black box disappears.

That does not make modern AI less impressive.

For me, it makes it more impressive.

There is no tiny intelligence hidden inside the executable.

There are well-defined mathematical operations connected to a large optimization process.

And yet, when enough parameters, data, compute, and structure come together, the resulting system can generate language.

That is an extraordinary engineering result.

---

About the Book

Pure C++ Transformers is written for developers who want to move beyond AI APIs and examine the machinery underneath a decoder-only language model.

The project built throughout the book uses:

• C++20
• LibTorch
• CMake
• Ninja
• PowerShell
• MSVC on Windows
• optional CUDA execution
• optional SentencePiece tokenization

The model architecture covers:

• RMSNorm
• RoPE
• causal self-attention
• Grouped-Query Attention
• SwiGLU
• tied embeddings
• AdamW
• warmup and cosine decay
• gradient clipping
• checkpoints and resume
• Top-k and Top-p sampling
• repetition control
• KV caching
• command-line chat
• runtime qualification

The goal is not to pretend that a small workstation can reproduce a frontier-scale language model.

The goal is to make the complete mechanism understandable, executable, testable, and extensible.

If you can already call an AI model but have started wondering what happens beneath generate(), the journey becomes much more interesting once you follow the tensors yourself.

Because there is a significant difference between saying:

“I know how to use an AI model.”

and saying:

“I understand how its major systems fit together, and I can build one.”
Get full Book