Karpathy published nanochat on October 13, 2025. Four hours on an 8×H100 node, roughly $100 of cloud spend, a ChatGPT-shaped model with a web UI at the end of it. The bit that got the least attention on release is the one I want to talk about: the optional reinforcement learning step in scripts/chat_rl.py.
It is 300-ish lines. It is labeled GRPO. It does not do what GRPO does. And once you read what it actually does, a lot of the recent "we trained our own reasoning model with GRPO" claims start to look less impressive, because the honest version fits in a script that Karpathy left out of the speedrun.
What the RL step is trying to fix
Post-SFT, the model can hold a conversation. It can answer greetings, follow the tool-use grammar, look competent. Hand it a GSM8K word problem and it will set up the equation cleanly and then get the arithmetic wrong. The setup improved from pre-training. The last-mile numeric answer did not.
The RL step targets this one gap. It picks GSM8K because the answers are machine-checkable: every GSM8K entry ends in a #### 42-style marker, so the reward function is a regex, a string comparison, and a float cast. No reward model. No human preference labels. Nothing you could have written wrong without noticing.
GSM_RE = re.compile(r"#### (\-?[0-9\.\,]+)")
def extract_answer(completion):
match = GSM_RE.search(completion)
if match:
return match.group(1).strip().replace(",", "")
return None
The reward is 1 if the extracted number matches the ground truth. 0 if it does not. That is the entire supervision signal for the reinforcement learning step, and it is enough to move accuracy noticeably. The lesson is not "GSM8K is easy," it is that when the reward is unambiguous the algorithm barely matters.
The loop: sample 16, subtract the mean, update
The training loop has three lines that carry the whole idea. For each problem:
- Render the prompt through the tokenizer up to the assistant's turn.
-
Engine.generate_batchproduces 16 completions (default;--num-samples). - Score each one, subtract the mean, use the result as an advantage.
rewards = torch.tensor(rewards, dtype=torch.float, device=device)
mu = rewards.mean()
advantages = rewards - mu
That is it. If all 16 completions get it right, mean is 1, advantages are all 0, and the update is a no-op. If all 16 get it wrong, mean is 0, advantages are all 0, and the update is a no-op. Only the rounds where some got it right and some got it wrong produce gradient. The group is doing the work a value function normally does in PPO.
That grouping is the one thing that connects the code to the paper GRPO comes from. DeepSeek introduced GRPO in the DeepSeekMath paper and then used it to train R1, and the "G" is exactly this: no critic, score each response relative to the group mean.
Why Karpathy put "GRPO" in quotes in his own comments
The chat_rl.py docstring says, in more words than this, "this is called GRPO but it is really REINFORCE with a mean baseline." He lists four reasons the implementation is stripped down compared to the DeepSeek recipe:
| GRPO paper feature | What nanochat does | What is missing |
|---|---|---|
| KL penalty against a reference model | Nothing | No trust region, no anchor |
| PPO-style ratio + clipping | Nothing | On-policy only, no clipping |
| Advantage normalization per-sequence | Per-token (DAPO-style) | Different denominator |
| Advantage = (r - μ) / σ | Advantage = (r - μ) | No z-score, just subtract mean |
Each row is a safety rail the full GRPO recipe uses. Each is missing here. What is left is the plainest possible policy gradient: sample, score, subtract the average, push good samples up and bad samples down. This is roughly what Snorkel calls the "no critic" pillar of GRPO with the trust-region pillar removed as well.
The objective itself is one short block:
logp = -model(inputs, targets, loss_reduction='none').view_as(inputs) # (B, T)
pg_obj = (logp * advantages.unsqueeze(-1)).sum()
num_valid = (targets >= 0).sum().clamp(min=1)
pg_obj = pg_obj / (num_valid * num_passes * examples_per_rank)
loss = -pg_obj
loss.backward()
Log-likelihood times advantage, summed, divided by the count of valid tokens, negated. The REINFORCE update from a 1990s textbook, in modern PyTorch. No KL term. No ratio. No clip. Not because the author does not know what those are; because for a machine-checkable reward, they are optional, and every optional part costs you complexity.
The tell: this step is not in the speedrun
runs/speedrun.sh is nanochat's headline path. It runs the tokenizer, pretraining, SFT, and eval. It does not call scripts.chat_rl. The README makes the reason explicit: float16 training relies on GradScaler to avoid gradient underflow, and the note says "SFT supports this, RL does not, currently."
Pretraining and SFT ship the guardrails. RL is opt-in. You run it by hand with torchrun --standalone --nproc_per_node=8 -m scripts.chat_rl if you feel like closing the GSM8K gap, and you skip it if the base chat quality is what you cared about.
This is a design choice worth naming. The speedrun defines what nanochat is claiming to be. RL is an experiment shelved next to it, useful, not load-bearing. That framing is honest in a way most repos are not.
The evaluator is the training loop with a different accumulator
run_gsm8k_eval samples device-batch-size completions per problem (default 8) and computes pass@k for k = 1 through 8. pass@k is "did any of the first k completions have the right answer." No ranking, no self-consistency, just the raw hit rate at increasing budget.
The eval loop is the training loop with the gradient replaced by a counter. Same regex. Same completion extraction. Same "generate several, look at the group." Training pushes the model in the direction of a higher pass@k. Evaluation measures whether that push landed. When your reward function and your metric are the same function called with different accumulators, the alignment problem is trivially solved. Nothing in the pipeline can drift, because the target and the measurement are the same code.
What the honest RL step teaches you about the recent GRPO wave
Since DeepSeek R1 dropped in January 2025, "we used GRPO" has become a project-page claim on par with "we used a transformer." HuggingFace TRL has a GRPO implementation, so do vLLM, so do dozens of research repos. The gap between all of those and nanochat is which of the four rows in the table above you actually implemented, plus how much your reward is a regex versus a reward model plus a bunch of heuristics stacked on top.
Karpathy's version is the minimum: a group baseline, a machine-checkable reward, an on-policy update, and that is the RL. The gap to a full GRPO implementation is measured in engineering, not intelligence. The gap to something that looks like R1 is measured in compute, data, and whether your reward is unambiguous.
The takeaway I ended up with: if your reward is not clean, the trust region and the KL term are what stop your model from wandering into whatever exploits the reward model has. If your reward is a regex over #### 42, you can skip most of the paper and still learn arithmetic. If your reward is human preferences, you can't.
Reading nanochat is faster than reading the papers
I read the four papers first (PPO, GRPO, DPO, DAPO). I understood roughly half of what they were doing. Then I read chat_rl.py for an hour and understood the shape of what the algorithm class is actually trying to compute, because the code is 300 lines and the papers are 300 pages. Every optional trick in the papers corresponds to a specific line that could go into nanochat and did not.
This is the most useful thing about small honest implementations: they force the authors to say what the algorithm actually needs versus what a well-written paper thought would be nice to have.
The version of this I wrote for the book covers the harness that runs around this — how the RL step gets composed with the SFT pipeline, what breaks when you swap the reward function, and the choices that let a $100 speedrun ship at all: The Harness Engineering Guide. The RL loop above is one chapter; the pattern generalizes to any pipeline where the reward is cheaper than the model.
Top comments (0)