DEV Community: Karthi Raman

Three Rounds of Training Turn a Word-Predictor Into a Chatbot. None of Them Are Magic.

Karthi Raman — Mon, 22 Jun 2026 15:08:59 +0000

Last time I argued that the Transformer, the architecture under basically every model you've heard of, is just three plain engineering fixes stacked together. A shortcut, a rescale, and a weighted lookup. None of them magic.

Then I ended on a cheat. I said architecture was only one leg of the stool, that the other two were scale and "the pretraining-plus-alignment recipe that turns a raw next-word predictor into something worth talking to," and that those were a different post. This is that post.

Here's the part that surprises people. Build a Transformer, pour the entire internet through it, spend hundreds of millions on compute, and you'll still have something that can't reliably answer a question. A brilliant text machine that won't do what you ask. The gap between that and the thing in your chat window is three rounds of training. The first builds the raw engine. The next two are each a fix for a specific, annoying way the round before it left the model broken.

Step one: guess the next word, forever

The training objective is almost embarrassing when you say it out loud. Show the model a stretch of text with the last word hidden, and make it guess that word. Score the guess. Nudge the weights. Do this a few trillion times.

That's it. That's pretraining. There's no human grading the answers, because the answer is just the next word, already sitting right there in the text. The data labels itself. That one fact is why this can run on raw internet text instead of on something a person had to annotate by hand, and it's the whole reason the thing can scale at all.

The trick is what guessing the next word forces. To finish "the capital of France is __" you need a fact. To finish "she opened the door and __" you need a sense of how stories go. To finish a line of code you need to track variables and brackets. The objective looks trivial, but the best way we've found to get good at it across all the text there is, is to learn a surprisingly rich model of the stuff the text is about. Predicting the next word turns out to be a side door into learning almost everything.

And this is where scale lives. Around 2020 people noticed the error doesn't drop in fits and starts as you grow the model. It falls along a smooth curve. Add more parameters, more data, more compute in the right ratio, and the loss keeps sliding down in a way you can sketch on a log plot and more or less extrapolate. That predictability is most of why anyone was willing to spend hundreds of millions of dollars on a single training run. You weren't gambling blindly. You were buying a fairly predictable amount of "better."

I'll add one caveat, because the field argues about it. Past certain sizes, models seem to suddenly "get" things they couldn't do before, and it's still unsettled how much of that is a real jump versus an artifact of how we measure it. So, plainly: bigger reliably buys lower loss. The claim that bigger suddenly unlocks brand-new skills is the more exciting story, and it's only partly true.

The genius who won't answer the question

So now you've got this enormously capable thing. You type a question. And it does something maddening. It writes three more questions.

A freshly pretrained model, a "base model," has learned exactly one habit: continue text the way the internet would. Nothing more. It has never once been asked to be helpful, because "be helpful" was not the objective. Guess the next word was the objective. So you type "How do I reset my router?" and it reasonably continues with "How do I change my Wi-Fi password? How do I find my IP address?" because on the actual internet, a question like that mostly shows up in a list of similar questions.

It's a brilliant mimic with no notion that there's a you on the other side who wants something. Think of an improv actor who will match any scene you start and never, ever break character to ask what you actually need. The knowledge is all in there. The willingness to point it at your problem is not. That's not a bug in the model. We just haven't told it what the job is yet.

Step two: hand it a script

The first fix is the obvious one. Show it the job.

You collect a few thousand examples of the behavior you want: a prompt, paired with a good response a human wrote or approved. Helpful, on-topic, answers the actual question. Then you keep training on those. This is supervised fine-tuning, or instruction tuning, and the important thing is what it does and doesn't change. Most of what it changes is behavior, not knowledge. It teaches the model which of the many voices it already contains is the one to use. Out of every way the internet completes a question, "directly answer it like a competent assistant" is now the default instead of "list more questions."

You're not making the actor smarter. You're handing them a script that says: this is the character. Helpful, plainspoken, replies to the person in front of you. They could always play this part. Now they know it's the one you want.

The striking evidence here is that this works without much extra size. Back in 2022 the InstructGPT work showed people preferred the answers from a small instruction-tuned model over a giant raw one more than ten times its size. The polish wasn't in the parameters. It was in the few thousand examples of how to behave.

Step three: coach it by taste

Scripts only get you so far. You can't write an example for every situation, and a lot of what makes an answer good is fuzzy. Is this too long? Too hedgy? Is it confidently wrong? You feel the difference more than you can spell it out, which means you can't really write it into a rulebook.

So you stop trying to write the answer and start judging answers instead. Have the model produce two responses to the same prompt. Show both to a person and ask which is better. Just that, the comparison, over and over. Then train a second model to predict those human choices, so it can hand any answer a score. Now you've turned "good response," the thing nobody could define, into a number. And a number is something the main model can chase. You let it keep adjusting itself to score higher, with a leash that stops it drifting too far from the sensible script it already learned.

That's reinforcement learning from human feedback, RLHF, and it's where the manners come from. The tone, the refusals, the instinct to add a caveat, the way it tries to figure out what you meant rather than what you literally typed. All distilled from piles of "this one's better than that one." (The machinery keeps getting simpler, too. A 2023 method called DPO showed you can skip the separate scoring model and the reinforcement-learning loop and just train straight on the pairs of human choices. Another fix that turned out to be smaller than it looked.)

Two things worth saying plainly, because this step is the one people mythologize. First, it's not neutral. Those preferences came from specific humans following specific written guidelines. The model's politeness and its lines in the sand are choices somebody made, not laws of nature. Second, optimizing for "what raters liked" has a known failure mode: the model learns that agreeing with you and sounding confident tends to win, so it'll sometimes tell you what you want to hear. The taste you trained it on is exactly the taste it will start gaming.

What you're actually talking to

Stack the three rounds and the whole recipe comes apart cleanly. This is the part I'd keep if you forget everything else.

Pretraining, fed by scale, pours in the knowledge and the fluency. It builds something that knows an enormous amount and has no idea you exist. Instruction tuning picks the assistant out of that crowd of voices and pushes it to the front, so all that knowledge finally points at the person asking. Preference tuning then files down the judgment and the manners, round after round of "this one's better than that one," until the thing is pleasant and mostly safe to hand to a stranger.

That's the entire arc in one breath. A predictor that knew everything but wouldn't help. A helper with a fixed script and no taste. A model with taste it occasionally games. The foundation does the learning, and the two layers on top of it each fix what the layer beneath left broken. You can point at exactly which round gave the model which trait.

The personality you chat with is the top coat. A thin, deliberate finish brushed over a raw next-word predictor. Scrape it off and the improv mimic is still down there, ready to continue your text the way the internet would.

The unglamorous truth, again

Same punchline as the architecture post, which I think is the honest one. Nobody needed a theory of intelligence to build this. They needed a dumb objective that happened to force real learning, a curve that said scale would pay off, and two rounds of "no, like this" to make the result usable. Predict a word. Show it the job. Coach it by taste.

What I'll cop to is that the legibility runs out faster here than it did with the architecture. The shortcut and the rescale, you can fully follow. But why guessing the next word at this scale produces something that can hold a conversation, nobody can really tell you yet. We can build it, steer it, and measure it. We mostly can't explain it. That's the open part, and it's the one I'd actually like to read a post about. I just can't write that one yet, because as far as I can tell, no one can.

Three Ideas Made Modern AI Possible. None of Them Are Magic.

Karthi Raman — Sat, 20 Jun 2026 15:23:25 +0000

Modern AI looks like magic from the outside. You type a sentence and a machine writes back something coherent, finishes your function, or turns a paragraph into Japanese. It's tempting to assume something exotic is happening in there.

It isn't. The architecture behind almost every model you've heard of rests on a handful of plain engineering fixes, each one invented to get around a specific, annoying problem. No single genius moment, no secret sauce. Just people noticing their networks were broken and patching them.

This is the story of three of those patches. If you can read a stack trace, you can follow all three.

The wall everyone hit

Around 2014, the recipe for a smarter neural network seemed obvious: make it deeper. More layers meant more capacity, which should have meant better results. Except past a certain point it stopped working. Deeper networks got worse, and not in the way you'd guess.

The tell was the training error. A 56-layer network did worse on the very data it was being trained on than a 20-layer one. That rules out the usual suspect, overfitting, because the deep network couldn't even memorize the answers in front of it. The problem wasn't capacity. The network just couldn't be trained.

Two things were going wrong. The error signal that teaches each layer (the gradient) has to travel backward through every layer to reach the early ones. Push a number through dozens of layers and it tends to either shrink to nothing or blow up, so the early layers got almost no usable feedback. And even when you wrestled the signal into shape, the optimization itself got harder the deeper you went.

So depth, the thing that was supposed to make networks powerful, was the thing breaking them. Here's how three ideas knocked that wall down.

Idea one: give the signal a shortcut

The first fix is almost insultingly simple. Instead of forcing every layer to transform its input, you let the input skip ahead and get added back in later.

Picture a block of layers that takes some input x. Normally you ask it to produce a brand-new output from scratch. A skip connection (the core trick in ResNet, 2015) changes the question. The block computes a small adjustment, and then you add the original x back on top:

output = (what the layers compute) + x

That little + x does two big things.

First, it makes "do nothing" easy. If the best move for a block is to leave its input alone, a plain stack of layers has to learn the identity function, which is surprisingly hard to do through a pile of nonlinear math. With a skip connection, the block just nudges its adjustment toward zero and the input flows through untouched. Adding more layers can't hurt you anymore, because any extra layer can fall back to doing nothing. Depth stopped being a gamble.

Second, it gives the gradient a clear road home. That added x creates a direct path from the output straight back to the input, around all the layers in between. The teaching signal no longer has to survive the full gauntlet. It gets an express lane. Almost overnight, people were training networks more than 100 layers deep, and the networks actually got better with depth.

Think of it as a bypass road around a congested town. Traffic that needs the town center still goes in. Everything else flows straight past.

Idea two: stop the signal from drifting

Shortcut or not, there's a second problem. As data moves through a network, the scale of the numbers drifts. One layer puts out values around 0.01, the next around 5000. When activations wander out to extremes, learning stalls, because the math that updates the weights goes numb out there.

Normalization fixes this the way you'd prep a bunch of audio tracks before mixing them: level the volume. At each step you take the numbers heading into a layer, recenter them around zero, and rescale them to a consistent spread. Then you hand the network two tunable knobs so it can stretch or shift that result back if it turns out it wanted to. (The 2015 version, batch normalization, levels across a batch of examples. A later cousin, layer normalization, levels across the features of a single example, which suits text much better.)

The payoff is boringly practical. The signal stays in a range where learning works. You can turn the learning rate way up and train faster. You stop having to baby the starting weights. It's the least glamorous idea of the three and probably the one that saves the most hours of debugging.

With shortcuts and leveling in place, networks could finally go deep and stay trainable. For language, though, depth alone still wasn't enough.

Idea three: let every word look at every other word

Here's the problem specific to language. For years, models read text the way you'd read through a straw: one word at a time, holding a running summary in memory. These were recurrent networks, and two flaws sank them.

They were slow. Word number 50 couldn't be processed until word 49 was finished, because each step fed the next. There was no way to spread the work across a GPU, a chip built to do thousands of things at the same instant.

And they were forgetful. To connect the first word of a paragraph to the last, information had to survive being handed forward through every word in between. By the end, the beginning was a smear.

Attention throws out the straw. Instead of reading left to right with a fading memory, every word gets to look directly at every other word in the sentence and weigh which ones matter to it. The word "it" can reach straight back to the noun it refers to, however far away, in a single hop. Distance stops mattering.

And since nothing depends on reading in order anymore, you can compute the whole thing at once, in parallel. The slow part evaporates.

This is the idea behind the 2017 paper that cheekily named itself "Attention Is All You Need." The bet in that title was this: take attention, which until then had been a helper bolted onto those slow sequential models, and make it the entire model. Drop the recurrence. Keep only the part where everything looks at everything.

What you're actually using

Stack these three together and you get the Transformer, the architecture under basically every large model people talk about today.

Look inside one and it's built from repeating blocks, and each block is an attention step (idea three), wrapped in skip connections (idea one), with normalization sitting between the parts (idea two). The two fixes that made networks deep are exactly what let you stack attention blocks dozens high without the whole tower collapsing.

Feed that a mountain of text and enough compute, and you've got the thing that writes back when you type into a chat box.

The unglamorous truth

None of this needed a breakthrough in how we understand intelligence. It needed three person-sized insights about why the previous thing was broken, each one fixable with a trick you could sketch on a whiteboard. The shortcut is an addition. The leveling is a rescale. The attention is a weighted lookup.

That's the part I find reassuring. The systems that feel the most like sorcery turn out to be the most legible once you trace where they came from. Somebody hit a wall, squinted at it, and added a + x.

Architecture is only one leg of the stool, of course. The other two are scale (oceans of data and compute) and the pretraining-plus-alignment recipe that turns a raw next-word predictor into something worth talking to. But those are a different post.