<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Walid Hadri</title>
    <description>The latest articles on DEV Community by Walid Hadri (@walid_hadri).</description>
    <link>https://dev.to/walid_hadri</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4242816%2Fa89fc242-4a39-4336-bc30-3dd490240ce4.jpg</url>
      <title>DEV Community: Walid Hadri</title>
      <link>https://dev.to/walid_hadri</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/walid_hadri"/>
    <language>en</language>
    <item>
      <title>You Will Understand 10 Years of LLMs</title>
      <dc:creator>Walid Hadri</dc:creator>
      <pubDate>Sun, 15 Nov 2026 15:41:53 +0000</pubDate>
      <link>https://dev.to/walid_hadri/you-will-understand-10-years-of-llms-f3c</link>
      <guid>https://dev.to/walid_hadri/you-will-understand-10-years-of-llms-f3c</guid>
      <description>&lt;p&gt;My new video, &lt;a href="https://youtu.be/cLWAiVIVr4U" rel="noopener noreferrer"&gt;&lt;strong&gt;In 19 Minutes, You'll Understand 10 Years of LLMs&lt;/strong&gt;&lt;/a&gt;, follows one question through every era of language models: &lt;em&gt;"What is the capital of France?"&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;This post is the developer version. It goes through every chapter of the video with the actual terms and numbers. Where the video shows a model's output, we ran an open model ourselves, so you get the code to reproduce it.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip install torch transformers accelerate
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;All outputs below are what we got on an RTX 4070 Ti. Yours may differ slightly with hardware and precision.&lt;/p&gt;

&lt;h2&gt;
  
  
  TL;DR timeline
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Year&lt;/th&gt;
&lt;th&gt;Milestone&lt;/th&gt;
&lt;th&gt;What it fixed&lt;/th&gt;
&lt;th&gt;What it left open&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1948–51&lt;/td&gt;
&lt;td&gt;Shannon's next-letter game&lt;/td&gt;
&lt;td&gt;"predict what comes next" as a measurable idea&lt;/td&gt;
&lt;td&gt;no machine could play it well&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2013&lt;/td&gt;
&lt;td&gt;word2vec&lt;/td&gt;
&lt;td&gt;words become vectors that capture meaning&lt;/td&gt;
&lt;td&gt;one vector per word, whatever the context&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1997–2014&lt;/td&gt;
&lt;td&gt;RNN, LSTM, seq2seq&lt;/td&gt;
&lt;td&gt;read sentences, translate&lt;/td&gt;
&lt;td&gt;one fixed-size summary per sentence&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2014&lt;/td&gt;
&lt;td&gt;attention (Bahdanau)&lt;/td&gt;
&lt;td&gt;the decoder looks back at every word&lt;/td&gt;
&lt;td&gt;still reads one word at a time&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2017&lt;/td&gt;
&lt;td&gt;the Transformer&lt;/td&gt;
&lt;td&gt;no recurrence: parallel training on GPUs&lt;/td&gt;
&lt;td&gt;writing is still one token at a time&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2018–19&lt;/td&gt;
&lt;td&gt;BERT, GPT-1, GPT-2&lt;/td&gt;
&lt;td&gt;pretraining on raw text&lt;/td&gt;
&lt;td&gt;the model continues text, it doesn't answer&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2020–22&lt;/td&gt;
&lt;td&gt;scaling laws, GPT-3, Chinchilla&lt;/td&gt;
&lt;td&gt;predictable gains from size and data&lt;/td&gt;
&lt;td&gt;still not helpful&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2022&lt;/td&gt;
&lt;td&gt;InstructGPT / RLHF, ChatGPT&lt;/td&gt;
&lt;td&gt;a second training stage teaches it to answer&lt;/td&gt;
&lt;td&gt;serving at scale is expensive&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2023–24&lt;/td&gt;
&lt;td&gt;LLaMA, KV-cache sharing, mixture of experts&lt;/td&gt;
&lt;td&gt;cheaper, faster, longer contexts&lt;/td&gt;
&lt;td&gt;running out of human text&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2024–25&lt;/td&gt;
&lt;td&gt;o1, DeepSeek-R1&lt;/td&gt;
&lt;td&gt;models learn to think before answering&lt;/td&gt;
&lt;td&gt;thinking helps, doesn't guarantee&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026&lt;/td&gt;
&lt;td&gt;agents, hybrid attention&lt;/td&gt;
&lt;td&gt;plan, use tools, act&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  0. The quiz page (Experiment 1)
&lt;/h2&gt;

&lt;p&gt;Before chatbots, a language model's only job was to continue text. Ask GPT-2 XL (1.5B parameters, 2019):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;transformers&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;AutoTokenizer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;AutoModelForCausalLM&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;torch&lt;/span&gt;

&lt;span class="n"&gt;name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"openai-community/gpt2-xl"&lt;/span&gt;
&lt;span class="n"&gt;tok&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;AutoTokenizer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;AutoModelForCausalLM&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;eval&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;ids&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;tok&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"What is the capital of France?"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;return_tensors&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"pt"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;input_ids&lt;/span&gt;
&lt;span class="n"&gt;out&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;generate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ids&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_new_tokens&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;do_sample&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;pad_token_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;tok&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;eos_token_id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tok&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;ids&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;shape&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;The capital of France is Paris.

What is the capital of the United States?

The capital of the United States is Washington, D.C.

What is the capital of the United Kingdom?

The capital of the United Kingdom is London.

What
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It's not answering; it's writing a quiz page. We'll come back to why.&lt;/p&gt;




&lt;h2&gt;
  
  
  1. Before 2017: the road to attention
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Shannon's game (1948, 1951)
&lt;/h3&gt;

&lt;p&gt;Claude Shannon asked how predictable English is. Letters don't come at random: counted in &lt;em&gt;Pride and Prejudice&lt;/em&gt;, after a T comes an H 29% of the time; after a Q, a U 99.85% of the time.&lt;/p&gt;

&lt;p&gt;In 1948 he generated text that follows these odds with a book: pick a letter, open another page, find that letter, copy the one after it, repeat. Letter pairs give almost-words; word pairs give &lt;em&gt;"The head and in frontal attack on an English writer that the character of this point…"&lt;/em&gt; Not English, but closer. The more context, the closer.&lt;/p&gt;

&lt;p&gt;In 1951 he made it a game with people: guess the next letter. On one passage they were right on the first try &lt;strong&gt;79 times out of 102&lt;/strong&gt;. He estimated that a letter of English carries about &lt;strong&gt;1 bit&lt;/strong&gt; of new information, against about &lt;strong&gt;4.75 bits&lt;/strong&gt; if letters were random.&lt;/p&gt;

&lt;p&gt;That's the objective LLMs still train on: predict the next token.&lt;/p&gt;

&lt;h3&gt;
  
  
  Words become vectors (2013)
&lt;/h3&gt;

&lt;p&gt;A word ID (&lt;code&gt;king = #4812&lt;/code&gt;) says nothing about meaning. &lt;strong&gt;word2vec&lt;/strong&gt; (Mikolov et al., 2013) gives each word a vector of a few hundred numbers (300 in the classic setup), learned by predicting a word's neighbours over billions of words. Words used in similar contexts end up close, and the space has structure: &lt;code&gt;king − man + woman ≈ queen&lt;/code&gt; (the nearest word, excluding the three inputs).&lt;/p&gt;

&lt;p&gt;The limit: one vector per word. The "bank" of a river and the bank with your money get the same vector.&lt;/p&gt;

&lt;h3&gt;
  
  
  RNN, LSTM, seq2seq (1997–2014)
&lt;/h3&gt;

&lt;p&gt;A &lt;strong&gt;recurrent neural network&lt;/strong&gt; reads one token at a time and updates a hidden state, a running memory. Two problems: the state is overwritten at every step, so early words fade; and in training, the error signal has to flow backwards through every step and shrinks on the way (vanishing gradients).&lt;/p&gt;

&lt;p&gt;The &lt;strong&gt;LSTM&lt;/strong&gt; (Hochreiter &amp;amp; Schmidhuber, 1997) adds a cell state, a memory lane that runs straight through every step, with gates that decide what to write, what to read and (since 2000) what to forget.&lt;/p&gt;

&lt;p&gt;By 2014, &lt;strong&gt;sequence to sequence&lt;/strong&gt; translated with two of them: an encoder packs the English sentence into one fixed vector (8,000 numbers in Google's 4-layer version), and a decoder unpacks it into French, one word at a time. On basic encoder-decoders, quality dropped as sentences got longer: one fixed vector is a bottleneck.&lt;/p&gt;

&lt;h3&gt;
  
  
  Attention (2014)
&lt;/h3&gt;

&lt;p&gt;Bahdanau, Cho and Bengio kept a vector for &lt;strong&gt;every&lt;/strong&gt; source word, and at each output step let the decoder compute a weighted mix of all of them. To write "zone", it weighs "Area" most. Long sentences stopped falling apart.&lt;/p&gt;

&lt;p&gt;One problem remained: everything was still sequential. Step 50 needs the state from step 49, while GPUs are built to do thousands of operations at once.&lt;/p&gt;




&lt;h2&gt;
  
  
  2. 2017: the Transformer
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;"Attention Is All You Need"&lt;/strong&gt; (Vaswani et al., Google, June 2017; author order random, the paper says so) kept attention and removed recurrence. Every token attends to every other token, all at once: &lt;strong&gt;self-attention&lt;/strong&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  What attention does (Experiment 2)
&lt;/h3&gt;

&lt;p&gt;Google's example: &lt;em&gt;"The animal didn't cross the street because it was too tired"&lt;/em&gt; (it = the animal) vs &lt;em&gt;"…too wide"&lt;/em&gt; (it = the street). Each token produces a &lt;strong&gt;query&lt;/strong&gt;; every token has a &lt;strong&gt;key&lt;/strong&gt; and a &lt;strong&gt;value&lt;/strong&gt;. Query-key matches (dot products, scaled, then softmax) give each token a share, and "it" takes in a weighted mix of the values.&lt;/p&gt;

&lt;p&gt;You can watch this in BERT (12 layers × 12 heads = 144 heads). We searched for the head whose focus flips most between the two sentences: layer 7, head 11 (index 6 and 10 in code):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;transformers&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;AutoModel&lt;/span&gt;

&lt;span class="n"&gt;btok&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;AutoTokenizer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"bert-base-uncased"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;bert&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;AutoModel&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"bert-base-uncased"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;output_attentions&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;eval&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;end&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"tired"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"wide"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"The animal didn't cross the street because it was too {end}."&lt;/span&gt;
    &lt;span class="n"&gt;enc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;btok&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;return_tensors&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"pt"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;toks&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;btok&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;convert_ids_to_tokens&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;enc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;input_ids&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;torch&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;no_grad&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;att&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;bert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;enc&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;attentions&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;        &lt;span class="c1"&gt;# layer 7, head 11&lt;/span&gt;
    &lt;span class="n"&gt;it&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;toks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;index&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"it"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;end&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;att&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;it&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;toks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;index&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;item&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"animal"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"street"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;tired {'animal': 0.284, 'street': 0.067}
wide {'animal': 0.012, 'street': 0.59}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;We picked this head because it flips the most; other heads learn other links (the previous word, the next word, sentence boundaries). Running many heads side by side is &lt;strong&gt;multi-head attention&lt;/strong&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  The whole machine (Experiment 3)
&lt;/h3&gt;

&lt;p&gt;Text is cut into &lt;strong&gt;tokens&lt;/strong&gt;, pieces from a fixed vocabulary (byte-pair encoding; 50,257 pieces for GPT-2):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;gtok&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;AutoTokenizer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"openai-community/gpt2"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"The capital of France is Paris."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"unbelievably"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"strawberry"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;" strawberry"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;repr&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;gtok&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;gtok&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;input_ids&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;'The capital of France is Paris.' ['The', ' capital', ' of', ' France', ' is', ' Paris', '.']
'unbelievably' ['un', 'bel', 'iev', 'ably']
'strawberry' ['st', 'raw', 'berry']
' strawberry' [' strawberry']
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;each token becomes an &lt;strong&gt;embedding&lt;/strong&gt; vector (word2vec's idea, learned with the model) plus a &lt;strong&gt;positional encoding&lt;/strong&gt;, since attention alone has no notion of order;&lt;/li&gt;
&lt;li&gt;each &lt;strong&gt;layer&lt;/strong&gt; has an attention block (tokens share information) and a &lt;strong&gt;feed-forward&lt;/strong&gt; block (each token processed on its own);&lt;/li&gt;
&lt;li&gt;the 2017 model stacks 6 encoder layers (read the English) and 6 decoder layers (write the French);&lt;/li&gt;
&lt;li&gt;the output is a probability for every possible next token.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Why it won: parallel training
&lt;/h3&gt;

&lt;p&gt;In training, the whole target sentence is known, so the model learns from every position at once and the GPUs are fully used. The big model trained in &lt;strong&gt;3.5 days on one machine with 8 P100 GPUs&lt;/strong&gt; and set records (BLEU 28.4 English–German, 41.8 English–French) at a fraction of the previous training cost.&lt;/p&gt;

&lt;p&gt;Writing stayed sequential: the next token doesn't exist until it's chosen.&lt;/p&gt;




&lt;h2&gt;
  
  
  3. 2018–2019: reading vs writing
&lt;/h2&gt;

&lt;p&gt;The blueprint split in two.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;BERT&lt;/strong&gt; (Google, 2018): encoder only. Training game: mask a word and predict it from both sides. Great at understanding, can't generate.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;GPT&lt;/strong&gt; (OpenAI): decoder only, with causal masking. Training game: Shannon's, predict the next token from the left.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Next-token prediction needs no labels: any text is a lesson. That's &lt;strong&gt;pretraining&lt;/strong&gt;, and it turns the internet into training data.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;GPT-1&lt;/strong&gt; (2018): about 117M parameters, 12 layers, 7,000 unpublished books.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;GPT-2&lt;/strong&gt; (2019): same objective, 1.5B parameters, 48 layers, 8 million web pages. OpenAI initially held back the full model citing "concerns about malicious applications".&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Experiment 1 now makes sense: GPT-2 continues documents, and on the web a question is often followed by its answer, then another question. It could also do untrained tasks by prompt alone: add &lt;code&gt;TL;DR:&lt;/code&gt; after an article and it writes a rough summary.&lt;/p&gt;




&lt;h2&gt;
  
  
  4. 2020–2022: scale
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Scaling laws&lt;/strong&gt; (Kaplan et al., OpenAI, January 2020): train many models from tiny to large, and on log-log axes, loss falls along straight lines as parameters, data and compute grow. Not a law of nature, but a pattern you can plan around.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;GPT-3&lt;/strong&gt; (2020): nearly the same architecture as GPT-2, but 175B parameters, 96 layers, trained on 300B tokens. It showed &lt;strong&gt;in-context learning&lt;/strong&gt;: give a few examples in the prompt and it continues the pattern.&lt;/p&gt;

&lt;h3&gt;
  
  
  Experiment 4: in-context learning, already in GPT-2
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;p_next&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;word&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;" Paris"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;ids&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;tok&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;return_tensors&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"pt"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;input_ids&lt;/span&gt;
    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;torch&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;no_grad&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;probs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;torch&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;softmax&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;model&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ids&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;logits&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kt"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;probs&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;tok&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;word&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;item&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;zero&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"Q: What is the capital of France?\nA:"&lt;/span&gt;
&lt;span class="n"&gt;few&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Q: What is the capital of Germany?\nA: Berlin\n"&lt;/span&gt;
       &lt;span class="s2"&gt;"Q: What is the capital of Italy?\nA: Rome\n"&lt;/span&gt;
       &lt;span class="s2"&gt;"Q: What is the capital of France?\nA:"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"zero-shot: {p_next(zero):.2f}   few-shot: {p_next(few):.2f}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# GPT-2 XL from experiment 1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;zero-shot: 0.57   few-shot: 0.99
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Chinchilla&lt;/strong&gt; (DeepMind, 2022) showed the balance was off: models were too big for their data. Chinchilla (70B parameters, 1.4T tokens) beat Gopher (280B, 300B tokens) almost everywhere. Data started to matter as much as size.&lt;/p&gt;




&lt;h2&gt;
  
  
  5. 2022: teaching it to help
&lt;/h2&gt;

&lt;p&gt;Scale didn't fix behaviour. OpenAI's own example: prompt GPT-3 with &lt;em&gt;"Explain the moon landing to a 6 year old in a few sentences"&lt;/em&gt; and it may continue with &lt;em&gt;"Explain the theory of gravity to a 6 year old…"&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;InstructGPT&lt;/strong&gt; (January 2022) added a second training stage:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Supervised fine-tuning&lt;/strong&gt;: people write good answers; the model imitates them.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Reward model&lt;/strong&gt;: the model writes several answers, people rank them, and a second model learns to predict the ranking.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Reinforcement learning&lt;/strong&gt;: the language model is optimized against the reward model.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;That's &lt;strong&gt;RLHF&lt;/strong&gt;. People preferred a 1.3B InstructGPT model to the 175B GPT-3.&lt;/p&gt;

&lt;h3&gt;
  
  
  Experiment 5: the two stages in one small model
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"Qwen/Qwen2.5-0.5B-Instruct"&lt;/span&gt;
&lt;span class="n"&gt;ctok&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;AutoTokenizer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;chat_model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;AutoModelForCausalLM&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;eval&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;q&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"What is the capital of France?"&lt;/span&gt;

&lt;span class="n"&gt;chat&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ctok&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;apply_chat_template&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"user"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;add_generation_prompt&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;return_tensors&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"pt"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;out&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;chat_model&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;generate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;chat&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_new_tokens&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;do_sample&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"CHAT:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ctok&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;chat&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;shape&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;skip_special_tokens&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;raw&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;ctok&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;return_tensors&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"pt"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;input_ids&lt;/span&gt;
&lt;span class="n"&gt;out&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;chat_model&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;generate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_new_tokens&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;do_sample&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"RAW:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ctok&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;shape&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;CHAT: The capital of France is Paris.
RAW:  The capital of France is Paris.

To verify this, I will use a simple Python code snippet to confirm ...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With the chat template, it answers and stops. Without it, it rambles. Knowledge comes from pretraining; manners come from the second stage.&lt;/p&gt;

&lt;p&gt;On November 30, 2022, OpenAI released &lt;strong&gt;ChatGPT&lt;/strong&gt;, trained this way. A million users in five days, and serving these models at scale became the next problem.&lt;/p&gt;




&lt;h2&gt;
  
  
  6. 2023–2024: bigger, cheaper, longer
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;GPT-4&lt;/strong&gt; (March 2023): its report "contains no further details about the architecture (including model size)". The frontier went closed; open models like Meta's &lt;strong&gt;LLaMA&lt;/strong&gt; (2023) show what changed inside the 2017 block:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;RoPE&lt;/strong&gt;: positions as rotations applied inside attention, instead of added vectors;&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;RMSNorm&lt;/strong&gt; and &lt;strong&gt;SwiGLU&lt;/strong&gt;: leaner normalization and feed-forward blocks;&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;FlashAttention&lt;/strong&gt;: the same attention, computed to fit the GPU's fastest memory.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Two harder problems:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Memory: the KV cache.&lt;/strong&gt; During generation, each new token attends to all previous ones, so the model caches every past token's keys and values in every layer. The cache grows with context length and, for long conversations, can outgrow the model's own weights. Fixes: &lt;strong&gt;grouped-query attention&lt;/strong&gt; (groups of heads share one set of keys/values) and DeepSeek's &lt;strong&gt;multi-head latent attention&lt;/strong&gt; (store a compressed summary). Smaller caches are part of how context windows went from a few thousand tokens to about a million.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Compute: mixture of experts.&lt;/strong&gt; Every token normally goes through every parameter. A &lt;strong&gt;MoE&lt;/strong&gt; layer splits the feed-forward block into many experts and a router sends each token to just a few. &lt;strong&gt;DeepSeek-V3&lt;/strong&gt; (December 2024): 671B parameters stored, 37B used per token.&lt;/p&gt;




&lt;h2&gt;
  
  
  7. 2024–2025: thinking
&lt;/h2&gt;

&lt;p&gt;In December 2024, Ilya Sutskever said at NeurIPS: &lt;em&gt;"Pre-training as we know it will unquestionably end"&lt;/em&gt; and &lt;em&gt;"We have but one internet."&lt;/em&gt; Epoch AI estimates models will have used up the public human text somewhere between 2026 and 2032.&lt;/p&gt;

&lt;p&gt;Gains moved to inference time. Kojima et al. (2022) found that adding &lt;em&gt;"Let's think step by step"&lt;/em&gt; took one maths benchmark (MultiArith) from 18% to 79%. A model spends roughly the same compute per token, so writing out steps buys more forward passes, and each step is easier than the whole jump.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;o1&lt;/strong&gt; (OpenAI, September 2024): trained with reinforcement learning to reason before answering; accuracy rose with thinking time. On the AIME 2024 maths competition: GPT-4o 12%, o1 74%.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;DeepSeek-R1&lt;/strong&gt; (January 2025), recipe published: reinforcement learning with &lt;strong&gt;verifiable rewards&lt;/strong&gt; (maths with known answers, code with tests), rewarding only correct outcomes. R1-Zero's responses grew from about 500 to 9,500 tokens over training, and it started checking its own work: &lt;em&gt;"Wait, wait. Wait. That's an aha moment I can flag here."&lt;/em&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Experiment 6: a small model that thinks
&lt;/h3&gt;

&lt;p&gt;DeepSeek also released distilled students of R1. The 1.5B one, with the model card's sampling settings:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B"&lt;/span&gt;
&lt;span class="n"&gt;rtok&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;AutoTokenizer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;rmodel&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;AutoModelForCausalLM&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;torch_dtype&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;torch&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;bfloat16&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;eval&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;q&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"How many r letters are in the word strawberry?"&lt;/span&gt;
&lt;span class="n"&gt;chat&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rtok&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;apply_chat_template&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"user"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;add_generation_prompt&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;return_tensors&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"pt"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;out&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rmodel&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;generate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;chat&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_new_tokens&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;900&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;do_sample&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;temperature&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.6&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;top_p&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.95&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rtok&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;chat&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;shape&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On the capital question, it thought for &lt;strong&gt;255 to 659 tokens&lt;/strong&gt; and answered "Paris" four times out of five. On strawberry, across five runs: &lt;strong&gt;3, 2, 3, 3, and once 8&lt;/strong&gt;. Remember experiment 3: " strawberry" is one token, so the model never sees the letters. Thinking helps; it doesn't guarantee. Many models now decide for themselves whether to think, and for how long.&lt;/p&gt;




&lt;h2&gt;
  
  
  8. Now: senses and agents
&lt;/h2&gt;

&lt;p&gt;Images and audio can be tokenized too, so the same architecture handles them: &lt;strong&gt;GPT-4o&lt;/strong&gt; (2024) took text, images and audio in one network.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Agents&lt;/strong&gt; run the model in a loop: plan, call a tool (search, code execution, file edits), read the result, repeat. METR measures the length of tasks (in human working time) that agents complete with 50% success, mostly on software tasks: GPT-2 a few seconds, GPT-4 (2023) about four minutes, early 2025 about an hour, and by April 2026 around 17 hours, beyond what their test suite measures reliably.&lt;/p&gt;

&lt;p&gt;The architecture keeps moving too. Full attention costs more as context grows, so many new open models keep it in only some layers and use cheaper, recurrent-style layers in between, the RNN idea made trainable in parallel. And in September 2026, DeepSeek described a new model as a "causal encoder-decoder". Nine years later, the encoder is back.&lt;/p&gt;




&lt;h2&gt;
  
  
  9. 2017 vs today
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;2017 Transformer&lt;/th&gt;
&lt;th&gt;Today&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Structure&lt;/td&gt;
&lt;td&gt;encoder + decoder&lt;/td&gt;
&lt;td&gt;mostly decoder-only&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Positions&lt;/td&gt;
&lt;td&gt;added sinusoidal stamps&lt;/td&gt;
&lt;td&gt;rotations inside attention (RoPE)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Attention memory&lt;/td&gt;
&lt;td&gt;every head its own keys/values&lt;/td&gt;
&lt;td&gt;shared (GQA) or compressed (MLA)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Feed-forward&lt;/td&gt;
&lt;td&gt;one block per layer&lt;/td&gt;
&lt;td&gt;many experts with a router (MoE)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Attention coverage&lt;/td&gt;
&lt;td&gt;every layer&lt;/td&gt;
&lt;td&gt;often only some layers&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Training&lt;/td&gt;
&lt;td&gt;supervised translation&lt;/td&gt;
&lt;td&gt;pretraining on trillions of tokens, then human preferences, then reasoning and tool use&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Almost every part has been replaced. Two things remain: &lt;strong&gt;attention&lt;/strong&gt;, in some layers at least, and &lt;strong&gt;the objective&lt;/strong&gt;: predict the next token, Shannon's game from 1948.&lt;/p&gt;




&lt;p&gt;▶ &lt;strong&gt;The full story, animated (19 min):&lt;/strong&gt; &lt;a href="https://youtu.be/cLWAiVIVr4U" rel="noopener noreferrer"&gt;In 19 Minutes, You will understand 10 years of LLMs&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;▶ &lt;strong&gt;What happens inside one model when it answers:&lt;/strong&gt; &lt;a href="https://youtu.be/GWjA5eHIXXk" rel="noopener noreferrer"&gt;What Actually Happens When You Press Enter on ChatGPT&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Sources: Shannon (1948, 1951); Mikolov et al. (2013); Hochreiter &amp;amp; Schmidhuber (1997); Sutskever, Vinyals &amp;amp; Le (2014); Bahdanau, Cho &amp;amp; Bengio (2014); Vaswani et al. (2017); Devlin et al. (2018); Radford et al. (2018, 2019); Kaplan et al. (2020); Brown et al. (2020); Hoffmann et al. (2022); Ouyang et al. (2022); Kojima et al. (2022); OpenAI GPT-4 Technical Report (2023) and o1 (2024); Touvron et al. (2023); DeepSeek-AI, DeepSeek-V3 (2024) and DeepSeek-R1 (2025); METR; Epoch AI.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>nlp</category>
      <category>machinelearning</category>
    </item>
    <item>
      <title>From my desk in Paris, one ChatGPT question went to the bottom of the Atlantic and back in about a second. I followed it all the way: the cables, the GPUs, the tokens, and how the model picks each word. With a 14-minute animated video.</title>
      <dc:creator>Walid Hadri</dc:creator>
      <pubDate>Fri, 13 Nov 2026 12:15:30 +0000</pubDate>
      <link>https://dev.to/walid_hadri/from-my-desk-in-france-one-chatgpt-question-went-to-the-bottom-of-the-atlantic-and-back-in-about-a-12df</link>
      <guid>https://dev.to/walid_hadri/from-my-desk-in-france-one-chatgpt-question-went-to-the-bottom-of-the-atlantic-and-back-in-about-a-12df</guid>
      <description>&lt;div class="ltag__link--embedded"&gt;
  &lt;div class="crayons-story "&gt;
  &lt;a href="https://dev.to/walid_hadri/what-actually-happens-when-you-press-enter-on-chatgpt-n34" class="crayons-story__hidden-navigation-link"&gt;What Actually Happens When You Press Enter on ChatGPT&lt;/a&gt;


  &lt;div class="crayons-story__body crayons-story__body-full_post"&gt;
    &lt;div class="crayons-story__top"&gt;
      &lt;div class="crayons-story__meta"&gt;
        &lt;div class="crayons-story__author-pic"&gt;

          &lt;a href="/walid_hadri" class="crayons-avatar  crayons-avatar--l  "&gt;
            &lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4242816%2Fa89fc242-4a39-4336-bc30-3dd490240ce4.jpg" alt="walid_hadri profile" class="crayons-avatar__image" width="449" height="449"&gt;
          &lt;/a&gt;
        &lt;/div&gt;
        &lt;div&gt;
          &lt;div&gt;
            &lt;a href="/walid_hadri" class="crayons-story__secondary fw-medium m:hidden"&gt;
              Walid Hadri
            &lt;/a&gt;
            &lt;div class="profile-preview-card relative mb-4 s:mb-0 fw-medium hidden m:inline-block"&gt;
              
                Walid Hadri
                
                
              
              &lt;div id="story-author-preview-content-5029583" class="profile-preview-card__content crayons-dropdown branded-7 p-4 pt-0"&gt;
                &lt;div class="gap-4 grid"&gt;
                  &lt;div class="-mt-4"&gt;
                    &lt;a href="/walid_hadri" class="flex"&gt;
                      &lt;span class="crayons-avatar crayons-avatar--xl mr-2 shrink-0"&gt;
                        &lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4242816%2Fa89fc242-4a39-4336-bc30-3dd490240ce4.jpg" class="crayons-avatar__image" alt="" width="449" height="449"&gt;
                      &lt;/span&gt;
                      &lt;span class="crayons-link crayons-subtitle-2 mt-5"&gt;Walid Hadri&lt;/span&gt;
                    &lt;/a&gt;
                  &lt;/div&gt;
                  &lt;div class="print-hidden"&gt;
                    
                      Follow
                    
                  &lt;/div&gt;
                  &lt;div class="author-preview-metadata-container"&gt;&lt;/div&gt;
                &lt;/div&gt;
              &lt;/div&gt;
            &lt;/div&gt;

          &lt;/div&gt;
          &lt;a href="https://dev.to/walid_hadri/what-actually-happens-when-you-press-enter-on-chatgpt-n34" class="crayons-story__tertiary fs-xs"&gt;&lt;time&gt;Nov 13&lt;/time&gt;&lt;span class="time-ago-indicator-initial-placeholder"&gt;&lt;/span&gt;&lt;/a&gt;
        &lt;/div&gt;
      &lt;/div&gt;

    &lt;/div&gt;

    &lt;div class="crayons-story__indention"&gt;
      &lt;h2 class="crayons-story__title crayons-story__title-full_post"&gt;
        &lt;a href="https://dev.to/walid_hadri/what-actually-happens-when-you-press-enter-on-chatgpt-n34" id="article-link-5029583"&gt;
          What Actually Happens When You Press Enter on ChatGPT
        &lt;/a&gt;
      &lt;/h2&gt;
        &lt;div class="crayons-story__tags"&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/ai"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;ai&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/machinelearning"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;machinelearning&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/llm"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;llm&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/chatgpt"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;chatgpt&lt;/a&gt;
        &lt;/div&gt;
      &lt;div class="crayons-story__bottom"&gt;
        &lt;div class="crayons-story__details"&gt;
            &lt;a href="https://dev.to/walid_hadri/what-actually-happens-when-you-press-enter-on-chatgpt-n34#comments" class="crayons-btn crayons-btn--s crayons-btn--ghost crayons-btn--icon-left flex items-center"&gt;
              

              &lt;span class="hidden s:inline"&gt;Add&amp;nbsp;Comment&lt;/span&gt;
            &lt;/a&gt;
        &lt;/div&gt;
        &lt;div class="crayons-story__save"&gt;
          &lt;small class="crayons-story__tertiary fs-xs mr-2"&gt;
            7 min read
          &lt;/small&gt;
          
          &lt;div class="only-sidebar-menu-item trusted-visible-block print-hidden align-center"&gt;
                
                    
                
          &lt;/div&gt;

        &lt;/div&gt;
      &lt;/div&gt;
    &lt;/div&gt;
  &lt;/div&gt;
&lt;/div&gt;


&lt;/div&gt;


</description>
    </item>
    <item>
      <title>What Actually Happens When You Press Enter on ChatGPT</title>
      <dc:creator>Walid Hadri</dc:creator>
      <pubDate>Fri, 13 Nov 2026 11:40:14 +0000</pubDate>
      <link>https://dev.to/walid_hadri/what-actually-happens-when-you-press-enter-on-chatgpt-n34</link>
      <guid>https://dev.to/walid_hadri/what-actually-happens-when-you-press-enter-on-chatgpt-n34</guid>
      <description>&lt;p&gt;&lt;em&gt;One question, followed from the keyboard to a GPU in America and back: about 13,600 km, one second, and a model that never knows its answer in advance.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fq550r4ru447ngax7r0ro.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fq550r4ru447ngax7r0ro.png" alt="A gold line of light crossing the Atlantic on a night globe" width="800" height="450"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Your question, as light, crossing the Atlantic.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;You type “What is the capital of France?” and press Enter. About a second later, ChatGPT answers: “The capital of France is Paris.”&lt;/p&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/GWjA5eHIXXk" width="710" height="399"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;

&lt;p&gt;We do this dozens of times a day, and the answer feels instant, almost local. &lt;strong&gt;It isn’t.&lt;/strong&gt; From my desk in France, that one question went to the bottom of the Atlantic, into a hall full of GPUs in America, through every layer of a neural network, and back, in about one second.&lt;/p&gt;

&lt;p&gt;I wanted to see all of it, not just “the model” or just “the internet”, so I followed one question from start to finish and turned it into a &lt;strong&gt;14-minute animated video&lt;/strong&gt;. This article is the written tour. The video shows every step.&lt;/p&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/GWjA5eHIXXk" width="710" height="399"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;

&lt;p&gt;I’ve always learned best by teaching: having to explain something is the fastest way to find out what you don’t understand yet. This video is the first one on my channel, &lt;strong&gt;The Next Token Lab&lt;/strong&gt;.&lt;/p&gt;




&lt;h2&gt;
  
  
  1. Leaving your laptop (0 ms)
&lt;/h2&gt;

&lt;p&gt;Before anything moves, your browser does three things. It &lt;strong&gt;packs&lt;/strong&gt; the question into a small structured message (JSON). It &lt;strong&gt;looks up&lt;/strong&gt; where chatgpt.com lives in the internet’s phone book (DNS). And it &lt;strong&gt;locks&lt;/strong&gt; the message (TLS), so nobody on the way can read it.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F78mh161yz5z3hcgi8nbp.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F78mh161yz5z3hcgi8nbp.png" alt="A browser checklist: pack the message (JSON), find the address (DNS), lock it (TLS)" width="800" height="450"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Pack the message, find the address, lock it: three steps before your question even leaves the laptop.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Then it leaves as &lt;strong&gt;pulses of light in a glass fibre 0.125 mm across&lt;/strong&gt;, barely thicker than a human hair.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fntr2euc9pgydezvrbegr.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fntr2euc9pgydezvrbegr.png" alt="Pulses of gold light inside a glass fibre, next to a human hair" width="800" height="450"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Your question, as light in a glass fibre.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Paris, two milliseconds later
&lt;/h2&gt;

&lt;p&gt;From my connection, the question needed &lt;strong&gt;9 hops and about 2 milliseconds&lt;/strong&gt; to reach the nearest edge server, in Paris. That server, run by Cloudflare for ChatGPT, is the front door. It checks that you’re not a bot or an attack, and it’s where the encrypted connection is unlocked.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Under the Atlantic
&lt;/h2&gt;

&lt;p&gt;OpenAI doesn’t say which data centre answers a given question. But for a question from Europe, it’s &lt;strong&gt;most likely in the United States&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;So the message heads to the Atlantic coast, to a small building by the beach where a transatlantic cable goes into the sea. On the ocean floor, kilometres down, that cable is &lt;strong&gt;about as thin as a garden hose&lt;/strong&gt;. Inside are 12 pairs of glass fibres, wrapped in steel wires for strength and copper that carries power to repeaters every 60 to 100 km. The repeaters boost the light along the way.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F6ioa5epxlrsxkfgrpty7.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F6ioa5epxlrsxkfgrpty7.png" alt="Cutaway of a subsea cable: plastic, copper, steel wires and glass fibres" width="800" height="450"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Inside a subsea cable: 24 glass fibres (12 pairs), steel wires, and copper carrying power from the shore.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;In glass, light travels at about 200,000 km per second. The cable is about 6,400 km long, so that’s &lt;strong&gt;at least 32 milliseconds: the longest stretch of the whole trip&lt;/strong&gt;. About 45 milliseconds after you pressed Enter, your question comes ashore in Virginia.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. OpenAI’s front door
&lt;/h2&gt;

&lt;p&gt;Now it reaches OpenAI’s servers. They receive tens of thousands of messages every second, so a &lt;strong&gt;load balancer&lt;/strong&gt; sends yours to a server that isn’t too busy. Those servers run in huge clusters managed by a system called &lt;strong&gt;Kubernetes&lt;/strong&gt;. (In December 2024, one bad update overloaded those clusters and ChatGPT was down for hours.)&lt;/p&gt;

&lt;p&gt;Before anything else, OpenAI checks who you are and that you haven’t hit your limit. Your message is saved to your history: ChatGPT’s main database keeps one master copy and about fifty copies around the world. Automatic safety classifiers look for harmful requests. A question about capitals goes straight through.&lt;/p&gt;

&lt;p&gt;Then comes something most people don’t know: &lt;strong&gt;the model never sees your question alone.&lt;/strong&gt; ChatGPT wraps it in a longer text, with instructions about who it is and how to behave, what it remembers about you if memory is on, and the conversation so far.&lt;/p&gt;

&lt;p&gt;And now, the first step that’s really about AI: &lt;strong&gt;tokenization&lt;/strong&gt;. &lt;strong&gt;The model can’t read letters.&lt;/strong&gt; It only works with numbers. The text is cut into tokens, and each token gets its number from a fixed list of about 200,000. Common words are one token, usually with the space in front. Rare words are cut into pieces: “strawberry” becomes &lt;strong&gt;st · raw · berry&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffa965ady3fna4cdf60qp.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffa965ady3fna4cdf60qp.png" alt="The word strawberry split into three tokens: st, raw, berry, with their numbers" width="800" height="450"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Three pieces, not ten letters: why models stumble when you ask them to count the R’s in “strawberry”.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Your question becomes &lt;strong&gt;7 numbers&lt;/strong&gt;. With a minimal set of instructions and the little markers that say who’s talking, the whole text is already &lt;strong&gt;74 tokens&lt;/strong&gt;. (ChatGPT’s real instructions are longer and not public.)&lt;/p&gt;

&lt;h2&gt;
  
  
  5. A room full of GPUs
&lt;/h2&gt;

&lt;p&gt;The machine that writes the answer may be in another building, even another state, like the new AI campuses in Texas: eight buildings, about a gigawatt of power. Inside, there are rows and rows of racks. &lt;strong&gt;One rack holds 72 GPUs&lt;/strong&gt;, chips built to do huge numbers of multiplications at the same time. Together, a rack draws &lt;strong&gt;about as much power as a hundred homes&lt;/strong&gt;, so it’s cooled with liquid, not air.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fbygw4ckyinz4u47oc4lv.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fbygw4ckyinz4u47oc4lv.png" alt="A corridor of liquid-cooled GPU racks with orange and blue coolant pipes" width="800" height="450"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Liquid-cooled racks: about 120 kW each, the power of roughly 100 homes.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Your question doesn’t get the chips to itself. &lt;strong&gt;It rides along with thousands of other conversations&lt;/strong&gt;, and at every step, the machine works on all of them at once.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Inside the model
&lt;/h2&gt;

&lt;p&gt;Nobody outside OpenAI can open ChatGPT’s model. But almost every chatbot today is built on the same architecture, called the &lt;strong&gt;Transformer&lt;/strong&gt;. So in the video I opened a small public one, &lt;strong&gt;GPT-2 from 2019&lt;/strong&gt;, and every number shown is real: I ran GPT-2 on our sentence.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A model is a long list of numbers&lt;/strong&gt;, called weights, learned during training. GPT-2 has about 124 million of them (half a gigabyte), loaded into the GPU’s own memory before any answer.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Each token becomes an address.&lt;/strong&gt; A token’s number picks one row in a table of about 50,000 rows, and that row is a list of 768 numbers: an address in a space of meaning. Training placed related words close together. Next to “France” sit “French”, “Germany”, “Spain”, “Italy” and “Europe”.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Attention lets words look at each other.&lt;/strong&gt; To guess what comes after “is”, the last word has to look back at the words before it. It’s a bit like searching a library: each word makes a &lt;em&gt;query&lt;/em&gt;, compares it with every earlier word’s &lt;em&gt;key&lt;/em&gt;, and takes in a blend of their &lt;em&gt;values&lt;/em&gt;. GPT-2 has 144 of these attention heads, each with its own job.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fl15q8bcfp7gv7dhq9p61.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fl15q8bcfp7gv7dhq9p61.png" alt="An attention pattern: each word attends to the word just before it" width="800" height="450"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;One of GPT-2’s 144 attention heads, measured on our sentence: every word looks at the word just before it.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Then each word thinks on its own&lt;/strong&gt;, in a block of 3,072 detectors, each tuned to a pattern. For “is”, only about 3 in 100 fire strongly. This block holds two thirds of each layer’s weights, and researchers have found that facts, like the capital of a country, are largely stored there.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Attention, then the detectors: that’s one layer.&lt;/strong&gt; GPT-2 stacks twelve of them, like a draft going through twelve editors. At the start, “Paris” is ranked around 28,000th among possible next words. Layer by layer, it climbs.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;At the top, the model votes.&lt;/strong&gt; The last word’s list is compared with every token GPT-2 knows, and the scores become percentages. Small GPT-2’s leaders are “the” at 16%, “Paris” at 11% and “France” at 6%. That’s not a confident answer, because this model is tiny. The largest GPT-2 already gives Paris 78%.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fj0zs98z3ulvxn8rvsq42.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fj0zs98z3ulvxn8rvsq42.png" alt="A weighted wheel of next-word chances, with Paris at 11%" width="800" height="450"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Decoding: instead of always taking the top word, chatbots spin a weighted wheel. Paris comes up about one time in nine.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Always taking the top word is called &lt;strong&gt;greedy decoding&lt;/strong&gt;, and here it would write “the capital of France” and go round in circles. So chatbots usually &lt;strong&gt;spin a weighted wheel&lt;/strong&gt; instead. A setting called &lt;strong&gt;temperature&lt;/strong&gt; shapes that wheel: low, and the favourite becomes even more likely; high, and the text gets creative, then strange.&lt;/p&gt;

&lt;p&gt;And here’s the most important idea of all: &lt;strong&gt;the model never writes the answer in one go.&lt;/strong&gt; It only ever guesses the next token. That token is added to the text, and the whole model runs again: “Paris”, then a full stop, then a special token that means “end of answer”. Two tricks make this fast. The model keeps earlier words’ keys and values in memory (the &lt;strong&gt;KV cache&lt;/strong&gt;), and it reads your whole prompt in a single pass.&lt;/p&gt;

&lt;h2&gt;
  
  
  7. The way back
&lt;/h2&gt;

&lt;p&gt;Each token is sent back as soon as it’s chosen, back under the ocean, to your screen. That’s why the answer appears word by word. &lt;strong&gt;About one second, start to finish.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flwj0rd56a8hz2yfzpo7o.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flwj0rd56a8hz2yfzpo7o.png" alt="A printed receipt: distance, hops, time, tokens in and out, energy" width="800" height="450"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;The receipt for one easy question.&lt;/em&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Distance:&lt;/strong&gt; about 13,600 km, there and back, if the answer is written in Virginia&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;To the edge server:&lt;/strong&gt; 9 hops, about 2 ms (measured)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tokens in:&lt;/strong&gt; 74. &lt;strong&gt;Tokens out:&lt;/strong&gt; 7.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Energy:&lt;/strong&gt; about 0.34 Wh, OpenAI’s average per query&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What’s measured, and what isn’t
&lt;/h2&gt;

&lt;p&gt;OpenAI’s internals aren’t public. The network numbers come from my own connection in France. The model numbers were measured on GPT-2, an open model with the same design. The steps inside OpenAI come from what OpenAI has said publicly. Wherever something is likely or illustrative, the video says so.&lt;/p&gt;




&lt;p&gt;If you’d like to see all of this move, the full video is here: &lt;strong&gt;&lt;a href="https://youtu.be/GWjA5eHIXXk" rel="noopener noreferrer"&gt;https://youtu.be/GWjA5eHIXXk&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;My channel, &lt;strong&gt;The Next Token Lab&lt;/strong&gt;, is where I’ll keep explaining how the tech we use every day actually works: &lt;a href="https://www.youtube.com/@TheNextTokenLab" rel="noopener noreferrer"&gt;https://www.youtube.com/@TheNextTokenLab&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What should I explain next?&lt;/strong&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>llm</category>
      <category>chatgpt</category>
    </item>
  </channel>
</rss>
