DEV Community

Cover image for From Zero to Shakespeare GPT: I Built a Language Model in JavaScript
Roshan Shikhare
Roshan Shikhare

Posted on Originally published at Medium

From Zero to Shakespeare GPT: I Built a Language Model in JavaScript

By the time I reached this stage, I had spent a considerable amount of time learning how neural networks work.

I had implemented neurons, trained MLPs, worked through backpropagation, experimented with activation functions, and eventually built the components of a Transformer.

But there was still one thing missing.

I hadn't brought everything together into a language model that could learn from actual text.

That was the goal from the beginning.

I wanted to take everything I had learned and build a tiny GPT from scratch using JavaScript.

Not a chatbot powered by an API. Not a wrapper around a pretrained model.

An actual neural network with its own trainable weights, attention mechanism, training loop and text-prediction pipeline.

And for the dataset, I chose Shakespeare.

1. Why Shakespeare?

I needed a real text corpus to train the model.

I didn't want to keep testing everything with small artificial examples. I wanted the model to learn from actual sequences of language.

Shakespeare was a good starting point for a few reasons:

  • It provides a substantial amount of continuous text.
  • It contains dialogue, punctuation, names and varied sentence structures.
  • Its relatively small character vocabulary makes it manageable for a first character-level language model.
  • The generated output is fun to inspect, even when the model makes mistakes.

I used the Tiny Shakespeare corpus from Andrej Karpathy's "char-rnn" project.

The dataset contains roughly 1.1 million characters and, with our character-level tokeniser, produces a vocabulary of 65 unique characters.

That meant our model had 65 possible character IDs to work with.

This was the first time our learning experiments were connected to a meaningful language dataset.

2. Designing the project before writing more code

One thing I had learned from the earlier experiments was that putting everything into one enormous file would make the project difficult to understand.

I wanted the architecture to reflect the actual concepts we were learning.

So I separated the implementation into modules.

tiny-shakespeare-gpt/
│
├── data/
│   ├── shakespeare.txt
│   ├── nextTokenDataset.js
│   └── prepareData.js
│
├── tokenizer/
│   └── tokenizer.js
│
├── tensor/
│   ├── Tensor.js
│   ├── operations.js
│   ├── autograd.js
│   └── random.js
│
├── layers/
│   ├── embedding.js
│   ├── positionalEmbedding.js
│   ├── linear.js
│   ├── layerNorm.js
│   ├── gelu.js
│   ├── causalAttention.js
│   ├── selfAttentionHead.js
│   ├── multiHeadAttention.js
│   ├── feedForward.js
│   └── transformerBlock.js
│
├── models/
│   └── gptModel.js
│
├── training/
│   ├── loss.js
│   ├── optimizer.js
│   ├── trainStep.js
│   ├── trainLoop.js
│   └── checkpoint.js
│
└── generation/
    └── generate.js
Enter fullscreen mode Exit fullscreen mode

Each folder had a responsibility.

The tokeniser handled text. The tensor module handled numerical operations and gradients. The layers contained the neural-network components. The model assembled the architecture. The training module handled learning.

It might seem like ordinary software organisation, but it mattered a lot here.

When something went wrong, I could investigate a particular operation or layer instead of debugging one massive file.

Project module map
The project is split into one folder per concept: data, tokenizer, tensor, layers, models, training and generation.

3. Building the tensor engine

This was one of the most important implementation decisions.

Machine-learning frameworks normally provide tensors and automatic differentiation.

I wasn't using one.

So I needed my own basic Tensor structure.

At its simplest, a Tensor stores:

  • Numerical data.
  • Its shape.
  • A gradient array.
  • References to the operations that produced it.
  • A backward function.

Conceptually:

class Tensor {
  constructor(data, shape) {
    this.data = data;
    this.shape = shape;
    this.grad = new Array(data.length).fill(0);

    this.parent = [];
    this.backward = () => {};
  }
}
Enter fullscreen mode Exit fullscreen mode

The parent references allow us to track the computation graph.

When we perform an operation, we create a new Tensor and record which earlier tensors contributed to it.

For example:

A ----\
       Add ----> C
B ----/
Enter fullscreen mode Exit fullscreen mode

If we want to calculate gradients for C, we need to propagate them back to A and B.

I implemented operations such as addition, multiplication, matrix multiplication, transpose, exponential, logarithm and square root, with their corresponding backward functions.

Then I implemented a topological traversal of the computation graph so that backward operations could execute in the correct order.

This is the foundation of automatic differentiation.

Computation graph with forward and backward passes
Each Tensor records its parents. The forward pass builds the graph (A + B → C), and backward functions run in topological order to send gradients from C back to A and B.

It was also a very different experience from calling a framework's backward() function without knowing what happened underneath.

4. Building the GPT layers

With tensors and gradients available, I could assemble the actual model components.

The implementation included:

Embedding: Converts token IDs into learned vectors.

Positional embedding: Adds learned position information to the token representations.

Linear layers: Apply trainable weight matrices and biases.

LayerNorm: Normalises feature values and learns scale and shift parameters.

GELU: Adds the nonlinearity used in the feed-forward network.

Causal attention: Calculates attention while preventing access to future positions.

Multi-head attention: Combines multiple attention heads.

Feed-forward network: Applies learned transformations to each token representation.

Transformer block: Combines attention, residual connections, normalisation and feed-forward layers.

The most interesting part was not writing each class independently.

It was making sure their dimensions and gradients connected correctly.

For example, our multi-head attention had to combine the outputs of individual heads and project the concatenated representation back into the model dimension.

The Transformer block then had to preserve the input shape so that residual connections worked.

A mistake in any of these interfaces could break the entire model.

5. How does GPT learn to predict the next character?

Now came the actual language-modelling task.

We don't need to teach the model to generate an entire paragraph in one operation.

Instead, we train it to predict the next character.

Consider:

Input:  HELLO
Target: ELLO!
Enter fullscreen mode Exit fullscreen mode

For a sequence of input characters:

H E L L O
Enter fullscreen mode Exit fullscreen mode

The targets are the next characters:

E L L O !
Enter fullscreen mode Exit fullscreen mode

Every position in the sequence contributes a prediction task.

This is called next-token prediction.

Our dataset helper creates input and target sequences by shifting the token IDs by one position.

In simplified JavaScript:

function createNextTokenPairs(tokenIds, start, length) {
  const inputs = [];
  const targets = [];

  for (let i = 0; i < length; i++) {
    inputs.push(tokenIds[start + i]);
    targets.push(tokenIds[start + i + 1]);
  }

  return { inputs, targets };
}
Enter fullscreen mode Exit fullscreen mode

Our actual batching implementation also samples multiple random starting positions and combines them into batches.

For example, with:

Batch size: 32
Sequence length: 256
Enter fullscreen mode Exit fullscreen mode

The model receives 32 sequences, each containing 256 characters.

The positional embedding matrix is sized to the configured maximum sequence length, so here it has 256 rows.

It predicts the next character at every position.

This gives us 32 × 256 prediction targets in one training step.

And because we use causal attention, the model cannot look at future characters when making those predictions.

Input and target shifted by one
Next-token prediction: the target sequence is the input shifted by one position. With batch size 32 and sequence length 256, one step has 32 × 256 prediction targets.

6. From logits to probabilities: the loss function

The GPT model produces logits.

For our 65-character vocabulary, every sequence position produces 65 scores.

These aren't probabilities yet.

We apply softmax to turn the scores into a probability distribution:

Pi=ezi∑jezj P_i = \frac{e^{z_i}}{\sum_j e^{z_j}}

For training, we use cross-entropy loss.

For a single correct target token:

L=−log⁡(Pcorrect) L = -\log(P_{\text{correct}})

If the model assigns a high probability to the correct next character, the loss is low.

If it assigns a very low probability, the loss is high.

Softmax and cross-entropy loss
Logits become probabilities through softmax, and cross-entropy is −log of the probability given to the correct character, averaged over the target positions.

For a batch of next-character predictions, we average the loss across the target positions.

I implemented stable softmax cross-entropy using a log-sum-exp approach to avoid numerical overflow.

This was important because numerical stability isn't optional when you're training a model.

A mathematically correct formula can still produce problematic numerical values if implemented carelessly.

7. The first real Shakespeare training step

This was a milestone.

We loaded the actual Shakespeare dataset, built the tokeniser, prepared a batch and passed it through our GPT model.

The first real training experiment used:

Vocabulary size: 65
Batch size: 32
Sequence length: 256
Enter fullscreen mode Exit fullscreen mode

The model processed actual Shakespeare passages.

We also decoded the input and target IDs back into text to verify that the dataset was aligned correctly.

Then we ran the first training step.

The output included:

Running one training step...

Loss: 5.4462

PASS: First Shakespeare training step completed.
Enter fullscreen mode Exit fullscreen mode

That number wasn't something to celebrate as a good language model yet.

It was something more fundamental:

The entire forward pass, loss calculation, backpropagation and parameter update had worked on real text.

The model had completed an actual learning step.

After all the individual mathematical experiments, seeing the components work together was a satisfying moment.

8. The first training experiment: does the loss decrease?

Getting one training step to work was only the beginning.

The next question was whether the model could actually learn.

I ran an experiment where the model repeatedly trained on the same fixed batch.

This was a separate run, which is why its starting loss differs from the single training step above.

The results:

INITIAL LOSS: 4.9393

Step 10  | Loss: 4.6922
Step 20  | Loss: 4.4776
Step 30  | Loss: 4.3167
Step 40  | Loss: 4.1928
Step 50  | Loss: 4.0918
Step 60  | Loss: 4.0043
Step 70  | Loss: 3.9256
Step 80  | Loss: 3.8529
Step 90  | Loss: 3.7848
Step 100 | Loss: 3.7202

FINAL LOSS: 3.7139
Enter fullscreen mode Exit fullscreen mode

The loss decreased from approximately 4.94 to 3.71.

Loss curve over 100 steps on one fixed batch
Loss on one fixed batch went from 4.9393 at the start to 3.7202 at step 100, with a final loss of 3.7139. Because it was the same batch every step, this doesn't show generalisation.

That was encouraging.

But there was an important catch.

We were repeatedly training on the same batch.

The model was getting better at predicting that particular batch, but that didn't prove it had learned patterns that generalised to unseen Shakespeare passages.

This is where training loss and validation loss become important.

9. Training loss versus validation loss

A model can become very good at the examples it has already seen without becoming equally good at new examples.

That's overfitting.

To investigate this, we split the dataset into two parts:

  • Training data: used to update model weights.
  • Validation data: used to evaluate the model without updating its weights.

The separation is important.

During validation, we perform a forward pass and calculate the loss. We do not call backpropagation or the optimizer.

This lets us ask two different questions:

  1. Is the model getting better at the training examples?
  2. Is the model also getting better at examples it hasn't trained on?

If training loss decreases while validation loss starts increasing, that can be a sign of overfitting.

Training versus validation
Training steps run forward, loss, backprop and the optimizer. Validation runs only the forward pass and loss, so the weights don't change.

I wanted the project to include this distinction from the beginning because a decreasing training loss alone is not enough to claim that a language model is learning well.

10. Saving the model: don't lose what it learned

Another practical requirement appeared once training started working.

If I trained the model and then closed the program, I didn't want to lose the learned weights.

I implemented checkpoint saving.

A checkpoint stores the model's configuration and its trainable parameter values.

In simplified form:

const checkpoint = {
  config,
  parameters: model.parameters().map(parameter => ({
    shape: parameter.shape,
    data: parameter.data
  }))
};
Enter fullscreen mode Exit fullscreen mode

Why save the shape as well as the data?

Because a parameter's dimensions are part of the model architecture.

A matrix with shape [65, 32] is not interchangeable with one having shape [32, 65], even if both contain the same number of values.

I also implemented checkpoint loading, including checks for parameter counts, tensor shapes and data lengths.

Then came the verification.

We trained Model A, saved its weights, created a completely fresh Model B, and loaded the checkpoint into it.

Both models received the same input.

The result:

CHECKPOINT VERIFICATION

Model A loss: 5.11065033
Model B loss: 5.11065033

Maximum output difference: 0

PASS: Model weights saved and restored correctly.
Enter fullscreen mode Exit fullscreen mode

This was a particularly useful test.

It demonstrated that we could take the learned parameters from one model instance and restore them into another without changing the model's predictions.

That is the foundation for a model that can be trained once and reused later.

Checkpoint save and load verification
Model A's config and parameters (shape + data) are saved and loaded into a fresh Model B. Both give a loss of 5.11065033 with a maximum output difference of 0.

11. What happens when we actually generate text?

Training and generation are two different processes.

During training, the model receives an input sequence and learns to predict the next character at every position.

During generation, we start with a prompt and repeatedly ask the model for the next character.

The process looks like this:

Prompt: "ROMEO:"
       |
       v
Convert prompt to token IDs
       |
       v
Pass tokens through GPT
       |
       v
Read logits at final position
       |
       v
Apply temperature and softmax
       |
       v
Sample next character
       |
       v
Append character to prompt
       |
       v
Repeat
Enter fullscreen mode Exit fullscreen mode

One parameter that makes generation interesting is temperature.

Before sampling, we divide logits by temperature:

zi′=ziT z_i' = \frac{z_i}{T}

Then apply softmax.

A lower temperature makes the probability distribution more concentrated, favouring higher-scoring characters.

A higher temperature makes it more spread out, allowing more varied choices.

This doesn't make the model more intelligent. It changes how we sample from its predictions.

Autoregressive generation loop with temperature
Generation is a loop: encode the prompt, run GPT, read the final position's logits, apply temperature and softmax, sample, append and repeat. Lower T concentrates the distribution, and higher T spreads it out.

There is also a context-window limit.

Our model can only process a configured maximum number of characters at once. As generation continues, we keep the most recent part of the context within that limit.

This is one of the practical differences between the simple idea of next-character prediction and a working autoregressive generation system.

12. What this project is — and what it isn't

It's important to be realistic about what we've built.

This is a small, character-level GPT trained on a limited Shakespeare corpus.

It is not a ChatGPT replacement.

It doesn't have the scale, knowledge, instruction-following abilities or reasoning capabilities of a modern large language model.

It learns statistical patterns in the training text and uses those patterns to predict the next character.

But that's exactly why it is useful as a learning project.

We can inspect its tensors. We can examine its weights. We can follow the forward pass. We can understand the attention calculation. We can see how the loss changes when the model learns.

There isn't a pretrained model hiding all those details from us.

13. Turning the model into something usable

The next stage is to move beyond running isolated training scripts.

I want the final project to behave like an actual application.

The plan is to build a Node.js server that:

  • Loads the tokeniser and trained checkpoint when it starts.
  • Exposes an API for text generation.
  • Allows another training run to be started when needed.
  • Reports training progress and validation loss.
  • Saves updated checkpoints.
  • Provides a small browser interface for interacting with the model.

The intention is simple:

Train once, save the model, restart the server, and continue using the trained model without starting from random weights every time.

I also want to publish the project on GitHub so other developers can explore the implementation and experiment with it.

14. What I learned from this entire journey

When I started on 30 July 2026, I had a question about how AI models work.

I thought I would probably spend some time reading about neural networks and Transformers.

Instead, the journey took me through:

  • Implementing trainable neurons and MLPs.
  • Understanding vectors, matrices, weights and biases.
  • Calculating gradients and implementing backpropagation.
  • Debugging exploding gradients and NaN values.
  • Comparing analytical and numerical gradients.
  • Experimenting with ReLU, tanh, Leaky ReLU and GELU.
  • Building embeddings and positional embeddings.
  • Understanding Query, Key and Value projections.
  • Implementing causal and multi-head attention.
  • Building Transformer blocks with residual connections and LayerNorm.
  • Implementing a small GPT model.
  • Training it on real Shakespeare text.
  • Evaluating loss and saving checkpoints.

The biggest lesson wasn't that I could now build a large language model.

I couldn't, and that was never the goal.

The lesson was that a technology that once seemed like a black box became understandable when I broke it into smaller pieces and implemented those pieces myself.

I also learned that knowing a formula and understanding an implementation are two very different things.

You can read about attention in five minutes. Understanding the tensor shapes, gradients, masking and numerical stability takes much more work.

And that work is worth it.


One final thought

I started this project because I didn't want to remain someone who only knows how to use AI tools.

I wanted to understand what happens behind them.

Building a tiny GPT didn't answer every question I have about AI. In fact, it gave me many more questions to explore.

But it changed how I look at these models.

The next time I use an LLM, I won't see only a prompt and a response. I'll think about tokenisation, embeddings, attention, learned weights, probability distributions and all the computation happening underneath.

And that's what I wanted from this journey.

Not to build the biggest model. To understand the model.

Explore the project

  • Project: Tiny Shakespeare GPT
  • Language: JavaScript
  • Approach: From-scratch character-level Transformer
  • Dataset: Tiny Shakespeare

Source code: github.com/Roshan190/tiny-shakespeare-gpt-js

Thanks for following Beyond the Prompt. This is the end of the first journey, not the end of my exploration of AI.

Project journey started: 30 July 2026.

Top comments (0)