How to Train Your Own ChatGPT for $50?

TL;DR

nanochat is Andrej Karpathy’s open-source LLM training framework for training a GPT-2-level chatbot for under $50 in about two hours. It runs on a single 8×H100 GPU node, keeps the core model to roughly 500 lines, and uses one main configuration dial (--depth) to derive the remaining training settings. Current records report a 1.65-hour run with a CORE score of 0.2626, exceeding OpenAI’s 2019 GPT-2 score while using far less time and cost.

Try Apidog today

Introduction

Training a language model once required large budgets, specialized hardware, and a research team. nanochat makes the full workflow approachable for developers who want to run, inspect, and modify an LLM training stack.

The project covers tokenization, pretraining, supervised fine-tuning (SFT), evaluation, inference, and a ChatGPT-like web UI. Its reference setup runs on one 8×H100 node and can finish GPT-2-level pretraining in under two hours.

Why this matters for developers

The cost and iteration-time difference is significant:

• OpenAI GPT-2 in 2019: approximately 168 hours and $43,000
• nanochat record run: 1.65 hours and approximately $48 on-demand

For teams building AI features, this makes it practical to:

• Test changes to model architecture and training recipes.
• Learn how an LLM pipeline works end to end.
• Run smaller experiments before committing to larger infrastructure.
• Build and validate APIs around a self-trained model.

You can pair a local or hosted nanochat inference service with an API platform such as Apidog to document, test, and share your AI endpoints.

What you’ll learn

By the end of this post, you will know how to:

• Set up nanochat and download its training data.
• Train a tokenizer and a base model.
• Fine-tune the model for chat.
• Run CLI and web-based inference.
• Use small models for fast architecture experiments.
• Interpret the practical limits of GPT-2-level capability.

What is nanochat?

nanochat is a minimal LLM training harness. Rather than exposing a large set of configuration files, it encodes a training recipe directly in readable Python modules and derives most hyperparameters from model depth.

The project is intended to be readable, hackable, and forkable.

Core claim

nanochat can train a GPT-2-capability model with roughly 1.6B parameters for:

• About $48 on demand: roughly two hours at approximately $24/hour for 8×H100.
• About $15 on spot instances.

For comparison, OpenAI’s original GPT-2 training reportedly used 32 TPU v3 chips for around seven days and cost approximately $43,000.

Pipeline components

Stage	Script	Description
Tokenization	scripts.tok_train	Train a BPE tokenizer with a 32,768-token vocabulary
Pretraining	scripts.base_train	Train the base GPT model
Fine-tuning	scripts.chat_sft	Run supervised fine-tuning for chat
Evaluation	scripts.base_eval	Run CORE metric and bits-per-byte evaluation
Inference	scripts.chat_cli	Use a command-line chat interface
Web UI	scripts.chat_web	Launch a ChatGPT-like web interface

The one-dial configuration model

The main nanochat sizing parameter is --depth, the number of transformer layers.

# GPT-1-size model
torchrun --standalone --nproc_per_node=8 -m scripts.base_train -- \
  --depth=12

# GPT-2-capability model
torchrun --standalone --nproc_per_node=8 -m scripts.base_train -- \
  --depth=24

# Larger experimental configuration
torchrun --standalone --nproc_per_node=8 -m scripts.base_train -- \
  --depth=26

When you set --depth, nanochat derives related settings, including:

• Transformer width and embedding dimension.
• Number of attention heads.
• Learning rates by parameter group.
• Total training steps.
• Weight-decay schedules.
• Batch sizes.

This removes the need to manually tune every model dimension and optimization setting for each experiment.

Why depth can drive the recipe

nanochat’s defaults are based on measured scaling relationships across many runs. Depth, width, batch size, and training duration follow predictable relationships, so the code encodes those relationships rather than exposing every value as a separate option.

For implementation work, this means you can start with one controlled variable—model depth—then focus your experiments on the architectural or data changes you actually want to evaluate.

The “time to GPT-2” leaderboard

nanochat tracks the time needed to exceed OpenAI’s original GPT-2 CORE score of 0.256525. CORE evaluates 22 tasks, including ARC, MMLU, and other benchmarks from the DCLM suite.

Run	Model	Time	CORE score	Key innovation
Original GPT-2	1.6B	168 hours	0.2565	OpenAI 2019 baseline
Run 1	d24	3.04 hours	0.2585	Initial baseline
Run 2	d26	2.91 hours	0.2578	FP8 training
Run 3	d26	2.76 hours	0.2602	1M-token batch size
Run 4	d24	2.02 hours	0.2571	ClimbMix dataset
Run 5	d24	1.80 hours	0.2690	AI-discovered optimizations
Run 6	d24	1.65 hours	0.2626	Improved smear/backout

AI-assisted optimization

Runs 5 and 6 incorporated changes from Karpathy’s autoresearch system. The system tested architecture modifications on smaller d12 models, where each experiment took about five minutes, then applied successful changes to the d24 configuration.

The reported improvements included:

• A better backout mechanism for mid-layer residual subtraction.
• A more efficient smear implementation for mixing bigram information from previous tokens.

These changes reduced training time from 2.02 hours to 1.65 hours, a 19% improvement.

How nanochat works

The codebase has roughly 3,000 lines across its core modules. The following sections highlight the implementation details most relevant when you want to modify or extend the system.

1. GPT model: nanochat/gpt.py

The transformer implementation uses several modern design choices:

• Rotary embeddings (RoPE) for relative position encoding.
• QK normalization to stabilize attention training.
• Untied weights for token embeddings and output projection.
• ReLU² activations in the MLP.
• Grouped Query Attention (GQA) to reduce KV-head overhead.
• Sliding-window attention patterns such as SSSL.
• Flash Attention 3 on supported Hopper GPUs, with PyTorch SDPA fallback.

Value embeddings

Alternating layers can include learnable value embeddings mixed through input-dependent gates:

# Value residual: mix in value embedding with per-head gate
if ve is not None:
    ve = ve.view(B, T, self.n_kv_head, self.head_dim)
    gate = 3 * torch.sigmoid(self.ve_gate(x[..., :self.ve_gate_channels]))
    v = v + gate.unsqueeze(-1) * ve

This adds capacity with limited additional compute.

Training-dynamics mechanisms

nanochat also includes learned residual and token-mixing mechanisms:

# 1. Per-layer residual scaling
x = self.resid_lambdas[i] * x + self.x0_lambdas[i] * x0

# 2. Smear: mix previous token embedding for bigram information
gate = self.smear_lambda * torch.sigmoid(self.smear_gate(x[:, :, :24]))
x = x + gate * x_pre_smear

# 3. Backout: subtract mid-layer residual
x = x - self.backout_lambda * x_backout

If you are testing architecture ideas, change one mechanism at a time and validate it first on a d12 run. That keeps feedback loops short and makes regressions easier to isolate.

2. Optimizer: nanochat/optim.py

nanochat uses different optimizers for different parameter types.

Parameter type	Optimizer	Purpose
Embeddings and lm_head	AdamW	Standard adaptive optimization
Scalar parameters	AdamW	Learned scaling factors
2D matrices	Muon	Orthogonalized updates

Muon and Polar Express

Muon—MomentUm Orthogonalized by Newton-Schulz—orthogonalizes matrix updates using a quintic Newton-Schulz iteration called Polar Express.

# Polar Express coefficients (5 iterations)
polar_express_coeffs = [
    (8.156, -22.483, 15.879),
    (4.043, -2.809, 0.500),
    # ... more coefficients
]

# Orthogonalization loop
for a, b, c in polar_express_coeffs[:ns_steps]:
    A = X.mT @ X
    B = b * A + c * (A @ A)
    X = a * X + X @ B

NorMuon variance reduction

After orthogonalization, nanochat normalizes updates per neuron to avoid scale collapse:

v_mean = g.float().square().mean(dim=red_dim, keepdim=True)
v_norm = v_mean.sum(dim=(-2, -1), keepdim=True).sqrt()
final_scale = step_size * (v_norm / v_norm_new.clamp_min(1e-10))
g = g * final_scale.to(g.dtype)

Multi-GPU communication

For distributed runs, the optimizer uses ZeRO-2-style sharding with asynchronous communication:

1. Launch asynchronous reduce_scatter operations.
2. Wait for reductions, update weights, then launch all_gather operations.
3. Wait for gathers and copy updated parameters back.

The sequence is designed to overlap communication and computation.

3. Precision management: nanochat/common.py

nanochat manages compute precision explicitly rather than relying on torch.amp.autocast.

Hardware	Default dtype	Reason
CUDA SM 80+ such as A100 and H100	bfloat16	Native BF16 tensor cores
CUDA SM below 80 such as V100 and T4	float32	No BF16 support
CPU and MPS	float32	No reduced-precision tensor cores

The custom linear layer casts the weight to the input dtype during the forward pass:

class Linear(nn.Linear):
    def forward(self, x):
        return F.linear(x, self.weight.to(dtype=x.dtype))

Master weights remain in FP32 for optimizer precision. On H100 and Blackwell hardware, you can enable FP8 training with --fp8.

4. Data loader: nanochat/dataloader.py

The dataloader uses BOS-aligned best-fit packing:

• Every row starts with a BOS token.
• Documents are packed with a best-fit algorithm.
• When no complete document fits, one document is cropped to fill the remaining space.
• At sequence length 2048, this targets 100% utilization with approximately 35% token cropping.

The document-selection logic looks like this:

# Find largest document that fits entirely
best_idx = -1
best_len = 0

for i, doc in enumerate(doc_buffer):
    doc_len = len(doc)
    if doc_len <= remaining and doc_len > best_len:
        best_idx = i
        best_len = doc_len

if best_idx >= 0:
    doc = doc_buffer.pop(best_idx)
    # Add full document
else:
    # Crop shortest doc to fill remaining space

This design ensures tokens can attend back to a BOS token while preserving as much complete-document context as possible.

5. Flash Attention abstraction: nanochat/flash_attention.py

nanochat provides a shared attention interface that chooses the best available backend.

from nanochat.flash_attention import flash_attn

# Auto-selects the best backend for the current hardware
y = flash_attn.flash_attn_func(
    q,
    k,
    v,
    causal=True,
    window_size=window_size,
)

On Hopper GPUs with BF16, it uses Flash Attention 3. On other hardware, it falls back to PyTorch scaled dot-product attention.

6. Inference engine: nanochat/engine.py

The Engine class handles generation features such as:

• KV-cache prefill for prompts.
• flash_attn_with_kvcache when FA3 is available.
• Batch generation by cloning the KV cache.
• Calculator tool use triggered by special tokens.

The engine also controls conversation flow when the model invokes the calculator.

Step-by-step: train your own model

The complete reference pipeline lives in runs/speedrun.sh. The commands below break it into stages so you can run and inspect each part independently.

Prerequisites

Before starting, prepare:

• An 8×H100 GPU node, or similar hardware.
• Approximately 20 GB of disk space for the dataset.
• Python 3.10 or newer.
• The uv package manager.

You can still run nanochat on fewer GPUs, but total training time increases.

Step 1: Set up the environment

Install uv, create a virtual environment, and install GPU dependencies.

# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh

# Create and activate a virtual environment
uv venv
source .venv/bin/activate

# Install dependencies
uv sync --extra gpu

Step 2: Download pretraining data

Download approximately 2B characters from the ClimbMix dataset.

python -m nanochat.dataset -n 170

This downloads approximately 170 shards of about 100 MB each:

• Total size: approximately 17 GB compressed.
• Data loader behavior: uses file locking to coordinate multiple ranks.

Step 3: Train and evaluate the tokenizer

Train the 32,768-token BPE tokenizer:

python -m scripts.tok_train

Then evaluate tokenizer compression:

python -m scripts.tok_eval

The tokenizer uses a GPT-4-style split pattern with byte-fallback BPE. Training takes approximately 10 minutes on 2B characters.

Step 4: Pretrain a GPT-2-capability model

Use a d24 model for the GPT-2-capability configuration:

torchrun --standalone --nproc_per_node=8 -m scripts.base_train -- \
  --depth=24 \
  --target-param-data-ratio=8 \
  --device-batch-size=16 \
  --fp8 \
  --run=my-first-model

Key options:

Option	Meaning
--depth=24	GPT-2-size configuration
--target-param-data-ratio=8	Slightly undertrains for speed
--device-batch-size=16	Per-GPU batch size
--fp8	Enables FP8 training on H100-class hardware
--run=my-first-model	Names the run for tracking and checkpoints

Expected runtime: approximately two hours on the reference 8×H100 setup.

Step 5: Run supervised fine-tuning

Download the identity conversation dataset:

curl -L -o ~/.cache/nanochat/identity_conversations.jsonl \
  https://karpathy-public.s3.us-west-2.amazonaws.com/identity_conversations.jsonl

Run SFT to teach chat formatting, special tokens, and tool use:

torchrun --standalone --nproc_per_node=8 -m scripts.chat_sft -- \
  --device-batch-size=16 \
  --run=my-sft

Step 6: Chat with the model

Use the command-line client:

python -m scripts.chat_cli -p "Why is the sky blue?"

Or start the web interface:

python -m scripts.chat_web

The web UI runs on port 8000.

Research workflow: run fast experiments first

Do not start architecture research with d24 or d26 runs. Use d12 as a quick validation target first.

Run a small experiment

The following command disables frequent evaluation, samples, and checkpoints to minimize overhead:

OMP_NUM_THREADS=1 torchrun --standalone --nproc_per_node=8 -m scripts.base_train -- \
  --depth=12 \
  --run="d12-test" \
  --core-metric-every=999999 \
  --sample-every=-1 \
  --save-every=-1

A d12 run is suitable for testing changes such as:

• An alternative residual connection.
• A modified attention pattern.
• A new activation function.
• A different token-mixing mechanism.
• Optimizer changes.

Monitor these metrics

Track the following metrics in Weights & Biases:

Metric	What it indicates
val_bpb	Validation bits-per-byte; a vocabulary-size-independent loss metric
core_metric	DCLM CORE evaluation score
train/mfu	Model FLOPS utilization
train/tok_per_sec	Training throughput

Validate across depths

A change that helps only one model size may be overfit to that configuration. Validate promising changes across d12 through d26 before treating them as a general improvement.

Why nanochat matters

Lower training cost

Approach	Cost	Time	Hardware
OpenAI GPT-2, 2019	$43,000	168 hours	32 TPU v3
nanochat, 2026	$48	2 hours	8×H100
nanochat on spot instances	Approximately $15	2 hours	8×H100 spot

This makes full-pipeline LLM training more accessible to:

• Individual researchers.
• Small startups.
• University courses.
• Hobbyists.

Educational value

nanochat is useful as a code-reading project because it includes:

• Roughly 500 lines for the GPT model.
• Roughly 530 lines for the optimizer.
• Comments describing design decisions.
• Minimal configuration indirection.

You can inspect a complete LLM pipeline, make a change, and measure the result without navigating a large framework.

Faster research cycles

Reducing model training from days to hours enables:

• Faster hypothesis testing.
• More experiments per week.
• Lower cost of failed ideas.
• Easier community comparison through the leaderboard.

Transparent implementation choices

The project documents design decisions through:

• Scaling-law notes in dev/LOG.md.
• Ablation studies in GitHub Discussions.
• Reproduction details for leaderboard entries.
• Disclosure of AI-assisted contributions.

Limitations and reality check

nanochat is an effective training and education tool, but it has important constraints.

Hardware requirements

The approximate $48 figure assumes access to an 8×H100 node. Actual rental prices vary:

• Lambda Labs: approximately $25/hour for 8×H100.
• RunPod: approximately $15/hour for spot pricing.
• Full workflow: approximately two hours of pretraining plus SFT time.

Budget approximately $50–$100 for a full run, depending on the provider and availability.

Capability ceiling

nanochat targets GPT-2-level performance, not current frontier-model capability.

It can support:

• Basic conversation.
• Simple reasoning.
• Elementary math.
• Limited factual recall.

It is not expected to handle:

• Complex multi-step reasoning.
• Code generation beyond simple functions.
• Nuanced instruction following.
• Tasks competitive with GPT-4, Claude, or Gemini.

Treat it as a compact research model and learning environment rather than a drop-in production replacement for a frontier API.

Data requirements

The full speedrun downloads:

• Approximately 170 shards.
• Approximately 17 GB compressed.
• Approximately 2B characters total.

Plan for sufficient local storage and network bandwidth before starting a run.

Metric limitations

CORE covers 22 tasks, but it does not fully measure:

• Real-world conversation quality.
• Domain-specific knowledge.
• Fine-grained instruction following.
• Safety and alignment.

Random seeds can produce approximately 0.016 CORE variance, so do not interpret small score changes as definitive without repeated runs.

FAQ

How much does nanochat training cost?

Pretraining costs approximately $48 on demand, based on $24/hour for two hours, or approximately $15 on spot instances. Add about 30 minutes for SFT.

What GPU do I need?

The code can run on a single modern datacenter GPU. The reference setup is 8×H100, with 8×A100 also suitable for fast training. nanochat can scale from one to eight GPUs using automatic gradient accumulation.

How long does training take?

The expected range is 1.65 to 3 hours depending on configuration and hardware. The current d24 leaderboard record is 1.65 hours.

What is the CORE metric?

DCLM CORE evaluates models on 22 tasks, including ARC, MMLU, and other benchmarks. GPT-2 scored 0.256525; nanochat runs regularly exceed 0.26.

Can I train on one GPU?

Yes. Omit torchrun; the code uses gradient accumulation automatically. Training takes roughly eight times longer but should produce nearly identical results.

Which dataset does nanochat use?

The current best result uses ClimbMix, NVIDIA’s curated web dataset. Previous versions used FineWeb-EDU. The tokenizer is trained on approximately 2B characters from the first approximately eight shards.

Does nanochat work on Apple Silicon?

Yes. nanochat runs on MPS using float32. It is slower than CUDA but useful for local experimentation.

Can I resume from a checkpoint?

Yes. Use --resume-from-step=<step> to continue from a saved checkpoint. The dataloader state is also saved for exact resumption.

How does nanochat differ from nanoGPT?

nanoGPT focuses on pretraining. nanochat extends the workflow to tokenization, pretraining, SFT, RLHF, evaluation, inference, and a web UI.

Conclusion

nanochat demonstrates that GPT-2-level LLM training can be run as a practical engineering project rather than a large research program.

Its value is not only the reduced cost. The project provides a compact, inspectable stack where you can trace the path from raw training data to tokenizer, base model, chat fine-tuning, evaluation, and inference.

Key takeaways

• Cost reduction: approximately $43,000 to approximately $48 for GPT-2-level capability.
• Speed improvement: 168 hours to as little as 1.65 hours.
• Simple configuration: --depth drives the main training recipe.
• Complete workflow: tokenizer through web UI.
• Fast experimentation: use d12 runs before validating on larger depths.

Next steps

Start with the nanochat repository and inspect runs/speedrun.sh. Run a d12 experiment first, confirm that your environment and metrics work, then move to d24 when you need GPT-2-level capability.

For developers building AI-powered applications, the barrier to understanding and experimenting with LLM training has shifted from large infrastructure investment to a focused engineering project.