How much does it cost to train an LLM from scratch?

Training an LLM from scratch costs from a few hundred dollars for a small model to tens of millions for a frontier model, and almost all of it is GPU-hours times the hourly rate. At the $2.60/hr Nodus lists for an NVIDIA H100 on its pricing page, a compute-optimal 1B model costs about $292 and a 7B model about $14,300 in compute. The rest of this post shows the formula, checks it against what Meta and DeepSeek actually published, and lists the costs the headline numbers leave out.

How do you estimate what an LLM training run will cost?

You need three numbers: model size, training tokens and how much of the GPU's peak speed you will really get. Training a transformer takes about 6 floating-point operations per parameter per token, the rule from Kaplan et al.'s scaling-laws paper. So a run needs roughly 6 × parameters × tokens operations in total.

An NVIDIA H100 SXM peaks at about 989 teraFLOPS of dense BF16 math. NVIDIA's H100 page lists 1,979 teraFLOPS, but that figure assumes sparsity, so halve it for normal training. No real run gets the full peak. The share you do get is called utilization, and it is the number that moves your budget most.

Put together, the estimate is:

GPU-hours = (6 × parameters × tokens) / (989e12 × utilization × 3,600)
cost      = GPU-hours × hourly rate

At 30% utilization, one H100-hour delivers about 1.07 × 10^18 useful operations. Every cost in this post comes from that line and a rate.

How many tokens should a model train on?

A common starting point is about 20 tokens per parameter. DeepMind's Chinchilla paper found that model size and training tokens should grow together for the best model at a fixed compute budget, which works out to roughly that ratio: Chinchilla itself paired 70 billion parameters with 1.4 trillion tokens. A 1B model gets 20 billion tokens and a 7B model gets 140 billion.

Most open models now train far past that point. Meta trained every Llama 3.1 model on about 15 trillion tokens, which is over 1,800 tokens per parameter for the 8B model. Extra tokens cost more to train but give a better small model that is cheaper to serve. That choice can multiply your bill by ten or more, so decide it before you price anything.

What did real LLM training runs actually cost?

Published GPU-hour counts are the best reality check, and they run from about 1.5 million to over 30 million for recent open models. Meta's Llama 3.1 model card lists the H100-80GB hours for each size. DeepSeek's V3 technical report lists 2.788 million H800 GPU-hours for its full training, which it prices at $5.576 million assuming $2 per GPU-hour.

Model	Published GPU-hours	At $2.60/GPU-hour
Llama 3.1 8B	1.46M H100	$3.80M
Llama 3.1 70B	7M H100	$18.2M
Llama 3.1 405B	30.84M H100	$80.2M
DeepSeek-V3 (37B active)	2.788M H800	$7.25M

GPU-hours from the Llama 3.1 model card and the DeepSeek-V3 report; Nodus H100 list rate checked October 30, 2026. The last column is for scale only: runs this size need thousands of GPUs held for weeks, which is a reserved-capacity contract, not an hourly bill.

DeepSeek-V3 used fewer GPU-hours than Llama 3.1 70B because only 37B of its parameters are active per token.

DeepSeek-V3 is a mixture-of-experts model, so each token only touches 37 billion of its parameters. That is why its compute bill sits between the 8B and 70B dense Llama models even though it is much larger in total.

What utilization should you plan for?

Plan for 30% unless you have measured better on your own code. We back-solved the effective utilization of each published run from its parameters, tokens and GPU-hours, using the formula above. The results were about 14% for Llama 3.1 8B, 25% for Llama 3.1 70B and 33% for Llama 3.1 405B.

The best small-scale result we found is higher. Andrej Karpathy's llm.c reproduction of the 1.6B GPT-2 trained on 33.6 billion tokens in 24 hours on one 8 × H100 node. That is 192 GPU-hours, which back-solves to about 46% utilization, close to the roughly 50% he reports.

The same formula on four real runs: plan for 30% and treat 45% as a well-tuned best case.

The gap between running speed and the full bill is the useful part. The Llama 3 paper reports 38% to 43% utilization while the 405B model was training, yet the whole GPU-hour count back-solves to 33%. Over a 54-day stretch of that run, Meta logged 466 job interruptions, 419 of them unexpected. Restarts, redone work and idle time all land on the bill, and that is why we plan at 30%, not at peak.

How much does it cost to train a small LLM from scratch?

A small model is cheap enough to train for learning or for a narrow domain. Here is the formula at 30% utilization and the Nodus H100 list rate of $2.60/hr.

Model and tokens	Total operations	H100 GPU-hours	Compute cost
1B on 20B tokens	1.2 × 10^20	112	$292
7B on 140B tokens	5.9 × 10^21	5,505	$14,313
7B on 2T tokens	8.4 × 10^22	78,643	$204,471
70B on 1.4T tokens	5.9 × 10^23	550,500	$1.43M

Our arithmetic from 6 × parameters × tokens, 989 teraFLOPS dense BF16, 30% utilization and $2.60 per H100-hour (checked October 30, 2026).

To check the method against a real bill, take the llm.c GPT-2 run. Karpathy priced its 192 H100-hours at $672, which is $3.50 per GPU-hour. At $2.60, the same 192 GPU-hours cost $499.20. If your code reaches 45% instead of 30%, every row above falls by a third.

What costs do these numbers leave out?

The formula prices one clean run, and real projects are never one clean run. DeepSeek says plainly that its $5.576 million covers only the official training run and excludes prior research and ablation experiments on architecture, algorithms and data. Expect to spend real money on small test runs before the big one.

Four other costs show up on most projects:

• Lost work after an interruption. If an 8-node, 64-GPU run loses 6 hours of progress, that is 384 GPU-hours, or $998.40 at $2.60. With a checkpoint no more than 30 minutes old, the same failure costs 32 GPU-hours, or $83.20.
• Idle GPUs. A node that sits waiting for data, a fix or a person bills the same as one that trains.
• Data preparation and evaluation. Cleaning and tokenizing a large corpus and running benchmarks takes its own compute.
• Storage for checkpoints and datasets, which grows with model size.

The first two are the ones you control with tooling. That is where most of the gap between the formula and the final invoice comes from.

Should you train from scratch at all?

Usually not. Train from scratch only when you need your own architecture or tokenizer, a model with no inherited weights, or research results about pretraining itself. For most teams, continuing to pretrain an open model on your own text, or fine-tuning it, gets a better model for a small fraction of the cost.

Our own continued-pretraining benchmark shows the scale difference. Continuing to train Qwen2.5-0.5B on raw text on one NVIDIA L4 cut held-out perplexity by 21.8% for a recorded charge of $0.121164. That was a small pilot on a tiny model. Still, it shows the order of magnitude: well under a dollar to adapt a small model, against hundreds to train even a 1B model from nothing. If you are fine-tuning instead, the cheapest way to fine-tune Llama on an H100 walks through that cost.

How do you run a from-scratch pretraining job on Nodus?

Nodus runs pretraining as a TrainingJob with task: Pretrain and initialization: Scratch, which keeps only the model's architecture and tokenizer. This manifest follows the training guide, with a spending cap and two 8-GPU nodes added:

apiVersion: nodus.dev/v1beta1
kind: TrainingJob
metadata:
  name: scratch-pretrain
spec:
  runtime: nodus/sft
  task: Pretrain
  model: {uri: hf://Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca}
  data: {volume: corpus, format: JSONL}
  parameters: {steps: 10000, initialization: Scratch, packing: true}
  resources: {gpu: {type: [h100-sxm-80g], count: 8}, nodes: 2}
  maxCostUSD: "500.00"

Price it before anything runs:

$ nodus apply -f scratch-pretrain.yaml --dry-run -o yaml
$ nodus apply -f scratch-pretrain.yaml

The dry run's estimate is the number of steps times the measured seconds per step on the slowest GPU type you allowed. maxCostUSD caps the whole run, including retries; when it is reached the run is suspended, not failed, and raising the cap resumes it from its checkpoint. If a node is lost, the gang restarts from the latest checkpoint every rank committed, up to three times. Multi-node TrainingJobs are in Beta and go up to 8 nodes, as the multi-node guide explains.

FAQ

How much did it cost to train GPT-4?

OpenAI has not published GPT-4's training compute or cost, so any figure you see is an outside estimate. The best public anchors are open models that publish GPU-hours, such as Llama 3.1 405B at 30.84 million H100-hours.

Can you train an LLM on one GPU?

Yes, for small models. Karpathy notes the 1.6B GPT-2 run would take about 8 days on one GPU instead of 1 day on eight, for the same GPU-hours. Nodus lists prices per GPU-hour, so the GPU-hours, not the GPU count, set the compute cost.

Is a B200 cheaper than an H100 for pretraining?

Only if it finishes the job more than proportionally faster. Nodus lists the NVIDIA B200 at $5.50/hr against $2.60/hr for the H100, so the B200 needs over 2.1 times the H100's throughput on your model to cost less per run. Measure seconds per step on both before you commit.

Does a mixture-of-experts model cost less to train?

Per token, yes, because compute scales with active parameters, not total ones. DeepSeek-V3 has 37B active parameters and needed 2.788 million GPU-hours, fewer than the dense Llama 3.1 70B at 7.0 million. You still need enough GPU memory to hold all the experts.

Originally published on the Nodus blog.