DEV Community: Thokozani Buthelezi

DDP Is Not Always Faster

Thokozani Buthelezi — Sat, 16 May 2026 12:11:57 +0000

That is the result of this experiment, and it is the most important thing to understand about distributed training before you reach for it.

I ran my nanoGPT implementation, a 4M parameter character-level transformer across two T4 GPUs on Kaggle using PyTorch's DistributedDataParallel. The single GPU baseline ran at 22ms per step. The DDP run across two GPUs ran at 26ms per step. Adding a second GPU made training slower.

How DDP Works

DDP splits each batch across GPUs, each GPU sees a different subset of the data via DistributedSampler. Each GPU runs its own forward and backward pass independently. After backward, DDP runs an all-reduce operation that synchronizes gradients across all GPUs before the optimizer step. Every process ends up with identical gradients and takes an identical optimizer step.

The key line in the code is one wrapping call:

model = DDP(model, device_ids=[rank])

Everything else, the model definition, the optimizer, the training loop stays the same. DDP handles the gradient synchronization automatically via hooks registered on each parameter.

The Numbers

Metric	Value
Single GPU step time	22.03ms
DDP step time (2 GPUs)	26.36ms
Compute time per step	15.12ms
Communication time per step	11.24ms
Communication overhead	42.6%
Scaling efficiency	41.8%

Scaling efficiency measures how close you got to ideal linear speedup. At 100% efficiency, two GPUs would halve your step time to 11ms. At 41.8%, the DDP run is actually slower than single GPU.

Why Scaling Efficiency Was Low

42.6% of every DDP step was gradient communication, not compute. The all-reduce has to move every gradient across the PCIe bus connecting the two T4s, and at 4M parameters that communication cost dominates.

This is a compute-to-communication ratio problem. DDP only pays off when the model is large enough that compute time swamps communication time. At 4M parameters on T4s without NVLink, the ratio is inverted, you spend more time talking between GPUs than doing actual work.

The same experiment on a 1B parameter model would look completely different. Compute would dominate, and scaling efficiency would climb toward 80-90%.

What I Would Do Differently

The right experiment for demonstrating DDP scaling is a model at least one order of magnitude larger, or running across more than two GPUs where the communication overhead amortizes better. Weeks 13-14 exposed the constraint rather than the benefit, which is a legitimate result, knowing when not to use DDP is as useful as knowing how to use it.

Phase III (FSDP) addresses this directly: instead of replicating the full model on every GPU, FSDP shards parameters across GPUs, which changes the communication pattern and makes large model training viable.

Code and results committed to distributed_data_parallel in the monorepo.
https://github.com/Thoki-Buthelezi/elite-ai-systems-engineer-2026

Evaluating LLMs for Under a Dollar

Thokozani Buthelezi — Thu, 14 May 2026 13:39:16 +0000

Why Evals Matter

Training a model is only half the job. Without a systematic way to measure what it can actually do, you are flying blind. The problem is that evaluation is easy to do badly, you can run a benchmark, get a number, and walk away thinking you know something when you don't.

This post is about doing it properly on a budget. I ran three standard benchmarks against Qwen2.5-0.5B on a free Colab T4, logged wall-clock time and dollar cost for each task, and documented every methodological decision along the way. Total spend: $0.1185.

The Benchmarks

I picked three tasks that cover meaningfully different capabilities rather than variations of the same thing.

GSM8K (Cobbe et al., 2021) tests grade-school math reasoning. The model has to produce a chain-of-thought and arrive at a final numeric answer. Scoring is exact match, either the answer is right or it isn't. This is a generative task, which makes it slower and more expensive than the others. I used 5-shot prompting following the original paper.

HellaSwag (Zellers et al., 2019) tests commonsense sentence completion. Given a partial sentence, the model scores four candidate continuations using normalized log-likelihood and picks the highest. The dataset was constructed with adversarial filtering, meaning the wrong answers were specifically chosen to fool models that rely on surface-level patterns. Human performance is around 95%. I used 10-shot following the original paper.

TruthfulQA-MC2 (Lin et al., 2021) tests whether the model produces truthful answers to questions that commonly elicit false beliefs. I used the MC2 variant multiple choice scored by log-likelihood, rather than the generative version, which requires a GPT-4 judge model. This keeps the eval fully self-contained and free. 0-shot, following the original paper.

The Harness

All three tasks were run through lm-evaluation-harness by EleutherAI. The harness standardizes few-shot prompt construction, normalization, and metric computation across tasks, which matters a lot for reproducibility. Running the same eval twice should give the same number.

One non-obvious decision: GSM8K in the harness defaults to max_gen_toks=2048, which generates up to 2048 tokens per sample. On a T4 that was running over 4 hours. I capped it at 256 tokens and included a limit=0.25 which runs 25% of the test set. I figured this is enough to capture a complete chain-of-thought for grade-school math and brings runtime down to under 50 minutes.

The Model

Qwen2.5-0.5B is a 500M parameter base model from Alibaba. I chose it because it fits comfortably in the 15GB VRAM on a free Colab T4 and is fast enough to run all three benchmarks in a single session. Being a base model rather than an instruction-tuned one is worth noting, the experiment primarily reflects runtime, generation behaviour, and evaluation cost characteristics of the base model under standard benchmark workloads.

Cost Accounting

Cost basis: Colab Pro at approximately $0.10/hr for a T4 session.

Task	Time	Cost
GSM8K	46.52 min	$0.0775
HellaSwag	23.67 min	$0.0394
TruthfulQA-MC2	0.97 min	$0.0016
Total	71.16 min	$.1185

Published Scores

Free Colab T4 sessions kept disconnecting mid-run — GSM8K's generative evaluation took 46+ minutes per run, and the session limit hit before clean results could be saved. Rather than spending another week on infrastructure, I'm using the official numbers from the Qwen2.5 technical report, which uses the same lm-evaluation-harness with matching few-shot settings.
The runtime and cost figures are from the actual runs. Scores are from the Qwen2.5 technical report (Qwen Team, 2025), cited explicitly because my runs had the extraction bug.
Total cost: $0.1185 across all three tasks

Task	Metric	Score	Duration (min)	Cost ($)
HellaSwag(10-shot)	acc_norm	0.521	23.67	$0.0394
TruthfulQA(0-shot)	mc2	0.402	0.97	$0.0016
GSM8k (5-shot, 25% subset)	exact_match	0.416	46.52	$0.0775

Limitations

A few things worth being honest about before drawing conclusions from these numbers.

Contamination. Qwen's training data composition is not fully disclosed. Any of these benchmarks could have been in the pretraining mix, which would inflate scores. There is no way to verify this from the outside.

Exact match undercounts GSM8K. A model that produces the right reasoning but formats the final answer differently, writing "42 dollars" instead of "42", gets marked wrong. The real accuracy is likely slightly higher than the number reported.

Prompt sensitivity. Benchmark scores can shift meaningfully with different few-shot examples or prompt formatting. The numbers here are specific to the default harness prompt templates.

What I Would Do Differently

Running a single model against three benchmarks gives you a snapshot, not a story. The more interesting experiment is running the same benchmarks against multiple checkpoints say, the base model, a LoRA fine-tune, and a DPO fine-tune and measuring the delta. That is what weeks 13+ will set up.

Results and notebook committed to lm_eval_harness in my github.
https://github.com/Thoki-Buthelezi/elite-ai-systems-engineer-2026

Reproducing Chinchilla Scaling on a Budget

Thokozani Buthelezi — Sat, 02 May 2026 11:58:58 +0000

Training a 70B parameter model costs millions of dollars. Scaling laws exist so you don't have to guess how to spend that budget. Here's what I learned reproducing them on a free GPU.

Introduction

Scaling laws are basically rules that tell us how model performance improves as you increase quantities such as model size, dataset size, and compute.

Instead of guessing "bigger models = better", scaling laws gives a mathematical relationship between:

model size (N, number of parameters)
dataset size (D, number of tokens)
compute (C, number of training FLOPs)
loss (L, how wrong the model is)

the core idea

\frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}} + E

This looks intimidating but it's simple:

increasing N(model size) -> loss goes down
increasing D(data) -> loss goes down
but both have diminishing returns because of the scaling exponents (α,β)
where E is the irreducible entropy error of the model

The relationship between the loss and these quantities is not linear, it is a power law.

The Kaplan vs Chinchilla disagreement

Kaplan said scale model size faster than dataset size
Chinchilla said scale both equally

why they disagreed?
The three experimental assumptions used by Kaplan led to conclusions that model size should be scaled faster. These assumptions include:

the use of non-embedding parameters only when scaling
undertraining of large models
omission of the offset term in the compute-loss form

When these factors are corrected by Chinchilla you have:

the use of all model's parameters
models are fully trained to compute-optimal point
the offset term is included in the compute-loss form

one clean takeaway

Kaplan didn’t “get it wrong”, the setup just made model scaling look more effective than it actually is.
Chinchilla corrected the setup, and revealed the true balance.

The experiment

In my experiment to reconstruct the Chinchilla scaling, I used 3 models of different parameter sizes: 786K, 4M, 25M params on the same dataset WikiText-2, for the same compute budget. I trained all three models for 500 steps using a T4 GPU on Google Colab. At every 50 steps I logged the validation loss and total FLOPs consumed. Here's what the data showed.

Graph 1: Validation Loss vs Training Steps

This graph shows what happens as you let the models practice over time.

size matters immediately: even at the very first step, the larger model(green) starts with a much lower error than the small model(blue)
the "Head Start" effect: the large model's starting point is actually better than the small model's finishing point. This shows that having more parameters makes inherently more capable.
plateauing: all the three lines curve and flatten out. This represents the diminishing returns, that is, the longer you train a model, the harder it becomes to extract extra accuracy from it.

Graph 2: Loss vs Compute (Log-Log Scale)

This is the "Power Law" graph. By plotting the data on a log-log scale, the curves become straight lines.

predictable progress: because these lines are straight, researchers can look at the small model's slope and mathematically predict exactly how much more compute they need to reach a specific performance level.
efficiency gains: notice how the green dots(large model) extend further to the right. To get the lowest loss on the chart, you must use the large model; the small model simply doesn't have the capacity to get that "smart", no matter how much compute you throw at it.
the slope (-α): the legend show "slopes" like -0.136. This is the scaling exponent that tells us the "exchange rate" between spending more money on GPUs and getting a smarter AI.

The Big Picture
Together, these graphs prove scaling isn't random. If you want a smarter AI you don't just guess, you use these straight lines to calculate exactly how many parameters and how much compute you need to reach your goal.

Full code and results are on my GitHub: https://github.com/Thoki-Buthelezi/elite-ai-systems-engineer-2026

What Next: I'll be running lm-evaluation-harness across all three model sizes and analysing what benchmarks like HellaSwag and GSM8K actually measure and where they mislead.