Warm vs Cold: Why a Single Trial Misleads Performance Claims

Why this lesson exists

Performance claims that rely on a single run of a binary are like a snapshot of a moving train – you see a moment, but you miss the whole journey. In the first episode we warned that a one‑shot timing can be wildly off because of warm‑up effects, OS noise, or just a lucky cache state.

This episode digs into the warm‑up trap and shows how csperf turns a single measurement into a statistically sound artifact.

Recap — where we are in the series

Ep 1: Why a Single ./a.out Time Misleads Your Performance Claims – we saw that a single run can be 10‑30 % off the true steady‑state performance and that reproducibility is impossible without a repeatable protocol.

Ep 2: Warm vs Cold: Why a Single Trial Misleads Performance Claims – we are now looking at how to structure a run so that the first few executions warm the cache, branch predictor, and other micro‑architectural state, and how to capture the resulting data.

The misconception

Many developers think “run once, read the number, publish.” The problem is that the first execution of a program after compilation is usually cold: the code is fetched from disk, the instruction cache is empty, the data cache is cold, and the CPU’s micro‑architectural state (branch predictor, prefetcher, etc.) is uninitialized. Subsequent runs see a different performance profile.

A single measurement cannot distinguish between a warm run that reflects the steady‑state performance and a cold run that is an outlier.

What problem csperf solves (this episode's slice)

csperf automates the warm‑up / repeat pattern and stores the raw data in a JSON artifact. The artifact contains:

• the machine metadata (CPU, OS, compiler, etc.)
• the command line used for compilation and execution
• a list of execution times for each repeat
• summary statistics (min, max, mean, stdev)
• a full set of hardware counters collected by perf or papi

With this data you can:

• prove that your performance claim is reproducible
• compare different compiler flags or backends
• detect regressions in a CI pipeline

Mental model

Think of a performance run as a warm‑up phase followed by a steady‑state phase.

┌───────────────────────┐
│  Warm‑up runs (2)      │
│  (discarded)           │
└─────────────┬─────────┘
              │
              ▼
┌───────────────────────┐
│  Repeat runs (5)       │
│  (kept for analysis)   │
└───────────────────────┘

The warm‑up runs are executed to bring the program into a stable state. They are not part of the final statistics. The repeat runs are what you publish.

Lab: install and first commands

1. Install csperf (if you haven’t already):

   pip install csperf

1. Compile and run a simple matrix traversal (the same example used in Ep 1):

   csperf run \
     --input examples/cpp/matrix_traversal.cpp \
     --backend cpu \
     --warmup-runs 2 \
     --repeat-runs 5 \
     --output results/warm.json

This command:

• compiles the C++ file with clang++ -O3
• runs the binary twice as warm‑up (discarded)
• runs it five times, recording each execution time
• writes a JSON artifact to results/warm.json
  1. Profile the artifact to see the raw numbers:

   csperf profile results/warm.json

The profile command prints a concise table of the summary statistics and the raw per‑run times.

Lab: what we ran on this machine

Item	Value
Hostname	f4c59d864117
Date	2026‑09‑26T19:21:22+05:30
CPU	AMD Ryzen 7 9700X (8‑core, 16‑thread)
OS	Ubuntu 24.04 (Linux‑7.0.0‑31‑generic)
Compiler	clang++ (LLVM 18)
Backend	CPU (native)
Warm‑up runs	2
Repeat runs	5
Artifact	results/warm.json

The machine metadata is embedded in the artifact; you can inspect it with csperf profile or by opening the JSON file.

Results (real numbers only)

The csperf run produced the following execution times (in milliseconds) for the five repeat runs:

Run	Time (ms)
1	5.151
2	5.165
3	5.163
4	5.182
5	5.165

Summary statistics (from the artifact):

• Count: 5
• Min: 5.151 ms
• Max: 5.182 ms
• Mean: 5.1634 ms
• Median: 5.165 ms
• Standard Deviation: 0.0127 ms

Hardware counters (subset):

Counter	Value
CPU cycles	5 407 786
Reference cycles	6 685 659
Instruction count	20 043 266
Branch instructions	3 125 327
Branch mispredictions	35 076
Cache references	1 243 626
Cache misses	85 371
IPC	3.706

These numbers come straight from csperf/warm.json and are verifiable.

How to read the artifacts

The JSON artifact is a self‑contained record of the experiment. Key sections:

• pipeline – shows the compiler steps and the profiler used.
• commands – the exact shell commands that were executed.
• metrics – raw measurements and summary statistics.
• artifacts – paths to the binary and any exported CSV/Excel files.

You can use csperf profile to pretty‑print the summary, or csperf export --format csv results/warm.json to get a CSV for spreadsheet analysis.

Common mistakes (teacher checklist)

1. Skipping warm‑up runs – always set --warmup-runs to at least 1.
2. Using a single repeat run – set --repeat-runs to ≥ 5 for a stable mean.
3. Ignoring the artifact – publish the JSON, not just the printed numbers.
4. Running on a shared machine – background processes can skew the results; use a dedicated test rig.
5. Not pinning the binary to a CPU – csperf can set CPU affinity; otherwise the OS may migrate the process.

Try this next (homework)

Run the same experiment on a different compiler backend (e.g., gcc -O3) and compare the mean execution time and IPC. Document the differences in a short markdown file.

Do this tonight — Episode 3 starts by assuming you did.

Closing

The warm‑up / repeat pattern is the foundation of any credible performance measurement. csperf removes the guesswork and gives you a machine‑aware, repeatable artifact that anyone can audit.

The series so far

• Ep 1: Why a Single ./a.out Time Misleads Your Performance Claims (this article)
• Ep 2: Warm vs Cold: Why a Single Trial Misleads Performance Claims

Teaser for next episode

This episode builds on Ep 1’s warning about single‑shot timings and sets the stage for next time when we show why screenshots aren’t evidence and how to keep your artifacts in a reproducible repository.