Why a Single ./a.out Time Misleads Your Performance Claims

Why this lesson exists

Single‑shot timings from time ./a.out or screenshots are seductive but fragile. They ignore machine state, compiler flags, and the stochastic nature of modern CPUs. A single number cannot be shared, diffed, or reproduced reliably.

What we're building toward

We want a reproducible, machine‑aware evidence trail that anyone can inspect, compare, and trust. csperf is that observatory.

The misconception

People think “my program runs in 5 ms” is enough proof. In reality, that 5 ms hides cache warm‑ups, branch predictor state, and even the current frequency governor.

What problem csperf solves (this episode's slice)

csperf turns a raw run into a structured artifact: a CSV, JSON, and optional Excel sheet that record the machine, compiler, and timing statistics. It also runs a sanity check (csperf doctor) to make sure the environment is ready.

Mental model

Think of a single time call as a photograph. csperf is a full‑body scan that records every relevant metric and the context that produced it.

Lab: install and first commands

pip install compilersutra-perf
csperf doctor
csperf quickstart --output-dir results/quickstart

The quickstart builds matrix_traversal.cpp with clang++ -O3, runs it three times, and writes the artifacts under results/quickstart.

Lab: what we ran on this machine

• Host: f4c59d864117
• OS: Ubuntu 24.04 (Linux 7.0.0‑31‑generic)
• CPU: AMD Ryzen 7 9700X, 16 cores, 5582 MHz max
• Compiler: clang++ 18.1.3
• csperf version: 0.2.0

Results (real numbers only)

Trial	Time (ms)
1	5.162
2	5.144
3	5.148
Mean	5.151333
Std dev	0.009452

All numbers come from results/quickstart/quickstart.csv.

How to read the artifacts

• CSV – human‑readable table of trials and summary stats.
• JSON – machine‑friendly representation, useful for CI pipelines.
• Excel – for quick visual inspection.

Each artifact includes a machine.txt snapshot that records the exact CPU model, clock, and available counters.

Common mistakes (teacher checklist)

1. Relying on a single time invocation.
2. Forgetting to run csperf doctor before measuring.
3. Ignoring the runs field; 1‑shot runs are not statistically sound.
4. Not checking the cpu_model in the artifact to ensure consistency.

Try this next (homework)

Do this tonight — Episode 2 starts by assuming you did.

Closing

A single ./a.out time is a myth. csperf gives you a verifiable, repeatable evidence trail. Next time, we’ll explore why warm‑up and repeat runs matter.