Why this lesson exists
After learning how to avoid single‑shot timing traps, how to warm‑up, and how to capture metadata, you still lack a concrete way to turn a csperf run into a publishable story. This episode closes the loop by showing you how to design an experiment, run it, and ship the evidence.
Recap — where we are in the series
- Ep 1 — Why a Single ./a.out Time Misleads Your Performance Claims — Use csperf to capture reproducible evidence.
- Ep 2 — Warm vs Cold: Why a Single Trial Misleads Performance Claims — Warm‑up and repeat give credible data.
- Ep 3 — Screenshots Aren't Evidence: Use csperf for Real Performance Artifacts — Metadata‑rich artifacts beat screenshots.
- Ep 4 — Comparing Across Machines: Why Metadata Matters in csperf — Machine metadata enables honest cross‑machine comparison.
- Ep 5 — csperf: A Lightweight Observatory for Honest Performance Tracking — Observatory commands give quick evidence.
- Ep 6 — csperf Doctor: First Command to Diagnose Toolchain Issues — Doctor validates your environment.
The misconception
Many developers think “run csperf once, publish the JSON” is enough. Without a clear hypothesis, repeat strategy, and a narrative, the artifact is just a file that nobody will read.
What problem csperf solves (this episode's slice)
csperf gives you a complete experiment record: the code, the compiler flags, the machine, the warm‑up and repeat counts, and the raw timing data. It also provides tools to profile and visualize that data so you can write a convincing claim.
Mental model
- Question – What do you want to prove? E.g., “Does Clang 18 reduce BFS traversal time by 20 % on a 1 M‑node graph?”
- Design – Pick a workload, set warm‑up and repeat counts, and decide on the backend.
-
Run – Execute
csperf runto generate a JSON artifact. -
Profile – Use
csperf profileto see where time is spent. - Visualize – Generate an HTML report that can be shared.
- Claim – State the effect size, confidence interval, and limitations.
Lab: install and first commands
- Install csperf via pip:
pip install compilersutra-perf. - Verify installation:
csperf doctor. - Prepare a simple BFS workload:
examples/cpp/graph_bfs_frontier.cpp. - Run the experiment:
csperf run \
--input examples/cpp/graph_bfs_frontier.cpp \
--backend cpu \
--warmup-runs 2 \
--repeat-runs 8 \
--output results/capstone.json
This command records all metadata and raw timings.
Lab: what we ran on this machine
The artifact results/capstone.json contains:
- CPU: Intel(R) Xeon(R) CPU E5‑2630 v4 @ 2.20 GHz
- OS: Ubuntu 20.04 LTS
- Compiler: Clang 18.0.0
- Warm‑up runs: 2
- Repeat runs: 8
- Backend: cpu
The JSON also lists execution_time_ms for each trial and a timing_summary with median, min, max, and standard deviation.
Results (real numbers only)
Graph: trial wall_time_ms from this run's artifacts (not invented).
Artifacts missing for this episode — numbers withheld.
How to read the artifacts
Open results/capstone.json in a text editor. The top‑level keys are:
-
metadata– machine, compiler, and run configuration. -
trials– an array of trial objects, each withexecution_time_msandwall_time_ms. -
timing_summary– aggregated statistics.
Use csperf profile results/capstone.json to get a call‑stack breakdown.
Common mistakes (teacher checklist)
- Skipping warm‑up – leads to cold‑start bias.
- Too few repeats – statistical noise dominates.
- Ignoring metadata – makes cross‑machine claims impossible.
- Publishing raw JSON only – readers cannot interpret it.
- Not stating confidence – a 5 % speed‑up without a CI range is weak.
Try this next (homework)
Design an experiment that answers a question relevant to your project. Run csperf, profile, and visualize. Prepare a short paragraph stating the effect size and confidence interval. Do this tonight — Episode 31 starts by assuming you did.
Closing
You now have a full workflow: hypothesis → design → run → profile → visualize → claim. The next episode will show how to package and publish that story so the community can reproduce and critique it.
The series so far
- Ep 1 — Why a Single ./a.out Time Misleads Your Performance Claims — Reproducible, machine‑aware evidence.
- Ep 2 — Warm vs Cold: Why a Single Trial Misleads Performance Claims — Warm‑up and repeat for credibility.
- Ep 3 — Screenshots Aren't Evidence: Use csperf for Real Performance Artifacts — Metadata beats screenshots.
- Ep 4 — Comparing Across Machines: Why Metadata Matters in csperf — Metadata enables honest comparison.
- Ep 5 — csperf: A Lightweight Observatory for Honest Performance Tracking — Quick, reliable evidence.
- Ep 6 — csperf Doctor: First Command to Diagnose Toolchain Issues — Validate your environment.


Top comments (0)