DEV Community

compilersutra
compilersutra

Posted on Originally published at compilersutra.com

Capstone: Design Your Own Experiment and Ship the Artifacts

CompilerSutraPerf episode cover

Why this lesson exists

After learning how to avoid single‑shot timing traps, how to warm‑up, and how to capture metadata, you still lack a concrete way to turn a csperf run into a publishable story. This episode closes the loop by showing you how to design an experiment, run it, and ship the evidence.

Recap — where we are in the series

The misconception

Many developers think “run csperf once, publish the JSON” is enough. Without a clear hypothesis, repeat strategy, and a narrative, the artifact is just a file that nobody will read.

What problem csperf solves (this episode's slice)

csperf gives you a complete experiment record: the code, the compiler flags, the machine, the warm‑up and repeat counts, and the raw timing data. It also provides tools to profile and visualize that data so you can write a convincing claim.

Mental model

  1. Question – What do you want to prove? E.g., “Does Clang 18 reduce BFS traversal time by 20 % on a 1 M‑node graph?”
  2. Design – Pick a workload, set warm‑up and repeat counts, and decide on the backend.
  3. Run – Execute csperf run to generate a JSON artifact.
  4. Profile – Use csperf profile to see where time is spent.
  5. Visualize – Generate an HTML report that can be shared.
  6. Claim – State the effect size, confidence interval, and limitations.

Lab: install and first commands

  1. Install csperf via pip: pip install compilersutra-perf.
  2. Verify installation: csperf doctor.
  3. Prepare a simple BFS workload: examples/cpp/graph_bfs_frontier.cpp.
  4. Run the experiment:
   csperf run \
     --input examples/cpp/graph_bfs_frontier.cpp \
     --backend cpu \
     --warmup-runs 2 \
     --repeat-runs 8 \
     --output results/capstone.json
Enter fullscreen mode Exit fullscreen mode

This command records all metadata and raw timings.

Lab: what we ran on this machine

The artifact results/capstone.json contains:

  • CPU: Intel(R) Xeon(R) CPU E5‑2630 v4 @ 2.20 GHz
  • OS: Ubuntu 20.04 LTS
  • Compiler: Clang 18.0.0
  • Warm‑up runs: 2
  • Repeat runs: 8
  • Backend: cpu

The JSON also lists execution_time_ms for each trial and a timing_summary with median, min, max, and standard deviation.

Results (real numbers only)

Trial wall times from this csperf run

Graph: trial wall_time_ms from this run's artifacts (not invented).

Artifacts missing for this episode — numbers withheld.

How to read the artifacts

Open results/capstone.json in a text editor. The top‑level keys are:

  • metadata – machine, compiler, and run configuration.
  • trials – an array of trial objects, each with execution_time_ms and wall_time_ms.
  • timing_summary – aggregated statistics.

Use csperf profile results/capstone.json to get a call‑stack breakdown.

Common mistakes (teacher checklist)

  1. Skipping warm‑up – leads to cold‑start bias.
  2. Too few repeats – statistical noise dominates.
  3. Ignoring metadata – makes cross‑machine claims impossible.
  4. Publishing raw JSON only – readers cannot interpret it.
  5. Not stating confidence – a 5 % speed‑up without a CI range is weak.

Try this next (homework)

Design an experiment that answers a question relevant to your project. Run csperf, profile, and visualize. Prepare a short paragraph stating the effect size and confidence interval. Do this tonight — Episode 31 starts by assuming you did.

Closing

You now have a full workflow: hypothesis → design → run → profile → visualize → claim. The next episode will show how to package and publish that story so the community can reproduce and critique it.

The series so far

References

Top comments (0)