DEV Community

compilersutra
compilersutra

Posted on Originally published at compilersutra.com

csperf Diff: Comparing Two Runs Without Spreadsheet Chaos

#csperf #diff #benchmarking #performance

Why this lesson exists

In Slack threads you’ll often see two people paste a mean execution time and immediately start a debate. One says “my run is faster” the other counters “but your mean hides the variance”. The problem is that a single number is a summary that can be misleading without context. csperf’s built‑in diff gives you a side‑by‑side, line‑by‑line comparison of two full JSON artifacts, turning the mean‑vs‑mean argument into a data‑driven discussion.

Recap — where we are in the series

  1. Why a Single ./a.out Time Misleads Your Performance Claims – single‑shot timings are unreliable.
  2. Warm vs Cold: Why a Single Trial Misleads Performance Claims – warm‑up and repeat runs are essential.
  3. Screenshots Aren’t Evidence: Use csperf for Real Performance Artifacts – artifacts beat screenshots.
  4. Comparing Across Machines: Why Metadata Matters in csperf – metadata enables honest cross‑machine comparison.
  5. csperf: A Lightweight Observatory for Honest Performance Tracking – observatory commands give quick evidence.
  6. csperf Doctor: First Command to Diagnose Toolchain Issues – validate the toolchain.
  7. csperf Quickstart: 30‑Second First Win – first reproducible artifact in half a minute.
  8. Reading csperf JSON: The Real Performance Artifact – JSON is the source of truth.
  9. Backends 101: Choosing the Right Measurement Surface – pick the right backend.
  10. Stability Metrics: Min, Max, and Standard Deviation as First-Class Citizens – mean hides volatility.
  11. Warmup Deep Dive: What You’re Throwing Away and Why – quantify warm‑up impact.
  12. Row vs. Column: Measuring Locality with csperf – locality measured, not assumed.
  13. Unmasking the O‑3 Myth: Real‑World Optimizer Impact with csperf – validate optimizer claims.
  14. Turning on Linux perf Counters with csperf (honestly) – enable, validate, read counters.
  15. Noise Sources: Affinity, Frequency, and Background Load — How to isolate and quantify non‑compiler variance with csperf – isolate noise.
  16. Evidence Levels: What a Single JSON Can and Cannot Claim – evidence levels separate truth from hype.

The misconception

People think a single mean value is enough to prove a performance win. In reality, the mean is just one statistic; it can mask outliers, skewed distributions, or systematic noise. When two teams post their means in a chat, the conversation devolves into “your mean is lower” without anyone looking at the underlying data.

What problem csperf solves (this episode's slice)

csperf’s diff command compares two JSON artifacts and produces a CSV (or JSON) that lists every metric side‑by‑side, along with the delta and the percent change. This turns a vague “mean is lower” claim into a concrete, machine‑verified statement: Metric X decreased by 0.3 % and Metric Y increased by 1.2 %.

Mental model

  1. Run 1 – baseline run, artifact a.json.
  2. Run 2 – new configuration, artifact b.json.
  3. Diff – csperf diff a.json b.json --csv a-vs-b.csv.
  4. Interpret – look at the CSV: each row shows metric, value in A, value in B, delta, percent change.
  5. Decide – if a metric improves, you have evidence; if it degrades, you have evidence to investigate.

Lab: install and first commands 

# Ensure csperf is installed in your venv
source .venv/bin/activate
pip install csperf
Enter fullscreen mode Exit fullscreen mode

Lab: what we ran on this machine

We ran the same matrix traversal benchmark twice, once producing a.json and once b.json.

csperf run \
  --input examples/cpp/matrix_traversal.cpp \
  --backend cpu \
  --warmup-runs 1 \
  --repeat-runs 5 \
  --output results/a.json

csperf run \
  --input examples/cpp/matrix_traversal.cpp \
  --backend cpu \
  --warmup-runs 1 \
  --repeat-runs 5 \
  --output results/b.json

csperf diff results/a.json results/b.json --csv results/a-vs-b.csv
Enter fullscreen mode Exit fullscreen mode

All commands were executed on f4c59d864117 (AMD Ryzen 7 9700X) with the same compiler flags (-O3 -std=c++17).

Results (real numbers only)

The two runs produced identical mean execution times:

Metric a.json b.json
execution_time_ms 5.134 5.134

The diff CSV (excerpt) shows zero delta for every metric because the runs were identical. In a real scenario where you change an optimization flag or a compiler version, the CSV would reveal the exact numeric impact.

How to read the artifacts

  • a.json and b.json contain full metadata, including compiler steps, hardware counters, and statistical summaries.
  • The diff CSV has columns: metric, value_a, value_b, delta, percent_change.
  • A positive percent_change indicates an increase; negative indicates a decrease.

Common mistakes (teacher checklist)

  1. Running on different machines – always capture machine metadata.
  2. Ignoring warm‑up – keep warm‑up runs consistent.
  3. Comparing different backends – ensure the backend flag is identical.
  4. Using the wrong diff command – csperf diff expects JSON, not CSV.
  5. Reading only the mean – look at min, max, and stdev for context.

Try this next (homework)

Run the benchmark twice, once with -O2 and once with -O3, and use csperf diff to compare the artifacts. Submit the CSV to the team and discuss the percent changes.

Do this tonight — Episode 18 starts by assuming you did.

Closing

We’ve moved from a vague mean‑vs‑mean debate to a concrete, artifact‑driven comparison. The next episode will show how to present these results in a human‑friendly HTML report.

The series so far

  • Ep 1: Why a Single ./a.out Time Misleads Your Performance Claims (this article)
  • Ep 2: Warm vs Cold: Why a Single Trial Misleads Performance Claims
  • Ep 3: Screenshots Aren’t Evidence: Use csperf for Real Performance Artifacts
  • Ep 4: Comparing Across Machines: Why Metadata Matters in csperf
  • Ep 5: csperf: A Lightweight Observatory for Honest Performance Tracking
  • Ep 6: csperf Doctor: First Command to Diagnose Toolchain Issues
  • Ep 7: csperf Quickstart: 30‑Second First Win
  • Ep 8: Reading csperf JSON: The Real Performance Artifact
  • Ep 9: Backends 101: Choosing the Right Measurement Surface
  • Ep 10: Stability Metrics: Min, Max, and Standard Deviation as First-Class Citizens
  • Ep 11: Warmup Deep Dive: What You’re Throwing Away and Why
  • Ep 12: Row vs. Column: Measuring Locality with csperf
  • Ep 13: Unmasking the O‑3 Myth: Real‑World Optimizer Impact with csperf
  • Ep 14: Turning on Linux perf Counters with csperf (honestly)
  • Ep 15: Noise Sources: Affinity, Frequency, and Background Load — How to isolate and quantify non‑compiler variance with csperf
  • Ep 16: Evidence Levels: What a Single JSON Can and Cannot Claim
  • Ep 17: csperf Diff: Comparing Two Runs Without Spreadsheet Chaos

Top comments (0)