DEV Community

compilersutra
compilersutra

Posted on Originally published at compilersutra.com

Baselines: Pinning a Golden JSON Without Lying to Yourself

#baseline #csperf #methodology #performance

Why this lesson exists

When you hand off a golden JSON to a teammate, you assume the numbers will stay trustworthy. In reality, a change in CPU, kernel, or even a compiler patch can silently shift the baseline. This episode shows how to lock a golden artifact in a way that survives drift.

Recap — where we are in the series

  1. Why a Single ./a.out Time Misleads Your Performance Claims – single‑shot timing is unreliable.
  2. Warm vs Cold: Why a Single Trial Misleads Performance Claims – warm‑up and repeat runs are essential.
  3. Screenshots Aren't Evidence: Use csperf for Real Performance Artifacts – artifacts beat screenshots.
  4. Comparing Across Machines: Why Metadata Matters in csperf – metadata enables honest cross‑machine comparison.
  5. csperf: A Lightweight Observatory for Honest Performance Tracking – observatory commands give quick evidence.
  6. csperf Doctor: First Command to Diagnose Toolchain Issues – validate the toolchain.
  7. csperf Quickstart: 30‑Second First Win – first reproducible artifact in 30 s.
  8. Reading csperf JSON: The Real Performance Artifact – JSON is the source of truth.
  9. Backends 101: Choosing the Right Measurement Surface – pick the right backend.
  10. Stability Metrics: Min, Max, and Standard Deviation as First-Class Citizens – mean + stdev.
  11. Warmup Deep Dive: What You’re Throwing Away and Why – quantify warm‑up impact.
  12. Row vs. Column: Measuring Locality with csperf – locality measured, not assumed.
  13. Unmasking the O‑3 Myth: Real‑World Optimizer Impact with csperf – validate optimizer claims.
  14. Turning on Linux perf Counters with csperf (honestly) – enable and validate counters.
  15. Noise Sources: Affinity, Frequency, and Background Load – isolate noise.
  16. Evidence Levels: What a Single JSON Can and Cannot Claim – evidence levels.
  17. csperf Diff: Comparing Two Runs Without Spreadsheet Chaos – artifact‑driven diff.
  18. HTML Reports: Share Performance Artifacts Your Team Will Actually Open – human‑friendly dashboards.
  19. Regression Language: When a Slower Mean is Actionable – isolate actionable regressions.
  20. CI‑Shaped Workflows: Store What Matters, Fail on What Counts – full artifacts in CI.

The misconception

A golden JSON is often treated as a static truth. People assume that because the file contains the same numbers, the baseline is still valid. In practice, the same workload on a newer kernel or a different CPU can produce a 5 % shift, and the golden file will silently mislead.

What problem csperf solves (this episode's slice)

csperf lets you pin a golden artifact to a specific machine, compiler, and backend configuration. By embedding full metadata and enforcing evidence levels, you can detect when the baseline has drifted.

Mental model

  1. Artifact – the JSON file that records metrics and metadata.
  2. Pinning – recording the exact machine, compiler, and backend.
  3. Evidence level – a flag that indicates the artifact’s confidence.
  4. Diff – comparing a new run against the pinned golden to spot drift.

Lab: install and first commands 

# Ensure csperf is on PATH
export PATH=$HOME/.venv/bin:$PATH

# Run the golden build
csperf run \
  --input examples/cpp/matrix_traversal.cpp \
  --backend cpu \
  --warmup-runs 2 \
  --repeat-runs 10 \
  --output results/golden.json
Enter fullscreen mode Exit fullscreen mode

The command creates results/golden.json and stores machine metadata in csperf/machine.txt.

Lab: what we ran on this machine

Item Value
Hostname f4c59d864117
CPU AMD Ryzen 7 9700X 8‑Core
Kernel 7.0.0-34-generic
Compiler clang++ 18.1.0
Backend cpu
Warm‑up runs 2
Repeat runs 10

Results (real numbers only)

From csperf/golden.json:

  • execution_time_ms mean: 5.1542 ms
  • execution_time_ms min: 5.134 ms
  • execution_time_ms max: 5.209 ms
  • cpu_cycles: 5,842,890
  • ipc: 2.100664
  • cache_misses: 84,263
  • l1_cache_misses: 238,030

How to read the artifacts

  1. Open csperf/golden.json – the top‑level keys are input_file, backend, metrics, and hardware.
  2. The hardware section lists the exact CPU model, kernel, and compiler.
  3. metrics.execution_time_summary_ms contains the statistical summary.
  4. The artifacts section points to CSV/Excel copies for quick spreadsheet use.

Common mistakes (teacher checklist)

  • Skipping machine.txt – always keep the machine snapshot.
  • Ignoring evidence level – a low level means the run may be unreliable.
  • Forgetting to pin the backend – a different backend (e.g., gpu) changes the metric distribution.
  • Overlooking warm‑up – without warm‑up, the first few runs skew the mean.

Try this next (homework)

Run the same workload on a different machine (e.g., a newer Ryzen 9) and generate a new golden JSON. Use csperf diff golden.json new.json to see how the metrics drift. Do this tonight — Episode 22 starts by assuming you did.

Closing

A golden JSON is only as trustworthy as the context it carries. By pinning machine, compiler, and backend metadata, csperf turns a static file into a living artifact that flags drift before it becomes a silent performance bug.

The series so far

  • Ep 1: Why a Single ./a.out Time Misleads Your Performance Claims (this article)
  • Ep 2: Warm vs Cold: Why a Single Trial Misleads Performance Claims
  • Ep 3: Screenshots Aren't Evidence: Use csperf for Real Performance Artifacts
  • Ep 4: Comparing Across Machines: Why Metadata Matters in csperf
  • Ep 5: csperf: A Lightweight Observatory for Honest Performance Tracking
  • Ep 6: csperf Doctor: First Command to Diagnose Toolchain Issues
  • Ep 7: csperf Quickstart: 30‑Second First Win
  • Ep 8: Reading csperf JSON: The Real Performance Artifact
  • Ep 9: Backends 101: Choosing the Right Measurement Surface
  • Ep 10: Stability Metrics: Min, Max, and Standard Deviation as First-Class Citizens
  • Ep 11: Warmup Deep Dive: What You’re Throwing Away and Why
  • Ep 12: Row vs. Column: Measuring Locality with csperf
  • Ep 13: Unmasking the O‑3 Myth: Real‑World Optimizer Impact with csperf
  • Ep 14: Turning on Linux perf Counters with csperf (honestly)
  • Ep 15: Noise Sources: Affinity, Frequency, and Background Load
  • Ep 16: Evidence Levels: What a Single JSON Can and Cannot Claim
  • Ep 17: csperf Diff: Comparing Two Runs Without Spreadsheet Chaos
  • Ep 18: HTML Reports: Share Performance Artifacts Your Team Will Actually Open
  • Ep 19: Regression Language: When a Slower Mean is Actionable
  • Ep 20: CI‑Shaped Workflows: Store What Matters, Fail on What Counts
  • Ep 21: Baselines: Pinning a Golden JSON Without Lying to Yourself

Teaser

Last time we pinned a golden JSON; next we’ll learn how a teammate can read that evidence in a PR. Stay tuned for the perf review checklist.

Top comments (0)