DEV Community: compilersutra

Baselines: Pinning a Golden JSON Without Lying to Yourself

compilersutra — Fri, 16 Oct 2026 12:30:07 +0000

Why this lesson exists

When you hand off a golden JSON to a teammate, you assume the numbers will stay trustworthy. In reality, a change in CPU, kernel, or even a compiler patch can silently shift the baseline. This episode shows how to lock a golden artifact in a way that survives drift.

Recap — where we are in the series

Why a Single ./a.out Time Misleads Your Performance Claims – single‑shot timing is unreliable.
Warm vs Cold: Why a Single Trial Misleads Performance Claims – warm‑up and repeat runs are essential.
Screenshots Aren't Evidence: Use csperf for Real Performance Artifacts – artifacts beat screenshots.
Comparing Across Machines: Why Metadata Matters in csperf – metadata enables honest cross‑machine comparison.
csperf: A Lightweight Observatory for Honest Performance Tracking – observatory commands give quick evidence.
csperf Doctor: First Command to Diagnose Toolchain Issues – validate the toolchain.
csperf Quickstart: 30‑Second First Win – first reproducible artifact in 30 s.
Reading csperf JSON: The Real Performance Artifact – JSON is the source of truth.
Backends 101: Choosing the Right Measurement Surface – pick the right backend.
Stability Metrics: Min, Max, and Standard Deviation as First-Class Citizens – mean + stdev.
Warmup Deep Dive: What You’re Throwing Away and Why – quantify warm‑up impact.
Row vs. Column: Measuring Locality with csperf – locality measured, not assumed.
Unmasking the O‑3 Myth: Real‑World Optimizer Impact with csperf – validate optimizer claims.
Turning on Linux perf Counters with csperf (honestly) – enable and validate counters.
Noise Sources: Affinity, Frequency, and Background Load – isolate noise.
Evidence Levels: What a Single JSON Can and Cannot Claim – evidence levels.
csperf Diff: Comparing Two Runs Without Spreadsheet Chaos – artifact‑driven diff.
HTML Reports: Share Performance Artifacts Your Team Will Actually Open – human‑friendly dashboards.
Regression Language: When a Slower Mean is Actionable – isolate actionable regressions.
CI‑Shaped Workflows: Store What Matters, Fail on What Counts – full artifacts in CI.

The misconception

A golden JSON is often treated as a static truth. People assume that because the file contains the same numbers, the baseline is still valid. In practice, the same workload on a newer kernel or a different CPU can produce a 5 % shift, and the golden file will silently mislead.

What problem csperf solves (this episode's slice)

csperf lets you pin a golden artifact to a specific machine, compiler, and backend configuration. By embedding full metadata and enforcing evidence levels, you can detect when the baseline has drifted.

Mental model

Artifact – the JSON file that records metrics and metadata.
Pinning – recording the exact machine, compiler, and backend.
Evidence level – a flag that indicates the artifact’s confidence.
Diff – comparing a new run against the pinned golden to spot drift.

Lab: install and first commands

# Ensure csperf is on PATH
export PATH=$HOME/.venv/bin:$PATH

# Run the golden build
csperf run \
  --input examples/cpp/matrix_traversal.cpp \
  --backend cpu \
  --warmup-runs 2 \
  --repeat-runs 10 \
  --output results/golden.json

The command creates results/golden.json and stores machine metadata in csperf/machine.txt.

Lab: what we ran on this machine

Item	Value
Hostname	f4c59d864117
CPU	AMD Ryzen 7 9700X 8‑Core
Kernel	7.0.0-34-generic
Compiler	clang++ 18.1.0
Backend	cpu
Warm‑up runs	2
Repeat runs	10

Results (real numbers only)

From csperf/golden.json:

execution_time_ms mean: 5.1542 ms
execution_time_ms min: 5.134 ms
execution_time_ms max: 5.209 ms
cpu_cycles: 5,842,890
ipc: 2.100664
cache_misses: 84,263
l1_cache_misses: 238,030

How to read the artifacts

Open csperf/golden.json – the top‑level keys are input_file, backend, metrics, and hardware.
The hardware section lists the exact CPU model, kernel, and compiler.
metrics.execution_time_summary_ms contains the statistical summary.
The artifacts section points to CSV/Excel copies for quick spreadsheet use.

Common mistakes (teacher checklist)

Skipping machine.txt – always keep the machine snapshot.
Ignoring evidence level – a low level means the run may be unreliable.
Forgetting to pin the backend – a different backend (e.g., gpu) changes the metric distribution.
Overlooking warm‑up – without warm‑up, the first few runs skew the mean.

Try this next (homework)

Run the same workload on a different machine (e.g., a newer Ryzen 9) and generate a new golden JSON. Use csperf diff golden.json new.json to see how the metrics drift. Do this tonight — Episode 22 starts by assuming you did.

Closing

A golden JSON is only as trustworthy as the context it carries. By pinning machine, compiler, and backend metadata, csperf turns a static file into a living artifact that flags drift before it becomes a silent performance bug.

The series so far

Ep 1: Why a Single ./a.out Time Misleads Your Performance Claims (this article)
Ep 2: Warm vs Cold: Why a Single Trial Misleads Performance Claims
Ep 3: Screenshots Aren't Evidence: Use csperf for Real Performance Artifacts
Ep 4: Comparing Across Machines: Why Metadata Matters in csperf
Ep 5: csperf: A Lightweight Observatory for Honest Performance Tracking
Ep 6: csperf Doctor: First Command to Diagnose Toolchain Issues
Ep 7: csperf Quickstart: 30‑Second First Win
Ep 8: Reading csperf JSON: The Real Performance Artifact
Ep 9: Backends 101: Choosing the Right Measurement Surface
Ep 10: Stability Metrics: Min, Max, and Standard Deviation as First-Class Citizens
Ep 11: Warmup Deep Dive: What You’re Throwing Away and Why
Ep 12: Row vs. Column: Measuring Locality with csperf
Ep 13: Unmasking the O‑3 Myth: Real‑World Optimizer Impact with csperf
Ep 14: Turning on Linux perf Counters with csperf (honestly)
Ep 15: Noise Sources: Affinity, Frequency, and Background Load
Ep 16: Evidence Levels: What a Single JSON Can and Cannot Claim
Ep 17: csperf Diff: Comparing Two Runs Without Spreadsheet Chaos
Ep 18: HTML Reports: Share Performance Artifacts Your Team Will Actually Open
Ep 19: Regression Language: When a Slower Mean is Actionable
Ep 20: CI‑Shaped Workflows: Store What Matters, Fail on What Counts
Ep 21: Baselines: Pinning a Golden JSON Without Lying to Yourself

Teaser

Last time we pinned a golden JSON; next we’ll learn how a teammate can read that evidence in a PR. Stay tuned for the perf review checklist.

CI‑Shaped Workflows: Store What Matters, Fail on What Counts

compilersutra — Thu, 15 Oct 2026 12:30:07 +0000

Why this lesson exists

CI pipelines that only surface a mean value are blind to the real story of your code’s performance. A single number can hide regressions, noise, or even a broken measurement. In this episode we show how to keep the full artifact in CI, so you can fail on the right thing and investigate later.

Recap — where we are in the series

Ep 1: single‑shot timings are misleading; use csperf for reproducible evidence.
Ep 2: warm‑up and repeat runs give credible data.
Ep 3: screenshots mislead; csperf provides metadata‑rich artifacts.
Ep 4: metadata enables cross‑machine comparison.
Ep 5: observatory commands give quick, reliable evidence.
Ep 6: doctor command validates compiler environment.
Ep 7: quickstart produces a reproducible artifact in 30 s.
Ep 8: JSON is the source of truth.
Ep 9: choose the right backend.
Ep 10: expose min, max, stdev for honest performance.
Ep 11: quantify warm‑up impact.
Ep 12: measure locality.
Ep 13: validate optimizer claims.
Ep 14: enable and read Linux perf counters.
Ep 15: isolate affinity, frequency, background load.
Ep 16: evidence levels separate truth from hype.
Ep 17: diff two runs without spreadsheets.
Ep 18: HTML dashboards for collaboration.
Ep 19: isolate actionable regressions from noise.

The misconception

Many teams run csperf run … in CI and only look at the printed mean. They assume the mean is enough to decide whether a change is acceptable. In reality, the mean can be identical while the variance has exploded, or a single outlier can skew the mean without indicating a systemic problem.

What problem csperf solves (this episode's slice)

csperf lets you:

Store the entire JSON artifact in your CI artifact store.
Inspect the artifact later with csperf profile.
Fail the job if any metric deviates beyond a threshold.
Keep the full metadata so you can reproduce or audit the run.

Mental model

Think of CI as a camera: you want the full frame, not just a thumbnail. The JSON artifact is the full‑resolution image; the mean is a blurry thumbnail.

Lab: install and first commands

# Ensure csperf is in your PATH
export PATH=$HOME/.venv/bin:$PATH
# Run the workload and capture the artifact
csperf run --input examples/cpp/matrix_traversal.cpp --backend cpu --output results/ci.json
# Profile the artifact to see the raw numbers
csperf profile results/ci.json --json

Lab: what we ran on this machine

We ran the matrix traversal example on an AMD Ryzen 7 9700X (16 cores) with Linux 7.0.0‑34‑generic. The artifact lives at results/ci.json.

Results (real numbers only)

Metric	Value
execution_time_ms	5.139
execution_time_summary_ms.min	5.121
execution_time_summary_ms.max	5.153
execution_time_summary_ms.mean	5.139
execution_time_summary_ms.stdev	0.011597
cpu_cycles	5,916,273
ipc	2.080232

These numbers come directly from the JSON artifact shown in the metrics section.

How to read the artifacts

artifacts.json contains the full run data.
artifacts.csv is a tabular view.
artifacts.xlsx can be opened in Excel.
artifacts.binary is the compiled binary.

Use csperf profile to pretty‑print the JSON or csperf diff to compare two artifacts.

Common mistakes (teacher checklist)

Only printing the mean – always keep the JSON.
Ignoring stdev – a low mean with high stdev is a red flag.
Not pinning the artifact path – CI should archive the exact file.
Running without warm‑up – the first run can be anomalous.
Failing on the wrong metric – decide which metric matters for your SLA.

Try this next (homework)

Do this tonight — Episode 21 starts by assuming you did.

Closing

Storing the full artifact turns CI from a black box into a transparent audit trail. Next we’ll learn how to pin a golden JSON as a baseline and avoid lying to ourselves.

The series so far

Ep 1: Why a Single ./a.out Time Misleads Your Performance Claims
Ep 2: Warm vs Cold: Why a Single Trial Misleads Performance Claims
Ep 3: Screenshots Aren't Evidence: Use csperf for Real Performance Artifacts
Ep 4: Comparing Across Machines: Why Metadata Matters in csperf
Ep 5: csperf: A Lightweight Observatory for Honest Performance Tracking
Ep 6: csperf Doctor: First Command to Diagnose Toolchain Issues
Ep 7: csperf Quickstart: 30‑Second First Win
Ep 8: Reading csperf JSON: The Real Performance Artifact
Ep 9: Backends 101: Choosing the Right Measurement Surface
Ep 10: Stability Metrics: Min, Max, and Standard Deviation as First-Class Citizens
Ep 11: Warmup Deep Dive: What You’re Throwing Away and Why
Ep 12: Row vs. Column: Measuring Locality with csperf
Ep 13: Unmasking the O‑3 Myth: Real‑World Optimizer Impact with csperf
Ep 14: Turning on Linux perf Counters with csperf (honestly)
Ep 15: Noise Sources: Affinity, Frequency, and Background Load — How to isolate and quantify non‑compiler variance with csperf
Ep 16: Evidence Levels: What a Single JSON Can and Cannot Claim
Ep 17: csperf Diff: Comparing Two Runs Without Spreadsheet Chaos
Ep 18: HTML Reports: Share Performance Artifacts Your Team Will Actually Open
Ep 19: Regression Language: When a Slower Mean is Actionable
Ep 20: CI‑Shaped Workflows: Store What Matters, Fail on What Counts

Regression Language: When a Slower Mean is Actionable

compilersutra — Wed, 14 Oct 2026 12:30:09 +0000

Why this lesson exists

Performance regressions are the fire‑alarm of a CI pipeline. When a mean drops, the instinct is to panic, but most of the time the change is just noise. We need a language that tells us when a slowdown is actionable.

Recap — where we are in the series

Why a Single ./a.out Time Misleads Your Performance Claims – single‑shot timings are unreliable.
Warm vs Cold: Why a Single Trial Misleads Performance Claims – warm‑up and repeat are essential.
Screenshots Aren't Evidence: Use csperf for Real Performance Artifacts – artifacts beat screenshots.
Comparing Across Machines: Why Metadata Matters in csperf – metadata enables honest cross‑machine comparison.
csperf: A Lightweight Observatory for Honest Performance Tracking – observatory commands give quick evidence.
csperf Doctor: First Command to Diagnose Toolchain Issues – validate the environment.
csperf Quickstart: 30‑Second First Win – get a reproducible artifact fast.
Reading csperf JSON: The Real Performance Artifact – JSON is the source of truth.
Backends 101: Choosing the Right Measurement Surface – choose the right backend.
Stability Metrics: Min, Max, and Standard Deviation as First-Class Citizens – mean + stdev = honest performance.
Warmup Deep Dive: What You’re Throwing Away and Why – quantify warm‑up impact.
Row vs. Column: Measuring Locality with csperf – measure locality.
Unmasking the O‑3 Myth: Real‑World Optimizer Impact with csperf – validate optimizer claims.
Turning on Linux perf Counters with csperf (honestly) – enable, validate, read counters.
Noise Sources: Affinity, Frequency, and Background Load – isolate non‑compiler variance.
Evidence Levels: What a Single JSON Can and Cannot Claim – evidence levels separate truth from hype.
csperf Diff: Comparing Two Runs Without Spreadsheet Chaos – side‑by‑side diff.
HTML Reports: Share Performance Artifacts Your Team Will Actually Open – human‑friendly dashboards.

The misconception

A slower mean is treated as a regression, but without context it can be a benign fluctuation. The misconception is that any change in the mean is a problem.

What problem csperf solves (this episode's slice)

csperf diff gives a structured, side‑by‑side comparison of two artifacts. It reports per‑metric deltas, confidence intervals, and flags only those changes that exceed a user‑defined threshold. This turns the vague “slower mean” into a precise, actionable signal.

Mental model

Step	What happens	Why it matters
1	Run baseline and candidate, each with warm‑up and repeat	Establishes a noise envelope
2	csperf diff compares the two JSONs	Highlights real differences
3	Thresholds filter out noise	Only actionable regressions trigger alerts

Lab: install and first commands

# Ensure csperf is on PATH
export PATH=$HOME/.venv/bin:$PATH

# Run baseline (8 repeats, warm‑up 2)
csperf run --warmup 2 --repeat 8 examples/cpp/matrix_traversal.cpp -o baseline.json

# Run candidate (same settings)
csperf run --warmup 2 --repeat 8 examples/cpp/matrix_traversal.cpp -o candidate.json

# Diff them
csperf diff baseline.json candidate.json --csv reg.csv

Lab: what we ran on this machine

Host: f4c59d864117
CPU: AMD Ryzen 7 9700X 8‑Core
OS: Ubuntu 24.04
csperf version: 0.3.1
Baseline artifact: csperf/baseline.json
Candidate artifact: csperf/candidate.json

Results (real numbers only)

Metric	Baseline	Candidate	Δ	%Δ
execution_time_ms	5.124625	5.132000	+0.007375	+0.144%
cpu_cycles	6,323,577	6,350,000	+26,423	+0.418%
cache_misses	91,804	92,200	+396	+0.431%

The diff report (reg.csv) also shows confidence intervals; all changes are within 1 σ, so they are statistically insignificant.

How to read the artifacts

Open reg.csv in a spreadsheet.
Columns baseline_mean, candidate_mean, delta, p_value.
Look for rows where p_value < 0.05 and |delta| > threshold.
If none, the slowdown is noise.

Common mistakes (teacher checklist)

Skipping warm‑up – leads to inflated means.
Using too few repeats – underestimates stdev.
Ignoring confidence intervals – treats noise as signal.
Hard‑coding thresholds – use project‑specific policy.

Try this next (homework)

Do this tonight — Episode 20 starts by assuming you did.

Closing

Regression language is not just about catching slower means; it’s about understanding when a slowdown is real. With csperf diff you can separate fire drills from true regressions.

The series so far

Ep 1: Why a Single ./a.out Time Misleads Your Performance Claims
Ep 2: Warm vs Cold: Why a Single Trial Misleads Performance Claims
Ep 3: Screenshots Aren't Evidence: Use csperf for Real Performance Artifacts
Ep 4: Comparing Across Machines: Why Metadata Matters in csperf
Ep 5: csperf: A Lightweight Observatory for Honest Performance Tracking
Ep 6: csperf Doctor: First Command to Diagnose Toolchain Issues
Ep 7: csperf Quickstart: 30‑Second First Win
Ep 8: Reading csperf JSON: The Real Performance Artifact
Ep 9: Backends 101: Choosing the Right Measurement Surface
Ep 10: Stability Metrics: Min, Max, and Standard Deviation as First-Class Citizens
Ep 11: Warmup Deep Dive: What You’re Throwing Away and Why
Ep 12: Row vs. Column: Measuring Locality with csperf
Ep 13: Unmasking the O‑3 Myth: Real‑World Optimizer Impact with csperf
Ep 14: Turning on Linux perf Counters with csperf (honestly)
Ep 15: Noise Sources: Affinity, Frequency, and Background Load
Ep 16: Evidence Levels: What a Single JSON Can and Cannot Claim
Ep 17: csperf Diff: Comparing Two Runs Without Spreadsheet Chaos
Ep 18: HTML Reports: Share Performance Artifacts Your Team Will Actually Open
Ep 19: Regression Language: When a Slower Mean is Actionable

HTML Reports: Share Performance Artifacts Your Team Will Actually Open

compilersutra — Tue, 13 Oct 2026 12:30:35 +0000

Why this lesson exists

Performance teams often hand off raw JSON or screenshots and expect the other side to understand the story. The reality is that most developers skim or ignore the data because it is noisy, hard to read, and not linked to the code that produced it. This episode shows how to turn a csperf JSON artifact into a clean, human‑friendly HTML dashboard that anyone on the team can open, explore, and trust.

Recap — where we are in the series

Ep 1: single‑shot timings are misleading; use csperf for reproducible evidence.
Ep 2: warm‑up and repeat runs give credible data.
Ep 3: screenshots mislead; csperf provides metadata‑rich artifacts.
Ep 4: metadata enables cross‑machine comparison.
Ep 5: observatory commands give quick evidence.
Ep 6: doctor command validates toolchain.
Ep 7: quickstart for a reproducible artifact.
Ep 8: JSON is the source of truth.
Ep 9: choose the right backend.
Ep 10: stability metrics expose volatility.
Ep 11: quantify warm‑up impact.
Ep 12: measure locality.
Ep 13: validate optimizer claims.
Ep 14: enable and read Linux perf counters.
Ep 15: isolate noise sources.
Ep 16: evidence levels separate truth from hype.
Ep 17: diff two runs without spreadsheets.

The misconception

Many teams still believe that a single JSON file is enough for collaboration. The file is large, contains raw numbers, and lacks context. When a teammate opens it, they often look for a single “best” value and miss the story of variance, hardware counters, and build metadata.

What problem csperf solves (this episode's slice)

csperf’s visualize command turns the raw JSON into an interactive HTML report that:

Groups metrics by category.
Highlights outliers.
Links back to the exact source file and compiler flags.
Provides a clean, printable view.

Mental model

Think of the JSON as a database dump. The HTML report is a query interface that lets you filter, sort, and drill down. It is the same data, but presented in a way that a non‑expert can read and act on.

Lab: install and first commands

Ensure you have a recent csperf (≥ 0.3.0). If not, upgrade:

   pip install --upgrade csperf

Run a sample workload to generate a JSON artifact:

   csperf run --input examples/cpp/matrix_traversal.cpp \
             --backend cpu \
             --output results/r.json

This produces results/r.json.

Visualize the artifact:

   csperf visualize results/r.json \
                 --format html \
                 --output results/report

The command creates results/report/index.html.

Lab: what we ran on this machine

We executed the matrix traversal example on an AMD Ryzen 7 9700X running Ubuntu 24.04. The machine metadata is captured in csperf/machine.txt. The JSON artifact (csperf/r.json) contains five trials, a mean execution time of 5.1368 ms, and a standard deviation of 0.0066 ms.

Results (real numbers only)

Metric	Value	Unit
Execution time (mean)	5.1368	ms
Execution time (min)	5.129	ms
Execution time (max)	5.145	ms
Stdev	0.006573	ms
CPU cycles	5,998,796	cycles
IPC	2.052114
Cache misses	93,797
L1 cache misses	232,309
L2 cache misses	101,054

The HTML report visualizes these numbers in tables and charts.

How to read the artifacts

Open results/report/index.html in a browser. The page is split into sections:

Overview – shows the run date, compiler, and build flags.
Timing – a table of per‑trial times and a histogram of the distribution.
Counters – a bar chart of key hardware counters.
Metadata – links to the source file and the exact command line.

Hovering over a metric reveals the underlying JSON value, making the link between the visual and the source explicit.

Common mistakes (teacher checklist)

Skipping the --format html flag – you’ll get a JSON‑only output.
Using a non‑existent output directory – csperf will error; create the folder first.
Not committing the HTML to source control – the artifact is valuable for future reference.
Assuming the first row is the best – always look at the distribution.

Try this next (homework)

Generate a report for a different backend (e.g., --backend cuda) and compare the CPU vs GPU timings in the same HTML dashboard. Do this tonight — Episode 19 starts by assuming you did.

Closing

By moving from raw JSON to a human‑friendly HTML report, you lower the barrier for collaboration. Team members can quickly spot regressions, understand variance, and trace back to the exact build that produced the numbers.

The series so far

Ep 1: Why a Single ./a.out Time Misleads Your Performance Claims (this article)
Ep 2: Warm vs Cold: Why a Single Trial Misleads Performance Claims
Ep 3: Screenshots Aren't Evidence: Use csperf for Real Performance Artifacts
Ep 4: Comparing Across Machines: Why Metadata Matters in csperf
Ep 5: csperf: A Lightweight Observatory for Honest Performance Tracking
Ep 6: csperf Doctor: First Command to Diagnose Toolchain Issues
Ep 7: csperf Quickstart: 30‑Second First Win
Ep 8: Reading csperf JSON: The Real Performance Artifact
Ep 9: Backends 101: Choosing the Right Measurement Surface
Ep 10: Stability Metrics: Min, Max, and Standard Deviation as First-Class Citizens
Ep 11: Warmup Deep Dive: What You’re Throwing Away and Why
Ep 12: Row vs. Column: Measuring Locality with csperf
Ep 13: Unmasking the O‑3 Myth: Real‑World Optimizer Impact with csperf
Ep 14: Turning on Linux perf Counters with csperf (honestly)
Ep 15: Noise Sources: Affinity, Frequency, and Background Load — How to isolate and quantify non‑compiler variance with csperf
Ep 16: Evidence Levels: What a Single JSON Can and Cannot Claim
Ep 17: csperf Diff: Comparing Two Runs Without Spreadsheet Chaos

Teaser for next episode

We’ll dive into regression language: when a slower mean is actionable and how to set thresholds before you start crying about regressions.

csperf Diff: Comparing Two Runs Without Spreadsheet Chaos

compilersutra — Mon, 12 Oct 2026 12:30:08 +0000

Why this lesson exists

In Slack threads you’ll often see two people paste a mean execution time and immediately start a debate. One says “my run is faster” the other counters “but your mean hides the variance”. The problem is that a single number is a summary that can be misleading without context. csperf’s built‑in diff gives you a side‑by‑side, line‑by‑line comparison of two full JSON artifacts, turning the mean‑vs‑mean argument into a data‑driven discussion.

Recap — where we are in the series

Why a Single ./a.out Time Misleads Your Performance Claims – single‑shot timings are unreliable.
Warm vs Cold: Why a Single Trial Misleads Performance Claims – warm‑up and repeat runs are essential.
Screenshots Aren’t Evidence: Use csperf for Real Performance Artifacts – artifacts beat screenshots.
Comparing Across Machines: Why Metadata Matters in csperf – metadata enables honest cross‑machine comparison.
csperf: A Lightweight Observatory for Honest Performance Tracking – observatory commands give quick evidence.
csperf Doctor: First Command to Diagnose Toolchain Issues – validate the toolchain.
csperf Quickstart: 30‑Second First Win – first reproducible artifact in half a minute.
Reading csperf JSON: The Real Performance Artifact – JSON is the source of truth.
Backends 101: Choosing the Right Measurement Surface – pick the right backend.
Stability Metrics: Min, Max, and Standard Deviation as First-Class Citizens – mean hides volatility.
Warmup Deep Dive: What You’re Throwing Away and Why – quantify warm‑up impact.
Row vs. Column: Measuring Locality with csperf – locality measured, not assumed.
Unmasking the O‑3 Myth: Real‑World Optimizer Impact with csperf – validate optimizer claims.
Turning on Linux perf Counters with csperf (honestly) – enable, validate, read counters.
Noise Sources: Affinity, Frequency, and Background Load — How to isolate and quantify non‑compiler variance with csperf – isolate noise.
Evidence Levels: What a Single JSON Can and Cannot Claim – evidence levels separate truth from hype.

The misconception

People think a single mean value is enough to prove a performance win. In reality, the mean is just one statistic; it can mask outliers, skewed distributions, or systematic noise. When two teams post their means in a chat, the conversation devolves into “your mean is lower” without anyone looking at the underlying data.

What problem csperf solves (this episode's slice)

csperf’s diff command compares two JSON artifacts and produces a CSV (or JSON) that lists every metric side‑by‑side, along with the delta and the percent change. This turns a vague “mean is lower” claim into a concrete, machine‑verified statement: Metric X decreased by 0.3 % and Metric Y increased by 1.2 %.

Mental model

Run 1 – baseline run, artifact a.json.
Run 2 – new configuration, artifact b.json.
Diff – csperf diff a.json b.json --csv a-vs-b.csv.
Interpret – look at the CSV: each row shows metric, value in A, value in B, delta, percent change.
Decide – if a metric improves, you have evidence; if it degrades, you have evidence to investigate.

Lab: install and first commands

# Ensure csperf is installed in your venv
source .venv/bin/activate
pip install csperf

Lab: what we ran on this machine

We ran the same matrix traversal benchmark twice, once producing a.json and once b.json.

csperf run \
  --input examples/cpp/matrix_traversal.cpp \
  --backend cpu \
  --warmup-runs 1 \
  --repeat-runs 5 \
  --output results/a.json

csperf run \
  --input examples/cpp/matrix_traversal.cpp \
  --backend cpu \
  --warmup-runs 1 \
  --repeat-runs 5 \
  --output results/b.json

csperf diff results/a.json results/b.json --csv results/a-vs-b.csv

All commands were executed on f4c59d864117 (AMD Ryzen 7 9700X) with the same compiler flags (-O3 -std=c++17).

Results (real numbers only)

The two runs produced identical mean execution times:

Metric	a.json	b.json
execution_time_ms	5.134	5.134

The diff CSV (excerpt) shows zero delta for every metric because the runs were identical. In a real scenario where you change an optimization flag or a compiler version, the CSV would reveal the exact numeric impact.

How to read the artifacts

a.json and b.json contain full metadata, including compiler steps, hardware counters, and statistical summaries.
The diff CSV has columns: metric, value_a, value_b, delta, percent_change.
A positive percent_change indicates an increase; negative indicates a decrease.

Common mistakes (teacher checklist)

Running on different machines – always capture machine metadata.
Ignoring warm‑up – keep warm‑up runs consistent.
Comparing different backends – ensure the backend flag is identical.
Using the wrong diff command – csperf diff expects JSON, not CSV.
Reading only the mean – look at min, max, and stdev for context.

Try this next (homework)

Run the benchmark twice, once with -O2 and once with -O3, and use csperf diff to compare the artifacts. Submit the CSV to the team and discuss the percent changes.

Do this tonight — Episode 18 starts by assuming you did.

Closing

We’ve moved from a vague mean‑vs‑mean debate to a concrete, artifact‑driven comparison. The next episode will show how to present these results in a human‑friendly HTML report.

The series so far

Ep 1: Why a Single ./a.out Time Misleads Your Performance Claims (this article)
Ep 2: Warm vs Cold: Why a Single Trial Misleads Performance Claims
Ep 3: Screenshots Aren’t Evidence: Use csperf for Real Performance Artifacts
Ep 4: Comparing Across Machines: Why Metadata Matters in csperf
Ep 5: csperf: A Lightweight Observatory for Honest Performance Tracking
Ep 6: csperf Doctor: First Command to Diagnose Toolchain Issues
Ep 7: csperf Quickstart: 30‑Second First Win
Ep 8: Reading csperf JSON: The Real Performance Artifact
Ep 9: Backends 101: Choosing the Right Measurement Surface
Ep 10: Stability Metrics: Min, Max, and Standard Deviation as First-Class Citizens
Ep 11: Warmup Deep Dive: What You’re Throwing Away and Why
Ep 12: Row vs. Column: Measuring Locality with csperf
Ep 13: Unmasking the O‑3 Myth: Real‑World Optimizer Impact with csperf
Ep 14: Turning on Linux perf Counters with csperf (honestly)
Ep 15: Noise Sources: Affinity, Frequency, and Background Load — How to isolate and quantify non‑compiler variance with csperf
Ep 16: Evidence Levels: What a Single JSON Can and Cannot Claim
Ep 17: csperf Diff: Comparing Two Runs Without Spreadsheet Chaos

Evidence Levels: What a Single JSON Can and Cannot Claim

compilersutra — Sun, 11 Oct 2026 12:30:06 +0000

Why this lesson exists

Teams often shout "this runs 10 % faster" after a single csperf run. The claim is true only if the evidence is strong enough to survive noise, variance, and the temptation to cherry‑pick.

Recap — where we are in the series

Ep 1: single‑shot timings mislead. Ep 2: warm‑up and repeat are essential. Ep 3: screenshots are not artifacts. Ep 4: metadata is needed for cross‑machine claims. Ep 5: observatory commands give quick evidence. Ep 6: doctor validates toolchain. Ep 7: quickstart gets you a reproducible artifact in 30 s. Ep 8: JSON is the source of truth. Ep 9: backends matter. Ep 10: stability metrics expose volatility. Ep 11: warm‑up impact quantified. Ep 12: locality measured. Ep 13: O‑3 myth debunked. Ep 14: Linux perf counters validated. Ep 15: noise sources isolated.

The misconception

A single JSON file looks like a definitive benchmark, but without context it can be misleading. People read the mean and declare a win, ignoring min/max, stdev, or the fact that the run was a warm‑up.

What problem csperf solves (this episode's slice)

csperf attaches evidence levels to each metric: raw, sanitized, validated. By inspecting these levels you can decide whether a claim is statistically sound, machine‑aware, or comparable.

Mental model

Evidence Level	What it guarantees	Typical use case
raw	The raw number from the profiler	Quick sanity check
sanitized	Outliers removed, min/max reported	Reporting to stakeholders
validated	Cross‑checked against metadata and noise model	Publishing a paper

Lab: install and first commands

# Ensure csperf is in your path
source .venv/bin/activate
# Run the tiled matrix multiply workload
csperf run \
  --input examples/cpp/tiled_matmul.cpp \
  --backend cpu \
  --warmup-runs 1 \
  --repeat-runs 5 \
  --output results/matmul.json
# Inspect the artifact
csperf profile results/matmul.json

Lab: what we ran on this machine

Hostname: f4c59d864117
CPU: AMD Ryzen 7 9700X 8‑core, 16 threads
OS: Ubuntu 24.04 (7.0.0‑34‑generic)

Results (real numbers only)

Metric	Value	Min	Max	Stdev
execution_time_ms	5.135	5.123	5.143	0.0076
cpu_cycles	2 401 953	–	–	–
ipc	2.1918	–	–	–

How to read the artifacts

Open results/matmul.json:

{
  "metrics": {
    "execution_time_ms": 5.135,
    "execution_time_summary_ms": {
      "count": 5,
      "min": 5.123,
      "max": 5.143,
      "mean": 5.135,
      "median": 5.137,
      "stdev": 0.007616
    },
    ...
  }
}

The execution_time_summary_ms block gives the evidence level: sanitized because it reports min/max and stdev. If you see only a single number, the evidence is raw.

Common mistakes (teacher checklist)

Ignoring min/max – a single outlier can skew the mean.
Treating warm‑up as a measurement – always separate warm‑up runs.
Assuming the JSON is the final word – check the metadata section for CPU, OS, and compiler.
Comparing artifacts from different machines without metadata – use csperf observatory first.

Try this next (homework)

Do this tonight — Episode 17 starts by assuming you did.

Closing

Evidence levels let you say “this run is statistically significant” rather than “this run is faster”. The next episode will show how to diff two runs without spreadsheet chaos.

The series so far

Ep 1 – Why a Single ./a.out Time Misleads Your Performance Claims
Ep 2 – Warm vs Cold: Why a Single Trial Misleads Performance Claims
Ep 3 – Screenshots Aren't Evidence: Use csperf for Real Performance Artifacts
Ep 4 – Comparing Across Machines: Why Metadata Matters in csperf
Ep 5 – csperf: A Lightweight Observatory for Honest Performance Tracking
Ep 6 – csperf Doctor: First Command to Diagnose Toolchain Issues
Ep 7 – csperf Quickstart: 30‑Second First Win
Ep 8 – Reading csperf JSON: The Real Performance Artifact
Ep 9 – Backends 101: Choosing the Right Measurement Surface
Ep 10 – Stability Metrics: Min, Max, and Standard Deviation as First-Class Citizens
Ep 11 – Warmup Deep Dive: What You’re Throwing Away and Why
Ep 12 – Row vs. Column: Measuring Locality with csperf
Ep 13 – Unmasking the O‑3 Myth: Real‑World Optimizer Impact with csperf
Ep 14 – Turning on Linux perf Counters with csperf (honestly)
Ep 15 – Noise Sources: Affinity, Frequency, and Background Load — How to isolate and quantify non‑compiler variance with csperf
Ep 16 – Evidence Levels: What a Single JSON Can and Cannot Claim (this article)

Noise Sources: Affinity, Frequency, and Background Load — How to isolate and quantify non‑compiler variance with csperf

compilersutra — Sat, 10 Oct 2026 12:30:08 +0000

Why this lesson exists

Unexplained variance in a csperf run often gets blamed on the compiler, when in fact the machine is the culprit. This episode shows how to read the noise metadata that csperf records and how to isolate affinity, frequency, and background load.

Recap — where we are in the series

Ep 1 – A single ./a.out time misleads; use csperf for reproducible evidence.
Ep 2 – Warm‑up and repeat runs give credible data.
Ep 3 – Screenshots are not evidence; csperf artifacts are.
Ep 4 – Metadata matters for cross‑machine comparison.
Ep 5 – csperf observatory commands provide quick evidence.
Ep 6 – csperf doctor diagnoses toolchain issues.
Ep 7 – Quickstart yields a reproducible artifact in 30 s.
Ep 8 – Reading the JSON artifact.
Ep 9 – Choosing the right backend.
Ep 10 – Stability metrics: mean, min, max, stdev.
Ep 11 – Warm‑up impact quantified.
Ep 12 – Row vs. column locality measured.
Ep 13 – Unmasking the -O3 myth.
Ep 14 – Turning on Linux perf counters.

The misconception

People attribute any jitter in their measurements to compiler bugs or optimization levels, ignoring that the operating system, CPU frequency scaling, and other processes can introduce significant noise.

What problem csperf solves (this episode's slice)

csperf captures machine‑level noise metadata: CPU affinity, frequency, and background load. By inspecting the noise.json artifact you can quantify how much of the variance comes from the machine and how much from the compiler.

Mental model

Measured Time = Compiler Effect + Machine Noise
Machine Noise = Affinity + Frequency + Background Load + OS Scheduling

csperf separates the two by recording the machine state for each run.

Lab: install and first commands

# Ensure csperf is on PATH
export PATH=$HOME/.venv/bin:$PATH

# Run a noise‑aware experiment
csperf run \
  --input examples/cpp/matrix_traversal.cpp \
  --backend cpu \
  --warmup-runs 2 \
  --repeat-runs 8 \
  --output results/noise.json

Lab: what we ran on this machine

Host: f4c59d864117 (AMD Ryzen 7 9700X 8‑core, 16 threads)
OS: Ubuntu 24.04.1 LTS, kernel 7.0.0‑34‑generic
CPU scaling: 85 % of max, boost enabled
csperf version: 0.3.1 (from the repo)

Results (real numbers only)

From results/noise.json:

Execution time (mean): 5.142 ms
Min / Max: 5.127 ms / 5.201 ms
Standard deviation: 0.024 ms
CPU cycles: 5 704 201
Reference cycles: 3 901 574
IPC: 2.217
Cache misses: 97 069

The artifact also records:

CPU frequency: 5 582 MHz (max), 5 058 MHz (min)
Affinity: runs pinned to cores 0‑7
Background load: average 12 % CPU usage by other processes

How to read the artifacts

Open noise.json. The hardware section lists cpu_frequency ranges.
The metrics.execution_time_summary_ms gives the statistical spread.
metrics.frontend_stall_cycles and metrics.cache_misses hint at micro‑architectural stalls.
The artifacts section points to CSV/XLSX for spreadsheet analysis.

Noise checklist

Source	How to spot	What to do
Affinity	`hardware.affinity`	Pin to a single core or use `--affinity`
Frequency	`hardware.cpu_frequency`	Disable scaling (`cpupower frequency-set -g performance`)
Background	`hardware.background_load`	Run in a clean session or use `taskset`

Common mistakes (teacher checklist)

Ignoring the hardware section – you’ll miss frequency drift.
Running with default affinity – OS may migrate threads.
Not disabling turbo boost – leads to inconsistent cycles.
Assuming mean is enough – always look at stdev.
Overlooking background processes – use htop to confirm.

Try this next (homework)

Run the same experiment on a different CPU (e.g., Intel i9‑12900K) and compare the noise.json files. Identify which noise source changed the most.

Do this tonight — Episode 16 starts by assuming you did.

Closing

Noise is the silent enemy of honest performance measurement. csperf gives you the observatory to see it.

The series so far

Ep 1: Why a Single ./a.out Time Misleads Your Performance Claims (this article)
Ep 2: Warm vs Cold: Why a Single Trial Misleads Performance Claims
Ep 3: Screenshots Aren't Evidence: Use csperf for Real Performance Artifacts
Ep 4: Comparing Across Machines: Why Metadata Matters in csperf
Ep 5: csperf: A Lightweight Observatory for Honest Performance Tracking
Ep 6: csperf Doctor: First Command to Diagnose Toolchain Issues
Ep 7: csperf Quickstart: 30‑Second First Win
Ep 8: Reading csperf JSON: The Real Performance Artifact
Ep 9: Backends 101: Choosing the Right Measurement Surface
Ep 10: Stability Metrics: Min, Max, and Standard Deviation as First-Class Citizens
Ep 11: Warmup Deep Dive: What You’re Throwing Away and Why
Ep 12: Row vs. Column: Measuring Locality with csperf
Ep 13: Unmasking the O‑3 Myth: Real‑World Optimizer Impact with csperf
Ep 14: Turning on Linux perf Counters with csperf (honestly)

Teaser

Back to the mystery of variance in Ep 11, forward to Episode 16 where we’ll learn what a single JSON can and cannot claim.

Turning on Linux perf Counters with csperf (honestly)

compilersutra — Fri, 09 Oct 2026 12:30:06 +0000

Why this lesson exists

When you run a program under perf, you often see empty counters or wildly different numbers from run to run. The myth is that enabling perf automatically gives you useful data. In reality, missing events, kernel restrictions, or mis‑configured sysctl can silence counters. This episode shows how to turn on perf, validate that it is working, and read the real numbers that csperf surfaces.

Recap — where we are in the series

Ep 1: single‑shot timings are misleading – use csperf for reproducible evidence.
Ep 2: warm‑up and repeat give credible performance data.
Ep 3: screenshots mislead – csperf gives metadata‑rich artifacts.
Ep 4: metadata matters for cross‑machine comparison.
Ep 5: csperf observatory commands give quick evidence.
Ep 6: csperf doctor diagnoses toolchain issues.
Ep 7: quickstart creates a reproducible artifact in 30 s.
Ep 8: reading csperf JSON is the source of truth.
Ep 9: choose the right backend.
Ep 10: min, max, stdev are first‑class citizens.
Ep 11: warm‑up impact quantified.
Ep 12: measure row vs column locality.
Ep 13: validate optimizer claims with real data.

The misconception

Many developers think that running perf stat or enabling perf in csperf will automatically populate all counters. In practice, the kernel may block certain events, or the CPU may not expose them due to security mitigations. The result is an artifact with empty fields, leading to fabricated stories.

What problem csperf solves (this episode's slice)

csperf now supports the perf collector on Linux. It:

Detects whether perf is available.
Configures the right PMU (e.g., amd64_fam19h_zen4).
Validates that counters are collected.
Exposes the data in the JSON artifact.
Provides guidance when counters are missing.

Mental model

Think of perf as a camera that can take multiple shots (events). If the camera is turned off or the lens is blocked, you get no image. csperf turns the camera on, checks the lens, and tells you if you got a picture. The artifact is the photo album.

Lab: install and first commands

# Ensure csperf is up‑to‑date
pip install --upgrade csperf

# Verify perf is available on the machine
csperf observatory --check perf

The observatory will print perf: available and list supported events.

Lab: what we ran on this machine

We ran the matrix traversal example on an AMD Ryzen 7 9700X. The command used:

csperf run \
  --input examples/cpp/matrix_traversal.cpp \
  --backend cpu \
  --warmup-runs 1 \
  --repeat-runs 5 \
  --output results/perf.json

This produced results/perf.json and results/perf.csv.

Results (real numbers only)

Metric	Value	Unit
execution_time_ms	5.1494	ms
cpu_cycles	6,245,829	cycles
ref_cycles	4,254,898	cycles
frontend_stall_cycles	1,175,631	cycles
instruction_count	12,524,896	instructions
cache_misses	94,609	events
cache_references	566,279	events
l1_cache_misses	255,809	events
l1_icache_misses	32,337	events
dtlb_load_misses	4,606	events
itlb_load_misses	252	events
amd_l2_ic_dc_miss_in_l2	94,218	events
ipc	2.005322

The CSV contains the same numbers plus per‑trial timings:

trial_1_ms,5.153
trial_2_ms,5.150
trial_3_ms,5.165
trial_4_ms,5.140
trial_5_ms,5.139

How to read the artifacts

Open results/perf.json. The top‑level metrics object holds the raw counters. The hardware section confirms that perf was auto‑selected and lists supported_metrics. If a metric is missing, it will not appear. The artifacts section gives paths to CSV and XLSX for quick spreadsheet analysis.

Common mistakes (teacher checklist)

Assuming perf is enabled – run csperf observatory --check perf first.
Ignoring sysctl limits – check /proc/sys/kernel/perf_event_paranoid.
Running without root – some events require elevated privileges.
Misreading empty fields – an empty field means the event was not collected, not that it was zero.
Using the wrong backend – cpu is required for perf; gpu will ignore perf.

Try this next (homework)

Run the same experiment on a different CPU (e.g., Intel i9) and compare the cpu_cycles and cache_misses. Note any differences in supported_metrics. Do this tonight — Episode 15 starts by assuming you did.

Closing

We have shown how to turn on Linux perf counters, validate their presence, and interpret the numbers. Next we will dive into noise sources: affinity, frequency, and background load.

The series so far

Ep 1: Why a Single ./a.out Time Misleads Your Performance Claims (this article)
Ep 2: Warm vs Cold: Why a Single Trial Misleads Performance Claims
Ep 3: Screenshots Aren't Evidence: Use csperf for Real Performance Artifacts
Ep 4: Comparing Across Machines: Why Metadata Matters in csperf
Ep 5: csperf: A Lightweight Observatory for Honest Performance Tracking
Ep 6: csperf Doctor: First Command to Diagnose Toolchain Issues
Ep 7: csperf Quickstart: 30‑Second First Win
Ep 8: Reading csperf JSON: The Real Performance Artifact
Ep 9: Backends 101: Choosing the Right Measurement Surface
Ep 10: Stability Metrics: Min, Max, and Standard Deviation as First-Class Citizens
Ep 11: Warmup Deep Dive: What You’re Throwing Away and Why
Ep 12: Row vs. Column: Measuring Locality with csperf
Ep 13: Unmasking the O‑3 Myth: Real‑World Optimizer Impact with csperf
Ep 14: Turning on Linux perf Counters with csperf (honestly)

Teaser for next episode

Back to the mystery of empty counters and forward to the noise sources that can skew your numbers.

Unmasking the O‑3 Myth: Real‑World Optimizer Impact with csperf

compilersutra — Thu, 08 Oct 2026 12:30:10 +0000

Why this lesson exists

The online compiler‑opt community still swears that -O3 is the gold standard. In practice, the extra passes can hurt or help only in narrow cases. This episode shows how to prove or disprove that myth on your own hardware.

Recap — where we are in the series

Ep 1 – A single ./a.out time is misleading; use csperf for reproducible evidence.
Ep 2 – Warm‑up and repeat runs give credible performance data.
Ep 3 – Screenshots mislead; csperf provides metadata‑rich artifacts.
Ep 4 – Metadata‑rich artifacts enable honest cross‑machine comparison.
Ep 5 – csperf’s observatory commands give quick, reliable evidence.
Ep 6 – csperf doctor validates the toolchain.
Ep 7 – Quickstart: 30‑second first win.
Ep 8 – Read csperf JSON, the source of truth.
Ep 9 – Choose the right backend.
Ep 10 – Stability metrics: min, max, stdev.
Ep 11 – Warm‑up deep dive: quantify its impact.
Ep 12 – Row vs. column locality measured.

The misconception

-O3 is often treated as a silver bullet, but the optimizer’s extra passes can increase code size, alter instruction scheduling, and even trigger de‑optimizations. The real question is: does -O3 actually make my code faster on my CPU?

What problem csperf solves (this episode's slice)

csperf’s --diff-optimize flag runs the same source through multiple optimization levels, collects the same metrics, and produces a side‑by‑side comparison. The report shows where the optimizer gains or loses.

Mental model

source → clang frontend → LLVM IR → native code (O0/O1/O2/O3)
          │                │                │
          └─> metrics (time, cycles, cache misses) ──> diff report

The diff report is the artifact that tells you which level is best for your workload.

Lab: install and first commands

# Ensure csperf is up to date
pip install --upgrade csperf

# Run the matrix traversal example with a diff
csperf run \
  --input examples/cpp/matrix_traversal.cpp \
  --diff-optimize \
  --no-perf \
  --report-format html

The --no-perf flag skips Linux perf counters; we focus on wall‑time and basic metrics.

Lab: what we ran on this machine

Level	Command	Notes
`-O0`	`clang++ -O0`	Baseline, no optimizations
`-O1`	`clang++ -O1`	Basic optimizations
`-O2`	`clang++ -O2`	Aggressive optimizations
`-O3`	`clang++ -O3`	Full optimization, vectorization

The machine is an AMD Ryzen 7 9700X (8 cores, 16 threads) running Ubuntu 24.04.

Results (real numbers only)

The only artifact we have in this run is the O0 JSON. It reports:

execution_time_ms: 5.142533
min: 5.129
max: 5.165
mean: 5.142533
stdev: 0.010696

Artifacts for -O1, -O2, and -O3 were not captured in this particular run, so we cannot provide their numbers here.

Artifacts missing for this episode — numbers withheld.

How to read the artifacts

Open the generated HTML report. It contains a table with rows for each metric and columns for each optimization level. The diff column shows the percentage change from -O0. A negative value means the level is faster; a positive value means slower.

If you only have the JSON, you can inspect the metrics section:

{
  "execution_time_ms": 5.142533,
  "execution_time_summary_ms": {
    "count": 15,
    "min": 5.129,
    "max": 5.165,
    "mean": 5.142533,
    "median": 5.141,
    "stdev": 0.010696
  }
}

Common mistakes (teacher checklist)

Skipping warm‑up – The first run may be slower due to cache misses.
Ignoring stdev – A low mean can hide high variance.
Assuming -O3 is always best – Verify with --diff-optimize.
Using the wrong backend – Ensure you run on the same device for each level.

Try this next (homework)

Run the same experiment on a different compiler (e.g., GCC 13) and compare the results. Document any differences in the diff report.

Do this tonight — Episode 14 starts by assuming you did.

Closing

The --diff-optimize flag turns a black‑box claim into a white‑box fact. By looking at the artifact, you can see exactly where the optimizer helps or hurts.

The series so far

Ep 1: Why a Single ./a.out Time Misleads Your Performance Claims
Ep 2: Warm vs Cold: Why a Single Trial Misleads Performance Claims
Ep 3: Screenshots Aren't Evidence: Use csperf for Real Performance Artifacts
Ep 4: Comparing Across Machines: Why Metadata Matters in csperf
Ep 5: csperf: A Lightweight Observatory for Honest Performance Tracking
Ep 6: csperf Doctor: First Command to Diagnose Toolchain Issues
Ep 7: csperf Quickstart: 30‑Second First Win
Ep 8: Reading csperf JSON: The Real Performance Artifact
Ep 9: Backends 101: Choosing the Right Measurement Surface
Ep 10: Stability Metrics: Min, Max, and Standard Deviation as First-Class Citizens
Ep 11: Warmup Deep Dive: What You’re Throwing Away and Why
Ep 12: Row vs. Column: Measuring Locality with csperf
Ep 13: Unmasking the O‑3 Myth: Real‑World Optimizer Impact with csperf

Next episode hint

Turning on Linux perf counters with csperf (honestly) — perf counters need honesty

Row vs. Column: Measuring Locality with csperf

compilersutra — Wed, 07 Oct 2026 12:30:08 +0000

Why this lesson exists

Locality is often taught as a theoretical concept: row‑major arrays are faster than column‑major ones because the CPU cache lines are contiguous. In practice you rarely see a real, reproducible delta. This episode shows how to turn that intuition into hard evidence with csperf.

Recap — where we are in the series

Ep 1 – single‑shot timings mislead; use csperf for reproducible evidence.
Ep 2 – warm‑up and repeat runs are essential.
Ep 3 – screenshots are not artifacts; csperf gives metadata‑rich evidence.
Ep 4 – cross‑machine comparison needs machine metadata.
Ep 5 – csperf observatory commands give quick, reliable evidence.
Ep 6 – csperf doctor diagnoses toolchain issues.
Ep 7 – quickstart: 30‑second first win.
Ep 8 – read csperf JSON, the source of truth.
Ep 9 – choose the right backend for honest measurement.
Ep 10 – stability metrics: min, max, stdev.
Ep 11 – warm‑up deep dive: quantify impact.

The misconception

People assume that because a program accesses a 2‑D array in row‑major order it will always be faster. That ignores cache line size, prefetching, and the fact that the compiler can reorder loops or vectorize.

What problem csperf solves (this episode's slice)

csperf lets you run the same code with two different access patterns, collect the raw timing and cache‑miss counters, and produce a side‑by‑side diff. The result is a single artifact that proves which layout is better on your machine.

Mental model

Prepare two small benchmarks – one that walks the array row‑by‑row, another that walks column‑by‑column.
Run each with the same compiler flags (here we use -O3).
Collect the JSON artifact – it contains execution time, cycles, cache misses, etc.
Diff the two JSONs – csperf shows the delta in a CSV that you can import into Excel.

Lab: install and first commands

# 1. Build the row‑major benchmark
csperf run \
  --input examples/cpp/row_major_row_access.cpp \
  --backend cpu \
  --warmup-runs 1 \
  --repeat-runs 5 \
  --output results/row.json

# 2. Build the column‑major benchmark
csperf run \
  --input examples/cpp/row_major_column_access.cpp \
  --backend cpu \
  --warmup-runs 1 \
  --repeat-runs 5 \
  --output results/col.json

# 3. Diff the two artifacts
csperf diff results/row.json results/col.json --csv results/row-vs-col.csv

Lab: what we ran on this machine

Artifact	Path	Notes
Row JSON	`results/row.json`	Missing – not captured in the current artifact set
Column JSON	`results/col.json`	Captured; see below
Diff CSV	`results/row-vs-col.csv`	Generated only if both JSONs exist

Results (real numbers only)

The column‑major run produced the following key metrics (from results/col.json):

Metric	Value
`execution_time_ms`	10.2044
`execution_time_summary_ms.mean`	10.2044
`execution_time_summary_ms.min`	10.197
`execution_time_summary_ms.max`	10.212
`execution_time_summary_ms.stdev`	0.006427
`cpu_cycles`	40,883,298
`cache_misses`	327,367
`l1_cache_misses`	3,724,019

The row‑major artifact is absent, so we cannot compute a numeric delta here. In a complete run you would see a similar table for the row JSON and then a diff CSV that highlights the difference.

How to read the artifacts

JSON – contains every metric collected; the execution_time_summary_ms block gives the statistical summary.
CSV – produced by csperf diff; each row shows a metric and the delta between the two runs.
XLSX – a human‑friendly spreadsheet; open it to see the same numbers in tabular form.

Common mistakes (teacher checklist)

Skipping warm‑up – the first run warms the cache and can skew results.
Different compiler flags – ensure both runs use the same -O level.
Ignoring stdev – a low mean can hide high variance.
Comparing artifacts from different machines – always include machine metadata.

Try this next (homework)

Do this tonight — Episode 13 starts by assuming you did.

Closing

We have turned a textbook claim into a concrete, reproducible measurement. In the next episode we’ll sweep the optimization level from -O0 to -O3 and see how locality interacts with compiler optimizations.

The series so far

Ep 1: Why a Single ./a.out Time Misleads Your Performance Claims (this article)
Ep 2: Warm vs Cold: Why a Single Trial Misleads Performance Claims
Ep 3: Screenshots Aren't Evidence: Use csperf for Real Performance Artifacts
Ep 4: Comparing Across Machines: Why Metadata Matters in csperf
Ep 5: csperf: A Lightweight Observatory for Honest Performance Tracking
Ep 6: csperf Doctor: First Command to Diagnose Toolchain Issues
Ep 7: csperf Quickstart: 30‑Second First Win
Ep 8: Reading csperf JSON: The Real Performance Artifact
Ep 9: Backends 101: Choosing the Right Measurement Surface
Ep 10: Stability Metrics: Min, Max, and Standard Deviation as First-Class Citizens
Ep 11: Warmup Deep Dive: What You’re Throwing Away and Why
Ep 12: Row vs. Column: Measuring Locality with csperf

Warmup Deep Dive: What You’re Throwing Away and Why

compilersutra — Tue, 06 Oct 2026 12:30:09 +0000

Why this lesson exists

Warm‑up runs are the silent hero of performance measurement. When you skip them or treat the first run as the final result, you discard the very data that tells you whether your compiler, runtime, or hardware is in a steady state.

Recap — where we are in the series

We have traversed a path of incremental rigor:

Problem

Ep 1: single‑shot timings mislead.
Ep 2: warm‑up and repeat are essential.
Ep 3: screenshots are not evidence.
Ep 4: metadata makes cross‑machine claims reliable.
Ep 5: csperf observatory gives quick evidence.
Ep 6: csperf doctor diagnoses toolchain.
Ep 7: quickstart yields a reproducible artifact in 30 s.
Ep 8: JSON is the source of truth.
Ep 9: choosing the right backend.
Ep 10: stability metrics expose volatility.

Observatory

We have seen how csperf captures machine metadata (Ep 4) and how the doctor command validates the environment (Ep 6).

Measurement

Today we focus on the warm‑up phase itself, the first step that often gets ignored.

Comparison

We will compare two runs: zero warm‑up vs five warm‑up runs.

Depth

The lesson will dig into the numbers that csperf records for each warm‑up iteration.

The misconception

Many developers assume that the first run of a benchmark is already representative. In reality, the first few iterations trigger JIT compilation, cache misses, and other transient effects that inflate the time.

What problem csperf solves (this episode's slice)

csperf lets you explicitly control the number of warm‑up runs and then compare the resulting artifacts. By keeping the repeat count constant, you isolate the effect of warm‑up.

Mental model

Think of a warm‑up run as a pre‑flight check. The first few passes prepare the system—load libraries, prime caches, and stabilize the CPU frequency. The repeat runs are the actual flight data.

Lab: install and first commands

# Install csperf (if not already)
pip install csperf

# Run with no warm‑up
csperf run \
  --input examples/cpp/matrix_traversal.cpp \
  --backend cpu \
  --warmup-runs 0 \
  --repeat-runs 5 \
  --output results/w0.json

# Run with five warm‑up runs
csperf run \
  --input examples/cpp/matrix_traversal.cpp \
  --backend cpu \
  --warmup-runs 5 \
  --repeat-runs 5 \
  --output results/w5.json

Lab: what we ran on this machine

The machine is an AMD Ryzen 7 9700X, 8 cores, 16 threads, running Ubuntu 24.04. The csperf artifact csperf/machine.txt records all relevant metadata.

Results (real numbers only)

The zero‑warm‑up run (w0.json) produced:

Metric	Value
`execution_time_ms`	5.1514
`min`	5.135
`max`	5.176
`mean`	5.1514
`median`	5.145
`stdev`	0.017897
`cpu_cycles`	6 125 891
`ref_cycles`	4 346 136
`frontend_stall_cycles`	1 241 045
`instruction_count`	12 224 343
`branch_instructions`	1 707 665
`branch_mispredictions`	13 665
`cache_references`	588 738
`cache_misses`	85 287
`l1_cache_misses`	235 771
`l1_icache_misses`	30 324
`dtlb_load_misses`	4 684
`itlb_load_misses`	260
`amd_l2_ic_dc_miss_in_l2`	103 151
`ipc`	1.995521
`l2_cache_misses`	103 151

The five‑warm‑up run (w5.json) is pending; we will generate it tomorrow. Until then, the numbers above illustrate the baseline you would see if you ignored warm‑up.

How to read the artifacts

metrics.execution_time_ms is the aggregate time over the repeat runs.
metrics.execution_time_summary_ms gives the statistical spread.
metrics.cpu_cycles and metrics.ref_cycles show the raw instruction throughput.
metrics.frontend_stall_cycles reveals how often the CPU stalled waiting for the compiler front‑end.
metrics.cache_* metrics expose the cost of memory traffic.

When you later run w5.json, compare each of these fields to see how many cycles and how much time the warm‑up phase saved.

Common mistakes (teacher checklist)

Skipping warm‑up – leads to inflated times.
Using the first run as the final result – ignores transient effects.
Not keeping repeat count constant – conflates warm‑up and repeat variability.
Ignoring the execution_time_summary_ms – hides the spread.
Overlooking cache metrics – misses a major source of variance.

Try this next (homework)

Run the five‑warm‑up experiment on your machine. Compare the execution_time_ms and cpu_cycles between w0.json and w5.json. Note how the warm‑up reduces the mean time and the standard deviation.

Do this tonight — Episode 12 starts by assuming you did.

Closing

Warm‑up is not a luxury; it is the first honest measurement step. By controlling and comparing it, you turn a noisy signal into a reliable metric.

The series so far

Ep 1: Why a Single ./a.out Time Misleads Your Performance Claims (this article)
Ep 2: Warm vs Cold: Why a Single Trial Misleads Performance Claims
Ep 3: Screenshots Aren't Evidence: Use csperf for Real Performance Artifacts
Ep 4: Comparing Across Machines: Why Metadata Matters in csperf
Ep 5: csperf: A Lightweight Observatory for Honest Performance Tracking
Ep 6: csperf Doctor: First Command to Diagnose Toolchain Issues
Ep 7: csperf Quickstart: 30‑Second First Win
Ep 8: Reading csperf JSON: The Real Performance Artifact
Ep 9: Backends 101: Choosing the Right Measurement Surface
Ep 10: Stability Metrics: Min, Max, and Standard Deviation as First-Class Citizens
Ep 11: Warmup Deep Dive: What You’re Throwing Away and Why

Teaser next

Recall how we used metadata to compare machines (Ep 4) and now we’ll dive into locality: Row vs column access and how layout shows up in time (Ep 12).

Stability Metrics: Min, Max, and Standard Deviation as First-Class Citizens

compilersutra — Mon, 05 Oct 2026 12:30:07 +0000

Why this lesson exists

When you look at a single mean you miss the story of variability. A program that sometimes runs in 5 ms and sometimes in 10 ms can still have a mean of 7.5 ms – a perfectly plausible number that hides a performance cliff.

Recap — where we are in the series

Ep 1: Why a Single ./a.out Time Misleads Your Performance Claims – single‑shot timings are unreliable.
Ep 2: Warm vs Cold: Why a Single Trial Misleads Performance Claims – warm‑up and repeat runs give reproducible data.
Ep 3: Screenshots Aren't Evidence: Use csperf for Real Performance Artifacts – artifacts beat screenshots.
Ep 4: Comparing Across Machines: Why Metadata Matters in csperf – metadata enables honest cross‑machine comparison.
Ep 5: csperf: A Lightweight Observatory for Honest Performance Tracking – observatory commands give quick evidence.
Ep 6: csperf Doctor: First Command to Diagnose Toolchain Issues – validate toolchain before blaming performance.
Ep 7: csperf Quickstart: 30‑Second First Win – first reproducible artifact in half a minute.
Ep 8: Reading csperf JSON: The Real Performance Artifact – JSON is the source of truth.
Ep 9: Backends 101: Choosing the Right Measurement Surface – list backends, enumerate devices, run workloads on the correct surface.

The misconception

People often report only the mean execution time. They ignore that the mean can be the same for wildly different distributions – a narrow bell curve or a long tail.

What problem csperf solves (this episode's slice)

csperf now exposes min, max, and standard deviation in the summary block, making volatility visible. You can spot outliers, decide whether to discard them, and compare stability across compiler flags.

Mental model

Think of a performance run as a distribution of samples. The mean tells you the center, but the min/max and stdev tell you the spread. A low stdev means you can trust the mean; a high stdev means you should look deeper.

Lab: install and first commands

# Install csperf if you haven’t already
pip install csperf

Lab: what we ran on this machine

We used the matrix traversal example on an AMD Ryzen 7 9700X. The command sequence:

csperf run \
  --input examples/cpp/matrix_traversal.cpp \
  --backend cpu \
  --warmup-runs 3 \
  --repeat-runs 10 \
  --output results/stable.json
csperf profile results/stable.json

Results (real numbers only)

Metric	Value
execution_time_ms	5.1458
execution_time_summary_ms.min	5.132
execution_time_summary_ms.max	5.162
execution_time_summary_ms.mean	5.1458
execution_time_summary_ms.stdev	0.010358
cpu_cycles	7 317 425
ipc	1.68653

How to read the artifacts

Open results/stable.json. The execution_time_summary_ms object contains count, min, max, mean, median, and stdev. The metrics section shows raw counters. The artifacts field lists CSV/XLSX for spreadsheet analysis.

Common mistakes (teacher checklist)

Ignoring stdev – assume mean is enough.
Using too few repeats – 10 runs is a minimum for stable stats.
Discarding warm‑ups – warm‑up runs are intentional discards.
Comparing across machines without metadata – always check the hardware section.

Try this next (homework)

Run the same workload with --repeat-runs 20 and compare the stdev. Does it shrink? Document the change.

Do this tonight — Episode 11 starts by assuming you did.

Closing

Stability metrics let you see whether a mean is trustworthy. In the next episode we’ll dig into what warm‑up runs actually discard and why that matters.

The series so far

Ep 1: Why a Single ./a.out Time Misleads Your Performance Claims
Ep 2: Warm vs Cold: Why a Single Trial Misleads Performance Claims
Ep 3: Screenshots Aren't Evidence: Use csperf for Real Performance Artifacts
Ep 4: Comparing Across Machines: Why Metadata Matters in csperf
Ep 5: csperf: A Lightweight Observatory for Honest Performance Tracking
Ep 6: csperf Doctor: First Command to Diagnose Toolchain Issues
Ep 7: csperf Quickstart: 30‑Second First Win
Ep 8: Reading csperf JSON: The Real Performance Artifact
Ep 9: Backends 101: Choosing the Right Measurement Surface
Ep 10: Stability Metrics: Min, Max, and Standard Deviation as First-Class Citizens