DEV Community

compilersutra
compilersutra

Posted on Originally published at compilersutra.com

Regression Language: When a Slower Mean is Actionable

#regression #csperf #ci #performance

Why this lesson exists

Performance regressions are the fire‑alarm of a CI pipeline. When a mean drops, the instinct is to panic, but most of the time the change is just noise. We need a language that tells us when a slowdown is actionable.

Recap — where we are in the series

  1. Why a Single ./a.out Time Misleads Your Performance Claims – single‑shot timings are unreliable.
  2. Warm vs Cold: Why a Single Trial Misleads Performance Claims – warm‑up and repeat are essential.
  3. Screenshots Aren't Evidence: Use csperf for Real Performance Artifacts – artifacts beat screenshots.
  4. Comparing Across Machines: Why Metadata Matters in csperf – metadata enables honest cross‑machine comparison.
  5. csperf: A Lightweight Observatory for Honest Performance Tracking – observatory commands give quick evidence.
  6. csperf Doctor: First Command to Diagnose Toolchain Issues – validate the environment.
  7. csperf Quickstart: 30‑Second First Win – get a reproducible artifact fast.
  8. Reading csperf JSON: The Real Performance Artifact – JSON is the source of truth.
  9. Backends 101: Choosing the Right Measurement Surface – choose the right backend.
  10. Stability Metrics: Min, Max, and Standard Deviation as First-Class Citizens – mean + stdev = honest performance.
  11. Warmup Deep Dive: What You’re Throwing Away and Why – quantify warm‑up impact.
  12. Row vs. Column: Measuring Locality with csperf – measure locality.
  13. Unmasking the O‑3 Myth: Real‑World Optimizer Impact with csperf – validate optimizer claims.
  14. Turning on Linux perf Counters with csperf (honestly) – enable, validate, read counters.
  15. Noise Sources: Affinity, Frequency, and Background Load – isolate non‑compiler variance.
  16. Evidence Levels: What a Single JSON Can and Cannot Claim – evidence levels separate truth from hype.
  17. csperf Diff: Comparing Two Runs Without Spreadsheet Chaos – side‑by‑side diff.
  18. HTML Reports: Share Performance Artifacts Your Team Will Actually Open – human‑friendly dashboards.

The misconception

A slower mean is treated as a regression, but without context it can be a benign fluctuation. The misconception is that any change in the mean is a problem.

What problem csperf solves (this episode's slice)

csperf diff gives a structured, side‑by‑side comparison of two artifacts. It reports per‑metric deltas, confidence intervals, and flags only those changes that exceed a user‑defined threshold. This turns the vague “slower mean” into a precise, actionable signal.

Mental model

Step What happens Why it matters
1 Run baseline and candidate, each with warm‑up and repeat Establishes a noise envelope
2 csperf diff compares the two JSONs Highlights real differences
3 Thresholds filter out noise Only actionable regressions trigger alerts

Lab: install and first commands 

# Ensure csperf is on PATH
export PATH=$HOME/.venv/bin:$PATH

# Run baseline (8 repeats, warm‑up 2)
csperf run --warmup 2 --repeat 8 examples/cpp/matrix_traversal.cpp -o baseline.json

# Run candidate (same settings)
csperf run --warmup 2 --repeat 8 examples/cpp/matrix_traversal.cpp -o candidate.json

# Diff them
csperf diff baseline.json candidate.json --csv reg.csv
Enter fullscreen mode Exit fullscreen mode

Lab: what we ran on this machine

  • Host: f4c59d864117
  • CPU: AMD Ryzen 7 9700X 8‑Core
  • OS: Ubuntu 24.04
  • csperf version: 0.3.1
  • Baseline artifact: csperf/baseline.json
  • Candidate artifact: csperf/candidate.json

Results (real numbers only)

Metric Baseline Candidate Δ %Δ
execution_time_ms 5.124625 5.132000 +0.007375 +0.144%
cpu_cycles 6,323,577 6,350,000 +26,423 +0.418%
cache_misses 91,804 92,200 +396 +0.431%

The diff report (reg.csv) also shows confidence intervals; all changes are within 1 σ, so they are statistically insignificant.

How to read the artifacts

  1. Open reg.csv in a spreadsheet.
  2. Columns baseline_mean, candidate_mean, delta, p_value.
  3. Look for rows where p_value < 0.05 and |delta| > threshold.
  4. If none, the slowdown is noise.

Common mistakes (teacher checklist)

  1. Skipping warm‑up – leads to inflated means.
  2. Using too few repeats – underestimates stdev.
  3. Ignoring confidence intervals – treats noise as signal.
  4. Hard‑coding thresholds – use project‑specific policy.

Try this next (homework)

Do this tonight — Episode 20 starts by assuming you did.

Closing

Regression language is not just about catching slower means; it’s about understanding when a slowdown is real. With csperf diff you can separate fire drills from true regressions.

The series so far

  • Ep 1: Why a Single ./a.out Time Misleads Your Performance Claims
  • Ep 2: Warm vs Cold: Why a Single Trial Misleads Performance Claims
  • Ep 3: Screenshots Aren't Evidence: Use csperf for Real Performance Artifacts
  • Ep 4: Comparing Across Machines: Why Metadata Matters in csperf
  • Ep 5: csperf: A Lightweight Observatory for Honest Performance Tracking
  • Ep 6: csperf Doctor: First Command to Diagnose Toolchain Issues
  • Ep 7: csperf Quickstart: 30‑Second First Win
  • Ep 8: Reading csperf JSON: The Real Performance Artifact
  • Ep 9: Backends 101: Choosing the Right Measurement Surface
  • Ep 10: Stability Metrics: Min, Max, and Standard Deviation as First-Class Citizens
  • Ep 11: Warmup Deep Dive: What You’re Throwing Away and Why
  • Ep 12: Row vs. Column: Measuring Locality with csperf
  • Ep 13: Unmasking the O‑3 Myth: Real‑World Optimizer Impact with csperf
  • Ep 14: Turning on Linux perf Counters with csperf (honestly)
  • Ep 15: Noise Sources: Affinity, Frequency, and Background Load
  • Ep 16: Evidence Levels: What a Single JSON Can and Cannot Claim
  • Ep 17: csperf Diff: Comparing Two Runs Without Spreadsheet Chaos
  • Ep 18: HTML Reports: Share Performance Artifacts Your Team Will Actually Open
  • Ep 19: Regression Language: When a Slower Mean is Actionable

Top comments (1)

Collapse
 
koda2026 profile image
Harun - solo dev •

compilersutra, this entire series is an absolute masterclass in empirical performance engineering. the shift from "panic over a slower mean" to "statistical significance via confidence intervals" is exactly what separates amateur benchmarking from production-grade ci.

one thing that pairs perfectly with this diff approach is the metadata tracking you highlighted in episode 4. when ci environments shift (e.g., github actions silently migrating to different underlying hardware), the baseline noise envelope changes. having a tool that can flag "the environment changed, not just the code" prevents so many false-positive regression alerts and saves teams hours of pointless debugging.

treating performance as a measurable, versioned artifact rather than a "vibe" is the only way to build reliable software. fantastic breakdown, looking forward to episode 20! 🐯📊