DEV Community

compilersutra
compilersutra

Posted on Originally published at compilersutra.com

CI‑Shaped Workflows: Store What Matters, Fail on What Counts

#ci #csperf #automation #performance

Why this lesson exists

CI pipelines that only surface a mean value are blind to the real story of your code’s performance. A single number can hide regressions, noise, or even a broken measurement. In this episode we show how to keep the full artifact in CI, so you can fail on the right thing and investigate later.

Recap — where we are in the series

  • Ep 1: single‑shot timings are misleading; use csperf for reproducible evidence.
  • Ep 2: warm‑up and repeat runs give credible data.
  • Ep 3: screenshots mislead; csperf provides metadata‑rich artifacts.
  • Ep 4: metadata enables cross‑machine comparison.
  • Ep 5: observatory commands give quick, reliable evidence.
  • Ep 6: doctor command validates compiler environment.
  • Ep 7: quickstart produces a reproducible artifact in 30 s.
  • Ep 8: JSON is the source of truth.
  • Ep 9: choose the right backend.
  • Ep 10: expose min, max, stdev for honest performance.
  • Ep 11: quantify warm‑up impact.
  • Ep 12: measure locality.
  • Ep 13: validate optimizer claims.
  • Ep 14: enable and read Linux perf counters.
  • Ep 15: isolate affinity, frequency, background load.
  • Ep 16: evidence levels separate truth from hype.
  • Ep 17: diff two runs without spreadsheets.
  • Ep 18: HTML dashboards for collaboration.
  • Ep 19: isolate actionable regressions from noise.

The misconception

Many teams run csperf run … in CI and only look at the printed mean. They assume the mean is enough to decide whether a change is acceptable. In reality, the mean can be identical while the variance has exploded, or a single outlier can skew the mean without indicating a systemic problem.

What problem csperf solves (this episode's slice)

csperf lets you:

  1. Store the entire JSON artifact in your CI artifact store.
  2. Inspect the artifact later with csperf profile.
  3. Fail the job if any metric deviates beyond a threshold.
  4. Keep the full metadata so you can reproduce or audit the run.

Mental model

Think of CI as a camera: you want the full frame, not just a thumbnail. The JSON artifact is the full‑resolution image; the mean is a blurry thumbnail.

Lab: install and first commands 

# Ensure csperf is in your PATH
export PATH=$HOME/.venv/bin:$PATH
# Run the workload and capture the artifact
csperf run --input examples/cpp/matrix_traversal.cpp --backend cpu --output results/ci.json
# Profile the artifact to see the raw numbers
csperf profile results/ci.json --json
Enter fullscreen mode Exit fullscreen mode

Lab: what we ran on this machine

We ran the matrix traversal example on an AMD Ryzen 7 9700X (16 cores) with Linux 7.0.0‑34‑generic. The artifact lives at results/ci.json.

Results (real numbers only)

Metric Value
execution_time_ms 5.139
execution_time_summary_ms.min 5.121
execution_time_summary_ms.max 5.153
execution_time_summary_ms.mean 5.139
execution_time_summary_ms.stdev 0.011597
cpu_cycles 5,916,273
ipc 2.080232

These numbers come directly from the JSON artifact shown in the metrics section.

How to read the artifacts

  • artifacts.json contains the full run data.
  • artifacts.csv is a tabular view.
  • artifacts.xlsx can be opened in Excel.
  • artifacts.binary is the compiled binary.

Use csperf profile to pretty‑print the JSON or csperf diff to compare two artifacts.

Common mistakes (teacher checklist)

  1. Only printing the mean – always keep the JSON.
  2. Ignoring stdev – a low mean with high stdev is a red flag.
  3. Not pinning the artifact path – CI should archive the exact file.
  4. Running without warm‑up – the first run can be anomalous.
  5. Failing on the wrong metric – decide which metric matters for your SLA.

Try this next (homework)

Do this tonight — Episode 21 starts by assuming you did.

Closing

Storing the full artifact turns CI from a black box into a transparent audit trail. Next we’ll learn how to pin a golden JSON as a baseline and avoid lying to ourselves.

The series so far

  • Ep 1: Why a Single ./a.out Time Misleads Your Performance Claims
  • Ep 2: Warm vs Cold: Why a Single Trial Misleads Performance Claims
  • Ep 3: Screenshots Aren't Evidence: Use csperf for Real Performance Artifacts
  • Ep 4: Comparing Across Machines: Why Metadata Matters in csperf
  • Ep 5: csperf: A Lightweight Observatory for Honest Performance Tracking
  • Ep 6: csperf Doctor: First Command to Diagnose Toolchain Issues
  • Ep 7: csperf Quickstart: 30‑Second First Win
  • Ep 8: Reading csperf JSON: The Real Performance Artifact
  • Ep 9: Backends 101: Choosing the Right Measurement Surface
  • Ep 10: Stability Metrics: Min, Max, and Standard Deviation as First-Class Citizens
  • Ep 11: Warmup Deep Dive: What You’re Throwing Away and Why
  • Ep 12: Row vs. Column: Measuring Locality with csperf
  • Ep 13: Unmasking the O‑3 Myth: Real‑World Optimizer Impact with csperf
  • Ep 14: Turning on Linux perf Counters with csperf (honestly)
  • Ep 15: Noise Sources: Affinity, Frequency, and Background Load — How to isolate and quantify non‑compiler variance with csperf
  • Ep 16: Evidence Levels: What a Single JSON Can and Cannot Claim
  • Ep 17: csperf Diff: Comparing Two Runs Without Spreadsheet Chaos
  • Ep 18: HTML Reports: Share Performance Artifacts Your Team Will Actually Open
  • Ep 19: Regression Language: When a Slower Mean is Actionable
  • Ep 20: CI‑Shaped Workflows: Store What Matters, Fail on What Counts

Top comments (0)