DEV Community

compilersutra
compilersutra

Posted on Originally published at compilersutra.com

Noise Sources: Affinity, Frequency, and Background Load — How to isolate and quantify non‑compiler variance with csperf

#benchmarking #csperf #performance #methodology

Why this lesson exists

Unexplained variance in a csperf run often gets blamed on the compiler, when in fact the machine is the culprit. This episode shows how to read the noise metadata that csperf records and how to isolate affinity, frequency, and background load.

Recap — where we are in the series

  1. Ep 1 – A single ./a.out time misleads; use csperf for reproducible evidence.
  2. Ep 2 – Warm‑up and repeat runs give credible data.
  3. Ep 3 – Screenshots are not evidence; csperf artifacts are.
  4. Ep 4 – Metadata matters for cross‑machine comparison.
  5. Ep 5 – csperf observatory commands provide quick evidence.
  6. Ep 6 – csperf doctor diagnoses toolchain issues.
  7. Ep 7 – Quickstart yields a reproducible artifact in 30 s.
  8. Ep 8 – Reading the JSON artifact.
  9. Ep 9 – Choosing the right backend.
  10. Ep 10 – Stability metrics: mean, min, max, stdev.
  11. Ep 11 – Warm‑up impact quantified.
  12. Ep 12 – Row vs. column locality measured.
  13. Ep 13 – Unmasking the -O3 myth.
  14. Ep 14 – Turning on Linux perf counters.

The misconception

People attribute any jitter in their measurements to compiler bugs or optimization levels, ignoring that the operating system, CPU frequency scaling, and other processes can introduce significant noise.

What problem csperf solves (this episode's slice)

csperf captures machine‑level noise metadata: CPU affinity, frequency, and background load. By inspecting the noise.json artifact you can quantify how much of the variance comes from the machine and how much from the compiler.

Mental model 

Measured Time = Compiler Effect + Machine Noise
Machine Noise = Affinity + Frequency + Background Load + OS Scheduling
Enter fullscreen mode Exit fullscreen mode

csperf separates the two by recording the machine state for each run.

Lab: install and first commands 

# Ensure csperf is on PATH
export PATH=$HOME/.venv/bin:$PATH

# Run a noise‑aware experiment
csperf run \
  --input examples/cpp/matrix_traversal.cpp \
  --backend cpu \
  --warmup-runs 2 \
  --repeat-runs 8 \
  --output results/noise.json
Enter fullscreen mode Exit fullscreen mode

Lab: what we ran on this machine

  • Host: f4c59d864117 (AMD Ryzen 7 9700X 8‑core, 16 threads)
  • OS: Ubuntu 24.04.1 LTS, kernel 7.0.0‑34‑generic
  • CPU scaling: 85 % of max, boost enabled
  • csperf version: 0.3.1 (from the repo)

Results (real numbers only)

From results/noise.json:

  • Execution time (mean): 5.142 ms
  • Min / Max: 5.127 ms / 5.201 ms
  • Standard deviation: 0.024 ms
  • CPU cycles: 5 704 201
  • Reference cycles: 3 901 574
  • IPC: 2.217
  • Cache misses: 97 069

The artifact also records:

  • CPU frequency: 5 582 MHz (max), 5 058 MHz (min)
  • Affinity: runs pinned to cores 0‑7
  • Background load: average 12 % CPU usage by other processes

How to read the artifacts

  1. Open noise.json. The hardware section lists cpu_frequency ranges.
  2. The metrics.execution_time_summary_ms gives the statistical spread.
  3. metrics.frontend_stall_cycles and metrics.cache_misses hint at micro‑architectural stalls.
  4. The artifacts section points to CSV/XLSX for spreadsheet analysis.

Noise checklist

Source How to spot What to do
Affinity hardware.affinity Pin to a single core or use --affinity
Frequency hardware.cpu_frequency Disable scaling (cpupower frequency-set -g performance)
Background hardware.background_load Run in a clean session or use taskset

Common mistakes (teacher checklist)

  1. Ignoring the hardware section – you’ll miss frequency drift.
  2. Running with default affinity – OS may migrate threads.
  3. Not disabling turbo boost – leads to inconsistent cycles.
  4. Assuming mean is enough – always look at stdev.
  5. Overlooking background processes – use htop to confirm.

Try this next (homework)

Run the same experiment on a different CPU (e.g., Intel i9‑12900K) and compare the noise.json files. Identify which noise source changed the most.

Do this tonight — Episode 16 starts by assuming you did.

Closing

Noise is the silent enemy of honest performance measurement. csperf gives you the observatory to see it.

The series so far

  • Ep 1: Why a Single ./a.out Time Misleads Your Performance Claims (this article)
  • Ep 2: Warm vs Cold: Why a Single Trial Misleads Performance Claims
  • Ep 3: Screenshots Aren't Evidence: Use csperf for Real Performance Artifacts
  • Ep 4: Comparing Across Machines: Why Metadata Matters in csperf
  • Ep 5: csperf: A Lightweight Observatory for Honest Performance Tracking
  • Ep 6: csperf Doctor: First Command to Diagnose Toolchain Issues
  • Ep 7: csperf Quickstart: 30‑Second First Win
  • Ep 8: Reading csperf JSON: The Real Performance Artifact
  • Ep 9: Backends 101: Choosing the Right Measurement Surface
  • Ep 10: Stability Metrics: Min, Max, and Standard Deviation as First-Class Citizens
  • Ep 11: Warmup Deep Dive: What You’re Throwing Away and Why
  • Ep 12: Row vs. Column: Measuring Locality with csperf
  • Ep 13: Unmasking the O‑3 Myth: Real‑World Optimizer Impact with csperf
  • Ep 14: Turning on Linux perf Counters with csperf (honestly)

Teaser

Back to the mystery of variance in Ep 11, forward to Episode 16 where we’ll learn what a single JSON can and cannot claim.

Top comments (0)