<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Mukhammaddiyor Abdulazimov</title>
    <description>The latest articles on DEV Community by Mukhammaddiyor Abdulazimov (@diyor-ai).</description>
    <link>https://dev.to/diyor-ai</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4252140%2Fe8683c55-9901-4739-a3b3-6cb7b3b3d11b.jpg</url>
      <title>DEV Community: Mukhammaddiyor Abdulazimov</title>
      <link>https://dev.to/diyor-ai</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/diyor-ai"/>
    <language>en</language>
    <item>
      <title>Why my first CPU benchmark was too noisy to use, and how I fixed it</title>
      <dc:creator>Mukhammaddiyor Abdulazimov</dc:creator>
      <pubDate>Wed, 18 Nov 2026 14:52:42 +0000</pubDate>
      <link>https://dev.to/diyor-ai/why-my-first-cpu-benchmark-was-too-noisy-to-use-and-how-i-fixed-it-4m14</link>
      <guid>https://dev.to/diyor-ai/why-my-first-cpu-benchmark-was-too-noisy-to-use-and-how-i-fixed-it-4m14</guid>
      <description>&lt;p&gt;My first benchmark numbers looked reasonable, until I ran the same thing twice and got different answers. This post is about how I found out why, and what I changed before trusting any number.&lt;/p&gt;

&lt;p&gt;All numbers come from the repo's README (including its "First attempt (archived)" section) and &lt;code&gt;results/archive_summary.json&lt;/code&gt; in &lt;a href="https://github.com/diyor-ai/cpu-inference-bench" rel="noopener noreferrer"&gt;cpu-inference-bench&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The first attempt was invalid
&lt;/h2&gt;

&lt;p&gt;I started by running ResNet-18 on PyTorch, ONNX Runtime and OpenVINO with default settings on a laptop CPU (Intel i7-12650H). To check repeatability I ran each configuration twice. In 12 of 18 repeated rows, the two identical runs differed by more than 5%, up to 48.27% for one row (PyTorch, 12 threads, batch 1). The results were not usable as a comparison, so I archived them.&lt;/p&gt;

&lt;p&gt;A laptop has package power and thermal limits, so the clock changes during a run. Without control over that, a difference of a few percent between frameworks would be within the noise.&lt;/p&gt;

&lt;h2&gt;
  
  
  Finding the cause
&lt;/h2&gt;

&lt;p&gt;The first attempt used default frequency scaling, so the clock was free to move. This CPU has a sustained package power limit of 45 W and a short-term limit of 115 W, and power and thermal limits like these can change the clock during a run. Background services and other processes sharing the cores can add noise as well.&lt;/p&gt;

&lt;p&gt;The data from that attempt is archived locally and is not in the public repo, so I can't show its per-run frequencies or temperatures here. Instead of guessing which factor dominated, I changed the setup so that each one is either controlled or visible: every repeat now records the package temperature, the mean and minimum P-core frequency, and the kernel's thermal-throttle counters.&lt;/p&gt;

&lt;h2&gt;
  
  
  A pilot that was still not enough
&lt;/h2&gt;

&lt;p&gt;I capped the P-cores at 3000 MHz and ran a small pilot. Two of four rows still raised the kernel's thermal-throttle counters, so I chose 2600 MHz as a clock the cores could sustain.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I changed
&lt;/h2&gt;

&lt;p&gt;All reported runs use:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;a fixed 2600 MHz clock on the 6 P-cores, with SMT siblings offline, a sustained clock the cores can hold rather than the peak clock;&lt;/li&gt;
&lt;li&gt;pinned cores, AC power and background services stopped;&lt;/li&gt;
&lt;li&gt;a cooldown wait until the package temperature is below 60 °C before every repeat;&lt;/li&gt;
&lt;li&gt;interleaved rounds, so drift affects all frameworks equally;&lt;/li&gt;
&lt;li&gt;a stability rule: a row is stable only if its spread over 5 repeats is below 5% and no thermal-throttle counter increased.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Speedup ratios are printed only between two stable rows. Unstable rows are shown and marked, not dropped.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this gave me
&lt;/h2&gt;

&lt;p&gt;With these controls, ResNet-18 INT8 vs FP32 at batch 1 came out at 2.34× (PyTorch), 2.74× (ONNX Runtime) and 3.48× (OpenVINO, 2.6 ms). Five of 36 BERT rows and 2 of 18 ResNet rows (both FP32) were still unstable, and the README says so.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it does not show
&lt;/h2&gt;

&lt;p&gt;Thermal counters do not detect power-limit throttling, so I also checked the frequency columns. A VPN client was running during all runs and I did not measure its effect. This is one laptop and one campaign.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I took from it
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Run the same configuration twice before comparing anything.&lt;/li&gt;
&lt;li&gt;Report unstable results instead of hiding them.&lt;/li&gt;
&lt;li&gt;Write down what you did not control (here: power limits, VPN).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The code was written with Claude Code; the methodology, decisions and verification are mine.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://diyor.dev/blog/cpu-benchmark-noise/" rel="noopener noreferrer"&gt;diyor.dev&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>machinelearning</category>
      <category>python</category>
      <category>performance</category>
      <category>benchmarking</category>
    </item>
  </channel>
</rss>
