While measuring how my search engine scales with thread count (results in the companion article), I threw away my entire first data set. Here's why — and if you ever benchmark anything on a fanless laptop, this is not someone else's problem.
The first data set was clean, and wrong
I measured 1 through 10 threads, in order. The curve looked beautiful, including a shiny record: "11 threads, fastest at 5.2 seconds." Right before writing the article, I re-ran the same conditions as a sanity check.
The same conditions came back up to 54% different. 4 threads: 11.0s became 9.0s. 11 threads: 5.2s became 8.0s. The record run did not reproduce.
No rogue background process. The culprit was the chassis.
The culprit: thermal history on a fanless machine
The test machine is a MacBook Air (Apple M4) — fanless. Under sustained load it heats up and thermal-throttles. Which means: every measurement depends on what you measured just before it.
The nasty part is that this is not random noise — it's a systematic error. Sweep 1→2→...→10 in order and the later thread counts always run on a hotter chip. The high-thread results are penalized, consistently, in the same direction. Averaging repeated sweeps doesn't remove it, because the bias repeats too. That "fastest 5.2s at 11 threads"? A single lucky run on a cold chip. The cleaner the graph, the more suspicious you should be.
The fix: round-robin
Stop measuring each thread count in a block. Instead: sweep 1-16 threads once, repeat the whole sweep for 4 rounds, and take the per-thread-count median across rounds. Every thread count now experiences cold and hot states equally; the thermal history no longer piles onto specific configurations.
The corrected protocol (128 measurements) produced the companion article's conclusion: a plateau at 8 threads, 3.0x. The 5.2s ghost vanished; 7.4s became a reproducible floor.
[Update, Aug 29, 2026] Being precise about "experiences cold and hot equally"
The sentence above is only true at the round level. Within a sweep it does not hold. Re-reading the results CSV (
results_threads16_rr.csv, 128 rows) shows that all four rounds swept 1→16 in ascending order. Nothing was randomized, and the direction was never alternated. So within a single sweep, the higher thread counts are still measured on a hotter chip. That bias remains.Round totals for the search term "Tokyo": round1 127.1 s, round2 155.1 s, round3 152.4 s, round4 161.7 s. Only the first sweep runs on a cold machine. The round-to-round spread is 1% at 1 thread, 55% at 8 threads, and 56% at 13 threads — the heavier the load, the more heat shows up.
So what round-robin plus median actually achieved is three things: (1) it stopped block-measuring one configuration at a time, so no single configuration builds up its own heat; (2) every thread count now gets the cold first round and the hot fourth round exactly once; (3) the median — with 4 points, the mean of the middle two — drops both the coldest and the hottest value.
The ascending-order bias inside each sweep is still there. Removing it would require alternating the sweep direction per round, or randomizing the order outright. The next run will do that.
One more correction: "if heat or any other disturbance were still leaking in, there's no reason two independent series would line up that well" was overstated. Both series were measured inside the same sweeps, under the same thermal history — if both are skewed in the same direction, they can line up while still being skewed. What this establishes is that the re-run is reproducible, not that the disturbance is gone.
The article's conclusion — a plateau at 8 threads, 3.0x over a single thread, thread count decided automatically — is unchanged.
How do you know the fix worked?
Claiming "the new numbers are right" needs evidence. Mine: two search patterns with hit counts differing by more than an order of magnitude (10,967 vs. 722 hits) produced nearly identical scaling curves. If heat or any other disturbance were still leaking in, there's no reason two independent series would line up that well.
I kept the first CSV for reference but decided it never appears in an article. Publish a retracted number "for reference" and someday someone will cite it.
Lessons
- On a fanless machine, measurement order leaks into results. A sequential sweep systematically penalizes the later configurations
- Distrust beautiful single records. One cold run masquerades as a best case
- Round-robin + median distributes cold and hot fairly across all configurations
- Validate with independent series, not repetition. Re-running the same biased protocol reproduces the same bias
The benchmark's enemy wasn't a rival product. It was my own chassis.
From the developer: a list of my apps, Kindle books and open-source projects is on GitHub: amru195704.
A note
The information in this article is provided for reference purposes only, and its accuracy or completeness is not guaranteed. If you notice any errors or inaccuracies, please let us know in the comments and we will review and correct them.
Top comments (0)