DEV Community 👩‍💻👨‍💻

Discussion on: An unexpected performance regression

sharkdp profile image
David Peter Author

Thank you for the feedback! Yes - I am using Travis and Appveyor for CI. I'm sure we could somehow install hyperfine. But there are many open questions:

  • First and foremost: can we expect benchmarks on the CI VMs to be reproducible? Wouldn't they be influenced by the general workload on that particular machine?
  • Can we expect benchmarks on the CI VMs to be representative? What kind of filesystems are they using?
  • What would we do with the benchmark results? Compare them against some hard-coded time limits? Do we manually lower these hard-coded values for every performance improvement that we make?
rhymes profile image

Agreed on the first two, I wouldn't expect a docker machine in testing to behave similarly to a physical one but maybe you can rent some dedicated server? Though it could be expensive...

saidaspen profile image
Said Aspen • Edited on

How about running it on a VM with limited resources. Probably easier to get consistent performance, at least decoupled from the performance of the host system, if the VMs performance is limited. Simulated hardware?

Or maybe, a single purpose Raspberry Pi with some stripped down Linux variant?