loading...

re: An unexpected performance regression VIEW POST

TOP OF THREAD FULL DISCUSSION
re: Are you using some kind of CI tool? Maybe your hyperfine tool could be integrated to it? (I have very little experiencie with CI and rust, but this...
 

Thank you for the feedback! Yes - I am using Travis and Appveyor for CI. I'm sure we could somehow install hyperfine. But there are many open questions:

  • First and foremost: can we expect benchmarks on the CI VMs to be reproducible? Wouldn't they be influenced by the general workload on that particular machine?
  • Can we expect benchmarks on the CI VMs to be representative? What kind of filesystems are they using?
  • What would we do with the benchmark results? Compare them against some hard-coded time limits? Do we manually lower these hard-coded values for every performance improvement that we make?
 

How about running it on a VM with limited resources. Probably easier to get consistent performance, at least decoupled from the performance of the host system, if the VMs performance is limited. Simulated hardware?

Or maybe, a single purpose Raspberry Pi with some stripped down Linux variant?

 

Agreed on the first two, I wouldn't expect a docker machine in testing to behave similarly to a physical one but maybe you can rent some dedicated server? Though it could be expensive...

Code of Conduct Report abuse