Benchmarked new perl class feature with many class builders[Updated]

[Updated]

• (2023-08-22 21:30 JST) Added the following pull requests, then benchmarked. Thanks @bbrtj
  • Add XS variants for Moose and Moo
• (2023-08-22 15:30 JST) Added the following pull requests, then benchmarked. Thanks @bbrtj
  • Add immutable for Moose class #1
  • Add two extra fields for each class #2

What's this?

In Perl 5.38, the class feature was integrated into the core, so I compared it with many class builders. Ovid, who is main designer of new class feature, blogged the follwing thing:

Note that it’s not taking anything away from Perl; it’s adding a core object system for better memory consumption, performance, and elegance.
( https://ovid.github.io/articles/corinna-in-the-perl-core.html )

The class feature is still in the experimental stage, so I think the results may change depending on future development.

In the following execution environment, the class feature had the best memory efficiency, and the results of the constructor of an object and access to object fields are equivalent to those of objects blessed with array references.

Execution Environment

  ❯ inxi -SCm
  System:
    Host: kfly8.local Kernel: 22.3.0 arch: arm64 bits: 64 Console: s004 OS: Darwin 22.3.0
  Memory:
    System RAM: total: N/A available: N/A used: N/A
    RAM Report: missing: Required tool dmidecode not installed. Check --recommends
  CPU:
    Info: 10-core model: Apple M2 Pro bits: 64 type: MCP
    Speed: N/A min/max: N/A cores: No OS support for core speeds.

Benchmark memory size

Size	Compare	Title
135.8 KB	--	class feature (perl: 5.038000)
257.9 KB	89.9%	bless arrayref
265.7 KB	95.6%	Object::Pad@0.79
359.5 KB	164.8%	bless hashref
359.5 KB	164.8%	Moo@2.005004
359.5 KB	164.8%	Class::Accessor::Lite@0.08
359.5 KB	164.8%	Moose@2.2206
359.5 KB	164.8%	Moo@2.005004 (XSConstructor + XSAccessor)
359.5 KB	164.8%	Object::Tiny@1.09
359.5 KB	164.8%	Mouse@v2.5.10
359.5 KB	164.8%	Moose@2.2206 (XSAccessor)
359.5 KB	164.8%	Class::Tiny@1.008

This result were calculated using bench-size.pl.

Benchmark object constructors

Rate	Compare	Title
473/s	-61%	Class::Tiny@1.008
636/s	-47%	Moose@2.2206
675/s	-44%	Moose@2.2206 (XSAccessor)
720/s	-40%	Object::Pad@0.79
777/s	-35%	Moo@2.005004
1053/s	-12%	Moo@2.005004 (XSConstructor + XSAccessor)
1141/s	-5%	Mouse@v2.5.10
1189/s	-1%	bless arrayref
1199/s	--	class feature (perl: 5.038000)
1658/s	38%	bless hashref
1662/s	39%	Class::Accessor::Lite@0.08
1690/s	41%	Object::Tiny@1.09

This result were calculated using bench-new.pl.

Benchmark access to object fields

Rate	Compare	Title
81146/s	-41%	Object::Pad@0.79
97206/s	-29%	Class::Accessor::Lite@0.08
111203/s	-19%	Moose@2.2206
117507/s	-14%	Moo@2.005004
119218/s	-13%	bless hashref
121662/s	-11%	bless arrayref
130453/s	-4%	Class::Tiny@1.008
136532/s	--	class feature (perl: 5.038000)
165414/s	21%	Object::Tiny@1.09
214369/s	57%	Moose@2.2206 (XSAccessor)
229681/s	68%	Mouse@v2.5.10
273066/s	100%	Moo@2.005004 (XSConstructor + XSAccessor)

This result were calculated using bench-field.pl.

Repository

https://github.com/kfly8/bench-perl-class-builder

Comments

[bbrtj]

Thanks for the benchmarks. Your Moose case for constructor is flawed, as you didn't make the class meta immutable: metacpan.org/dist/Moose/view/lib/M...

Also, I would suggest having at least a couple of fields in your class. Having just one may favor some simpler solutions which may not scale well. In my own benchmarks it seemed like Moo constructor did not scale as good as Moose one, while being faster for super simple cases.

[kobaken]

Thanks for your comment!

I agree that Moose (and also Mouse) would be better if they were immutable.
And you are right, I think the benchmark would be better if it handled cases where the fields are complex.

I would be happy to send me a pull request!

[kobaken]

Thanks for your pull requests! I updated this post and repository.

[Paul Evans]

Something to keep in mind with the field access benchmarks is that the very design of Object::Pad and the core class system offers direct access to the fields of the object to the class's own methods, as simple lexicals. Code inside methods does not have to use the accessors just to see or modify those values. This is a feature unique to those two systems.

Therefore, in real-world code scenarios, the faster more-direct access these provide means that overall code can run faster in a way that simple benchmarks "from outside the class" such as your bench-field.pl do not manage to capture. It would not be easily possible to write a "universal" test case in the same style as your scripts to demonstrate this, exactly because of this unique ability.

As an example, consider this silly class:

class RGBColour {
  field $red :param;
  field $green :param;
  field $blue :param;

  method as_css () {
    sprintf "rgb(%f%%,%f%%,%f%%)", $red, $green, $blue;
  }
}

The ->as_css method here has direct access to the $red, $green and $blue lexicals, and so invoking that would only involve one actual method call at runtime. Whereas, this class based on any of the other systems would need code something like:

sub as_css ($self) {
  sprintf ..., $self->red, $self->green, $self->blue;
}

and thus any call to the outer method would involve three inner calls, a total of four. Once you get into real-world scenarios with code that contains real-world behaviour, the cost of all these little inner $self->field accessor calls quickly adds up and overtakes the original cost of the outside method call or even the object constructor.

Perhaps you could add a new category of benchmark, one that constructs an object with a few nontrivial fields and then performs a calculation method based on the values of those fields - feel free to steal my example code above if it helps. I'd be very interested to see benchmarks of that kind of case too.

[kobaken]

Thanks for your kind comments!

I will try the new benchmark case!

[Bruno Meneguele]

Hey, really nicely done! Hope we can keep this benchmark alive as long as possible!

One suggestion: it would be really nice to keep a "history" graph to follow core class enhancements in future Perl versions. AFAIK only the MVP was introduced to the core, but many optimizations are still to be done, including (as you mentioned in one of your comments) :reader and :writer accessors and, possibly, many other low-level object handling optimizations. So I really expect to see core class feature numbers to decrease, but who knows right? :D

[kobaken]

Thank you for your comment!

You are right, it would be nice to keep a history! I look forward to following future Perl updates.

[Dimitrios Kechagias]

It's interesting to see Mouse still leading the pack of the Moose compatible crowd. We switched to it a few years ago after benchmarking, and we've kept it even though Moo became more popular. It even manages to pull a win over the current version of class, but the latter is sure looking good otherwise, especially for something newly added to core!

[kobaken]

As you say, by far the best performing accessor was Mouse. I think the difference is whether the accessor works with XS or not. class feature will be adding a :reader modifier in the future that will make the accessor live, and I will try to benchmark it again when that is added.

[Ed Sabol]

Very nice!