DEV Community

Ragekill3377
Ragekill3377

Posted on

Really fast columnar analytics engine

it’s not a database, just a bytecode vm engine. has 2 modes: interpreter and JIT. I managed to get the JIT to run at 1.4-1.6 Billion elements per second processing on an intel core i3 4th gen which I think is pretty good. It depends on the cpu. I worked really hard on the JIT and currently it’s only x86_64 right now. to test it I benchmarked it against DuckDB on nyc taxi data. You can view it on the GitHub link. :)VoxelVM

Top comments (2)

Collapse
 
renolu profile image
Reno Lu

The wins in columnar come from vectorized execution and late materialization, plus encodings the CPU can scan without fully decoding. Curious where this sits, a from-scratch engine or a layer over a kernel like Arrow or DuckDB. What matters is selective scans, not full aggregates.

Collapse
 
ragekill3377 profile image
Ragekill3377

it’s a from scratch C++ engine. I did not make it a layer or wrapper over DuckDB or Arrow like you said. Voxel does everything you stated like vectorised execution, cpu being able to understand encodings without decoding them and late materialisation so im glad im good on those parts. Thanks to the JIT, it only executes the instructions so i have practically 0 overhead and it’s really fast. I should work on improving the interpreter further, but theres a limit to everything. Any tips?