DEV Community

Zenql
Zenql

Posted on

50 Million Records in Under One Second — Inside ZenQL’s New Collection Engine

#zenql #go #collections #linq

With the release of version 1.7.9, ZenQL’s Collection API, Thor, received substantial performance improvements, largely driven by a series of memory optimization enhancements. These changes reduced unnecessary allocations, improved data handling efficiency, and significantly accelerated query execution, particularly when working with large in-memory datasets.

From the early stages of development, we established a baseline benchmark to measure both correctness and performance consistently: filtering a slice of 50 million items and validating the result.

The original Collection API completed this benchmark in approximately 9 seconds.

Later, we introduced Thor as a replacement for the default Collection API, reducing the benchmark time to around 4 seconds.

With the latest round of memory optimizations and internal improvements, Thor now completes the same benchmark in less than one second.

benchmark:

  goos: linux
  goarch: amd64
  pkg: github.com/malikhan-dev/zenql/collections/Thor
  cpu: 12th Gen Intel(R) Core(TM) i7-12700H
  BenchmarkQueryEngine
  BenchmarkQueryEngine-20               88          13636313 ns/op         22727310 B/op          0 allocs/op
Enter fullscreen mode Exit fullscreen mode 
at Thor_Engine__test.go 

func BenchmarkQueryEngine(b *testing.B) {

    result := From(&items).Where(func(search ComplexObjectToSearch) bool {
        return search.Name == "Jane" && search.Flag == false
    }).Collect()

    result2 := From(&result).Any(func(search ComplexObjectToSearch) bool {
        return (search.Name != "Jane") || (search.Flag != false)
    }).Assert()

    if result2 {
        b.Error("result should be false")
    }

}
Enter fullscreen mode Exit fullscreen mode

Join the Journey

We are committed to making ZenQL the fastest and most developer-friendly query engine for Go. As we continue to grow and push the boundaries of performance, we need your support!

If you find ZenQL useful, please star our repository on GitHub. Your support helps us reach more developers and keep the project moving forward.

Thank you!

Top comments (0)