Sergey Nikolaev

Posted on Feb 10, 2023 • Edited on Nov 6 • Originally published at manticoresearch.com

Manticore Search 6

Today, the Manticore team is thrilled to announce the release of Manticore Search 6.0.0. This release has been a labor of love, taking over half a year, almost 800 commits, and more than 80,000 lines of code to complete. We would like to extend our deepest gratitude to all of our:

who have helped us along the way. Please join us in welcoming the new version of Manticore Search 6.0.0!

Manticore Buddy - a PHP extension of Manticore Search

Starting with this release, Manticore Search comes with Manticore Buddy, a sidecar daemon written in PHP that handles high-level functionality that does not require super low latency or high throughput. Manticore Buddy operates behind the scenes, and you may not even realize it is running. Although it is invisible to the end user, it was a significant challenge to make Manticore Buddy easily installable and compatible with the main C++-based daemon. This major change will allow the team to develop a wide range of new high-level features, such as shards orchestration, access control and authentication, and various integrations like mysqldump, DBeaver, Grafana mysql connector etc. For now it already handles

SHOW QUERIES allows you to view the queries currently running in Manticore Search
BACKUP enables you to create a backup of tables in your Manticore Search instance
and Auto schema allows you to create a table from the first inserted document without having to manually define the schema beforehand

Also, we eagerly look forward to your contributions. Now, if you want to extend Manticore with a new functionality and can write in PHP, it should be fairly easy. You can find instructions in the Manticore Buddy GitHub project.

Elasticsearch-compatible INSERTs and REPLACEs

You can now execute Elasticsearch-compatible insert and replace JSON queries, which enables the use of Manticore with tools such as Logstash and Filebeat, as well as other tools from the Beats family. This brings Manticore closer to becoming a replacement for Elasticsearch in the ELK stack. Our next goal is to complete the integration of Kibana/Opensearch Dashboards.

BTW, you may want to read this article which discusses Manticore as a faster alternative to Elasticsearch.

Secondary indexes are on by default

If you use our Manticore Columnar Library, which is highly recommended, secondary indexes are now ON by default. After their introduction in the previous major release they were significantly improved and we now believe having them enabled by default makes sense for most users. There’s also a new command ALTER TABLE table_name REBUILD SECONDARY to rebuild your secondary indexes, e.g. when you upgrade from a previous version.

Auto-schema

You can now skip creating a table. Just insert the first document, and Manticore will create the table automatically based on its fields. Read more about this in detail here.

Revamp of Cost-based optimizer

When Manticore executes a fullscan query, it can either use plain scan to check every document against the filters, or it can use additional data and/or algorithms to speed up the query execution. To decide which approach to take, Manticore uses a query cost-based optimizer (also known as “CBO” or “query optimizer”). This release includes a lot of improvements in how it works. For more information, read the documentation.

Telemetry

We are excited to announce the addition of telemetry in this release. This feature allows us to collect anonymous and depersonalized metrics that will help us improve the performance and user experience of our product. Rest assured, all data collected is completely anonymous and will not be linked to any personal information. This feature can be easily turned off in the settings if desired.

Tools for backup

It is crucial to regularly back up your tables in order to be able to recover them in the event of problems such as system crashes, hardware failure, or data corruption/loss for any reason. Backups are also essential before upgrading Manticore Search to a new version that introduces a change in table format, and can also be used to transfer your data to another system when you want to migrate to a new server.

The manticore-backup tool, which is included in the Manticore Search official packages, can help you automate the process of backing up tables for a Manticore instance running in RT mode, giving you peace of mind in case of any outages or crashes.

You can also back up your data through SQL by running the simple command BACKUP TO /path/to/backup.

SHOW QUERIES and KILL

The new SQL command, SHOW QUERIES, is an easy way to view running queries. It can be used in conjunction with the KILL command to terminate a long-running SELECT statement.

Dynamic max_matches and accurate aggregation

The max_matches option specifies the maximum number of matches that the server will retain in RAM for each table and return to the client. As of this release, its value is dynamic and can be increased automatically up to the max_matches_increase_threshold to improve the accuracy of aggregation queries. Additionally, it can also be automatically lowered if the number of unique values of a groupby attribute is less than the threshold.

arm64 support

Arm64 packages are now live. This means that users can now take advantage of the powerful performance and energy efficiency of arm64 processors when running Manticore Search and Manticore Columnar Library. With the growing popularity of arm64 architecture, it’s more important than ever for software providers to offer support for this platform. By offering arm64 packages in our new release, we are ensuring that our users can stay current and take advantage of the latest hardware advancements.