DEV Community: Sergey Nikolaev

Manticore Search 29.9.0: Chunked auto-embeddings and mmap columnar access

Sergey Nikolaev — Fri, 11 Sep 2026 16:05:04 +0000

Originally published on https://mnt.cr/go/c72RPs on September 11, 2026

Manticore Search 29.9.0: Chunked auto-embeddings and mmap columnar access

Manticore Search 29.9.0 adds chunked and multi-vector auto-embeddings, embedding input limits, UTF-8 identifiers, mmap columnar access by default, and fixes for hybrid search, KNN, bulk ingestion, and grouping.

[Manticore Search 29.9.0](https://mnt.cr/go/LfXmU1 has been released. The largest change is in auto-embeddings: long documents can now be split into searchable chunks, and a document can keep several vectors instead of compressing all of its content into one. This release also makes mmap the default access mode for columnar attributes, adds safer controls for embedding workloads, UTF-8 identifiers, better backup support, and fixes across hybrid search, KNN, bulk ingestion, grouped search, and schema changes.

This post covers everything shipped after 29.0.2, from 29.0.3 through 29.9.0.

❤️ We’d like to thank [@tudorvasinca](https://mnt.cr/go/Vn5RXy for their work on [PR #4857](https://mnt.cr/go/DdBMb7, [PR #4859](https://mnt.cr/go/jZePzd, and [PR #4873](https://mnt.cr/go/eQNGL6

Upgrade notes

There is no release-wide mandatory data migration. Existing tables and configuration can be upgraded normally, but a few fixes need follow-up if the earlier behavior already affected your data:

German sharp-s normalization is opt-in. Switching an existing table to lemmatize_de_v2 or lemmatize_de_v2_all changes indexed terms, so rebuild a plain table or replay documents into a new RT table.
Authenticated BACKUP introduces the backup authorization action. Before downgrading to 29.3.12 or earlier, remove any backup grants.
Columnar attributes now use mmap by default instead of buffered file reads. Set access_columnar_attrs='file' explicitly if you need to retain the previous access mode.

Long documents can keep more than one embedding

The old auto-embedding path produced one vector per document and truncated text that did not fit the model input window. That works for titles and short descriptions, but it means a relevant paragraph near the end of a long article may never reach the index.

Manticore Search now supports five [chunking strategies](https://mnt.cr/go/BXIJ8c

truncate keeps the previous behavior and remains the default.
mean embeds every chunk and averages the results into one vector.
fixed splits text into fixed-size token windows.
recursive prefers paragraph, line, sentence, and space boundaries in that order.
sentence groups complete sentences up to the configured limit.

The multi-vector strategies use [float_vector_array](https://mnt.cr/go/Y31JKu Each chunk competes independently during KNN search, but Manticore returns the document once and uses its closest chunk for knn_dist():

CREATE TABLE articles (
  title text,
  content text,
  chunks float_vector_array knn_type='hnsw' hnsw_similarity='cosine'
    model_name='Xenova/all-MiniLM-L6-v2' from='title,content'
    chunk_strategy='sentence' max_tokens='256' overlap_tokens='32'
);

INSERT INTO articles (id, title, content)
VALUES (1, 'Rotating certificates', 'A long guide with many sections ...');

SELECT id, knn_dist()
FROM articles
WHERE knn(chunks, 5, 'how do I rotate a certificate');

MAX_TOKENS, OVERLAP_TOKENS, and MAX_CHUNKS control chunk size, shared context at boundaries, and the maximum vector count. Declare a model-backed float_vector_array when creating the table: adding one later with ALTER TABLE ... ADD COLUMN and rebuilding its embeddings are not supported yet.

Put a ceiling on local embedding work

Long-context models can make a single large input unexpectedly expensive, especially on CPU. The new [MAX_INPUT_TOKENS](https://mnt.cr/go/sX4dok column option caps how much of each input is sent to a local embedding model:

ALTER TABLE articles
MODIFY COLUMN chunks MAX_INPUT_TOKENS='512';

The change applies to embeddings generated afterward; existing vectors stay as they are. Set it during CREATE TABLE or change it later without re-embedding the table. A value of 0, or leaving the option out, uses the model's own limit.

This release also fixes two less visible problems around this path. Models no longer remain cached after an embedding column is modified, and configurations with different API_TIMEOUT or MAX_INPUT_TOKENS values no longer collide and reuse the wrong cached model.

UTF-8 identifiers and better German matching

Table, field, and attribute names now follow one consistent [safe UTF-8 identifier syntax](https://mnt.cr/go/mfKS2T RT, percolate, distributed, template, and plain tables can use localized names such as Chinese or Cyrillic identifiers across DDL, expressions, field selectors, and inferred source schemas.

German AOT morphology also gains opt-in sharp-s normalization. With charset_table=non_cont,german and morphology=lemmatize_de_v2 or lemmatize_de_v2_all, forms such as Straße, Strasse, and STRAẞE match in ordinary whole-word searches. If index_exact_words=1 is enabled, exact-word queries can still distinguish the ß and ss forms.

Better load testing and backups

[manticore-load](https://mnt.cr/go/cnoLS6 can now benchmark through the HTTP JSON API with --http. Writes use /bulk, searches use /search, and --table selects the target table. Its reports now include local searchd RSS during the run plus peak RSS, disk, and CPU statistics at the end, including aggregate monitoring for multi-command workloads.

[Manticore Backup](https://mnt.cr/go/t4acW4 now works with authenticated Manticore Search installations through username/password or bearer-token credentials. SQL [BACKUP](https://mnt.cr/go/6dxdrT has a dedicated authorization action and checks read access to the selected tables.

S3 backups and restores can also use the AWS SDK credential provider chain when static keys are not set. That includes IRSA, shared credentials, ECS task roles, and EC2 instance profiles. Temporary credentials can supply AWS_SESSION_TOKEN.

Columnar attributes use mmap by default

Manticore Search now defaults [access_columnar_attrs](https://mnt.cr/go/sdnJ5l to mmap. The operating system maps and caches *.spc columnar-attribute files on demand, without prereading the whole file at startup. The previous buffered path remains available with access_columnar_attrs='file'.

ALTER TABLE also reopens replaced columnar storage with the configured access mode, so an altered table no longer falls back to a different reader than the one requested.

Vector and grouped-search fixes

Several fixes target queries that were valid but could return incomplete results or fail under a particular table layout:

Distributed and sharded KNN queries with a local shard no longer rescore merged 1-bit-quantized results twice, which could crash the coordinator or return the wrong nearest neighbor. ([Issue #4791](https://mnt.cr/go/LfXW1t
KNN queries with additional filters avoid a redundant knn_dist prefilter when HNSW already excludes documents without vectors. ([PR #4861](https://mnt.cr/go/bZtpIJ
LENGTH() on a float_vector_array now reports the number of vectors rather than its internal storage-word count. ([PR #4879](https://mnt.cr/go/8be2oV
Hybrid search with GROUP BY retains all buckets, including MVA groups, and follows both final and within-group ordering. ([Issue #4639](https://mnt.cr/go/b5Hmg8
Hybrid filters on weight() and expressions or aliases derived from it now run after fusion against the final text weight instead of being ignored. Weight-dependent filters inside OR trees remain unsupported and return an explicit error. ([Issue #4889](https://mnt.cr/go/OMu9qt
Multi-chunk RT grouping no longer risks duplicate groups, incorrect split counts, or a hang while ordering COUNT(DISTINCT ...) results. ([Issue #4856](https://mnt.cr/go/KtrFXK
Document-ID filters whose signed representation is negative now follow the lookup index's unsigned ordering. ([Issue #4774](https://mnt.cr/go/nEO1vc

There are crash fixes here too: a second hybrid-search statement in a multi-statement request ([PR #4864](https://mnt.cr/go/RJ5hGi, distributed JSON aggregation sorted by a string attribute ([Issue #4822](https://mnt.cr/go/Jdpl0M, and dropping a table during auto-embedding precommit ([Issue #4860](https://mnt.cr/go/Edp7gc are all handled safely now.

Bulk ingestion behaves predictably

Elasticsearch-compatible /_bulk requests now return HTTP 200 once a batch has been processed, while individual failures remain visible through errors: true and per-item statuses. Duplicate create actions return per-item 409 version_conflict_engine_exception errors, including duplicates within the same batch. This prevents clients such as Fluent Bit from retrying writes that already succeeded.

Fixed-length gzip-compressed /bulk bodies are also decoded correctly when they arrive across multiple socket reads. And if native bulk processing fails because the target table does not exist, the request can again reach Manticore's auto-schema fallback with valid NDJSON while preserving the expected bulk response envelope.

More reliability fixes

The rest of the release closes a broad set of operational and compatibility problems:

searchd --stopwait no longer hangs while a sharded table is being rebalanced after a node rejoins. ([Issue #3905](https://mnt.cr/go/ZVFkNe
Compatible older binlogs replay safely during upgrade and completed RT chunks are published before clean shutdown. ([Issue #4808](https://mnt.cr/go/O0ZRHJ Fatal replay diagnostics also name the relevant recovery flag. ([Issue #4811](https://mnt.cr/go/pzGwEm
indexer creates missing parent directories for plain-table paths when the nearest existing parent is writable. ([Issue #4793](https://mnt.cr/go/eIFSQf
UUID document IDs no longer cause stored text fields to come back empty. ([Issue #4833](https://mnt.cr/go/otelYC
ALTER TABLE ... RENAME preserves hidden remote embedding API keys without exposing them in SHOW CREATE TABLE. ([Issue #4842](https://mnt.cr/go/XkT4iw
Sequel Ace 5.3.1+ compatibility probes work again. ([Issue #4828](https://mnt.cr/go/zjvXbT
JSON /search keeps distances for negated NEAR and proximity operators. ([Issue #4784](https://mnt.cr/go/NgB0Q7
Internal string-sort helper columns no longer leak from LEFT JOIN results. ([Issue #4788](https://mnt.cr/go/rxNeFg
Malformed binary API SEARCH element counts are rejected instead of terminating searchd. ([PR #4790](https://mnt.cr/go/ufPVLK

For the complete list, see the [Version 29.9.0 changelog](https://mnt.cr/go/gl5v6N

Get Manticore Search 29.9.0

Install or upgrade Manticore Search with the [installation guide](https://mnt.cr/go/XPyNrv Review the upgrade notes above if you use columnar attributes, German AOT morphology, or authenticated backups.

Need help or want to connect?

Join our [Slack](https://mnt.cr/go/v4HPeK
Visit the [Forum](https://mnt.cr/go/R7u2vk
Report issues or suggest features on [GitHub](https://mnt.cr/go/CURvTu
Email us at contact@manticoresearch.com

Explore Manticore Data with OpenSearch Dashboards

Sergey Nikolaev — Tue, 08 Sep 2026 10:00:00 +0000

Originally published on https://manticoresearch.com/blog/manticore-opensearch-dashboards-integration/ on September 8, 2026

Explore Manticore Data with OpenSearch Dashboards

Connect OpenSearch Dashboards to Manticore Search, explore data in Discover, and build visualizations and dashboards using a familiar interface.

OpenSearch Dashboards provides a familiar visual interface for exploring data, building charts, and assembling interactive dashboards. Although it is normally paired with OpenSearch, you can also connect it directly to Manticore Search.

The integration lets you keep Manticore as the search and analytics backend while using Discover, Visualize, and Dashboards as the user interface. This is especially useful for log and event data: tools such as Logstash, Filebeat, Fluent Bit, and Vector can send data to Manticore, while OpenSearch Dashboards gives your team a convenient way to inspect it.

In this tutorial, we will configure the connection, add a small sample dataset, explore it in Discover, and create a visualization.

How the integration works

OpenSearch Dashboards communicates with its backend through an HTTP API. Manticore exposes a compatible API on its HTTP listener, which uses port 9308 by default.

The compatibility layer is provided by the EmulateElastic plugin in Manticore Buddy. It handles the requests that OpenSearch Dashboards needs for startup, index discovery, searches, filters, and supported aggregations. Buddy normally starts automatically with searchd, so a standard Manticore installation already has the required component.

This is an API compatibility integration, not an embedded OpenSearch cluster. Manticore stores and queries the data, and OpenSearch Dashboards provides the visual interface.

Prerequisites

For this walkthrough, you need:

A running Manticore Search instance with its HTTP endpoint available at http://localhost:9308
Manticore Buddy installed and running
OpenSearch Dashboards 3.4.0, the currently tested and recommended version
Manticore running in real-time mode

Other OpenSearch Dashboards versions may work, but they have not been tested to the same extent. The version reported by Manticore must also match the version of OpenSearch Dashboards.

Step 1: Configure Manticore

Open your Manticore configuration file and make sure that the HTTP listener is enabled. Set kibana_version_string to the version of OpenSearch Dashboards that you plan to run:

searchd {
    listen = 127.0.0.1:9308:http
    pid_file = /var/run/manticore/searchd.pid
    data_dir = /var/lib/manticore
    kibana_version_string = 3.4.0
}

The name kibana_version_string is retained for compatibility with the existing Elasticsearch-style API. OpenSearch Dashboards checks the backend version during startup, so a mismatch can produce warnings or prevent the application from starting.

Restart Manticore after changing the configuration.

If Manticore and OpenSearch Dashboards run on different hosts or in different containers, do not bind the HTTP listener only to 127.0.0.1. Bind it to an address that OpenSearch Dashboards can reach, and restrict access with your network or firewall configuration.

Step 2: Configure OpenSearch Dashboards

Open opensearch_dashboards.yml. In a tarball installation, it is usually located at config/opensearch_dashboards.yml; packages may place it at /etc/opensearch-dashboards/opensearch_dashboards.yml.

Point opensearch.hosts to Manticore's HTTP endpoint:

opensearch.hosts: ["http://localhost:9308"]

Manticore does not provide the OpenSearch Security plugin. The corresponding Dashboards plugin must therefore be disabled.

For a tarball installation, stop OpenSearch Dashboards and remove the plugin:

./bin/opensearch-dashboards-plugin remove securityDashboards

If you run OpenSearch Dashboards in Docker, provide both settings through environment variables:

environment:
  OPENSEARCH_HOSTS: '["http://manticore:9308"]'
  DISABLE_SECURITY_DASHBOARDS_PLUGIN: "true"

Here, manticore is the hostname or container name reachable from the OpenSearch Dashboards container. Using localhost inside that container would refer to the container itself, not to Manticore.

Start OpenSearch Dashboards and open http://localhost:5601.

Step 3: Add sample data to Manticore

If you already have a real-time table, you can skip this step. Otherwise, connect to Manticore through its MySQL-compatible port:

mysql -h127.0.0.1 -P9306

Create a table for a small set of application events:

CREATE TABLE app_events (
    message text,
    service string,
    status_code uint,
    response_time float,
    event_time timestamp
);

Add a few documents:

INSERT INTO app_events
    (message, service, status_code, response_time, event_time)
VALUES
    ('Request completed', 'catalog', 200, 0.12, 1786924800),
    ('Request completed', 'checkout', 200, 0.31, 1786928400),
    ('Upstream timeout', 'checkout', 504, 2.75, 1786932000),
    ('Product not found', 'catalog', 404, 0.08, 1786935600),
    ('Request completed', 'catalog', 200, 0.15, 1786939200),
    ('Payment rejected', 'payments', 422, 0.44, 1786942800);

This schema gives us text to search, dimensions to group by, metrics to aggregate, and a timestamp for time-based charts.

Step 4: Create an index pattern

OpenSearch Dashboards uses an index pattern to select the data source shown in Discover and visualizations. In this integration, the pattern matches a Manticore table name.

Open Management > Dashboards Management.
Go to Index Patterns and choose Create index pattern.
Enter app_events.
Select event_time as the time field.
Save the index pattern.

The app_events fields should now be available in OpenSearch Dashboards.

Step 5: Explore the data in Discover

Open Discover and select the app_events index pattern. You can now:

Inspect individual documents
Search the message field
Filter by fields such as service or status_code
Change the time range
Add and remove columns from the results

For example, add a filter where status_code is greater than or equal to 400 to focus on failed requests. You can then add service, message, status_code, and response_time as columns to get a compact error view.

Simple Dashboard Query Language searches work with the integration. Advanced DQL features—including nested field searches, regular expressions, fuzzy searches, proximity searches, and term boosting—may not be compatible with Manticore.

Step 6: Build a visualization

Open Visualize, create a new visualization, and choose the app_events index pattern. OpenSearch Dashboards can use the following bucket aggregations with Manticore:

terms
histogram
date_histogram
range
date_range

Supported metric aggregations include:

max
min
sum
avg

As a first chart, create a bar chart of events by service:

Use Count as the metric.
Add a Terms bucket using the service field.
Apply the changes.
Save the visualization as Events by service.

You can also create a time-series chart with a Date Histogram on event_time, or chart average latency by using the avg metric on response_time.

After saving your visualizations, open Dashboards, create a dashboard, and add them. Filters applied at dashboard level let you focus the whole view on one service, status code, or time range.

What is supported

The integration covers the main data exploration workflow:

Searching and filtering documents in Discover
Creating index patterns for Manticore tables
Building visualizations with supported bucket and metric aggregations
Saving visualizations and combining them into dashboards
Managing index patterns and saved objects in Dashboards Management

Manticore also emulates the stack-level requests needed during OpenSearch Dashboards startup, including node version information, cluster settings, configuration objects, and index listings.

Current limitations

OpenSearch Dashboards includes many features that depend on OpenSearch-specific APIs or field types. Those features are outside the scope of this integration.

Unsupported field types include:

Geographic and Cartesian fields such as geo_point, geo_shape, xy_point, and xy_shape
Range types such as integer_range, ip_range, and date_range
Specialized search types such as semantic, rank_feature, and percolator
OpenSearch vector types such as knn_vector and sparse_vector
Relational types such as nested and join
Advanced string types such as completion and search_as_you_type

Plain text and keyword fields are supported. Manticore's own vector search remains available through its SQL and JSON APIs, but OpenSearch Dashboards cannot represent it through OpenSearch's vector field type.

Nested aggregations—an aggs block inside another aggs block—are not supported. Metric functions are limited to those implemented by Manticore.

OpenSearch-specific applications and administration tools are also not available, including:

Geospatial visualizations
Observability and trace analytics
Alerting and Anomaly Detection
Security Analytics
Index State Management and Index Management
Performance Analyzer
OpenSearch Security plugin workflows

These limitations do not affect the core Discover, visualization, and dashboard workflow described above.

Bringing in real data

Once the connection works, you can replace the sample table with your own data pipeline. Manticore integrates with Logstash, Filebeat, Fluent Bit, and Vector. These tools can collect and transform logs or events before sending them to Manticore's Elasticsearch-compatible HTTP endpoint.

The resulting workflow is straightforward:

An agent or pipeline collects your data.
Manticore indexes it in real time.
OpenSearch Dashboards queries Manticore over HTTP.
Users explore the data and build dashboards in a familiar interface.

Conclusion

The OpenSearch Dashboards integration gives Manticore users a practical visual layer for log analysis and data exploration. With a small amount of configuration, you can search documents in Discover, build charts from supported aggregations, and combine them into dashboards while Manticore handles storage and query execution.

For the latest compatibility details and configuration notes, see the OpenSearch Dashboards integration documentation.

How to return top-N from each group with GROUP_CONCAT()

Sergey Nikolaev — Fri, 04 Sep 2026 10:00:05 +0000

Originally posted by Stanislav Klinov on https://manticoresearch.com/blog/group-concat-top-n-per-group/ on September 4, 2026

How to return top-N from each group with GROUP_CONCAT()

A practical example of how to select several most recent items from each group in Manticore Search and combine them into a single string.

Imagine a customer support screen. An agent searches for refund-related events and, instead of a long log, wants a short summary: one row per user, the total number of matches, and the five most recent events. If more details are needed, the application can load them by ID.

Two obvious approaches don't quite give us what we need:

regular GROUP_CONCAT() collects all values in the group together,
while GROUP N BY returns several rows per user.

Starting with Manticore Search 28.6.6, values inside GROUP_CONCAT() can be sorted and limited to the number you need:

GROUP_CONCAT(id ORDER BY event_ts DESC, id DESC LIMIT 5)

Let's see how it works. We'll use SphinxQL and an explicit GROUP BY - the new form of GROUP_CONCAT() doesn't work without it.

Events we'll work with

Let's create an activity table where each document is a single event. The event text is stored in body, the user in user_id, and the timestamp in event_ts.

CREATE TABLE activity (
    body text,
    user_id int,
    event_ts bigint,
    event_type string
);

INSERT INTO activity (id, body, user_id, event_ts, event_type) VALUES
    (1001, 'refund requested for order 501',       101, 1770000010, 'requested'),
    (1002, 'user logged in',                       101, 1770000020, 'login'),
    (1003, 'refund approved for order 501',        101, 1770000030, 'approved'),
    (1004, 'refund email sent for order 501',      101, 1770000040, 'email'),
    (1005, 'refund status checked for order 501',  101, 1770000050, 'checked'),
    (1006, 'refund payout queued for order 501',   101, 1770000060, 'queued'),
    (1007, 'refund webhook retried for order 501', 101, 1770000060, 'retried'),
    (2001, 'refund requested for order 601',       202, 1770000015, 'requested'),
    (2002, 'refund approved for order 601',        202, 1770000025, 'approved'),
    (2003, 'shipping address changed',             202, 1770000035, 'shipping'),
    (2004, 'refund payout queued for order 601',   202, 1770000045, 'queued'),
    (2005, 'refund completed for order 601',       202, 1770000055, 'completed'),
    (3001, 'refund requested for order 701',       303, 1770000012, 'requested'),
    (3002, 'invoice downloaded',                   303, 1770000022, 'invoice'),
    (3003, 'refund rejected for order 701',        303, 1770000032, 'rejected');

A search for refund will find six events for user 101, four for user 202, and two for user 303. We deliberately gave events 1006 and 1007 the same timestamp: later you will see why sorting also needs id.

What's wrong with the old approaches

Let's start with regular GROUP_CONCAT(). The response format is what we need - one row per user:

SELECT
    user_id,
    COUNT(*) AS matched_events,
    GROUP_CONCAT(id) AS all_event_ids
FROM activity
WHERE MATCH('refund')
GROUP BY user_id
ORDER BY matched_events DESC, user_id ASC;

But for user 101, the string will contain all six IDs, for example 1001,1003,1004,1005,1006,1007, while we only need the five most recent ones. Also, without internal sorting, the order of the values is not guaranteed.

+---------+----------------+-------------------------------+
| user_id | matched_events | all_event_ids                 |
+---------+----------------+-------------------------------+
|     101 |              6 | 1001,1003,1004,1005,1006,1007 |
|     202 |              4 | 2001,2002,2004,2005           |
|     303 |              2 | 3001,3003                     |
+---------+----------------+-------------------------------+

Another option is to ask GROUP N BY to select the five most recent documents from each group:

SELECT
    id,
    user_id,
    event_ts
FROM activity
WHERE MATCH('refund')
GROUP 5 BY user_id
WITHIN GROUP ORDER BY event_ts DESC, id DESC
ORDER BY user_id ASC;

The oldest event for user 101 disappears, but each remaining document is returned as a separate row. So instead of three rows, we get eleven: 5 + 4 + 2. This is convenient when the client needs the documents themselves, but it doesn't work for our compact summary.

+------+---------+------------+
| id   | user_id | event_ts   |
+------+---------+------------+
| 1007 |     101 | 1770000060 |
| 1006 |     101 | 1770000060 |
| 1005 |     101 | 1770000050 |
| 1004 |     101 | 1770000040 |
| 1003 |     101 | 1770000030 |
| 2005 |     202 | 1770000055 |
| 2004 |     202 | 1770000045 |
| 2002 |     202 | 1770000025 |
| 2001 |     202 | 1770000015 |
| 3003 |     303 | 1770000032 |
| 3001 |     303 | 1770000012 |
+------+---------+------------+

Combining only the five most recent IDs

Now let's combine both steps: sort the documents directly inside GROUP_CONCAT() and limit the list to five values there as well.

SELECT
    user_id,
    COUNT(*) AS matched_events,
    GROUP_CONCAT(
        id
        ORDER BY event_ts DESC, id DESC
        LIMIT 5
    ) AS recent_event_ids
FROM activity
WHERE MATCH('refund')
GROUP BY user_id
ORDER BY matched_events DESC, user_id ASC;

+---------+----------------+--------------------------+
| user_id | matched_events | recent_event_ids         |
+---------+----------------+--------------------------+
|     101 |              6 | 1007,1006,1005,1004,1003 |
|     202 |              4 | 2005,2004,2002,2001      |
|     303 |              2 | 3003,3001                |
+---------+----------------+--------------------------+

First, MATCH('refund') selects refund events, then GROUP BY user_id groups them by user. COUNT(*) counts all matching events, while GROUP_CONCAT() takes only the first five from each group after sorting.

The second sort key - id DESC - is especially important here. Events 1006 and 1007 have the same event_ts, so without it their relative order would be undefined. Sorting by ID ensures that event 1007 always comes first.

Notice that the query has two ORDER BY clauses. The one inside GROUP_CONCAT() determines the order of IDs in the string. The final ORDER BY sorts the completed rows: first by the number of matches, then by user_id.

The inner LIMIT does not change COUNT(*) or affect pagination of the overall result. So user 101 still has six matches even though only five IDs are shown next to it. That's exactly what we need for this summary.

One more detail: GROUP_CONCAT() always returns a string, even when it contains numeric IDs. If the API needs to return an array of numbers or objects with several fields, the string has to be parsed on the client side, or a different response format should be used.

The query works the same way with distributed tables. Manticore collects candidates from all local and remote tables, then selects the overall top-N for each group.

If a comma doesn't work

By default, values are separated by commas. Sometimes a more readable string is useful - for example, showing the event type next to its ID. You can set a custom separator with SEPARATOR:

SELECT
    user_id,
    GROUP_CONCAT(
        CONCAT(event_type, ':', TO_STRING(id))
        ORDER BY event_ts DESC, id DESC
        SEPARATOR ' / '
        LIMIT 3
    ) AS recent_events
FROM activity
WHERE MATCH('refund')
GROUP BY user_id
ORDER BY user_id ASC;

+---------+----------------------------------------------+
| user_id | recent_events                                |
+---------+----------------------------------------------+
|     101 | retried:1007 / queued:1006 / checked:1005    |
|     202 | completed:2005 / queued:2004 / approved:2002 |
|     303 | rejected:3003 / requested:3001               |
+---------+----------------------------------------------+

In this syntax, SEPARATOR comes before LIMIT. Manticore doesn't escape anything or add quotes: the result is a regular string, not a JSON array.

This works well for event_type because those values are controlled by the application. Be more careful with arbitrary text: if the separator appears in the data itself, the result can no longer be parsed reliably. In that case, it's better to return separate rows or use a structured format.

Where the new approach doesn't work

This form of GROUP_CONCAT() has several limitations. It works only in SQL queries with an explicit GROUP BY and does not support:

DISTINCT, OFFSET, or combining multiple expressions at once;
JOIN, FACET, outer SELECT queries, or table functions;
KNN and hybrid queries, or scroll;
implicit grouping or equivalent aggregation syntax in the JSON API.

Its alias cannot be used in HAVING or the final ORDER BY. The groups themselves can still be sorted by the grouping key and regular aggregates - for example, by user_id and COUNT(*), as in the query above.

Memory usage is another consideration. For each such expression, Manticore stores a separate top-N for every group that remains in the result. The more groups there are, the higher N is, and the more expressions you use, the more memory is required. The size of the values and sort keys also affects memory usage, so it is best not to set an unnecessarily high limit "just in case".

Full documentation for the new functionality is available here.

More examples

Support events are just one possible scenario. The new mode can be useful anywhere a document already contains a group key, the value you want to collect, and a field to sort by:

Product images: collect the first few IDs or paths for each product_id, sorted by display_order.
Priority tasks: return up to six task IDs for each assignee_id, sorted by a precomputed priority.
Server errors: show the latest N errors for each server while keeping their total count separately.

If you need a short list of IDs, names, or paths, GROUP_CONCAT(... ORDER BY ... LIMIT N) now lets you get it in a single query. We hope you find this useful.

How to search for long hashes and IDs with dict='keywords_32k'

Sergey Nikolaev — Wed, 02 Sep 2026 07:30:01 +0000

Originally posted by Manticore Search on https://manticoresearch.com/blog/dict_keywords_32k/ on August 31, 2026

How to search for long hashes and IDs with dict='keywords_32k'

A practical guide to searching long hashes, event IDs, message IDs, and email addresses in Manticore Search: limits, exact matching, wildcard search, tokenization, migration, and limitations.

Full-text search usually works with ordinary words: product names, titles, comments, and descriptions. Such tokens are rarely longer than a few dozen characters.

Logs and technical data are different. A SHA-256 hash is 64 characters long, while message IDs, correlation keys, event identifiers, and some email addresses can be even longer. The value is often meaningful only as a whole: if its tail is lost, one ID can easily be confused with another.

Manticore Search provides dict='keywords_32k' for these cases.

keywords_32k is available starting with Manticore Search 27.1.1. Version 27.1.5 or newer is recommended when converting an existing table from keywords.

The problem with the regular dictionary

By default, Manticore uses dict='keywords'. Its maximum token length is 42 bytes after normalization.

The limit is measured in bytes, not characters. One ASCII character takes one byte, but a UTF-8 character can take several bytes.

If a token exceeds 42 bytes, Manticore truncates it:

when indexing a document;
when processing a search query.

As a result, querying the complete long value does not necessarily return zero results. Because the query is truncated too, the document may be found-but only by the first 42 bytes.

This creates a more serious problem: two different IDs with the same first 42 bytes become indistinguishable to full-text search. It is also impossible to find a value by a fragment located after the 42nd byte.

What `keywords_32k` changes

dict='keywords_32k' increases the maximum normalized token length to 32768 bytes, or 32 KB.

Behavior	`dict='keywords'`	`dict='keywords_32k'`
Maximum token length	42 bytes	32768 bytes
Token exceeding the limit	Truncated	Skipped with a warning
Prefix and infix search	Supported	Supported
Morphology for tokens longer than 42 bytes	Token is already truncated	Not applied
RT tables	Supported	Supported
Plain tables	Supported	Supported

The setting applies to the entire table:

CREATE TABLE events (
  message text,
  event_id text
)
dict='keywords_32k';

Regular short words in the same table continue to use the configured morphology. Tokens longer than 42 bytes are stored in their original normalized form, without stemming or lemmatization.

For machine identifiers, this is usually exactly what you need: a hash or event ID has no useful word stem.

When to use `keywords_32k`

Use it when both conditions are true:

The value can exceed 42 bytes after tokenization.
It must be searchable through MATCH(), by prefix, or by substring.

Typical examples include:

SHA-256 and other long hashes;
event IDs and message IDs;
request IDs, trace IDs, and other technical identifiers;
long record keys;
email addresses with a long local or domain part;
technical values from logs;
long identifiers containing separators.

However, keywords_32k is not required for every ID lookup.

If you only need exact equality

If the application always receives the complete ID and only needs to check exact equality, a string attribute is sufficient:

CREATE TABLE events (
  message text,
  event_id string
);

Then use a regular filter:

SELECT id, message
FROM events
WHERE event_id =
  '9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08';

If you need both exact matching and full-text search

Use string attribute indexed:

CREATE TABLE events (
  message text,
  event_id string attribute indexed
)
dict='keywords_32k';

In this case, Manticore:

stores the original value as a string attribute;
lets you filter it with WHERE;
also indexes it for MATCH() and wildcard search.

This is usually the most convenient schema for technical identifiers.

Searching by the complete token

Create a table and add a 64-character SHA-256 hash:

DROP TABLE IF EXISTS events;

CREATE TABLE events (
  message text,
  event_id string attribute indexed
)
dict='keywords_32k';

INSERT INTO events (id, message, event_id) VALUES
(
  1,
  'delivery accepted',
  '9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08'
);

Use the complete normalized token in a full-text search:

SELECT id, message
FROM events
WHERE MATCH(
  '@event_id 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08'
);

The @event_id operator restricts the search to the required field. Without it, Manticore searches for the value in all full-text fields of the table.

Quotes are not needed around a single simple alphanumeric token here. Quotes denote a phrase search; they do not turn MATCH() into a byte-for-byte comparison of the original string.

A full-text match is not the same as exact equality

MATCH() operates on the result of tokenization and normalization. It can be affected by:

charset_table;
conversion to lowercase;
blend_chars;
ignore_chars;
word forms and other text-processing settings.

For a strict comparison of the stored value, use a string attribute:

SET collation_connection='binary';

SELECT id, message
FROM events
WHERE event_id =
  '9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08';

binary enables byte-for-byte string comparisons in the current SQL session. This setting does not affect full-text search behavior.

The practical rule is simple:

WHERE event_id = ... - a strict comparison of the stored string;
MATCH('@event_id ...') - a normalized-token search;
MATCH('@event_id prefix*') - a prefix search;
MATCH('@event_id *fragment*') - a substring search.

How to inspect tokenization

Before loading a large volume of data, check that Manticore actually sees the value as a single token:

CALL KEYWORDS(
  '9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08',
  'events'
);

The normalized column should contain the complete 64-character hash.

CALL KEYWORDS is especially useful for values containing periods, hyphens, @, colons, slashes, and characters from different writing systems. This lets you inspect the actual token boundaries before indexing the data.

Prefix and substring search

To search inside a token, enable min_infix_len:

DROP TABLE IF EXISTS events_infix;

CREATE TABLE events_infix (
  message text,
  event_id string attribute indexed
)
dict='keywords_32k'
min_infix_len='4';

Prefix search:

SELECT id, message
FROM events_infix
WHERE MATCH('@event_id 9f86d081*');

Substring search:

SELECT id, message
FROM events_infix
WHERE MATCH('@event_id *b2b0b822*');

A positive min_infix_len also enables prefix search. This example uses 4 to prevent excessively short and broad patterns.

For production systems: do not allow excessively short fragments, choose min_infix_len based on real data, use expansion_limit, test performance with a production-sized dictionary, and restrict search to a specific field with @field.

Email addresses and other values with separators

keywords_32k changes only the maximum token length. It does not determine where a token begins and ends.

By default, a period, @, hyphen, and other characters may split a value into multiple parts. If an email address or message ID must also be indexed as a whole, you can use blend_chars:

DROP TABLE IF EXISTS mail_events;

CREATE TABLE mail_events (
  sender string attribute indexed,
  subject text
)
dict='keywords_32k'
blend_chars='., @, -'
min_infix_len='4';

Blended characters are indexed in two ways: as part of the complete token and as separators between regular parts of the value. This makes it possible to search both the complete email address and individual words within it.

How to convert an existing table

For an RT table, you can change the setting with ALTER TABLE:

ALTER TABLE events dict='keywords_32k';

However, this affects only documents added or replaced after the setting is changed. Existing documents are not automatically tokenized again. Their long tokens remain in the old truncated form until the documents are reindexed.

Follow these steps:

Update dict.
Verify the setting with SHOW CREATE TABLE.
Reindex or reload the existing documents.
Check several long values with CALL KEYWORDS and MATCH().

Why `dict='crc'` does not solve this problem

dict='crc' stores keyword checksums instead of their original text. However, it does not increase the allowed token length. The exception to the regular 42-byte limit is implemented specifically by dict='keywords_32k'.

Current limitations

At the time of publication, dict='keywords_32k' has several limitations:

CALL SUGGEST and CALL QSUGGEST are not supported;
it cannot be used in percolate tables;
tokens longer than 42 bytes are not highlighted in snippets or highlights;
indextool --dumpdict cannot dump this type of dictionary;
the full-text REGEX operator works with dict='keywords', but not with keywords_32k.

Do not index secrets

Being able to search a long value does not mean that you should store it in a search index. Do not index API keys, bearer tokens, session cookies, private keys, passwords, reset tokens, or other data that grants access to the system unless absolutely necessary.

If a secret must be matched by its exact value, it is safer to calculate a suitable hash in advance and store only the hash.

Quick checklist

Before enabling keywords_32k, check:

Is the normalized token actually longer than 42 bytes?
Do you need full-text or wildcard search rather than only WHERE value = ...?
Does CALL KEYWORDS see the entire value as one token?
Do you need blend_chars for periods, hyphens, @, and other separators?
Is min_infix_len large enough?
Is the number of wildcard expansions limited?
Have the old documents been reindexed?
Is the field free of secret data?
Does the application avoid relying on highlighting, SUGGEST, percolate, or full-text REGEX?

Summary

dict='keywords_32k' solves a specific problem: it allows a full-text index to store normalized tokens up to 32768 bytes long instead of the regular 42 bytes.

It is well suited to long hashes, event IDs, message IDs, email addresses, and other machine identifiers. Keep three points in mind:

keywords_32k increases the token length but does not change tokenization rules;
MATCH() on a complete token is not the same as a strict comparison of the original string;
existing documents must be reindexed after the setting is changed.

If you only need exact equality, use a string attribute. If you need both exact matching and partial-value search, use string attribute indexed together with dict='keywords_32k'.

Documentation

How KupujemProdajem searches 5.6 million ads at ~10 ms with Manticore Search

Sergey Nikolaev — Mon, 31 Aug 2026 07:00:01 +0000

Originally posted by Manticore Search on https://manticoresearch.com/blog/kupujemprodajem/ on August 31, 2026

How KupujemProdajem searches 5.6 million ads at ~10 ms with Manticore Search

How Serbia's largest online classifieds marketplace uses Manticore Search to search 5.6 million active ads at ~10 ms, process real-time updates, and prepares for hybrid search.

For an online marketplace, having millions of listings is useful only if buyers can find the right one.

At KupujemProdajem, Serbia's largest online classifieds marketplace, search is the primary way users discover ads. Buyers search from the web and mobile applications, while internal teams use search for moderation and support. Because of that, both relevance and latency directly affect user engagement and sellers' ability to be discovered.

Today, KupujemProdajem uses Manticore Search to search approximately 5.6 million active ads, while separate administrative indexes contain around 70 million rows of current and historical data. The production cluster handles around 150 search requests per second on average, with an average Manticore query time of approximately 10 ms.

And the same search infrastructure is now giving the team a path toward something more ambitious: hybrid lexical and semantic search designed specifically for Serbian-language marketplace content.

Search is part of the core product

KupujemProdajem maintains two main kinds of search indexes. The first powers the customer-facing experience. It contains approximately 5.6 million active classified ads, including titles, descriptions, categories, and structured attributes. Users search these indexes while browsing the marketplace on web and mobile.

The second set of indexes is used internally. In addition to current data, these indexes retain part of the history of deleted ads, allowing moderation and support teams to search information that may no longer be visible on the public marketplace.

This makes search important on both sides of the product. For buyers, it determines how quickly they can get from an idea of what they want to a relevant listing. For sellers, search affects whether their ads are discovered. Internally, it gives employees access to historical marketplace data when investigating problems.

As Dimitrije Marinković, Backend Developer at KupujemProdajem, puts it:

“Search is the primary discovery mechanism on the platform, so relevance and latency directly affect user engagement and seller success.”

Choosing a search engine that could grow with the product

KupujemProdajem had already been running a dedicated search engine for years. When its previous engine, Sphinx, stopped being actively developed, the team needed a maintained alternative that would fit the existing architecture without requiring the entire search integration to be rebuilt.

But compatibility was only part of the decision. The new engine also needed to support:

consistently low query latency;
real-time changes to millions of ads;
replication;
SQL that worked naturally with the team's MySQL-centric PHP stack;
full-text search together with structured filters;
flexible relevance tuning;
and a path toward vector and hybrid search.

Manticore provided those capabilities in one system. That last requirement is becoming increasingly important.

KupujemProdajem plans to combine traditional lexical search with vector search using its own embedding model. Because Manticore supports KNN search alongside full-text search, the team can build that functionality inside the search infrastructure it already operates instead of synchronizing its data with a separate vector database.

Keeping millions of changing ads searchable in real time

MySQL remains the source of truth for marketplace data. When an ad is created, edited, or removed, the corresponding change is pushed to Manticore in real time. The production workload averages approximately 40 REPLACE operations per second, with additional UPDATE and DELETE operations.

This is particularly useful for classifieds. Marketplace inventory changes continuously. Sellers publish new ads, update prices and descriptions, change status, and remove products that are no longer available.

A search system therefore has two jobs at once:

return relevant results quickly;
make sure those results reflect the current marketplace.

KupujemProdajem uses Manticore's real-time updates to keep the searchable representation of the marketplace synchronized with MySQL as these changes happen.

More than matching words

Marketplace search also has very different requirements from searching a collection of plain documents. A listing contains text, but it also has structured information such as category, price, location, status, and other properties. KupujemProdajem combines full-text matching with filters over these typed attributes in Manticore.

The team also controls how different parts of the listing affect relevance. The title, for example, receives significantly more weight than the description or category. This makes intuitive sense for classified ads: a product named directly in the title is normally a stronger signal than the same word appearing somewhere in a longer description.

On top of this, KupujemProdajem uses a custom ranker tuned for its relevance requirements and groups results by ad. This gives the team the ability to encode marketplace-specific relevance rules rather than relying entirely on a generic ranking formula.

A small replicated cluster

The entire Manticore deployment runs on four virtual machines. Three nodes form the production replication cluster, while a fourth VM serves as a backup. All writes are sent to one node and replicated to the others. Read requests are distributed among all three production nodes by application-level logic.

The architecture is relatively simple: MySQL → real-time updates → Manticore write node → replicated Manticore nodes → web/mobile/admin search

That simplicity matters operationally. The team does not need a large distributed data platform merely to support marketplace search. The same cluster handles ingestion, full-text retrieval, structured filtering, custom ranking, replication, and the foundation for future vector search.

5.6 million active ads, 70 million admin rows

The user-facing indexes currently contain around 5.6 million active ads and occupy approximately 9 GB. The larger administrative indexes contain approximately 70 million rows, including some historical deleted-ad data, and occupy around 90 GB.

Across the three-node cluster, KupujemProdajem sees approximately:

Metric	Production scale
Active ads	~5.6 million
Admin index rows	~70 million
User-facing index size	~9 GB
Admin index size	~90 GB
Search traffic	~150 queries/sec average
Search traffic per node	~50 queries/sec average
`REPLACE` traffic	~40/sec
Average query time	~10 ms
Infrastructure	3 production VMs + 1 backup

The cluster remains stable under normal operation, without stuck or long-running queries. Short latency spikes of approximately 150 ms occur occasionally, while the average query execution time remains around 10 ms. Those numbers are particularly useful because this is not a synthetic benchmark. They describe an actual production workload behind the primary discovery experience of a large marketplace.

What users get from this architecture

Infrastructure metrics matter, but only because of what they enable at the product level. For buyers, Manticore helps KupujemProdajem keep search fast even while querying millions of listings and applying relevance logic and structured filters. Because changes are continuously pushed into the index, search can also reflect changes in marketplace inventory quickly instead of relying on occasional large reindexing jobs.

For sellers, this infrastructure supports the main mechanism through which buyers discover their listings. For moderation and support teams, the much larger internal indexes make historical ad information searchable even when it is no longer part of the live marketplace.

The next challenge: understanding Serbian-language intent

The next project for the KupujemProdajem team is hybrid search. Traditional full-text search works particularly well when the words entered by a user also appear in an ad. But marketplace users and sellers do not always describe the same thing in exactly the same way.

That problem becomes even more interesting in Serbian. The team highlights several challenges:

rich morphology;
both Latin and Cyrillic scripts;
synonyms;
different ways buyers and sellers can express the same intent.

KupujemProdajem is therefore preparing its own embedding model for Serbian marketplace content. The plan is to retrieve results using both lexical matching and KNN vector search, then combine the two sets of signals at query time.

The user benefit is straightforward. Someone may search for an item using words that never literally appear in the seller's listing, while the listing is nevertheless highly relevant. Semantic retrieval can provide another signal for recognizing that relationship.

As Dimitrije describes the expected result:

“Semantic matching will let users find relevant ads even when their wording doesn't literally match the listing.”

Perhaps just as important for the engineering team, they do not need to introduce another database to do it. The vector representation can live alongside the text and structured attributes already searchable in Manticore, giving the team one engine in which to combine lexical relevance, filters, and semantic similarity.

A foundation for better marketplace search

KupujemProdajem already uses Manticore for one of the most important workflows on its marketplace. Approximately 5.6 million active ads are searchable with an average engine query time of around 10 ms. Updates arrive continuously. Replication provides multiple search nodes. Internal teams search a much larger historical dataset using the same technology.

But the more interesting part may be what comes next. The team can now experiment with semantic retrieval and hybrid relevance without replacing the search infrastructure or operating an additional vector database.

For KupujemProdajem, that means Manticore is not only keeping today's marketplace search fast. It is also giving the team a practical way to make tomorrow's search better.

Full-Text Search Still Works. It Just Doesn’t Get You to an Answer

Sergey Nikolaev — Mon, 24 Aug 2026 10:34:39 +0000

Originally posted by Klim Todrik on https://manticoresearch.com/blog/conversational-search/ on Aug 20, 2026

Imagine a typical online shoe store.

A shopper opens search and types:

I need black waterproof running shoes for daily runs on wet pavement. What would you recommend?

A few years ago, almost no one expected this from a search box. The query would have been shortened to something like:

black waterproof running shoes

Then the shopper would open several product cards, compare descriptions, materials, intended use, and price, and make the decision alone.

Today, people increasingly expect search itself to do part of that work. Not because full-text search has become worse. It still performs very well with exact names, SKUs, product codes, brands, and keywords. What has changed is what people expect to be able to ask a search system.

For example, Google reported in May 2026 that AI Mode had surpassed one billion monthly users. The company notes that people are asking longer, more complex questions that previously did not fit into conventional search. (blog.google)

The same thing happens in an online store.

A query such as:

I need black waterproof running shoes for daily runs on wet pavement.

only looks like one sentence. For a search system, it contains several tasks.

It needs to extract constraints such as color and waterproofing; understand that this is about running rather than walking; account for price, size, and availability; find suitable products; and, if the user asks “which are better,” explain the differences.

No single algorithm solves all of that.

Full-text search, vector search, filters, hybrid search, and a language model each solve different parts of the problem. It is far more effective to combine them than to choose between them.

That is exactly why Manticore has Conversational Search.

Word search, semantic search, and conversation are different tasks

Start with a simple query:

Nike Pegasus 41 black

Here, the system barely needs to interpret the user’s intent. Full-text search handles it directly.

Or something even simpler:

SKU 123456

Semantic methods are not needed here.

Now consider another example:

light shoes for long summer walks

A product card may not contain the words “summer” or “long walks,” but it may include details such as “breathable material” or “lightweight construction.”

This is where vector search becomes useful.

Real queries often fall between these extremes:

black Gore-Tex shoes for everyday running

Some parameters — black and Gore-Tex — need to be preserved. Everyday running describes the user’s intent rather than an exact attribute.

For such cases, Manticore uses hybrid search, combining full-text and vector search through result ranking.

But even hybrid search returns only a list of results.

At that point, search considers its job done. The user usually does not.

It does not answer questions such as:

Which of these models are better suited to rain?

And it certainly does not handle a follow-up such as:

Which of those cost less than $120?

That is a conversation. It needs another layer.

What we built

To test this in practice, we used ConvApparel, a dataset of conversations about choosing apparel. After cleanup, it contained 82,524 products: footwear, pants, tops, and outerwear. Each product has a description, category, images, and attributes. We built Manticore Apparel Shop on this data.

For example, you can type:

I need black waterproof running shoes for jogging

The system first finds suitable products, then a language model generates an answer using them as context, while the interface shows the products themselves.

Try the demo: Manticore Apparel Shop generates a random product and a query that should retrieve it, then demonstrates that the same query does retrieve that product through Manticore.

It is important to keep the connection between the answer and the data. If the system claims that a model is suitable for rain, the user should be able to open the product and verify the source of that claim.

In this approach, the language model does not replace search. It interprets its results.

How it works

Two main commands are used:

CREATE CHAT MODEL

and

CALL CHAT(...)

First, you create a Conversational Search model and set the rules it follows.

CREATE CHAT MODEL assistant (
    model='openrouter:google/gemma-4-26b-a4b-it',
    timeout=60,
    retrieval_limit=5,
    max_document_length=3000,
    custom_prompt='You are a context-only shopping assistant.

Answer using only the provided context.
Do not use outside knowledge or unsupported assumptions.

Recommend only products supported by the retrieved context.
For every recommended product, briefly explain why it matches the request.

End every recommendation with the corresponding
context source ID in the format [ref:<id>].

If none of the retrieved products support the request,
say that you do not have enough information.'
);

retrieval_limit determines how many documents enter the context. max_document_length limits the amount of text from each document.

If there is too little context, the model will not see the right products. If there is too much, latency and query cost increase. Like a person, a language model does not become smarter just because it has been given everything to read.

You can then run a query:

CALL CHAT(
    'I need black waterproof running shoes for jogging',
    'convapparel_products',
    'assistant',
    'demo-session-001',
    'embedding_vector'
);

You can then continue the conversation:

CALL CHAT(
    'Which of these are better for daily use?',
    'convapparel_products',
    'assistant',
    'demo-session-001',
    'embedding_vector'
);

The system uses conversation history, so the user does not need to repeat the context.

Through the HTTP API

Conversational Search is also available through the JSON API:

{
  "chat": {
    "query": "I need black waterproof running shoes for jogging",
    "table": "convapparel_products",
    "model_name": "assistant",
    "conversation_uuid": "demo-session-001",
    "vector_field": "embedding_vector"
  }
}

The request is sent to /search.

What the system returns

The response contains:

conversation_uuid
user_query
search_query — the search query generated by the system
response
sources

search_query is particularly important.

If the user writes:

Which of these would work better in rain?

On its own, this query makes no sense without context. The system therefore forms a complete search query using the conversation history.

This also simplifies debugging: you can trace the entire chain from query to answer.

How retrieval works

Conversational Search uses vector search over an embedding field. The flow looks like this:

user question
→ conversation history
→ search query
→ vector search
→ retrieved documents
→ language model
→ answer and sources

Where search ends and conversation begins

Full-text search works well for exact queries. Vector search works with semantic ones. Hybrid search works with their combination. Conversational Search is needed when the result must be explained, compared, or refined.

Quality

We tested this on our conversational search quality benchmark, using 200 deterministic shopping queries from ConvApparel. The benchmark evaluates the product IDs returned as sources, rather than the wording of the generated answer.

In the current run, Manticore scored 0.3650 Hit@3, 0.4250 Hit@5, 0.5250 Hit@10, and 0.2790 MRR — the best result on each of those metrics among the engines tested. The full repository includes the dataset-building rules, engine configuration, smoke tasks, and raw results.

Conclusion

Search in modern systems is not one algorithm but several layers:

full-text search
vector search
hybrid search
Conversational Search

Each solves its own task. A language model does not replace search; it works on top of it.

Without good search, you do not get a smart assistant; you get a very talkative consultant that barely knows its own catalog.

Want to test the approach in practice? Try Manticore Apparel Shop: choose a random product, ask a question based on it, and confirm that the same product is found and suggested in the answer.

If you want to run it locally or explore the code, check the GitHub repository.

UUID in Manticore: A Practical Guide

Sergey Nikolaev — Thu, 06 Aug 2026 05:20:04 +0000

In the overview article we explained why it makes sense to use the same UUID in search and in the primary database, when one already exists. Here we will go straight to practice: create a table, run the core operations through SQL and the JSON API, and then load several documents through /bulk.

All examples are for Manticore Search 28.5.0 or later. The <generated UUID> value in the responses means the UUID that Manticore creates when processing the request. You do not need to copy this string into the next request: substitute the actual id from your own response.

Table for all examples

To make UUID the document ID, first declare the field as id uuid. Aside from the id type, the RT table schema does not change:

CREATE TABLE products_uuid (
    id uuid,
    title text,
    sku string,
    price int
);

DESC products_uuid will show that the id field has the uuid type. Both SQL and the HTTP API use this value as the document ID, so there is no need to copy the UUID into a string attribute.

Working with UUID through SQL

First, insert a product with a prebuilt UUID:

INSERT INTO products_uuid (id, title, sku, price)
VALUES (
    '550e8400-e29b-41d4-a716-446655440000',
    'Mechanical keyboard',
    'KB-001',
    149
);

In SQL, the UUID must be enclosed in single quotes. You can find the document with a regular WHERE id = '...' condition:

SELECT id, title, sku, price
FROM products_uuid
WHERE id = '550e8400-e29b-41d4-a716-446655440000';

Manticore can also generate the UUID. To do that, simply do not pass any id value:

INSERT INTO products_uuid (title, sku, price)
VALUES ('USB microphone', 'MIC-001', 89);

SELECT LAST_INSERT_ID();

LAST_INSERT_ID() will return the UUID created by this request:

+--------------------------------------+
| last_insert_id()                     |
+--------------------------------------+
| <generated UUID>                     |
+--------------------------------------+

INSERT for multiple documents and the @@session.last_insert_id variable are covered in detail in the documentation section on adding documents.

Let's add another document with a known ID to check IN:

INSERT INTO products_uuid (id, title, sku, price)
VALUES (
    '550e8400-e29b-41d4-a716-446655440001',
    'USB-C dock',
    'DOCK-001',
    119
);

SELECT id, sku, price
FROM products_uuid
WHERE id IN (
    '550e8400-e29b-41d4-a716-446655440000',
    '550e8400-e29b-41d4-a716-446655440001'
);

You can change attributes with a regular UPDATE. The id itself stays the same:

UPDATE products_uuid
SET price = 139
WHERE id = '550e8400-e29b-41d4-a716-446655440000';

An INSERT with an already existing UUID will not overwrite the document. Manticore will return a duplicate error:

INSERT INTO products_uuid (id, title, sku, price)
VALUES (
    '550e8400-e29b-41d4-a716-446655440000',
    'Duplicate keyboard',
    'KB-DUP',
    1
);

To create a new version of the document with the same UUID, use REPLACE:

REPLACE INTO products_uuid (id, title, sku, price)
VALUES (
    '550e8400-e29b-41d4-a716-446655440000',
    'Mechanical keyboard, revised',
    'KB-001',
    129
);

To change only the price, UPDATE is enough. Full-text fields and columnar attributes require REPLACE: it marks the old version of the document with the same ID as deleted and writes the new one. If that ID does not exist yet, Manticore simply adds the document. More details are in the documentation: UPDATE and REPLACE.

Delete the document by the same UUID:

DELETE FROM products_uuid
WHERE id = '550e8400-e29b-41d4-a716-446655440001';

Let's check the current state of the document whose UUID we set on insert:

SELECT id, title, sku, price
FROM products_uuid
WHERE id = '550e8400-e29b-41d4-a716-446655440000';

The SQL client sends and receives a UUID as a string. Pass it as a string parameter, and read the id from SELECT as a string. There is no need to convert it to a number or BINARY(16). Code written for a numeric document ID will need to be adjusted.

The same operations through the JSON API

When writing through the JSON API, the id field is passed alongside table, not inside doc. In the first example, we use an uppercase UUID:

curl -sS http://localhost:9308/insert \
  -H 'Content-Type: application/json' \
  -d '{
    "table": "products_uuid",
    "id": "AAAAAAAA-AAAA-4AAA-8AAA-AAAAAAAAAAAA",
    "doc": {
      "title": "Wireless keyboard",
      "sku": "JSON-KB-001",
      "price": 159
    }
  }'

Manticore accepts UUID in uppercase, stores it in lowercase, and returns it lowercased in the response:

{
  "table": "products_uuid",
  "id": "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
  "created": true,
  "result": "created",
  "status": 201
}

For automatic generation, remove the id field entirely:

curl -sS http://localhost:9308/insert \
  -H 'Content-Type: application/json' \
  -d '{
    "table": "products_uuid",
    "doc": {
      "title": "Portable speaker",
      "sku": "JSON-SPK-001",
      "price": 79
    }
  }'

The response contains the ID that should be saved for subsequent operations:

{
  "table": "products_uuid",
  "id": "<generated UUID>",
  "created": true,
  "result": "created",
  "status": 201
}

In /search, you can use UUID in the equals filter. If you request id in _source, the result will contain the same UUID in both _id and _source.id:

curl -sS http://localhost:9308/search \
  -H 'Content-Type: application/json' \
  -d '{
    "table": "products_uuid",
    "query": {
      "equals": {
        "id": "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa"
      }
    },
    "_source": ["id", "title", "sku", "price"]
  }'

{
  "timed_out": false,
  "hits": {
    "total": 1,
    "total_relation": "eq",
    "hits": [
      {
        "_id": "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
        "_score": 1,
        "_source": {
          "id": "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
          "title": "Wireless keyboard",
          "sku": "JSON-KB-001",
          "price": 159
        }
      }
    ]
  }
}

_id is part of the search result metadata, while _source.id is the document field. For a UUID table, they contain the same string.

Now let's call the remaining endpoints for modifying data one by one. UPDATE changes only the price:

curl -sS http://localhost:9308/update \
  -H 'Content-Type: application/json' \
  -d '{
    "table": "products_uuid",
    "id": "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
    "doc": {"price": 149}
  }'

To replace the document completely, call /replace:

curl -sS http://localhost:9308/replace \
  -H 'Content-Type: application/json' \
  -d '{
    "table": "products_uuid",
    "id": "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
    "doc": {
      "title": "Wireless keyboard, revised",
      "sku": "JSON-KB-001",
      "price": 139
    }
  }'

Now delete the replaced document:

curl -sS http://localhost:9308/delete \
  -H 'Content-Type: application/json' \
  -d '{
    "table": "products_uuid",
    "id": "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa"
  }'

After /delete, there is no document with this UUID left in the table. A /search by the same ID will return total: 0:

curl -sS http://localhost:9308/search \
  -H 'Content-Type: application/json' \
  -d '{
    "table": "products_uuid",
    "query": {
      "equals": {
        "id": "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa"
      }
    }
  }'

{
  "timed_out": false,
  "hits": {
    "total": 0,
    "total_relation": "eq",
    "hits": []
  }
}

Examples of requests for deleting by ID and by condition are collected in the documentation section on deleting documents.

Where to generate UUID

If the UUID is already issued by the primary database, just pass it to Manticore as id. If the request is repeated, the UUID will stay the same. INSERT will report a duplicate, and REPLACE will write a new version of the document under the same ID. The same rule applies to insert and replace in /bulk.

Manticore can also generate the UUID itself: just do not pass id. But resending such a request will create another document, so for automatic retries it is better to set the UUID explicitly.

For an explicit ID, any UUID version from v1 to v8 is suitable. For automatic generation, Manticore uses its own UUIDv8 structure, but it does not convert UUIDs received from the client into it.

Batch loading through `/bulk`

/bulk accepts NDJSON: each line contains a separate operation. The UUID is passed in the id field, just like in the other JSON API requests:

POST /bulk
Content-Type: application/x-ndjson

{"insert":{"table":"products_uuid","id":"bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb","doc":{"title":"USB hub","sku":"BULK-HUB-001","price":49}}}
{"insert":{"table":"products_uuid","id":"bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbc","doc":{"title":"Laptop stand","sku":"BULK-STAND-001","price":39}}}

After the last data line, a trailing newline is required. With curl, it is convenient to pass the body through --data-binary so it does not strip newlines.

Both operations belong to one table, so Manticore executes them in a single transaction. The shortened response shows how many documents were added and how the entire batch finished:

{
  "items": [
    {
      "bulk": {
        "created": 2,
        "status": 201
      }
    }
  ],
  "current_line": 2,
  "skipped_lines": 0,
  "errors": false
}

On an error, current_line shows the line where processing stopped, and skipped_lines shows the number of skipped lines. If an empty line or a table switch split the request into multiple transactions, Manticore will not roll back the ones that already completed.

When reloading a document, it is important to choose the right operation. If the UUID already exists, insert will fail with a duplicate error. replace will write the document again, and if no such ID exists, it will add a new document.

The id field in /bulk can also be omitted, and Manticore will generate a UUID. But the /bulk response contains only the overall transaction result, without separate results for each inserted row. If you need to keep the ID of each document, it is more convenient to generate the UUID before the batch request or add documents one by one through /insert.

Validation and common errors

Manticore validates the UUID before adding the document. SQL, the JSON API, and /bulk return errors differently, so do not bind your code to the exact wording of the message.

What you sent	What Manticore will do
`550e8400-e29b-41d4-a716-446655440000`	Add the document
`AAAAAAAA-AAAA-4AAA-8AAA-AAAAAAAAAAAA`	Add the document, store the UUID, and return it in lowercase
A string without hyphens or with an invalid character	Return an error because the UUID format is invalid
`00000000-0000-0000-0000-000000000000`	Return an error: zero UUID cannot be used
A number, including `0`	Return an error: `id` must be a string
A UUID with a version outside the 1-8 range or an RFC-invalid `variant` value	Return a validation error
`INSERT` or `insert` in `/bulk` with an already existing ID	Return a duplicate error

A few points:

The canonical form consists of 36 characters split into 8-4-4-4-12 groups. UUID versions from 1 to 8 are valid; in the RFC variant position, one of these characters must appear: 8, 9, a, or b.
Manticore validates only the ID format. Correct generation is the application's responsibility: for UUIDv4, randomness matters, and for UUIDv7, the time component and compliance with the rules matter. Manticore does not convert passed v4 and v7 values into v8.
In a table with UUID IDs, id = 0 does not trigger automatic generation unlike numeric IDs: the request will fail. But if you do not pass id, Manticore will generate its own UUIDv8 structure and return it in the response. Keep in mind that it can be used as a document ID, but not as, for example, an access token.
You can validate the UUID with a standard library on input, but keep in mind that Manticore still performs its own validation.
In the /bulk response, check errors, current_line, and skipped_lines: they show where processing stopped and which part of the batch could not be committed. In SQL batch requests, the entire request is rolled back if an error occurs.

Columnar RT and replication

UUID can also be used as the document ID in RT tables with columnar storage. Only the engine declaration changes:

CREATE TABLE products_uuid_columnar (
    id uuid,
    title text,
    sku string,
    price int
) engine='columnar';

For regular and columnar RT tables, INSERT, REPLACE, and DELETE, exact-match searches, and IN conditions are written the same way. The client code does not depend on the table engine. But UPDATE does not change columnar attributes, so, for example, the price in such a table can only be updated together with the whole document through REPLACE.

Replication also supports UUID. Suppose the catalog cluster has already been created, the nodes have joined, and the local products_uuid table exists. Let's add it to the cluster:

ALTER CLUSTER catalog ADD products_uuid;

In SQL, a colon goes between the cluster name and the table name:

INSERT INTO catalog:products_uuid (id, title, sku, price)
VALUES (
    '550e8400-e29b-41d4-a716-446655441000',
    'Replicated keyboard',
    'REPL-KB-001',
    169
);

SELECT id, sku, price
FROM catalog:products_uuid
WHERE id = '550e8400-e29b-41d4-a716-446655441000';

In the JSON API, the table name is passed unchanged, while the cluster goes in a separate field. For example, the following operation will update a document previously added through SQL:

curl -sS http://localhost:9308/update \
  -H 'Content-Type: application/json' \
  -d '{
    "cluster": "catalog",
    "table": "products_uuid",
    "id": "550e8400-e29b-41d4-a716-446655441000",
    "doc": {"price": 159}
  }'

After replication, the document keeps the same UUID on all nodes. You can run exact-match searches, UPDATE, REPLACE, and DELETE by it.

What to consider before rollout

The uuid type can be assigned only to the id field. It is supported in RT tables, including those with columnar storage and replication, but not in plain, percolate/PQ, or shard tables.
An existing table cannot be switched from a numeric ID to a UUID or back with ALTER TABLE. This decision has to be made when creating the new schema.
A UUID can be used in = and IN conditions. The <, <=, >, and >= ranges, as well as arithmetic on id, are not supported.
UUIDv7 contains time, but filtering id by range is still not allowed. For date-based selection, add an attribute such as created_at timestamp and filter by it.
The document ID cannot be changed with UPDATE. To give an object a different UUID, you will need to create a document with a new ID and delete the old one separately.

The behavior of SQL sessions and the full list of limitations are described in the documentation: UUID document IDs. Support for UUID as a document ID appeared in Manticore Search 28.6.6.

UUID in Manticore: a single ID for the primary database and Manticore

Sergey Nikolaev — Wed, 05 Aug 2026 04:21:12 +0000

Suppose your product already has the ID 550e8400-e29b-41d4-a716-446655440000 in the primary database. It shows up in events, logs, and API responses. But when you load that same product into Manticore, the application still has to assign it another, numeric ID.

Before Manticore Search 28.5.0, the document ID was an unsigned 64-bit number. A UUID could be stored in a separate string attribute, but that did not make it the document ID. UPDATE, REPLACE, and DELETE still required a numeric id.

As a result, you had to keep a mapping between the UUID from the primary database and the numeric ID in the Manticore table. Now you can do without it: an RT table in Manticore can use a UUID as the document ID.

Why a second ID is a problem

A mapping table between IDs is not exactly complicated, but only as long as the application is loading data and searching. The problems start when documents change.

A worker receives an event with a UUID, finds the matching numeric ID, and only then sends the update to Manticore. Deletion follows the same path. If the mapping record is missing or stale, the wrong document may be updated, or the changes may never reach Manticore at all.

Another option is to turn the UUID into a 64-bit hash. Then the application itself has to account for possible collisions. You can also introduce a separate sequence counter, but then it has to be coordinated across all processes that create documents.

There is another subtle point with numeric IDs, and it appears at the API level. Inside Manticore it is a uint64, while SQL shows it as a signed BIGINT. Because of that, SQL can return values greater than 2^63-1 as negative numbers, and the client has to convert them carefully. A UUID is passed and returned as a string, so there is no need to worry about signed range or overflow.

id uuid solves the problem at the root: the object identifier no longer needs conversion. The same UUID is used in the primary DB, the queue, Manticore, logs, and the external API.

What a UUID is

UUID is essentially a 128-bit identifier that does not require a central registry to create. In text form, it usually consists of 36 characters: 32 hexadecimal digits and four hyphens.

550e8400-e29b-41d4-a716-446655440000

A UUID contains a version and a variant. The version defines how the remaining bits are formed. UUIDv4 is based on random or pseudorandom data. UUIDv7 includes a time component and preserves the chronological order of identifiers. In UUIDv8, the placement of the remaining bits is defined by a specific implementation.

The lack of a central registry makes it possible to assign an ID before writing to a shared database. For example, two independent services can create objects in parallel, and a mobile client with no LTE signal can prepare data without a server connection. After synchronization, the object keeps the same identifier.

However, a UUID should not be treated as an absolute guarantee of uniqueness. In practice, that depends on the correctness of the chosen generator. Also keep in mind that a UUID is not a secret and does not replace a password, token, or permission check.

What changed in Manticore 28.5.0

For an RT table, you can now explicitly set the document ID type:

CREATE TABLE products_uuid (
    id uuid,
    title text,
    sku string,
    price int
);

INSERT INTO products_uuid (id, title, sku, price)
VALUES (
    '550e8400-e29b-41d4-a716-446655440000',
    'Mechanical keyboard',
    'KB-001',
    149
);

After insertion, the same UUID can be used in equality and IN filters, as well as in UPDATE, REPLACE, and DELETE. SQL returns the ID as a string. You can also pass an explicit UUID or ask Manticore to generate an ID through the JSON API; we will cover detailed requests and responses in the next article.

If you run INSERT again with a UUID that already exists, Manticore rejects it, just as it used to with a numeric ID. Use REPLACE to overwrite a document by ID.

Note that the uuid type applies only to the document ID. You cannot declare a regular user attribute with the uuid type.

Who creates the UUID

There are two options:

How the document is written	Who creates the ID	What Manticore does
The `id` field is provided	The primary DB, a client library, or another system component	Validates the format, version, and `variant`, then stores the UUID in lowercase
The `id` field is missing	Manticore	Creates a UUIDv8 and encodes its internal numeric auto-ID into it

If your application already creates the UUID, you do not need to adapt it to Manticore. The format is standard: five groups of 8-4-4-4-12. The version can be any from v1 to v8, and the variant position must contain 8, 9, a, or b. Case does not matter: you can send the UUID in uppercase, but Manticore will store it in lowercase.

That is where validation ends: Manticore checks the UUID format, but not the quality of the generator. The client library is responsible for the randomness of UUIDv4 and the correct time component in UUIDv7.

Server-side generation works differently. If the id field is not provided, Manticore creates a UUIDv8 with its own structure and encodes the built-in numeric auto-ID into it. This is not a random UUIDv4 and it is not a secret value.

A UUIDv8 created by the application will be validated and stored by Manticore as is; the server will not reshape it according to its own scheme.

Technical note. The full UUID remains the external identifier: Manticore does not replace it with a 64-bit hash and does not require the application to store a mapping table. How the UUID is laid out inside the engine is an implementation detail and does not affect the external contract.

Where it works

You can use a UUID as the document ID in regular RT tables, RT tables with engine='columnar', and tables in a replication cluster. With such IDs, exact match search, IN filters, and the usual document operations are available: INSERT, REPLACE, UPDATE, and DELETE.

Keep these limitations in mind:

the uuid type is not suitable for regular attributes: Manticore will reject a declaration such as guid uuid;
in plain, percolate/PQ, and shard tables, UUIDs cannot be used as the document ID;
an existing table cannot be switched from a numeric ID to a UUID, or back, with ALTER TABLE;
range conditions <, <=, > and >=, as well as arithmetic operations on IDs of this type, are not supported;
the document ID cannot be changed through UPDATE - this applies to both UUIDs and numeric IDs;
for automatic generation, the id field must be omitted: the value 0 does not work here as a special marker.

This is easy to forget when porting legacy code. For a numeric RT table, 0 can mean “create the ID automatically.” In a UUID table, you simply need to leave out the id field.

UUIDv7 also does not make id suitable for time-based filters. You can use it as an external identifier, but Manticore does not yet support queries like id > ....

Summary

If the UUID already exists in the primary DB, you can now pass it to Manticore together with the document. If the document appears in Manticore first, do not specify id and take the generated UUID from the response.

From there, use the same UUID in UPDATE, REPLACE, and DELETE. There is no longer any need to give the document a separate numeric ID and keep a mapping between the two identifiers.

The full contract and current limitations are documented here: UUID document IDs. Support for UUID as a document ID appeared in Manticore Search 28.5.0.

In the next article, a practical guide to using UUIDs as document IDs, we will step through how to set and generate IDs with SQL and JSON, search, update, replace, and delete documents, and handle errors.

Manticore Search 28.6.6: UUID document IDs, ordered GROUP_CONCAT(), and 16 fixes

Sergey Nikolaev — Fri, 31 Jul 2026 09:33:57 +0000

Manticore Search 28.6.6 has been released. The headline additions are UUID document IDs for real-time tables and ordered, limited GROUP_CONCAT() for grouped queries. The release also includes 16 fixes for backups, replication, query processing, SQL compatibility, and secondary indexes.

This post covers everything shipped from 28.4.5 through 28.6.6.

Upgrade notes

There are no new mandatory data migrations in this release. UUID IDs are an opt-in table definition: existing numeric-ID tables keep working as they are. If you want UUID identifiers, create a real-time table with id uuid; ALTER TABLE cannot convert an existing table between numeric and UUID IDs.

Two fixes are particularly useful for production installations. Successful backups now always unfreeze real-time tables when they finish (previously in rare cases they didn't), rather than leaving writes blocked. And authenticated replication can again add an existing populated RT table with ALTER CLUSTER ... ADD.

UUID document IDs for real-time tables

Applications often already have UUID identifiers from the system of record. Until now, using them with Manticore Search meant maintaining a separate numeric ID mapping. Real-time tables can now use UUID document IDs directly:

CREATE TABLE products_uuid(id uuid, title text, price int);

Manticore accepts an explicit UUID string, or generates one when id is omitted from an insert or replace. UUID equality and IN filters work in queries, and UUID IDs can be used with REPLACE, UPDATE, and DELETE.

This is currently a real-time-table capability, including columnar and replicated RT tables. Plain, percolate, and sharded tables continue to use their existing ID models.

Ordered and limited GROUP_CONCAT()

Grouped results often need a compact preview of the most relevant values in each group. GROUP_CONCAT() can now sort values and retain only the requested number of them in explicit SQL GROUP BY queries:

SELECT category,
       GROUP_CONCAT(title ORDER BY price DESC SEPARATOR ', ' LIMIT 3)
FROM products
GROUP BY category;

The new ORDER BY, SEPARATOR, and LIMIT options let the query return the top values in the requested order instead of concatenating an arbitrary-sized set and sorting it again in application code. See the GROUP_CONCAT() reference for the full syntax.

Safer table, backup, and cluster maintenance

A few changes remove failures that tend to appear during routine operations rather than ordinary searches:

Manticore Backup 1.10.2 fixes a path where a successful backup could leave an RT table frozen and future writes blocked.
Authenticated replication state transfer now works when ALTER CLUSTER ... ADD is used to add a populated RT table.
RT lifecycle operations now remove obsolete external files after ALTER, TRUNCATE, chunk removal, and optimization. The same release preserves Jieba settings during unrelated ALTER TABLE operations and rejects unsupported effective setting changes explicitly.
OPTIMIZE TABLE accepts qualified system.<table> names, so physical sharded tables can be optimized.

Query processing and client compatibility

The fixes are broad, but several stand out for busy search services:

CALL SNIPPETS no longer spends seconds simplifying some complex boolean queries before highlighting them when boolean_simplify is enabled.
Long exact and phrase-like queries no longer risk a query-stack underestimate that could corrupt memory or crash searchd; repeated keyword statistics are also deduplicated for local_df work on multi-chunk RT and distributed tables.
The updated columnar library fixes a StreamVByte decoder over-read that could crash secondary-index queries, including GROUP BY on JSON fields. indextool --check now reports corrupted secondary-index files instead of crashing.
USER() remains usable after USER became an authentication keyword, restoring MySQL client status command compatibility. SHOW INDEX ... STATUS is also protected from a concurrent percentile-calculation race.
Document-ID IN(...) filters correctly accept valid unsigned 64-bit IDs above Long.MAX_VALUE, and a LEFT JOIN with multiple FACET clauses no longer crashes.

The release also fixes a SphinxQL SSL connection-handling race that could consume workers at 100% CPU, and fixes PHP API response parsing and the Ruby API fixture so valid response chunks ending in "0" are preserved.

For the complete list, see the Version 28.6.6 changelog.

Get Manticore Search 28.6.6

Install or upgrade Manticore Search with the installation guide. If you are upgrading an authenticated or replicated deployment, review the existing authentication rollout checklist before changing production traffic.

Need help or want to connect?

Join our Slack
Visit the Forum
Report issues or suggest features on GitHub
Email us at contact@manticoresearch.com

Faceted search under active filters

Sergey Nikolaev — Thu, 30 Jul 2026 04:54:21 +0000

Facets in an online store seem simple until the first filter is selected.

In a catalog, they are part of the navigation. A selected color should not disappear from the list. Other colors should remain available: the user may want to switch to one or broaden the selection. Options with no matching products are more useful when shown as unavailable. Within a single facet, OR usually applies: red or blue. Across different facets, it is AND: brand, color, and size at the same time.

The difficult part begins after the first selection. Imagine a product catalog where the user has selected a brand, color, and size. The main result set must remain narrow and show only products matching all three conditions. But the filter panel follows different rules. It needs to preserve the selected color while also showing the colors the user can switch to under the same brand and size.

If every facet inherits all filters from the main query, it quickly collapses to the values already selected. If the filters for every facet have to be rebuilt manually, you end up with separate query branches for color, size, brand, availability, seller, and every other attribute.

Manticore Search 25.12.0 introduced facet_filter_mode, which moves this behavior into the facet API. A query can now describe how the store's filter panel should behave, without the application manually building nearly identical queries for every facet.

What e-commerce facets need

For a filter panel, answering "how many products have this value?" is not enough. It needs statuses that the interface can use directly:

selected values remain visible and can be cleared quickly;
other values in the same facet remain available because they broaden the filter through IN (...);
values in other facets indicate whether selecting them would return at least one product;
counts remain predictable: the interface knows whether a number refers to the current result set or a broader selection;
search within long brand lists, categories, price ranges, SEO rules, and query analytics still matter; they are a separate part of search design.

facet_filter_mode handles the most common part of this problem: recalculating facets after the user has already selected several filters.

What changed

Facets now have three filter inheritance modes:

Mode	Behavior
`strict`	Applies all filters from the main query to the facet. This is the previous behavior and remains the default.
`auto`	Applies every filter except filters on the facet itself and adds `status`: selected values get `selected`, while other values get `available`.
`max`	Calculates buckets over a broad base result set and adds `status` to distinguish selected, available, and unavailable values.

You can also control this manually:

ALL FILTERS - apply all filters to the facet;
FILTERS color_id, size_id - apply only the listed filters;
EXCLUDE FILTERS color_id - apply every filter except those listed;
ZEROES - starting with Manticore Search 27.3.0, preserve buckets from the broad max scope in SQL max mode even when their count(*) in the current facet is 0.

The same options are available in the JSON API through facet_filter_mode, mode, filters, exclude_filters, and zeroes: true.

In short, strict answers "what is in the current result set?", auto answers "what happens if this particular filter changes?", and max shows a broad list of buckets with a status field: selected, available, or unavailable.

Minimal example

Consider a small catalog with a brand, color, size, and SKU:

CREATE TABLE products(
  title text,
  brand_id int,
  color_id int,
  color_name string,
  size_id int,
  size_name string,
  sku string
);

INSERT INTO products(id,title,brand_id,color_id,color_name,size_id,size_name,sku) VALUES
(1,'p1',7,1,'red',10,'small','sku1'),
(2,'p2',7,1,'red',20,'large','sku2'),
(3,'p3',7,2,'blue',10,'small','sku3'),
(4,'p4',7,3,'green',30,'xlarge','sku4'),
(5,'p5',8,1,'red',10,'small','sku5'),
(6,'p6',8,4,'black',20,'large','sku6'),
(7,'p7',9,5,'white',10,'small','sku7');

The user selected:

brand_id=7 AND color_id=1 AND size_id=10

Under these conditions, the result set contains one product: p1. Let's see what happens to the facets.

strict: the previous behavior

Without additional options, Manticore Search uses strict: every facet receives all filters from the main query.

SELECT count(*)
FROM products
WHERE brand_id=7 AND color_id=1 AND size_id=10
LIMIT 0
FACET color_id ORDER BY color_id ASC
FACET size_id ORDER BY size_id ASC;

Response:

+----------+----------+
| color_id | count(*) |
+----------+----------+
|        1 |        1 |
+----------+----------+
+---------+----------+
| size_id | count(*) |
+---------+----------+
|      10 |        1 |
+---------+----------+

This is an accurate SQL response, but it is often too narrow for an online store. The user sees only the already selected color_id=1 and size_id=10. It looks as if there are no other options, even though the data contains a product with the same brand and size but a different color (color_id=2), and another with the same brand and color but a different size (size_id=20).

auto: a facet ignores its own filter

The auto mode keeps every filter except the filter on the facet currently being calculated.

SELECT count(*)
FROM products
WHERE brand_id=7 AND color_id=1 AND size_id=10
LIMIT 0
OPTION facet_filter_mode='auto'
FACET color_id ORDER BY color_id ASC
FACET size_id ORDER BY size_id ASC;

Response:

+----------+----------+-----------+
| color_id | count(*) | status    |
+----------+----------+-----------+
|        1 |        1 | selected  |
|        2 |        1 | available |
+----------+----------+-----------+
+---------+----------+-----------+
| size_id | count(*) | status    |
+---------+----------+-----------+
|      10 |        1 | selected  |
|      20 |        1 | available |
+---------+----------+-----------+

Here is what happened:

FACET color_id applied brand_id=7 AND size_id=10, but not color_id=1;
FACET size_id applied brand_id=7 AND color_id=1, but not size_id=10.

This gives the interface alternative values without a separate query for every facet. Selected buckets are marked selected, while other values in the same facet are marked available: they can be added to the current filter as a broader selection through IN (...).

max: a broad bucket list with statuses

auto shows only values from the current filter scope for a particular facet. max goes further: it calculates buckets over the base result set and marks their status separately for the interface.

SELECT count(*)
FROM products
WHERE brand_id=7 AND color_id=1 AND size_id=10
LIMIT 0
OPTION facet_filter_mode='max'
FACET color_id ORDER BY color_id ASC
FACET size_id ORDER BY size_id ASC;

Response:

+----------+----------+-----------+
| color_id | count(*) | status    |
+----------+----------+-----------+
|        1 |        3 | selected  |
|        2 |        1 | available |
|        3 |        1 | available |
|        4 |        1 | available |
|        5 |        1 | available |
+----------+----------+-----------+
+---------+----------+-----------+
| size_id | count(*) | status    |
+---------+----------+-----------+
|      10 |        4 | selected  |
|      20 |        2 | available |
|      30 |        1 | available |
+---------+----------+-----------+

status comes directly from Manticore Search:

selected - the value is already present in the filter on this facet;
available - the value can be selected; for another value in the same facet, this broadens the filter through IN (...);
unavailable - the value exists in the facet's broad result set, but selecting it would return no documents under the current filters.

In this example, color_id=1 occurs in three products across the catalog, so its count in max is 3. It is marked selected because it already participates in the filter. The other colors are marked available: if the user selects one, the color filter becomes broader, for example color_id IN (1,2). Unavailable values appear when a bucket in the broad result set cannot return documents under the current filters; the sku example below demonstrates this case.

Manually setting the filter scope

The global facet_filter_mode covers most common cases, but sometimes different facets need different behavior. For example, color can remain strictly constrained by all filters, size can use max, SKU can be calculated using only color and size, and brand can be calculated without the color filter.

SELECT count(*)
FROM products
WHERE brand_id=7 AND color_id=1 AND size_id=10
LIMIT 0
OPTION facet_filter_mode='max'
FACET color_id ALL FILTERS ORDER BY color_id ASC
FACET size_id ORDER BY size_id ASC
FACET sku FILTERS color_id, size_id ORDER BY sku ASC
FACET brand_id EXCLUDE FILTERS color_id ORDER BY brand_id ASC;

Response:

+----------+----------+-----------+
| color_id | count(*) | status    |
+----------+----------+-----------+
|        1 |        1 | selected  |
+----------+----------+-----------+
+---------+----------+-----------+
| size_id | count(*) | status    |
+---------+----------+-----------+
|      10 |        4 | selected  |
|      20 |        2 | available |
|      30 |        1 | available |
+---------+----------+-----------+
+------+----------+-------------+
| sku  | count(*) | status      |
+------+----------+-------------+
| sku1 |        1 | available   |
| sku5 |        1 | unavailable |
+------+----------+-------------+
+----------+----------+----------+
| brand_id | count(*) | status   |
+----------+----------+----------+
|        7 |        2 | selected |
+----------+----------+----------+

How to read this query:

FACET color_id ALL FILTERS applies every filter and returns only the selected color;
FACET size_id inherits the query-level max mode;
FACET sku FILTERS color_id, size_id applies only the color and size filters;
FACET brand_id EXCLUDE FILTERS color_id applies every filter except color.

This mode is useful when a filter panel contains both regular and technical facets, and some values need to be calculated according to special rules.

ZEROES: zero-count buckets in max

ZEROES is useful when counts need to remain strict but the list of values needs to stay broad. It works with max, either through OPTION facet_filter_mode='max' or through MODE max on an individual facet.

Without ZEROES, a facet with ALL FILTERS shows only the bucket that passes every filter:

SELECT count(*)
FROM products
WHERE brand_id=7 AND color_id=1 AND size_id=10
LIMIT 0
OPTION facet_filter_mode='max'
FACET size_id ALL FILTERS ORDER BY size_id ASC;

Response:

+---------+----------+----------+
| size_id | count(*) | status   |
+---------+----------+----------+
|      10 |        1 | selected |
+---------+----------+----------+

With ZEROES, Manticore Search keeps the same visible counts but returns the remaining buckets from the broad max scope with a count of zero:

SELECT count(*)
FROM products
WHERE brand_id=7 AND color_id=1 AND size_id=10
LIMIT 0
OPTION facet_filter_mode='max'
FACET size_id ALL FILTERS ZEROES ORDER BY size_id ASC;

Response:

+---------+----------+-----------+
| size_id | count(*) | status    |
+---------+----------+-----------+
|      10 |        1 | selected  |
|      20 |        0 | available |
|      30 |        0 | available |
+---------+----------+-----------+

The interface can then show large and xlarge alongside the selected small: the number refers to the current strict result set, while status shows that these values can still be selected to broaden the filter.

The same query through the JSON API

SQL is shorter here and shows the mechanics more clearly, but the JSON API supports the same approach:

POST /search
{
  "table": "products",
  "limit": 0,
  "query": {
    "bool": {
      "must": [
        { "equals": { "brand_id": 7 } },
        { "equals": { "color_id": 1 } },
        { "equals": { "size_id": 10 } }
      ]
    }
  },
  "facet_filter_mode": "max",
  "aggs": {
    "colors": {
      "terms": { "field": "color_id", "size": 10 },
      "sort": [ { "color_id": { "order": "asc" } } ]
    },
    "sizes": {
      "terms": { "field": "size_id", "size": 10 },
      "sort": [ { "size_id": { "order": "asc" } } ]
    }
  }
}

Response:

{
  "took": 0,
  "timed_out": false,
  "hits": {
    "total": 1,
    "total_relation": "eq",
    "hits": []
  },
  "aggregations": {
    "colors": {
      "buckets": [
        { "key": 1, "doc_count": 3, "status": "selected" },
        { "key": 2, "doc_count": 1, "status": "available" },
        { "key": 3, "doc_count": 1, "status": "available" },
        { "key": 4, "doc_count": 1, "status": "available" },
        { "key": 5, "doc_count": 1, "status": "available" }
      ]
    },
    "sizes": {
      "buckets": [
        { "key": 10, "doc_count": 4, "status": "selected" },
        { "key": 20, "doc_count": 2, "status": "available" },
        { "key": 30, "doc_count": 1, "status": "available" }
      ]
    }
  }
}

In JSON aggregations, mode, filters, exclude_filters, and zeroes can also be set for each individual aggregation.

How this differs from Meilisearch, Elasticsearch, and OpenSearch

We tested the same scenario in Manticore Search, Meilisearch, Elasticsearch, and OpenSearch: the brand_id=7, color_id=1, and size_id=10 filters are active; the result set remains strict, while the facets need to show options that appear when their own filter is excluded.

Engine	Excludes its own filter in one query	Native bucket status	What remains for the application
Manticore Search	Yes, through `auto`/`max`	Yes, in `auto`/`max`; `unavailable` only in `max`	Rendering the interface, SEO, analytics, and custom rules.
Meilisearch	Not in this form	No	Run additional queries and assemble facets in the application.
OpenSearch	Yes, but manually through `global` + `filter` aggregations	No	Duplicate filter branches and calculate `status` in the application.
Elasticsearch	Yes, but manually through `global` + `filter` aggregations	No	The same: maintain separate aggregation branches and client-side logic.

Basic facets are easy to use in Meilisearch, but a query such as:

{
  "filter": ["brand_id = 7", "color_id = 1", "size_id = 10"],
  "facets": ["color_id", "size_id"]
}

returns counts for the already filtered result set. In our dataset, that means only color_id=1 and size_id=10. The alternative color_id=2 and size_id=20 values do not appear in this response. If the interface needs them, the application must make additional queries and merge the results.

In Elasticsearch and OpenSearch, similar behavior can be built in one query, but each facet needs its own explicit aggregation branch. For color, you keep brand_id and size_id; for size, you keep brand_id and color_id; and so on. This works, but the query grows quickly, and bucket statuses still have to be calculated in the application.

The main difference is where this logic lives. In Manticore Search, it is configured directly in the facet API; in other systems, it is usually assembled from several similar filter trees.

Performance and limitations

strict remains the default and preserves the previous behavior. If you only need facets within the current result set, nothing needs to change.

auto is usually the better fit for e-commerce filters: it shows alternative values within each facet, marks selected values as selected, and marks other values as available.

Use max when the interface needs value lists broader than the current result set and needs to show unavailable options. This mode costs more: Manticore Search calculates buckets over a broad scope and then determines their status separately. This is worth considering with large datasets and many facets.

There are also some limitations:

local filter rewriting for a facet supports only attribute filters combined with AND;
complex AND/OR trees are not rewritten automatically for individual facets;
selected values are currently matched reliably only for explicit value filters such as = and IN;
statuses are not yet calculated automatically for ranges such as prices.

For prices, discounts, and ratings, it is better to define separate ranges explicitly. For example, keep a numeric field for sorting and sliders, and add price_band or discount_band for the facet. This lets the interface show clear statuses for predefined ranges.

SEO for faceted URLs, A/B tests, merchandising rules, and query analytics remain the responsibility of the application or platform around search.

In practice, start with auto when you need alternative values, and move to max when the interface needs a broad bucket list with selected, available, and unavailable statuses.

Install Manticore Search with one command

Sergey Nikolaev — Wed, 29 Jul 2026 03:24:47 +0000

There is now a one-line installer for Manticore Search:

curl https://manticoresearch.com | sh

It is still a normal package-manager install. The script checks whether the machine uses APT, YUM/DNF, or Homebrew, configures the Manticore repository when needed, installs the manticore package, and starts the service unless you tell it not to.

Use it when you want to bring up Manticore Search quickly and do not need to walk through the package-manager steps by hand. If you prefer to control the repository setup, package install, service start, or deployment method yourself, the manual installation pages are still the right place to go.

After it runs

A successful run leaves Manticore installed through the system package manager. Upgrades, service management, uninstall, and host inspection use the usual system tools.

By default, the installer:

installs from the stable Manticore package repository
installs the manticore package
starts the manticore service
keeps the default listeners on 9306, 9308, and 9312
writes an install log for troubleshooting

Check the service

Once the service is started, a quick SQL-over-HTTP check is enough:

curl -sS "http://127.0.0.1:9308/sql?mode=raw" -d "SELECT VERSION()"

Or use the MySQL client:

mysql -P9306 -h0 -e "SELECT VERSION()"

If that returns a version, the package is installed and the service is answering.

Why add another install path?

The manual instructions are not going away. They are still better when you need Docker, Kubernetes, Windows, a pinned package-manager flow, or a deployment policy that does not allow remote shell scripts.

The one-line installer covers the shorter path. Instead of asking the reader to pick between APT, YUM/DNF, and Homebrew first, it detects the package manager and runs the same standard setup:

Debian or Ubuntu: add the APT repo, update metadata, install the package, start the service.
RHEL-compatible distribution: install the repository RPM, install the package, start the service.
macOS: use Homebrew, then start the service through brew services.
Other environments: stop and point to the manual options.

Small difference, but useful when you just want a running local or test instance.

What the script does

The installer is a shell script. If you want to read it first, the source is installer/bootstrap-standalone.sh in the Manticore Search GitHub repository.

On a system it supports, it:

detects APT, YUM/DNF, or Homebrew
checks that the CPU architecture is x86-64 or ARM64
installs the Manticore repository package when needed
installs the manticore package
starts the manticore service by default
checks the default listener ports before startup when ss or netstat is available
writes an install log to /var/log/manticore_install.log when possible, otherwise to /tmp/manticore_installer.log

The default listener ports are 9306, 9308, and 9312. If one of them is already occupied, Manticore is not already running, and the system has ss or netstat, the installer fails instead of editing the configuration for you. If neither utility is available, the pre-check cannot run; service startup may still fail if a port is already taken.

Port changes are left to you because changing them can affect clients, service files, and monitoring.

Where it applies

The installer currently supports:

Debian, Ubuntu, Linux Mint, and Debian-like systems using APT
RHEL, CentOS, Fedora, Rocky Linux, AlmaLinux, Amazon Linux, and RHEL/Fedora-like systems using YUM or DNF
macOS when Homebrew is installed
x86-64 and ARM64 CPUs

The command shown above uses curl. You can also fetch the script with wget:

wget -O- https://manticoresearch.com | sh

Bash must be installed. The downloaded script starts under /bin/sh, checks for Bash, and then runs the main installer payload with Bash. You also need the usual permissions to install packages and start services, either as root or through sudo.

If the operating system, package manager, or CPU architecture is not supported, the installer stops and prints a message pointing to the manual installation guide.

Default behavior

With no extra options, the installer uses the stable release channel, installs the package, starts the service, and keeps the standard listener ports:

stable packages are used unless you pass dev
the service starts unless you pass no-start
listener ports stay at 9306, 9308, and 9312
existing data is not removed

Other checks apply only in specific cases. Before an upgrade, the installer backs up configuration. It copies data directories only if you pass backup-data. During uninstall and purge, it keeps data in place. If it finds an existing searchd binary outside the package-managed install, it prints a warning.

The destructive option is still separate: purge-all removes packages, repository state, configuration, and data. Without non-interactive mode, it asks you to type DELETE before removing /etc/manticoresearch and /var/lib/manticore.

Common commands

Most installer behavior is controlled by passing options after sh -s:

# Install the latest stable package and start the service
curl https://manticoresearch.com | sh

# Install but do not start the service
curl https://manticoresearch.com | sh -s no-start

# Upgrade an existing package-managed installation
curl https://manticoresearch.com | sh -s upgrade

# Upgrade and include the data directory in the backup
curl https://manticoresearch.com | sh -s upgrade backup-data

# Choose where upgrade backups are written
curl https://manticoresearch.com | sh -s upgrade backup-data backup-dir /path/to/backups

# List available stable versions
curl https://manticoresearch.com | sh -s list-versions

# List versions from the development repository
curl https://manticoresearch.com | sh -s dev list-versions

# Install or switch to a specific version without starting the service
curl https://manticoresearch.com | sh -s version 25.0.0 no-start

# Show installer help
curl https://manticoresearch.com | sh -s help

Specific version selection is for Linux package-manager installs. Homebrew installs and upgrades follow the Homebrew formula and do not support version through this installer.

The dev option switches to the development repository. The default is the release channel.

CI and non-interactive use

For automation, use silent or yes only when you want the installer to run without prompts and accept the script's defaults:

curl https://manticoresearch.com | sh -s silent no-start

silent and yes are aliases for the same non-interactive mode. They do not mean "no output": the installer still writes progress and errors, but without interactive prompts and color. They also do not request an upgrade by themselves. If an existing installation should be upgraded, pass upgrade explicitly:

curl https://manticoresearch.com | sh -s upgrade silent

Be careful with cleanup commands in non-interactive mode. In the current installer, silent also skips the "DELETE" prompt for purge-all, so it can delete configuration and data without asking again.

Cleanup options

Uninstalling is split into levels:

# Remove packages, keep configuration, data, and repository state
curl https://manticoresearch.com | sh -s uninstall

# Remove packages and the repository bootstrap package
curl https://manticoresearch.com | sh -s purge

# Remove packages, repository state, configuration, and data
curl https://manticoresearch.com | sh -s purge-all

purge-all is the destructive option. Do not combine it with silent or yes unless you intentionally want to remove /etc/manticoresearch and /var/lib/manticore without a confirmation prompt:

curl https://manticoresearch.com | sh -s purge-all silent

Removing software and deleting data are not the same operation, so the installer keeps those paths separate.

If your policy forbids pipe-to-shell

Some environments do not allow curl | sh. That is fine. Use the manual package-manager instructions, or review the script source first and run it only after inspection.

For manual package-manager commands, Docker, Kubernetes, Windows, and separate packages, see the Manticore Search installation guide.

Manticore Search authentication rollout checklist for production

Sergey Nikolaev — Tue, 28 Jul 2026 06:15:49 +0000

In production, "turn it on and done" almost never works. On a standalone node, the technical sequence is short: enable auth, restart Manticore, create an admin user, and update clients. A topology with distributed tables or replication clusters needs extra preparation because nodes must authenticate to each other as well.

Handle the rollout like a small release. Inventory clients and nodes, prepare the auth data, rehearse the procedure for your topology, and then cut over. A rehearsal exposes failures before the maintenance window.

This checklist is for users who plan to enable authentication and want to roll it out as safely as possible in an existing system. Remember that authentication is disabled until you configure auth; after cutover, clients that still omit credentials will fail.

Choose the procedure by topology:

Topology	Migration requirement
standalone node	bootstrap the first administrator after enabling `auth`
distributed tables and remote agents	distribute one canonical auth store before authenticated remote queries begin
replication cluster	prepare the persisted cluster user and auth store before startup, then recover the cluster in a controlled order

Phase 1: inventory before changing anything

Start by listing every client that connects to Manticore Search, including each application and independently deployed application component. Do this before editing the config.

Common things to check:

application search frontends
ingest workers
cron jobs
dashboards and BI tools
support or admin tools
schema change/update scripts
backup and maintenance scripts
local scripts run manually

For each client, record something like this:

Client	Protocol	Tables or clusters	Needed actions	Notes
product search app	HTTP	`products`	`read`	will use Bearer token
catalog ingest worker	SQL	`products`	`write`	uses password auth
schema migration job	SQL	`products`	`schema`	run only during deploy
access administrator	SQL	`*`	`admin`	manages users and permissions

Then confirm the deployment basics:

Is this a standalone node, a distributed-table topology, a replication cluster, or a combination of them?
Are you running RT mode or plain mode?
In plain mode, where should the auth file live?
Is pid_file configured in the config? Bootstrap needs it.
Are all participating nodes running a version that supports the same authentication protocol?
Will SQL clients use SSL, and will HTTP clients use HTTPS when credentials cross a network?
Where will credentials, including temporary Bearer tokens, be stored safely?
Who is allowed to see the first admin password?

If replication is present, also record every cluster name and its persisted user in each node's <data_dir>/manticore.json. Back up the complete data directory, configuration, and any existing auth store before the rehearsal and again before production cutover.

Do not skip the credential-handling questions. CREATE USER returns a raw Bearer token; TOKEN generates a new token for the specified user. SHOW TOKEN later shows the stored token hash, not the raw token. If the raw token is lost, rotate it with TOKEN and update the token in your application.

Phase 2: set up least-privilege users

Create users based on specific workloads, rather than granting permissions based on what someone might need later.

Use a small matrix like this:

Workload	User	Permissions
search frontend	`app_read`	`GRANT read ON 'products' TO 'app_read'`
ingest worker	`app_ingest`	`GRANT write ON 'products' TO 'app_ingest'`
schema migration job	`schema_job`	`GRANT schema ON 'products' TO 'schema_job'`
auth operator	`security_admin`	`GRANT admin ON * TO 'security_admin'`
replication operator	`cluster_repl`	`GRANT replication ON 'posts' TO 'cluster_repl'`

Keep in mind that admin is a narrow permission. It only allows managing authentication and authorization state; it does not imply read, write, schema, or replication.

That matters because a person who manages credentials does not automatically need to read business data. A service that searches products does not need to write documents. A migration job does not need to manage users.

Plan negative permission tests as well. For every user you create, choose at least one thing it should be able to do and one thing it should not be able to do.

Examples:

app_read can search products.
app_read cannot insert into products.
app_ingest can write to products.
app_ingest cannot manage auth.
schema_job can change schema for products.
schema_job cannot read tables unless you grant read.

Phase 3: test in staging

Use a staging environment to rehearse the sequence of steps you plan to follow in production.

In RT mode:

searchd {
    data_dir = /var/lib/manticore
    auth = 1
    auth_log_level = info
    ...
}

To turn auth off explicitly in RT mode:

searchd {
    data_dir = /var/lib/manticore
    auth = 0
    ...
}

In plain mode:

searchd {
    auth = /var/lib/manticore/auth.json
    auth_log_level = info
    ...
}

Keep the auth file private. Before the first bootstrap, Manticore may create an empty auth file. After bootstrap, that file stores auth data and credential hashes.

This bootstrap sequence works on a standalone node or on an isolated temporary daemon that prepares auth data for several nodes. Never start an existing replication data directory with an empty auth store. Its persisted cluster user will be missing, and Manticore may skip the cluster descriptor.

Start searchd, then bootstrap the first administrator:

searchd --config /etc/manticoresearch/manticore.conf --auth

For scripted setup:

printf 'admin\nStrongPass#2026\nStrongPass#2026\n' | \
  searchd --config /etc/manticoresearch/manticore.conf --auth-non-interactive

⚠️ Warning: The command above includes the password in plain text. In production automation, feed the three input lines to standard input from your secret-management system.

Bootstrap creates the first administrator with all actions, including replication; no additional grant is needed. The command does not return a bearer token. If the administrator needs HTTP Bearer access, connect as that user and run TOKEN, or use the HTTP POST /token endpoint.

For a multi-node deployment, create the auth store once. Use a temporary daemon with its own empty data directory, PID file, and listeners. Bootstrap the administrator and shared service users, then stop the daemon cleanly. Copy the resulting auth store to every participating node before enabling authenticated node-to-node traffic. Do not recreate the same users independently: matching names and passwords can still produce different stored authentication material.

Next, create staging users based on the notes from the previous phases. For example:

CREATE USER 'app_read' IDENTIFIED BY 'ReadPass#2026';
GRANT read ON 'products' TO 'app_read';

CREATE USER 'app_ingest' IDENTIFIED BY 'IngestPass#2026';
GRANT write ON 'products' TO 'app_ingest';

CREATE USER 'schema_job' IDENTIFIED BY 'SchemaPass#2026';
GRANT schema ON 'products' TO 'schema_job';

CREATE USER 'security_admin' IDENTIFIED BY 'AdminPass#2026';
GRANT admin ON * TO 'security_admin';

Store the returned Bearer tokens in secure storage. Remember: raw tokens must not be left in logs, shell history, or unprotected files. If you need to rotate one, use:

TOKEN 'app_read';

SHOW TOKEN is not a way to recover the raw token:

SHOW TOKEN FOR 'app_read';

Review users and permissions:

SHOW USERS;
SHOW PERMISSIONS;
SHOW PERMISSIONS FOR 'app_read';

Run one allow and one deny test for every user. For the read-only user:

curl -H "Authorization: Bearer <app_read_token>" \
  http://127.0.0.1:9308/sql?mode=raw \
  -d "SELECT * FROM products LIMIT 10"

Then verify that an unauthorized operation is denied:

curl -H "Authorization: Bearer <app_read_token>" \
  http://127.0.0.1:9308/sql?mode=raw \
  -d "INSERT INTO products(id,title) VALUES(1,'test')"

For this HTTP request, expect a 403 Forbidden response. Over SQL/MySQL, a permission denial returns ERROR 1045 with a permission-denied message.

SQL clients should connect with a Manticore user name and password. The SQL/MySQL protocol in Manticore supports mysql_native_password.

MYSQL_PWD=ReadPass#2026 \
  mysql -h127.0.0.1 -P9306 -uapp_read \
  -e "SELECT * FROM products LIMIT 10"

HTTP clients can use Basic authentication or Bearer tokens:

curl -u app_read:ReadPass#2026 \
  http://127.0.0.1:9308/sql?mode=raw \
  -d "SELECT * FROM products LIMIT 10"

HTTP authentication schemes (Basic, Bearer) are case-insensitive; user names are case-sensitive.

If you edit the auth file outside the daemon during maintenance, reload it:

RELOAD AUTH;

Phase 4: production rollout checklist

Use this checklist for every deployment, then follow the procedure for your topology.

[ ] Confirm you have a current configuration and data-directory backup.
[ ] Back up manticore.json and the existing auth store separately.
[ ] Confirm existing network protections stay in place.
[ ] Confirm pid_file is set in the config.
[ ] Confirm where the auth file will be created or loaded from.
[ ] Confirm password and token storage is ready.
[ ] Confirm every participating node runs a compatible Manticore version.
[ ] Rehearse the same topology and restart order in staging.
[ ] Prepare one canonical auth store for users shared across nodes.
[ ] Choose the standalone, distributed, or replication procedure below.
[ ] Enable auth during a planned maintenance window.
[ ] Expect unauthenticated clients to fail after auth is enabled.
[ ] Store tokens in a protected secrets store immediately after they are issued. Do not store raw tokens in files, shell history, or logs.
[ ] Update SQL connection code to send user names and passwords.
[ ] Update HTTP connection code to use Basic authentication or Bearer tokens.
[ ] Run the allow and deny tests from staging.
[ ] Verify internal node-to-node operations when the deployment has remote agents or replication.
[ ] Check the auth log.
[ ] Run application stress tests that cover search, ingest, dashboards, and maintenance scripts.
[ ] Rotate any temporary rollout credentials.
[ ] Keep the first admin credential out of normal application use.

Standalone node

For a standalone node, the direct bootstrap sequence is sufficient:

Stop Manticore cleanly and take the final backup.
Configure auth and start searchd.
Bootstrap the first administrator with searchd --config <path> --auth.
Create the production users and permissions.
Update clients and run the planned allow and deny tests.
Confirm that existing tables and known rows are still available.

Distributed tables and remote agents

Distributed queries send remote-agent requests as the current session user. Each remote node must have the same stored authentication material for that user and grant the required permission on the remote table.

For a new rollout across distributed nodes:

Create the shared users once in the isolated bootstrap daemon from Phase 3.
Stop the affected agents and masters for the coordinated cutover.
Configure auth and place the same canonical auth store on every participating node. Preserve restrictive ownership and permissions, and compare checksums.
Start remote agents before the masters that query them.
Test a direct authenticated query on each agent, followed by the equivalent distributed query through the master.
Create node-local users only after shared traffic works, and keep shared user records synchronized whenever their passwords, tokens, or permissions change.

Create shared users only once. Independently created accounts can have different stored authentication material even when their names and passwords match.

Existing replication cluster

Moving an existing unauthenticated replication cluster to auth requires a coordinated restart. Do not enable auth and bootstrap the first user against the existing cluster data. The empty store does not contain the persisted cluster user, so Manticore may skip the cluster descriptor.

Use this sequence:

While the unauthenticated cluster is healthy, choose its future replication identity and persist it:

   ALTER CLUSTER products UPDATE user 'cluster_repl';

UPDATE user writes the name to the cluster metadata. Authentication is still disabled, so Manticore neither creates nor checks the account at this point. Create it in step 3, before restarting any real cluster node with authentication enabled.

Verify that every node's <data_dir>/manticore.json now stores "user": "cluster_repl" for the cluster. If a node hosts several clusters, update each cluster to a user that will exist in the new auth store, or create and grant every persisted user before cutover.

Stop all cluster nodes cleanly and use the replication state to select the safe primary. After a clean shutdown, this is normally the node stopped last, with safe_to_bootstrap: 1 in its <data_dir>/grastate.dat. See Restarting a cluster.
Bootstrap cluster_repl as the first administrator in the isolated temporary daemon. The first administrator already has every action, including replication, so it needs no additional grant. If the cluster will use a separate least-privilege identity, create it once and grant replication before distributing the store.
Stop the temporary daemon. Configure auth on every real cluster node and copy the exact generated auth store to each one. Keep the files private and byte-identical. Do not start a real cluster node before this store is in place.
Restart a two-node cluster in this order:
1. Start the non-primary peer normally and wait only for its daemon listener. Its cluster can report closed at this point; do not write to it.
2. Start the recorded safe primary with --new-cluster, or use the corresponding manticore_new_cluster service action.
3. Wait for the safe primary to report cluster_products_status=primary and cluster_products_node_state=synced.
4. Stop the first peer cleanly and start it normally again. Wait for the same primary and synced values on that peer. Never use --new-cluster on it.

The safe primary reads the persisted cluster user from a descriptor peer during startup, so the initial peer must already be listening. Starting the safe primary alone can fail with failed to fetch donor user from any node even when the auth store is correct. For a larger cluster, rehearse the sequence in staging. Use one non-primary node as the initial metadata peer, establish the safe primary, and then start or restart the remaining peers normally.

On each node, check that the cluster component is primary, the local node is synchronized, and the pre-migration data is present:

   SHOW STATUS LIKE 'cluster_products_status';
   SHOW STATUS LIKE 'cluster_products_node_state';
   SELECT COUNT(*) FROM products:existing_table;

Wait until the two status values are primary and synced, respectively, before treating the node as writable.

Before returning traffic, make a disposable one-row table and add it to the recovered cluster:

   CREATE TABLE migration_control (id bigint, body text);
   INSERT INTO migration_control VALUES (1, 'post-auth control');
   ALTER CLUSTER products ADD migration_control;

Confirm that the peer returns one row from products:migration_control. If this check fails after the pre-migration data checks passed, investigate table transfer rather than the migration itself.

After the cluster is healthy, create the remaining administrative users and, if needed, a dedicated least-privilege replication user. Change the stored cluster identity only after the new user and its auth data are visible on every node:

CREATE USER 'repluser' IDENTIFIED BY '<strong-secret>';
GRANT replication ON * TO 'repluser';
ALTER CLUSTER products UPDATE user 'repluser';

Do not remove the bootstrap administrator until another administrator and the final replication identity have both been verified.

When a node joins an authenticated cluster, the donor's auth data replaces the joining node's local auth data. At info or a more verbose auth log level, Manticore writes the previous data to searchd.log.auth as a backup. The log can contain salts and credential hashes, so restrict access and redact it before sharing.

Authentication logging during cutover

Authentication events are written to a separate auth log when authentication is enabled. If the daemon log is /var/log/manticore/searchd.log, the auth log is /var/log/manticore/searchd.log.auth.

The auth_log_level values are:

disabled
error
warning
info
all
trace

The default is info. Start there unless you have a reason to reduce or increase the logging detail. Use trace only for diagnostics; it also includes all successful internal auth traffic.

Useful cleanup and maintenance commands:

SET PASSWORD 'NewReadPass#2026' FOR 'app_read';
REVOKE read ON 'products' FROM 'app_read';
DROP USER 'app_read';

SET PASSWORD changes the password used by SQL/MySQL and HTTP Basic auth. It does not revoke existing bearer tokens. To rotate Bearer access, create a new token with TOKEN or POST /token and update the client.

Phase 5: rollback and troubleshooting

For a standalone node, restore the previous configuration and network restrictions, restart Manticore, and revert the client configuration if necessary.

Roll back all communicating nodes together. A mix of authenticated and unauthenticated nodes will not work. Restore the same configuration and auth state on each node, then restart remote agents before their masters.

Keep the pre-cutover manticore.json backup for every replication cluster. If a node started with auth before the persisted cluster user existed, stop it and compare the current descriptor with the backup. A clean stop may have saved the skipped state without the cluster descriptor; restore the backed-up descriptor before retrying. Do not recreate clustered tables or delete their data.

Do not delete the rollout notes. They are usually the fastest way to see which client was updated, which token was stored where, and which permissions were created.

Symptom	Likely cause	Check
SQL access denied	Wrong user, wrong password, or client auth mismatch	Check the configured user and confirm the client can use `mysql_native_password`.
HTTP 401	Missing or invalid credentials	Check the `Authorization` header and whether the client uses Basic auth or Bearer token auth.
HTTP 403	User authenticated but lacks permission	Check `SHOW PERMISSIONS FOR '<user>'`.
Bearer token does not work	Token was lost, copied incorrectly, or already revoked	Run `TOKEN '<user>'`, store the returned token, and update the client.
User has fewer permissions than expected	Missing action grant	Check whether the operation needs `read`, `write`, `schema`, `replication`, or `admin`.
User has more permissions than expected	Broad target or missing explicit deny	Check wildcard grants, exact target grants, and any `WITH ALLOW 0` rules.
Distributed query is denied remotely	Shared user is missing, differs, or lacks permission on the agent	Compare the auth stores and permissions on the master and agent.
Existing cluster is skipped at startup	Persisted cluster user is missing from the auth store or lacks `replication`	Check `manticore.json`, `SHOW PERMISSIONS`, and the pre-cutover backup before restarting.
`failed to fetch donor user from any node`	No descriptor peer is available, or daemon-to-daemon authentication failed	Check the restart order, peer availability, and `searchd.log.auth` on both nodes.

Permission rules are determined by action type. When rules conflict, an explicit deny always takes precedence over an allow, even if the allow is more specific. If no matching allow exists, access is denied.

Final check

Before calling the rollout done, confirm that:

[ ] You used the procedure for the deployment's topology.
[ ] Every isolated part of the system that uses Manticore Search has its own user.
[ ] Every user has only the actions it needs.
[ ] Users shared between nodes have the same stored authentication material.
[ ] Bearer tokens are stored in a protected secrets store; raw tokens are not saved outside a controlled environment.
[ ] The operations team knows that SHOW TOKEN does not return the raw token, but shows its hash; use TOKEN or the HTTP endpoint to get a new token.
[ ] SQL and HTTP clients have been updated.
[ ] Expected denials were tested.
[ ] You tested distributed queries or replication operations when applicable.
[ ] Every migrated replication cluster is synced, and its existing data is present on every node.
[ ] Auth logs are visible.
[ ] The rollback procedure is clear and documented.

We wish you a smooth authentication and authorization rollout!

DEV Community: Sergey Nikolaev

Manticore Search 29.9.0: Chunked auto-embeddings and mmap columnar access

Manticore Search 29.9.0: Chunked auto-embeddings and mmap columnar access

Upgrade notes

Long documents can keep more than one embedding

Put a ceiling on local embedding work

UTF-8 identifiers and better German matching

Better load testing and backups

Columnar attributes use mmap by default

Vector and grouped-search fixes

Bulk ingestion behaves predictably

More reliability fixes

Get Manticore Search 29.9.0

Need help or want to connect?

Explore Manticore Data with OpenSearch Dashboards

Explore Manticore Data with OpenSearch Dashboards

How the integration works

Prerequisites

Step 1: Configure Manticore

Step 2: Configure OpenSearch Dashboards

Step 3: Add sample data to Manticore

Step 4: Create an index pattern

Step 5: Explore the data in Discover

Step 6: Build a visualization

What is supported

Current limitations

Bringing in real data

Conclusion

How to return top-N from each group with GROUP_CONCAT()

How to return top-N from each group with GROUP_CONCAT()

Events we'll work with

What's wrong with the old approaches

Combining only the five most recent IDs

If a comma doesn't work

Where the new approach doesn't work

More examples

How to search for long hashes and IDs with dict='keywords_32k'

How to search for long hashes and IDs with dict='keywords_32k'

The problem with the regular dictionary

What keywords_32k changes

When to use keywords_32k

If you only need exact equality

If you need both exact matching and full-text search

Searching by the complete token

A full-text match is not the same as exact equality

How to inspect tokenization

Prefix and substring search

Email addresses and other values with separators

How to convert an existing table

Why dict='crc' does not solve this problem

Current limitations

Do not index secrets

Quick checklist

Summary

Documentation

How KupujemProdajem searches 5.6 million ads at ~10 ms with Manticore Search

How KupujemProdajem searches 5.6 million ads at ~10 ms with Manticore Search

Search is part of the core product

Choosing a search engine that could grow with the product

Keeping millions of changing ads searchable in real time

More than matching words

A small replicated cluster

5.6 million active ads, 70 million admin rows

What users get from this architecture

The next challenge: understanding Serbian-language intent

A foundation for better marketplace search

Full-Text Search Still Works. It Just Doesn’t Get You to an Answer

Word search, semantic search, and conversation are different tasks

What we built

How it works

Through the HTTP API

What the system returns

How retrieval works

Where search ends and conversation begins

Quality

Conclusion

UUID in Manticore: A Practical Guide

Table for all examples

Working with UUID through SQL

The same operations through the JSON API

What `keywords_32k` changes

When to use `keywords_32k`

Why `dict='crc'` does not solve this problem

Batch loading through `/bulk`