How to use Elasticdump with Manticore

Elasticdump is a tool used for managing and migrating data in Elasticsearch. Elasticdump allows users to move and save indices by exporting data to a JSON file and then importing it into another place. This functionality is particularly useful for backup and recovery purposes, and for migrating data between different environments (such as from development to production).

Now you can use Elasticdump to import your Elasticsearch indices to Manticore as well. You can choose one of two options available: importing a whole index or importing only its schema.

This functionality is only available in the development version as of now (February 27th, 2024). Here's how you can install development packages: https://mnt.cr/dev/nightly.

Importing schema and data

To copy an Elasticsearch index to Manticore including the data you can use the following command:

elasticdump --input=http://localhost:9200/your_elasticsearch_index  --output=http://localhost:9308/your_manticore_table --type=data

Let's look at a simple example of using it. Here is the original Elasticsearch index named test with this schema:

# curl -sX GET localhost:9200/test/_mapping

{
  "test": {
    "mappings": {
    "properties": {
          "price": {
          "type": "float"
          },
          "title": {
          “type": "text"
          }
    }
    }
  }
}

and 300 documents in it:

# curl -sX GET localhost:9200/test/_count

{
  "count": 300,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  }
}

So far, we don't have any tables in Manticore:

# curl -sX GET localhost:9308/cli -d 'SHOW TABLES'

Empty set (0.000 sec)

Now, let's run Elasticdump:

# elasticdump --input=http://localhost:9200/test --output=http://localhost:9308/test_imported --type=data

Tue, 26 Feb 2024 16:18:52 GMT | starting dump
(node:156) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
Tue, 26 Feb 2024 16:18:52 GMT | got 300 objects from source elasticsearch (offset: 0)
Tue, 26 Feb 2024 16:18:52 GMT | sent 300 objects to destination elasticsearch, wrote 300
Tue, 26 Feb 2024 16:18:52 GMT | got 0 objects from source elasticsearch (offset: 300)
Tue, 26 Feb 2024 16:18:52 GMT | Total Writes: 300
Tue, 26 Feb 2024 16:18:52 GMT | dump complete

And check the results of the import we've made:

# curl -sX GET localhost:9308/cli -d 'DESC test_imported'

+-------+--------+----------------+
| Field | Type   | Properties     |
+-------+--------+----------------+
| id    | bigint |                |
| title | text   | indexed stored |
| price | float  |                |
+-------+--------+----------------+
3 rows in set (0.000 sec)

# curl -sX GET localhost:9308/cli -d 'SELECT COUNT(*) FROM test_imported'

+----------+
| count(*) |
+----------+
| 300      |
+----------+
1 row in set (0.000 sec)

We can see that the table has been copied.

Import schema only

To import only the table schema, run:

# elasticdump --input=http://localhost:9200/your_elasticsearch_index --output=http://localhost:9308/your_manticore_table --type=mapping

You will get a new empty Manticore table with the structure imported from the Elasticsearch node.

Note that in the examples above the default Elasticsearch and Manticore ports are used: localhost:9200 and localhost:9308, respectively. If you use other hosts/ports for connecting to Elasticsearch or Manticore, you should change your command appropriately.

Also, keep in mind that Manticore converts some Elasticsearch data types to the ones of its own, if it does not support them internally:

• aggregate_metric => json
• binary => string
• boolean => bool
• byte => int
• completion => string
• date => timestamp
• date_nanos => bigint
• date_range => json
• dense_vector => json
• flattened => json
• float => float
• float_range => json
• geo_point => json
• geo_shape => json
• half_float => float
• histogram => json
• integer => int
• integer_range => json
• ip => string
• ip_range => json
• keyword => string
• long => bigint
• long_range => json
• match_only_text => text
• object => json
• point => json
• scaled_float => float
• search_as_you_type => text
• shape => json
• short => int
• text => text
• unsigned_long => int
• version => string

For example, if you import an Elasticsearch index as follows:

{
 "test": {
   "mappings": {
     "properties": {
       "location": {
         "type": "geo_point"
       },
       "title": {
         "type": "text"
       }
     }
   }
 }
}

the imported Manticore table will have the following schema:

+----------+--------+----------------+
| Field    | Type   | Properties     |
+----------+--------+----------------+
| id       | bigint |                |
| title    | text   | indexed stored |
| location | json   |                |
+----------+--------+----------------+

Conclusion

In conclusion, Manticore Search now supports Elasticdump, making it easier for users to move from Elasticsearch to Manticore. This update simplifies the migration process, allowing users to transfer data and schema smoothly. It's a big step for those looking to switch to Manticore Search without the hassle of a complicated data transfer. This integration means a straightforward migration that keeps your data intact, making Manticore Search an appealing choice for enhancing search capabilities with ease.