DEV Community

Cover image for How to use Elasticdump with Manticore
Sergey Nikolaev
Sergey Nikolaev

Posted on

How to use Elasticdump with Manticore

Elasticdump is a tool used for managing and migrating data in Elasticsearch. Elasticdump allows users to move and save indices by exporting data to a JSON file and then importing it into another place. This functionality is particularly useful for backup and recovery purposes, and for migrating data between different environments (such as from development to production).

Now you can use Elasticdump to import your Elasticsearch indices to Manticore as well. You can choose one of two options available: importing a whole index or importing only its schema.

This functionality is only available in the development version as of now (February 27th, 2024). Here's how you can install development packages: https://mnt.cr/dev/nightly.

Importing schema and data

To copy an Elasticsearch index to Manticore including the data you can use the following command:

elasticdump --input=http://localhost:9200/your_elasticsearch_index  --output=http://localhost:9308/your_manticore_table --type=data
Enter fullscreen mode Exit fullscreen mode

Let's look at a simple example of using it. Here is the original Elasticsearch index named test with this schema:

# curl -sX GET localhost:9200/test/_mapping

{
  "test": {
    "mappings": {
    "properties": {
          "price": {
          "type": "float"
          },
          "title": {
          “type": "text"
          }
    }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

and 300 documents in it:

# curl -sX GET localhost:9200/test/_count

{
  "count": 300,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  }
}

Enter fullscreen mode Exit fullscreen mode

So far, we don't have any tables in Manticore:

# curl -sX GET localhost:9308/cli -d 'SHOW TABLES'

Empty set (0.000 sec)
Enter fullscreen mode Exit fullscreen mode

Now, let's run Elasticdump:

# elasticdump --input=http://localhost:9200/test --output=http://localhost:9308/test_imported --type=data

Tue, 26 Feb 2024 16:18:52 GMT | starting dump
(node:156) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
Tue, 26 Feb 2024 16:18:52 GMT | got 300 objects from source elasticsearch (offset: 0)
Tue, 26 Feb 2024 16:18:52 GMT | sent 300 objects to destination elasticsearch, wrote 300
Tue, 26 Feb 2024 16:18:52 GMT | got 0 objects from source elasticsearch (offset: 300)
Tue, 26 Feb 2024 16:18:52 GMT | Total Writes: 300
Tue, 26 Feb 2024 16:18:52 GMT | dump complete
Enter fullscreen mode Exit fullscreen mode

And check the results of the import we've made:

# curl -sX GET localhost:9308/cli -d 'DESC test_imported'

+-------+--------+----------------+
| Field | Type   | Properties     |
+-------+--------+----------------+
| id    | bigint |                |
| title | text   | indexed stored |
| price | float  |                |
+-------+--------+----------------+
3 rows in set (0.000 sec)
Enter fullscreen mode Exit fullscreen mode
# curl -sX GET localhost:9308/cli -d 'SELECT COUNT(*) FROM test_imported'

+----------+
| count(*) |
+----------+
| 300      |
+----------+
1 row in set (0.000 sec)
Enter fullscreen mode Exit fullscreen mode

We can see that the table has been copied.

Import schema only

To import only the table schema, run:

# elasticdump --input=http://localhost:9200/your_elasticsearch_index --output=http://localhost:9308/your_manticore_table --type=mapping
Enter fullscreen mode Exit fullscreen mode

You will get a new empty Manticore table with the structure imported from the Elasticsearch node.

Note that in the examples above the default Elasticsearch and Manticore ports are used: localhost:9200 and localhost:9308, respectively. If you use other hosts/ports for connecting to Elasticsearch or Manticore, you should change your command appropriately.

Also, keep in mind that Manticore converts some Elasticsearch data types to the ones of its own, if it does not support them internally:

  • aggregate_metric => json
  • binary => string
  • boolean => bool
  • byte => int
  • completion => string
  • date => timestamp
  • date_nanos => bigint
  • date_range => json
  • dense_vector => json
  • flattened => json
  • float => float
  • float_range => json
  • geo_point => json
  • geo_shape => json
  • half_float => float
  • histogram => json
  • integer => int
  • integer_range => json
  • ip => string
  • ip_range => json
  • keyword => string
  • long => bigint
  • long_range => json
  • match_only_text => text
  • object => json
  • point => json
  • scaled_float => float
  • search_as_you_type => text
  • shape => json
  • short => int
  • text => text
  • unsigned_long => int
  • version => string

For example, if you import an Elasticsearch index as follows:

{
 "test": {
   "mappings": {
     "properties": {
       "location": {
         "type": "geo_point"
       },
       "title": {
         "type": "text"
       }
     }
   }
 }
}
Enter fullscreen mode Exit fullscreen mode

the imported Manticore table will have the following schema:

+----------+--------+----------------+
| Field    | Type   | Properties     |
+----------+--------+----------------+
| id       | bigint |                |
| title    | text   | indexed stored |
| location | json   |                |
+----------+--------+----------------+
Enter fullscreen mode Exit fullscreen mode

Conclusion

In conclusion, Manticore Search now supports Elasticdump, making it easier for users to move from Elasticsearch to Manticore. This update simplifies the migration process, allowing users to transfer data and schema smoothly. It's a big step for those looking to switch to Manticore Search without the hassle of a complicated data transfer. This integration means a straightforward migration that keeps your data intact, making Manticore Search an appealing choice for enhancing search capabilities with ease.

Top comments (0)