Journey from Elastic Cloud to AWS Elastic Search

This article demonstrates the steps that we should follow when we migrate Elastic search data from Elastic cloud to AWS.

Let's get started.

The Elastic Cloud supports Major.Minor.Patch (eg: x.y.z) version of Elastic Search, where AWS supports only Major. Minor (eg: x.y).

When you build the target cluster on AWS, ensure you created the cluster using the next latest Minor version of the source cluster. (If the source cluster version is 7.7.1, the target cluster version must be 7.8).

Elastic Search doesn’t allow migrating from the upper version to the lower version.

{"error":{"root_cause":[{"type":"snapshot_restore_exception","reason":"[test:cloud-snapshot-2021.06.01-ucssrgkdq9oyci-vbbfyfw/QYwmbA6TTJOXG2T83AtTTw] the snapshot was created with Elasticsearch version
 [7.7.1] which is higher than the version of this node [7.7.0]"}],"type":"snapshot_restore_exception","reason":"[test:cloud-snapshot-2021.06.01-ucssrgkdq9oyci-vbbfyfw/QYwmbA6TTJOXG2T83AtTTw] the snapsh
ot was created with Elasticsearch version [7.7.1] which is higher than the version of this node [7.7.0]"},"status":500}

At first, create an S3 bucket in your AWS account as a custom repository to store ES snapshots/backups.

You could find the steps to set up a custom repository on the Elastic Cloud at their site. Therefore, I am not going to describe them here.

Create an EC2 instance which will be used to execute commands during the data restoration. (Even t2.nano is sufficient).

Create two roles such as Role-S3, Role-ES and two policies such as Policy-S3 and Policy-ES and attach them to the roles respectively.

Policy-S3

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [ 
 "arn:aws:s3:::<es-snapshot-bucket-name>"
            ]
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Effect": "Allow",
            "Resource": [ 
  "arn:aws:s3::: <es-snapshot-bucket-name>/*”
            ]
        }
    ]
}

Policy-ES

{
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": "iam:PassRole",
                "Resource": "arn:aws:iam::<account_id>:role/Role-S3"  
            },
            {
                "Effect": "Allow",
                "Action": "es:ESHttpPut",
                "Resource": [
                         "arn:aws:es:<region>:<account_id>:domain/<es_domain_name>/*"
                  ]
            }
        ]
    }

Attach the role Role-ES to the EC2 instance and log in to the EC2.

Trigger a curl command to AWS ES endpoint to check if the cluster is reachable.

curl <es_endpoint>

Install the following python libs in the EC2.

$ pip3 install boto3
$ pip3 install requests
$ pip3 install requests_aws4auth
$ pip3 install --upgrade request

Copy and Paste the given python script, change the appropriate values and run the script.

This script will register the S3 bucket as a snapshot repository for the ES cluster that we created.

import boto3
import requests
from requests_aws4auth 
import AWS4Auth

host = <es_domain_endpoint/>    # Enter the ES domain endpoint and trailing ‘/’
region = <region>       
service = 'es'

credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

# Steps to Register snapshot repository
path = '_snapshot/<repository-name>'   # the ES API endpoint
url = host + path
payload = {
  "type": "s3",
  "settings": {
    "bucket": <es_snapshot_bucket_name>, 
    "region": <region>,  # Specify region for S3 bucket. If the S3 bucket is in the us-east-1 region use endpoint
    "endpoint": "s3.amazonaws.com", 
    "role_arn": "arn:aws:iam::<account_id>:role/Role-S3
  }
}
headers = {"Content-Type": "application/json"}
r = requests.put(url, auth=awsauth, json=payload, headers=headers)
print(r.status_code)
print(r.text)

The script would print output as shown below if it is executed successfully.

200
{"acknowledged":true}

List down the snapshots.

curl <es_endpoint>/_snapshot/<repository_name>/_all?pretty

Restore a snapshot

curl -XPOST <es_endpoint>/_snapshot/<repository-name>/<snapshot_name>/_restore

Restore a specific index from a snapshot with settings

curl -XPOST <es_endpoint>/_snapshot/<repository-name>/<snapshot_name>/_restore -d '{
"indices": "<index_name>",
"index_settings": {
  "index.routing.allocation.require.data": null
  }
}' -H 'Content-Type: application/json'

List down the indices.

curl -XGET <es_endpoint>/_cat/indices

You'll be able to view the indices, if the restoration is successful.

** Key Points **

Don't restore the .kibana or system indices. It could throw errors.
Don't delete .kibana index in target cluster when restoring.