Creating and restoring manual AWS Elasticsearch snapshots in Ruby

AWS makes regular automated snapshots of your hosted Elasticsearch services. But if you for whatever reason wanna create a manual snapshot, you can only do this scripted. At the time of writing this there's no way of doing this from the AWS console.

This Ruby script will help you creating manual snapshots of an AWS hosted Elasticsearch service using signed requests.

I had to create a manual snapshot of my ES service to migrate to a new VPC. You cannot change the VPC for an existing service, so I had no other option than creating a new ES service, manually snapshotting my old service and restoring this snapshot on the new service. And btw this is the documented way to do it by AWS.

Before you begin, start by reading the guide from AWS to familiarize yourself with the process:
Creating Amazon ES Index Snapshots

These are the steps you need to take to prepare yourself to create the manual snapshots and restoring them.

• Start by creating an AWS S3 bucket for your snapshots.
• Next go to the AWS IAM console and create a new policy. The content of the policy is also listed in the above link:

{
  "Version": "2012-10-17",
  "Statement": [{
      "Action": [
        "s3:ListBucket"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::s3-bucket-name"
      ]
    },
    {
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::s3-bucket-name/*"
      ]
    }
  ]
}

Now create an AWS IAM Role - this will be used in the script to permit the snapshot.

• The console will ask for a use case. Select DMS for now - we'll change this later as Elasticsearch is not listed.
• Click next to set the role permissions. Here you can search for the policy you just created.
• Name your role something meaningful and create it. You'll have to reference it in the script.
• Once created find your role and open it in the console.
• Click on the Trust Relationsships tab and click Edit trust relationship
• Replace the existing policy with the following:

{
 "Version": "2012-10-17",
 "Statement": [{
   "Sid": "",
   "Effect": "Allow",
   "Principal": {
     "Service": "es.amazonaws.com"
   },
   "Action": "sts:AssumeRole"
 }]
}

The script

Now we are ready to configure the script.
I'm using the aws-sdk-elasticsearchservice gem - also listed in the script. Start by installing it:

gem install aws-sdk-elasticsearchservice

You need to do some small configuration in the script. The AwsEsSnapshot class defines a number of constants that you should replace with your own values:

• AWS bucket name and region (notice you can only migrate Elasticsearch to a new service in same region)
• Role arn for your freshly created role. It's listed in the top of the summary screen in the AWS console.
• Finally you'll have to name a repository and snapshot. For a one time migration these values are not that important, but if you wanna continue to create manual snapshots you might wanna set some meaningful names.

Finally I'm fetching access key id and secret from environment variables. You can run your script like this:
AWS_ACCESS_KEY_ID=xxx AWS_SECRET_ACCESS_KEY=xxx ruby miration.rb

gem 'aws-sdk-elasticsearchservice'
require 'aws-sdk-elasticsearchservice'

class AwsEsSnapshot
  attr_accessor :logger
  REPO = 'my-repository-name'
  SNAPSHOT_NAME = 'my-snapshot-name'
  ROLE_ARN = 'arn:aws:iam::123456789012:role/TheSnapshotRole'
  BUCKET = 's3-bucket-name'
  REGION = 'eu-west-1'
  ACCESS_KEY_ID = ENV['AWS_ACCESS_KEY_ID']
  SECRET_ACCESS_KEY = ENV['AWS_SECRET_ACCESS_KEY']

  def initialize(logger: Logger.new(STDOUT))
    @logger = logger
    @logger.info('AwsEsSnapshot') { <<~HEREDOC
        Initialized with:
        Repository: #{REPO}
        Snapshot name: #{SNAPSHOT_NAME}
        Role ARN: #{ROLE_ARN}
        S3 bucket: #{BUCKET}
        AWS Region: #{REGION}
      HEREDOC
    }
  end

  # Creating a signed request
  def signer
    @signer ||= Aws::Sigv4::Signer.new(
      service: 'es',
      region: REGION,
      access_key_id: ACCESS_KEY_ID,
      secret_access_key: SECRET_ACCESS_KEY
    )
  end

  def signature(method:, url:, payload: nil)
    signer.sign_request(
      http_method: method.to_s.upcase,
      url: url,
      body: payload ? payload.to_json : nil
    )
  end

  def request(method: :put, host: nil, path:, payload: nil)
    url = host + '/' + path
    signature = signature(method: method, url: url, payload: payload)
    uri = URI(url)

    logger.debug("AwsEsSnapshot#request") { "#{method.to_s.upcase}ing #{payload} to #{url}"}

    Net::HTTP.start(uri.host, uri.port, :use_ssl => true) do |http|
      request = send(method, uri)
      request.body = payload.to_json if payload
      request['Host'] = signature.headers['host']
      request['X-Amz-Date'] = signature.headers['x-amz-date']
      request['X-Amz-Security-Token'] = signature.headers['x-amz-security-token']
      request['X-Amz-Content-Sha256']= signature.headers['x-amz-content-sha256']
      request['Authorization'] = signature.headers['authorization']
      request['Content-Type'] = 'application/json'
      response = http.request request
      if response.code.to_i == 200
        logger.debug("AwsEsSnapshot#request") { response.body }
      else
        logger.fatal("AwsEsSnapshot#request") { response.body }
        raise "Received unexpected status code #{response.code} in response."
      end
      response
    end
  end

  def post(uri)
    Net::HTTP::Post.new uri
  end

  def put(uri)
    Net::HTTP::Put.new uri
  end

  def get(uri)
    Net::HTTP::Get.new uri
  end

  def register(host:)
    payload = {
      type: 's3',
      settings: {
        bucket: BUCKET,
        region: REGION,
        role_arn: ROLE_ARN
      }
    }

    request host: host, path: "_snapshot/#{REPO}", payload: payload
    logger.debug("AwsEsSnapshot#request") { "Registered S3 bucket on #{host}" }
  end

  def create_snapshot(host:)
    # First we'll ensure our S3 bucket is registered
    register(host: host)

    request host: host, path: "_snapshot/#{REPO}/#{SNAPSHOT_NAME}"
    logger.debug("AwsEsSnapshot#request") { "Completed snapshot for #{SNAPSHOT_NAME}" }
  end

  def restore(host:)
    # First we'll ensure our S3 bucket is registered
    register(host: host)

    payload = {
      indices: "-.kibana*,-.opendistro_security",
      include_global_state: false,
      ignore_unavailable: true,
    }

    request method: :post, host: host, path: "_snapshot/#{REPO}/#{SNAPSHOT_NAME}/_restore", payload: payload
    logger.debug("AwsEsSnapshot#request") { "Restored snapshot for #{SNAPSHOT_NAME}" }
  end

  def status(host:)
    request method: :get, host: host, path: "_snapshot/_status"
  end
end

Now you are ready to use the script.
By default it's logging to STDOUT, but you can initialize with a custom logger:

es = AwsEsMigration.new(logger: MyCustomLogger.new)

There are tree methods of interest.
AwsEsMigration#create_snapshot
Expects a host: argument. This is your existing Elasticsearch service endpoint URL from where you wanna create a snapshot.

es.create_snapshot(host: 'https://my-legacy-endpoint-url.eu-west-1.es.amazonaws.com')

AwsEsMigration#status
will tell you the current snapshot progress. It's my experience the snapshot is completed fairly quickly.
Like litereally in minutes for a 10gb ES storage.

puts es.status(host: 'https://my-legacy-endpoint-url.eu-west-1.es.amazonaws.com').body

AwsEsMigration#restore
Finally if you like me need to restore a snapshot:

es.restore(host: 'https://my-new-endpoint-url.eu-west-1.es.amazonaws.com')

I hope this will help somebody out there :)