DEV Community ๐Ÿ‘ฉโ€๐Ÿ’ป๐Ÿ‘จโ€๐Ÿ’ป

adithyasrinivasan
adithyasrinivasan

Posted on • Originally published at adithya.dev on

S3 Snapshots for Elasticsearch via Kibana

S3 Snapshots for Elasticsearch via Kibana

I'm terrified of racking up big bills with AWS so I don't work a lot with it, aside from Amazon Simple Email Service (SES). For my ElasticSearch 7.14 cluster that I run alongside with Kibana, I wanted to setup Snapshot and Restore.

My understanding was that with Snapshots, you obtain a copy of your cluster data at that point and you can restore it when you would like to. I already had an AWS account setup before starting with this. For this, you needed to setup a storage repository and then configure via Kibana.

This is how I went about it.

  • Create a S3 Bucket (this is the storage repository we will use with Elasticsearch)
  • Create an AWS policy to allow the IAM User to access the S3 Bucket
  • Create an IAM User and attach the policy we created
  • Install the S3 Repository Plugin
  • Register the repository with Kibana
  • Setup an automatic policy with Kibana / Elasticsearch so it backs up data via cron

Create a S3 Bucket

  • Open AWS Console -> S3 -> Create Bucket
  • Enter a bucket name: elasticsearch-s3-bucket-snapshot , other defaults are good - make sure you select " Block all public access" and then go ahead with creating the bucket (or saving)

S3 Snapshots for Elasticsearch via Kibana
Enter a bucket name: elasticsearch-s3-bucket-snapshot, other defaults are good - make sure you select "Block all public access"

Create an AWS policy

  • Open AWS Console -> IAM -> Policies
  • Click on "Create Policy"
  • Switch to JSON editor and paste the following policy (that Elasticsearch recommends for S3 permissions)
{
  "Statement": [
    {
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads",
        "s3:ListBucketVersions"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::snaps.example.com"
      ]
    },
    {
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::snaps.example.com/*"
      ]
    }
  ],
  "Version": "2012-10-17"
}

Enter fullscreen mode Exit fullscreen mode

S3 Snapshots for Elasticsearch via Kibana

  • Click on "Next: Tags", tags can be default so click next again
  • Enter a policy name elasticsearch-s3-policy-snapshot and hit " Create policy"

S3 Snapshots for Elasticsearch via Kibana
Enter a policy name elasticsearch-s3-policy-snapshot and hit "Create policy"

Create an IAM User

  • Open AWS Console -> IAM-> Access Management -> Users
  • Click on "Add Users"
  • Enter an user name elasticsearch-s3-user (customizable), check Programmatic access and click Next: Permissions

S3 Snapshots for Elasticsearch via Kibana
Enter an user name elasticsearch-s3-user (customizable), check Programmatic access and click Next: Permissions

  • Switch to " Attach existing policies directly" , search and select " elasticsearch-s3-policy-snapshot"

S3 Snapshots for Elasticsearch via Kibana

  • Click " Next: Tags", defaults are fine. Review the details and click Create User.

S3 Snapshots for Elasticsearch via Kibana
Review screen for Elasticsearch IAM user

  • This will generate a "Access Key ID" and "Secret access key" - both of which we will need soon to setup repository in Kibana, so keep them handy. There's a CSV option to download.

S3 Snapshots for Elasticsearch via Kibana
Credentials for IAM User

Install the S3 Repository Plugin

  • On your machine or server, go to where the Elasticsearch bin folder is. Mine was at /usr/share/elasticsearch
  • Run sudo bin/elasticsearch-plugin install repository-s3, answer with y or Y when asked about permissions
  • Restart your cluster!
  • We will configure the client settins for the plugin. Using elasticsearch-keystore, set the credentials for your IAM User
bin/elasticsearch-keystore add s3.client.default.access_key
# enter your "Access Key ID", hit enter
bin/elasticsearch-keystore add s3.client.default.access_key
# enter your "Secret Access Key", hit enter

Enter fullscreen mode Exit fullscreen mode

Register the repository with Kibana

  • Go to your Kibana dashboard -> Management -> Stack Management -> Snapshot & Restore -> Repositories -> " Register a repository"

S3 Snapshots for Elasticsearch via Kibana
Snapshot & Restore on Kibana

  • Name it as elasticsearch-repository-snapshot , select AWS S3 as repository type, click Next. If you don't see AWS S3, make sure you restarted your cluster!

S3 Snapshots for Elasticsearch via Kibana
Repository name & Type

  • Client name is default (this is part of your keystore secrets s3.client. default.access_key) and your S3 Bucket Name. Review other defaults and hit Register.

S3 Snapshots for Elasticsearch via Kibana
Repository client name & S3 Bucket name

  • Once this is done, it pops up a details screen for the repository. Click on Verify repository

S3 Snapshots for Elasticsearch via Kibana
Verify Repository to make sure everything is good

Setup an automatic policy with Kibana

  • On the same screen, switch to Policies -> Create Policy <!--kg-card-begin: markdown-->
  • Enter these details
    • elasticsearch-weekly-snapshot as the name
    • as the snapshot name, appends the date at the end
    • Select elasticsearch-repository-snapshot
    • Set frequency to every week
    • Click Next <!--kg-card-end: markdown-->

S3 Snapshots for Elasticsearch via Kibana
Details for Policy - Cron

  • Click Next, and on review screen, hit Create Policy

S3 Snapshots for Elasticsearch via Kibana
Review Policy

  • You can run the policy and when it's done, you will see the snapshot

S3 Snapshots for Elasticsearch via Kibana
Run now for the policy

  • You should see something like this once snapshot is complete

S3 Snapshots for Elasticsearch via Kibana
snapshot is complete

Top comments (0)

18 Useful Github Repositories Every Developer Should Bookmark

18 Useful GitHub repositories every developer should bookmark: everything from learning resources and roadmaps to best practices, system designs, and tools.