15-Minute RPO to 7-Year Retention: Elasticsearch Backups on S3 with SLM (No Scripts, No Keys)

• Four Elasticsearch SLM policies (15-min, hourly, monthly, yearly) write to four repositories in one S3 bucket.
• Nodes authenticate with an EC2 instance profile. Zero access keys in the keystore.
• SLM handles retention, from 7 days up to ~7 years. S3 lifecycle rules stay off, on purpose.
• The whole thing is about 30 lines of JSON and a handful of AWS CLI commands.

Three questions every backup should answer

Elasticsearch cluster runs on EC2 in a regulated environment. I wanted backups that answer three questions in one sentence each: how much data can we lose, how far back can we go, and what credentials does the job use?

Elasticsearch already ships a scheduler for this: Snapshot Lifecycle Management (SLM). So I moved everything into SLM and let S3 be dumb, durable storage.

The architecture

Read it left to right. Nodes get temporary credentials from an IAM role. SLM fires snapshots on cron schedules. The S3 repository writes objects into an SSE-S3 encrypted bucket, one folder per tier. SLM comes back around to delete expired snapshots, and restores go the other way.

Tier	Runs	Keeps	What's in it
15min/	every 15 min	7 days	critical app indices only
daily/	hourly	35 days	everything + cluster state
monthly/	1st of month	~13 months	everything + cluster state
yearly/	Jan 1	~7 years	everything + cluster state

The daily/ folder holds the hourly snapshots. It's the day-to-day recovery tier.

Step 1: IAM, the keyless way

One role, one instance profile, attached to every node: data, master and Kibana. The permissions policy needs bucket-level and object-level actions in separate statements:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads",
        "s3:ListBucketVersions"
      ],
      "Resource": "arn:aws:s3:::my-es-snapshots"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Resource": "arn:aws:s3:::my-es-snapshots/*"
    }
  ]
}

Create the role with an EC2 trust policy, then wire it to each node:

aws iam create-role --role-name es-snapshot-role \
  --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"ec2.amazonaws.com"},"Action":"sts:AssumeRole"}]}'

aws iam put-role-policy --role-name es-snapshot-role \
  --policy-name es-snapshot-s3 --policy-document file://s3-policy.json

aws iam create-instance-profile --instance-profile-name es-snapshot-profile
aws iam add-role-to-instance-profile \
  --instance-profile-name es-snapshot-profile --role-name es-snapshot-role

# Repeat for every data, master and Kibana node
aws ec2 associate-iam-instance-profile --instance-id i-0123456789abcdef0 \
  --iam-instance-profile Name=es-snapshot-profile

If a node already has an instance profile, add the S3 policy to that role instead. An instance can only have one profile.

Then leave the Elasticsearch keystore alone. With no s3.client.default.access_key configured, the S3 client uses the instance profile automatically.

Step 2: Repositories

On 7.x, run this on each node and do a rolling restart. On 8.x, S3 support is built in, so skip it.

sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install repository-s3

Then register one repository per tier:

PUT _snapshot/es-backup-15min
{
  "type": "s3",
  "settings": {
    "bucket": "my-es-snapshots",
    "base_path": "15min",
    "server_side_encryption": true
  }
}

Do the same for daily, monthly and yearly.

Step 3: SLM policies

The 15-minute policy covers only the critical indices:

PUT _slm/policy/snap-15min
{
  "schedule": "0 5/15 * * * ?",
  "name": "<snap-15min-{now/d}>",
  "repository": "es-backup-15min",
  "config": {
    "indices": ["app-critical-*"],
    "ignore_unavailable": true,
    "include_global_state": false
  },
  "retention": { "expire_after": "7d", "min_count": 10 }
}

The hourly policy snapshots everything:

PUT _slm/policy/snap-hourly
{
  "schedule": "0 0 * * * ?",
  "name": "<snap-hourly-{now/d}>",
  "repository": "es-backup-daily",
  "config": { "indices": ["*"], "include_global_state": true },
  "retention": { "expire_after": "35d", "min_count": 24 }
}

The monthly and yearly policies are the same shape, with schedules like 0 0 2 1 * ? and 0 0 3 1 1 ?, and expire_after values of 396d and 2575d. SLM appends a unique suffix to every snapshot name, so the date-based names never collide.

Kick the tires:

POST _slm/policy/snap-15min/_execute
GET _slm/policy/snap-15min?human
GET _slm/stats
GET _cat/snapshots/es-backup-15min?v&s=end_epoch:desc

The policy response shows last_success, last_failure and next_execution. Those are the fields to alert on.

Five things that bit me (so they don't bite you)

1. "Cannot create snapshot repository" on a healthy cluster.
Repository registration is verified from every master and data node. One node without the instance profile fails the whole request. Check every node, not just the one you're curling.

2. Access denied with a policy that looked right.
I had object permissions on bucket/* but forgot the bucket ARN itself. ListBucket and GetBucketLocation live at the bucket level.

3. I wanted a 5-minute RPO.
SLM refuses schedules under 15 minutes by default (slm.minimum_interval). The fix was scoping: a 15-minute policy for only the critical indices, and hourly for everything else. If you need tighter than 15 minutes, you need replication, not snapshots.

4. I almost added an S3 lifecycle rule.
Don't. Snapshots are incremental and share files, so expiring objects by age can corrupt the repository. SLM retention knows which files are still referenced. If cleanup seems slow, remember it runs once a day by default; force it with POST _slm/_execute_retention.

5. Schedules colliding at the top of the hour.
Starting the 15-minute policy at minute 5 (0 5/15 * * * ?) keeps it out of the hourly run's way.

Bonus check: the 15-minute tier keeps about 672 snapshots (96 a day for 7 days). Make sure your version's repository snapshot limit can hold that.

Restoring (the part that actually matters)

Restore to new names first, compare, then swap:

POST _snapshot/es-backup-daily/<snapshot-name>/_restore
{
  "indices": "app-critical-orders",
  "rename_pattern": "(.+)",
  "rename_replacement": "restored-$1",
  "include_global_state": false
}

For a full-cluster restore, close or delete the target indices first, then restore without the rename.

What I'd add next

Alerting on last_failure and on the failure counts in _slm/stats, plus a monthly restore drill into a scratch cluster. A backup you haven't restored is a hypothesis.

If you run self-managed Elasticsearch on EC2, I'd love to hear how you handle long-term retention. Drop it in the comments.