DEV Community: Hari Karthigasu

Boost Developer productivity and DBOps efficiency with AWS Aurora Cloning

Hari Karthigasu — Wed, 12 Nov 2025 10:39:41 +0000

Context

There are numerous situations where we need an independent and isolated DB. For example, developers want to test their code or the Ops team needs to run upgrade tests and etc. The common approach is restoring a snapshot. However, it can become expensive and inefficient over time for repetitive tasks.

AWS Aurora Clone

The "AWS Aurora Clone" becomes handy in this situation, where it provides a more efficient solution in terms of operation and cost.

The Aurora clone uses copy-on-write protocol and works in a way that it shares the same volume between the source and clone cluster(s), but the updates are only visible to the respective cluster.

I have shared, how the Aurora clone has been integrated into our GH actions to enable developers, spin up an independent and isolated Aurora RDS cluster, which contains the same data as the given (source) cluster to test their code on demand.

Solution

In my organization, we have dedicated AWS accounts for GH runners and applications.

Application accounts have two lambda functions.

Aurora Clone - Which is invoked by the GH action and performs the Aurora clone operation.
Aurora Clone Purge - Which is responsible for purging the cloned DB cluster, after a certain period of days where it is no longer needed; Invoked by an Event bridge scheduler.

Workflow

A developer creates a PR with a label clone-db
GH action invokes the lambda function that requires the name of the SOURCE_DB_CLUSTER and PR_NUMBER as inputs.

aws lambda invoke \
    --function-name arn:aws:lambda:$AWS_REGION:$AWS_ACCOUNT_ID:function:aurora_clone \
    --region $AWS_REGION \
    --cli-binary-format raw-in-base64-out \
    --payload '{ "SOURCE_DB_CLUSTER":"$SOURCE_DB_CLUSTER", "PR_NUMBER":"$PR_NUMBER" }' response.json

The response/response.json file contains the endpoint of the clone cluster.
The deployment manifest will be updated with the clone cluster's endpoint.

Even though aws lambda invoke is a synchronous call, it won't freeze the pipeline, as the Aurora cloning call is asynchronous.

HAPPY CLONING!

References

Aurora clone documentation
Source code of Lambdas
Tofu code to deploy the stack

Why S3 Intelligent-Tiering Should Be Your Default Storage Class for Large-Scale Buckets?

Hari Karthigasu — Wed, 27 Aug 2025 20:46:52 +0000

Context

Storing objects in S3 for an extended period increases both storage size and cost. Especially when you don't access them frequently. You can configure life cycle rules or implement a custom solution to move them between the storage classes, but it may come with drawbacks.

S3 Intelligent-Tiering

S3 Intelligent-Tiering moves an object to low-cost S3 storage based on its access frequency pattern, while preserving low latency and high throughput.

Frequent Access, Infrequent Access, and Archive Instant Access tiers are the three access tiers that S3 Intelligent-Tiering storage class automatically stores the objects.

Frequent Access tier is optimized for frequent access, Infrequent Access tier is a lower-cost tier optimized for infrequent access, and Archive Instant Access, which is a very low-cost tier optimized for rarely accessed data.

Your object(s) will float across these 3 tiers depending on your object access frequency patterns. They all provide low latency and high-throughput performance to your objects.

If an object in the Infrequent Access tier or Archive Instant Access tier is accessed later, it is automatically moved back to the Frequent Access tier.

The Archive Access tier and the Deep Archive Access tier are optional and operate only when activated if you want to get the lowest storage cost for data that can be accessed in minutes to hours.

If not activated, your objects will stay in the Archive Instant Access tier, which provides you with low latency and high-throughput performance to your objects.

Frequent Access Tier
         |
- Object has not been accessed for 30 days
         |
Infrequent Access Tier
         |
- Object has not been accessed for 90 days
         |
Archive Instant Access Tier

Activate S3 Intelligent-Tiering

When you create a new S3 bucket, straight away you can select the storage class as S3 Intelligent-Tiering. For an existing bucket, create a life cycle rule to move the objects to S3 Intelligent-Tiering.

Visualization

After activating S3 Intelligent-Tiering for an existing bucket,

The (image below shows) objects have been transferred to the S3 Intelligent-Tiering while the bucket is scaling.

Orange - Standard Rest - S3 Intelligent-Tiering

However, you may be surprised after seeing the huge spike in the following month's bill. It'll be double the amount from the previous month.

S3 Intelligent-Tiering was enabled in month 7, and the cost doubled in month 8.

WHY?

During the first month(s), the life cycle rule transfers the objects from the STANDARD to the S3 INTELLIGENT-TIERING class.

$0.01 per 1,000 transitions to Intelligent Tiering -> So, if you have 200 million objects, you'll pay~ $2000. In addition to that,
S3 Intelligent-Tiering monitors objects that are more than 128KB in size to transfer among the storage classes. $0.0025 per 1,000 Objects per month in Intelligent-Tiering

Nevertheless, it'll be hugely beneficial in the long run for large-scale S3 buckets, which have objects of unpredictable/fluctuating access patterns.
If you see the graph, the cost is slowly going down after the 8th month and by the 16th month, it becomes lower than 0th month.

AWS MSK IAM Authentication CLI commands

Hari Karthigasu — Sun, 10 Aug 2025 21:02:01 +0000

When you have a Kafka cluster in AWS MSK with IAM auth, there will be situations where you need to interact with its CLI to view the resources or for troubleshooting. During authentication, you should pass a properties file containing auth parameters.

This bash script will set up the Kafka CLI to connect to the MSK cluster.

#!/bin/bash

# variables
BROKER_ENDPOINT=$MSK_ENDPOINT
KAFKA_VERSION=3.8.1
BINARY_VERSION=2.13
IAM_AUTH_CLI_VERSION=2.13.1

# Download Kafka Binary
wget https://archive.apache.org/dist/kafka/$KAFKA_VERSION/kafka_$BINARY_VERSION-$KAFKA_VERSION.tgz
tar -zxvf kafka_$BINARY_VERSION-$KAFKA_VERSION.tgz
cd kafka_$BINARY_VERSION-$KAFKA_VERSION
cd libs/

# Download AWS MSK IAM CLI
wget https://github.com/aws/aws-msk-iam-auth/releases/download/v$BINARY_VERSION/aws-msk-iam-auth-$IAM_AUTH_CLI_VERSION-all.jar
cd ../bin/

# AWS IAM Auth file 
cat <<EOF> client.properties
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
EOF

Test

cd kafka_$IAM_AUTH_CLI_VERSION-$KAFKA_VERSION/bin
./kafka-topics.sh --bootstrap-server $BROKER_ENDPOINT --command-config client.properties --list

Set custom configuration in AWS EKS CoreDNS Addon

Hari Karthigasu — Fri, 18 Jul 2025 21:32:40 +0000

When you enable managed addons in EKS, they come with predefined configurations. Nevertheless, there are situations where we have to override them. This gist shows how to set custom configuration for the CoreDNS addon using terraform-aws-modules/terraform-aws-eks and via the AWS console.

...
addons = {
    coredns = {
      addon_version = "v1.11.4-eksbuild.2"
      most_recent   = true
      configuration_values = <<EOT
      {
        "corefile": ".:53 {\n  errors\n  health {\n    lameduck 5s\n  }\n  ready\n  kubernetes cluster.local in-addr.arpa ip6.arpa {\n    pods insecure\n    fallthrough in-addr.arpa ip6.arpa\n  }\n  prometheus :9153\n  forward . /etc/resolv.conf\n  cache 30\n  loop\n  reload\n  loadbalance\n}",
        "autoScaling": {
          "enabled": true,
          "minReplicas": 4,
          "maxReplicas": 8
        },
        "tolerations": [
          {
            "key": "AppsOnly",
            "effect": "NoSchedule",
            "operator": "Equal",
            "value": "apps"
          },
          {
            "key": "CriticalAddonsOnly",
            "effect": "NoSchedule",
            "operator": "Exists"
          }
        ]
      }
      EOT
    }
  }
...

module "eks" {
  source = "terraform-aws-modules/terraform-aws-eks"
  ...
  cluster_addons = var.addons
  ...
}

OpsGist - Tried‑and‑worked snippets and insights I’ve come across.

AWS Cross-Account Read-Only RDS access via Private Link

Hari Karthigasu — Sun, 06 Jul 2025 20:21:33 +0000

CONTEXT

Cross-account resource sharing is one of the critical operations in AWS. I have elaborated on the solution to grant READ-ONLY RDS database access to an external AWS account.

SOLUTION

DESIGN RATIONALE

1.. VPC Endpoint Service with Private Link

AWS VPC Endpoint Service, powered by PrivateLink, enables secure and effortless connectivity between two VPCs with fine-grained access controls. Unlike VPC peering or Transit Gateway (TGW) integration, which provides broader network access, PrivateLink ensures a more restricted and secure connection.

Check out my article AWS VPC endpoint services for NLB powered by Private Link.

2.. RDS Proxy

RDS Proxy Read-Only endpoint

The most important requirement is to grant read-only access to the RDS. This can be achieved by creating a database user with SELECT-only privileges at the database level. However, human error could modify these permissions. The RDS Proxy read-only endpoint enforces read-only operations for clients by routing traffic exclusively to read replica instances in the backend. regardless of the user’s database-level permissions, providing an extra layer of protection.

Static IPs

An RDS Proxy endpoint maintains a static IP address throughout its lifecycle. This allows you to create a target group behind a Network Load Balancer (NLB) using the RDS proxy’s read-only endpoint IPs, enabling consistent and reliable connectivity.

3.. A Secrets Manager encrypted by a CMK

RDS Proxy requires database credentials to connect to the database. These credentials are stored in AWS Secrets Manager and encrypted using a CMK, as they need to be shared with an external account for access.

4.. Lambda function

To enhance security, the database credentials should be rotated on a regular schedule. A Lambda function handles the rotation by updating the credentials both in AWS Secrets Manager and the database.

Once the VPC Endpoint Service is established between two accounts or VPCs, the external account can connect to our database using one of the endpoints provided by the service.

CHALLENGES

Even though IAM authentication is enabled on the RDS cluster, the RDS Proxy (or any client connecting through it) still requires database credentials to establish a connection.
As a standard practice in my organization, we use SSM Parameter Store to manage and store secrets. In this case, using AWS Secrets Manager added an additional layer. To align with our standards, I tried to implement referencing AWS Secrets Manager secrets from Parameter Store parameters. However, the integration was unsuccessful due to a bug, and I have raised a ticket with AWS Support. Currently, there is no ETA for resolution.

If this approach had been successful, we could have created Advanced Tier Parameter Store entries that reference Secrets Manager, making it easy to share them with the external account. This would have reduced direct calls to Secrets Manager from the external application.

After a successful implementation in the development environment, the same setup encountered issues in production. A user was unable to execute queries when connecting to the RDS via RDS Proxy. The issue was resolved after recreating the RDS Proxy. Contacted AWS Support, although the root cause remains unclear.

AWS SSM Association - Schedule Stop and Start RDS

Hari Karthigasu — Sat, 14 Jun 2025 18:49:57 +0000

CONTEXT

AWS RDS is an essential and high-cost service. Improving its cost efficiency will help control an AWS account's overall expenses. For non-production environments, it is advisable to shut down RDS databases outside of working hours to reduce the unnecessary costs they incur.

Usually, we utilize an event bridge scheduler to start and stop an RDS service via a Lambda function. This post shows the step-by-step Terraform code that elaborates on implementing this solution using AWS Systems Manager (SSM) Association.

SOLUTION

data "aws_iam_policy_document" "iam_ssm_policy_stop_aurora_cluster" {
  count = var.environment == "prod" ? 0 : 1

  statement {
    sid    = "StopAuroraCluster"
    effect = "Allow"
    actions = [
      "rds:StopDBCluster",
      "rds:StartDBCluster"
    ]
    resources = ["arn:aws:rds:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:cluster:${var.rds_cluster_name}]
  }

  statement {
    sid    = "DescribeAuroraClusters"
    effect = "Allow"
    actions = [
      "rds:DescribeDBClusters"
    ]
    resources = ["*"]
  }
}

module "iam_ssm_policy_stop_aurora_cluster" {
  count   = var.environment == "prod" ? 0 : 1
  source  = "terraform-aws-modules/iam/aws//modules/iam-policy"

  name        = "rds-start-stop-aurora-cluster-policy"
  path        = "/"
  description = "IAM Policy to allow SSM to start and stop cluster."

  policy = data.aws_iam_policy_document.iam_ssm_policy_stop_aurora_cluster[0].json
}

module "iam_assumable_role_stop_aurora_cluster" {
  count   = var.environment == "prod" ? 0 : 1
  source  = "terraform-aws-modules/iam/aws//modules/iam-assumable-role"

  create_role             = true
  create_instance_profile = false

  role_name              = "start-stop-aurora-cluster-role"
  role_requires_mfa      = false
  allow_self_assume_role = false

  trusted_role_services = [
    "ssm.amazonaws.com"
  ]

  custom_role_policy_arns = concat(
    [
      module.iam_ssm_policy_stop_aurora_cluster[0].arn,
    ]
  )
}

resource "aws_ssm_association" "ssm_stop_aurora_cluster_association" {
  count = var.environment == "prod" ? 0 : 1

  name                = "AWS-StartStopAuroraCluster"
  association_name    = "stop-aurora-cluster"
  schedule_expression = "cron(0 18 * * ? *)"

  parameters = {
    ClusterName          = "${var.rds_cluster_name}"
    AutomationAssumeRole = module.iam_assumable_role_stop_aurora_cluster[0].iam_role_arn
    Action               = "Stop"
  }
}

resource "aws_ssm_association" "ssm_start_aurora_cluster_association" {
  count = var.environment == "prod" ? 0 : 1

  name                = "AWS-StartStopAuroraCluster"
  association_name    = "start-aurora-cluster"
  schedule_expression = "cron(0 8 * * ? *)"

  parameters = {
    ClusterName          = "${var.rds_cluster_name}"
    AutomationAssumeRole = module.iam_assumable_role_stop_aurora_cluster[0].iam_role_arn
    Action               = "Start"
  }
}

AWS VPC endpoint services for NLB powered by Private Link.

Hari Karthigasu — Tue, 07 Jan 2025 23:02:42 +0000

Recently in my organization, there was a requirement to connect to a private endpoint in Account A from Account B. When such a requirement comes, VPC peering is the first solution that comes to our mind. However, if the given endpoint is hosted behind an NLB, it can simply connected via a VPC endpoint service which is powered by AWS Private Link.

In Account A, create an NLB and service endpoints respectively.

resource "aws_vpc_endpoint_service" "this" {
  # The ARN of the NLB
  network_load_balancer_arns = [module.nlb.arn]

  # DNS of the private endpoint
  private_dns_name    = var.private_dns_name

  # Accept or Reject endpoint connections from other AWS accounts
  acceptance_required = true

  tags = {
    Name = "${terraform.workspace}-nlb"
  }
}

resource "aws_vpc_endpoint_service_allowed_principal" "this" {
  vpc_endpoint_service_id = aws_vpc_endpoint_service.this.id

  # Allow principal to create endpoint connection
  principal_arn = "arn:aws:iam::${var.account_b_id}:root"
}

The Service name is required when we configure the VPC endpoint in Account B.

Add the TXT record to your Domain. After a successful validation the Domain verification status will be shown as Verified.

In Account B, create a VPC endpoint for the VPC endpoint service created above.

module "vpc_endpoints" {
  source  = "terraform-aws-modules/vpc/aws//modules/vpc-endpoints"
  ...
  endpoints = {
    "nlb" = {
      service_name        = "com.amazonaws.vpce.eu-north-1.vpce-svc-0f61ad0e435a4680c"
      subnet_ids          = module.vpc.private_subnets
      private_dns_enabled = true
      service_type        = "Interface"
      tags                = { Name = "${terraform.workspace}-nlb" }
    }
  }
}

Go back to Account A and accept the endpoint connection request that comes from Account B, under the Endpoint connections tab in Endpoint services.

Now try to access the private endpoint hosted in Account A from Account B.

$ curl nlb.petproject.my
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Thank you for reading!

Understanding self-assumption and scoped-down policy in AWS IAM

Hari Karthigasu — Mon, 08 Jul 2024 21:13:55 +0000

AWS IAM is a fundamental service for all 200+ AWS services, as it enables interaction with AWS principals. An AWS IAM role consists of two components: a policy and a trust relationship. The trust relationship handles authentication, while the policy is for authorization.

The trust relationship has rules specifying which AWS principals are allowed to assume the role. What is assume the role? In a nutshell, entities can use AWS STS to assume the role by calling the aws sts assume-role. If an entity is able to assume the role, it can execute the actions specified in the attached policy. Therefore it's important to follow best practices and choose suitable patterns when implementing IAM.

Self-assumption

Have you ever encountered a scenario where an IAM role assumes itself? It may sound awkward, yet it's real. An IAM role needs to be explicitly allowed to assume itself as it doesn't have self-assumption capabilities by default. It is to improve consistency and visibility of a role's privileges.

To elaborate, I have an IAM role GHAction-Role with AssumeRoleWithWebIdentity to authenticate GithubActions in AWS and a github action respectively.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "GithubOidcAuth",
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:AssumeRoleWithWebIdentity"
            ],
            "Condition": {
                "StringLike": {
                    "token.actions.githubusercontent.com:sub": "repo:harik8/services:*"
                }
            }
        }
    ]
}

  STS:
    runs-on: ubuntu-latest
    needs: [CI]
    steps:
    - name: Git clone the repository
      uses: actions/checkout@v4

    - name: configure aws credentials
      uses: aws-actions/configure-aws-credentials@v4
      with:
        role-to-assume: ${{ vars.IAM_ROLE_ARN }} # ARN of the GHAction-Role
        aws-region: ${{ vars.AWS_REGION }}

Output of above action.

Run aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume: arn:aws:iam::123456789012:role/GHAction-Role
    aws-region: eu-north-1
    audience: sts.amazonaws.com
Assuming role with OIDC
Authenticated as assumedRoleId AROASIGA2HTHJOXZFKTPL:GitHubActions

The GitHub workflow is able to assume the role using WebIdentity. However, if the GitHub workflow tries to perform sts:AssumeRole against the GHAction-Role, it will encounter an issue.

aws sts assume-role --role-arn arn:aws:iam::123456789012:role/GHAction-Role --role-session-name GitHubActions

An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::12345678912:assumed-role/GHAction-Role/GitHubActions is not authorized to perform: sts:AssumeRole on resource: arn:aws:sts::12345678912:assumed-role/GHAction-Role/GitHubActions

The trust policy of the GHAction-Role doesn't currently allow it to assume the role. To resolve this, the GHAction-Role needs to be able to assume itself. Therefore, the GHAction-Role's ARN arn:aws:iam::123456789012:role/GHAction-Role should be added to the trust policy to permit this action as shown below.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "GithubOidcAuth",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789012:role/GHAction-Role",
                "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:AssumeRoleWithWebIdentity"
            ],
            "Condition": {
                "StringLike": {
                    "token.actions.githubusercontent.com:sub": "repo:harik8/services:*"
                }
            }
        }
    ]
}

Scoped down policy

If a given GitHub Action runs more than one job and requires different permissions for each, using a single GHAction-Role with maximum permissions is not a good design practice as it violates IAM's principle of least privilege. This is where scoped-down policies come into play.

A scoped-down policy refers to a policy that grants the minimum set of permissions required for a user, group, or role to perform their necessary tasks.

Instead of having one generic role, GHAction-Role, with all required policies attached, we should create a specific role for each job with the necessary least privileges. For example, we would have roles like GHAction-Role-S3, GHAction-Role-EC2, and GHAction-Role-EKS. These roles would be assumed by GHAction-Role.

So, the trust policy for the above roles will look like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "GHAction-S3",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789012:role/GHAction-Role"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Even though self-assumption is suitable for certain use cases. Scoped-down policies generally provide more secure, control and manageable permissions.

HAPPY ASSUMING!

Share Securely

Hari Karthigasu — Tue, 25 Oct 2022 18:34:19 +0000

Securely sharing confidential information between team members is one of the critical tasks we have to perform during our day-to-day life.

There are platforms we can use to share passwords or sensitive data. Such as onetimesecret.com, scrt.link and etc. Primarily they provide a one-time link to access your secret. The link will be disappeared once you access it.

In this article, I’ll be illustrating how we can implement a similar application via the AWS Serverless ecosystem.

As shown above in the diagram, the web application has been hosted in AWS Amplify. It allows users to store and read their secrets.

The web application is backed by two lambda functions. They manage the DB operations. (For demo purpose I have used lambda functionalURL.)

The data will be stored in DynamoDB.

Add a Secret
Adding a secret has three steps,

Enter the Message.
Enter a secret key to protect your message.
Select the expiration time for the secret.

After the submission, the web application will invoke a Lambda functional URL to insert the data into DynamoDB.

Read a Secret

Access the shared link
Enter the provided Secret Key
Your secret will be displayed.

After a successful retrieval of a secret. The secret will be deleted from the database immediately.

Workflow

DynamoDB has 4 attributes. SecretID (PK) and ExpirationTime (SK), Message, _and SecretKey_.

The TTL has been enabled on the ExpirationTime attribute. DyanmoDB deletes the record once it reaches the TTL value. This operation doesn’t consume a write capacity. However, DynamoDB TTL is not real-time. It’d take 24H-48H to remove a record from the DB. The lambda that reads the data has a logic to validate whether the requested secret is expired or not.

Demo

URL : https://secretshare.forexample.link

https://www.youtube.com/watch?v=q4W8R18ItzI

Source
https://github.com/harik8/temp-secret-share

Discount Portal - Send Discount Newsletters

Hari Karthigasu — Mon, 10 Jan 2022 19:51:35 +0000

Overview of My Submission

Discount Portal is Web based application that sends out discount newsletters to a subscriber or list of subscribers.

Submission Category:

Choose Your Own Adventure

Link to Code

harik8 / discount-portal

Discount Portal is a Web application send product discount newsletters to subscribers.

Discount Portal

Discount Portal is a Web application that sends product discount newsletters to subscribers. Click here to access the portal.

This is application is developed for Mongo Atlas Hackathon 2021/2022.

(The portal requires Google Auth to login. Currently it's only allowed to my user.)

Click here to view the dev.to page about this application.

Folder Structure

assets - Screenshots and images.
backend - Backend service (server.py) which add records to Mongo Atlas Database via Data API.
functions - AWS Lambda function to send mails and Realm scheduler function to clean DB.
templates - AWS SES Email template.

Technologies Used

React JS - Front end
Python - Back end
Mongo Atlas Services

Service Flow Chart

View on GitHub

Additional Resources / Info

React JS - Frontend
Python - Backend

Mongo Atlas
Realm Functions
Triggers
Data API
Diagrams and Image

Flow Chart Diagram

(Please click on the image to view clearly or view on Github).

Journey from Elastic Cloud to AWS Elastic Search

Hari Karthigasu — Tue, 26 Oct 2021 12:19:50 +0000

This article demonstrates the steps that we should follow when we migrate Elastic search data from Elastic cloud to AWS.

Let's get started.

The Elastic Cloud supports Major.Minor.Patch (eg: x.y.z) version of Elastic Search, where AWS supports only Major. Minor (eg: x.y).

When you build the target cluster on AWS, ensure you created the cluster using the next latest Minor version of the source cluster. (If the source cluster version is 7.7.1, the target cluster version must be 7.8).

Elastic Search doesn’t allow migrating from the upper version to the lower version.

{"error":{"root_cause":[{"type":"snapshot_restore_exception","reason":"[test:cloud-snapshot-2021.06.01-ucssrgkdq9oyci-vbbfyfw/QYwmbA6TTJOXG2T83AtTTw] the snapshot was created with Elasticsearch version
 [7.7.1] which is higher than the version of this node [7.7.0]"}],"type":"snapshot_restore_exception","reason":"[test:cloud-snapshot-2021.06.01-ucssrgkdq9oyci-vbbfyfw/QYwmbA6TTJOXG2T83AtTTw] the snapsh
ot was created with Elasticsearch version [7.7.1] which is higher than the version of this node [7.7.0]"},"status":500}

At first, create an S3 bucket in your AWS account as a custom repository to store ES snapshots/backups.

You could find the steps to set up a custom repository on the Elastic Cloud at their site. Therefore, I am not going to describe them here.

Create an EC2 instance which will be used to execute commands during the data restoration. (Even t2.nano is sufficient).

Create two roles such as Role-S3, Role-ES and two policies such as Policy-S3 and Policy-ES and attach them to the roles respectively.

Policy-S3

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [ 
 "arn:aws:s3:::<es-snapshot-bucket-name>"
            ]
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Effect": "Allow",
            "Resource": [ 
  "arn:aws:s3::: <es-snapshot-bucket-name>/*”
            ]
        }
    ]
}

Policy-ES

{
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": "iam:PassRole",
                "Resource": "arn:aws:iam::<account_id>:role/Role-S3"  
            },
            {
                "Effect": "Allow",
                "Action": "es:ESHttpPut",
                "Resource": [
                         "arn:aws:es:<region>:<account_id>:domain/<es_domain_name>/*"
                  ]
            }
        ]
    }

Attach the role Role-ES to the EC2 instance and log in to the EC2.

Trigger a curl command to AWS ES endpoint to check if the cluster is reachable.

curl <es_endpoint>

Install the following python libs in the EC2.

$ pip3 install boto3
$ pip3 install requests
$ pip3 install requests_aws4auth
$ pip3 install --upgrade request

Copy and Paste the given python script, change the appropriate values and run the script.

This script will register the S3 bucket as a snapshot repository for the ES cluster that we created.

import boto3
import requests
from requests_aws4auth 
import AWS4Auth

host = <es_domain_endpoint/>    # Enter the ES domain endpoint and trailing ‘/’
region = <region>       
service = 'es'

credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

# Steps to Register snapshot repository
path = '_snapshot/<repository-name>'   # the ES API endpoint
url = host + path
payload = {
  "type": "s3",
  "settings": {
    "bucket": <es_snapshot_bucket_name>, 
    "region": <region>,  # Specify region for S3 bucket. If the S3 bucket is in the us-east-1 region use endpoint
    "endpoint": "s3.amazonaws.com", 
    "role_arn": "arn:aws:iam::<account_id>:role/Role-S3
  }
}
headers = {"Content-Type": "application/json"}
r = requests.put(url, auth=awsauth, json=payload, headers=headers)
print(r.status_code)
print(r.text)

The script would print output as shown below if it is executed successfully.

200
{"acknowledged":true}

List down the snapshots.

curl <es_endpoint>/_snapshot/<repository_name>/_all?pretty

Restore a snapshot

curl -XPOST <es_endpoint>/_snapshot/<repository-name>/<snapshot_name>/_restore

Restore a specific index from a snapshot with settings

curl -XPOST <es_endpoint>/_snapshot/<repository-name>/<snapshot_name>/_restore -d '{
"indices": "<index_name>",
"index_settings": {
  "index.routing.allocation.require.data": null
  }
}' -H 'Content-Type: application/json'

List down the indices.

curl -XGET <es_endpoint>/_cat/indices

You'll be able to view the indices, if the restoration is successful.

Key Points

Don't restore the .kibana or system indices. It could throw errors.
Don't delete .kibana index in target cluster when restoring.

DEV Community: Hari Karthigasu

Boost Developer productivity and DBOps efficiency with AWS Aurora Cloning

Context

AWS Aurora Clone

Solution

Workflow

References

Why S3 Intelligent-Tiering Should Be Your Default Storage Class for Large-Scale Buckets?

Context

S3 Intelligent-Tiering

Activate S3 Intelligent-Tiering

Visualization

AWS MSK IAM Authentication CLI commands

Set custom configuration in AWS EKS CoreDNS Addon

AWS Cross-Account Read-Only RDS access via Private Link

CONTEXT

SOLUTION

DESIGN RATIONALE

CHALLENGES

AWS SSM Association - Schedule Stop and Start RDS

AWS VPC endpoint services for NLB powered by Private Link.

Understanding self-assumption and scoped-down policy in AWS IAM

Share Securely

Discount Portal - Send Discount Newsletters

Overview of My Submission

Submission Category:

Link to Code

harik8 / discount-portal

Discount Portal is a Web application send product discount newsletters to subscribers.

Discount Portal

Folder Structure

Technologies Used

Service Flow Chart

Additional Resources / Info

Flow Chart Diagram

Journey from Elastic Cloud to AWS Elastic Search

Policy-S3

Policy-ES

** Key Points **

Key Points