DEV Community: Hai Nguyen

How to automatically create S3 lifecycles using AWS CLI and AWS SDK

Hai Nguyen — Sun, 29 Sep 2024 15:58:45 +0000

Co-author: Le Hai Dang @natsu08122

1. Introduction
S3 lifecycle is a feature in Amazon S3 that allows you to automatically manage the storage of objects throughout their lifecycle. The main purpose of S3 lifecycle is help AWS user to optimize storage costs and automate data management. There are two main actions:

Transition: Move objects between storage classes.
Expiration: Delete objects after a specified time.

S3 lifecycle does support these storage classes: Standard, Intelligent-Tiering, Standard-IA (Infrequent Access), One Zone-IA, Glacier Instant Retrieval, Glacier Flexible Retrieval, Glacier Deep Archive. One important thing to note: S3 Policy cannot block S3 Lifecycle Rules. Here is the official AWS docs.

2. Problem statement
One day, we have a bunch of S3 buckets with the below folder structure: //

The retention directory of an object is defined by its expiration time. So, we need a solution to make a appropriate lifecycle of all S3 buckets in the corresponding folders. By utilizing S3 lifecycle management feature, we will create the rules using two main information:

Prefix: /
Object Tag: "isFile:True" (It means that all objects should be tagged with : that is "isFile:True"

We have several methods to do it using AWS management console, IaC, CDK, AWS CLI, AWS SDK. This blog will cover using AWS CLI and SDK to accomplish this task.

3. Solutions
3.1. Using AWS CLI
Prerequisites:

AWS CLI version 2
AWS Credentials In order to create S3 lifecycle rule, we use the following command:

aws s3api put-bucket-lifecycle-configuration --bucket my-bucket --lifecycle-configuration file://config.json

However, there is one point that we should pay our attention to:

All old rules will be overwritten by the new rule. In other words, all old rules will be deleted, new rules will be created when running the command.

This is an example of config.json file:

{
  "Rules": [
      {
          "Expiration": {
              "Days": 7,
          },
          "Filter": {
              "And":
                  {
                  "Prefix": "7days/",
                  "Tags": [{
                      "Key": "isFile",
                      "Value": "true"
                  }],
              }
          },
          "ID": "7days-retention-policy",
          "Status": "Enabled",
      },
  ],
}

Note: If we want to have multiple conditions in "Filter" block, just use "And". Otherwise, we will get an error "MalformedXML".

3.2. Using AWS SDK
Prerequisites: AWS SAM latest version

How to do:

Step 1: Clone this code into your environment.
Step 2: Follow README.md in above code.
Step 3: Run the command to setup AWS environment:

sam deploy --guided

After deployment is complete, the S3 lifecycle rules generation code will run on Lambda.

4. Solutions

This blog shows how to automatically create S3 lifecycles using AWS CLI and AWS SDK. I hope it will be useful for you. Thank you for your reading and feel free to leave your comment!

Mitigating disruption during Amazon EKS cluster upgrade with blue/green deployment

Hai Nguyen — Thu, 27 Jun 2024 04:43:13 +0000

Co-author @coangha21

Table of Contents

In-place and blue/green upgrade strategies
Upgrade cluster process
- Prerequisite
- Update manifests
- Bootstrap new cluster
- Re-deploy add-ons and third-party tools with compatible version
- Re-deploy workloads
- Verify workloads
- DNS switchover
Stateful workloads migration
Conclusion

Introduction
Upgrading your Amazon EKS cluster version is necessary for security, performance optimization, new features, and long-term support. Nowadays, Amazon EKS introduces extended support plan for Kubernetes version that will cost you remarkably. The upgrade is never a easy game and can feel like a business continuity nightmare. Some may feel tempted to postpone the inevitable. In this blog, we will walk you through our upgrade process using the Blue/Green deployment strategy.

We’ll demonstrate this on an EKS cluster with EC2 instances as worker nodes. This strategy can be also applied the same for Fargate, and we'll leverage the popular AWS Retail Store sample application to demonstrate the steps. For the code, head over to the AWS repository. By the end of this blog, you'll have a clear understanding of what an EKS upgrade entails and how to navigate it with confidence.

In-Place vs. Blue/Green upgrade strategies
Upgrading a cluster can be a balance between cost and risk. There are two common strategies that be widely used: in-place and blue/green upgrades.

In-Place Upgrades: Simpler and more cost-effective. This strategy will modify your existing cluster directly. While this minimizes resource usage, it carries the risk of downtime and limits upgrades to single versions at a time. Additionally, rolling back requires extra steps.
Blue/Green Upgrades: This strategy prioritizes zero downtime by creating a brand new, upgraded cluster (the "green" environment) alongside the existing one (the "blue" environment). Here, you can migrate workloads individually, enabling upgrades across multiple versions. However, blue/green deployment requires managing two clusters simultaneously, which can be costly and strain regional resource capacity. Additionally, API endpoints and authentication methods change, requiring updates to tools like kubectl and CI/CD pipelines.

In-place upgrade method is ideal for cost-sensitive scenarios where downtime is less critical or where the two versions don’t have breaking changes. For situations demanding high availability or the ability to jump multiple versions, the blue/green strategy provides a safer solution but is also more resource-intensive and costly. Thoroughly consider your specific needs, resource constraints, and infra cost to determine the best suitable upgrade method for your cluster.

Upgrade cluster process

1. Prerequisite

Explore your cluster: Before diving into your cluster upgrade, system inventory is a mandatory step in order to have insight of what is running in your cluster. Note down your cluster version, add-on versions, and the number of services and applications running. This intel helps you choose the right upgrade strategy, identify potential compatibility issues, and plan a smooth migration for all your workloads. It's like gathering intel before a mission - the more you know, the smoother the upgrade!

The current cluster’s version is 1.24 and it running on extended support

Currently 04 adds-on are running.

The cluster is using EC2 instances as worker nodes

Karpenter adds-on for node autoscaling.

Around 12 services found

The application UI

Assess the impact of new version upgrade: Thoroughly review the release notes for the EKS and Kubernetes versions you want to upgrade to in order to fully grasp important information such as breaking changes and deprecated APIs. For instance, if I want to upgrade to EKS 1.29, I will read the following documents:
- Release notes của EKS: https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions-standard.html
- Kubernetes change log: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.29.md
- Kubernetes new version release notes: https://kubernetes.io/blog/2023/12/13/kubernetes-v1-29-release/
Backup EKS cluster (Optional)
Review and address deprecated APIs:
- Kubernetes may deprecates some APIs in new version. So, we need to identify and fix any usage of deprecated APIs within our workloads to ensure compatibility in new EKS version.
- It’s worth to read this deprecation policy to understand how Kubernetes deprecate APIs.
- There are several tools that help us find out the API deprecations in our clusters. One of them is “kube-no-trouble” aka kubent. At the time I write this document, the latest ruleset is for 1.29 in kubent. I run kubent with target version of 1.29 and got the below result. As you can see, kubent shows the deprecated APIs.

Deprecated APIs found by kubent

2. Update manifests
When we have deprecated APIs in our hands. For next steps, we need to update those API version by manually or tools such as “kubectl convert” that actually depends on number of deprecated APIs. We recommend you to update the API version manually to avoid any unforeseen error. For example, based on above kubent result, we see that our HPA apiVersion will be removed since version 1.26. This is original HPA manifest in the current EKS cluster v1.24 and updated HPA manifest in new version, respectively:

Old version

New version

3. Bootstrap new cluster
There are some typical options for a new Amazon EKS cluster deployment with your desired Kubernetes version such as AWS Management Console, eksctl tool, or Terraform. In this blog, we have deployed a new cluster, namely "green-eks", using version v1.29 and EC2 worker nodes.

New EKS cluster

EC2 worker nodes

4. Re-deploy add-ons and third-party tools with compatible version
Once the "green-eks" cluster is ready, we've re-deployed required custom add-ons and third-party tools. It's crucial to ensure those adds-on and third-party tools version are compatible with new cluster. For instance, this document shows us the suggested version of the Amazon VPC CNI add-on to use for each cluster version.

EKS adds-on in new cluster

5. Re-deploy workloads
Now that the foundation is laid, we can begin redeploying our workloads to the new "green-eks" cluster.

Application deployment in new cluster

6. Verify workloads
Once our workloads are deployed successfully in the "green-eks" cluster, it's verification time! The specific tests you run will depend on your application development process. You might opt for smoke test, integration test, manual test, or even a simple UI check like we did in this blog for demo purpose only. The key purpose is to ensure everything functions as intended in the new environment.

Application in new cluster

We also would check EKS adds-on operation. For example, Karpenter works well by scaling node as expected.

Karpenter deployment logs

7. DNS Switchover
When application is ready to serve the client requests, the final step is to switch traffic over to the "green-eks" cluster. We achieved this by updating our DNS records in DNS management such as Amazon Route 53 or any other DNS provider. Amazon Route 53 provides weighted routing policy, so we can initially direct a small percentage of users to the new cluster. This allows us perform a staged rollout and verify everything functions smoothly before migrating all traffic.

Weighted routing policy (source)

Stateful workloads migration

During workload deployments to new Kubernetes clusters, specific considerations arise for stateful workloads. These workloads, such as Solr databases or monitoring stacks like Prometheus and Grafana, require data persistence and careful migration strategies. One proven and reliable migration approach for ensuring data integrity is the backup and restore method. We shared our experience in Solr database migration between EKS cluster in previous blog. The blog serves as a reference guide for migrating your stateful workloads.

Conclusion
By leveraging the Blue/Green deployment strategy, we've successfully navigated our EKS upgrade with minimal disruption. This approach offers several benefits:

Reduced Downtime: Since you maintain a fully functional "blue" cluster while deploying the upgrade on "green," user traffic experiences minimal interruption.
Phased Rollout: Weighted routing policy with Amazon Route 53 allows for a staged rollout, letting you test the new cluster with a small percentage of users before fully traffic migration.
Rollback: If any issues arise in the new environment, you can easily switch traffic back to the "blue" cluster with minimum overhead.

This blog provides a high-level guideline for EKS upgrade process using blue/green deployment to mitigate system disruption. Remember to tailor the specific steps to your application and infrastructure. Through a well-prepared planning and execution, blue/green deployment can make your EKS upgrade a breeze!

What's Gateway API and how to deploy on AWS?

Hai Nguyen — Mon, 15 Jan 2024 09:15:44 +0000

Co-author: @coangha21

Gateway API is recently standing out to be a promising project that will change the way we manage traffic in Kubernetes. It is looking forward to being the next generation of APIs used for Ingress, Load Balancing, and Service Mesh functionalities. In today's blog, we will discuss what Gateway API is, what it offers and finally we will get our hands dirty to gain better understanding of the service. Let’s get started.

Gateway API overview
The Gateway API is a recently graduated (version 1.0 in October 2023) official Kubernetes project that aims to revolutionize L4 and L7 traffic routing within Kubernetes. The goal is to simplify and standardize the way ingress and load balancing are configured and managed, addressing limitations of existing solutions like Ingress and Service APIs.

You can see above Gateway API logo, it already speak for it self, it illustrates the dual purpose of this API, enabling routing for both North-South (Ingress) and East-West (Mesh) traffic to share the same configuration.

Now, let’s take a look at some of the key features that Gateway API offers:
1. Extensible and Role-oriented:

Unlike the single-purpose Ingress controller, Gateway API is designed with flexibility and specialization in mind.
It offers various resource types like Gateway, GatewayClass, HTTPRoute, GRPCRoute, and Policy that work together to define specific roles and capabilities for different networking tasks.
This allows for building sophisticated networking configurations with greater control and clarity.

2. Advanced Traffic Routing:

Gateway API goes beyond simple load balancing and provides powerful routing capabilities based on HTTP routing rules, path matching, headers, and even gRPC service names.
This facilitates setting up complex traffic destinations, traffic splitting, and A/B testing scenarios.

3. Protocol-Aware and Scalable:

The API supports both L4 (TCP/UDP) and L7 (HTTP/gRPC) protocols, offering a unified platform for all your networking needs.
Additionally, it's designed for scalability and performance to handle large workloads and complex network topologies.

4. Community-Driven and Evolving:

Gateway API is a community-driven project under the Kubernetes SIG Network, actively maintained and constantly evolving.
New features and capabilities are being added regularly, making it a future-proof solution for your Kubernetes networking needs.

From my point of view, Gateway API represents a significant leap forward in Kubernetes service networking. Its dynamic capabilities, flexible routing, and robust policy tools will empower developers and operators to manage external traffic with greater control, precision, and agility. If you have time, try it yourself, it will be “worth your time”.

What is the differences between Gateway API and Ingress?
While both Gateway API and Ingress manage traffic routing in Kubernetes, there are several key differences between the Gateway API and the traditional Ingress API, let’s go through some of them:
Functionality:

Ingress: Primarily focused on exposing HTTP applications with a straightforward, declarative syntax.
Gateway API: A more general API for proxying traffic, supporting various protocols like HTTP, gRPC, and even different backend targets like buckets or functions.

Flexibility:

Ingress: Limited configuration options with heavy reliance on annotations for advanced features.
Gateway API: More fine-grained control with dedicated objects for defining routes, listeners, and backends, promoting cleaner configuration and extensibility.

Protocol Support:

Ingress: Only supports HTTP.
Gateway API: Supports multiple protocols beyond HTTP, like gRPC and WebSockets.

Scalability:

Ingress: Can become complex to scale, often requiring external load balancers or intricate configurations.
Gateway API: Designed with scalability in mind, easily integrating with various data plane implementations.

Security:

Ingress: Limited built-in security features, primarily relying on annotations for authentication and authorization.
Gateway API: Supports extensions for implementing enhanced security features like authentication and authorization.

Other Differences:

Portability: Gateway API configurations are more portable across data planes due to its separation of concerns.
Management: Gateway API allows for better cluster operator control with dedicated objects for managing various components.
Maturity: Ingress is a stable, GA (General Availability) API, while Gateway API is still under development but rapidly gaining traction.

In summary, Ingress is a basic but mature solution for exposing simple HTTP applications in Kubernetes. Gateway API is a more powerful and flexible API that caters to diverse use cases, supports broader protocols, and scales more efficiently. It offers greater control and extensibility at the cost of slightly increased complexity.

"Which one should I choose?", it depends on your use case:

For Ingress: If you need a simple solution for exposing an HTTP application and don't require advanced features.
For Gateway API: If you need flexibility for various protocols, backends, or require extensibility for security or advanced routing features.

Please keep in mind that, Gateway API is not meant to replace Ingress entirely, but rather provide a more comprehensive and future-proof option for complex traffic routing needs in Kubernetes.

How to deploy Gateway API on AWS EKS
Finally, this is probably the part you are waiting for . Let’s deploy a Gateway API on our AWS EKS cluster. I will only show high-level steps that need to be done. For manifest deployment, please refer to this repository.

Architecture demo: we have 02 services (user and post). We use the picture of VPC Lattice and Gateway API for your mapping overview.

You can see in the picture that the Gateway API is composed of three main components: GatewayClass(Controller), Gateway, HTTPRoute/GRPCRoute, each of them is related to VPC Lattice objects.

How it works
The AWS Gateway API controller (GatewayClass) integrates VPC Lattice with the Kubernetes Gateway API. When installed in your cluster, the controller watches for the creation of Gateway API resources such as gateways, routes, and provisions corresponding Amazon VPC Lattice objects. This enables users to configure VPC Lattice Service Networks using Kubernetes APIs, without needing to write custom code or manage sidecar proxies. The AWS Gateway API Controller is an open-source project and is fully supported by AWS team.

Now let’s go through step by step to set this up on our EKS cluster.

Step by step guide:

Create GatewayClass:
First we need to create a GatewayClass (Gateway API controller), we will using AWS Gateway API controller. Before you create the GatewayClass, you need to setup 2 following things:
Setup security groups to allow all Pods communicating with VPC Lattice to allow traffic from the VPC Lattice managed prefix lists.
Create IRSA for Gateway API Controller.
For those steps, please refer to this link.

After all of that is done, we will create our first GatewayClass. You can find all manifests used in this demo here.

The outcome should be look like this:

Service networks (Gateway): Next, we will create a Gateway. Gateway describes how traffic can be translated to Services within the cluster (through Load Balancer, in-cluster proxy, external hardware, etc.). In AWS, Gateway points to a VPC Lattice service network. Services associated with the service network can be authorized for discovery, connectivity, accessibility, and observability.

Services and HTTPRoute: Finally, we will define Services and Routes using K8s object Service and HTTPRoute to start routing traffic between services.

Service: User

Service: Post

Target groups for 02 services:

Result: Now let’s check if service “post” can called service “user” via domain name in VPC Lattice and vice versa.

It's worked!

Conclusion
Even though Gateway API is new and on it way to accomplish, it’s already showing lots of potentials. With more features and improvement coming in the future, we can expect it to be the future of APIs used for Ingress, Load Balancing, and Service Mesh functionalities.

References:

How to migrate Apache Solr from the existing cluster to Amazon EKS

Hai Nguyen — Thu, 31 Aug 2023 03:24:37 +0000

Co-author: @coangha21

Solr is an open-source enterprise-search platform, written in Java. Its major features include full-text search, hit highlighting, faceted search, real-time indexing, dynamic clustering, database integration, NoSQL features and rich document (e.g., Word, PDF) handling. Providing distributed search and index replication, Solr is designed for scalability and fault tolerance. Solr is widely used for enterprise search and analytics use cases and has an active development community and regular releases.

Solr runs as a standalone full-text search server. It uses the Lucene Java search library at its core for full-text indexing and search, and has REST-like HTTP/XML and JSON APIs that make it usable from most popular programming languages. Solr's external configuration allows it to be tailored to many types of applications without Java coding, and it has a plugin architecture to support more advanced customization.

In this article, I'll walk through the process of migrate Solr from a Kubernetes cluster to Amazon EKS (Elastic Kubernetes Service) using backup and restore method. Please take into account, depend on your Solr's version, system requirement, circumstance, etc, you will need to extra setup on EFS to ensure your K8s cluster is able to access to network file system on AWS. This will not be required if your Solr’s version can use S3 as backup repository. Please refer to the links below:

For the purpose of demonstrating, I'll migrate Solr from an EKS cluster to another one within the same region. However, you can apply this migration method to any Kubernetes cluster running on any platform.

Prerequisite:

Before you begin, make sure you have the following available:

AWS account and required permission to create resources
Terraform or AWS CLI, kubectl and helm installed on your machine

The following step is what we will do in this article:

Step 1: Create target EKS cluster.

Step 2: Install Solr using Helm.

Step 3: Setup Solr backup storage.

Step 4: Create backup from origin cluster.

Step 5: Restore Solr to EKS using backup.

Let's go in details!

Create target EKS cluster

There are many ways to create a cluster such as using eksctl. In my case, I will use terraform module cause it’s easy to reuse and comprehend.

This is my Terraform code template to create the cluster. You can just copy and run it or customize based on your desired configurations:



provider "aws" {
  region = "ap-southeast-1"
  default_tags {
    tags = {
      environment = "Dev"
    }
  }
}

provider "kubernetes" {
  host                   = module.eks.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
  token                  = data.aws_eks_cluster_auth.this.token
}

provider "helm" {
  kubernetes {
    host                   = module.eks.cluster_endpoint
    cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
    token                  = data.aws_eks_cluster_auth.this.token
  }
}

provider "kubectl" {
  apply_retry_count      = 10
  host                   = module.eks.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
  load_config_file       = false
  token                  = data.aws_eks_cluster_auth.this.token
}

data "aws_eks_cluster_auth" "this" {
  name = module.eks.cluster_name
}

data "aws_availability_zones" "available" {} 

locals {
  region = "ap-southeast-1"

  vpc_cidr = "10.0.0.0/16"
  azs      = slice(data.aws_availability_zones.available.names, 0, 3)
}

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.12"

  ## EKS Cluster Config
  cluster_name       = "solr-demo"
  cluster_version    = "1.25"

  ## VPC Config
  vpc_id                   = module.vpc.vpc_id
  subnet_ids               = module.vpc.private_subnets

  # EKS Cluster Network Config
  cluster_endpoint_private_access      = true
  cluster_endpoint_public_access       = true

  ## EKS Worker
  eks_managed_node_groups  = {
    "solr-nodegroup" = {
      node_group_name    = "solr_managed_node_group"
      # launch_template_os = "amazonlinux2eks"
      public_ip          = false
      pre_userdata       = <<-EOF
          yum install -y amazon-ssm-agent
          systemctl enable amazon-ssm-agent && systemctl start amazon-ssm-agent
        EOF
      desired_size       = 2
      ami_type           = "AL2_x86_64"
      capacity_type      = "ON_DEMAND"
      instance_types     = ["t3.medium"]
      disk_size          = 30
    }
  }
}

module "eks_blueprints_addons_common" {
  source  = "aws-ia/eks-blueprints-addons/aws"
  version = "~> 1.3.0"

  cluster_name      = module.eks.cluster_name
  cluster_endpoint  = module.eks.cluster_endpoint
  cluster_version   = module.eks.cluster_version
  oidc_provider_arn = module.eks.oidc_provider_arn

  create_delay_dependencies = [for ng in module.eks.eks_managed_node_groups: ng.node_group_arn]

  eks_addons = {
    aws-ebs-csi-driver = {
      service_account_role_arn = module.ebs_csi_driver_irsa.iam_role_arn
    }
    vpc-cni = {
      service_account_role_arn = module.aws_node_irsa.iam_role_arn
    }
    coredns = {
    }
    kube-proxy = {
    }
  }
  enable_aws_efs_csi_driver = true
}

## Resource for VPC CNI Addon
module "aws_node_irsa" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  version = "~> 5.20"

  role_name_prefix = "${module.eks.cluster_name}-aws-node-"

  attach_vpc_cni_policy = true
  vpc_cni_enable_ipv4   = true

  oidc_providers = {
    main = {
      provider_arn               = module.eks.oidc_provider_arn
      namespace_service_accounts = ["kube-system:aws-node"]
    }
  }
}

module "ebs_csi_driver_irsa" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  version = "~> 5.20"

  role_name_prefix = "${module.eks.cluster_name}-ebs-csi-driver-"

  attach_ebs_csi_policy = true

  oidc_providers = {
    main = {
      provider_arn               = module.eks.oidc_provider_arn
      namespace_service_accounts = ["kube-system:ebs-csi-controller-sa"]
    }
  }
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "solr-demo-subnet"
  cidr = local.vpc_cidr

  azs             = local.azs
  private_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 4, k)]
  public_subnets  = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k + 48)]

  enable_nat_gateway = true
  single_nat_gateway = true

  public_subnet_tags = {
    "kubernetes.io/role/elb" = 1
  }

  private_subnet_tags = {
    "kubernetes.io/role/internal-elb" = 1
  }
}

The following AWS resources will be created:

VPC with private and public subnets.
EKS cluster along with a node group (t3.medium x 01) and EKS adds-on (aws-ebs-csi-driver, vpc-cni, coredns, kube-proxy).

You can check AWS resources in AWS management console:

Install Solr using helm

Next step is to install Solr using helm:



### Install the Solr & Zookeeper CRDs
helm repo add apache-solr https://solr.apache.org/charts
helm repo update

### Install the Solr operator and Zookeeper Operator
kubectl create -f https://solr.apache.org/operator/downloads/crds/<version>/all-with-dependencies.yaml
helm install solr-operator apache-solr/solr-operator --version <version>

### Install the Solr, zookeeper
helm install solr apache-solr/solr -n solr --version <version>

Replace version with your chart version or chart version which contain your Solr’s version.

Next run this command to get admin’s password and access to Solr UI



### Get solr password
kubectl get secret solrcloud-security-bootstrap -n solr -o jsonpath='{.data.admin}' | base64 --decode
### Port forward Solr UI
kubectl port-forward service/solrcloud-common 3000:80 -n solr

Now open your browser and type http://localhost:3000, the result should be:

Then using the admin’s password we get above to login.

Setup Solr backup storage
After you have Solr installation done, it is time to setup Solr backup storage. At the time of writing this post, AWS supports 2 backup storage types: EFS and S3. Depending on your Solr’s version and system requirement, you can choose either of them. In this demo, I’ll use EFS as backup storage since this storage type is compatible to most Solr’s version. For more information, please visit this link.

To setup EFS as Solr’s backup storage, you need to create an EFS in AWS. This terraform code template will create EFS resource:



module "efs" {
  source  = "terraform-aws-modules/efs/aws"
  version = "1.2.0"

  # File system
  name           = "solr-backup-storage"

  performance_mode = "generalPurpose"
  throughput_mode  = "bursting"

  # Mount targets / security group
  mount_targets = {
    "ap-southeast-1a" = {
      subnet_id = module.vpc.private_subnets[0]
    }
    "ap-southeast-1b" = {
      subnet_id = module.vpc.private_subnets[1]
    }
    "ap-southeast-1c" = {
      subnet_id = module.vpc.private_subnets[2]
    }
  }

  deny_nonsecure_transport = false

  security_group_description = "EFS security group"
  security_group_vpc_id      = module.vpc.vpc_id
  security_group_rules       = {
    "private-subnet" = {
      cidr_blocks = module.vpc.private_subnets
    }
  }
}

Replace EFS-id with EFS resource ID (e.g. fs-1234567890abcdef) you just created in previous step and then run command:



kubectl apply -f solr-efs-pvc.yaml -n solr

From now on EFS is ready to use on your cluster. In next step, you need to upgrade Solr to take EFS as backup storage. First, create values.yaml file as below:



backupRepositories:
  - name: "solr-backup"
    volume:
      source: # Required
        persistentVolumeClaim:
          claimName: "solr-efs-claim"
      directory: "solr-backup/" # Optional

Note that you will need to do this for both origin and target cluster.

Second, you need to roll it out using helm, run command:



helm upgrade --install solr -f values.yaml apache-solr/solr -n solr --version <version>

Finally, you should see EFS claim name in your Solr pod using this command as an expected result:



kubeclt describe statefulset/dica-solrcloud -n solr | grep solr-efs-claim
    ClaimName:  solr-efs-claim

Create backup from source cluster

In order to restore Solr to new cluster, you definitely need to have a backup file in your hand. You are going to backup a collection using Solr API by using the following command:



curl --user admin:<password> https://<origin-solr-endpoint>/solr/admin/collections?action=BACKUP&name=<backup-name>&collection=<collection-name>&location=file:///var/solr/data/backup-restore/solr-backup&repository=solr-backup

If you have more than one collection, just repeat the process. Replace password with admin's password, origin-solr-endpoint for your origin Solr’s endpoint and as your choice.

You can check your backup progress by accessing to a pod and then check the directory:

Restore Solr to target EKS cluster

Since both origin and target Solr’s backup storage are using the same directory in AWS EFS as you setup in previous steps. You only need to invoke the restore API in your target cluster:



curl --user admin:<password> http://localhost:3000/solr/admin/collections?action=RESTORE&name=<backup-name>&location="/var/solr/data/backup-restore/solr-backup"&collection=<collection-name>

As I configured to use port-fowarding, I only need to replace Solr’s endpoint with localhost:3000. Finally, let’s go to Solr UI and you should see the collection have been restored successfully to your new EKS cluster.

After that, you can start setting up autoscaling, ingress, security, and other resources for Solr in new EKS cluster and connecting your application to the database.

Should you need any further information regarding Solr’s backup and restore API, please visit this link.

Conclusion

Solr is widely used in enterprise and SMB. Depending on your system requirements and circumstances, migrating Solr to Amazon EKS will require different setup and approaches. I hope this post will provide you with useful information about Solr migration using backup and restore. Any comments are welcomed. Thank you for your reading!

Thanks co-author @coangha21 for your effort in this post!

Automatically delete S3 buckets in all AWS regions

Hai Nguyen — Fri, 23 Sep 2022 10:06:15 +0000

In this blog, I will demonstrate how to automatically delete S3 buckets in all AWS regions based on a pre-defined schedule using EventBridge and Lambda.

1. Problem statement
We have many S3 buckets in several AWS regions in our account. Due to some reasons, our task is to delete all those S3 buckets in a pre-defined schedule such as once per week, once per month. A manual deletion using AWS management console will be a burden and time-consuming job. So, we have to figure it out in automatic way, which help us save our effort. Currently, AWS provides services that help us out.

2. Solution
We will use 02 services to automatically delete all existing S3 buckets in all AWS regions in a schedule. The workflow is shown as below:

EventBridge: A serverless event bus that let you receive, filter, transform, route, and deliver events. EventBridge can be considered a rule-driven router, which allows to define event patterns based on the actual content of the events to decide that targets receive each event passing through the bus. A target can be Lambda, AWS SNS, AWS SQS, etc. A rule in EventBridge that run on a schedule by using rate expression or cron expression can trigger a Lamda function to do a specific job. The following show a rule, which will run each 30 days using rate expression.

Lambda: A serverless service that run a custom code to delete all S3 buckets in all regions. It will be triggered periodically to run the function by a given EventBridge rule.

In order to delete a S3 bucket, all resources inside the bucket such as object, access points, etc. must be deleted in first. For multi-region access points, because all requests to create or maintain will be routed to the us-west-2 (Oregon region), the lambda function, which runs code to delete S3 buckets, should be deployed in us-west-2.
Refer to this for more information.

Lambda deployment:
My colleague and I wrote a Lambda function using Python to do the following steps:

List all bucket names in all regions.
Delete all multi-region access points.
Delete other resources (bucket policy, objects) and then permanently delete S3 buckets except the bucket name including a string "cloudtrail".
P/s: Bucket name, which contains a string "cloudtrail", stores the CloudTrail logs for our security audit. So, do not delete this bucket.

For source code reference, you can refer to this repository.

3. Conclusion
In this post, I showed a method that automatically delete all existing S3 buckets in all AWS regions in a specific schedule. Thanks @natsu08122 for your source code contribution. I hope this blog will be helpful for you. Should you have any question, feel free to leave your comment. Thank you for your reading!

AWS VPN site-to-site troubleshooting

Hai Nguyen — Thu, 22 Sep 2022 08:39:11 +0000

In this blog, I will show you how to troubleshoot a VPN site-to-site connection between AWS and other side.

1. Problem
Our customer wants to continuously backup their data from AWS Aurora MySQL to local provider cloud in private connection. So, a IPsec VPN site-to-site connection is required.
In AWS side, Virtual Private Gateway (VGW) provides dual-tunnels to Customer Gateway (CW) for high availability. If there is a device failure within AWS, the VPN connection automatically fails over to the second tunnel so that the connection is not interrupted. Meanwhile, local cloud provider's border router supports a single tunnel only. The below picture shows the scenario:

From time to time, AWS performs routine maintenance on the VPN connection such as tunnel endpoint replacement, which might briefly disable the tunnel. Refer to this for your information. It's going to interrupt the data replication progress.

In my case, VPN connection was configured using default parameters. It caused the tunnel down and could not be recovered automatically even the endpoint replacement finished. We can check VPN connection status metric in CloudWatch:

2. Solution
After the investigation, I found that the issue occurred by IKE_SA (Internet Key Exchange_Session Association) deletion between VGW and CGW. IKE is an IPsec based tunneling protocol that provides a secure VPN communication channel between peer VPN devices and defines negotiation and authentication for IPsec security associations (SAs) in a protected manner. So, if IKE_SA is deleted, the IPsec VPN connection will be down.

When AWS does the endpoint replacement, VPN connection will be interrupted. IKE_SA will be kept in a specific time, which defined by Dead Peer Detection (DPD) timeout parameter in both VGW and CGW. After DPD timeout occurs, VGW or CGW will send IKE_SA deletion request to the other. IKE_SA will be deleted then.

As soon as the endpoint replacement finishes, CGW or VGW should initiate the IKE negotiation to restart VPN tunnel. If both of them just keep waiting for the other, VPN tunnel will be down forever. Unfortunately, it is my case.

Because I could not change the configuration in CGW, which belongs to local cloud provider, I tried to change the setting in AWS side. There are 03 VPN tunnel options which we should consider, they are:

DPD timeout: it is 30 seconds in default. We can increase this value to cover the endpoint replacement time. But, we do not know how long it takes indeed and be careful it will affect the failover time to 2nd tunnel.
DPD timeout action: In default, the value is "Clear". It means end the IKE session and clear the routes. I changed to "Restart", which will restart the IKE session when DPD timeout occurs.
Startup action: "Add" is the default value, which request that CGW must initiate the IKE negotiation to bring the tunnel up. In my case, it should be "start". AWS will proactively initiates the IKE negotiation to bring the tunnel up instead of CGW.

For more information, you can refer to this.

3. Conclusion
In this blog, I showed the solution to deal with the AWS VPN site-to-site connection issue. Hopefully, it will be helpful for you. If you have any question, do not hesitate to leave your comment. Thank you for your reading!

Case Study: How to replicate database from AWS to outside?

Hai Nguyen — Sat, 17 Sep 2022 14:43:42 +0000

In this post, I will explain how to use AWS Database Migration Service (DMS) to replicate the data from AWS to outside.

1. The requirement

In my country - Vietnam, every company must to store the client information such as personal information in local for security concern. Please check this link for more information. Based on the government regulation, it does not matter whether you are a local company or international one, your client's personal data must be stored in Vietnam. In case your application is running on AWS or other public clouds, which currently does not have a region in Vietnam, how your company comply with our government regulation?

In this blog, I will show an approach which help you leverage AWS services and still comply with the government regulation when your company invests to Vietnam market.

2. Solution
Let's say you are running your application on top of AWS infrastructure. Your client data is stored in RDS instance. Based on the government regulation, you have to store the client data both in AWS and in the local environment. You rent a VM from local provider to store the data and replicate the data from AWS to VM.

The below diagram demonstrates the solution:

The components are:

AWS RDS for MySQL: storing the production database including the client's personal data.
AWS Database Migration Server (DMS): a managed service to help migrate commercial and open-source database to AWS quickly and securely. In this case, I used DMS to replicate continuously the data from a source (RDS instance) to a destination (VM in local cloud provider).
Virtual Private Gateway and firewall: establish a IPsec VPN site-to-site between AWS and local cloud environment. It provides a secured communication channel.
Virtual machine: running MySQL engine to store the replicated data from AWS RDS.

Let's go to the details:

a) AWS RDS instance: Assuming that I have a RDS instance for MySQL engine which stores the private client data.

b) Virtual machine in local provider:
A virtual machine is running MySQL engine.

c) DMS:
For DMS step-by-step configuration and best practices, you can refer to these link: Get started and best practices.

Step 1: Creating a replication instance to replicate the data from RDS to VM based on the pre-defined migration task.

Note: In production environment, a multi-AZ replication instance is highly recommended to provide high availability. In order to achieve high-performance replication, you should use instance type such as compute optimized or memory optimized instead of burstable one. I just use it for the demonstration only.

Step 2: Creating endpoints for source (RDS instance) and destination (virtual machine). You should run test the connection to make sure RDS instance can communicate with VM smoothly.

Step 3: Creating migration task:
There is a bunch of parameters which you should consider to configure. It depends on your requirements and your source database status. Refer to this for your information.

In migration task, validation can be enabled if you want DMS to compare the data between source and destination. Please keep in mind that this validation progress will make the replication task longer. Otherwise, you can compare the source data and destination one manually such as number of database, number of tables, and size of database, etc.

The migration task in running status shows that it is replicating the data based on the pre-defined parameters. For other status, refer to this link for your information.

In order to monitor the migration task, you can check CloudWatch Logs and Table statistics, which updates information regarding the state of your tables during replication. The possible table state can be:

Table does not exist: AWS DMS can't find the table on the source endpoint.
Before load: The full load process is enabled, but it hasn't started yet.
Full load: The full load process is in progress.
Table completed: Full load is completed.
Table cancelled: Loading of the table is canceled.
Table error: An error occurred when loading the table.

The following shows that full load is completed. It means all tables was replicated to the destination (VM).

Other parameters in table statistics such as Inserts, Deletes, Updates, and DDLs show the number of these statements that were replicated during the change data capture (CDC) phase.

3. Conclusion
In this blog, I demonstrated how to use AWS Database Migration Service (DMS) to replicate the data from a source (RDS instance) to a destination (VM in local provider) to comply the government regulation. Should you have any question, feel free to leave your comment. Thank you for your reading!

CloudFront with JWT authentication

Hai Nguyen — Thu, 25 Aug 2022 04:10:00 +0000

In this post, I will explain how CloudFront provides JWT authentication, which I applied successfully in the project. Hopefully, it will be helpful for you.

1. The requirement:
My customer provides LMS (Learning Management System) for language training on AWS. Their clients should buy training course to learn from recorded videos (Video on Demand). Videos are stored in S3 bucket as origin. After the client logins the web portal successfully, he will get a JWT (JSON Web Token) from webapp and then use the token to get video from origin via CloudFront.

2. Solution:
CloudFront authenticates using CloudFront Function the client's JWT and then do the following actions:

If the token is valid or the expiry time hasn't passed, CloudFront distribution checks its cache. In case of cache hit, CloudFront Edge delivers the video to client. Otherwise, CloudFront distribution request origin (S3 bucket) for the video.
If the token is invalid or the expiry time has passed or the token is not included in the client request, CloudFront Edge will send a response with error code 401 (Unauthorized) to the client.

The below shows the solution concept:

The components are:

CloudFront: CloudFront Functions runs JWT authentication source code. S3 bucket, which stores training videos, will be an origin of CloudFront distribution with Origin Access Identity (OAI) to restrict access to the bucket only through CloudFront.
S3 bucket: storing training videos. S3 bucket policy should be configured to allow only CloudFront distribution to access the bucket.

3. Introduction to CloudFront and CloudFront Functions:
3.1. CloudFront:
CloudFront is AWS CDN (Content Delivery Network), securely delivers content with low latency and high transfer speeds. It consists of a global network of CloudFront Edge locations that are distributed across the globe and use AWS Shield Standard in default to defend against DDoS attacks at no additional charge. Please check the below for more information:

3.2. CloudFront Functions:
CloudFront Functions are ideal for lightweight computation tasks on web requests. Some popular use cases are:

HTTP header manipulation: View, add, modify, or delete any of the request or response headers. For example, add HTTP Strict Transport Security (HSTS) headers to your response or copy the client IP address into a new HTTP header (like True-Client-IP) to forward this IP to the origin with the request.
URL rewrites and redirects : Generate a response from within CloudFront Functions to redirect requests to a different URL. For example, redirect a non-authenticated user from a restricted page to a paywall. You could also use URL rewrites for A/B testing a website.
Cache key manipulations and normalization : Transform HTTP request attributes (URL, headers, cookies, query strings) to construct the CloudFront cache key that is used for determining cache hits on future requests. By transforming the request attributes, you can normalize multiple requests to a single cache key, leading to an improved cache hit ratio.
Access authorization: Implement access control and authorization for the content delivered through CloudFront by creating and validating user-generated tokens, such as HMAC tokens or JSON web tokens (JWT), to allow or deny requests.

At the time I write this post, CloudFront Functions only supports JavaScript runtime. For more information, kindly refer to the link.

I referred to this AWS GitHub repository for JWT authentication source code. You can write the JWT authentication approach by yourself. It's up to you!

Here are some configurations in CloudFront:

a) CloudFront Functions after published in live deployment:

CloudFront Functions written in JavaScript to authenticate client request with JWT string parameter. It will decode JWT value using pre-defined key (in my case, it is AWS secret key). If JWT value is validated successfully by CloudFront Functions or the expiry time hasn't passed, the client request will be served. Otherwise, the response message “401 - Unauthorized” will be sent to the client.

CloudFront function can be also published in development environment for testing. Please remember that only in production environment, the function will be associated with CloudFront distribution.

b) In my case, the client request contains a JWT value in a query string parameter named jwt. CloudFront cache policy should be configured with cache key setting like this:

Note: query strings value may be different in your case. It depends on the parameter string which is defined in your backend code. In some cases, the parameter string is "token=xxxyyyzzz". So, query string value should be "token".

c) After creating CloudFront Functions, it should integrate with CloudFront behavior, in order to make the function in use:

Note: CloudFront Functions can be integrated with Viewer request and Viewer response. For Origin request and Origin response, only Lamda@Edge can be integrated.

After above steps done, I tested the function with Postman application before applying it into the project. A video was uploaded into S3 bucket (origin). An API request with valid JWT string parameter was passed from CloudFront’s authentication and received the response from CloudFront. Otherwise, the response message “401 - Unauthorized” was sent back. Here is the result:

The API with valid JWT parameter string. The response code was 200 OK.

The API request without JWT parameter string. The response code is 401 Unauthorized.

4.Conclusion:

In this post, I demonstrated how CloudFront can authenticate JWT from the client request to serve the video stored in S3 bucket. The solution may be helpful for who has a VoD application.

P/s: AWS document also provides the tutorial for hosting on-demand streaming video with S3, CloudFront and Route 53. If you are interested in this scenario, kindly refer to this link.

Thank you for your reading! I look forward to hearing any comments from you!

Case Study - AWS Application Migration Service (MGN)

Hai Nguyen — Tue, 23 Aug 2022 11:19:00 +0000

In this post, I will demonstrate how AWS Application Migration Service (MGN) helped me out from the issue which I got in the project.

1. My issue
As you may know, in order to launch the instance with these kind of OS (e.g. CentOS, SUSE, RedHat), an AMI subscription from AWS Market Place must be required. I subscribed AMI CentOS 7 Updates HVM published by AWS and then launched an EC2 instance with this OS. A lot of applications and middleware was installed on the instance, and an AMI was created for further use.

A few weeks later, I needed to share the AMI (including applications and middleware) to new AWS account. The customer tried to restore the instance using the shared AMI but it was failed because that CentOS 7 Updates HVM image published by AWS was no longer available for a new customer at that time.
(Note: At the time I write this post, CentOS 7 Updates HVM image by AWS come back available for new customer).

So, how could the customer restore the instance with unavailable base image? The only way was to create a new instance with available image in AWS Market Place. But, installing applications and middleware from scratch is a nightmare to our customer.

It's time for AWS MGN to action! Let's started.

2. Introduction to AWS MGN
Previously, CloudEndure (https://www.cloudendure.com/) or AWS Server Migration Service (AWS SMS) was migration tools to replicate the servers from on-premises and other clouds to AWS. In 2019, AWS acquired CloudEndure and continue developing the product. In March 2022, AWS introduced a new migration powerful tool, namely the AWS Application Migration Service (AWS MGN), which is now highly recommended as the primary migration service for lift-and-shift migration to AWS. Customers are encouraged to use AWS MGN for further migration.

AWS MGN enables the customers to migrate their applications to AWS with minimal downtime and without having to make any changes to the applications and the source servers.

Refer to the official link for more information: https://aws.amazon.com/application-migration-service/?nc1=h_ls

3. How I used MGN to replicate the data:
AWS MGN will replicate every applications, middleware, and config files from old server to new one. It helped my customer avoid their nightmare. The below diagram shows my scenario:

The components are:

Source server: The applications and middleware were running on. AWS replication agent has to be installed in the server to communicate with AWS MGN Service Manager via TCP port 443, with replication server via TCP port 1500 for data replication, and with S3 via VPC gateway endpoint.
Replication sever: The server stays in Staging Area to continuously replicate the data from source server. It will connect to MGN Service Manager via TCP port 443 and with S3 via VPC gateway endpoint. The replicated data will be stored in EBS volume.
Destination server: When data replication finish, I will do cut-over to launch a new server based on the pre-defined launch template. The EBS volume, which stores the replicated data, will be mounted to new server.

Let's go to step-by-step in AWS management console:

Step 1: Go to AWS Application Migration Service --> Add source server:

Installing replication agent in source server. The agent needs some permission to communicate with AWS MGN service Manager, so an IAM credentials (access key and secret key) is required. Following the guideline to create the required IAM credentials: https://docs.aws.amazon.com/mgn/latest/ug/credentials.html
or click on "Create IAM user":

Step 2: Download "aws-replication-installer-init.py" file:

wget -O ./aws-replication-installer-init.py https://aws-application-migration-service-ap-southeast-1.s3.ap-southeast-1.amazonaws.com/latest/linux/aws-replication-installer-init.py

Step 3: Running the file using python3:

sudo python3 aws-replication-installer-init.py --region ap-southeast-1

Following the guideline from CLI, entering the above IAM credentials and disks which will be replicated:

If the agent is installed successfully, it should be like this:

Step 4: Checking the source server status in AWS MGN console:

It shows the initial sync between source server and replication instance is finished and then the replication server is creating a EBS snapshot. Just wait for migration lifecycle status to be Ready.

Meanwhile, I had to configure EC2 launch template which defines instance specifications of destination server. Click on source server and go to tab "Launch setting", and then move to "Modify" of "EC2 Launch Template":

Note that you must mark the latest version of launch template to be default in order for AWS MGN to recognize it.

Step 5: Checking migration lifecycle of source server, making sure the status ready. It should be "Ready for testing" like this:

Step 6: I run "Launch test instances" to check the replication status:

You will see one m4.large instance in initializing status. It is AWS MGN conversion server, which converts the disks to boot and run on AWS. In particular, it makes bootloader changes, injects hypervisor drivers and installs cloud tools. Refer to the document: https://docs.aws.amazon.com/mgn/latest/ug/AWS-Related-FAQ.html#What-Conversion-Server-Do
The instance will be terminated as soon as the conversion job finishes.

Step 7: Make sure status of Alerts "launched". It means that successfully launched test/ cutover EC2 instance. Next, mark as "Ready for cutover":

Step 8: Launch cutover instance and just wait for the instance running:

You can check the cutover job status here:

Step 9: Make sure cutover progress finishes with successfully launched cutover EC2 instance like this:

Step 10: I finalized cutover by clicking on "Finalize cutover":

Finally, I connected to the destination server (cutover instance) to verify the data. It was amazing!!! Every data was replicated to the cutover instance same as source one.

4. Conclusion
In this post, I demonstrated my real case study regarding AWS Application Migration Service (MGN), which successfully replicated the data (applications and middleware) from source server to new one. It helped my customer save much efforts compared to installing everything from the scratch.