DEV Community: #developer #awsdeveloper

Effortlessly migrate your on-premise machines & application to any Cloud platform

#developer #awsdeveloper — Wed, 12 Apr 2023 08:57:55 +0000

Effortlessly migrate your on-premise machines & application to any Cloud platform

This blog is more about how we can effortlessly migrate our on-prem machines and Applications to the AWS cloud platform. In order to do this, we will be consuming the **CloudEndure **service which easily assists us to do our end to end migration.

CloudEndure is a SAAS service that automatically does a lift-and-shift (rehost) solution that simplifies, expedites, and reduces the cost of migrating applications to AWS.

In this article, we will be migrating our virtual machines from one region to another region. The source can vary in your case it can be your on-premies. For our demonstration, we are doing region to region migration.

Getting started with CloudEndure sign up for the **CloudEndure **Account. The CloudEndure is a separate portal not available directly on amazon. Below is the URL for the portal.

https://console.cloudendure.com/#/register/register

Once Registration is done you can login into the directly portal. In the portal, we have created a project called DemoMigration. For our use case, we have amazon Linux which is running in the EU(Ireland) and we will migrate the same in another region which is AWS EU(Stockholm)

CloudEndure consists of there parts which are the source, replication, and target. Once you have set-up the project configures the below settings.

The REPLICATION SETTINGS tab enables us to define your Source and Target environments, and the default Replication Servers in the Staging Area of the Target infrastructure.

Once all the required configuration is setup. Install the CloudEndure agent on the source machine to initiate replication. We have an amazon virtual machine VM with docker service installed in the AWS environment(source machine). See more on installing the CloudEndure agent here.

The replication instance will be launched on the target machine. The Replicated data will be in EBS snapshots.

Before proceeding make sure you have defined the blueprint for the target machine. Based on that the target machine will be created. Once all the required configuration is setup. Time to start the replication process to migrate the server.

CloudEndure gives the option to launch in test mode and cutover mode.

Test Mode-

Each Source machine for which a Target machine is launched will be marked as having a test Target machine launched on this date.

CutOver Mode-

Each Source machine for which a Target machine is launched will be marked as having a cutover Target machine launched on this date.

Once the migration is completed we can see the machine is getting created in the target region with the same set of configurations.

This migration is very much useful when an organization needs to migrate a lot of its servers and applications into the cloud.

Below you can find some helpful links and documentation regarding best practices and become more familiar with the CloudEndure Migration service: https://docs.cloudendure.com/Content/Configuring_and_Running_Migration/Migration_Best_Practices/Migration_Best_Practices.htm

Restructuring NAT Gateway Usage and replacing it with Squid Proxy

#developer #awsdeveloper — Thu, 06 Apr 2023 06:06:29 +0000

Restructuring NAT Gateway Usage and replacing it with Squid Proxy

We generally deploy our instances in a private network within our VPC in order to securely access them. However, for these instances, we require an outbound connection in order to have the latest OS security patches, and sometimes even some applications might need to connect to third-party URLs for various reasons. To have access to the internet we more often use cloud-provided services like NAT Gateway.

The below diagram depicts the typical deployment and usage of NAT Gateway

Nat gateway is highly available and takes care of most of the responsibility in terms of managing the service. But it becomes complex when organizations have several AWS accounts and then NAT gateway is deployed and managed in each of these accounts

Below are the few disadvantages with respect to use the Nat Gateway

Cost increases with the number of NAT Gateways deployed in each account
There is no ruleset to allow or deny traffic based on the rule list. This also includes access to malicious websites.
Detailed logging is missing.
Logs are not collected at a centralized location and no central control over them.

Solution Used

A centralized proxy solution is used to overcome the above-mentioned challenges
Squid Proxy(Open Source) is used for the outbound connection which gives more control at a centralized place
The solution is highly available, scalable, and is supported by different tenants.
This will save costs and operational effort while reducing the number of NAT gateways and the need for an Outbound proxy for each and every account.

This proxy will be deployed in a centralized account with a dedicated VPC. In order to have the Proxy VPC reachable from each tenant, VPC Interface endpoints powered by AWS private link will be used. Routing inside each VPC may be required in order to reach the VPC endpoint which will in fact be an ENI on a specific subset of the VPC.

To build this infrastructure solution. I have used terraform. In my opinion, terraform gives you more flexibility by writing less code compared to cloud formation.

Followed the concept of modules which gives more readability and easy understanding to the code. These modules can easily be reused. This also helps in reducing duplication, enable isolation, and enhance testability.

To over this challenge, SQUID Proxy is one of the possible ways to overcome

SQUID proxy is an open-source that can be used easily.
This proxy provides the filtering mechanism by which can easily restrict IPs and other websites.

This solution is designed to achieve high -availably using the autoscaling group with Network Load balancer.

The below is the structure for the Terraform modules for this use-case.

We can create different separations for each environment. This gives more control over the entire management of the infrastructure. With this approach, we can easily define have had lights configuration(Instance Type, high-availability) for dev and acceptance compared to the production environment.

One of the major challenges which we faced while building this solution was how we can consume the existing VPC and subnets which are part of the account itself. We wanted to avoid the hardcoding of the exiting VPC and subnets under variables files. For this, we use the filtering mechanism of the terraform. Based on the tag name we can easily search the desired VPC name and it’s respective subnets.

Conclusion

This overall solution personally I feel very useful and we can easily avoid the use of multiple NAT gateways

GitLab's Integration with AWS CodePipeline for ECS

#developer #awsdeveloper — Fri, 03 Mar 2023 09:01:11 +0000

GitLab's Integration with AWS CodePipeline for ECS

Aws codepipeline provides integration with most of the third-party repositories but for Gitlab’s the pipeline doesn’t supports the build in integration therefore it becomes challenging for developers to have a complete stack set of CICD pipeline.

This blog is more about how we can overcome this challenge and build a complete end-to-end pipeline. There are different ways to achieve this. Here we are using one of the easiest technique

The following diagram depicts how the CICD pipeline is set up for GitLab’s using s3 as a source and its different components

In order to integrate Gitlabs with Code Pipeline, the GitLab gives us a provision to do this by using the .gitlab-ci.yml file. The GitLab file should include the script with the required set of IAM Permission to S3 bucket which allows users to push the Object to s3 bucket. The file should be included under the project repository.

Therefore, whenever the developer commits the code into the GitLab’s repository with the use of GitLab-ci.yaml file which contains the script to push the repo code to the desired s3 bucket. Once the Object is uploaded into the s3 bucket the Code Pipeline has an event that is S3 as a source which in turn helps to trigger the pipeline and later executes the rest of the phases for the code pipeline will be executed.

Steps

Create the S3 Bucket. This bucket will be used to push the object.
Create the gitlab yaml file which includes the below scripts.

The above sample contains the variables and script. Script bascially includes the logic to upload the code to S3 bucket. Gitlabs provides the pre-defined variables like CI_COMMIT_SHORT_SHA. This commit code can be used to reterive the last commit hash made to the repository. We can used this commit hash code as a tagging to the bucket.

Create the IAM user which has permissions permisson to S3 bucket to put object as well as put object tagging.We are adding tag to a bucket with the commit hash code which help us in order to back trace any request in case of any issues.

Configure the AccesskeyID, AccessSecretkey in Gitlabs under Secret Variables.AWS Cli will automatically consume these varaibles whenever the scripts executes in order to upload the files as a zip to S3 bucket.

At last create the Codepipeline with S3 as source with Build and deploy steps.

Conclussion:

This is one of the easist techique which overcomes the challenges to setup end to end CICD pipeline quickly.

DEV Community: #developer #awsdeveloper

Effortlessly migrate your on-premise machines & application to any Cloud platform

Effortlessly migrate your on-premise machines & application to any Cloud platform

**Restructuring NAT Gateway Usage and replacing it with Squid Proxy**

Restructuring NAT Gateway Usage and replacing it with Squid Proxy

GitLab's Integration with AWS CodePipeline for ECS

GitLab's Integration with AWS CodePipeline for ECS

Conclussion:

Restructuring NAT Gateway Usage and replacing it with Squid Proxy