DEV Community: Hugo Thomaz

Zonal autoshift: Automatically redirecting traffic in response to potential issues across Availability Zones

Hugo Thomaz — Sun, 21 Jan 2024 20:40:18 +0000

Hello everyone! 🤓

Today, I've chosen to discuss one of the Networking & Content Delivery announcements from AWS re:Invent 2023.

The new feature of the Amazon Route 53 Application Recovery Controller allows AWS to shift our workloads from an Availability Zone when AWS identifies a potential failure affecting that Availability Zone, such as power, connectivity, network devices, and so on. It rarely happens, but it has already occurred, and it will happen. However, to avoid it, we should design and deploy our applications/services across multiple Availability Zones in a Region to ensure high availability.

The old feature, "Zonal shift," deployed before the new "Zonal autoshift," allows us to shift our workload to a new Availability Zone manually when the error originates from our side (customer side). However, when there is a potential issue at the Availability Zone level, it's challenging to identify because we don't monitor these resources managed by AWS. Typically, we check the "AWS Health Dashboard" or rely on others sharing their complaints about the issue. Consequently, during this period, we probably waste some time, and the application is out of service.

Now, with this release, you can configure Zonal autoshift to safeguard your workloads from potential failures in an Availability Zone. The AWS itself with their internal monitoring tools and metrics to determine when to initiate a network traffic shift.

Both Zonal shift and Zonal autoshift features, they operates exclusively at the Application Load Balancer (ALB) or Network Load Balancer (NLB) level, but only when cross-zone load balancing is disabled.

Ok, with these information let's see how to configure Zonal autoshift.

Note: We won't delve into the deployment of the Elastic Load Balancer, assuming familiarity on your part. Instead, let's concentrate on configuring the Zonal Autoshift settings.

1 - Seearch "Route 53 Application Recovery Controller", and open it.

2 - On the left pane, I select Zonal autoshift, and click in Configure zonal autoshift button.

3 - I've chosen the load balancer for my demo application. It's essential to note that currently, only load balancers with cross-zone load balancing turned off are eligible for zonal autoshift.

4 - To proceed with the settings, as you scroll down, you'll encounter the "Practice run blocked windows and dates" section, which is optional but crucial to configure. AWS typically tests the shift to another Availability Zone to ensure smooth application functioning in the event of an Availability Zone failure. Therefore, it's advisable to set the business hours during which you wouldn't want AWS to run this practice test, preventing any disruptions during peak business times. Additionally, you can block specific dates to avoid AWS conducting practice tests on public holidays, for example.

5 - In the final section of the settings, it's needed to configure at least the initial CloudWatch alarm ARN, but I recommend you create a CloudWatch alarm for both.

The primary purpose is to ensure that during a practice test, if the alarm doesn't transition into the ALARM state (down), AWS will consider it successful. Consequently, creating an alarm to monitor the health of the application or service is essential for this certification.

The second field, this alarm serves as a safeguard, preventing AWS from conducting the practice test if the specified alarm state is down. For instance, in exceptional scenarios, during business hours, your EC2 instance running the application might experience high CPU utilization due to increased customer access. To circumvent the inadvertent initiation of the Practice test during such atypical events, you can create an alarm to monitor and prevent the test accordingly.

Subsequently, tick the box to authorize AWS to shift our workloads from one Availability Zone to another when a failure at the Availability Zone level is identified.

Conclusion

In conclusion, I hope you liked it, I'd like to talk about this service, and also how we setup it. We can effectively deploy this solution to enhance our infrastructure. I trust that you found this discussion enjoyable.

Reference Link:
https://aws.amazon.com/blogs/aws/top-announcements-of-aws-reinvent-2023/
https://aws.amazon.com/blogs/aws/zonal-autoshift-automatically-shift-your-traffic-away-from-availability-zones-when-we-detect-potential-issues/

Connecting Your On-Premise Cisco Router to AWS Cloud Through a Site-to-Site VPN

Hugo Thomaz — Mon, 28 Aug 2023 00:34:23 +0000

Hello everyone! 🤓

As the Cloud grows, we Network Engineers need to create a connection between our On-Premises Data Center (DC) and the Cloud environment. There are many options to establish a connection between them,but now let's focus in create an AWS Site-to-Site VPN with a Cisco Router, and with BGP routing protocol enabled to exchange the network prefixes.

As I don't have a on-premise DC environment available, then we are going to simulate our DC inside of the AWS cloud, and our edge device on the DC side will be a Cisco CSR 1000V with IOS XE system. So if you need to create a Site-to-Site VPN between AWS cloud and your DC using a Cisco Router this post it's for you.

The diagram below show our proposal scenario, where on the left side we have the AWS environment with a VPC, private subnet, EC2 instance to test the connectivity, a Transit Gateway (TGW) to concentrate the connections with our VPC and site-to-site VPN. On the right side, we are going to simulate Corporate DC, as said before. As Customer Gateway or Egde device, we will use a Cisco CSR 1000V, and it will have an interface on the public subnet, which it's nedded to attach public IP address on it, and an interface on the Private subnet. This private subnet will simulate our internal network to allocate our internal services.

Proposed topology:

Note: I assume you know how to create the EC2 instance, VPC, subnets, SG, route tables, Transit Gateway, ... so this post will only concentrate in deploying the site-to-site VPN on the AWS and also the settings needed on the Cisco router to come up the tunnels.

Deployment steps:

1 - Create the customer gateway on the AWS side;
2 - Create a Site-to-Site VPN on the AWS side;
3 - Check the Transit Gateway Attachment created on the Site-to-Site VPN deployment;
4 - Create a Transit Gateway Route table for the Site-to-Site VPN;
5 - Setup the Site-to-Site VPN on the Cisco Router;
6 - Setup the BGP protocol on the Cisco Router;
7 - Check the routes learned on the AWS and Cisco router side;
8 - Test connectivity.

Those are the steps to deploy a site-to-site VPN as per the proposal scenario, so let's get start! 🙂

1 - Create the customer gateway on the AWS side

Our customer gateway is the Cisco CSR 1000V on the on-premises network. As all site-to-site VPN deployment, we need to know the Peer address of device that I would like to establish a VPN connection, then with this Public IP address, and also the BGP AS number - once we will use the BGP Routing protocol - in our hands, we can be able create the customer gateway.

Customer Gateway IP address: 3.132.131.135
BGP AS number: 65000

Now, let 's go to the VPC dashboard, and on the Virtual private network (VPN) session, clink on the "Customer gateways". For create the Customer gateways, click on the Create customer gateway button.

Define the name, Peer address and AS number as per the customer information, and at the end click on the Create customer gateway button.

Okay, once it's created, let's move on to the next step.

2 - Create a Site-to-Site VPN on the AWS side

On the VPC dashboard, and on the Virtual private network (VPN) session, clink on the "Site-to-Site VPN connections". For create a site-to-site VPN click on the Create VPN connection button.

Define the name, choose the Transit Gateway, and Customer gateway created on the before step.

Define BGP Route as routing protocol, protocol IPv4 address, there is more these other option to improve the VPN performance, and which networks will be allowed over VPN tunnels, but now as it's example, let's keep the option as default, but it's important to enable and define the settings for the production connections to improve the connection performance and retrict the networks allowed inside of the tunnels.

On the Tunnel 1 and Tunnel 2 options you can define the phases 1 and 2 settings like encryption, integrity, Diffie-Hellman group, IKE Version, lifetime, and a lot of stardand settings related to the IPSEC VPN, but now let's keep these options as default settings, and click on the Create VPN connection button

After some minutes the VPN connections move from pending to available status, but the Tunnels will still as down on the tunnels tab because we don't setup the VPN on the Router CSR1000V side.

For helping us on the router settings on the DC side, we download the configuration as you can see below.

There are a lot of the Vendor to choose, but now let's select generic device and choose the IKE version. In this case I defined the "ikev2" (safer than ikve1), and download the settings.

For assuranced the high available with the same Peer, the AWS create two tunnels for each VPN connection, then with this file you will see the informations about each tunnel as Pre-shared Key, Authentication, Encryption, Lifetime, DH, and other information about the VPN settings.

But before move to deploy router settings on the onpremise DC side, let's check other settings on the AWS side.

3 - Check the Transit Gateway Attachment created on the Site-to-Site VPN deployment

On the VPC dashboard, and on the Transit gateways session, clink on the Transit gateway attachments (TGW attachements). With the VPN ID realted the VPN connection created, you can filter the TGW attachement assigined to the VPN connection.

4 - Create a Transit Gateway Route table for the Site-to-Site VPN

On the VPC dashboard, and on the Transit gateways session, clink on the Transit gateway route tables (TGW route table). Let's create a TGW route table and assigned to the TGW attachement checked on the before step. Click on the Create transit gateway route table button.

Define a name, and assigned to the Transit Gateway.

After created it, let's assign the TGW attchement to this TGW route table. Filter the TGW route table, select it, click on the associations tab, and click on the create association button.

Select the TGW attachment to the VPN connection created and click on the create button.

Now, on the Propagations tab, let's propagate the routes related to the our internal VPC (10.100.0.0/16), and the customer routes learned by BGP routing protocol. So, let's assigned to the TGW attachement realted to the VPN and also the VPC application.

VPN TGW attachment:

VPC application TGW attachment:

TGW attachements assigned to the VPN TGW route table.

Now, let's to move the settings on the cisco router side on the On-premise DC.

5 - Setup the Site-to-Site VPN on the Cisco Router

Note: On the this part, let's focus on the IPSEC VPN on the Cisco router side, but the idea here it's not to detail each command.

Let's specify pre-shared key for each Peer of the AWS side. The IKEv2 keyring specifies the pre-shared keys used for IKEv2 negotiation.

Let's check the settings on the File downloaded on the Step #2.

Peer address and Pre-shared Key related to the tunnel #1:

Pre-shared Key related to the tunnel #2:

Based on these informations, let set these informations on the configuration mode.

!
crypto ikev2 keyring AWS-KEYRING
 # Tunnel 1
 peer vpn-aws-x-dc-tunnel1
  address 3.223.8.46
  pre-shared-key cN5zi29A3lJ_lqLUsOYiEkXFeypiW3F8
 !
 # Tunnel 2
 peer vpn-aws-x-dc-tunnel2
  address 34.236.4.200
  pre-shared-key GFfKIgr6LGmJlOR5bGCUAQq8pvEFHQ3q
 !
!

Let's create the IKEv2 profile:

!
crypto ikev2 profile AWS-US-EAST2-PROFILE
 match address local interface GigabitEthernet1
 match identity remote address 3.223.8.46 255.255.255.255 
 match identity remote address 34.236.4.200 255.255.255.255 
 authentication remote pre-share
 authentication local pre-share
 keyring local AWS-KEYRING
 lifetime 28800
 dpd 30 5 periodic
 exit
!

Note: Replace the "gigabitEthernet1" for your external interface.

Now, let's specify security policy for traffic.

!
crypto ipsec transform-set AWS-TS esp-aes esp-sha256-hmac 
 mode tunnel
exit
!

Create the The IPSec crypto profile that it is invoked in IKE Phase 2.

!
crypto ipsec profile AWS-VTI
 set transform-set AWS-TS 
 set pfs group20
 set ikev2-profile AWS-US-EAST2-PROFILE
!

Define the Interface tunnels:

Go back to thefile downloaded on the Step #2, and let's see the Inside IP Addresses related to the tunnel #1.

Let's see the Inside IP Addresses related to the tunnel #2

!
interface Tunnel30
 description AWS-US-EAST2-Tunnel1
 ip address 169.254.29.170 255.255.255.252
 ip mtu 1400
 ip tcp adjust-mss 1350
 keepalive 10 6
 tunnel source GigabitEthernet1
 tunnel mode ipsec ipv4
 tunnel destination 3.223.8.46
 tunnel protection ipsec profile AWS-VTI
 no shut
!
!
interface Tunnel40
 description AWS-US-EAST2-Tunnel2
 ip address 169.254.254.222 255.255.255.252
 ip mtu 1400
 ip tcp adjust-mss 1350
 keepalive 10 6
 tunnel source GigabitEthernet1
 tunnel mode ipsec ipv4
 tunnel destination 34.236.4.200
 tunnel protection ipsec profile AWS-VTI
 no shut
!
!

Afer that, we can see the interfaces tunnels up, and IPSeC VPN up as well.

6 - Setup the BGP protocol on the Cisco Router

Once VPC tunnels UP, we can run these command below to establish the BGP neighbors with both tunnels.

!
router bgp 65000
  bgp log-neighbor-changes
  bgp graceful-restart
  address-family ipv4 unicast
    !#Tunnel 1
    neighbor 169.254.29.169 remote-as 64512
    neighbor 169.254.29.169 ebgp-multihop 255
    neighbor 169.254.29.169 activate
    !
    !#Tunnel 2
    neighbor 169.254.254.221 remote-as 64512
    neighbor 169.254.254.221 ebgp-multihop 255
    neighbor 169.254.254.221 activate
    !
    network 10.200.50.0 mask 255.255.255.0
    network 10.200.1.0 mask 255.255.255.0
    no auto-summary
    no synchronization
  exit-address-family
 exit
!

7 - Check the routes learned on the AWS and Cisco router side

And after that, we can see the neighbor establish and the router 10.100.0.0/16 route learned via BGP protocol.

We can also check the status on the AWS side.

VPN tunnels and BGP running

and also see the route on the TGW route table assigned to the VPN connection.

8 - Test connectivity.

As the VPN tunnels are UP, and the routes were learned we can test the connectivity.

From a EC2 instance on the AWS side, which belong to the 10.100.1.0/24 network range, to the VM on the DC side which belong to the 10.200.1.0/24 network range. As you can see below on the screenshot the connection was established.

I hope you liked the procedure, and If you have any issue, please, let me know.

Deploying an Interface VPC Endpoint

Hugo Thomaz — Mon, 19 Jun 2023 00:25:34 +0000

Hello everyone!

In this post, we will discuss about the interface VPC Endpoint, and how it permit to access AWS services from the private subnet without Internet access. Additionally, I will guide you on deploying this resource using the AWS Console and provide the Terraform code from my GitLab repository for deploying the proposed topology.

Introduction:

Interface VPC endpoints permit us to access specific AWS Cloud Services without the need to NAT Gateway, Internet Gateway, VPN, or a Direct Connect connection. As opposed to Gateway VPC endpoints that use the route table and a prefix list, as you've already talked about it on this link, interface endpoints create a Elastic Network Interface (ENI) in your subnet, so when you need to access an AWS Services from a private subnet you call it through an internal IP address or DNS created by own interface endpoint. This simplifies the routing and allows for more flexibility.

Follow some AWS services supported by interfaces endpoints:

Objective:

After introduction about the Interface VPC endpoint, and some importants points that highlights its importance, let's move to our example scenario to learn in the practice. Now, we are going to create an interface VPC endpoint to access the AWS SQS Queue service through the endpoint, and also send a message from our private EC2 instance to SQS queue created for testing the connectivity.

Scenario proposed:

Note: I assume you know how to create the VPC, subnets, SG, route tables, so this post will only focus on deploying the resources to create the Gateway endpoint.

Deployment steps:

1 - Creating the Policy to allow send messages for the SQS Queue

In IAM dashboard, click on Policies, and then click create policy button

Click on JSON button, paste the policy and click in Next at the bottom of the page

Json policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "sqs:SendMessage"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

Define a name, description, check the access level, and click on the create policy button

Policy created

2 - Creating the Role, and assiging the policy created to it

On the IAM dashboard, click in Roles, and then create role button.

Select AWS service once that we will connect to the AWS SQS Queue service, and EC2 because this IAM Role will be used by EC2 Instaces to call AWS service. After select them, click in the next button.

Find the the policy name created on the step before, select it, and next.

Define a name, description, check the policy and click in create role

IAM Role created:

3 - Assigning the IAM Role created to the private EC2 instance

On the EC2 Instance dashboard, select the Private EC2 Instance, Action, Security and Modify IAM Role

Find the IAM Role created, select it and update IAM Role

3 - Creating the simple SQS Queue

Search for "sqs", and then click on the "Simple Queue Service"

Click on the Create queue button

Select the Standard queue, and define a name

At the Access Policy, select advanced mode, change the action from "SQS:*" to "SQS:SendMessage" what it will only permit to send message and let the SQS queue more safe, and then click on the create queue at the bottom of the page.

SQS Queue created:

4 - Creating the Interface VPC Endpoint for accessing the SQS Queue

In VPC dashboard, select Endpoint, and click in create endpoint button

Define a name, select AWS service, select com.amazonaws.us-east-1.sqs (type Interface), select the VPC, private subnet where the VPC endpoint will created a ENI, and select a Security Group with HTTPS traffic allowed to all Private subnet CIDR block.

At the end of the page click on the create endpoint button

Interface VPC endpoint created

5 - Check the connectivity

After some minutes, Interface VPC endpoint will be came available, so let's test the connectivity.

Accessing the Linux bastion host. I'm using my Key to access the Instances, but you should your own key.

Accessing the Private EC2 instance from the Bastion host

Now before send the message to the SQS Queue, let's validate some importants information.

On the VPC endpoint, the endpoint has a DNS name and it was assigned to a private subnet where an ENI as you can see below

Note: it's important to note that it was created many DNS records. the first one it's recommended to use when the source live in different Availability Zone of the VPC endpoint, but live at the same region. The second one it's recommended to use when the source live on the same Availability Zone of the VPC endpoint, and there is also the Private DNS names.

ENI assigned to the endpoint has an IP address what it belong to the private subnet what it's correct

Also the SQS Queue service created a DNS name too as you can see below

Now with these information in our hands, let's test the connectivity and send a message to the SQS Queue from private EC2 instance.

Example command to send a message to the SQS Queue, please, replacing the AWS Account ID and SQS Queue name at the URL (Documentation link)

aws sqs send-message --region us-east-1 --endpoint-url https://sqs.us-east-1.amazonaws.com/ --queue-url https://sqs.us-east-1.amazonaws.com/<AWS_ACCOUNT_ID>/<SQS_QUEUE_NAME> --message-body "Hello SQS, this is test send message."

Ok, I ran a nslookup to see if the DNS resolution was working fine, and it correct. It showed that the DNS resolution is translating to the IP address assigned to the the Interface VPC Endpoint. And after this test, it was be able to send a message to SQS Queue.

From SQS Queue dashboard we can validate if the message was arrived on the queue. Go to the SQS Queue dashboard,click on the "Send and receive messages" button, and then "Poll for messages".

After poll, you will see a message, and if you click on it you will see the message that it was used on the AWS CLI command.

After this validation we concluded our test.

If you would like to deploy this env via Terraform code as the purposed topology, so feel free to access my Gitlab repository to clone and deploy it. Before deploy via Terraform, replace the profile in provider.tf file and replace your Key_name at the EC2 instace code as per your own settings.

Conclusion:

In conclusion, we have discussed about the interface VPC Endpoint, and also we saw an example to deploy an endpoint. We can effectively deploy this solution to enhance our infrastructure. I trust that you found this discussion enjoyable.

Reference Link:
https://docs.aws.amazon.com/vpc/latest/privatelink/create-interface-endpoint.html
https://docs.aws.amazon.com/cli/latest/reference/sqs/send-message.html

Deploying the VPC Endpoint - Gateway for Amazon S3

Hugo Thomaz — Thu, 08 Jun 2023 19:48:41 +0000

Hello everyone!

In this post, we will discuss the VPC Endpoint - Gateway for Amazon S3, along with various scenarios that highlight its importance. Additionally, I will guide you on deploying this resource using the AWS Console and provide the Terraform code from my GitLab repository for deploying the proposed topology.

This EC2 instance is running inside of a Public subnet, and there are some data stored in S3 bucket. Let's say that the EC2 instance needs to upload and download some data stored in S3 bucket, so basically the EC2 instance would make an API call, and then it will access the data over the public internet through the Internet Gateway (IGW). It's obvious architecture, and it works perfectly there is no problem at all.

Now let's move on to an other scenario, where there is a similar architecture, but now the EC2 instance redise inside of the Private subnet. Now what happens? The EC2 instance needs to upload and download data in S3 bucket. We have learned on the old scenario that the EC2 instance needs to access the internet to upload/download the datas, but now as this EC2 instance reside into a private subnet, how will it be able to access the S3 bucket? So, that's why exist a resource called NAT Gateway. We would add a NAT Gateway in a public subnet, modify the private route table, and then the traffic will go to the S3 bucket over the public internet. Now the EC2 instance will be able to access the S3. So this works perfectly too. There is no problem with that too.

Yeah, probally you are asking yourself, why do I need to use VPC endpoint services? If you noticed, there is one thing in commun on this both scanarios. In both scanarios, the EC2 instance need to reach out the S3 bucket over the public internet, but what AWS says, if all the services are into the same region, as your VPC, including S3 service that we saw, there is a better way to access these services and that better way is called VPC endpoint services.

So what are VPC endpoints, endpoints allow you to connect to AWS services, using a private network instead of going for a public network. Then, we won't need to use the IGW or NAT Gateway to reach the S3 bucket, which it's good because we can save money, for example, NAT Gateway charge per hour running as well as data flow, our traffic wouldn't go out over public internet that it's not safe like a private network, and other benefits.

There are two types of the VPC endpoint, Gateway and Interface endpoint, but now, let`s focus on the Gateway endpoint.

Now we are going to see how we can deploy a Gateway endpoint to permit the private EC2 instance (ec2_app_web) inside of the Private subnet without Internet access can reach the S3 bucket through Gateway Endpoint. As the private EC2 instance is not reachable through the internet we are going to create a Linux Bastion Host (linux_bastion_host) to access the private EC2 instance, and then test the connectivity with S3 bucket, as we can see on the digram below.

Note: I assume you know how to create the VPC, subnets, SG, route tables, so this post will only focus on deploying the resources to create the Gateway endpoint.

Deployment steps:

1 - Creating the Policy to permit List, Get and Put data into the S3 bucket.

In IAM dashboard, click on Policies, and then click policy button

Click on JSON button, paste the policy and click in Next

Policy:

{ "Version": "2012-10-17", "Statement": [ { "Sid": "Statement1", "Effect": "Allow", "Action": [ "s3:Get*", "s3:List*", "s3:Put*", "s3-object-lambda:Get*", "s3-object-lambda:List*", "s3-object-lambda:Put*" ], "Resource": "*" } ] }

Define a name, check the access level, and click on the create policy button

Policy created

Now, let's create the IAM Role to attach the private EC2 instance (ec2_app_web). On the IAM dashboard, click in Roles, and then create role button.

Select these option, and click in next

Find the the policy name created on the step before, select it, and next.

Define a name, description, check the policy and click in create role

Role created

Now, let's assgin the IAM Role created to the private EC2 instance (ec2_app_web). Select the instance, Action, Security and Modify IAM Role

Find the IAM Role created, select it and update IAM Role

2 - Create the Gateway endpoint for accessing the S3.

In VPC dashboard, select Endpoint, and click in create endpoint button

Define a name, select AWS service, select com.amazonaws.us-east-1.s3 (type Gateway), select the VPC, and route table that is assign to private subnet because the route table will update with a prefix to reach S3 service through the VPC endpoint. At the bottom of the page click on the create endpoint button

VPC endpoint created

If you check the route table that belong to the private subnet, you will see the route to reach the VPC endpoint.

3 - Create S3 bucket.

On the S3 management console, click in create bucket

Define a name, and region where EC2 instance was deployed. At the bottom of the page click on the create bucket button

Bucket created

4 - Check the connectivity

Now, we are going to access the Linux bastion host (linux_bastion_host), and from it access the private EC2 instance (ec2_app_web) to list the S3 bucket.

Accessing the Linux bastion host

Accessing the Private EC2 instance, listing all buckets, creating a file, and uploading it to the bucket.

Based on this result, we completed the deployment and validated connectivity to the S3 bucket from an EC2 instance without internet access.

Conclusion:

In conclusion, we have explored different scenarios that demonstrate the significance of utilizing the VPC Endpoint - Gateway for Amazon S3. By understanding its role and benefits, we can effectively deploy this solution to enhance our infrastructure. I trust that you found this discussion enjoyable.

Reference Link:
https://docs.aws.amazon.com/vpc/latest/privatelink/gateway-endpoints.html
https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html

How can I read variables into Terraform from a YAML file to deploy the resources on the AWS Cloud?

Hugo Thomaz — Mon, 13 Feb 2023 22:40:51 +0000

Hello everyone!

I've been joining a IaC project (Infrastructure as Code) where the Terraform has been used to deploy our resources into the Cloud, and some variables are being to store in the YAML file. Participating on the project, I have noticed that this model has some advantages as you can see below:

- Readability: the YAML use more human-readable syntax which it makes easier to understand each varaibles and their values.

- Organization: as the YAML permit to organize the varaibles into groups, you can manage it better in a complex infrastructure with a lot the varaibles.

- Better unification: YAML is much used with other DevOps tools, as for example Ansible, then it become esier to share the varaibles with different tools.

So, if you use the YAML file to store the Terraform variables, it will offer more flexibility and scalability to manage your infrastructure as code.

Objective:

Here I'm going to using a public module from the Terraform registry to deploy a VPC network and subnets (private and public) on AWS cloud, and the Terraform varaibles will become store into YAML file. It'll be an example, but feel free to use it to deploy other resources.

Note: In this post, I will not comment how the Terraform and module works, or how the AWS network resource works. If desired, comment here because I can develop another post or share AWS documentation explaining about each resource.

Before start, there are some requirements to deploy this scenarios that will be expose here. It's needed to have the Terraform and AWS CLI installed, AWS account set, and in my case, I'm going to use the VScode as source-code editor.

So, let's get started?

Firstly, let's clone the my public terraform_study github repository where there are the terraform code stored. So you can use the the git clone command to download the Terraform codes or feel free to copy manually. Here we're going to use the git clone command in my Linux machine to become easier.

cd ~
git clone https://github.com/hugothomazpsouza/terraform_study.git
cd terraform_study/Deploy_VPC_and_Subnets_resources_with_Variables_store_on_the_YAML_file/
ls -la

Result:

After completed the commands, you will se the some Terraform files and one YAML file.

Let's get start to talk about the "locals.tf" Terraform file. It refer to local variables in Terraform, and the tip here is becasue I'm using the "yamldecode" function to read "variables.yaml" YAML file as input for Terraform.

Now, let's check the "variables.yaml" YAML file. This YAML file defines two VPCs: "vpc_core_network" and "vpc_app_network", and each one store the VPC CIDR block, Availability Zones, Private and Public Subnets varaibles.

As defined on this file, we're going to create two VPCs, 3 Private subnets each one in an Availability Zone and also 3 Public subnets each one in an Availability Zone.

The "vpc.tf" Terraform file, as I said before, I'm calling a public module from the Terraform registry to deploy a VPC on AWS Cloud, but I've adjusted the input variables to get from YAML file, and I've created for_each argument for setting to the value of local.config.vpcs, meaning that Terraform will run the module once for each key-value pair in this data structure.

Now, we're going to talk about of other three Terraform files.

outputs.tf - it is used to define the outputs variables in Terraform. The outputs values can be used in other Terraform configurations or scripts after the resources has been deployed. For example, if you would like to get the VPC_ID to create another resource you can get it through the Outputs variables instead of set the VPC_ID manually. It will become your code more dynamically and scalable.

provider.tf - It is used to configure the Terraform provider that is used, in this case we've used AWS provider, but we can use other Cloud provider, such as Google Cloud, and other. On this provider file I've also defined the authentication method and the configuration data needed to access the provider.

version.tf - It is used to specify the version of Terraform configuration syntax. It is recommended to specify the Terraform version to ensure that Terraform uses the correct syntax for the desired version.

Now, after completed the understand about the Terraform files and YAML file, we're going to deploy the resources.

Through the CLI, let's run the Terraform init command to load the modul, installs the necessary providers, and also sets the backend for storing the Terraform state file.

After that, let's run the Terraform plan command to see the changes that the Terraform will make to your infrastructure. I'm going to add the command below, but I won't show the output of the command because there will be many lines.

You will see how many resources will add, change or destroy.

And now, let's run Terraform apply to deploy the resources shown in the terraform plan command.

There will be a question to approve the deployment, then type "yes".

After completed it, you will see the information how many resources were added, changed or destroyed, and also the outputs variables.

Now, you can go to the AWS console to see the resources deployed.

Note: For deleting all resources deployed, please, run the Terraform destroy command. You will need to approve it, then type "yes".

Conclusion:

The idea here was shown how to read variables into Terraform from a YAML file to deploy the resources in the AWS Cloud. This was a tasting, but you reuse this idea to use according to your environments.

Well, I hope you enjoyed it!

Centralizing Internet Output in Multiple AWS Accounts with the AWS Transit Gateway

Hugo Thomaz — Sat, 04 Feb 2023 00:08:45 +0000

Hello everyone!

The objective of this theme is to share a network architecture tip and how to implement it within the AWS Cloud in order to centralize Internet output instead of having multiple NAT Gateways and Internet Gateways scattered throughout the environment.

The goal is to achieve the following benefits:

Simplify the network architecture by routing all outgoing traffic through a centralized point, the AWS Transit Gateway.
Reduce costs by centralizing network management and reducing the number of necessary Internet gateways.
Improve security by having a single point of control for Internet access, reducing the attack surface and allowing centralized security policies.

Note: In this post, I will not comment on the function of each network resource. If desired, comment here because I can develop another post or share AWS documentation explaining about each resource.

So, let's get started?

Objectives:

1 - Create the Internet Gateway and attach it;
2 - Create a NAT Gateway for each zone and attach it;
3 - Create the Transit gateway (TGW);
4 - Create the Transit gateway attachments with each VPCs;
5 - Create the Transit gateway route table for each Transit gateway attachment;
6 - Adding the routes on the TGW route tables;
7 - Adding the routes on the VPC route tables of each AWS account;
8 - Testing the connection to the internet.

Network diagram purpose:

This project consists of three AWS Accounts. The Core Network AWS Account will provide the interface to the internet for the PROD and QA AWS Accounts that do not have access to the internet through themselves, that is, they will need to Core Network AWS Account to update them Operation System and access the services outside of the AWS environment.

I consider that the VPC, subnets (private and public), and route tables have already been deployed so we can focus on the Transit Gateway, NAT Gateway, Internet Gateway, and also update the routes on the route tables.

With based on this diagram, let's deploy this environments step-by-step.

1 - Create the Internet Gateway and attach it to the vpc-core-network on the Core Network AWS Account.

On the AWS console, go to the VPC dashboard, click on the Internet gateways, and Create internet gateway botton.

Define the Internet gateway name, and click on the Create internet gateway.

After created Internet gateway, let's attach it to the vpc-core-network VPC

Now let's update the public subnets' routing table with a default route (0.0.0.0/0) as the next hop to the internet gateway created in the previous steps. Then, go to the Route Tables, select the route table and edit the routes.

Create the default route, and set the Internet gateway as target, after that save the change.

2 - Create a NAT Gateway for each zone (us-east-1a and us-east-1b), and attach it for its respective public subnets on the Core Network AWS Account.

AWS NAT Gateway is a zone resource, which mean that it has high availabilty in the Availability Zone (AZ) where it was deployed, but if there is a issue on the AZ where the NAT Gateway was deployed the resource on that zone will lose the internet access, that's why we are going to create a NAT Gateway in two AZs to ensure higy availability.

So, go to the NAT Gateway dashboard and click on the Create NAT Gateway botton

Define a name, select the public subnet that belong to the us-east-1a AZ, allocate a Elastic IP (public ip) and create NAT Gateway.

Now, let's create the NAT Gateway to the subnet that belong to the us-east-1b AZ.

The NAT Gateways created.

3 - Create the Transit Gateway (TGW) on the Core Network AWS Account and sharing it to the PROD and QA AWS Accounts

On the AWS console, go to the VPC dashboard, click on the Transit gateways, and Create transit gateway botton.

Define the Transit gateway name, uncheck the "Default route table association" and "Default route table propagation" options, check the "Auto accept shared attachments" option, keep the other settings as default and the click on the Create transit gateway.

Note: the "Default route table association" and "Default route table propagation" options will associate and propagate the routes, but some case there is a requirement where some environments can't establish a connection. Then, when this options is disable Network team there will have more control between communication of the environments.

Transit gateway available.

Now, let's share the Core Network Account's TGW to the PROD and QA AWS Accounts.

Type RAM on the search bar, and choose the "Resource Access Manager".

On the session "Shared by me", click in "Shared resources", and Create resource share botton.

On the Step 1, define the a name to the resource that will be share, select "Trasit Gateway" as resource that will share, select it on the Selected resources, and click in Next at the botton of the page.

On the Step 2, keep these default settings, and click in Next.

On the Step 3, as my other AWS Account belong of the same Organization, the "Allow sharing only within your organization" option was selected, but you can shared to anothe AWS Account that doesn't part of the owner Organization. Select the AWS Account, type AWS Account IDs, in this case it was added and selected the PROD and QA AWS Accounts as you can see on the diagram. After that, click in Next.

On the last Step, make a review, and click on the "Create resource share" at the botton of the page.

The sharing process can take a while, but once it's done, you can see its status on the "Resource shares".

4 - Create the Transit gateway attachments with each VPCs.

Still in Core Network AWS Account, go back to the VPC dashboard, and in Transit Gateway session select the "Transit gateway attachments" option. Click on the Create transit gateway attachments to assign the TGW to the vpc-core-network VPC.

On the first part of the settings, define a name for the TGW attachment, select the TGW-01 created on the previeus step, and as the attachment will be with VPC, select the VPC, but there is other options of attachment (feel free to see the other options).

On this last part, select the "vpc-core-network" VPC, the subnets will be load, and select Private subnets related to the each AZ as you on can see on the diagram.

Transit gateway attachment created.

Now, sign into each AWS account (PROD and QA account) and repeat the same Transit gateway attachment procedure for their respective VPC. As this is a simulation/study environment, the PROD and QA accounts only have one subnet (according to the diagram) in the part of selecting the subnets you will see only one subnet to select.

5 - Create the Transit Gateway route table for each Transit gateway attachment created in the previous step.

On the Core Network AWS Account, go to VPC dashboard and in the Transit Gateway click on the "Transit gateway route tables" option. Click in "Create transit gateway route table" botton to create a TGW route table for the vpc-core-network VPC.

Define a TGW route table name, select the TGW-01, and click in Create transit gateway route table button.

After created the TGW route table, let's assign the TGW attachment assigned the vpc-core-network VPC. Click on the Associations tab, and Create association button.

Select the TGW attachment assigned the vpc-core-network VPC, and click on the Create association.

Now, let's create a TGW route table for the PROD and QA accounts. Follw the table with TGW attachment ID respective for PROD and QA VPC.

PROD Account
Name: TGW-attachment-vpc-prod-us-east-1
ID: tgw-attach-09d24c8d49c4b0a76

QA Account
Name: TGW-attachment-vpc-qa-us-east-1
ID: tgw-attach-09d24c8d49c4b0a76

Note: The TGW route table is created in the TGW's own account, so in this case it's the Core Network AWS Account.

PROD Account

Define a TGW route table name, select the TGW-01, and click in Create transit gateway route table button.

Let's associate the created TGW route table with the TGW attachment assigned to VPC vpc-prod-us-east-1.

QA Account

Define a TGW route table name, select the TGW-01, and click in Create transit gateway route table button.

Let's associate the created TGW route table with the TGW attachment assigned to VPC vpc-qa-us-east-1.

The three TGW route tables created.

6 - Adding the routes on the TGW route tables

With based on the network diagram, and the resources created let me shared some information that will help to set the routes on the TGW route tables.

- Core Network AWS Account
VPC Name: vpc-core-network - CIDR Block: 10.100.0.0/16
TGW Attachment name: TGW-attachment-vpc-core-network
TGW Attachment ID: tgw-attach-0dcf709522c248333

- PROD AWS Account
VPC Name: vpc-prod-us-east-1 - CIDR Block: 10.10.0.0/16
TGW Attachment name: TGW-attachment-vpc-prod-us-east-1
TGW Attachment ID: tgw-attach-09d24c8d49c4b0a76

- QA AWS Account
VPC Name: vpc-qa-us-east-1 - CIDR Block: 10.20.0.0/16
TGW Attachment name: TGW-attachment-vpc-qa-us-east-1
TGW Attachment ID: tgw-attach-0d3bd1d648be67976

On the Core Network Account, let's create two routes to reach the network that belong PROD AWS Account and QA AWS Account.

Select the "tgw-rtb-vpc-core-network", click on the Routes tab, and then click in Create static route button.

Define the PROD CIDR Block and PROD TGW Attachment ID.

Create a new one rote for QA AWS Account. Define the QA CIDR Block and QA TGW Attachment ID.

Routes created.

Now, on the "tgw-rtb-vpc-prod-us-east-1" and "tgw-rtb-vpc-qa-us-east-1" TGW route tables, let's create a default route (0.0.0.0/0) and define as target the Core Network AWS Account TGW Attachment Id. As you can see on the next screenshot.

tgw-rtb-vpc-prod-us-east-1

tgw-rtb-vpc-qa-us-east-1

7 - Adding the routes on the VPC route tables of each AWS account

On the Core Network Account, let's create the routes on the route table that is attached to the Public subnet and Private subnet.

On the Public Subnet (rtb-vpc-core-network-public-subnets), add these routes as you can see on the table, and screenshot.

CIDR Block | Next Hop
10.100.0.0/16 | local
10.10.0.0/16 | TGW-01
10.20.0.0/16 | TGW-01
0.0.0.0/0 | IGW

On the both Private Subnets, add these routes as you can see on the screenshots.

rtb-vpc-core-network-private-subnets-us-east-1a

As this route table belong to the private subnet of the eu-east-1a Zone, It should be used the NAT Gateway deployed on the eu-east-1a Zone as target.

rtb-vpc-core-network-private-subnets-us-east-1b

The same thing for the route table that belong to the private subnet of the eu-east-1b Zone, It should be used the NAT Gateway deployed on the eu-east-1b Zone as target.

Now, let's create a route on the PROD and QA Accounts.

Create a default route (0.0.0.0/0), and as target set the Transit gateway.

PROD Account

QA Account

8 - Testing the connection to the internet

From QA Account, it was created a EC2 instance where I ran some commands as you can below on screenshot to show that a EC2 instance assigned to a private subnet can reach the internet throught on Core Network Account.

The "curl ifconfig.me" command come back with public IP address, and if you check the NAT Gateway public IP address will valideted that the public ip address belong to the NAT Gateway of the Core Network Account as you can see on the screenshot below.

Conclusion

NAT Gateway is a network resource that, when there is a lot of it, can bring high costs and your bill at the end of the month will get more expensive, in addition to leaving the Internet output from several places decentralized and also more complex to manage. Then, this proposed architecture comes to demonstrate a solution with high availability, low management complexity, having a single exit point and reducing the cost of the AWS account bill.

Well, I hope you enjoyed it!