DEV Community: Robin Ford

AWS Golden Jacket; Is it worth the race?

Robin Ford — Wed, 13 Sep 2023 16:06:50 +0000

So, I've just completed my AWS Machine Learning Specialty certification. This now brings my total active certifications to 12, the "full suite" of achievable certification. In doing so I'm now eligible for the elusive "golden jacket" awarded by AWS to holders of all certifications.

I've had a few people ask my about my certification journey and as often as the how did I do it, is they why did I do it.

This post will not look at how, but address they why, and is it worth it. It is subjective to me as a person, my current role, and my current employer. As such some of the justification might not match you or your situation. I acknowledge that and hopefully you will see why it was right for me at this point in time.

Please note that the order of reasons is not a ranking. It is just the reasons on my journey in the order that they became relevant for whether to go after more certifications.

Why number 1

To help stand out in the talent market

Considering I have been working with AWS since about 2011 with significant hands on experience from about 2015 , my first certification was not achieved till 2018. Why did it take me 3-7 years? Well I didn't need it before then. I was in a large enterprise that recognized my skills and did not see value in a certification that expires. However, in 2018 I was looking for work and the certification was something that validated my skills.

So off I went and got the Solution Architect Associate, and started working towards my Solution Architect Professional. The idea being that along with saying I had done work on AWS and I could talk about AWS, I also had a certificate that said I met a certain level of knowledge.

It must have worked because it helped get me several interviews and my job at Accenture.

So, if you are looking to stand out and prove you can walk the talk, certifications can help. However, it is in combination with experience. It allowed me to pass a "skills required" list and get to an interview to demonstrate my skills.

Why number 2

To help stand out in your company

So if you are in a job and not looking for a new job do certifications not matter?

Well, as with anything in technology, "It Depends".

At my first job using AWS it didn't matter. While they would send me on training for multiple technologies (CCSK, AWS, ServiceNow, etc.) the certifications, in their eyes, didn't add any additional value. As such they would not pay for them and I saw no benefit in paying for them myself when I saw myself being there for longer that the validity of the certification.

So, if your company doesn't see the value, by which I mean it wont improve your career opportunities, then it might not be worth your effort pursuing them. However, if they are not offering the upskilling and training I'd ask is it a good company to be with and maybe go back to why number 1 and get the cert to help you leave and go to somewhere that recognizes talent and development.

However, if you're in a company where certification help differentiate you, and/or they are willing to pay for them, then you might want to pursue them. For my team/role in Accenture, certifications are a differentiator and something that are discussed in performance reviews. Now, I'm lucky enough to have them provide vouchers, in addition to getting vouchers through the AWS Community Builders program. So for me there is no financial burden, although there is a time commitment over and above my day job. However, I think you'd need to weigh up the cost vs benefit. If you can gain a certification that help you get promotion is it worth it? Well if the exam is Professional or Specialty at $300 and you get a pay rise of even $500 a year I would say that a 400% ROI is a great return.

Why number 3:

To help your company stand out in the market

In a competitive market for companies, especially consulting/out-sourcing, certificates can help your organization stand out in the market.

The Accenture AWS Business Group prides itself on the breadth and depth of knowledge its team has. While this knowledge can be implied though case studies, thought leadership and references, a simple way to show it is by the number of certified individuals and the numbers of certifications held.

Similar to the partner accreditation the certified individuals can help customers decide who the right partner to engage with is. So while it might not improve your career outlook, similar to reason 2, if you company is willing to fund hte certifications why not go for it. After all they will do you no harm, and while not actively support career progression they will not count against it, especially if it is an external metric that is monitored.

This is also the reason that after completing the Solution Architect Associate, and Solution Architect Professional I went back and got the Cloud Practitioner certification. Not because I thought I needed it but it help with our total certification count.

Why number 4:

I had the knowledge

For some of the certifications I already had the knowledge that put me in good stead for the exams. While AWS focused, both the Advance Networking and Security exam outlines showed that good general knowledge in the areas was as importance on the AWS knowledge. My view having had experience in both security and networking was I knew the generic/industry why and how and some experience in implanting on AWS that it would be a relatively easy exam to pass. Again, with my employer paying for the exam, it was an relatively easy decision to make and I passed both with scores in the 800s.

This is also the reason I gained the Developer Associate, SysOps Associate, and DevOps Professional in quick succession. After a few years at Accenture and some projects with more hands on engineering, especially DevOps work, I thought I had the knowledge. I took a practice exam for the associate exams and got good scores so just went for it. After gaining scores in the 900s for both associate certifications it gave me the confidence that I knew what I was doing so went for the professional the following month.

When the SAP on AWS certification was announced in 2021 I thought it looked interesting. I had some experience in SAP and supported an SAP migration from on premise DB2 to a cloud x86 environment. After looking at the exam outline it seemed that it was mainly infrastructure focused. I thought with the overlap between the exam and the Solution Architect Professional, Advance Networking and Security specialties, I would not need to learn too much. So after spending some time looking at the specifics of SAP on AWS such as overlay networks, SAP tooling etc., I took the beta exam, and after a long 4 month wait for the results, successfully passed.

So, if you feel you have the knowledge then go for it. This is especially true for the specialty exams where 50% is topic specific understanding and not AWS. If you think you have the right level of understanding and experience based on your career then just go for it. I know people who have converted from say Cisco certifications to AWS Advance Networking with easy because they know so much of the core networking components that they just need to learn how to do it on AWS.

Why number 5:

As a method of learning

While certifications are not everything to everyone, I like the structured approach to learning. By having a goal and criteria to work against I can ensure my effort is spent on the right things. As I do more work as both an architect and engineer I realize there are new things I need to learn and have a deeper understanding of.

A big one for me was with database and the reason I went for the Database specialty. While designing solutions knowing the best database and best deployment method was something that cropped up more and more, especially with so many new options available from AWS.

So my goal was to go and learn about all the AWS database options, the features, and when and how each one should be used. While I did take a look at lots of different guides and documents I feel the structure of the domains help me focus my effort. For database the 5 domains are DB Design, Deployment/Migration, Management, Monitoring/Troubleshooting, and Security. Using this approach I worked through the different offerings and made sure I know enough in each of these areas.

As a result I ensured not only did I feel confident in gaining the certification but I felt confident in my knowledge and abilities when discussing databases. Again, through learning and certification I validated my knowledge and experience. I possess more than just the certification but the certification shows my minimum standard.

Why number 6:

"Gotta Catch 'Em All"

At some point you get to the "the point of no return". For me this was where the question changed from "why would I get them all" to "why wouldn't I get them all". I suffer a little from imposter syndrome and under-recognize my skills and expertise. While many people advocated on my behalf and extolled my skills and expertise it took me a long time to accept it. But when I got to 10 certificates its hard to deny I knew what I was talking about and could probably get them all.

So at that point I made the personal objective of achieving the final two certification within a year. During that year my 3 professional certifications would also needed renewing so just a little challenge.

Now, maybe because the order of certifications but the final two ended up being a little easier that I thought. Maybe some of this is also the fact that you learn lots of "generic" items across all the certifications that are applicable to all of them.

But when I looked at the overlap between Database and Data Analytics certification it is quite high in my opinion. So if you've got the knowledge on all the databases, how and when to use them etc. you probably have an easy step up to analytics because the difference is really, how to get data in, how to move it, and how to analyze and display it.

Similarly, in my view the overlap between Data Analytics and Machine Learning are quite high. A good third of the certification was around data management, manipulation, and topics that were also in the analytics exam. Another third was around standard AWS topics such as security, operations, and DevOps. So again only a relatively small increment of knowledge to gain. And with a passing score of 720 in theory I could get half the algorithm and model wrong and pass.

So the why might be to get them all, but in response to the why wouldn't I, it became clear that it would be easier to get the 11th and 12th than it was say the 3rd and 4th. So why not do it.

So, will I continue to maintain my certification level?

This is something I am considering and will depend on the certification and my situation when it comes up for renewal.

At present my plan is to let at least the SAP on AWS, and Machine Learning certification expire and possibly the Database and Data Analytics certifications. The reason is due to the direction I am going. My role as a Technical Architect needs to understand these areas but they are not core to my role. Having completed them has given me the knowledge I need and I can demonstrate it by saying I achieved the certification.

For me the two professional certifications are critical so I will ensure these don't lapse. They will also renew the practitioner and associate certifications so that will help keep my numbers up. I will then ensure I maintain the Advance Networking and Security specialties. I see these two areas as fundamental to the cloud. Security I think is the specialty everyone should get as it is critical to success in the cloud. For networking, the type of clients I work with are typically large scale organization and as such networking, especially complex and on-premise networking is a key area.

And don't forget, I don't just have certifications. I keep my knowledge up to date, and gain a few badges, with lots of other training offerings. If you take a look at my credly page you will see as well as certifications I have partner and technology badges. For me the biggest value is in the journey and knowledge I've gained along the way. The certification and badges are just a by product of that.

I hope this blog has been useful for explaining my journey and why you might want to go after all, or any, AWS Certification. If you decide you want to go for them good luck and I look forward to seeing your journey on becoming a golden jacket holder.

You put what in a public subnet‽

Robin Ford — Tue, 04 Apr 2023 11:15:00 +0000

Its great seeing peoples designs for modern solutions and especially serverless. What is more impressive is, where VPC services are in use, they are splitting them out into separate tiers and subnets.

😕 But why do so many people put things in public subnets that don't need to be?

In this article I'll look at what I think should be in public subnets and why you try not to put anything in a public subnet you don't need to.

What is a public subnet?

So first lets look at what a public subnet is.
The AWS VPC docs, a public subnet is defined as "[having] a direct route to an internet gateway. Resources in a public subnet can access the public internet."

Conversely, this also means that a resource in a public subnet is also accessible from the public internet. And, for me, that is a huge risk.

So why isn't this the best thing to do?

There are two reasons I think this is not the best thing to do.

Firstly is security.
If you put something in a public subnet it poses a significantly increased security risk. In general new resources are detected and scanned in less than 5 minutes of being available on the internet. This is a big risk for your services and data and if you don't have to put something in the public subnet I wouldn't.

Second is performance.
While you could secure you external facing devices, you risk increasing latency for internal services or under conditions of extra load by doing this for all traffic. I believe layering services gives the optimum solution with out trying to make one layer do all the hard work. This also means you can then bypass layers such as external egress for internal services or those over trusted connections such as a direct connect.

So what can I put in a public subnet?

There are only two services I would deploy in to a public subnet.

Firstly are gateways.
Whether these are NAT Gateways, IPv6 Egress Only Gateways, or Internet Gateways, these have to be in the public subnet if you want outbound internet access. Ideally though I would be looking at building solutions that don't need internet access from application/solution VPCs reducing the need for gateways as a whole.

Second are Layer 3/4 Load Balancers.
Whether that is a Gateway Load Balancers (GWLB), or Network Load Balancers (NLB) these act as the inbound gateway for traffic to your VPC. Use them to be the broker directing traffic to internal services. Placing these in front of Application Load Balancers (ALB) allows you to direct traffic to different load balancers and have an outer perimeter of security such as enforcing TLS1.3. While many will argue an ALB can also do this, by segregating the role of NLB and ALB, the ALB performance is increased, and internal traffic is not as impacted from the effect of external traffic load.

But what about... ?

So you will now probably have a "what about" component in mind that you think you should put in a public subnet. I'll look at the common ones I see and address both why I think its not a good idea and what you should be doing instead.

Bastion / Jump boxes
Why this is not a good idea:
These systems are often open to everywhere or have poor control to remove IP permissions. The issue here is that if someone breaches these boxes they generally then have a direct route to lots of other systems. They are also often configured with tools, and credentials to access data such as storage and databases. This puts a lot of risk of data exposure.

What to do instead:
If you need to access a EC2 instance I'd recommend migrating from bastions to SSM Sessions. This gives you control at a user level of what systems they can connect to and removes the need for extra infrastructure. It can also be logged in terms of connections and, if needed, commands performed. If you have to have a box to run tools within the environment, or bridge connectivity to AWS services, then migrate the box to a private subnet and use session manager to access via SSH or tunnel through to access items such as RDS.

Application Load Balancers (ALB)
Why this is not a good idea:
So many designs and documents will say just put ALB in public subnet. While this will work, and can be secured, in my view its not the best way to do things. This is especially true where you have services access directly over the internet and internally within either your AWS estate or On-Premise systems. By separating the logic and applying different controls on each you can better protect your service.

What to do instead:
Put your ALB in the same subnets as your application and allow internal services to talk directly to the ALB. Then have an NLB in the public subnet for external traffic. Ideally the NLB is behind CloudFront. Both ELBs can be protected by AWS WAF. This means that layer 4 protect can happen quickly by devices only exposed to the internet reducing load and impact on the layer 4 devices that are also used by internal traffic.

EC2 instances acting as a Firewall/IDS/IPS device
Why this is not a good idea:
While you might want this functionality on the perimeter, putting your EC2 instances directly in the path opens them up to attack. As with Bastions, an instance with a public IP will be constantly scanned and probed. The easiest way to reduce this is by not exposing the server.

What to do instead:
Put a Gateway Load Balancer (GWLB) in the public subnet and use it to divert traffic to the security appliance. Not only does this reduce the load on the appliance, it also means that the security devices can be deployed and manage centrally but service multiple VPCs. Take a look at this blog from AWS on how to implement a GWLB.

As always these are just my view based on using AWS services for over a decade and having to deal with the fall out of systems that are exposed and breached as well as troubleshooting operational performance and reliability.

I'd love your feedback on this article or suggestion on what you would put in a public subnet.

Troubleshooting with VPC Flow Logs

Robin Ford — Thu, 05 Jan 2023 12:20:57 +0000

So you built your secure VPC, but things are not working as expected.Or maybe something changed on the infrastructure and now things are not working.

And as any network engineer knows, every application fault is always due to the network! So how do we prove traffic is getting to our systems and it's not the network?

The answer is VPC Flow Logs.

There is great guidance on Flow Logs in the AWS VPC documentation so I will try not to cover that. What I will try and do is clarify some areas and explain how we can then use them to understand what is going on in our network. Specifically how we can use the AWS CloudWatch Logs console to find out what is happening in our VPC and give us some pointers on what might be wrong.

What is a flow?

The VPC guide defines a flow as "characterized by a 5-tuple on a per network interface basis", which if you have no idea about networks means nothing.
So, firstly what is a tuple, and then what is a 5-tuple? Well, according to good old wikipedia;

An n-tuple is a sequence (or ordered list) of n elements, where n is a non-negative integer

So a tuple is an ordered list and 5 denotes there are 5 items in the list. In networking the tuple is defined as;

source IP address, source port, destination IP address, destination port, transport protocol.

As such using the 5-tuple we can uniquely identifies any UDP or TCP session based on the source and destination (IP and Port) and protocol. So even if there are lots of connections between 2 machines we will be able to identify each of the flows.

So how do I read a flow?

So know we know what a flow is how do we understand them?
Well lets take a look at some flows for an Instance I've spun up for testing.

So, what are we seeing, as it is more than the 5-tuple defined as a flow.
If you look in the VPC documentation you will see what the fields are that are captured by default and those you can add for further information if needed.

But lets group the fields into 2 categories.

The first group are fields that you will typically get from all network devices that capture traffic flows. So if you enable NetFlow on a Cisco router, JFlow on a Juniper, or even Wire Shark on your PC, you will get these fairly standard set of fields and generally in this order.

source IP address
destination IP address
source port
destination port
protocol number
number of packets
total number of bytes
start time
end time

The second group of fields are AWS' custom field and help you understand the version of Flow Logs being used, where the flow has come from, was the traffic allowed, and was the log data collected successfully.

flow log version
AWS account id
interface id
action taken
log status

So, if we look at the first line in the flow logs we see the following:

2 67**0 eni-0**8ed0 94.102.61.6 10.0.7.253 57835 12333 6 1 44 1672869645 1672869668 REJECT OK

So, we can tell that it was generated using Flow Log version 2 (default). The record came from eni-0*edo in account 67*0, useful for consolidated logging solutions. We can then tell that the source IP was 94.102.61.6 (Some Dutch IP collector) and the target IP was 10.0.7.253 (the eni's private IP). We can also tell that the source port was 57835 and the destination was 12333. The protocol used was TCP (TCP = 6, UDP = 17 and ICMP = 1, full list at IANA). A single packet was sent and that packet was 44 bytes in size. We then also have the start and end time in Unix EPOC unixtimestamp. We can then see that the packet was rejected by either a NCAL or Security Group, and the flow record was captured completely.

Searching the Logs

So how do we find the flows we want in a sea of connections?

Luckily CloudWatch Logs has a built in search functionality as well as ways to filter down the data.

The first thing to do is see if you can find the interface details for the system having issues. If you know your client is talking to a specific host/endpoint than the first way to narrow down the search is to select the log stream for that eni. This will then only show flows that were to or from the specific eni. If you do not know the interface details then you need to use the "Search log group" option but this will have all your flows and be slower to search.

The second thing to do is narrow down the time window. If you know when an issue occurred or are doing live testing you can use the pre-defined time windows or a custom period. Again you can search all the logs but results will be quicker if a smaller set of flows to search.

Finally you can search based on know data from the flows using the search box. For example you could type the source or destination IP address. For example if I know the source system is 172.58.17.20 I could add this as a filter and only get records with that as the source or destination.

HINT Put any know search variable in quotes with preceding and tailing spaces to get an exact match. For example,
Searching for " 172.58.17.20 " will only return IP 172.58.17.20. Where as searching for 172.58.17.20 will return that IP and also IP addresses 172.58.17.200-209.

Proving connections

So know we know how to read the flows and find the flows we need to look at, how do we prove the network is not an issue?

I've enabled SSH from my local machine to my test instance via both NACL and Security Groups. I have then searched for records that match my source IP of 8*...*0 in the console. It has returned me 4 "messages" or flow log entries as seen below.

So what are we seeing?

Well, we can see there is a source IP 8*...0 talking to the instance. We can also see that the instance is talking to the same IP as there are entries where the IP 8...*0 is he destination.
We can then see that the source is talking to the host on TCP port 22 (SSH) and the host talks back to the source on the 57439.

What does this tell us?

The first thing this tell us is that communications have been established. The source has sent data to the host and the host has replied.
We can tell that communication has been established and it is not just coincidence that the source and host tried to talk to each other at the same time by two pieces of information.

Firstly the ports pairs (source/destination) are identical between the two records. When a system responds to a session it will always send data back the port that the request came from. These "high ports" or "ephemeral ports" are randomly assigned when a connection is made and a rarely open and listening for connections.

Secondly is the start and end time stamps. These are identical for each pair of flows. If this we two separate connections based on the number of packets and data sent we would expect to see variation in the end times, even if the start time happened to coincide.

Finally, we see two sets of flows with the same 5-tuple. This would indicate that all the records are flows that are part of the same session between the client and host.

What if I don't see this?

So what if you don't see these pairs of flow or you see flows with REJECT?
Well this means that the connection has not been established.

If the flow log entry shows REJECT this means that either the NACL or Security Group is blocking the flow. If there is a single entry that is showing REJECT (as in the first screen shot) then this means the inbound flow is being rejected. If there are two flows but it is the second flow showing REJECT then this means it is an outbound rule that is blocking the traffic.

If you are seeing a single log entry that is showing ACCEPT but no return flow then this means the application or service is not running or, if running, not listening for requests on the specified port. In these instance check the application/service is up and running and that it is configured to listen for requests on correct port.

Hope you have found this useful and helps you to understand your networks a little better. In a future article I'll take a look at how we can analyze and visualize the data in flow logs.

Till then, safe travels on your cloud journey and go rock your AWS solutions.

How to use NACLs and Security Groups

Robin Ford — Tue, 06 Dec 2022 21:43:33 +0000

Following up from my last post (here) on what Network Access Control Lists (NACLs) and Security Groups (SGs) are, I will now take a look at where and how I think you should use them to ensure you have a secure network.

I'll use a basic scenario of a VPC (10.0.0.0/16) split into two public subnets, with access to the internet (10.0.0.0/24 and 10.0.1.0/24), and two private subnets, with no route the the internet (10.0.10.0/24 and 10.0.1.0/24). The application is running on 2 EC2 behind an application load balancer to discuss the options.

For a more detailed implementation take a look at my post on Creating a Well-Architected VPC which has multiple tiers and controls.

Default Security

So lets start with taking a look at what security you would get if you just used the defaults that are provided when you create a VPC.

As you can see by default you get a single security group and NACL.

Default NACL

The NACL has an allow any port from anywhere rule, often refered to as an ANY-ANY rule, on both inbound and outbound connections. This means that all traffic is allowed and is the same as not having an NACL.

All subnets not explicitly associated with another NACL have the default NACL applied to them. One of the challenges with the default NACL is removing the initial rule when creating custom rules. While this is easy in the console it can be tricky in code and often leads to the rule not being removed. This is one reason I recommend to always create a custom NACL with needed rules and apply it to the required subnets. This way, as long as you create one custom rule, the default Allow ALL rule is never created.

Default Security Group

The Security Group is slightly more restrictive. It has an inbound rule that only allows traffic from other components with the same security group (see the source is the security group id) but the outbound allows all traffic to leave the instance.

Security Groups have to be selected when creating a resource so there has to be a conscious decision to use the default security group. However if selected at least only other AWS resources with the default security group can talk to the resource.

As with the default NACL removing the default rules is a little complicated via code so again, create a custom security group with the rules you need, even if a replica or the default security group.

Affect on Application

So if we just used the default NACL and Security group for our little application what would happen?

From a NACL perspective nothing would be blocked so all traffic could enter and exit the subnets and VPC as long as routing was in place. This would mean the resources would receive packets that are sent to them and the security group would decide what to do.

From a Security Group perspective the load balancer would be able to talk to the application and receive a response as they are both in the same security group. In addition the application instances would be able to talk to each other for the same reason. However, external users would not be able to communicate with the load balancer as it is blocking traffic not from resources in the same security group.

Note: The user would not be able to get to the application instances as they are in private subnets with no routes from/to the internet.

Fix Defaults

So what could we do with the defaults to make our application work but more secure?

NACL

So what might we want to do at the network layer?

Lets only change the inbound any-any rule to only allow https from anywhere.

This means that any traffic entering a subnet has to be on port 443 but it can be from any source IP address. By leaving the outbound rule as the default any-any we ensure all reply traffic can flow correctly.

Security Group

So what can we do with a single security group?

First we need to allow https from anywhere so that the user can get to the load balancer. Then we want to remove the default rule so that only port 443 is allowed into the resources.

For now we can leave the default outbound rule. This means that the instance can get to remote resources with HTTPS for things such as application updates.

Create New

So while we can fix the default NACL and Security Group there is a better way.

What we should be doing is creating custom NACLs and Security Groups and not using the defaults. There are several reasons behind this.

Firstly, if you use the defaults there is no segregation/differentiation between resources. All VPC Subnets have the have the same NACL and all resources the same Security Group. By creating new NACLS and Security Groups we can have different rules applied to different areas of the architecture.

For example, we can have a Public Subnet NACL that only allows the ports required from the internet (443 in, 1024-65535 out) and then only required ports to the private subnet (443 out, 1024-65535 in). We can then have a Private Subnet NACL with the required ports from the public subnet (443 in, 1024-65535 out).

The same for Security groups we can create a ALB Security group with inbound from internet on port 443 and outbound to a new Application Security Group. In the new Application Security Group, to be applied to the instance, we can just allow inbound on port 443 from the ALB Security Group.

Secondly, by creating new groups we make infrastructure as code (IaC) easier as we don't need complicated lookups of the default components. As we will create the resources we can easily refer to them in code. This means as we change our application over time we can update rules as needed.

So hopefully this gives you some idea of where and why to apply security features to your VPC. While something than many fine problematic, doing this improves your security and performance.

Why performance also? Well if you can reduce traffic hitting your resources by using NACLs, and blocking ports on the resource using Security Groups, you systems do not have to work as hard processing network traffic. Ideally you block as close to the source as possible using NACLs as this moves processing of rules from the resource to the network devices.

As always I'd love your thoughts on this post.
In the next post I'll take a look at how to troubleshoot when traffic is blocked by NACLs and Security Groups.

Network Access Control Lists vs Security Groups

Robin Ford — Tue, 06 Dec 2022 21:13:11 +0000

What are NACLs and Security Groups?
What are the differences?
How do they work?

This post hopes to explain the different mechanisms used so you have a better understanding of how to use them.

Both are used to protect networks and resources, but there is often confusion about the difference between Network Access Control Lists (NACLs) and Security Groups, and when each should be used.

This post, aims to demystify the two concepts.

The differences that we will cover are:

Stateful vs Stateless
Inbound vs Outbound
Allow vs Deny
Rule Order

Future post will then look at how to use this knowledge to apply both NACLs and Security Groups, and how to troubleshoot connectivity issues when NACLs and Security Groups are in place.

If you want to see how this forms part of a Well-Architected Network take a look at my post on creating a VPC.

How to represent network traffic.

So the first thing to understand is how data travels over a network.

I like to think of IT network the same as a road network, and in a lot of ways its very similar.

Long distance connections such as the international bearers are like large motorways/interstates/expressways. Junctions on these are the large network centers and carrier hotels with huge routers. These junctions connect with other large bearers but also to smaller "local" connections. Local connections such as your broadband or AWS Direct Connect, are like the highways and main roads. Junctions on these are connected by smaller routers, such as the one in your home, as less traffic is handled by them. These routers connect the local connection to the final network such as your home Wi-Fi. Within the local network connections such as small subnets/VLANs are like the town/estate roads. Junctions between each of these are managed by switches and hubs.

With this in mind we can think of NACLs as gates or barriers on these junctions that determine if you are allowed through based on the combination of the source (IP and Port), destination (IP and Port), and protocol (TCP/UDP/ICMP etc.).

Security Groups on the other hand are like the locks on your doors. A NACL might allow traffic onto the network but the resource doesn't have to accept it. Just like a community or office building might have a gate with a guard, you don't have to accept someone into your house or individual office. And vise versa, just because your neighborhood allows people to drive on the roads you can stop lock the garage to prevent your car being driven out.

Stateful vs Stateless

So we can see a difference in where NACLs and Security Groups are applied, network vs resource level, but there is also another major difference.

NACLs are stateless when processed where as Security Groups are Stateful. This is a term applied to other firewall functions and you will see in documentation on AWS Network Firewall and other firewall providers. So what do the two terms mean.

Stateful

This means that devices examine, and track, packets that flow through them. This process records the state of the packet and as such examination is said to be stateful as the processing is aware of the state. This means that outbound (reply ) traffic is inherently allowed based on the inbound part of the flow being allowed. So, if we take our house as an example, if I am allowed into your house I am also allowed to leave.

Stateless

This means that traffic flows are not examined or its state maintained. As a result, each packet is looked at in it's own merit based on a set of rules. Lets think about our network as a carpark. At the entrance there are barriers to restrict width and height, and we might have a barrier to control entry. When you enter you are given a ticket that raises the barrier. At the exit the barrier you have to provide a paid ticket for the barrier to raise. The barrier doesn't know anything about you and applies the same exit criteria to all cars.

Inbound vs Outbound

So I think this is where things get confusing, especially as on both NACLs and Security Groups on AWS label everything as Inbound and Outbound Rules.

Lets take a typical web flow, https on port 443, to demonstrate the components as they apply to NACLS. We have a single NACL on the VPC and a Security Group on the EC2 instance running a web server. We've put the instance behind a network load balance to restrict access and allow scaling.

NACLs

In terms of connection this is an inbound flow from an external user to the VPC. However, for NACLs, which we now know are stateless, we have to provide a rule for every packet that enters or leaves the network (VPC or Subnet). As such the direction of the connection doesn't matter, just the direction of the packet.

So in this example we would have to have inbound rules in the NACL to allow anyone to come in when going to something running on port 443.

We also then need an outbound rule to allow the return traffic. In this instance we have to set the port range on the destination which could be any of the high/ephemeral ports of the users device.

What you will see is that in a NACL we can only specify 4 things; Source or Destination IP, Protocol (TCP/UDP/ICMP etc.), Port/Type and Action (Allow or Deny). And this is where it starts to get complicated and we start to see gaps in the security.

We are saying that anyone (0.0.0.0/0) can enter the VPC as long as the destination is accessed on port 443 but also have to remember to allow the return traffic, which can be forgotten. But we are also saying that you can access the webserver but also any other server that might be listening on the same port.

Security Groups

In terms of connection, as for NACL, this is an inbound flow from an external user to the EC2 instance.

So in this example we would only need to have an inbound rule in the Security Group to allow anyone to come in when accessing port 443.

Allow vs Deny

Another big difference between NACLs and Security Groups is the fact that NACLs can have deny rules. You we see there is a default deny at the bottom of the NACL that says if you are not allowed further up then deny. As well as this you can also create other deny rules.

Security Groups on the other hand only have allowed rules. If it is not allowed it will be blocked.

The reason for this is due to where they are applied. As security groups are at the resource level there is no where further the packet can go. If it is not allowed the resource just drops it.

On a NACL as it can be applied to a VPC or Subnet we want to explicitly tell the network device (router/switch) to drop the packet. This means that no further processing will happen.

This has the benefit of reducing traffic on resources further in the network as traffic will never reach them.

Rule Order

One of the other differences between NACLs and Security Groups is how they apply the rules.

NACLs

NACLs apply their rules in strict order based on the rule number. This means that if something is allowed higher up the rule list it can not be denied later and vice versa, if something is denied it can then not be allowed. Again this can cause confusion but has benefits in managing networks. A useful example of this is denying all traffic from a specific IP ranges. By having these denies as the first rules there is no further processing of packets. This reduces the load on the network devices and allows other packets to be processed faster. Another example would be known good IPs. For example I have a rule at the top of some of my NACLs that allow my home IP (Static from provider) to access on any port. This means no mater what I build in the VPC i can always access, subject to security groups, with out modifying NACLs.

Security Groups

As security groups are only allow rules, and applied on resources, all rules are evaluated together. This means that if any of the rules mean I have access I would be permitted access. One disadvantage of this is that it has to apply all the rules to every packet and if there are large rule sets it slows down the traffic. It is also why there is a limit on the number of rules that can be configured on a resource (SG rules times interfaces has to be less that 1000). It is also why some instances have problems when there are not enough resources to process the state of packets due to large rules.

I hope you've found this useful and have a better understanding of the two concepts.

In my next post I'll look at how to use them to your advantage to ensure highest security in your AWS network.

Why use a Transit Gateway

Robin Ford — Wed, 13 Jul 2022 12:01:59 +0000

You may see in my designs and discussions that I always use an AWS Transit Gateway for connections outside of the VPC.
While there are use cases where this does not make sense, which I'll describe, for the majority of organisations' use cases I believe that using Transit Gateways is the preferable solution.

Cost 💰

Firstly lets address the cost implications of using an AWS Transit Gateway over VPC Peering, as many will use this to justify using peering because they see it directly on their bill.

Transit Gateways have 2 charging mechanisms. Firstly is VPC Attachment charge and second is data processing. A VPC Attachment is charged per hour, at between 5¢ and 7¢ dependent on region. This means that the use of a Transit Gateway can cost between $36.5 and $51.1 a month per VPC. As you can see if you start having 100's of VPC this can be expensive. VPC Peering on the other hand do not charge for peering to be in place. For many this can be seen as the sole reason to not use a gateway. For Data Processing there is a flat rate that is charged of 2¢/GB within region. If data leaves the region there is then a flat rate of 2¢/GB for inter region data transfer.

For example a Transit Gateway in Sydney region (ap-southeast-2) attached to 3 VPCs with combined monthly usage of 10GB over a month between all 3 VPCs and multiple AZs will cost $153.50.

VPC Peering only has a single charging method of data transfer, although it can get complicated. If data remains within the same AZ in a region then there are no transfer charged. However, if data does not stay in the same availability zone, then there are charges for Inter-Zonal data transfers. This is charged at 1¢/GB in each direction.

So for the same 3 VPCs in Sydney region (ap-southeast-2) connected by VPC Peering with 10GB over a month between all 3 VPCs the monthly cost would be $0.20.

As you can see the impact of Transit Gateway in "simple" networks is high from a cost perspective even with low network utilisation. This is due to the attachments costs which are static.

So if you are only focused on cost, and have a small network, then peering is probably a better option for you to manage networking in AWS.

Its important to note that costs for on-premise connectivity may be more when not using AWS Transit Gateway. (See below)

Network Throughput 🚿

It was always quoted that if you had large data volumes then use VPC Peering connections, even if you used a Transit Gateway for other traffic, to get the increased capacity that a VPC Peering could offer.

So lets look at a large data hungry application that is moving several GBs a second between EC2 instances in different VPCs, say logs being sent to central logging servers.

For peering connections there is no restriction to bandwidth between the two instances over the instance network capacity. However, if data is sent in a single flow then EC2 instance bandwidth will be applied. This would mean that EC2 would restrict to either 5GB or 10GB per flow (see documentation). So for our example if we had 20 collector nodes running on instances with 5Gb network capacity (ec2 network performance information) talking to a cluster of 8 central nodes running on instances with 12.5GB network capacity there would be no throttling from a network perspective, although throughput may not be 100Gbps.

For a Transit Gateway if we look at the quotas we can see that the bandwidth is per connection. So if traffic was just between 2 servers then, subject to instance and flow limits, we could achieve 50Gb/s transfer. However, if we look at our many to one example, while capacity would not be throttled from the collector VPCs to the Transit Gateway, the connection to the VPC with the central nodes would be limited to 50Gbps.

Although I have used logging in this example, where there may be alternatives to move large data volumes or latency might be tolerated, the principle is that if large data movements are needed then VPC Peering might be the only way to achieve the desired through put.

Connectivity - VPN 🔐

In a simple AWS only environment, cost and throughput might be all we need to worry about in terms of deciding how to implement networking. However, few AWS landscapes are simple, and as you look into more traditional enterprises, the requirement that is external connectivity begins to complicate networking.

First lets look at VPNs. These are often used in the infancy of a move to the cloud as the are relatively simple, quick, and cheap to implement. In these scenarios the cloud estate is usually small POC or Non-Production, and traffic is easy to manage with separate VPNs per VPC as traffic is tightly controlled. However, as organisations move to larger estates with more complicated workloads the number of VPCs increases.

If we only use VPC Peering, and VPN connectivity is still needed outside of AWS, there are only then two options we can implement. Either each VPC needs it's own VPN to be establish, or a central account is needed to house the VPNs and a proxy to manage the transit traffic between VPCs.

If each VPC has its own VPN set up then there is a management overhead and an increase in cost as VPN connections are charged at 5¢ an hour ($36.5 a month). In addition the on-premise solution needs to be able to support the increased number of VPNs. For an estate of 3 VPCs with resilient connections this would be an additional cost of $146.

If we decided to have a central "transit VPC" then we would need a proxy solution to manage the traffic and connectivity to more VPCs. Even using a relatively cheap t-family instance will cost around $100 a month, and add administrative overhead.

If however we use a Transit Gateway we can use the gateway to terminate the VPN. This means that the solution is scalable as an increase in VPCs does not require extra VPN and, although Transit Gateway Attachment costs, does not increase the cost to provide the connectivity.

It's also important to note that even if multiple VPNs are configured on a Virtual Private Gateway that the aggregate throughput is still limited to 1.25Gbps where the same Customer Gateway is used. For a Transit Gateway up to 10Gbps can be achieved using ECMP (Equal Cost Multi-path) routing combining multiple VPNs to a single Customer Gateway.

Connectivity - Direct Connects 🔌

Now lets look at the second option for connectivity and that is Direct Connect.

As organisations network requirements grow, the throughput and latency of a VPN is generally not sufficient for the demands of the workloads. We can combine multiple VPNs on a Transit Gateway, but latency would not be improved. However, by moving to Direct Connect, capacity can be increased up to 100Gbps. More importantly as a dedicated and private connection, the latency impact of an internet VPN can be removed, even on low speed (sub 1Gbps) hosted connections.

If we utilise VPC Peering, then there are two ways we can connect our Direct Connect to our VPCs. We can either link the Virtual Interface direct to the VPC, or we can terminate the connection on a Direct Connect Gateway and attach that to the VPC. In both scenarios, the hosted connection interface or a single interface on a dedicated connection is linked to a VPC's Virtual Private Gateway. This is critical to account planning as a hosted connection only has a single interface and a dedicated connection can only have 50 private interfaces. In addition, if using hosted connections the bandwidth is not shared between multiple connections, as a result you need to sure each connection is sized for peak work load.

Using a Transit Gateway simplifies this as a single Transit Interface is connected to the gateway, either directly or via a Direct Connect Gateway. As a result only a single link needs to be configured and the capacity can be shared between VPC reducing wasted capacity if on a hosted connection. Another benefit of a Transit Gateway for connections is as VPCs are created and deleted there is no configuration required from network teams as the connection between on-premise and the Transit Gateway remains.

Routing 📡

We've seen in the previous sections things can start to get complicated to understand and manage when using networking components. In this section lets look at the impact on routing for the various connectivity components.

We'll look at a two common scenario for AWS networking. The first scenario is where inbound and outbound internet is housed in the same VPC as the application and external VPC connectivity is only needed to talk to other applications or on-premise over a VPN. The second scenario is where we have centralised inbound and outbound internet solutions, as well as inter VPC and on-premise communications. In both scenarios we have 10 application VPCs in a single region, on-premise using a VPN, and for IP ranges we will use a Prefix List to simplify the management of on-premise IPs and VPCs will be in the region of 172.16/12 range with a /22 CIDR.

Scenario 1

For VPC peering we have what seems like a relatively simple route table. In addition to the VPC CIDR as a Local Route there would only be a few extra entries. The default route (0.0.0.0/0) will point to either the Internet Gateway or NAT Gateway in the account. On-Premise routing will use the Prefix List if the VPN is static or BGP if dynamic routing is in place. The only additional routes that are then needed would be for each of the 9 peering connection to point the VPC's CIDRs to the relevant connection. As a result the route tables (1 private and 1 public) would have 12 entries each.

For the Transit Gateway networking these route tables are simplified even further. The default routes would stay the same, as would the VPC CIDR route. However, now we would not need 9 routes for each of the peering but instead a single route pointing to the Transit Gateway. As all VPCs are in the same range we can just have a route for 172.16.0.0/12 which will mean traffic for any VPC will be directed to the Transit Gateway. For the VPN we will terminate this on the Transit Gateway so we also need a route for the Prefix List pointing to the Transit Gateway. As a result the route table has shrunk from 12 to 4 entries.

Scenario 2

As we move to a centralised solution, it appears to be a simple change to the VPC Peering Option. All we need to worry about is the connection to the new centralised solution. We just need to remove the public subnet and its route table, and then in the private routable update the default route to be the security VPC peering. If only it was that simple. If we just set the route to be the VPC it would land in the VPC and be rejected as the VPC will not allow transient routing. So what other options do we have. Well the only option in this case would be to have a load-balancer in the centralised solution VPC and use a VPC endpoint for this communication as you can not reference resources between VPCs in a route table. As we are using a VPC Endpoint we dont need the peering to the centralised solution so now the VPC has a single route table which has 11 routes defined.

For Transit Gateway we run into a similar issue but lets see how we address it. As with peering we remove the public subnet and public route table. We now turn our attention to the private route table which currently has 4 entries (Local CIDR, Default Route, Prefix List and 172.16.0.0/12). As the default route, on-premise and all VPC traffic is over the Transit Gateway we can simplify this from an application VPC perspective to just have it's default route point to the Transit Gateway. This has now reduced the route table to 2 IP addresses. So how will the traffic that was via a NAT/Internet gateway get to the central solution? Well there are 2 things that are needed. Firstly in the Transit Gateway Route Table we add an entry for the default route pointing to the solutions VPC. Then, in the VPC we amend the route table of the subnets where the gateway is attached and point it to the interfaces of the solution (Instance or Load Balancer) which will then act as the gateway.

Security 🔒

So let's look at the security implications of the two networking solutions as often network controls are used to secure workloads.

When we look at all the security that can be applied to routing such as Network Access Controls (NACLs) and Security Groups these can equally be applied to VPCs with both Transit Gateways and Peering. So are there any security benefits or disadvantages either way?

Well, the obvious one is that peering connections offer the ability to reference security groups between VPCs. Great! Every time a new workload comes up in a connected VPC I can updated my security groups to allow them in by referencing their security group. Well, that is true in principle but the question is how are you going to apply it? If we look at the move to architectures such as micro-services, event driven architecture, and API first communication, along with methodologies such as GitOps, DevSecOps, and Zero Touch Deployments this is very hard to achieve. With out building tightly coupled code one application may not even know of the existence of another, let alone that it is hosted in a connected VPC.

So what else is there?

Well for me there are several security benefits of a Transit Gateway, especially with centralised connectivity solution such as proxies, WAF, etc.

Firstly is the fact that I can centrally manage all networking. This allows we to block developers or whole accounts from creating components such as an Internet Gateways of VPN connections that could compromise an account. I can then also manage what they are allowed to get to by having approval for updates to central ingress/egress systems or route tables.

Second is the fact that I can force the segregation of environments such as Production and Non-Production. By blocking any VPC peering I ensure that VPCs can only talk to other VPCs that are connected to the same Transit Gateway and have the same Transit Gateway Route Table. The only thing I have to do is share the Transit Gateway if in a different account and give the ability to attach to the VPC.

Finally is I have greater speed to deal with threats or compromises. By managing routing in a TGW and centralised security I can block communication with suspicious IPs, isolate a VPC, inspect traffic logs at the gateway level plus enforce scanning for things such as virus and PCI.

Quotas 📃

So lets look at where quotas might play a part and mean that a Transit Gateway is the only possible method to connect.
First lets look at a scenario with just inter-VPC communications.

The number of peering connections allowed per VPC is adjustable from the default of 50 to a maximum of 125 (see quota). This means that in large networks of over 125 VPCs a full peering mesh would not be possible, and alternative method would be needed to allow all VPC to communicate. In addition, while peering connections can be cross-region, the quota is at the VPC level.

On the other hand a Transit Gateway can have up to 5,000 attachments (see quota) which means that 5,000 VPCs can connect to a single gateway. In addition Transit Gateways can be peered with up-to 50 other Transit Gateways. This would allow Transit Gateways to peer with other regions at a gateway level or, if attachment limits were hit, VPC connecting via peered gateways. This would increase the limit to a maximum of 250 thousand connected VPCs.

Now lets look at a scenario with on-premise communications.

Using the scenario of a single hosted Direct Connect, connected via Direct Connect Gateway, we could chose to attach to a Transit Gateway or directly to VPC via a Virtual Private Gateway. In these scenarios the Direct Connect quotas need to be looked at.

In the case of Virtual Private Gateways there is a limit of 10 connections per Direct Connect Gateway. This would mean that a maximum of 10 VPCs could be attached and over that there would need to be a routing solution implemented similar to the VPN scenario earlier. As above, even t-family instance will cost around $100 a month, and add administrative overhead.

For Transit Gateways, the limit is 3 connections per Direct Connect Gateway. Although this is significantly lower, as a Transit Gateway can have 5 thousand associations, even with a single Transit Gateway we can have 500 times more VPC connections that using Virtual Private Gateways.

Conclusion 🏆

So we've looked at a lot of items and hopefully can see the use cases for both VPC Peering and Transit Gateways. But in most cases you can only choose one option.

So which is it to be?

Lets use the trusted AWS Well-Architected Framework pillars to help us decide. We'll look at each pillar and see what solution comes top in that pillar and then crown and overall winner.

Operational Excellence 🛠️

So for me Transit Gateways win on this pillar. Implementation is significantly simpler as the number of workloads increases. That along with the ability to monitor traffic on the gateway ensures that monitoring and troubleshooting connectivity can be implemented separately or in addition to VPC level solution.

Security 🔒

So while it would seem beneficial to having the ability to cross-reference security groups between VPCs I see this as being less practical than it was 10 years ago. With many organisation "All-In on AWS" there can be too many team, and deployment methods, to benefit from this feature. In addition, I believe that the extra security Transit Gateways can offer outweighs the loss of the security group functionality

Reliability ⛈️

As both option are AWS services this pillar is quite a tight battle. For me they are equal on 4 out of 5 design principles. The tie-breaker principle is "Stop guessing capacity". On the one hand VPC Peering has no network capacity restrictions, but Transit Gateways can make better use of shared connections such as VPN and Direct Connect. Digging deeper into some of the best practices we see that Transit Gateway starts to sway the balance. When looking at Quota's Transit Gateway is a big winner with number of VPCs that can be connected. Taking these into account I think Transit Gateway just wins on this pillar.

Performance Efficiency 🚀

So performance efficiency is an interesting pillar for connectivity. Initially it would seem that VPC Peering would win as it has the greater throughput. But is that all that makes something performant? If we look at monitoring performance Transit Gateway significantly simplifies that. If we then look at everything apart from through-put again Transit Gateway makes that easier to manage and implement. So for me Transit Gateway wins but with the caveat of using a VPC Peering Connection if throughput capacity is required.

Cost Optimization 💰

So cost will always be the hard one to argue. You can look at the surface and say Transit Gateway cost on average $40 per VPC per Month and VPC Peering costs nothing. But what costs do we save? If we centralise connectivity solutions such as internet access and VPN access. Internet access could save ~$100 per VPC for the removal of NAT Gateways ($120 average 3 AZ NAT Gateway cost minus cost to provide central solution), a significant amount if 100's of VPC with just minor internet access. VPN de-duplication would save ~$40 per VPC with the same VPN. Finally, a shared Direct Connect could reduce over-provisioning links especially on Hosted Connections. As an example, rather than 20 50Mbps links a single 1Gbps would give savings of 50% or a reduce capacity 500Mbps would provide 66% savings.

Sustainability 🌿

The biggest thing for sustainability is to not have something running or reduce the effort needed to run it. And then once you've decided what to run make sure you maximise utilisation. For networking the effort principle manifests itself in two ways. Firstly the number of connections that are established and then the effort required to route the traffic. If we look at the number of connections for 50 VPCs in a single region to communicate with no external access (internet or on-premise) we can see that for VPC peerings we would have 1225 connections (each VPC connected to 49 others) and route tables with a total of 2450 entries. When using Transit Gateways this number would be reduced to 100 connections (VPC and Gateway ends of the connection) and 100 route table entries. This reduction greatly reduces the energy needed to perform routing. So again, for this pillar I think Transit Gateway is a clear winner.

And the winner is...

Transit Gateway 🥳

I'm always glad when my thoughts built up over the last decade are justified. But hopefully your all in agreement, at least based on the information I've provided, that Transit Gateways are the best solution, in most cases, for networking.

If you disagree with any of the points I've made please let me know and even after 10 years I'm always learning how new services or features change the possibilities.

For more information on transit gateways take a look at the AWS Documentation.

What it's like as an AWS Certification Subject Matter Expert (SME)

Robin Ford — Tue, 28 Jun 2022 19:53:26 +0000

So you may have seen some of the AWS Certification SME badges on peoples profiles and asked what is an SME?, how do I become one?, and why should I become one?

The goal of this post is to give you some insight into the programme, how and why you should apply, and what I've gained from participating.

What is an SME?

So the first real question is what is an AWS Certification Subject Matter Expert (SME)?

The main role of an SME is to assist in the creation and validation of exam materials. This entails two types of workshops/sessions.

The first is participating in exam writing workshops for a particular exam. This is where SMEs are tasked with the creation of questions linked to the domains of the exam. This is probably the biggest piece of work and the most frequent type of workshop I have been invited to. The workshops historically have been run in person over 3 days at an AWS facility. These seem to be making a comeback so hopefully as travel restrictions relax globally there will be more of these.

During Covid-19 when restriction prevented in person events the workshops moved on line. As a result there were 2 version. One was a week workshop with daily check in and writing exams in our own time with conversations on Chime. The second version was the same time commitment/question expectation but spread over a few weeks.

I personally liked the multi-week option as it could fit around other commitments with out taking time out from work projects. In addition, it meant I could spend a little more time to craft my questions.

The second type of sessions are items reviews. Sometimes these are added to the exam writing workshops but I have been invited to separate sessions just focused on this. In these sessions you look at questions others have written and decided if they are well written and should be passed to the next stage of the process. These are interesting because you get to see questions others have written and can learn skills on how to write better questions.

Applying to the Programme

Pre-Requisites 📃
There is really only 1 pre-requisite to applying to the SME programme, and that is you must hold a certification in the area you want to engage in. For example, if you select that you want to participate in the machine learning curriculum, you need to hold the machine learning certification. I suspect however that if you hold a professional level certification is covers you if you want to participate in associate level events.

Training 🏫
Either before you apply, after you reply, or after select you need to take the AWS Certification Subject Matter Expert (SME) Item Writing course on skillbuilder.aws.

This course goes through what is a good exam question, both in terms of structure and level, and how to align questions to the exam criteria.

Applying 📨
The application process is simple. All you need to do is head over to the AWS Certification SME site and request an application.

The application form is straight forwards and asks you information about your AWS certifications, industry experience and what areas you would like to participate in.

For each area of interest you are then presented with an essay type question you need to answer in a few hundred words. The idea of this is to show how you have demonstrated a level of usage in your area of expertise over and above just achieving the certification.

So what's in it for you?

In my mind there are several reasons to apply to the programme, over an above the desire to help contribute to the certification process.

Firstly is the recognition 🏅
Its always nice to get recognition for your knowledge and I've found that the SME programme has been a great way to share and show that knowledge. I think it's one of the great things I love about the whole AWS community ethos, and that is the ability to engage in all aspects, and being recognized for that.

For me, that recognition is more than just having that badge on LinkedIn or an eMail signature. I find that it gets a conversation started about my involvement in the certification process, and if you ask people who know me I do love talking about all things AWS.

Secondly is learning 🧑‍🎓
One of the big things I have found from the workshops is not only am I writing questions and contributing to the curriculum, but I am also learning as I go along.

The first big learning experience for me is how to be more concise whilst putting over technical topics. One thing the exam aims to be is fair, and ensuring concise wording with out technical or regional jargon is one way they do this. As a result I've had to learn to write things that can be understood globally and without bias to experiences.

I am also learning new things about service and configuration I didn't know. There have been several times I have queried if a question/answer is valid and had to go and search the docs only to find it was released months ago. One of the great advantage is you are also in a room/call with lots of other SMEs who can also help explain and give examples of how they have used a service or feature.

Finally is SWAG 😁
Now, who doesn't like a bit of AWS swag?

By participating in the SME progamme you gain points for various activities such as an exam workshop. And as any fan of British gameshows will know "Points make Prizes", or in this case swag.

With a dedicated swag store you can trade in your points for a wide range of goodies. As expected, there is a large amount of SME branded merchandise such as polo-shirts, jackets, travel mugs etc, but you can also trade in for Exam Vouchers and Amazon products such as Echos and Kindles.

Would I recommend being an SME?
Whole heartedly.

I've found it a great experience engaging with people from both within and outside of AWS. Each one of my fellow SME's has shared with me their experiences and knowledge and that have helped me not only become a better SME but a better engineer all round.

To find out more about the application and apply head over to the AWS SME site.

Creating a Well-Architected VPC

Robin Ford — Sat, 14 May 2022 21:53:04 +0000

So this is the first in my posts walking through how to deploy a solution in AWS. Hopefully you find it useful as VPCs are the foundation for private and secure networking in AWS and an area many struggle. This guide is designed to ensure that your VPC deployments can be Well-Architected and provide a base level of network security but is only one option for VPC layout. While it will meet a vast majority of workloads you might want to review the structure and reduce, or increase, the number of subnets as well as other components.

I will not go into CloudFormation templates structure, unless required, or deployment method. However, all templates are designed to be able to manually uploaded via the console and applied consecutively or the individual template you wish to deploy.

All code for this project can be found at my git organisation under the VPC 101 repository myawsrocks/vpc101.

What you'll create:

This solution will create the following topology with the option for a single (reduced redundancy) NAT Gateway. In addition we will enable encrypted VPC Flow logs that are stored in CloudWatch and S3. Generally the S3 bucket would be in a central security account but for our purpose we will create and secure it in the same account.

Well go through the each of the following steps with each step representing a unique CloudFormation template that you can deploy from my repo.

Create an empty VPC with an IPv4 CIDR range.
Create an IPv6 CIDR range and associate to the VPC.
Create 9 private subnets with IPv4 and IPv6 IP ranges allocated.
Create private route tables and associate to the private subnets.
Enable VPC Flow Logs and send to encrypted S3 Bucket and CloudWatch Log Group:
Deploy VPC Endpoints for common AWS Services.
Create public subnets and internet access.
Option for single NAT Gateway

Creating the base VPC

CloudFormation template: vpc_1.yaml

Unlike with the VPC wizard in the console, creating a VPC in code or via CLI only creates an empty VPC with the IP4 range (CIDR) you provide. In my example we'll use the default in the template of 172.16.0.0/20 but you can replace this with any range. Unless you have a large public IP range you want to consume I'd suggest an RCF1918 range (Private IPv4) which only has minor AWS restrictions in relation to secondary IP ranges (see the docs - working with vpcs).

Remember: If you are going to be connecting to other VPC's via VPC Peering or a Transit Gateway you need to ensure you do not assign the same IP range to multiple VPCs.

Once the CloudFormation Stack has completed you will be able to view your VPC in the console and see the attached IPv4 range.

Creating an IPv6 range and associate to VPC

CloudFormation template: vpc_2.yaml

Although IPv6 adoption is low, I promote its use as it offers many benefits if you are able to adopt it. Assigning and using an IPv6 range is simple and most services in AWS are able to run IPv4 and IPv6 and some services can now run IPv6 only.

As with IPv4, while you can use your own public IPv6 range in AWS, unless you have requirements to advertise these internally or externally it is simpler to utilize an AWS IPv6 range. These are public ranges so globally unique both between VPCs and on the internet.

Updating the stack with IPv6 should only take a minute or two and once complete you will be able to see the assigned AWS IPv6 range in both the CloudFormation Stack outputs and in the VPC configuration in the console.

Creating private subnets

CloudFormation template: vpc_3.yaml

So we now have our VPC with both an IPv4 and IPv6 CIDR attached. However we have no where to place resources. In order to that we need to create subnets within the VPC. Subnets are linked to Availability Zones and are a small segment of the larger VPC IP ranges. Take a look at the VPC User Guide - Subnets for more details on subnetting.

Although there are many ways to create/design your subnets in a VPC, I tend to lean towards splitting in line with the three tier architecture model (wiki article) as this tends to be the most common and flexible way of allocating space.

Note: It is not possible to resize a subnet one created. It has to be deleted and then recreated.

Based on using 3 Availability Zones for resiliency we will create 3 subnets in each tier giving us a total of 9 private subnets. For each of these subnets we will allocate a IPv4 and IPv6 range.

Below are the IP allocations in the template (available in excel in repo). Generally the application/logic tier tends to have more components so these subnets are sized larger that the presentation and data tiers. I have split across application layer first then allocated to each Availability Zone. This allows easier security for items such as Network Access Control Lists (NACLs) or on-premise firewalls as if required an individual tier can be allowed.

For IPv6 because this range is allocated by AWS we have to use calculation to allocate the subnet part of the IP range and add to the VPC allocation. In the IP allocation file I use a dummy IPv6 VPC range of fc00:1234:5678:9a::/56 to show how the subnets ranges are allocated. As you can see we assign the last 2 digits of the subnet range (in bold) to the provided VPC CIDR.

Note: Apart from the US West (Northern California) region all regions have 3 AZs for new customers

Once complete you will be able to see all the subnets and their associated ranges in both the CloudFormation Stack outputs and in the console. You will notice in the CloudFormation Stack outputs there are also some extra keys. These are to enable easy reference via other stacks, if you use CloudFormation Nested Stack, or resources if you add to this template.

Creating routing

CloudFormation template: vpc_4.yaml

So we now have our VPC and Subnets and can start creating resources if we wish. The resources would be able to talk to each other using the "Default Route Table" that is created when creating a VPC. However, this route table can not then be manipulated in code if we wish to modify those routes. In order to be able to create and amend routes we need to define route tables in code and attach it to the subnets we have created.

At this point we will only create a single private route table as it will only define a "local" route that can be used by all subnet to communicate between themselves. As the need for different routes per subnet/AZ is required we will create separate route tables and attach to the subnets required replacing this single route table.

One completed you will be able to see the new route table in the VPC console along with the attachments to our 9 subnets.

Enable VPC Flow Logs

CloudFormation template: vpc_5.yaml

One key item in becoming Well-Architected is ensuring security and operations are included in what we deploy. A key part for both of these items is knowing what is happening (observability) in the VPC. The first, and arguably most valuable, component in this is enabling VPC Flow Logs. Flow Logs can be enabled at the VPC, Subnet or Interface level but by enabling at the VPC level we ensure that logging is enabled on all traffic with out the need to be configured on subnets or components as they are deployed.

We will publish the logs to CloudWatch Logs, to enable easy analysis, as well as S3 for archive. Both CloudWatch Logs and S3 are encrypted using a Customer Managed Key (CMK) provided by KMS that we will create in the template.

The template defaults retention period for CloudWatch Log to 14 days and S3 to 365 days but as they are parameters these can easily be changed.

In an organization, I would expect the S3 bucket to be created in a separate account that is used purely for logging to ensure segregation of duties.

For more information on VPC Flow Logs take a look in the VPC User Guide.

Once the stack update is complete you will be able to see the flow log configuration against the VPC in the console. Usefully it also links to the CloudWatch Log Group and the S3 bucket where the logs are stored.

If you look at the CloudFormation Stack outputs you will also see details of the IAM role and Policy as well as the KMS key id.

Deploy VPC Endpoints

CloudFormation template: vpc_6.yaml

At present there is no access to components outside of the VPC. While this is extremely secure it does limit functionality. The easiest way to access AWS services is by using VPC Endpoints. These use the AWS network to connect to the services and are more secure than going over the internet. In addition endpoints have a security group and can have a resource policy to restrict access if required.

If you want to understand more about the benefits of endpoints I did a blog post on the subject here.

We will deploy all the VPC endpoints that would allow you to assign an IAM Role to an EC2 instance to interact with AWS if required, remote in to EC2 instances via Systems Manager - Session Manager, and use the CloudWatch Agent on EC2 instances to output OS metrics. Whilst there are over 170 service that support endpoints, these should give you enough to understand how they work and how you can provision them for your workloads.

As these are Interface Endpoints we have to create the network interface in a subnet. For these interfaces I will create them in the Application Subnet as this is primarily where the workloads using them will be created. If you chose to create a public subnet this would also be an ideal place to put all VPC endpoints.

Once the endpoints are created you can see them in the console and also the IP addresses of the interface within each subnet.

Now, it might seem that this would then take a lot to be able to use these endpoints. However, if you take a look at the Details tab of the endpoint you will see DNS names for the endpoint.

Again, the list of DNS looks daunting and while they can be used and will resolve to the endpoint IPs the item that is more critical is the Private DNS names. This will only resolve in the VPC but the name is the same as the public name for the service. So in this example the external resolution for the STS server sts.eu-west-2.amazonaws.com is 52.94.48.43. Within the VPC resources will use the private DNS resolution which would then resolve to the IPs in the VPC, in our case 172.16.4.206, 172.16.5.244, and 172.16.6.244.

Deploy Internet Access

CloudFormation template: vpc_7.yaml

So we now have a VPC with Subnet and Routing along with access to AWS service that we might require and a way to remote in to EC2 instances. While this might be sufficient for some solutions such as data processing where the output is then in an S3 bucket or sent via SNS or some other servers it will not work for all workloads. In addition some workloads will deploy services that will need access to the internet to either download and other workloads might need to be accessible in order to serve customers.

To enable this and increase security we will create separate public subnets to host internet facing components. For these components we will deploy an Internet Gateway with direct access.

For items in the private subnets that need internet access we will need to deploy a NAT Gateway for IPv4 traffic and an Egress Only Internet Gateway for IPv6 traffic. In addition we will need to create new route tables to allow subnets to access the relevant gateways. As NAT Gateways now support internet and private NAT options we have to specify that our gateway will be for public connectivity.

As mentioned above we will also move the VPC Endpoints to the Public Subnet.

Reduce NAT Gateway Costs

CloudFormation template: vpc_8.yaml

While most components have no charge NAT Gateways cost between 4¢ and 7¢ an hour dependent on the region you deploy to. While this might not seem much, as we are deploying in 3AZs (good practice) this is up to 21¢ and hour.
So in a month with 31 days (744 hours) that equates to $157 just to have them provisioned.

As such we will add an option to reduce this for workloads that do not need it. Using a CloudFormation Conditions we can decide which resources to build based on a input parameter. In our case we will have a parameter (ReducedNAT) that will then set whether we want a SingleNAT or MultiNAT solution. We have set the parameter input to only allow either Yes or No as an option to ensure that we always meet one of the 2 conditions.

If we select Yes we then only create a single NAT Gateway and private route table with all private subnets using the same route table pointing to the single NAT Gateway.

Network Access Control Lists (NACLs)

CloudFormation template: vpc_9.yaml

To improve security at the network level, we are going to create 4 NACLs to control traffic between the Subnets as well as in/out of the VPC. We will create NACLs as follows and associate to the relevant subnets:

Public Subnets NACL
Presentation Subnets NACL
Application Subnets NACL
Data Subnets NACL

The diagram below shows the flows that will be enabled by the NACLs. I use common rule numbers across NACLs as I believe it is simpler to be able to match them up. For example rule 1xx is used for outbound internet IPv4 traffic and it's reply traffic in all NACLs. I also group IPv4 and IPv6 so NACL entries together. For example rule 200,201,202, all provide the same access between the presentation and applications subnets.

In the Public Subnet NACL firstly we are going to create rules which will enable inbound traffic from the internet on port 443 (HTTPS) and its reply traffic on ports 1024 to 65535 (Ephemeral Ports) using IPv4 (rule10) or IPv6 (rule11). This will allow access to resources we might host in the public subnet such as load balancers or API Gateways, but only over the secure HTTPS protocol. Secondly we are going to create rules that allows all VPC components to access the public internet on port 443 (HTTPS) and the reply traffic on ports 1024 to 65535 using IPv4 (rule100) or IPv6 (rule105). This enables devices in any subnet to talk to the internet over HTTPS and the reply traffic. For those devices in the Private Subnets this is via the NAT devices.

In the Presentation Subnet NACL firstly we are going create the inbound internet rules that will allow devices in the public subnets to talk to the presentations subnet on port 443 and the reply traffic on ports 1024 to 65535 using IPv4 (rule10, 11, 12) or IPv6 (rule15, 16, 17). This will allow devices such as load balancers that are routing internet traffic to talk to components in the presentation subnet only over HTTPS matching rules 10 and 11 in the Public Subnet NACL. Secondly we are going to create rules that correspond to rules 100 and 105 in the Public Subnet NACL for outbound internet traffic. For IPv4 this is rules 100, 101, 102, and for IPv6 this is rules 105, 106, 107. Finally we are going to allow devices in the presentation subnets to talk to the application subnet on ports 1 to 1023 (Common Ports) and the reply traffic on ports 1024 to 65535 using IPv4 (rule200, 201, 202) or IPv6 (rule205, 206, 207). This will allow components such as web servers to talk to components in the application subnet and the reply traffic.

In the Application Subnet NACL firstly we are going to create the same outbound internet rules as in the presentation subnet (100, 101, 102, 105, 106, 107) that correspond to rules in the Public Subnet NACL for internet traffic. Secondly we are then going to create rules 200, 201, 202, 205, 206, and 207, which correspond to the same rules in the Presentation Subnet NACL to allow traffic between the presentation and application subnets. Finally we are going to allow devices in the application subnets to talk to the data subnet on ports 1 to 1023 and the reply traffic on ports 1024 to 65535 using IPv4 (rule300, 301, 302) or IPv6 (rule305, 306, 307). This will allow components such as application servers to talk to components in the application subnet such as databased and the reply traffic.

In the Data Subnet NACL, again we are firstly going to create rules 100, 101, 102, 105, 106, 107 for outbound internet traffic. that correspond to rules in the Public Subnet NACL. Secondly we are then going to create rules 300, 301, 302, 305, 306, and 307 that correspond to the same rules in the Application Subnet NACL to allow traffic between the application and data subnets.

Once deployed take a look at the NACLs, you should have the following rules in each list. Note, this relatively open but we would further control access with Security Groups around resources.

What's Next?

Clean Up

The cost of this solution according to the AWS Pricing Calculator is $126 for the London Region (see estimate). While that's not much it would equate to about $1,500 for the year. The reduced NAT option's estimate for the same region is $54/month or $650 a year. So, unless you need to use it I'd suggest deleting the stack once you've had a look at how a VPC and it's components work.

Future Updates

If you take a look at the Git repository for this post you can see the current roadmap. Once those items are created I will come back and update this post. At present the items I have plans for are:

IPv6 only subnets and networking devices.
Option for no NAT

If you want to contribute or recommend updates on the solution please feel free to either leave a suggestion here or create a pull request on GitHub.

myawsrocks / vpc101

Code to create a Well-Architected VPC

myaws.rocks vpc 101

How to build a VPC demo

Report Bug · Request Feature

Table of Contents

About The Project

This solution walks through the building and securing of the following VPC.
At the end of the deployment you will have a VPC spanning multiple Availability Zones (AZ) with both IPv4 and IPv6 ranges.

###Components built with the solution are:

Create an empty VPC with an IPv4 CIDR range - vpc_1.yaml
Create an IPv6 CIDR range and associate to the VPC - vpc_2.yaml
Create 9 private subnets with IPv4 and IPv6 IP ranges allocated: - vpc_3.yaml
- 3x Presentation
- 3x Application
- 3x Data
Create a private route tables and associate to the private subnets - vpc_4.yaml
Enable VPC Flow Logs and send to encrypted S3 Bucket and encrypted CloudWatch Log Group - vpc_5.yaml
Deploy VPC Endpoints for common AWS…

View on GitHub

Cheat Sheet or Quick Reference? What's in the name?

Robin Ford — Thu, 14 Apr 2022 18:02:17 +0000

Cheat Sheet, Quick Reference, Key Facts, Quick Start Guide, the list of names for similar types of reference documents are as varied as the topics they cover. So why do Cheat Sheets seem to be the most popular name and format?
In this post I'll give my view on the use of cheat sheets, and the term in general, and why I think they've become popular. I'll also explain why I hate them (cheat sheets) and the misuse of term.

The term cheat sheet originally comes from those tiny bits of paper crammed with the most useful bits of information needed to pass an exam. As the name suggested they were used to cheat and gave just enough information to prompt the individual's memory or a formula they never learnt.
And this is why I think they have become popular in recent years with the push for professional certifications. I believe that both writers of them and their audience see them as enough information to bluff your way through a given scenario or certification. In the case of certification it seems that many people now see the goal as being to learn the minimum possible to pass. As such a google search for "aws solution architect cheat sheet" returns over 166,000 results. The more worrying fact for me is that out of the 10 results on the first page 4 were from training/learning websites. These are clearly aimed at people who want the easy way to pass a certification with minimal effort.

So, you might be asking yourself, am I against all these sort of documents?
The answer is a resounding NO! I just don't like cheat sheets.
I often use quick reference guides, especially for things I am learning or don't use very often. I also refer to documentation more often than people think. My main issue is what cheat sheets specifically are aimed at.

So why am I against cheat sheets?
Well first of all I am against cheating as I believe it devalues the certifications and qualifications. This is inherently unfair to those who have worked hard at gaining the knowledge and expertise. This is one reason I report any suspected cheat sites/adverts or sites offering "exam dumps" which is against most exam providers terms and conditions.
The second reason is that I think it gives a false perspective of the certifications. I see people saying they have got a certification after a few months studying and how can they get a job with those skills. For me the certification shows you have learnt and demonstrated the skills so should be able to get a related job or have a related job anyway. Many of the certifications, especially AWS professional and speciality, are much easier to pass with experience and expected duration of experience is often given in the exam requirements.
Finally, I think it gives a bad image of reference documents. By calling documents cheat sheets it implies they shouldn't be used if you know what your doing. This shouldn't be the case. Skills should be more than just the ability to remember the format of a command you use only in rare cases or unique scenarios. Skills shoild be based on understanding whats needed or gone wrong. Anything that can be looked up in documents is not skills or experience, know which command to when is more important than remembering the structure.

So what should we call the documents that are not just for passing exams?
Well for me these general come in 2 forms based on their purpose.
Firstly is quick start guides. These documents are often a page or two long and are designed to get you up and running on a specific technology. Usualy a step by step guide they walk through the steps needed to get up to speed on something or get a solution ready for use. They offer explanations where needed but often refer to other documents for more information and are guide as a jump start to get going. These are often common in applications, such as for an IDE's showing how to connect to GitHub, or technical solutions such as how to spin up a basic VPC.
Second I think are reference guides or command guides. These can be simple one or two pages such the Git Cheat Sheet (yes I think this should be renamed) or the more complete reference guides such as AWS CloudFormation Guide that shows the structure for every resource type. Ether way they cover the key information needed on a topic. They can be written with learning in mind such as the Getting Started with AWS guide which talks through the basics of AWS, or more detailed documents that talk through specific solutions and deployments.

So why does it matter, and why write about it, in the grand scale of things?
Well I think in an industry where we are trying to be more inclusive, and encourage more people from under-represented backgrounds, vocabulary matters. For me the name cheat sheet has negative meanings. It implies that you are looking at something you should know. This can cause people to feel like they should not use them, when in fact they should be at everyone's' fingertips.
In Linux operating systems, most reference guides such as command structure are build into the program through either man or -help to ensure you can complete a task. So if these items are in some programs/commands why not have a pdf or web links to similar close to hand.
For me as mentioned earlier the skill and expertise is knowing what components to build into a solution or what to look at when troubleshooting. Yes, some certifications require hands on labs, and in some cases they are closed book/no manuals. If you're a specialist then this can make sense. If I hire an experienced Cisco engineer or Kubernetes developer I would expect them to know the majority of the command for general operations. But would I expect them to know every possible command and make up of parameter flags, probably not.
Engineering, which most modern computer roles are, is a mindset and process driven skill. Give me someone with the right mindset and I can teach them commands.

As always welcome your view and experience with this.

Useful AWS Training Resources

Robin Ford — Fri, 24 Dec 2021 14:45:20 +0000

One thing I think is important in technology is to never stop learning. For me change in technology is like a river, it never stops flowing, and at times such as re:invent it can feel like a flood. Things are changing so rapidly if you don't keep up to date you will drown in all the announcements and changes.

Hopefully you have all heard about the skillbuilder.aws website that has over two and a half thousand digital courses in 16 languages. What you may not know is that there is a raft of other training material out there either direct from AWS or others that can help with your learning journey.

My goal here is not to provide a definitive list or endorse any resources but to provide a list of resources that are often hard to find or topics I think are important to know. Hopefully you'll find this list useful. It is not meant to be exhaustive but it you feel there is a site or channel I really should include let me know.

Web Sites

AWS managed sites.
As well as the skill builder website and service documentation there are 3 resources libraries on the AWS website as well as over 400 hands on tutorials.
- AWS Developer Centre
- AWS Architecture Centre
- AWS Builders Library
- AWS hands-on Tutorials

AWS then do a really good job at providing labs. A lot of these are not highly publicized, although can be found on Google, but instead referenced in immersion days, AWS events and the like. A useful source for hands-on with explanation aimed at all levels of knowledge and skills.

AWS workshops is the main site with 100's of workshops you can follow in areas ranging from astronomy to zero-trust.
AWSworkshop.io is focused on modernization workshops and currently has just over 50 labs to follow.
AWS Security Workshops as the name indicates is focused purely on security related labs and workshops.
AWS Well-Architected Labs has labs in the 5 pillars, hopefully sustainability will be added next year. The goal for these labs is to improve your posture in each area and become "well architected".
Control Tower Tools for those of you wanting to gain more insight to AWS Control Tower this is the place to be. It's based on the immersion day but has so much more including integration with ITSM tools and customizations.

If your going serverless, which you probably should at least be thinking about, then Serverless Land is the place to head to. It is a hub of information and contains it's own content, such as over 100 design pattern templates, in addition to pointing to to other resources such as AWS Blogs and videos.

Other sites.
acloudguru and cloudacademy are two of the more popular learning platforms for both individuals and organizations. Although aiming at paid members they do offer an incredible amount for free.
codecademy is focused purely on code skills, but again has a free tier as well as paid options for individuals and businesses.
cloudonaut.io is two brothers with a passion for AWS. They have code examples, videos and a podcast.

YouTube Channels

As you would expect there is a lot of content on YouTube while the quality of some content is dubious there is a lot of good content both official from AWS or partners as well as individuals.

AWS Channels
Amazon Web Services is the main channel for AWS and is a mixture of PR and Technical content.
AWS Events Channel focuses on AWS events such as re:Invent and Summits. It has both keynotes and breakouts as well a AWS on Air.
AWS Online Tech Talks features talks with AWS experts in a wide range of topic.
AWS Elemental Tech is one of a couple of industry focused channels looking in this case at the Media and Entertainment industry.
AWS Public Sector is the other industry channel for Public Sector clients.

There are a number of other channels including regional channels for native language resources so be sure to look for related channels.

Other Channels
As well as the AWS channels there are numerous other channels that I often refer to. I've split these into 3 categories.

Training Providers
acloudguru is the channel for the acloud.guru learning site. They offer a lot for free and not just AWS but technologies such as Kubernettes and Networking.
Linux Academy although acquired by acloud.guru there is a lot of previous content that is very useful.
Cloudacademy is the another training provider that puts a lot of information on YouTube for free and well worth a look.

Vendors
UbuntuOS and Redhat both offer lots of content for use of their OS's.
Kubernetes Community channel is jam backed with advice and resources to get you started and progress your container knowledge.
Although you might only use Cisco on premise they offer a lot of networking fundamental videos.
Git Hub and Git Lab both offer both product resources and general DevOps advice.

Others
Network Chuck is not only just for networking but covers a range of topics in an relaxed and easy to follow manner.

How to Well-Architect network connectivity to AWS services.

Robin Ford — Sun, 05 Dec 2021 17:32:44 +0000

So a few weeks ago I was asked what my strategy was for accessing internal AWS resources such as S3, DynamoDB etc. where it is possible to access over VPC endpoints as well as the internet. My first point of reference for them was the great map by Corey Quinn, Chief Cloud Economist at The Duckbill Group which looks at the costs for moving data around AWS.

In my mind I also had thoughts around ease of use for developers, security and speed in addition to the cost that lead me to a single solution. However, when discussing with the individual I realize that I'd never formally assessed my view. So that's what I hope to achieve here. I'll look at each option and compare against the Well-Architected Framework and see how my initial thoughts hold up.

So what are the various options? I personally see 3 common patterns, and while there are probably more, these are the one's I'll focus on:

Direct internet via Internet Gateway
Internet access via NAT Gateway
Access via VPC Endpoint

I'm also not going to look at centralized solutions. For me there is a significant impact on cost, security and operations for these solutions and unless a really justifiable reason I would not recommend them.

When calculating costs I will be basing calculations on transferring 500GB to/from AWS S3 in 4 different regions (N. Virginia, Ireland, London and Sydney) This should give enough comparison as most costs are higher outside N. Virginia.

Direct Internet Connectivity ★★★★☆☆☆☆☆☆

So this is the simplest solution, and 15yrs ago was the only option for EC2 to access anything. With implementation of VPCs in 2009 this was still the simplest solution and any other access had to be via a EC2 based proxy or NAT solution.

Direct Internet is also the cheapest solution at $5 per 500GB with no change in costs across different regions. As such there is no arguing the 2 stars for Cost Optimization.

Arguably it is the most reliable and performant of the solutions as there are no additional devices in the network path to fail or impact traffic. However it is subject to general internet congestions and someone else's popular website could slow down API calls or transfers from AWS services. As such each of these pillars is given a single star in my view.

However, it is the most insecure. All instances would have to be in a public subnet and have a public IP. This is a huge attack vector for the environment. I have heard in the past that with no addition controls EC2 instances with public IPs will be compromised in less than 5 minutes. This means that the management of instances becomes very onerous and increases the operational controls and efforts required to maintain a secure and functioning.

In my view security is so fundamental that it carries more weight than other pillars. Is a solution is not secure it doesn't really matter to some extent if it is high performance and resilient. So in my view lack of security drops 2 stars along with dropping 2 stars for operational overheads.

This gives Direct Internet Access a score of 4 out of 10 stars.

Internet via NAT Gateway ★★★★★☆☆☆☆☆

In 2015 when NAT Gateways were introduced they were seen as a major step for networking. Self built NAT solutions that relied on alarms and user data scripts could be removed and a fully managed NAT solutions could be dropped in. Although a small cost difference compared to a self-managed solution, the improvements in operations meant most large organizations migrated quickly to NAT Gateways.

This improved the security of access to the Internet and AWS services dramatically by removing the need for a public IP. It also removed the self managed instances some organizations were using. However there is still little concern as NAT Gateways do not offer any content filtering. However NAT Gateways gets a respectable 1 star for the performance pillar.

So what's the impact on cost? Based on the 5 scenarios, NAT Gateways come in 3rd with a cost of $70 a month for 500GB of data. While not a significant amount if you are transferring a lot of data between your VPC and AWS services this could soon mount up so something to be aware of. Another thing to be aware of is that NAT gateways are multi-AZ but have a single route-table entry. So in some scenarios traffic might traverse AZs. This is a 10% overhead on charges and again while not significant at low data volumes if you are moving a petabyte of data that would be approximately $2000 extra. So for the cost optimization pillar NAT Gateway gets 1 star.

So what about reliability and performance. Although a managed service improving these areas over a self managed solution, as with direct internet traffic to AWS services through a NAT gateway go over the internet and can be subject to variance in performance. So again a each of these pillars is given a single star.

Finally operations. Well in this areas things are drastically improved. There is a reduction in operational management of security but still a lot that needs to be done to manage access to services. As such for me NAT gateways get 1 start for Operational Excellence.

This gives Internet access via NAT Gateway a respectable score of 5 out of 10 stars.

Direct via VPC Endpoint ★★★★★★★★★☆

2015 was a good year for networking and 7 months before NAT Gateways we released AWS released the VPC Endpoint service. Initially only for S3, 108 services as of 5th December 2021, it allowed for traffic to be routed over AWS' private network rather than the internet. As a managed service it simplified routing to S3 as well a host of other functionality.

For example, in addition to private routing, 86 of the current 108 services supporting VPC endpoints also support VPC endpoint policies. This means that access to AWS Services can be controlled at the endpoint level as well as the IAM level. For some services such as S3 VPC endpoints can also be used in the resource policy. This restricts access to the resource from specific VPC endpoints. For me this increased security pushes the security pillar to 2 stars as resource, principle and route can all have policies applied ensuring the highest level of control.

For performance as traffic is now routed over AWS' private network there is less impact from other customers or general internet loads. In addition each VPC Endpoint has 1oGbps capacity with bursting to 40Gbps. If we compare this to NAT Gateway that has a throughput of 5Gbps with bursting to 45Gbps this is a significant improvement as each service now has dedicated bandwidth which is not only more performant but more consistent. For me this pushed performance pillar and reliability pillar both to 2 stars.

For operations, whether traditional or DevOps, I see great benefits in VPC endpoints. They can be deployed along with a VPC to provide access to multiple AWS services and can remove the need for any internet access. In addition if provisioned in the same pipeline as the resources being access they can be referenced in security policies. There are then overheads for managing all these policies and ensuring they meet wider standards and for that reason I think VPC endpoints drops a star giving it 1 star in the operational excellence pillar.

So what's the impact on cost? Well surprisingly it is cheaper than a NAT Gateway. For 500GB of data a month the average cost is $13. Not a huge increase from the $5 for direct internet access and significantly cheaper than the $70 for NAT Gateway. Taking into account the increase functionality and performance this is a fair trade-off and for me gives VPC endpoints 2 stars for Cost Optimization.

All this gives access via VPC Endpoints a well deserved score of 9 out of 10 stars.

Conclusion

So I am glad that my initial recommendation of VPC endpoints stood up to scrutiny, even if my own. Many of my recommendations are based on using AWS services for a significant amount of time and seeing incremental changes. As such I sometimes don't always dig into why I am making a recommendation.

I've enjoyed writing this so might do a few more of this type of post so please let me know if you found it useful, interesting or if you disagree with some of the views.