DEV Community: Walid BATTOU

The Hidden Cost of Paying Twice for Inter-AZ Network Traffic in AWS

Walid BATTOU — Mon, 25 Nov 2024 00:49:16 +0000

Today, it's well known that network cost on AWS can be very expensive.

I would like to explain the hidden cost of Inter-AZs network traffic when an AWS managed service (EC2) communicates with other AWS services like FSx, Amazon MQ, Load Balancer and Fargate.

We have several categories of network costs. This article will not duplicate content from existing articles like below:

Starting with Cost Explorer

How to check the cost of network traffic between Availability Zones?!

Filter by "API operation": InterZone-In and InterZone-Out

If you group by Service, these operations will be under "EC2-Other".
If you group by Usage type, these operations will be under "Region-DataTransfer-Regional-Bytes".

Most of your Inter-AZs traffic related to EC2 instances appears under the EC2-Other service category.

Let's filter by this Usage type:

Filter by "Region-DataTransfer-Regional-Bytes"
Group by "Service"

What we can observe?

The "API operation" InterZone-In, InterZone-Out are under the service "EC2-Other" only and the Usage type "Region-DataTransfer-Regional-Bytes".
Under the Usage type "Region-DataTransfer-Regional-Bytes" we can have a lot of services, but what we have behind it?

Let's take an example with FSx

Filter by "Region-DataTransfer-Regional-Bytes"
Filter by Service "FSx"
Group by "API Operation"

Conclusion
For the FSx service, we are paying for the Usage type named ""Region-DataTransfer-Regional-Bytes" identified with the "API operation" "CreateFileSystem:Lustre". With Cost Explorer, we are not able to know if this traffic is related to Inter AZs, traffic Out of AWS, Inter Regions or other. We need to use the CUR(Cost & Usage Report) to understand what we have behind it.

Deep Dive into the CUR (CUDOS dashboard version 5.4)

If you're working on AWS and not aware of CUDOS and other dashboards provided by AWS, check this link.

Let's check the tab "Data Transfer & Networking"
On this tab, we have multiple choice to dig into the detail related to network cost. I checked the box "Inter AZ":

For clarity, I've added a column description to the 'Data Transfer Details Usage and Cost' visualization.

We find for FSx what we found in Cost Explorer:

The description clarifies that with FSx, we need to pay for Inter-AZs network traffic between FSx and:

EC2
an EIP (Elastic IP)
a Load Balancer

For my example, we need to pay twice the Inter AZs network traffic between an EC2 and a file system on FSx:

My first question when I discovered that was:

Is it mentioned in the AWS documentation?

The answer is yes for FSx but sometimes not clear for other services like Fargate.

The description in the CUR is the same one for any managed services with inter AZs traffic:

Take Away

Use CUDOS to check your Network Traffic on AWS
Keep resources within the same AZ (when it's possible)

AWS VPC Endpoint is JUST a FinOps topic!?

Walid BATTOU — Mon, 04 Mar 2024 19:39:31 +0000

This article follows up on the previous one, where we attempted to address the following question:
AWS VPC Endpoint is NOT a security topic!?

Clarification about my last article

Even if we demonstrate that the VPC Endpoint is not focused on security or performance because the traffic stays on the AWS backbone. In a real-world use case, we can have compliance requirements.

If you have compliance requirements such as HIPPA(health) or PCI-DSS(Bank) it is not feasible to tell the auditor "Hey, do not check the AWS documentation, VPC Endpoint is not about security, I checked it."

However, if you have no compliance requirements, be frugal! You will have the options described below.

Why is it a topic related to cost?

How VPC endpoints are billed?
They are billed based on the elastic network interfaces(ENI) per availability zone behind the endpoint. We can deploy it across 1 to 3 availability zones based on your infrastructure requirements. Of course, it is recommended to have at least two for high availability.

Split VPC endpoints into two categories!

You will find the list of AWS services that support VPC endpoints.

I split this list into two categories:

Endpoints related to an application (eg. API gateway, Athena).
Endpoints related to the infrastructure (eg. CloudWatch, SSM).

Based on that, we can start with the costs incurred when using or not using an "infrastructure" VPC endpoint like CloudWatch Log for example (price for eu-west-1).

Let's focus on the CloudWatchLog VPC Endpoint

Consider a scenario where we have 10 VPCs, each hosting multiple workloads that utilize CloudWatch Logs for application logging. On average, the monthly log volume is 1TB for all of them.

We have 3 options:

Without VPC endpoint (with NAT Gateway).
With VPC Endpoint (one per VPC).
With Centralized VPC Endpoint.

Cost without VPC Endpoint (with NAT Gateway)

What we pay in this case:

1TB(DataProcessing per month)*0.048=$48/month
Total: $48/month

Cost with VPC Endpoint

What we pay in this case:

VPCEndpoint/month for 3 AZs for each VPC:
- $0.011*730(hours in month)*3=$~24/month
VPCEndpoint data processed:
- $0.01*1000=$10/month
Total for 10VPCs: (24*10)+10=$250/month

Cost with Centralized Endpoint

On AWS, we usually have multiple accounts and VPCs. The general practice is to have a network account with a transit gateway in it. For more information, please refer to this documentation.

What we pay in this case:

VPCEndpoint/month for 3 AZs:
- $0.011*730(hours in month)*3=$~24/month
VPCEndpoint data processed:
- $0.01*1000=10$/month
Transit Gateway data processed:
- $0.02*1000=20$/month
Total: $54/month

Conclusion

With 1TB of data transfer for 10 VPCs. The best scenario is to not use the VPC endpoint and keep using the NAT Gateway.

The decision to utilize a VPC endpoint or not varies based on individual use cases. It's essential to evaluate the specific requirements and factors involved.

Make your own calculations!

When keeping a NAT Gateway is more cost-effective!?

When does it become more economical to use the centralized solution?

Based on the previous calculation, the following table shows that above approximately 1.3TB of DataTransfer, the centralized endpoint is more cost-effective. Below this amount, you can continue using the NAT Gateway.

In the next article, we will see how to deploy a centralized VPC Endpoint in an air-gapped environment on AWS.

AWS VPC Endpoint is NOT a security topic!?

Walid BATTOU — Fri, 16 Feb 2024 20:31:50 +0000

This article demystifies the security and performance implications of using AWS VPC Endpoints.

Spoiler: This is not about security for me.

Indeed, many companies have compliance constraints that mandate the implementation of an air-gapped environment.

Let's take a closer look at how VPC endpoints work.

Start with official AWS documentation

VPC Endpoint definition from AWS
VPC Endpoint doc

Takeaways from the docs:

Traffic stays on the AWS Network
Prevent availability risks or bandwidth constraints
Use AWS PrivateLink

AWS PrivateLink
If you're looking for VPC endpoints, you quickly come across the concept of AWS PrivateLink.
PrivateLink doc

Do we exit the AWS network without an endpoint?

AWS PrivateLink supports many AWS services. For this test, we will use SSM (System Manager).

Below are two diagrams showing the difference between having an endpoint and not having one.

Let's test it with the traceroute command!

Below are the results indicating the hops traversed from an EC2 instance to the SSM endpoint.

If we analyze the IP addresses of each hop. There is no IP outside of the AWS network. This is why we can conclude that we stay on the AWS network.

You can check below the reference of the RFC mentioned:

Network class E: RFC1112
CGNAT Shared Space: RFC6598

This last one is pretty interesting because AWS recommends using it in case of IP exhaustion for EKS.

What is intriguing here is the use of unusual CIDR blocks internally by AWS.

Are we experiencing better latency with the endpoint?

How to test this?

With a bash script of course :)

The following script measures the average latency of HTTP requests to the SSM endpoint over 30 minutes. It records timing metrics for each request and calculates the average latency from the results. Thanks to ChatGPT for the curl options.

end_time=$((SECONDS+1800)) 
while [ $SECONDS -lt $end_time ]; do
   curl -o /dev/null -s -w 'Connect: %{time_connect} TTFB: %{time_starttransfer} Total time: %{time_total}\n' https://ssm.eu-west-1.amazonaws.com;
   sleep 2; 
done >> output.log
count_lines=$(awk '{print $NF}' output.log | wc -l)
sum_lines=$(awk '{print $NF}' output.log | paste -sd+ - | bc)
average=$(echo "scale=4; $sum_lines / $count_lines" | bc)
echo "Average Latency: $average"

Result

WITHOUT endpoint --> ~55ms
WITH endpoint --> ~55ms

About bandwidth, based on a real-world use case using Athena with and without an endpoint, the performance remained the same.

Observation

Based on these tests we can conclude that network security and performance are the same with and without an endpoint.

AWS constructed its infrastructure using public DNS, but resolving a public IP doesn't necessarily mean traversing the internet to access it.

Example:
If you create a private load balancer on AWS, its DNS name is publicly resolvable.

Take Aways

Now, if using an endpoint is not focused on security or performance, which criteria can we rely on?

Let's delve into this topic in the next article of this series.

Detecting Incomplete AWS s3 Multipart Uploads: Storage Lens vs CUDOS

Walid BATTOU — Wed, 24 Jan 2024 00:14:59 +0000

The goal of this article is to explain the different ways provided by AWS to detect incomplete multipart uploads.

What is s3 multipart upload?

For objects bigger than 100MB, AWS provides a feature to chunk an object before uploading it to s3.

What is s3 incomplete multipart upload?

An incomplete multipart upload occurs when a user starts uploading a large object using multipart upload but does not finish the process.

The problem?

We need to pay for the incomplete parts stored in our s3 buckets.
AWS provides two ways to detect it:

Storage Lens
CUDOS dashboard

CUDOS

AWS provides CUDOS, a QuickSight dashboard part of the CID Framework(Cost Intelligence Dashboards) to dive deep into your cost usage based on the CUR(Cost & Usage Report) data.

One of the tabs of CUDOS is "Amazon s3". At the bottom of it, we have a view named "Buckets with Incomplete MultiPart Uploads Last 30 Days".

The source of this dashboard is the CUR (Cost Usage Report). When I saw this view, I wondered:
How can we retrieve this information from a cost perspective without access to S3 metrics?

Let's deep dive into the CUDOS template and what we have behind the view above.

From the analysis, if you click on the view, we will see the fields used in it.

Let's check what we have behind the "MPU Request Delta" calculated field.

In the CUR we have two operations:

InitiateMultipartUpload
CompleteMultipartUpload

The people who built this view must have thought like this:

"We cannot obtain the exact number of gigabytes of lost storage from the CUR. Nevertheless, we can still get an idea if we take the delta between 'Initiate' and 'Complete' requests we will know if some multipart uploads failed."

To know exactly the amount of gigabytes, we need to go through Storage Lens.

Storage Lens

For each of my articles, I tried to not duplicate existing content.
For s3 multipart upload with Storage Lens, you can check this blog post written by AWS employees.

Key take Aways from it:

Delegate Storage Lens administration to a member account in your AWS organization (Audit or Security account).
Create an organizational dashboard from the member account.

Observation

The default configuration for s3 buckets should include a lifecycle rule to automatically abort old incomplete multipart uploads. It would be good to just have a checkbox to activate it.

Lower your AWS CloudFront Bills with AWS PPA !

Walid BATTOU — Mon, 15 Jan 2024 21:20:16 +0000

AWS PPA (Private Pricing Addendum) is a discounted pricing model for a specific AWS service with high consumption (commitment per year). We are not talking about EDP, Savings plan, this is another discount.

There is not so much content about AWS PPA. I have practical experience with s3 and CloudFront PPA which is why I wanted to tackle this topic.

So let's deep dive into AWS PPA for CloudFront.

I checked the AWS documentation about this topic, and PPA is not mentioned in any of them.

However, when we are looking at the CloudFront pricing page, the following part corresponds to PPA I think:

As mentioned below, you can start talking to AWS about PPA when you have 10TB/month for Data Transfer Out to Internet.

The CloudFront Savings Bundle is another discount program. I won't explain this offer here.

What do we need to remember?

It is not possible to combine both PPA and CloudFront Savings Bundle. According to AWS, the PPA is more advantageous than the CloudFront Savings Bundle.

How does the Private Pricing Addendum apply to CloudFront?

You will find below to which CloudFront usage type the PPA is applied.

DataTransfer-Out-Bytes: From CloudFront to Internet (GB).
DataTransfer-Out-OBytes: From CloudFront to origin (GB).
Request-Tier1/2(HTTP/HTTPS): HTTP(S) GET and HEAD requests.
Request-HTTP(S)-Proxy: HTTP(S) DELETE, OPTIONS, PATCH, POST, and PUT requests from CloudFront to your origin.

Recommendations

AWS suggests a helpful tip to decrease your bills related to DOT (DataTransfer Out to Internet).

Because of the discount model of CloudFront, you can put it in front of any AWS services that transfer data to the internet. This way, we will use CloudFront as a simple proxy but with the lowest price for DOT.

HOWEVER, there is indeed some missing information regarding this recommendation.

What is missing on the CloudFront Pricing page?

As of writing these lines, the Pricing Example 2 from the documentation mentions this:

AWS forgot to mention two usage types here:

DataTransfer-Out-OBytes:from CloudFront to origin (GB).
Requests-HTTP(S)-Proxy: Forward from CloudFront to origin.

We can put CloudFront in front of any AWS services, and that is a good point. However, if your workload does not use any CloudFront features and uses it just as a proxy, you will have a high consumption of these two metrics.

IMPORTANT
Even with a PPA commitment, the usage type DataTransfer-Out-OBytes does not have PPA applied to it (cf. diagram above).

This point deserves a dedicated example on the pricing page!

Take Aways

Check the eligibility of your DataTransfer to Internet for PPA.
Do your own maths for DOT optimization with CloudFront.

Source:
https://aws.amazon.com/cloudfront/pricing/
https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/billing-and-usage-interpreting.html

Edge API Gateway Custom Domain with SAM: certificate in us-east-1 when using another region

Walid BATTOU — Thu, 28 Dec 2023 16:16:37 +0000

Purpose?

I want to deploy an edge-optimized API Gateway with a custom domain name.

The tool to build it?

My infrastructure background led me to CloudFormation when I started on AWS. Where some developers can start with SST, ServerlessFramework, or CDK.
In the serverless world, I use SAM (Serverless Application Model).

Problem?

To deploy an Edge API Gateway, it's mandatory to generate a certificate in us-east-1.
The problem with CloudFormation is that resources in one stack are deployed in a single region. We are not able to set another region for a specific resource.
Example: Create a Certificate with ACM (AWS Certificat Manager) in us-east-1.

How to solve it?

From this statement, we have multiple options to choose from:

To create a dedicated stack for the certificate in us-east-1, and use the certificate in our main stack, for me in eu-west-1.
To create a unique stack (in eu-west-1) and use CloudFormation custom resources to handle certificate creation in us-east-1.

To learn more about CloudFormation custom resources, you can check out this workshop.

Why the first option?

I chose the first option for simplicity and maintainability. Custom resources expand CloudFormation but I love to keep things simple. That means not creating too many resources via custom resources.

I will use custom resources in my solution to "GET" some values (not creating anything).

Stack

Prerequisites:

Having your Route53 zone in the same AWS Account.

Architecture

Let's go and Build !!!

1/ As you can see in the diagram above, the first step is to create the "certificate.yaml" file.

To create an ACM certificate on AWS with DNS as DomainValidationOptions, we need to provide the DomainName and the HostedZoneId. CloudFormation does not get the HostedZoneId dynamically.

I made a custom resource for a dynamic approach, not hardcoded. AWS should provide the support of DomainName only.

To get the ARN of this certificate from the other stack I store it in a ParameterStore.

 CertificateARN:
    Type: "AWS::SSM::Parameter"
    Properties:
      Name: !Ref ACMCertificate
      Type: "String"
      Value: !Ref Certificate

2/ Deploy the main stack with the template named "template.yaml".
Now we need to deploy the main stack, our EDGE API Gateway. We need to get the certificate we created in Step 1 and the HostedZoneId for our domain. To do this we will create a second custom resource.

The stack described in this article is available on my GitHub if you want to test or reuse it.

CloudWatch Metrics Pricing - FinOps

Walid BATTOU — Tue, 22 Nov 2022 11:31:56 +0000

In this article, I will talk about CloudWatch metrics. Like the other articles in my FinOps series, I will give you some tips about the cost-effective architecture design pattern.

Many AWS services provide metrics by default that are free of charge(cf:https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/aws-services-cloudwatch-metrics.html).

From AWS documentation :
"Many AWS services offer basic monitoring by publishing a default set of metrics to CloudWatch with no charge to customers."

Take care of the additional cost incure if you use external solution like DataDog or NewRelic (get metrics from APIs).

Many companies want to use a manage solution for their Observability strategies. These solutions are usually very expensive. When you make a forecast for this kind of solutions for AWS, you need to take care of the additional cost you will pay :

The amount of API calls to GET metrics.
The amount of Data Out from AWS to internet to upload your logs to your managed solution.

Build the dashboards you will often consult.

Every Ops needs some dashboards to consult in case of incident or performance issue. You have to define a clear strategy about this point.

When an Ops team checks the monthly cost of a dashboard, 3$
does not seem expensive. But if you have 20 members in your team and each of them creates multiple dashboards, the bill will increase.

Use some IaC tools like Terraform or CloudFormation to build your dashboards. In this case you will have the opportunity to destroy and recreate them when needed.

Use "Metrics Insights - query builder" which is free of charge.

CloudWatch Metrics Insights is a powerful SQL query engine that you can use to query your metrics. Unfortunately, for the moment it is not possible to save searches directly on it.

You can use the interactive builder view and save the SQL query generated in a git repository to share them with your team.

s3 Storage/DataRetrievals Pricing - FinOps tips

Walid BATTOU — Thu, 10 Nov 2022 07:24:28 +0000

In this article, I will talk about s3, a complex and highly available service. Like the other articles in my FinOps series, I will give you some tips about the cost-effective architecture design pattern.

Because s3 is a huge topic, today I will only talk about the cost of data storage and data retrieval. The other charges will be covered in a future article (Requests, Data transfer).

Monitor storage metrics with Storage Lens.

When you work on s3, you need some insights about the objects we store, for example:

The size of your buckets.
The total number of encrypted bytes.
How many objects do you have ?

With Storage Lens, you can analyze your storage with ~29 metrics and interactive dashboards to aggregate data for your entire organization.

Storage Lens provides metrics, you can use them in CloudWatch to create alarms and triggered actions.

A real case example :

Using multipart upload to backup your data on s3.

When you use s3 multipart upload (from an object size of 100 MB, you should consider using multipart uploads) and the uploading process fails, you pay for the partial data you have in your bucket.

Consequences :

You will not find any data in your bucket but a high cost in cost explorer.

Solution:

Use Storage Lens to identify this issue and publish the metrics to CloudWatch to be able to create an alarm.

Choose the right encryption strategy, do not use KMS for a bucket with a high volume of data (performance impact and high cost).

A real case example :

You have a data lake workload and use s3 to store the data.

As part of the security guideline of the company, you need to encrypt your data at rest and in transit. A CMK KMS key is used to encrypt all of your s3 buckets at rest.

Consequences :

After a period of time, you notice that KMS is very expensive.

Solution:

For big data workloads, I do not recommend using a CMK KMS key for encryption. Why? Because of the interaction between s3 and KMS to decrypt/encrypt objects when you access them.

For a bucket with TeraBytes or PetaBytes of data, you will have a performance impact and high cost for KMS service.

From my experience, with a very large s3 bucket, we saved 6K$/month by changing the encryption key from SSE-KMS to SSE-S3.

If the security guideline does not allow you to do that, consider using "S3 Bucket Keys for SSE-KMS".

Prefer intelligent Tiering class, depending on your retrieval performance requirements.

As you can see in the animation, with Intelligent Tiering you don't pay to retrieve data except for the expedited mode.

If your use case does not need specific performance and you don’t know how your workload will evolve, Intelligent Tiering is the right class to begin with.

AWS backup for s3 is expensive, do not choose it as default.

For the Ireland region, AWS backup for s3 costs 0.05$/GB per month. It is really expensive but you should take this option if your use case needs:

Need point-in-time recovery.
Protect from accidental objects/version deletion.
Want to use AWS backup policies at the organizational level.

I hope these tips will help you. Do not hesitate to give me some feedback.

Application Load Balancer Pricing - FinOps

Walid BATTOU — Wed, 02 Nov 2022 23:23:34 +0000

I would like to thank the people who gave me some feeback on my first article.

On this one, I made an animation about the pricing of an Application Load Balancer on AWS and my FinOps recommendations for it.

If you still have some Classic Load Balancer in place, you pay for DataProcessing and not LCU. Migrate CLB to ALB to save cost.

The CLB (Classic Load Balancer) was released in 2009, ALB (Application Load Balancer) was released in 2016. If you started on AWS before 2016, you had no choice but taking CLB.

Check your workloads and do the migration.

If you don’t use API Gateway features, challenge the migration to an ALB to reduce cost.

This recommendation applies to a Serverless use case. When you start using Lambda for building an API, the default choice is API Gateway.

Since 2018, it has been possible to use an ALB to invoke lambda functions and serve HTTP(S) requests, which can be cheaper.

When enabling access log, do not forget to define s3 retention policy.

When using an ALB, think about defining your observability strategy :

How do I check my logs ?
Which dashboard will I use ?
I store it on s3 but which period do I need to archive ?

CloudWatch Logs Pricing - FinOps

Walid BATTOU — Thu, 27 Oct 2022 06:00:11 +0000

I would like to share graphical content about some FinOps topics and what you need to know about AWS Bills. I am starting with a CloudWatch log animation, let me know if you are interested in this content.

You will find some explanations about my recommendations below.

Configure retention policy on your log groups.

You can directly configure your log group retention policy on your CloudWatch agent configuration file.

Concerning the archiving of your oldest logs, for the moment, you can periodically copy them to s3 but not schedule the copy.

Avoid selecting a long period of time when using Logs Insights queries.

You can use CloudWatch Logs Insights to do some queries and build some dashboards based on them. On dashboards or queries, do not select a large period of time when you have a lot of data in your log groups.

Think about using throttling/WAF on your internet facing workloads to avoid a high log volume ingestion.

For this recommendation, we can take API gateway for example.
Suppose that your company can’t have the time to invest on security for the moment, there aren’t any WAF rules or throttling in place.

What happens if your API Gateway gets attacked and logging is enable on it ?
The result will be a high volume of log. The most expensive part is logs ingestion (cf:animation below). In the cloud, other than causing some performance troubles, an attack will lead to an additional cost on logs and metrics.