DEV Community: Brian

5 Ways to Cut AWS Costs You Didn't Know About

Brian — Wed, 20 Sep 2023 14:29:41 +0000

If you’re an Amazon Web Services (AWS) customer, you’re probably in this repeating cycle of spiraling cost pain..

The bill comes in over budget every month.

The engineering team can’t answer “Why”?

And then you’re left searching for answers.

If this is you, you’re not alone, and it's not your fault.

There are entire categories of hidden costs in AWS’ billing data. Charges like Data Transfer, “EC2-Other”, and API activity are notoriously difficult to isolate and understand.

Strake’s platform was purpose built to solve this problem, leaving you with lower, more predictable, cloud costs.

Strake helps engineering teams identify and eliminate waste. That means tying every charge back to the resources that caused them and identifying exactly what they are paying for.

As an example of how our platform works, we documented some examples of Strake customers defeating these hidden costs in our new eBook: 5 Ways to Cut AWS Costs You Didn’t Know About.

Each chapter addresses a unique AWS billing issue with annual savings over $100,000:

– The Hidden EC2 Data Transfer Tax
– The NAT Gateway Processing Multiplier
– The S3 Request Flood
– The CloudWatch Rolex Watch
– The RedShift Data Hoarders

Take a look at this eBook and let us know about the hidden costs you'd like to see defeated next!

Download free eBook

AWS EC2 Instance and EBS Volume Pricing

Brian — Tue, 09 May 2023 23:42:37 +0000

Originally posted on Strake's Website.

AWS Pricing by Strake is an open-source project that allows anyone to bring AWS Public pricing into Google Sheets using Strake's custom functions. To help the community get started with this project, Strake published several free templates to help engineers answer critical questions about AWS Pricing.

Today, we will use Google Sheets templates to analyze Amazon EC2 Instance and EBS Pricing.

Amazon Web Services Pricing is Complicated!

Amazon Web Services operates in over 30 regions, has over 200 different services, and thousands of usage types. This results in a nauseating number of prices that go into calculating your monthly AWS bill.

For Amazon Elastic Compute Cloud (EC2), there are over 265,000 lines of data in the public AWS EC2 pricing file for a single region.

On top of all the operational complexities of AWS, understanding AWS pricing and pricing changes is a critical component of understanding your AWS bill. In this blog post, we will walk through how your team can quickly analyze AWS Pricing data and use this data to your advantage when planning infrastructure changes.

AWS Pricing by Strake

Strake recently launched our open-source project, AWS Pricing by Strake, on Product Hunt. This project is an add-on for Google Sheets that makes up-to-date AWS public pricing available using custom Google Sheets functions.

Before we go into too much detail, here is a 3-minute video on how to get started with AWS Pricing:

AWS EC2 Pricing

In previous posts on Strake’s blog, including the Developer’s Guide to AWS Costs: EC2 and What is EC2-Other?, I’ve broken down the different types of charges that come through your AWS bill under the ‘EC2’ service. Today, we will focus on understanding pricing for the hourly running cost of EC2 Instances.

Amazon EC2 Instance Types

Before using AWS Pricing by Strake to calculate EC2 pricing, we must understand the concept of EC2 instance types. EC2 instance type, region, platform, and purchase type are the key variables to determine what an EC2 Instance will cost per hour.

EC2 instance types combine an EC2 instance family and the EC2 instance size. For example, the instance type “m5.large” is the combination of the instance family “m5” (anything before the ‘.’) and the instance size “large” (after the ‘.’). You can repeat this methodology for any EC2 instance type offered by AWS:

On Demand and Reserved Instance EC2 Pricing

With the AWS Pricing by Strake project, our team has also shared several templates to help users calculate their infrastructure costs. The first template, “AWS Pricing by Strake - EC2 Pricing and Reserved Instances”, makes it simple for any user to calculate the On Demand and Reserved Instance costs for their EC2 instances.

DOWNLOAD: AWS Pricing: EC2 Template

First, If you have not already, you can download AWS Pricing by Strake from the Google Workspace Marketplace. Downloading this add-on takes less than 15 seconds!

Second, you will need to make a copy of the EC2 Template and open the sidebar to view our formula documentation. Once you have downloaded a copy of the template, click Extensions → AWS Pricing by Strake → How to use AWS Pricing.

Third, Once the add-on has been downloaded and the sidebar is open, you can start the use the template. Select the region, platform, and instance type of the EC2 instance you want to calculate pricing for. After the formulas calculate, you will have the On Demand and Reserved Instance prices and discount rates to analyze in Google Sheets.

Now that we understand how the template works, we can use this template to answer some critical business questions, for example:

How do the prices for the new c6i instances compare to the c5a instances? The price is often lower when AWS releases new instance families compared to the current generation. Engineers must understand the cost and operational benefits of moving to a new instance family before starting the migration.

Should we make convertible or reserved instance reservations? How do the discount rates compare for different payment options and commitment terms? I discussed this in another blog post, A Complete Guide to AWS Reservations. Three key variables determine how large a discount will be: length of commitment, specificity of the commitment, and the amount of money paid upfront. This EC2 template will let your business understand the different discount rates to weigh cost benefits and operational complexity.

We're starting a new development project. What instance types should we use? How does pricing compare across EC2 instance types? When your engineering team starts a new project, it is crucial to estimate your costs from the beginning. Maybe $0.01 per hour doesn't seem like a big difference, but when you're running thousands of instances in production, that can be a meaningful cost.

EC2 and EBS Volume Pricing

The second template we have created and shared with the AWS Community, “AWS Pricing by Strake - EC2 & EBS Calculator”, combines the hourly EC2 instance cost with any attached EBS volume cost. These two costs combined give users a much better understanding of what their development project will cost.

DOWNLOAD: AWS Pricing: EC2 and EBS Template

To use this template, First, select the details for the EC2 instances. We discussed calculating the cost for different EC2 Instance Types with the “EC2 Pricing and Reserved Instance Template”. The only difference with this template is that the user must also select the payment type and any related selections. For example, if you choose the “Purchase Type” as Reserved, you will be prompted to fill out the Offering Class, Term, and Payment Option.

Second, select the details for the EBS volumes that will be attached to your instances. This step is optional. If you don’t want to calculate the cost of EBS volumes, leave those cells blank.

EBS volumes have pricing variables similar to EC2 Instances that must be chosen before AWS Pricing by Strake can reveal a price. These are Volume Type, Storage Type, Region” and Volume Size (GB).

Third, select the count of EC2 Instances and EBS Volumes. This value is used to calculate the total cost of your infrastructure. This cell needs to be filled. The default value for this template is “1”.

With the template filled out, the calculations will show Hourly, Daily, and Monthly total costs as well as the individual costs for EC2 and EBS. This template, in particular, is suited for when engineers are making EC2 compute or EBS storage changes and need to calculate cost changes, such as the following:

Upgrading EBS volume types and want to understand the total cost impact. Storage costs are just as complex as EC2 Instance Types. When your team is planning infrastructure changes, understand the cost before it happens so there are no surprises on your AWS bill.

Expanding production infrastructure due to customer demand and need to estimate the image to their Gross Margins. If your production environment is based on EC2 Instances with EBS volumes as the primary form of storage, this template can help you build out your Cost of Goods Sold (COGS) for your production environment. COGS is a compelling metric to track as a business to understand how your costs will scale as you win new customers.

Calculating the total cost impact of changing infrastructure from On Demand to Reserved Instance pricing. Total cost reduction from reservations is an important metric to track so the business understands what percentage cost reduction Reservations can actually provide. Unfortunately, Reserved Instances don't apply to EBS Volumes. This template will allow your team to combine EC2 Reserved Instances and your existing EBS volumes to calculate the total cost reduction to your infrastructure.

Learn More

If you're interested in learning more about AWS Pricing by Strake, here are some additional resources:

🖥️ Visit our website to learn about features and download AWS Pricing templates.
🚀 Create a GitHub Issue for bugs or if you have ideas to improve the project.
💭 Join the Strake community if you have questions or want to stay updated with Strake.

AWS Pricing by Strake on Product Hunt!

Brian — Mon, 20 Mar 2023 07:49:27 +0000

A couple of weeks ago, Strake made an announcement that we have entered into an agreement with Mike Heffner to assume ownership and maintenance of the AWS Pricing add-on for Google Sheets. You can read that full announcement on Strake's website.

Today, Strake is launching 'AWS Pricing by Strake' on Product Hunt! 🚀🚀🚀

AWS Pricing by Strake is an open-source project that allows users to use AWS public pricing data in Google Spreadsheets. The add-on is available from the Google Workspace Marketplace.

AWS pricing is highly complicated. Did you know there are over 300,000 line items in the most recent EC2 pricing file from AWS? I know this is a constant pain point for me - How am I supposed to know when prices change, how they impact our AWS bill, or which infrastructure changes our team should make?

We’re excited to collaborate with the Product Hunt & AWS community through this project.

Please share your feedback on the product on Product Hunt. We'd love your input on the roadmap! 🚀

AWS Pricing plug-in for Google Sheets

Brian — Tue, 14 Mar 2023 16:32:55 +0000

Strake, the AWS cost management platform built for engineers, is excited to announce that we have entered into an agreement with Mike Heffner to assume ownership and maintenance of the AWS Pricing plugin for Google Sheets. The original plugin has been partially restored, and is available for free from the Google Workspace Marketplace.

AWS customers are always looking for ways to optimize their infrastructure and achieve more accurate forecasting. Unfortunately, this work requires manual effort and removes engineering resources from important feature work. In addition to Strake's core platform, we're committed to maintaining the AWS Pricing plugin for Google Sheets as an open-source tool to help AWS users understand their cloud costs.

Our team has spent the last several months working with the creator, Mike Heffner, to understand the project and how to best serve the community:

Nearly four years ago I created the aws-pricing Google Sheets add-on because I was tired of copying and pasting AWS pricing data when compiling infrastructure cost estimates. Since then I've been amazed by the adoption and interest from the AWS community for the add-on. That's why I'm excited that the team at Strake has graciously offered to take over ownership and maintenance of the add-on moving forward. I'm grateful that the team has restored the add-on for all users and I'm excited to see what new features they'll add in the future. The add-on will be able to continue as a useful resource for the AWS community.
– Mike Heffner

The AWS Pricing plugin for Google Sheets allows any Google Sheets user to pull the current public AWS pricing information into their spreadsheet using Google Sheets functions.

In its current state, there are functions for Elastic Compute Cloud (EC2), Elastic Block Store (EBS). In the coming weeks, we will have an update about the second version of the AWS Pricing Plugin that features streamlined formulas, restored RDS functions, and an updated backend to work with the new format of public AWS Pricing files.

Strake is committed to maintaining the AWS Pricing plugin as an open-source project and expanding the project with feedback from the open-source community.

If you want to learn more about the AWS Pricing project, you can get it from the Google Workspace Marketplace.

For questions, comments, or feedback on the project roadmap, join the Strake Community.

AWS Cost and Usage Report Documentation

Brian — Tue, 29 Nov 2022 18:07:52 +0000

This blog was originally published on getstrake.com

The Cost and Usage Report is the standard report from AWS for customers to understand and manage their costs. The Cost and Usage Report drives most cloud cost management tools, including AWS Cost Explorer. This documentation will outline the essential fields in the Cost and Usage Report, explain how those fields can be used for cloud cost management, and provide sample values users can use to build queries based on usage in their accounts.

The Cost and Usage Report

The Cost and Usage Report is the standard billing report for AWS Customers. This report is free (except for the S3 storage costs) and can be created by anyone with the proper billing permissions. For more details on the Cost and Usage Report, how to make a report, and getting started with analysis, check out this overview in the Developer's Guide to AWS Costs.

This report contains over 200 fields and can be millions of records for a single month of usage. Using the documentation below, we are taking the ~200 fields in the Cost and Usage Report and filtering it down to only 30 of the most critical fields. These 30 fields will answer most of your cost management questions and greatly simplify your cloud cost management practices.

Cost and Usage Report Field Categories

There are seven categories of cost fields across all Cost and Usage Reports: Bill, Identity, Pricing, Line Item, Product, Reservation, and Savings Plans. Below, we will break out the essential fields by field categories, provide details about what these fields describe, and provide sample values that will show up in your cost and usage report.

Continue Reading @ Strake

A Complete Guide to AWS Reservations

Brian — Mon, 21 Nov 2022 14:08:24 +0000

This blog was originally posted on Strake's Website.

Summary

In this overview of AWS Reservations, we will discuss the most popular reservation products: Compute Savings Plans, EC2 Savings Plans, EC2 Reserved Instances, and RDS Reserved Instances. In addition, we will mention the other reservation products offered by AWS and where you can find more information on those products.

What are AWS Reservations?

AWS Reservations offer customers the ability to commit to using certain resources for an extended period of time in exchange for a steep discount. Discounts range from 20-75% depending on how much risk your organization is willing to take.

Compute Savings Plans

Compute Savings Plans are the most flexible reservation tool offered by Amazon Web Services. Using a compute savings plan from AWS, you can commit to a certain amount of 'compute' spending and receive a significant discount.

Compute Savings Plans reduce the engineering overhead of using AWS Reservations. Compute Savings Plans are automatically applied across your EC2, Fargate, and Lambda usage regardless of instance type, Region, operating system, or tenancy.

Compute Savings Plans commitments are made in dollars per hour. If you execute a Compute Savings Plan commitment of $1 for a 1-year term, that is a commitment to spending $1 on compute (either EC2, Lambda, or Fargate) at discount prices every hour for the following year.

Compute Savings Plan Options

Term Length

A Compute Savings Plan is either a 1-year or 3-year commitment. The longer the commitment, the more significant the discount. Committing to a three-year term will generally cause the discount rate to double.

Payment Options

Payment terms for Compute Savings Plans determine whether the contract is paid upfront or hourly for the term of the agreement. The more money paid upfront will increase the discount for the Compute Savings Plan. There are three options:

All Upfront - The contract is paid upfront before any usage. All Upfront payment terms provide the highest discount rate.
Partial Upfront - Half of the contract is paid upfront, and half is paid hourly over the agreement term. Partial upfront is a lower discount rate than All Upfront but a higher discount than No Upfront.
No Upfront - No money paid upfront and the lowest discount rate. The cost of the contract is paid over the agreement term.

Compute Savings Plan Reservation Calculation

To calculate the compute savings plan commitment your team should make, you need to understand how much for On-Demand usage is in your account. To do that, you can either use the AWS Cost Explorer or query your Cost and Usage Report. For a full walkthrough of how to query your Cost and Usage Report, check out this walkthrough on the Developer's Guide to AWS Costs.

Once you understand the On-Demand usage in your account, you need to calculate what that usage would cost if a Compute Savings Plan covered the usage.

For example, let's say that over the last seven days, we had 10,000 hours of m5.large On-Demand usage in the US East, North Virginia Region, needs to be covered with a three-year, All Upfront, Compute Savings plan. To calculate the Compute Savings Plan commitment to cover this usage, we must:

Multiply the On Demand hours by the Savings Plan rate:

Compute Savings Plan rate = $0.044

Number of m5.large hours = 10,000
10,000 hours * $0.044 per hour = $440

Take the $440 for the last seven days, and identify the spend per hour:
$440 for 7 days = $62.85 per day = $2.61 per hour

Assuming your team wants to cover 100% of this usage, the hourly commitment executed in the AWS Console is $5.71 per hour.

EC2 Savings Plans

EC2 Savings Plans provide a deeper discount than Compute Savings Plans but only apply to EC2 usage and require you to make a more detailed commitment. EC2 Savings Plans require AWS customers to commit to usage in a specific instance family in an AWS Region. Once this commitment is made, the savings are automatically applied to the instance families you chose, regardless of the instance type.

The additional restrictions on EC2 Savings Plans require more planning and alignment by engineering teams. Once this commitment is made, it will not automatically be applied if the team decides to change their instance family or to operate in a different region.

Similar to Compute Savings Plans, an EC2 Savings Plan commitment is made in dollars per hour.

EC2 Savings Plan Options

EC2 Savings Plans require the user to select the AWS Region, Instance Family, Term Length, and Payment Option before committing.

AWS Region

EC2 Savings Plans commitments are made by Region. For example, a Region could be US East (N. Virginia).

Instance Family

EC2 Savings Plans require the user to select an instance family that is covered. For example, this would be 'm5' or 'c5'. It is important to note that EC2 Savings Plans are size flexible, so users can avoid committing to individual instance types.

Term Length

Same as with the Compute Savings Plan, the term length options for an EC2 Savings Plan are either 1-year or 3 years. On average, going from a 1-year to a 3-year commitment with EC2 Savings Plans will increase your discount by 50%.

Payment Options

Same as the Compute Savings Plan, the payment options for an EC2 Savings Plan are:

All Upfront - The contract is paid upfront before any usage. All Upfront payment terms provide the highest discount rate.
Partial Upfront - Half of the contract is paid upfront, and half is paid hourly over the agreement term. Partial upfront is a lower discount rate than All Upfront but a higher discount than No Upfront.
No Upfront - No money paid upfront and the lowest discount rate. The cost of the contract is paid over the agreement term.

EC2 Savings Plan Reservation Calculation

The calculation for EC2 Savings Plans Reservations is very similar to Compute Savings Plans, except we don't need to worry about Fargate or Lambda usage.

Let's use the same example of 10,000 hours of m5.large On-Demand usage in the US East North Virginia Region over the past seven days. That usage needs to be covered with a three-year, All Upfront, EC2 Savings plan. To calculate the required Compute Savings Plan commitment for this usage, we must:

Multiply the On Demand hours by the Savings Plan rate:
Compute Savings Plan rate = $0.036
Number of m5.large hours = 10,000
10,000 hours * $0.036 per hour = $360

Take the $360 for the last seven days, and identify the spend per hour:
$360 for 7 days = $51.42 per day = $2.14 per hour

Assuming your team wants to cover 100% of this usage, the reservation must be $2.14 per hour. The EC2 Savings Plan provides an additional 18% discount compared to the Compute Savings plan.

Reserved Instances

Reserved Instances are the previous generation of Reservations offered by Amazon Web Services. Reserved Instances are still alive and well, but require significantly more overhead than Compute or EC2 Savings Plans.

Reserved Instances are a commitment made against resources. For example, if you are running 100 m5.large instances, you will need to commit to 100 m5.large reserved instances.

Amazon offers Reserved Instances for Elastic Compute Cloud (EC2), Relational Database Service (RDS), Elasticache, Amazon OpenSearch, Amazon Redshift, and DynamoDB. All these products require the buyer to choose a Term of 1-year or three years and Payment Options of All Upfront, Partial Upfront, or No Upfront.

Elastic Compute Cloud (EC2) Reserved Instances

EC2 Reserved Instances are the most common type of Reserved Instance. EC2 Reserved Instances come in two categories: Standard and Convertible Reserved Instances.

With the creation of Compute Savings Plans and EC2 Savings Plans, it makes sense for most AWS customers to purchase those products instead of EC2 Reserved Instances. These products offer similar discounts with significantly less developer overhead.

Standard Reserved Instances

Standard Reserved Instances lock in a price on instance usage and do not allow the user to change the instance family, operating system, or tenancy of the reservation. If changes need to be made to the AWS availability zone, EC2 instance size, or networking type, you can make those changes via the AWS APIs or in the AWS Management Console.

Convertible Reserved Instances

Convertible Reserved Instances offer additional flexibility over Standard Reserved Instances by allowing users to change instance family, operating system, tenancy, and payment options and give customers benefits when infrastructure prices are reduced.

Relational Database Service (RDS) Reserved Instances

Relational Database Service (RDS) Reserved Instances significantly discount your RDS usage. AWS customers can not cover RDS with Savings Plans products, so Reserved Instances are your only option.

Here are some of the key details of RDS Reserved Instances:

RDS Reserved Instances are available for all supported DB Engines and in all AWS Regions.
Only MySQL, MariaDB, PostgreSQL, Amazon Aurora DB Engines, and BYOL for Oracle offer instance size flexibility. There is no size flexibility for other DB engines, meaning users who have to reserve the exact instance type they plan to use.
Terms offered for 1 or 3 years
Payment Options are All Upfront, Partial Upfront, and No Upfront. The No Upfront payment option is only available for a 1-year term for RDS.

All Other Reserved Instances

Amazon Web Services does offer reservation tools for other AWS products: AWS OpenSearch Service, Amazon Redshift, Amazon ElastiCache, and Amazon DynamoDB. These offerings are more complex than any of the reservations we've discussed before and require a deep understanding of the engineering roadmap before committing.

AWS OpenSearch Service Reserved Instances

OpenSearch Service Reserved Instances are identical to on-demand instances. They offer a 1 or 3-year commitment with an all upfront, partial upfront, and no upfront payment option.

OpenSearch Service Reserved Instances are not size-flexible. Suppose a reservation is made for c5.large.search instances that will not cover usage for any c5.2xlarge.search instances.

For more information about OpenSearch Service Reserved Instances, check out the Reserved Instances in Amazon OpenSearch Service AWS documentation.

Amazon Redshift Reserved Nodes

Amazon Redshift Reserved Nodes offer the ability to reserve nodes in return for a discount. Reserved nodes come in 1 or 3-year commitments with all upfront, partial upfront, and no upfront payment options. Similar to RDS, the no upfront payment option is only available for a 1-year term.

Redshift Reserved Nodes are not size-flexible. When making a reservation, you must specify the node type you want to be covered. If this node type changes, you will pay on-demand rates for the new nodes and still be charged for your reservation.

For more information about Redshift Reserved Nodes, check the Purchasing Amazon Redshift reserved nodes AWS documentation.

Amazon ElastiCache Reserved Nodes

Amazon ElastiCache Reserved Nodes offer customers the ability to reserve Cache node capacity for 1 or 3 years with either an all upfront, partial upfront, or no upfront pricing model. These are available for Redis and Memcached nodes and apply to all AWS Regions.

To make a reservation, you must select the Region, Term Length, and Cache Node Class, which you cannot change later. There is a maximum purchase limit of 300 cache nodes until you have to request a service level increase manually.

For more information about Amazon ElastiCache Reserved Nodes, check out the Amazon ElastiCache Reserved Nodes AWS documentation.

Amazon DynamoDB Reserved Capacity

Amazon DynamoDB Reserved Capacity offers significant savings for customers that can predict their read-and-write throughput. Users of DynamoDB Reserved Capacity pay a one-time upfront fee and a minimum usage level for the reservation duration.

DynamoDb Reserved Capacity only makes sense for AWS customers with extremely predictable DynamoDB usage or who have a minimum amount of usage they want to commit to in return for a slight reduction in spending. There is a big risk that Engineering optimizations could reduce your read-and-write throughput, and you would waste money on your reservation.

For more information on Amazon DynamoDB Reserved Capacity, check out the Amazon DynamoDB reservations AWS documentation.

How many AWS Services Can You Name?

Brian — Wed, 12 Oct 2022 15:53:05 +0000

Yesterday, I was having a conversation with my co-founder, @aryounce, and we fell down a rabbit hole of trying to figure out how many AWS Services we could name...

That conversation + my secret love of Sporcle quizzes led to this:

Brian Regan

@brianpregan

Think you've spent too much time in #AWS lately?

What about being able to list the 50 oldest services (still active) by memory?

sporcle.com/games/brianreg…

#DevOps #SRE

15:42 PM - 12 Oct 2022

Now, why the 50 oldest AWS Services? Honestly, I wanted to do ALL AWS Services, but that quiz wouldn't be any fun. Who has heard of "AWS Resilience Hub" or who uses "Amazon Corretto" anyway?

Even if you didn't think of all of these as 'Serivices', I bet you know all these answers. Give it a shot, and let me know what you think!

Quiz Link: 50 Oldest AWS Services (still active)

What is EC2-Other?

Brian — Tue, 20 Sep 2022 15:20:02 +0000

This post was originally posted on getStrake.com.

EC2-Other is a cost category calculated by Amazon that causes a lot of heartburn for the Engineering community. If you are a heavy EC2 user, this could be your second largest cost category on your AWS Bill.

This post will dive into this category to help AWS users understand what they are getting billed for and how to isolate individual cost categories within 'EC2-Other'.

Understanding EC2 Costs

Before we can unpack what costs live in the 'EC2-Other' category, you have to understand what this category means to total EC2 costs and how EC2 costs are calculated in various AWS reporting tools.

After researching this problem, I put together this table for how different AWS reporting tools display total EC2 costs. Below the table, I summarize the different reporting methodologies and how these values relate. Please note that this is only true for on-demand EC2 usage.

Calculating Total EC2 Costs

AWS Bill	CUR (ServiceCode)	AWS Cost Explorer	CUR (ProductCode)
Elastic Compute Cloud	AmazonEC2	EC2-Instances	AmazonEC2
Data Transfer (partial)	AWSDataTransfer (partial)	EC2-Other	AmazonEC2

AWS Bill

The AWS Bill category 'Elastic Compute Cloud' combines EC2 instance running costs, NAT Gateway, EBS, and all EC2 costs except data transfer. A second category in the AWS Bill called 'Data Transfer' combines data transfer costs for many different AWS Services. Unfortunately, EC2 Data Transfer costs cannot be isolated using only your AWS Bill.

AWS Cost Explorer

When you search for 'EC2' under 'Service' in Cost Explorer, there are four options that come up:

EC2-Instances (Elastic Compute Cloud - Compute)
EC2-Other
EC2-ELB (Elastic Load Balancing)
EC2 Container Registry (ECR)

The first two combined, EC2-Instances and EC2-Other, make up your total EC2 costs (including data transfer for EC2). The second two, EC2-ELB and EC2 Container Registry, can be ignored because they are different services altogether.

The EC2-Instances category isolates the hourly cost of running your EC2 instances. One of the benefits of this category is that customers can quickly identify the instance costs that can be covered with reservations. EC2-Other is a calculation that combines all other costs driven by EC2 usage: EBS, NAT Gateway, EC2's portion of Data Transfer, and others.

With these two categories, EC2-Instances and EC2-Other, customers can get the total cost of running their EC2 instances. However, since this cost includes EC2's portion of data transfer costs, this will not be the same as the AWS Bill's cost for 'Elastic Compute Cloud' if there is any data transfer cost.

AWS Cost and Usage Report (CUR)

The CUR is the source of truth for all AWS billing. In the Developer's Guide to AWS, I go into much more detail about the CUR, provide a data dictionary, and share instructions for setting up the CUR in your AWS Account.

For this discussion about EC2-Other costs, there are two fields that matter:

[lineItem/ProductCode] - the value 'AmazonEC2' captures all costs associated with running EC2 instances (including Data Transfer).
[product/ServiceCode] - the valued 'AmazonEC2' will show all EC2 costs other than Data Transfer, and 'AWSDataTransfer' is used for data transfer costs by any AWS Service.

Data Sources Key Takeaways

There are three key takeaways from the different AWS data sources before we dig into the cost categories that make up EC2-Other.

EC2-Other is a calculated metric from Cost explorer. This metric calculates the total cost of running your EC2 instances, less the fixed hourly instance running costs.
If you want to dig into costs for categories on your AWS Bill, the top cost categories on the AWS Bill align to the CUR's [product/ServiceCode] fields:

AWS Bill Categories	[product/ServiceCode] Values
Elastic Compute Cloud	AmazonEC2
Data Transfer	AWSDataTransfer

The sum of EC2-Instances and EC2-Other from cost explorer:
- IS equal to [lineItem/ProductCode] is 'AmazonEC2'
- IS NOT equal to 'Elastic Compute Cloud' on the AWS Bill because this field does not include EC2's portion of Data Transfer costs.

Visibility into EC2-Other Costs

In the Developer's Guide to AWS Costs, the first service I posted a guide for was EC2. In this guide, I broke EC2 costs into three logical categories:

Existence Cost - EC2-Instances in cost explorer - This identifies the cost of having an EC2 instance running regardless of the resource activity. This cost is based on an hourly rate determined by your instance type, region, and whether you're using contractual reservations or spot instances.
Utilization Cost - part of EC2 - Other in cost explorer - On top of the flat rate for having an instance running, some costs come from how an EC2 instance is utilized. These costs are driven by a usage-based metric such as $ per GB-month. The most common Utilization cost for EC2 is data transfer.
Subresource Cost - the second part of EC2 - Other in cost explorer - This is incurred for resources attached to or directly interacting with your EC2 instance. An easy example of EC2 is EBS Volumes used for EC2 instance storage.

I broke down the 'EC2-Other' category from Cost Explorer into Utilization and Subresource costs. Utilization focuses on how your EC2 resources are used, while Subresources isolates other resources (NAT Gateways or EBS Volumes, for example) that contribute to your total EC2 cost.

Total EC2-Other Costs

Utilization Costs

To isolate Utilization Cost, use the query below. This query isolates your EC2 instances using the [lineItem/ResourceId] field and filters out types [lineItem/UsageType] and [lineItem/Operation] fields that indicate Existence cost.

#Utilization Costs
##Utilization costs for EC2 by line item operation
SELECT
    [lineItem/Operation],
    round(sum([lineItem/UnblendedCost]), 4) as utilization_cost
FROM CUR
WHERE
    [lineItem/ProductCode] is 'AmazonEC2'
    and [lineItem/ResourceId] LIKE 'i-%'
    and [lineItem/UsageType] NOT LIKE '%BoxUsage%'
    and [lineItem/Operation] NOT LIKE 'RunInstances%'
    and [lineItem/LineItemType] is 'Usage'
GROUP BY
    [lineItem/Operation]
ORDER BY
    sum([lineItem/UnblendedCost]);

Some of the sample values from [lineItem/Operation] that indicate Utilization costs include:

VPCPeering-In and VPCPeering-Out - VPC Peering connections between the VPCs allow you to route traffic using private IP Addresses. All VPC Peering connections in the same AWS Region are charged at $0.01/GB.
PublicIP-In and PublicIP-Out - These track the costs of transferring data 'in' or 'out' from a public or Elastic IPv4 address. This traffic is charged at $0.01/GB in each direction.
InterZone-In and InterZone-Out - These are the costs associated with transferring data 'In' or 'Out' of EC2. Data transfer 'In' from the internet is free, but there are charges for data transfer 'out' to either the internet or other AWS Services. If you want to see the costs from AWS that show data transfer costs from EC2 to each region or product, those are posted on the EC2 pricing page under Data Transfer.

If you add the [product/ServiceCode] field to the SELECT statement of this query, you will be able to see which costs are categorized as 'AWSDataTransfer' and which are 'AmazonEC2'.

#Utilization Costs
##Utilization costs for EC2 by product service code line item operation
SELECT
    [product/ServiceCode],
    [lineItem/Operation],
    round(sum([lineItem/UnblendedCost]), 4) as utilization_cost
FROM CUR
WHERE
    [lineItem/ProductCode] is 'AmazonEC2'
    and [lineItem/ResourceId] LIKE 'i-%'
    and [lineItem/UsageType] NOT LIKE '%BoxUsage%'
    and [lineItem/Operation] NOT LIKE 'RunInstances%'
    and [lineItem/LineItemType] is 'Usage'
GROUP BY
    [lineItem/LineItemType],
    [lineItem/Operation]
ORDER BY
    sum([lineItem/UnblendedCost]);

Subresource Cost

There are three main categories of Subresources customers should have the ability to analyze in more detail:
EBS Volumes

EBS volumes are identified by filtering the [lineItem/ResourceID] field for items that contain vol-%.
There are two main types of costs for EBS Volumes that show up in the [lineItem/Operation] field:
- IO Usage: read and write costs.
- CreateVolume: costs for creating provisioned EBS Volumes.

#Subresource Costs
##Subresource costs for EBS Volumes
SELECT DISTINCT
    [lineItem/ResourceID],
    [lineItem/LineItemType],
    [lineItem/Operation],
    round(sum([lineItem/UnblendedCost]), 4) as subresource_cost
FROM CUR
WHERE
    [lineItem/ProductCode] is 'AmazonEC2'
    and [lineItem/ResourceId] LIKE 'vol-%'
GROUP BY
    [lineItem/ResourceID],
    [lineitem/lineitemtype],
    [lineItem/Operation]
ORDER BY
    sum([lineItem/UnblendedCost]);

EBS Volume Snapshots

EBS volumes are identified by filtering the [lineItem/ResourceID] field for items that contain vol-%.
There are two main types of costs for EBS Volumes that show up in the [lineItem/Operation] field:
- IO Usage: read and write costs.
- CreateVolume: costs for creating provisioned EBS Volumes.

#Subresource Costs
##Subresource costs for EBS Volume Snapshots
SELECT DISTINCT
    [lineItem/ResourceID],
    [lineItem/LineItemType],
    [lineItem/Operation],
    round(sum([lineItem/UnblendedCost]), 4) as subresource_cost
FROM CUR
WHERE
    [lineItem/ProductCode] is 'AmazonEC2'
    and [lineItem/ResourceId] LIKE '%snapshot%'
GROUP BY
    [lineItem/ResourceID],
    [lineitem/lineitemtype],
    [lineItem/Operation]
ORDER BY
    sum([lineItem/UnblendedCost]);

NAT Gateways

NAT Gateways are identified by filtering the [lineItem/ResourceID] field for items that contain %natgateway%.
There are many different cost categories for NAT Gateways, but they are primarily driven by data transfer or data processing fees.

#Subresource Costs
##Subresource costs for NAT Gateways
SELECT DISTINCT
    [lineItem/ResourceID],
    [lineItem/LineItemType],
    [lineItem/Operation],
    round(sum([lineItem/UnblendedCost]), 4) as subresource_cost
FROM CUR
WHERE
    [lineItem/ProductCode] is 'AmazonEC2'
    and [lineItem/ResourceId] LIKE '%natgateway%'
GROUP BY
    [lineItem/ResourceID],
    [lineitem/lineitemtype],
    [lineItem/Operation]
ORDER BY
    sum([lineItem/UnblendedCost]);

Community Cost and Usage Report Documentation

Brian — Tue, 06 Sep 2022 13:10:34 +0000

If you haven’t seen any of my previous posts, I’ve been sharing the work I’ve been doing with the Developer’s Guide to AWS Costs on Github. This project shares with SQL code developers can run on their own Cost and Usage Report to learn what exactly is driving costs in their AWS environment.

I’m always looking for feedback on the project and input on our roadmap in the GitHub discussions. Give us a ⭐️ if you like what you see!

CUR Documentation

This week, I added a file to the Developer’s Guide to AWS Costs on GitHub with the goal of building out a more in-depth version of AWS’ cost and usage report data dictionary with input from the developer community.

Right now, this has the very eloquent name: Cost and Usage Report Documentation.

One of the most frequent questions I’ve gotten about the project is ‘how do you know what these fields mean?’. Unfortunately, the answer has multiple parts:

There is some decent documentation from AWS.
There are some great authors out there that dive into the specifics of AWS Services. I’ve learned a lot from these folks!
I’ve been analyzing CUR data for years.. and there are some issues you only learn by messing them up many times 😃

With this documentation, the goal is to build out definitions and sample values for each of the 200+ fields in the cost and usage report. While this documentation will never be ‘complete’, an engineer can reference the documentation to understand the values they are finding in their own billing data. This should remove a lot of guesswork from understanding AWS costs and save your engineers precious time!

The Best Place to Start

In the table below you’ll see there are 7 groups of billing fields. Since there are over 200 fields in total, I decided to start with the Bill, Identity, and Pricing categories to help ground users in the data that describes the CUR itself and AWS’ pricing model.

Field Category	Status
Bill	Up to date
Identity	Up to date
Pricing	Up to date
Line Item	Not Started
Product	Not Started
Reservation	Not Started
Savings Plan	Not Started

If you want to see the documentation as it stands today, check out the file here. Below I’ve added two example fields from the documentation to give an idea of the types of detail I starting with.

bill/BillingEntity

This is the seller of the services you are being billed for. This can be different from the invoicing entity. There are three possible values for this field:

AWS - Amazon Web Services sells the AWS services.
AISPL - Amazon Internet Servies Pvt. Ltd which is the local reseller for AWS services in India.
AWS Marketplace - the AWS Marketplace is the billing entity for any third-party software purchased in the AWS Marketplace.

Sample Values
AWS Marketplace
AWS

pricing/PurchaseOption

Purchase option is for reservations and savings plans. There are only three options for purchase reservations that can be populated here.

Sample Values
All Upfront
Partial Upfront
No Upfront

Thank you!

This is just a start, with much more detail to come. Check out GitHub if you’d like to contribute to the project or have questions about anything we’ve shared. Look forward to working with all of you on this project in the future!

Stop Recurring AWS Bills

Brian — Tue, 30 Aug 2022 13:38:07 +0000

Across AWS Social Media the number of users that are stuck with recurring monthly bills is much higher than I thought it would be:

Andras Szabo 🇭🇺🇩🇪

@nomoreconfig

AWS sends me a bill of 0.95 $ every month. But I'm too lazy to figure out why 😂

16:07 PM - 10 Aug 2022

Tanya Reilly

@whereistanya

After several attempts to stop paying AWS 80c every month I spent an hour searching the console and finally found the stray service I hadn't deleted. And I was *sure* I had it this time until... I just got an AWS bill for 23c. This thing is the goddamn Hotel California.

16:32 PM - 03 Jan 2019

Jonathan Kozolchyk

@seakoz

My personal AWS bill is $0.13 every month. I put some stuff in S3 a long time ago and don't remember what. Wedding photos maybe? It's going to have to get a lot more expensive before I go looking.

21:43 PM - 04 Mar 2021

isai.btc

@isaibcortez

I got random AWS messages on Saturday didn’t think anything of it. All of a sudden I got hit with a $9k and $2k AWS elastic compute bill for Friday / Saturday, but my devs didn’t work those days. @AWSSupport please help 😖😖😖 my average is $450 per month 😭😭😭

23:13 PM - 08 Aug 2022

For those who find themselves in this trap, here are some basic steps you can take to finally stop these recurring bills or find what is driving your cost spike.

Reminder: ALWAYS Set up MFA

Take a few minutes, go to r/aws, and look at the posts over the past week. I bet at least one post sounds like this: "help, I've been hacked, and someone spent thousands of dollars on my account!!". Here's one if you aren't able to find any on your own.

The first question is: "Have you set up MFA on your account?"
The answer is ALWAYS: "No."

Don't be one of the people begging AWS to refund $5K in usage because you didn't set up MFA! What if AWS stops canceling bills from 'hackers' if you don't have MFA? You don't want to be stuck holding someone else's bar tab. Setting up MFA will take ~5 minutes and could save you thousands of dollars or a major headache.

Create a Cost and Usage Report

The first step to fixing a cost problem is figuring out what is driving your cost problem. To do this, you need visibility into what is running in your environment. The Cost and Usage Report is the best resource for AWS users to get visibility into what is running in their accounts.

The Cost and Usage Report (CUR) is the most detailed billing report Amazon has to offer. You can see a line item for every resource, for every hour during a month, with over 100 fields that describe the resources, resource operations, and pricing structure. The CUR contains the same data that drives visualization in Cost Explorer (we all know it isn't a good idea to rely on Cost Explorer.

In my GitHub project, the Developer's Guide to AWS Costs, I created a guide for creating and analyzing your Cost and Usage Report. There is no cost to create a CUR. The only fee you will incur is for s3 storage. This cost will scale with the size of your business: for an individual account or startup, this will be ~$1 per month. This cost would still be less than $~25 per month for large enterprise AWS customers.

To be clear, For an individual user looking to get rid of their lingering AWS costs, this report running for a single day would be less than $0.01 in storage costs. Pay $0.01 for getting rid of a $5 recurring monthly bill? Yes, please!

How to Find the Resources

With the cost and usage report active, here are two straightforward SQL queries to investigate which resources are driving your costs.

First, this query is a high-level summary of the costs for a particular account by AWS Service. Use this query as a high-level view to figure out which AWS Services need to be investigated further.

SELECT
    [lineItem/UsageAccountID],
    [lineItem/ProductCode],
    sum([lineItem/UnblendedCost])
FROM cur;
GROUP BY
    [lineItem/UsageAccountID],
    [lineItem/ProductCode]
ORDER BY
    sum([lineItem/UnblendedCost]);

Now we understand which services are causing our cost issue. Next, we can add the [lineItem/ResourceID] field to see a list of all the resources that are still running. For this example, we are filtering for all EC2 Resources.

SELECT
    [lineItem/ResourceID]
    sum([lineItem/UnblendedCost])
FROM cur
WHERE
    [lineItem/ProductCode] is 'AmazonEC2'
GROUP BY
    [lineItem/ResourceID]
ORDER BY
    sum([lineItem/UnblendedCost]);

Some costs in AWS aren't tied to a resource (of course), so if that last query wasn't able to answer your question, we could replace the [lineItem/ResourceID] field with [lineItem/Operation] and [lineItem/UsageType] to describe the cost categories that are driving cost.

SELECT
    [lineItem/UsageAccountID],
    [lineItem/ProductCode],
    sum([lineItem/UnblendedCost])
FROM cur;
GROUP BY
    [lineItem/UsageAccountID],
    [lineItem/ProductCode]
ORDER BY
    sum([lineItem/UnblendedCost]);

The two queries above aggregate costs for the duration of your report. Adding in the [lineItem/UsageStartDate] field will show costs by hour. This field adds a lot of detail to the report and will help you determine if this is a monthly service charge or a recurring fee for usage.

SELECT
    [lineItem/UsageStartDate],
    [lineItem/UsageAccountID],
    [lineItem/ProductCode],
    sum([lineItem/UnblendedCost])
FROM cur;
GROUP BY
    [lineItem/UsageAccountID],
    [lineItem/ProductCode]
ORDER BY
    sum([lineItem/UnblendedCost]);

The Last Step

Once you figure out and terminate the straggling resources, the last two things you will need to do are:

Delete your Cost and Usage Report
Delete the S3 bucket where your Cost and Usage Report is stored.

Delete the Cost and Usage Report

Search for 'billing' to go to the billing console and click on 'Cost & Usage Reports' on the left banner of your screen. Once on this screen, you will see all the active Cost and Usage Reports in your account. Select the report you created, and click on the 'delete' button (see the screenshot below).

Delete the S3 bucket.

Search for S3 to go to the S3 console. Once in the S3 console, look for the S3 bucket where your S3 bucket is being saved. Once this bucket is empty, you can select the bucket and click 'Delete'.

Clicking 'delete' will open a new screen where you must type the bucket's name and confirm you want to delete the s3 bucket.

Billing Freedom

These steps will allow you to see what you're being billed for and have the information you need to stop these recurring AWS bills. If you want a more detailed analysis of your favorite AWS services, check out our project on GitHub. If there are any questions about this process, let me know in the comments below ⬇️.

Content powered by Strake

Cost Explorer Isn't the Answer

Brian — Tue, 23 Aug 2022 14:45:00 +0000

I recently launched and started sharing the Developer's Guide to AWS. The Developer's Guide to AWS is my open-source project where I'm sharing techniques for analyzing AWS billing data using SQL.

The feedback I've gotten from the development community has been incredible. I appreciate the time people have taken to check out the code and provide some input on the roadmap!

If you're interested in following the project and providing some feedback, you can check it out on GitHub. If you like the analysis, give us a star ⭐️!

Why Not Cost Explorer?

One piece of feedback I've been getting pretty regularly is: "Why can't you just use Cost Explorer?" (check out this Reddit thread). I decided to summarize my opinions here so I can share this post instead of having to explain the reasoning every time I get this question in the future. 🤣

A little background on me: I'm a co-founder of Strake, and before this, I worked with companies helping finance and engineering teams understand their AWS billing. One of the biggest problems for AWS customers across the board is a lack of real cost visibility.

The truth is, your engineering teams can't understand and fix a cost problem if they don't have access to the right data. I've seen startups and multi-billion dollar enterprises trying to manage their costs using Cost Explorer, and here are some of the biggest problems we've encountered:

Exports, exports, and more exports.
What is EC2-Other?
Where are my reservations?

Exports, exports, and more exports

The single most significant drawback of Cost Explorer is the lack of the ability to filter data & visualizations by more than one variable.

Let's say, for example, I have 10 AWS accounts, and I want to understand my on-demand costs by AWS Service for each of my 10 accounts. I would not be able to do that in a single Cost Explorer visualization. My options are:

Create views for each AWS Account, grouped by Service
Create views for each Service, grouped by AWS Account.

For this simple example, a user would have to create a minimum of 10 views, export those views to CSV, combine the data and make a visualization. This process is a massive waste of engineering time when you compare all that work to this simple query from your cost and usage report:



##On-demand costs by Account, AWS Service
SELECT
    [lineItem/UsageAccountID],
    [lineItem/ProductCode],
    sum([lineItem/UnblendedCost])
FROM cur
GROUP BY
    [lineItem/UsageAccountID],
    [lineItem/ProductCode];

Can you get to the correct answer using Cost Explorer? Yes, eventually. I will argue all day that creating these views, exporting the data, and combining the reports in Excel is not a valuable exercise for anyone in your company.

What is EC2-Other?

I love EC2-Other because we've all been in conference rooms when bad decisions were made. I like to think EC2-Other was discussed back in 2006 as a placeholder, and here we are in 2022 STILL trying to figure out what it means.

First, it's good to discuss which costs are considered EC2-Other. I go into more detail on this topic in my first EC2 code guide, but here is the summary:

EBS volumes and snapshots
Nat Gateway resources and fees
Data Transfer costs

So now that we know what costs we're looking for, how can we break down these cost categories in Cost Explorer? It is possible to get the correct answer in Cost Explorer if you manually filter the 'Usage Type' and 'Usage Type Group' fields for each EC2-Other cost type. Then you would repeat the process: create views for each cost category, export the data from those views to CSV, and develop visualizations as required.

To circumnavigate all this work, I've written queries for the Cost and Usage Report that break down the various categories of EC2-Other costs. This query is an example of how to isolate EBS Snapshot costs:



#Subresource Costs
##Subresource costs for EBS Volume Snapshots
SELECT DISTINCT
    [lineItem/ResourceID],
    [lineItem/LineItemType],
    [lineItem/Operation],
    round(sum([lineItem/UnblendedCost]), 4) as subresource_cost
FROM CUR
WHERE
    [lineItem/ProductCode] is 'AmazonEC2'
    and [lineItem/ResourceId] LIKE '%snapshot%'
GROUP BY
    [lineItem/ResourceID],
    [lineitem/lineitemtype],
    [lineItem/Operation]
ORDER BY
    sum([lineItem/UnblendedCost]);

Unfortunately, EC2-Other isn't going anywhere. Fortunately, using the CUR instead of Cost Explorer, we can circumnavigate the issue and get to the bottom of what we are really being charged for.

Where are my Reservations?

One of the most confusing things I've seen engineering teams try to overcome is the lack of visibility to where their reservations are being applied. Making commitments in AWS is one of the few ways to keep AWS costs manageable. These commitments come in three primary forms: EDPs, Reserved Instances, and Savings Plans.

For another example: an AWS customer with 1 management account and 9 member accounts makes reservations on their EC2 usage. This reservation is made in the management account, so the cost savings can be applied to all AWS accounts. How does this actually show up in Cost explorer?

The monthly cost of the reservations would hit the management account, even if the reservation was used by a member account.
Any reservation usage in the member accounts shows up as $0.

This is a massive problem for engineering cost visibility. Overnight, some engineers will see their costs drop by $0, while others will not get any benefit from the reserved instances being purchased. The only answer for this get using Cost Explorer would be pulling EC2 instance hours by instance type for every account and reverse-engineering a $-per-hour rate that can be applied to the EC2 usage. I've done this before, and it is terrible.

A much easier solution would be to use this query on your CUR. This query breaks out the usage by instance type and region for each of the EC2 pricing models (on-demand, spot, reserved instance, and savings plans).



#Total existence cost for EC2
##Total existence cost by resource ID
WITH on_demand_existence AS (
    SELECT
        [lineItem/ResourceId],
        round(sum([lineItem/UnblendedCost]), 4) as existence_cost
    FROM CUR
    WHERE
        [lineItem/ProductCode] = 'AmazonEC2'
        and [product/instanceType] <> ""
        and [lineItem/LineItemType] is 'Usage'
        and [lineItem/UsageType] LIKE '%BoxUsage%'
        and [lineItem/Operation] LIKE 'RunInstances%'
    GROUP BY
        [lineItem/ResourceId]
)
, spot_existence AS (
    SELECT
        [lineItem/ResourceId],
        round(sum([lineItem/UnblendedCost]), 4) as existence_cost
    FROM CUR
    WHERE
        [lineItem/ProductCode] = 'AmazonEC2'
        and [product/instanceType] <> ""
        and [lineItem/LineItemType] is 'Usage'
        and [lineItem/UsageType] LIKE '%SpotUsage%'
        and [lineItem/Operation] LIKE 'RunInstances%'
    GROUP BY
        [lineItem/ResourceId]
)
, reserved_existence AS (
    SELECT
        [lineItem/ResourceId],
        round(sum([reservation/EffectiveCost]), 4) as existence_cost
    FROM CUR
    WHERE
        [lineItem/ProductCode] = 'AmazonEC2'
        and [product/instanceType] <> ""
        and [lineItem/LineItemType] is 'DiscountedUsage'
    GROUP BY
        [lineItem/ResourceId]
)
, savings_plan_existence AS (
    SELECT 
        [lineItem/ResourceId],
        round(sum([savingsPlan/SavingsPlanEffectiveCost]), 4) as existence_cost
    FROM CUR
    WHERE
        [lineItem/ProductCode] = 'AmazonEC2'
        and [product/instanceType] <> ""
        and [lineItem/LineItemType] is 'SavingsPlanCoveredUsage'
    GROUP BY 
        [lineItem/ResourceId]
)
SELECT
    CUR.[lineitem/ResourceId],
    CUR.[product/instanceType],
    CUR.[product/region],
    COALESCE(on_demand_existence.existence_cost, 0) as on_demand_existence_cost,
    COALESCE(spot_existence.existence_cost, 0) as spot_existence_cost,
    COALESCE(reserved_existence.existence_cost, 0) as reserved_existence_cost,
    COALESCE(savings_plan_existence.existence_cost, 0) as savings_plan_existence_cost,
    (COALESCE(on_demand_existence.existence_cost, 0) + COALESCE(spot_existence.existence_cost, 0) + COALESCE(reserved_existence.existence_cost, 0) + COALESCE(savings_plan_existence.existence_cost, 0)) AS total_existence_cost
FROM CUR
LEFT JOIN 
    on_demand_existence
    ON on_demand_existence.[lineItem/ResourceId] = CUR.[lineItem/ResourceId]
LEFT JOIN 
    spot_existence
    ON spot_existence.[lineItem/ResourceId] = CUR.[lineItem/ResourceId]
LEFT JOIN 
    reserved_existence
    ON reserved_existence.[lineItem/ResourceId] = CUR.[lineItem/ResourceId]
LEFT JOIN 
    savings_plan_existence
    ON savings_plan_existence.[lineItem/ResourceId] = CUR.[lineItem/ResourceId]
WHERE 
    CUR.[lineItem/ProductCode] is 'AmazonEC2'
    and CUR.[product/instanceType] <> ""
    and CUR.[lineitem/ResourceId] <> ""
GROUP BY
    CUR.[lineitem/ResourceId],
    CUR.[product/instanceType],
    CUR.[product/region]
ORDER BY
    total_existence_cost;

Final Thoughts

Cost Explorer does not meet the engineering teams' need to understand costs. The three examples discussed in this blog represent three significant roadblocks that create unnecessary work for engineering teams.

Any engineer can easily replace Cost Explorer for free by creating a Cost and Usage Report and doing your own SQL analysis. If there are any specific cost problems your team is fighting, let me know in the comments, and I'll add the analysis to the Developer's Guide to AWS Costs project!

Content powered by Strake

It's a Trap? (EC2 Spot Instances)

Brian — Thu, 18 Aug 2022 14:15:00 +0000

TLDR: Spot is an opportunity for big savings with no commitment, but costs are unpredictable. Be wary of using Spot too much and driving the price up on yourself (not kidding). Check out the guide on our website to analyze your own spot usage.

Is it a Trap?

All jokes aside, AWS does a great job marketing EC2 Spot Instances. When you do a quick google search, the promise of 'up to 90% savings' is thrown on the table. Who isn't going to click on that?

The best part? This is technically true. There are times when you can get massive discounts using spot instances. So why would AWS do this?

Take a moment and think about how much time and energy has been spent trying to forecast cloud usage or the server needs at your own data center. To say the least, this is a painful process. To say more, we (read: I) never get it right.

So how can AWS promise international availability for hundreds of services to millions of customers? If you strip AWS' business model down to the core, they are the foremost data center capacity planning experts. However, even the experts need to build some cushion into their plan every once in a while. For EC2, when there is excess capacity, that is sold on the market as Spot Instances.

The thought process from AWS is probably: We already bought it, might as well try to get some money for it!

How do you use Spot Instances

You submit a 'Spot Fleet Request' when you want to use spot instances. This request contains the following parameters:

AMI
Target capacity (# of instances)
- Maintain target capacity? Allow AWS to automatically replace interrupted Spot Instances
- Set a maximum cost for Spot Instances
Network and AZ details
Instance type requirements: either specify the instance type or the compute requirements for your workload
Select an allocation strategy

Once all these parameters are filled out, you will be given a 'Fleet Strength' score to describe how likely you will find a match.

How does Spot Compare?

On average, here is what you should expect financially from Spot Instances:

Average of 70% discount when comparing Spot to On-Demand pricing.
Average additional 10% discount when comparing Spot with 3yr no-upfront Compute Savings Plans and NO COMMITMENT

I said on average because the fun part about spot instances is how frequently the pricing can change!

The harsh fact about Spot is there is only so much excess EC2 capacity. The more customers try to use Spot instances, the higher the price goes (classic supply and demand economics). Check out this example from the Spot Instance pricing history tool in the EC2 console that highlights the pricing variance over the last three months.

There are two key takeaways from this image:

You will sometimes get a better discount for Savings Plans than Spot.
This price is highly variable. This will throw your team for a loop when you're trying to figure out what your costs will be in the future.

If you're using your cost and usage report for cost analysis, I've written some SQL code to help you identify your spot instance rates over time:

#Spot existence cost
##Spot existence by usage start date, instance type, region

SELECT
    [lineItem/UsageStartDate],
    [product/instanceType],
    [product/region],
    [lineItem/UnblendedCost]/[lineItem/UsageAmount] as spot_rate
FROM CUR
WHERE
    [lineItem/ProductCode] = 'AmazonEC2'
    and [product/instanceType] <> ""
    and [lineItem/LineItemType] is 'Usage'
    and [lineItem/UsageType] LIKE '%SpotUsage%'
    and [lineItem/Operation] LIKE 'RunInstances:SV%'
GROUP BY
    [product/instanceType],
    [lineItem/UsageStartDate],
    [product/region]
ORDER BY
    [lineItem/UnblendedCost]/[lineItem/UsageAmount];

When to use Spot Instances:

Spot instances are a no-brainer for dev clusters, research projects, and short-term workloads that don't have a tight deadline. You can set a cost target, pick from various instance types, and allow AWS to interrupt (pause) your work when the price gets too high. The work will eventually get done, and you will come in under budget.

The real opportunity is for the engineers who build an architecture that can handle some of their infrastructure running on spot instances. The combination of Spot, On-Demand, and Savings Plans will provide the optimal balance of low-cost and reliable infrastructure.

If you can do this successfully, your biggest problem will become your own spot usage throwing off the supply/demand equation and increasing your own costs. Think about that one for a bit..