DEV Community: Mike Graff

Has Generative AI ruined the re:Invent experience?

Mike Graff — Thu, 03 Oct 2024 21:12:48 +0000

Introduction

I've been attending re:Invent every year since 2015, and while the conference has its share of issues, I've always returned because I've felt the benefits far outweigh the drawbacks. It seems as if Generative AI related sessions have taken over re:Invent, to the detriment of other topics.

AWS re:Invent - a learning conference

AWS re:Invent is...a lot to deal with. Yes, the session catalog is terrible to navigate. So much so that it has inspired some fellow Community Builders to step in with the creation of third party tools to fill the gap. Sure, the session registration system is a total mess and has been for years. Not to mention it completely disadvantages folks from different countries that live in different time zones. Lastly, the huge crowds and navigating across multiple venues up and down the strip makes things logistically painful.

However at the end of the day it has always been the high quality and deeply technical nature of the content, the variety of topics covered, and the outstanding networking opportunities that kept me coming back year after year. (Not to mention the fantastic musical performances at re:Play).

However in 2024 Generative AI has become the dominant topic in the tech world. It has sucked all the oxygen out of the room at most tech conferences, and that trend seems to have extended to re:Invent as well. Let's dive in...

Generative AI related sessions have taken over re:Invent

I'm always excited when the re:Invent session catalog gets released, and this year was no different. However that excitement diminished significantly when I started looking at the breakdown of session subjects. I quickly discovered that the catalog is dominated by AI topics. More worrisome, several of the areas I'm passionate about (like Networking and Storage) have a shockingly low number of sessions this year. Here's a breakdown of the 2378 sessions currently listed in the re:Invent catalog by session topic:

Topic	Number of Sessions	Percentage of Total
AI/ML	820	34%
Analytics	223	9%
Architecture	243	10%
Business Applications	129	5%
Cloud Operations	237	10%
Compute	153	6%
Content Delivery	31	1%
Customer Enablement	34	1%
Databases	138	6%
DevOps & Developer Experience	214	9%
End User Computing	18	0.8%
Hybrid Cloud	37	2%
IoT	65	3%
Kubernetes	77	3%
Migration & Modernization	234	10%
Networking	70	3%
New to AWS	46	2%
Security, Compliance, and Identity	227	10%
Serverless & Containers	185	8%
Storage	111	5%
Training & Certification	29	1%

re:Invent 2024 session catalog breakdown by topic area. Note that sessions are often tagged to more than one topic area.

As you can see AI/ML topics currently make up whopping 34% of the session catalog. In addition, because AWS will tag the same session with multiple topics, a fair number of the sessions in other topic areas are also AI related.

Historical Trend

The AI takeover trend is even better illustrated when you compare the current session catalog to the catalogs from the past few years. (A big shout out to my fellow AWS Community Builder Raphael Manke for providing 2022 and 2023 session catalog data). Take a look at the trend graph for the various topic areas:

Here's a further detailed breakdown of selected session topic areas and the percentage change from 2022 to 2024:

Topic Area	2022	2023	2024	Change 2022 to 2024	Percentage change
AI/ML	417	768	820	403	52%
Architecture	314	309	243	-71	-23%
Cloud Operations	367	304	236	-131	-43%
Compute	237	209	153	-84	-40%
Customer Enablement	165	76	34	-131	-172%
Databases	272	200	137	-135	-68%
DevOps	370	263	214	-156	-59%
EUC	67	33	18	-49	-148%
Hybrid Cloud	164	82	36	-128	-156%
Networking & Content Delivery	238	128	101	-137	-107%
Serverless & Containers	311	292	185	-126	-43%
Storage	291	168	112	-179	-107%
Training & Certification	102	92	30	-72	-78%

re:Invent Session Catalog for the past three years, broken down by topic area

The number of AI/ML sessions has essentially doubled since 2022. Now I can't really blame AWS for this, as I already stated that is where all the buzz has been for the past 18 months so it makes sense. However, my real concern is the precipitous drop in some key topic areas that are essential to building and working in the cloud computing.

My typical go to topic areas at re:Invent have been Architecture (23% decrease), Cloud Operations (43% decrease), Networking (107% decrease) and Storage (107% decrease). The huge decreases in the areas of Customer Enablement and Hybrid Cloud are also alarming for folks who are coming to re:Invent for the first time and trying to figure out their cloud migration plans.

Conclusions

Look, I get it, Generative AI is the new hotness, and AWS has to react to that given how often they are getting beat up for being "behind in AI." However there needs to be balance, as a large number of folks who are building on AWS (including me) are not building anything that has to do with AI. If we are going to spend $2000 and a week of our lives suffering through Las Vegas, our time needs to be rewarded with a plethora of deep technical sessions on the topics that we care about.

Cloud Economist (and master of cloud snark) Corey Quinn had a blog post reviewing the AWS New York summit earlier this year and I found this quote summed things up nicely:

And so, the hyperfocus on GenAI is concerning to me because of what’s being shunted aside to create room for it. They’re Amazon WEB Services, not Amazon GenAI Services. I fix large AWS bills for large enterprises for a living; my customers have a raft of very large-scale challenges that don’t involve GenAI in the slightest.
- Corey Quinn

Will I be at re:Invent this year? Absolutely, for the reasons I outlined at the start of this post. Will this be my last re:Invent? I'll let you know after December.

What are your thoughts on the makeup of the re:Invent session catalog? Am I overreacting? Let me know in the comments.

Use IAM Roles Anywhere to reduce the use of IAM keys

Mike Graff — Mon, 06 Nov 2023 00:34:17 +0000

Abstract

Exposed static IAM keys are one of the most common security risks for AWS accounts. Avoiding the use of static keys is a best practice that can be hard to achieve in hybrid cloud environments where access needs to be given to external systems. IAM Roles Anywhere is a service that help mitigate this risk. In this article I detail the problems and risk associated with static keys, how IAM Roles Anywhere can solve this problem, and walk through how to setup the complete solution.

The Keys Problem
- Why this is bad
- The Toil of Managing Keys
- What about IAM Instance Roles?
- No Solution for External Systems
IAM Roles Anywhere Solves the Keys Problem
How does IAM Roles Anywhere Work?
- IAM Roles Anywhere Components
- IAM Roles Anywhere Architecture
Setting Up IAM Roles Anywhere
- Setup AWS Certificate Manager Private CA
  - Steps to Setup the Root CA
  - Important Note on CA Hierarchies
- Configure IAM Roles Anywhere
  - Establish Trust Anchor
  - Configure Role
  - Configure a Profile
Configuring the External System
- Generate Client Certificate
- Export the Certificate
- Decrypt the Private Key
- Install IAM Credentials Helper
Making the Connection
- Manually request and set credentials
- Leverage Roles Anywhere in AWS CLI Config
Closing Thoughts
References

The Keys Problem

One of the more pesky problems to solve when managing an AWS environment is granting access to external systems in a secure manner. External access to the AWS console can be handled via username and password (or even better, leverage SSO login via IAM Identity Center!). Unfortunately, this method doesn't work for API access from machines and applications outside your AWS environment.

Why this is bad

The most common solution for providing external access is not awesome. You generate an IAM access key pair for the application or system and pass those credentials over to your developer for use. Afterwords, you keep your fingers crossed that they do not share these keys with other folks or store them in plain text on their computer. Even worse would be if they put these credentials directly into their code and then commit it to a git repo.

Without a doubt this last scenario happens all the time, and consequently it is something that hackers are constantly scanning GitHub repos to find and exploit to get access to your AWS accounts.

The Toil of Managing Keys

Let's say you are able to avoid these credentials getting exposed accidentally. You still have to make sure you are performing good security practices on these keys by rotating them regularly. For example, in my environment we rotate keys every 75 days. Then you have to go through the process of securely exchanging the keys with your developers again, and they have to update their application to use the new keys. Wash, rinse, and repeat this process every few months and that adds up to a lot of toil just to keep your keys secure.

What about IAM Instance Roles?

This problem has been solved for years for systems running inside of AWS with IAM Instance Roles. This feature allows you to assign an IAM Role directly to an EC2 instance (or container or Lambda function). As a result, the EC2 instance then dynamically retrieves an access token for the role via the EC2 Instance Metadata service. Very slick and no static keys required!

No Solution for External Systems

However, the very nature of the way this service works requires that the calling system be inside AWS. For external systems you had no option beyond static keys. Thankfully, that all changed in July 2022 when AWS announced IAM Roles Anywhere.

IAM Roles Anywhere Solves the Keys Problem

IAM Roles Anywhere makes it possible to use IAM Roles on systems outside of AWS. It provides a mechanism for external servers, containers, and applications to obtain temporary AWS credentials in a manner similar to EC2 Instance Roles. Thus it eliminates the need to create and manage static access keys and dramatically improves the security posture of your AWS accounts.

How does IAM Roles Anywhere Work?

IAM Roles Anywhere leverages public key infrastructure (PKI) as a mechanism to establish trust between your external system and your AWS Account. Systems sitting outside of AWS hold X.509 Certificates that they present as part of a CreateSession request. Next these certificates are validated by IAM Roles Anywhere and then a temporary set of credentials are returned to the client.

IAM Roles Anywhere Components

There are six basic components to the IAM Roles Anywhere architecture.

Certificate Authority (CA). The CA is the heart of your public key infrastructure and is responsible for issuing certificates. For IAM Roles Anywhere you can use a CA provided by Amazon Certificate Manager, or you can use an existing External CA.
Certificates are digital documents that securely associate cryptographic key pairs with a system. A certificate contains the public key and a signature that has been encrypted with the private key. This allows a third party to verify that a certificate is valid by decrypting the signature using the public key. Certificates also contain a trust chain that links the certificate back to the CA that issued it.
A Trust Anchor is used to establish a trust relationship between IAM Roles Anywhere and your Certificate Authority.
An IAM Role. You probably already know that a Role is an IAM identity that contains specific permissions you wish to grant and that can be assumed by anyone who needs it. In order to use a role with IAM Roles Anywhere, your role must be configured to trust the IAM Roles Anywhere service principal.
Profiles are used to specify which roles IAM Roles Anywhere can assume and what your workloads can do with the temporary credentials that are issued. You can specify a session policy to limit the permissions created for the session.
Lastly, the IAM Roles Credential Helper is a downloadable tool provided by AWS that allows you to request temporary security credentials via the CreateSession API.

IAM Roles Anywhere Architecture

Here is a diagram that illustrates the components of the IAM Roles Architecture:

The components of the architecture work together in the following way:

A Trust Anchor is established between IAM Roles Anywhere and the Certificate Authority (CA). This CA can be running inside of AWS or externally in your own data center or even another cloud provider.
The CA issues an x.509 Certificate to the external system, where it is installed in the local store.
The IAM Role is created and configured to trust the IAM Roles Anywhere service principal.
A Roles Anywhere Profile associates the IAM Role with Roles Anywhere and can set session restrictions if desired.
The External Server issues a CreateSession request and provides it's Certificate along with specifying the role it wishes to assume.
IAM Roles Anywhere validates the Certificate is valid and tied to the CA contained in the trust anchor.
Once these validations are complete, the system is now authenticated and IAM Roles Anywhere will create a role session via STS and pass the session credentials back to the external system.
The external system can now use these temporary credentials to make any AWS API calls that are allowed in the IAM Role.

Setting Up IAM Roles Anywhere

Now that I've gone over the basic architecture of IAM Roles Anywhere, next I'm going to walk you through an end to end deployment of this solution. For my lab deployment I will use AWS Certificate Manager Private CA as our Certificate Authority. As a result the first thing I will walk through is how to stand up the Private CA.

Setup AWS Certificate Manager Private CA

AWS Certificate Manager is AWS' fully managed Certificate Authority service. You are probably most familiar with this service as a method for issuing public certificates for use with other services like CloudFront and API Gateway, but the service also has the capability to act as a Private CA, allowing you to setup entire managed CA hierarchies without the headache of managing your own CA (and trust me, having setup my own private PKI hierarchies in the past, it neither fun to setup nor maintain).

Steps to Setup the Root CA

First, we need to setup the Root CA

To get started, login to your AWS account with the appropriate credentials and navigate to the Certificate Manager service. From the Certificate Manager home page, click the link for AWS Private CA on the left hand navigation bar

On the resulting page, click the orange Create a Private CA button to get started creating your new Private CA.

After clicking the button, you will be presented with the Create a private certificate authority wizard page. Here we will be configuring all the required options to stand up our private Certificate Authority.

For your CA Mode options, select General Purpose. NOTE: if you are following along and setting this up in a lab environment be aware of Amazon Private CA pricing. A General Purpose private CA will cost you $400/month! I have not tested the option to use the short-lived certificate mode, and it certainly would not work for a production deployment as the certificates would expire too quickly.
For CA type, select Root
Fill in your Subject DN options as you please with Organization, Organization Unit, Country, State, Locality and Common Name.
Set your key algorithm as desired, e.g. RSA 2048

Next we need to setup a Certificate Revocation List distribution point so we can have a place to publish certificates that are no longer valid. All properly configured CAs must have a CRL location for the CA to be trusted.
Check the boxes for Activate CRL Distribution and Create a new S3 bucket.
Enter a new bucket name in the S3 bucket name field.
You can leave the other CRL settings as default.
Add tags as desired
Under CA permissions options, ensure that the check box for Authorize ACM access to renew certificates is checked.
Finally, check the box acknowledging the pricing for ACM and click the Create CA button.

Our Root CA is now created, but to finish activating it we need to Install a CA Certificate. From the Actions menu in the top right corner of the Certificate Authority homepage, you should choose Install CA Certificate. On the resulting screen accept the default 10 year validity period and SHA256 signing algorithm and click the Confirm and install button.

After completing these steps, you now have a functioning Root CA that is able to start issuing private certificates.

Important Note on CA Hierarchies

From a PKI best practices perspective, you normally would never issue client certificates from your Root CA. Instead you would setup a subordinate "Issuing CA" that trusts the Root CA and that would be used to issue all your client certificates. However in the interest of simplifying this blog post and avoiding the additional cost of running multiple CAs, I'm going to skip that step here and move on to configuring IAM Roles Anywhere. Just be aware of this for your production Certificate Authority deployment.

Configure IAM Roles Anywhere

Now that we have a functioning Certificate Authority, let's get into setting up IAM Roles Anywhere. To get started with this task, navigate to the IAM service page in your AWS Console. Next, click on Roles on the left side navigation bar. At the bottom of the resulting page you will see a section for Roles Anywhere. Click the Manage button to get started setting it up.

Establish Trust Anchor

The Roles Anywhere page has a very nice Setup wizard that will walk you through the process. The first thing we need to do is create a trust anchor to the Private CA we setup in the previous step. Click the orange Create a trust anchor button to get started.

You will be taken to the Create a trust anchor wizard page.

First, enter a friendly name for your trust anchor.
Next you will choose your CA source. Here is where you can specify an external CA to trust via uploading a certificate bundle, or you can go with AWS Private Certificate Authority, which is the default option.
Below in the next section you will see a list of AWS Private Certificate Authorities in the account. Click the radio button next to the root CA we created in the previous step.
Leave notification options as default and add tags as appropriate.
Finally, click the orange Create a trust anchor button to complete the process.

Configure Role

After you create your trust anchor, the Setup wizard will move to the next step, Configuring roles. If you have an existing role, you can easily configure it for Roles Anywhere by adding the required trust statements to the role. For our demo, let's make things simple and create a new role for use with Roles Anywhere. You can do this by clicking the Create a new role button.

Clicking that button will take you directly to the IAM Role creation wizard. Step 1 is Select trusted entity.

Click the button next to AWS Service.
From the Use case drop down menu, select Roles Anywhere.
Click Next.

From here we need to add permissions to the role. This will control what API actions the external system is able to use via Roles Anywhere.

You can attach any existing IAM policy in your account, AWS Managed policies, and even attach multiple policies.
For this demonstration I'm going to grant the AmazonS3ReadOnlyAccess managed policy and click Next.

To complete the wizard, give your new Role a name and description, review the trust policy and permissions you just setup, assign tags and then click the orange Create Role button.

Configure a Profile

After you role is created, navigate back to the Manage Roles Anywhere screen. The final setup step is to create a Profile that will associated your IAM Role for use with Roles Anywhere using a role session policy. To complete this step, click the button that says Configure a Profile:

Time for yet another wizard page! Here you will complete the following steps:

Give your Profile a name
In the Roles section, chose the role we created in the previous step
In the Session Policies section, you can optionally limit what permissions are granted by the role for the session created via Roles Anywhere. For our case we will remove the default inline policy that is present, as it grants ALL access.
Apply Tags
Finally click the Create a profile button to finish the wizard.

At this point we have successfully completed all the setup tasks to get our IAM Roles Anywhere backend configured.

Configuring the External System

With our Roles Anywhere backend up and running, let's move on to configuring the external system for access. The first step for this is to issue an x.509 certificate for our external system to use for authenticating to IAM Roles Anywhere.

Generate Client Certificate

In the AWS console, navigate back to Amazon Certificate Manager service. From the service home page, click Request Certificate on the left side navigation bar (or click the big red orange button on the right):

Under Certificate type, choose the option Request a private certificate and click Next:

You guessed it, it's wizard time! From this wizard we will be completing the required properties for our certificate request (CSR).

In the Certificate Authority drop down menu, select the private CA we created earlier.
Enter your domain name for the certificate in the Domain Name section.
Choose your Key algorithm. In this case I'm going with the default of RSA 2048.
Add tags as desired.
Under Certificate Renewal Permissions, click the box to acknowledge that ACM will need permissions to rewew this cert. (you may remember we already granted this permission when we setup the CA originally).
Click the orange Request button to complete the request.

Export the Certificate

Your certificate will get created and after a few moments the certificate status should update from Pending validation to Issued in the Certificates list. Once the certificate is issued, we need to download it so we can place it on our external system. Click on the certificate in the list and from the details page click the Export button.

On the Export Certificate page you will need to add a passphrase so that the private key can be encrypted. Next, click the checkbox to acknowledge that you will be charged for exporting this certificate:

After you click Generate PEM encoding, the service will return to you three items, each with Download links:

Certificate body
Certificate chain
Certificate private key

Now you will need to download each of these to your local system. You can do this by either clicking the download buttons for each item, or creating text files on your local system and copying the text into them. Place all these files in a folder on your system. After downloading the files, you will need to rename certificate.txt to certificate.pem and private_key.txt to private_key.pem.

Decrypt the Private Key

Once the files are downloaded, next we need to decrypt the private key using the passphrase we entered when we performed the export. On a Mac or Linux system we can do that easily with openssl. Here is the command to use:

openssl rsa -in private_key.pem -out decrypted_private_key.pem

Openssl will run and prompt you to enter the passphrase you created, and then decrypt the key to the designated output file:

Enter pass phrase for private_key.pem:
writing RSA key

Success! We've now created a new certificate for our system in Amazon Private CA, exported it our system have it ready for use with the IAM Roles Credential Helper.

Install IAM Credentials Helper

As mentioned before the IAM Roles Credentials Helper is a downloadable tool maintained by AWS. The tool is used to create a SigV4 signature with your certificate and make a call to the Roles Anywhere endpoint to obtain session credentials. Next, the Roles Anywhere endpoint then returns the credentials to the calling process in JSON format. You can download the latest version of the tool from the IAM Roles Anywhere User Guide, where there are packages for Linux, Windows, and Darwin (MacOS). Download the helper and place it in the same directory where you placed your certificate and private key. First, you'll need to make the package executable before you can use it:

chmod +x aws_signing_helper

With this last step done, we've completed all the end to end setup work needed to use Roles Anywhere. To wrap up the article, I'm going to demonstrate how to make a CreateSession API request using the credentials helper.

Making the Connection

There are three things we will need to have in order to use the AWS IAM Roles Anywhere Credentials Helper to make a CreateSession request.

Roles Anywhere Trust Anchor Amazon Resource Name (ARN)
Profile ARN
IAM Role ARN

The first two items can be obtained from the IAM Roles Anywhere management page, while the IAM Role ARN can be found by examining the details of the Role in the IAM Roles page. You will also need the filenames for your certificate and decrypted private key that we downloaded in the previous step.

Manually request and set credentials

You will combine all these items as the options for our rather long aws_signing_helper command:

./aws_signing_helper credential-process \
--trust-anchor-arn arn:aws:rolesanywhere:us-east-1:111111111111:trust-anchor/49d455a6-deec-4cfc-9c12-2c75217ea49a --profile-arn arn:aws:rolesanywhere:us-east-1:111111111111:profile/d0119c28-ecfa-4ad2-88c7-cacfd61bb268 \
--role-arn arn:aws:iam::111111111111:role/cloudyadvice-roles-anywhere-role \
--certificate certificate.pem --private-key decrypted_private_key.pem

Note I've modified the account numbers here, you should replace with the proper values from your ARNs. If all goes well, AWS will return you an Access Key ID, Secret Access Key, and Session token in JSON format, like so:

{"Version":1,"AccessKeyId":"ASIAVFPLXXXXXXXXXXXX","SecretAccessKey":"+ZCVCbiRHXwCOBZCi0JVv/XXXXXXXXXXXXX","SessionToken":"IQoJb3JpZ2luX2VjEC4aCXVzLWVhc3QtMSJGMEQCIFgE1/1OZjanZDOiMnVkxgKMXsoZVlQl3QJy2POXadb/AiAm7EqPHvXslugCqcaS6wCtquf/XIaMTg1YbJg3ywfXDCqIBAhmEAIaDDM1NTM2NDgyOTg2NCIM0BYAV8uBV1afYXAuKuUDMXS8VhT4SnBqg+S4yqtOj0TSZdbp/z4AKJeovB6u9g/5f2SA1tAG3hkByq/PEiuLFGN0kIgkWT66CPDNHS5iuPVXhiHIghaNqVDXzGXvWcxp5SPLbIIAQxfyQTmNiBhqg82VDaEUEutSh2kd8CdvTtlP39tEFE0q8poWbNCnDCwg1Wj50LaH7IQb3jYoTxHAnHEkTmtlqmoeYIM7UVc1RQ2Fox37nnMZVqaeY0tCyxyfmQn8tCEmaFRTxYWweCfxB7WIjptGwQGUchEBbISHnJulKU9Pv8lgct3HENWqyQcVKSWRZDvIAuFjpfq8cahcAVHrzM20KNMO3OxfZyXhBgSNalxO6c0pGoBDgmlH1o8yEVky2GSzYncSwyL++KPAzb/+FeNiT4aJeSyd3ns0Ewp/s30JbC8fuID6UuuuSfNa7VetSFBanSfIkjDhUnefjyYFyz2f+7k2X5HPYeAmT/zjwOGpNY7KcIJqZQHERhaMWn5Gy0RvjvAoVVAgyEwqwBff76TEwmxh6DMN4o/LRuOnflommJojT0BmVtfBD5Ykp53xZXPCQctF7ijN40Ud18w8ZIBF/6QUAdQe1Lp32kIH9HPq3Lg4YlvJzdsMX6BXemDWHCFLCE1BfFQXXBRRs0p8Ci8wgueaqgY6lAHD/crexcd3uBsB87czp6kyYV8iH4BMPPpxbIs3fAgjOkJMta2pROF72YKR9FRF9ixciFkBCpnACjWGSDSuwx/+elwoDJ3zlCOad7SYX0J8CBxAKKAQ564MOiYrB9wdf3N2sQd+dKJkmPSZxEN1sWnMnD/D+HrWB7A0ULsW3phER50gisns8G/KsE4q4gtz1QOrOIP6","Expiration":"2023-11-04T22:11:30Z"}%

As you can see, Roles Anywhere has sent you back a send of temporary credentials in the form of an Access Key ID, a Secret Access Key, and a Session Token. You can simply use these values returned here to set as temporary environment variables for your AWS credentials, and then issue calls to AWS using those environment variables. For example:

export AWS_ACCESS_KEY_ID=access-key-id-value
export AWS_SECRET_ACCESS_KEY=secret-access-key-value
export AWS_SESSION_TOKEN=session-token-value

Once that is done you can issue AWS CLI commands, e.g. get a list of S3 buckets:

aws s3api list-buckets

Which should return you a list of S3 buckets in JSON format.

Leverage Roles Anywhere in AWS CLI Config

Manually requesting credentials is a great way to test, but rather laborious to do on a regular basis. Fortunately you can leverage the aws_signing_helper as a custom credential_process for the AWS CLI. To do this, you simply need to add it as a new profile entry in your AWS CLI credentials file to invoke it. First, copy the certificate files and the aws_signing_helper executable to the same directory where your AWS CLI is located, typically ~./aws. Next, you add a new profile to your ~/.aws/config file that leverages IAM Roles Anywhere. The profile entry should look something like this:

[profile rolesanywhere]
credential_process = /users/cloudyadvice/.aws/aws_signing_helper credential-process --trust-anchor-arn arn:aws:rolesanywhere:us-east-1:111111111111:trust-anchor/49d455a6-deec-4cfc-9c12-2c75217ea49a --profile-arn arn:aws:rolesanywhere:us-east-1:111111111111:profile/d0119c28-ecfa-4ad2-88c7-cacfd61bb268 \
--role-arn arn:aws:iam::111111111111:role/cloudyadvice-roles-anywhere-role \
--certificate certificate.pem --private-key decrypted_private_key.pem

Obviously, you would need to replace the dummy values shown here with the proper values from your own environment. Once you've added this new profile to your ~/.aws/config file, you can call it for use with any AWS CLI command with the --profile flag, for example:

aws s3 ls --profile rolesanywhere

Now AWS CLI will leverage Roles Anywhere to mint dynamic temporary credentials when you execute commands using this profile switch. Totally awesome!!

Closing Thoughts

I was so excited when IAM Roles Anywhere was announced because it solves a real world problem I often have to deal with in my day job. As a result I was glad to finally got around to testing it out and writing up this blog post.

Obviously, this was a fairly long blog post covering a pretty complicated setup. AWS IAM by itself is a very complicated service. Then when you add public key infrastructure into the mix, things get even more complicated. To help with this complexity, I've done my best here to walk through the steps in an easy to follow fashion so you can start experimenting with this powerful service.

I hope that you will see the benefit of this great service and use it to make your own AWS environments more secure. Are you already using IAM Roles Anywhere in your environment? Share your experiences and thoughts in the comments!

References

AWS Security Blog - IAM Roles Anywhere Launch Post

AWS Security Blog - IAM Roles Anywhere with an external certificate authority

AWS Security Blog - Enable external pipeline deployments to AWS Cloud by using IAM Roles Anywhere

AWS IAM Roles Anywhere User Guide

YouTube - AWS re:Inforce 2023 Managing hybrid workloads with IAM Roles Anywhere, featuring Hertz

Hacker Loi YouTube - AWS IAM Roles Anywhere Full Tutorial

GitHub AWS repo - roles anywhere credential helper

Build Easy Chargeback Reports using AWS Cost Categories

Mike Graff — Thu, 06 Oct 2022 15:08:09 +0000

Introduction

At my day job, we first started using public cloud back in 2015, and at the time we were really just starting to crawl with cloud financials. Our finance department was not a fan of chargeback and as a result we ended up partnering with a third party reseller to help us with our cloud billing. We built a complicated system where we leveraged the reseller for consolidated billing, and each business team would issue a separate PO to the reseller for their AWS usage. The reseller was using a commercial tool called CloudHealth to slice and dice our bill and generate 70+ invoices each month back to us for each team.

Fast forward to the second half of 2021, and at this point our increased maturity around managing public cloud meant that it was time to start walking on our own and bring the cloud billing in-house. In April 2022 I wrapped up a six month project that had three main objectives:

Develop and implement new internal tools to replace the capabilities of CloudHealth
Design and implement a new cost allocation scheme for our AWS and GCP bills
Negotiate a new EDP with AWS and migrate away from the third party reseller.

The result? A lot less paperwork in the form of purchase orders, invoices, and payments, more transparency on AWS costs for our account holders, and stronger FinOps muscles for our team.

Part of my solution was to implement the CUDOS Cost Intelligence Dashboards to replace the reporting capabilities of CloudHealth. You can also read about how I implemented CUDOS for my enterprise by reading these earlier blog posts:

Implementing CUDOS for the Enterprise – Part 1
Implementing CUDOS for the Enterprise – Part 2

In this post, I’m going to talk about our solution for creating a chargeback report for allocating our AWS spend back to business groups. I will provide a deep dive into the AWS Cost Categories tool, which we are using for cost allocation. I’ll talk about how it works, how we are using it at my company, and some of the lessons learned along the way.

AWS Cost Categories

AWS Cost Categories is a feature within AWS’ Cost Management suite that allows you to group cost and usage information into meaningful categories based on your needs.

You can map AWS charges in whatever way makes the most sense for your business, for example business units, environments, product lines, etc. In our case, we were primarily looking to catagorize charges based on internal cost centers defined by our Finance group.

After you create these cost categories, they become part of the Cost and Usage Report and are thus available for use in other parts of the Cost Management suite like Cost Explorer, AWS Budgets and my personal favorite the CUDOS Cost Intelligence dashboards.

Implementation

So how to get started?

First, think about what categories might make the most sense for your organization. Your groupings should align with whatever dimensions you are looking to aggregate your spend data For my situation, our primary grouping was Cost Center.

Also be aware that you can build as many different cost categories as you like. Maybe you want cost center and business group and product line…no problem. There is a limit of 50 categories per payer account but there is no charge for this service

Creating your Cost Category

Start by logging into your payer account and navigating to the Billing Dashboard. From there choose Cost Categories from the menu on the left hand side:

You will be taken to the Cost Categories home page. To create your new Cost Category, click the orange Create Cost Category button, which will launch the Cost Category creation “wizard”.

Step 1 – Name

Step 1 of this wizard is to give your Cost Category a meaningful name, e.g. Cost Center, and apply any desired tags to the Cost Category. Then click Next to proceed to the next step of the wizard.

Step 2 – Define Rules

Once you’ve created your new category, you then need to define rules for how the tool will categorize your cost. The workflow is pretty simple: you create your category values, and then define rules for each value. Rules determines which charges will be assigned to that category.

These rules can be based on several different dimensions like AWS account, a specific AWS service or charge type, a tag, or even other cost categories.

Rules can also have multiple dimensions in them, for example “all charges from account X for AWS S3”. You can have up to 100 rules per cost category and they are executed in priority order from top to bottom. At the bottom of the ruleset you can also define a Default category to capture any charges that are not captured by your defined rules. I recommend you do this to avoid any uncategorized charges.

For my use case the requirements were pretty simple, I built my cost center category based on AWS account boundaries, each cost center value had one or more AWS accounts assigned to it.

Step 3 – Define Split Charges

If you want to take a specific service charge and spread it across multiple categories, you can leverage the split charge feature. To achieve this you define a category rule to capture the charges that you want to split, and then use split charge rules to decide how you want the charges to be spread across all the remaining categories. You can read more about this particular feature in the AWS user guide on the topic.

Save and Finish

Once you have finished defining all your category rules and (optional) split charge rules, click the orange Save Changes button to finish creating your cost category. You will be returned to the Cost Categories home page and your new Cost Category will show a status of Processing while the service crunches through your CUR data. When finished, the status will change to Applied. The time it takes to show a status of Applied very much depends on how many accounts you have and where you are in the billing cycle for the month. In my environment I find it can take anywhere from 6-10 hours for a category to change from processing to applied.

Viewing Cost Category Data

Once your cost category shows a status of applied, you can then view it in the cost categories console. Simply click on the cost category you wish to view. On the resulting page you are presented with a graphical overview of the category distribution, a table with absolute amounts and percentage breakdowns for every category as well as any uncategorized costs. Finally, you can download a CSV file for analysis with your spreadsheet tool of choice.

Caveat on Cost Categories reports

I want to call out one annoyance with the native Cost Categories reports that I discovered after I started using the tool. The reports that are shown in the cost categories console are all based on Amortized Charges rather than the Unblended costs that are on your AWS invoices. (Unclear on the difference? Check this blog post).

This caught me out when I first tried reconciling my cost categories report against my invoice. Most of my category values were off by a few hundred dollars from the invoice. I spent a week or so trying to figure this out before my AWS TAM let me know that Cost Categories is only capable of showing Amortized Charges. Sadly there is currently no way to change this inside the cost categories tool. You can bet I’ve got my name on a PFR to fix this though! I’ll talk about how I worked around this problem in the next section.

Tips & Tricks

Here are a few tips and tricks I’ve picked up over the past few months of working with Cost Categories that you might find useful.

Multi-level hierarchies

I mentioned before that one of the rule dimensions you can use in your category values is other cost categories. This allows you to leverage groupings you have already created to build hierarchical structures.. For example in my environment I built my initial “Cost Center” cost category based on AWS account values. From there, I built a new cost category called Business Group that grouped multiple Cost Centers together into each business group.

Cost Categories as Code

Once I had been working with cost categories for a while I started to find the graphical rule builder kind of tedious. I was happy to find that you also have the ability to edit your categories as a JSON file. This makes editing of your category rules a lot faster. Not to mention, it also gives you the option to place that JSON file in a version control system. From there, you can manage the ruleset like any other code in your environment. For you Terraform pros out there, cost categories is also supported natively in the Terraform AWS provider!

Cost Categories in Cost Explorer

So as I mentioned before, once created Cost Categories become available for use inside of Cost Explorer. You can use it for filtering and grouping on spend data just like any other dimension available inside cost explorer.

I leveraged this capability to get around the amortized costs limitation of the native cost categories report. Cost Explorer allowed me to build a report that I could use for my monthly invoice reconciliation process. You can perform the following steps to get a downloadable file to use for cost allocation.

Use Cost Explorer to generate a monthly report grouped by my CostCenter cost category.

Make sure to go into advanced options and choose the setting to have cost explorer show costs as unblended costs.

Once you have the data the way you want it, use the Download CSV function to get a CSV file. From there, you can then use that file as input data for your cost allocation process.

Cost Categories and CUDOS

As I mentioned at the start of this article, once Cost Categories are created they become part of your CUR. As a result they become available for use in other AWS Cost Management tools. They can even be used by other third party tools that are leveraging the CUR. In my case this allowed me to bring my Cost Categories into the CUDOS Dashboard. As a result, I was able to use the CostCenter cost category I created to build new views into CUDOS. Below are some example screenshots where I’m leveraging Cost Categories to build new views of my CUR Data.

First is a view of monthly invoiced spend grouped by individual cost center. This allows you to see a month over month trend of how much each cost center is spending.

Here is a view showing EC2 running hours for the top 10 cost centers grouped by consumption type. In this way you can see how each cost center is performing with respect to targets for using savings instruments like reserved instances, savings plans, and spot instances.

Finally, below you can see an analysis of daily data transfer out grouped by Cost center. This allows you to see which cost centers are responsible for the bulk of your data transfer out charges.

Conclusion

In this blog post, I’ve provided you with a dive into the AWS Cost Categories feature and how to set it up. I’ve also shared how we are using it in my company and some of the lessons learned along the way. You can leverage this feature to build your own chargeback reports based on whatever model makes the most sense for your organization. You can even use the split charges feature to take shared costs and spread them across different business units.

I hope you have found this post useful. Have you tried using Cost Categories in your environment? Please share your experiences or give me your feedback in the Comments section.

UPDATE - Cost Categories can now be Retroactive

An exciting update to the Cost Categories service was announced on September 27, 2022. Cost Categories now supports retroactive application of cost category rules. When creating a new cost category, you can apply cost category rules starting any month from the previous 12 months. Similarly, while editing a cost category, you can apply cost category rules starting any month from the previous 12 months. These changes will also be reflected in AWS Cost Explorer.
More details at this blog post.

References

AWS Cost Categories – Launch Blog Post

AWS Cost Categories User Guide

CloudyAdvice – Using AWS Cost Explorer for Forecasting

AWS Cloud Financial Management

CFM Talks Video – How to Design your AWS Cost Allocation Strategy

FinOps Foundation

Easily Query your Cloud Inventory with Steampipe

Mike Graff — Tue, 27 Sep 2022 22:05:21 +0000

Cloud Inventory is a Challenge

Anyone who has worked with AWS for a while will be quite familiar with the difficulties of getting a complete picture of the inventory of resources in your accounts. While AWS has made some limited strides in this area with tools like the EC2 Global View and the ability to get a count of all VPCs in the VPC dashboard, the bottom line is that AWS does not provide a comprehensive inventory solution for an AWS account or group of accounts.

While there are many excellent third party commercial tools you can use to solve this problem, not everyone (including me!) has the budget available to pay for these tools. So for the past month or two I've been spending time looking at some open source tools to add to my team's toolchain to help us with the inventory problem. One of the tools I've been looking at is an interesting solution called Steampipe. In this post I'm going to dive into this tool and share some of the things I've learned about it.

Steampipe - Query your cloud inventory like it's 1992

Steampipe.io is an open source tool that is maintained by the commercial tool company Turbot. In a nutshell, the software allows you to query your favorite cloud services with SQL. By providing a consistent command line interface that works across multiple IaaS, PasS and SaaS services, Steampipe aims to reduce the time wasted on context switching between different cloud provider's native interfaces. As the end user, you write your queries in a consistent SQL syntax, and Steampipe translates that query into native API calls that are executed in real time against the cloud service's API.

Interaction with various cloud service's APIs is handled via Steampipe plugins for each service. As of this writing there are 86 different plugins available on the Steampipe Hub, including plugins for all the major cloud players (AWS, Alibaba, Azure, Cloudflare, GCP, Digital Ocean, IBM, Oracle, Heroku), SaaS services (Datadog, PagerDuty, Salesforce, Stripe, Twilio, Zendesk, Zoom), as well as some more intriguing options like VMware vSphere, Terraform, Reddit, and Slack.

Getting Started

The easiest way to get started playing with Steampipe is to install it on your local workstation. The Steampipe Downloads page provides step by step instructions for installation on MacOS, Linux, and Windows operating systems.

On a Mac with the Homebrew package manager installed, the process could not be any simpler:

Tap Turbot's Cask: ~$brew tap turbot/tap
Install Steampipe: ~$brew install steampipe
Validate the installation is working by checking the installed version of Steampipe: ~$steampipe - v

Once you have the core Steampipe engine installed, you need to install plugins for whichever cloud services you want to query. First, let's install the Steampipe plugin which will allow us to query Steampipe components, such as the available plugins in the Steampipe hub.

~$ steampipe plugin install steampipe

steampipe            [====================================================================] Done                

Installed plugin: steampipe@latest v0.5.0
Documentation:    https://hub.steampipe.io/plugins/turbot/steampipe

Next, let's install the AWS plugin so we can start querying our AWS environments. Use the following command:

~$ steampipe plugin install aws

aws                  [====================================================================] Done                

Installed plugin: aws@latest v0.78.0
Documentation:    https://hub.steampipe.io/plugins/turbot/aws

Now that we have our AWS plugin installed, we need to do configure some credentials for the plugin so we can start querying our AWS accounts.

Configure Credentials

Basic Setup

When you install the Steampipe AWS plugin, the tool will automatically create a configuration file for you located at the following path: ~/.steampipe/config/aws.spc

Within this configuration file you can setup one or more AWS accounts to query with Steampipe. You can setup credentials directly in the aws.spc file if you want, but even more handy is the fact that Steampipe can refer to credential profiles you've already setup for AWS CLI. Best of all, you can even use AWS SSO profiles with Steampipe!

Credentials statements in my aws.spc file look like this:

connection "itprod" {
  plugin = "aws"
  profile = "sso-itprod"
  regions = ["*"]
}
connection "itshared"
  plugin = "aws"
  profile = "sso-itshared"
  regions = ["*"]
}

As you can see, each connection entry in my aws.spc starts with a name definition, followed by which plugin you are using, the profile name in my ~./aws/config file, and finally a region specification. You can put a list of individual regions here or use * to query all regions simultaneously.

Advanced Configuration

If you have a multi account setup, and you've configured multiple connection entries in your aws.spc file, you can also create what is called an "aggregator connection" which allows you to query against multiple connections from a plugin in a single query.

connection "aws_all" {
  type = aggregator
  plugin = aws
  connections = ["itprod", "itshared"]
}

You can even use wildcards in your aggregator specification, like so:

connection "aws_it_all" {
  type = aggregator
  plugin = aws
  connections = ["it*"]
}

Another handy capability comes into play if you are running Steampipe on an EC2 instance inside your AWS Account. If you have an IAM Instance profile attached to the instance with appropriate permissions, Steampipe will automatically use that IAM role with no other credentials being configured.

Now that we have some credentials configured for Steampipe, let's see what we can do with this tool!

Basic Queries

The Steampipe AWS plugin organizes the many AWS services into discrete tables that you can then write queries against. As of this writing there are 344 discrete tables defined in the AWS plugin, you can review them all here. You can run queries in two different modes with Steampipe:

Enter an interactive query console by entering the command steampipe query
Run a specific query directly by entering it all in a single command: steampipe query "select * from cloud"

I believe it is easier to get started using the interactive query console as it provides you with hints and autocomplete as you write your queries. Let's enter the console and see what commands are available:

➜  steampipe query
Welcome to Steampipe v0.15.4
For more information, type .help
> .help
Welcome to Steampipe shell.

To start, simply enter your SQL query at the prompt:

  select * from aws_iam_user

Common commands:

.help     Show steampipe help                           
.inspect  View connections, tables & column information 
.exit     Exit from steampipe terminal                  

Advanced commands:

.cache [mode]        Enable, disable or clear the query cache                                                     
.clear               Clear the console                                                                            
.connections         List active connections                                                                      
.header on|off       Enable or disable column headers                                                             
.multi on|off        Enable or disable multiline mode                                                             
.output [mode]       Set output format: csv, json, table or line                                                  
.quit                Exit from steampipe terminal                                                                 
.search_path         Display the current search path, or set the search-path by passing in a comma-separated list 
.search_path_prefix  Set a prefix to the current search-path                                                      
.separator           Set csv output separator                                                                     
.tables              List or describe tables                                                                      
.timing on|off       Enable or disable query execution timing

As the help file suggests, you can use the .tables command to get a list of all the tables available in Steampipe:

Steampipe .tables command

You can use the .inspect command to view details on connections and tables. Inspect is particularly powerful as you can use it to drill into a specific table to see what fields are available. For example here's a screenshot showing the .inspect aws_vpc command result:

Steampipe .inpsect aws_vpc command

Once you've seen the connections and tables, you can get an idea of what types of queries you can run. As the help file suggests, you can start by getting a list of all the IAM users in all your configured accounts with the command select * from aws_iam_user which returns a table like this:

Steampipe query for all IAM users

If you have an aggregator configuration setup, you will get results for all configured AWS accounts by default. If you want to limit your scope to a specific AWS Account, simply prefix the table name with the connection name e.g. select * from itshared.aws_ec2_instance:

Steampipe query for EC2 instances in a specific account

Advanced Queries

If you review the table definitions published on the AWS plugin page at Steampipe Hub, you will see lots of more advanced examples of what you can do with Steampipe. For example, let's say you want to get a count of IAM access keys that are between 45 and 90 days old:

Steampipe query of access keys between 45 and 90 days old

Or perhaps a list of your S3 buckets that are not encrypted:

Maybe you'd like to get a quick count of EC2 instances grouped by region across all your accounts?

Steampipe query counting number of EC2 instances by region

Lastly, let's get really fancy and create an age table of all our EC2 instances:

Steampipe EC2 instance age table

Wrap-up

As you can hopefully see from this brief introduction, there is a ton of things you can do with this tool, especially leveraging the power of SQL. While I personally do not have a lot of SQL experience, I was still able to get up and running pretty quickly following the excellent documentation and examples provided by the Steampipe project.

In the future I hope to do a follow up post on some of the more advanced capabilities of Steampipe including the ability to serve custom dashboards based on the Steampipe engine. In the meantime, if you've tried this tool yourself, please share your experience with me in the comments section.

Happy querying!

Implementing CUDOS for an Enterprise - Part 2: Filtering Data using Row Level Security

Mike Graff — Fri, 08 Jul 2022 20:21:53 +0000

Introduction

In my first post on the CUDOS solution, I showed you how you can implement implement SSO access to CUDOS using AWS Single Sign-on service. Leveraging this capability you can enable users to sign in to QuickSight using their corporate credentials and avoid having to give them full access to the AWS console. Now that we've got our users setup via SSO, let's leverage these same user objects to enable additional functionality.

By default, when a user accesses your CUDOS dashboard, they will have the ability to see billing data for every account in your AWS organization. If you have a large environment with dozens or even hundreds of AWS accounts, this may be less than desirable. Users may have a hard time figuring out how to just see the data for accounts they are interested in, or as an admin you may have a desire to limit what data they can see.

In this article, I'm going to show you a method to limit what account data any given user can see in the CUDOS dashboard. Once implemented, the user will only be able to see billing data for the accounts that you have granted them access to.

The Solution

There are two primary components to this solution. The first is QuickSight Groups. QuickSight groups are just what you would expect, a feature that allows you to organize sets of users into groups that make it easier to manage access and security. When I first started working with QuickSight, groups were an absolute pain to manage. Originally they could not be managed via the AWS console...in fact they were not even visible in the console and could only be created, managed or listed using AWS CLI commands. Good news for you however is that as of March 2022 you can now manage QuickSight Groups in the QuickSight console.

The second component of our solution is something called row-level security (RLS). This is a feature of the Enterprise Edition of QuickSight that allows you to restrict access to a dataset using a set of dataset rules. Once implemented, dataset owners can still see all the data, but dataset readers will be restricted based on the criteria in your ruleset. Access to a dataset can be restricted by user/group based rules, tag-based rules, or both. Tag based rules are really targeted towards anonymous users if you are doing something like an embedded dashboard, so we will be focusing on user/group based rules for our architecture.

The diagram below outlines how the components outlined below come together for our solution.

QuickSight users are placed into QuickSight Groups
An input file is prepared in CSV format containing the dataset rules and stored in an S3 bucket
The input file is imported into a QuickSight dataset
The rules dataset is used to apply row-level security against the appropriate CUDOS datasets

Now that I've given you a high level overview of the solution, let's walk through the implementation in detail.

Implementation

Step 1 - Create QuickSight groups

The foundation for implementing your filtering is the creation of one or more QuickSight groups. These groups will be used later in our row-level security dataset to tell QuickSight how to filter data set views. You will need to create a QuickSight Group for each set of accounts that you want to group together.

To create a QuickSight group:

Sign in to the AWS Console in the account where you are hosting QuickSight, navigate to the QuickSight service.
On the QuickSight home page, click the menu in the upper right and select Manage QuickSight.
On the QuickSight Management page, select Manage Groups from the left hand menu.
On the Manage Groups page click the blue New Group button.
On the Create a New Group screen, enter a Group name. Note the group name cannot contain any spaces or certain special characters. Optionally enter a group description and click the blue Create button.

6. After you create your new group, you will be take to the group membership page where you can now add QuickSight users to the group. Click the blue Add User button and add new users by user name or email address.

Repeat this process for each group that you need to create. My recommendation is to create groups for each logical grouping of accounts in your AWS Organization, could be by project team, business group, or whatever works best for you. Note also that a user can be a member of more than one QuickSight group, for example if you want to give them access to more than one set of accounts. In addition to team specific groups, I recommend creating a group called "FullAccess" for any QuickSight users that you want to be able to see all your AWS Accounts in the dataset. Once you've created your QuickSight Groups, it's time to setup your rules mapping file.

Step 2: Row Level Security Rules Input file

As documented in the QuickSight user guide, you can setup RLS with user-based rules by creating a query or a file that feeds the rules dataset. I'm not really a SQL guy, so I find the file based option to be simplest to setup and it makes it easy for me to change my ruleset as needed. I just edit the file and replace the version stored in the S3 bucket (more on this process later). To get started, create a new comma-separated value (CSV) file with two columns in it. Column A header should be "GroupName" and column B header should be "account_id".

Next, we will populate the GroupName column with the QuickSight Group names we created in step 1. In the account_id column, you will enter the AWS Account number you want that group to have access to, enclosed in double quotes. If you want to give a single QuickSight group access to more than one account, you will need to create a row for each account_id, all using the same GroupName. I was originally told by some folks at AWS that you could enter multiple account ids in a single column, separated by commas, but I could never get that to work and settled on this file format through trial and error. To grant your "FullAccess" group access to all data in the dataset, create a row with that group name and leave the account_id column empty. See the screenshot at right shows an example of what your file should look like.

Save your file, giving it a name like "cudos_dataset_rls.csv" or something similar. With our file created, we are now ready for the next step, creating the rules QuickSight dataset.

Step 3: Create a CUDOS RLS Rules Dataset

We need to create a new QuickSight Dataset to store our RLS rules so that we can then apply them against our CUDOS datasets. Since it is very likely we will want to make changes to these rules over time, we want to create a dataset that is looking at specific location in an S3 bucket rather than just uploading the file into a dataset. By using the S3 method, whenever we want to update our RLS rules, all we have to do is upload a new version of the file to S3.

In preparation for creating the S3 dataset, upload the CSV file you created in step 2 to a private S3 bucket in the AWS account where you are running QuickSight. This can be a new purpose-created S3 bucket or you can use an existing bucket, for example the bucket you use to store your QuickSight queries results. Then, we need to create an S3 manifest file that will tell QuickSight where our file is located in S3.

An S3 manifest is a JSON file that contains the S3 url of the file location. Create a new file called "CUDOS_manifest.json" and set it up like the example below, replacing "YOUR_BUCKET_NAME" and "YOURFILENAME" with the proper values for your file.

{
    "entries": [{
        "url": "s3://YOUR_BUCKET_NAME/YOURFILENAME.csv",
        "mandatory": true
    }]
}

You can store the manifest file in the same S3 bucket where you placed your CSV file. With the files in S3 prepared, you can follow this process to create the new dataset.

Log in to the AWS Console in the account where you are running QuickSight, navigate to the QuickSight service.
From the QuickSight home page, click Datasets on the left hand menu bar.
On the Create Datasets screen, click S3.
On the Datasets page, click the blue New dataset button.

5. For data source name enter "CUDOS_Dataset_rules" and enter the S3 URL of your manifest file in the "Upload a manifest file" field. Make sure the "URL" radio button is selected and click Connect

6. On the Finish dataset creation screen, click the Edit/Preview data button.

7. In the dataset preview screen, we need to verify the the account_ID field is a String datatype. If it appears as an Integer, change the data type for account_id to String. Follow the screenshots below for how to do this.

It is very important to make this change because it matches the same data type as the account_id field in all of the CUDOS datasets and prevents the loss of any leading or trailing zeros on an account_id. After you've finished changing the field type, click Save & publish in the upper right to save your dataset.

With our rules dataset prepared, we can move on to the last step, applying these rules against the CUDOS datasets.

Step 4: Apply Row Level Security to CUDOS datasets

On each of the datasets that the CUDOS dashboard is using, we will define row level security for the dataset by following these steps:

In the QuickSight management console, choose Datasets from the left hand navigation menu.
Click the dataset name, e.g. "cudos_summary_view"
A dataset details window will pop open showing you more information about the dataset.
Click the button bottom left labeled Row-level security

5. You will be taken to the Set up row-level security screen. Click the double-caret to expand the User-based rules section.

6. You will see that the selected dataset rules is currently set to None, and below you will see a list of all QuickSight datasets in your environment.

7. Click the radio button next to the CUDOS_dataset_rules dataset you created in Step 3 to select it.

8. Click the blue Apply dataset button to apply these rules against your dataset.

9. A confirmation window will open, confirming that the dataset rules will use the GroupName column to find matching rules. Click Apply and Activate.

10. You will be taken back to the row-level security screen, where you should now see your dataset listed under the Selected Dataset Rules.

To ensure the security filtering is consistent across all the CUDOS dashboard tabs, you will need to repeat these steps for all of the CUDOS related datasets. As of this writing, these are the CUDOS datasets that I applied the rules against:

cudos_summary_view
cudos_compute_savings_plan_eligible_spend
cudos_s3_view
cudos_ec2_running_cost
cctd_view
cudos_cur

Test!

Once you have completed applying the RLS ruleset against each of the CUDOS datasets, you should test to confirm that row-level security is working correctly. You can do this by logging into the QuickSight dashboard using a user that is a member of group that has limited access, and validate that the AWS Account IDs shown are only the ones specified in your rules dataset. You should also access the dashboard as a user who is a member of the FullAccess group and validate that all AWS Account IDs are still visible.

Updating RLS rules

Over time you may find that you wish to update your row-level security rules to reflect new AWS accounts or new QuickSight groups. The process to update your rules is simple:

Update your CSV file with the new rules, adding new QuickSight groups or account_id values as needed.
Upload the updated CSV file to your S3 bucket and replace the existing file.
Navigate to the QuickSight management console
Click on Datasets in the left hand menu.
On the datasets screen, click your CUDOS_dataset_rules dataset.
On the dataset summary screen, click the Refresh now button

7. A dialog box will open, click the blue Refresh button to complete the refresh of your dataset.

8. A "Warning" dialog will open, click Ok to confirm the dataset refresh. Afterwards you will get a Success dialog box confirming that the refresh has been queued. Click Ok.

QuickSight will automatically apply the new rules against your CUDOS dataset when the refresh has completed.

Conclusion

In this post, I've shown you a method to filter what data your QuickSight users can view in your CUDOS dashboard. Implementing this feature was incredibly useful in my environment where I have over 150 AWS accounts and I might only want a particular user to be able to view billing data for just a few accounts. It also makes it easier for your users since the data is pre-filtered for them and they don't have to manually filter out accounts they are not interested in.

I hope you've found this article useful, and please leave a comment if you have any questions or suggestions for improvement. Once again, I also have to give a big shout out to Aaron Edell, GTM for Customer Cloud Intelligence for AWS, who guided me on implementing the row-level security solution for QuickSight and answered the many questions I had.

References:

Amazon QuickSight User Guide - Creating and managing groups in Amazon QuickSight

Amazon QuickSight User Guide - Using row-level security (RLS) in Amazon QuickSight

Implementing CUDOS Cost Intelligence Dashboards for an Enterprise

Mike Graff — Fri, 29 Apr 2022 22:29:06 +0000

Part 1 - Deployment and SSO Integration

At my day job I recently led a project to move away from CloudHealth for cost reporting and implement some new tools to replace the functionality of CloudHealth. CloudHealth is a great tool but my company no longer wanted to pay a percentage of our spend just to get the reporting capabilities, especially as our spend continues to grow. While I found that native tools like AWS Cost Explorer and Cost Categories met a lot of our requirements, I wanted some additional analytics tools that allowed us to see billing data across multiple AWS accounts in a single dashboard, as well as provide a means to restrict users to only be able to see certain AWS account

To solve this requirement, my AWS Technical Account Manager let me know about a set of open source QuickSight Cost Intelligence dashboards developed by AWS. The suite includes the CUDOS Dashboard, Cost Intelligence Dashboard, Trusted Advisor Dashboard, and Trends dashboard and are all built using a combination of AWS services including the Cost and Usage Report, S3, Glue, Athena and QuickSight. You can read up more on these offerings and the underlying architecture in this AWS blog post.

This article is not going to be a walkthrough on how to setup these dashboards from scratch. There is already an excellent guide on this in the form of a series of AWS Well Architected Labs that guide you through the whole process. You can also use this demo site from AWS to get hands on with the dashboards to check out their capabilities. I suggest you go through these labs to get the dashboards setup and then come back here and read my posts for a series of tips for how to implement this solution for a large enterprise supporting multiple different cloud teams.

For this series of articles I'm going to be primarily focusing on the CUDOS dashboard which looks like this:

Part 1 of this series will be on Implementing AWS SSO for QuickSight. Future posts will cover topics like customizing and publishing dashboards to users, as well as how to implement row level security in QuickSight to filter access for different account owners.

Implementing AWS SSO for QuickSight

To make things simple, I deployed my QuickSight dashboards in my AWS organization payer account aka the management account. However I don't want to allow everyone who needs access to these dashboards to login to the AWS Console on my management account. Instead, I wanted them to login to QuickSight directly via SSO. In my environment I already have AWS SSO implemented to allow users to access the AWS Console using their corporate credentials, so I was able to extend that capability to provide a login for QuickSight. If you need guidance on setting up AWS SSO for the first time, I refer you to AWS' documentation on the subject.

AWS also provides pretty good documentation on how to implement various IdPs with Amazon QuickSight, but in this post I've distilled down a step-by-step guide for implementing AWS Single Sign-On as your IdP for QuickSight (assumes you already have AWS Single Sign-On implemented for AWS Console logins). To complete the setup, there are four tasks we much complete:

Configure a new Application in the IdP (AWS SSO)
Create new SAML IdP in AWS IAM and configure it for QuickSight
Update attribute mappings in AWS SSO
Assign users to your new Application in AWS SSO

Create new Application in AWS Single Sign-On

Login to your AWS Organization Management account, navigate to the AWS Single Sign-on service
On the left hand bar, click on Applications.
On the Applications screen, click the blue Add a new application button

4. In the search box, type "QuickSight"

5. When Amazon QuickSight is displayed, click on it to select it and then click the blue Add Application button at the bottom of the screen

6. The next screen will allow you to enter configuration details for your application. Click on the View instructions link to get the specific configuration instructions for Amazon QuickSight, which I will walk you through below

7. Under Details, enter a display name and description for your app. You can accept the defaults if you wish.

8. In the AWS SSO metadata section, click the link to download the AWS SSO Metadata file to your computer. We will use this file later when we complete the IAM configuration.

9. Under Application properties, ensure Relay State has the value `https://quicksight.aws.amazon.com` and set the Session duration to whatever value you prefer...I recommend keeping it to 4 hours or less.

10. Under Application metadata, ensure that the Application ACS URL is set to `https://signin.aws.amazon.com/saml` and the SAML audience should be set to `urn:amazon:webservices`

Click Save Changes to save your SAML configuration. Now we are ready to move on to IAM configuration

Create new IAM SAML Identity Provider

First we need to create a new Identity Provider in IAM for the Application we just created.

If you are not already signed in, sign in to the AWS Management Console in your Management account
Navigate to the IAM service
On the left hand navigation bar, click Identity Providers
Click the blue button Add Provider on the upper right
On the resulting screen, leave the provider type as SAML
Enter a unique provider name
Click the Choose file button and upload the metadata file you downloaded in the earlier section
Add tags as appropriate and click the blue Add Provider button

Next, we need to implement a SAML Federation IAM role. The purpose of this role is to establish a trust relationship between IAM and AWS SSO. To implement, navigate to the Roles section of IAM and click Create Role. I will walk you through each step of the Create Role wizard.

Step 1 - Select Trusted Entity

Click the Radio button next to "SAML 2.0 federation"
From the SAML 2.0 Provider drop down, select the IDP you just created
Select the radio button for "Allow programmatic and AWS Management Console access"
Click the blue Next button

Step 2 - Add permissions

Now we need to attach a policy to this new role. We will be building a custom policy so click the Create Policy button on the upper right. A new window will pop open for you to create a policy, click the JSON tab and enter a policy in JSON format using the following as a guide, replacing the account ID with your own:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "quicksight:CreateReader"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:quicksight::accountid:user/${aws:userid}"
        }
    ]
}

Note that for my purposes, I only want SSO users to have the QuickSightReader permission, so my policy only includes that permission. This QuickSight documentation page provides details on additional permissions you can grant if desired.

Give your policy a name and then click the blue Create Policy button.

Go back to the window where you were creating the new Role, click refresh on the policy screen, and then select the new policy you just created and click Next.

Step 3 - Name, review and create

Give your role a name, a description and validate the trust and permissions are correct. Add tags as needed and then click the blue Create role button.

Update Attribute Mapping in AWS SSO

With our IDP and IAM role created, we need to go back to AWS Single Sign On and update our QuickSight application with the proper attribute mappings to use the IAM role and IDP.

While signed in to the AWS Management Console in the Organizations Management account, navigate back to the AWS Single Sign-On (SSO) service
On the left hand menu, select Applications
Click on the Amazon QuickSight application you previously created
Choose the Attribute Mappings tab
Click Add new attribute mapping
In the first field, enter `https://aws.amazon.com/SAML/Attributes/Role`
In the Value field, enter both the ARNs for the IAM role and the SAML provider separated with a comma, e.g. `arn:aws:iam::ACCOUNTID:role/ROLENAME,arn:aws:iam::ACCOUNTID:saml-provider/SAMLPROVIDERNAME`
Leave Format as unspecified
Click the blue Save Changes button

Assign Users to Application

With all the back-end setup work completed, we can now assign users to our QuickSight application!

While signed in to the AWS Management Console in the Organizations Management account, navigate back to the AWS Single Sign-On (SSO) service
On the left hand menu, select Applications
Click on the Amazon QuickSight application you previously created
Choose the Assigned Users tab
Click the blue Assign users button
In the assign Users screen you can toggle between Users and Groups, select the User or Group you wish to assign to the Application and then click the blue Assign users button

NOTE: I'd recommend you create a new AWS SSO group for this purpose and assign the application to the group. That way when you onboard more users you can just add them to the group.

Test!

With all the configuration out of the way, you can now test the SSO configuration. Login to your AWS SSO portal using the user you assigned to the QuickSight Application. In addition to the AWS Account application they should now see an application button for Amazon QuickSight.

Clicking on the Amazon QuickSight button will launch QuickSight in a new window and the user will be walked through the first time login screen. Note when they initially login they will not have any dashboards visible until an admin assigns them. I will cover this topic in my next post in this series so stay tuned!

Conclusion

In this article I gave you a quick introduction to the QuickSight Cost Intelligence dashboards, and walked you through how you can connect AWS Single Sign-on to QuickSight so that users can use their corporate credentials to access QuickSight directly. In future articles in this series I will be covering customizing and publishing dashboards to users, as well as how to implement QuickSight Groups and use row level security in QuickSight to filter access for different account owners. I hope you found this information useful, and please feel free to send me feedback or ask questions in the comments below.

Finally, I have to give a huge thanks to Jenny Oshima, my AWS Technical Account Manager, as well as Aaron Edell, GTM for Customer Cloud Intelligence for AWS who helped me out a ton during the implementation phase of this project.

References:

AWS Blog - Visualize and gain insights into your AWS cost and usage with Cloud Intelligence Dashboards and CUDOS using Amazon QuickSight

AWS Blog - A Detailed Overview of the Cost Intelligence Dashboard

Well Architected Labs - Cloud Intelligence Dashboards

Github - CUDOS Deployment Framework

Automated Enterprise Deployment of AWS Config

Mike Graff — Thu, 14 Apr 2022 15:35:48 +0000

AWS Config - powerful tool but a deployment headache

Seasoned users of AWS will be familiar with the AWS Config service. Config is a powerful tool that lets you continuously monitor and record the configuration of AWS resources in your account. Using Config, you can easily see changes to configuration of a given AWS resource on a timeline, see who made a given configuration change, and see relationships between resources in your account. Config also offers a powerful feature that allows you to setup governance rules to alert you when resources don't match your desired guidelines.

AWS recommends setting up AWS Config in each of your AWS account(s) as a best practice. Experienced users of Config know that this can rapidly become a cumbersome process if you have more than a few AWS accounts. AWS Config is a regional service which means you need to enable it in every region in which you operate. This can be a mind-numbing process using the AWS Console, and even using AWS CLI it is a complicated multi-step affair:

Create an S3 bucket
Create an IAM Role
Create an SNS topic
Create configuration recorder
Create a delivery channel
Start the Configuration recorder

Now imagine doing this doing this twenty times (once for each region) across 150 AWS accounts and I think you can begin to imagine the pain and suffering involved.

For admins of large multi-account environments that live within an AWS Organization, I'm happy to share there is a much easier way to deploy Config across all accounts and regions at once and for new accounts in your organization to have Config enabled automatically. The magic? CloudFormation StackSets to the rescue!

The Solution

In this solution, we will be setting up an AWS Organization wide Config service deployment, with all configuration recorder data being logged to a single centralized Amazon S3 bucket. Config service will be enabled in every account and region leveraging CloudFormation StackSets.

Prerequisites

In order to deploy this solution successfully there are a few requirements that must be taken care of first.

AWS Organizations

In order to leverage AWS StackSets, your AWS accounts must be members of an AWS Organization. Your accounts should be organized into one or more Organizational Units (OUs). I also recommend creating a "Testing" OU where you can try out your StackSet deployments on a test account before unleashing them on all accounts in your organization. In addition your AWS Organization should have all features enabled, not just consolidated billing.

Enable Stack Sets Trusted Access

We will be creating a StackSet with "service-managed" permissions. I highly recommend this option because with service-managed permissions, you do not need to worry about creating IAM roles in each account you will be deploying in...StackSets will take care of that for you automatically. This AWS documentation page provides more details on the different permissions models for StackSets deployments. Before you can start deploying StackSets with service-managed permissions, you must enabled trusted access with AWS Organizations. There are multiple ways to enable this, but the simplest way to do so is via the CloudFormation console.

Sign into your Organizations management account as a user with Administrator Access
Navigate to the AWS CloudFormation console
In the left-side navigation pane, choose StackSets. If trusted access is not currently enabled, a banner will appear at the top of the page that prompts you to enable trusted access.
Click the Enable trusted access button to enable this feature.

Trusted access is successfully enabled when the following green banner displays:

Centralized S3 Bucket

Rather than deploying an S3 bucket for each linked member account, this architecture relies on a single S3 bucket that all accounts write their logs to. If you are managing a large multi-account environment, you probably already have a centralized logging account, and this is where you should place this new bucket for config logging. Create an appropriately named S3 bucket in your logging account and ensure that the Block Public Access settings are enabled for the bucket. Then apply a bucket policy on the bucket that allows every account in the organization to access the bucket as well as the AWS Config service to access the bucket.

Below is a example bucket policy that demonstrates this configuration by leveraging the aws:PrincipalOrgID conditional to allow access from every account in the org and then uses Principal "Service": "config.amazonaws.com" to allow AWS Config the access it needs to the bucket.

{
    "Version": "2008-10-17",
    "Id": "Policy1357935677554",
    "Statement": [
        {
            "Sid": "ListBucketAccess",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::my-config-bucket",
                "arn:aws:s3:::my-config-bucket/*"
            ],
            "Condition": {
                "StringEquals": {
                    "aws:PrincipalOrgID": "o-1111111111"
                }
            }
        },
        {
            "Sid": "PutObjectAccess",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::my-config-bucket/AWSLogs/*",
            "Condition": {
                "StringEquals": {
                    "aws:PrincipalOrgID": "o-1111111111"
                }
            }
        },
        {
            "Sid": "AWSConfigBucketPermissionsCheck",
            "Effect": "Allow",
            "Principal": {
                "Service": "config.amazonaws.com"
            },
            "Action": "s3:GetBucketAcl",
            "Resource": "arn:aws:s3:::my-config-bucket"
        },
        {
            "Sid": "AWSConfigBucketExistenceCheck",
            "Effect": "Allow",
            "Principal": {
                "Service": "config.amazonaws.com"
            },
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::my-config-bucket"
        },
        {
            "Sid": "AWSConfigBucketDelivery",
            "Effect": "Allow",
            "Principal": {
                "Service": "config.amazonaws.com"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::my-config-bucket/AWSLogs/*",
            "Condition": {
                "StringEquals": {
                    "s3:x-amz-acl": "bucket-owner-full-control"
                }
            }
        }
    ]
}

With these prerequisites out of the way, we can now proceed with deployment of the solution using CloudFormation StackSets.

StackSet Deployment

StackSets are created in the AWS Organizations management account, using the same CloudFormation template concept as a normal CloudFormation stack. A StackSet is created and targeted at one or more Organizational Units (OUs), and CloudFormation deploys a Stack in each targeted account and region.

For this deployment, we will be leveraging a pre-built template from AWS, making our job easier. Follow this procedure to create a new CloudFormation StackSet in your AWS Organizations Management account

Log into the AWS Console in your Management account with a user that has Administrator Access permissions
Navigate to the CloudFormation service console
In the left-side navigation pane, choose StackSets.
Click the orange Create StackSet button to start the new StackSet deployment workflow

Below I will walk you through each step of the deployment workflow

Step 1: Choose a template

Ensure Service-managed permissions is selected and select the option to Use a sample template. From the Sample templates drop down, select the option "Enable AWS Config with central logging".

Then click the orange Next button.

Step 2: Specify Stack Set Details

Give your StackSet a meaningful name. You can leave all Parameters as default, with the following exception.

Under Central S3 bucket, enter the Name of the S3 bucket you created to store your config logs. NOTE: do not enter an ARN here, just the name of the bucket

Optionally, you can modify the following:

You can choose to include global resource types by changing that setting to true.
You can modify the frequency of with which AWS config delivers configuration snapshots by modifying the Snapshot delivery frequency.
You can specify an email address for AWS Config notifications in the Notification Email field.

When finished configuring options, click Next.

Step 3: Configure Stack Set Options

Apply any tags that you want for your StackSet in this screen. You can leave Managed Execution set to Inactive or change it to Active if you want CloudFormation to perform non-conflicting operations concurrently. When finished click the orange Next button.

Step 4: Set deployment options

Leave the deployment option set to Deploy new stacks.

For deployment targets, I always recommend starting out with a testing OU when deploying new StackSets to make sure there are no issues before targeting the entire organization. Select the radio button for Deploy to organizational units and enter the AWS OU ID in the text box. (You can retrieve OU IDs from the AWS Organizations console).

Leave Automatic deployment set to enabled, in this way if a new account is added to the OU a new stack is automatically pushed to that account. Account removal behavior should be set to delete stacks.

In the Specify Regions section, you can choose the regions you want to target from the drop down menu or click the Add all regions. Note that you should only target regions that are enabled in your target accounts, and those regions must also be enabled in your management account. See this note from AWS for more details on this.

Finally for Deployment options, choose the Maximum concurrent accounts you wish the stack to be deployed to at once. You can leave this at 1 or set a higher value.

For Fault tolerance, I recommend setting this a value higher than 0, otherwise your whole deployment will stop if a single account fails in a single region. I typically set this value to 2 or 3 so that a single error will not cause the entire StackSet deployment to fail.

Region concurrency controls how many regions will be deployed at the same time for a given account. For faster operations, set this to Parallel.

When finished configuring options, click Next.

Step 5: Review

Review all your settings to ensure they are accurate, then click the orange Submit button. You will be taken to the Operations tab for the new StackSet you just deployed, and it should have a status of Running

If all goes well, the Operation status will change from Running to Succeeded, indicating the stack was deployed successfully in all targeted accounts and regions.

You can then click on the Stack Instances tab to see the individual CloudFormation stack deployments in each account and region. You can also use the Stack Instances tab in the case of a deployment failure to see what went wrong.

Conclusion & Next Steps

To wrap up, in this post I've shown you how you can automatically deploy the AWS Config service across all regions for all AWS accounts in your AWS Organization, and have all configuration recorder data stored in a centralized S3 bucket. By deploying this architecture you will have the ability to assess, audit and evaluate the configuration of all AWS resources in your account(s), and track changes to those resources over time. In addition with this service enabled across your accounts, you can then implement additional governance across your AWS environments by leveraging AWS Config Rules.

Some of you may have already deployed AWS Config across your accounts in a decentralized fashion, and may be wondering how to migrate to this centralized architecture. Stay tuned as I will be covering that in my next post in this series.

I hope you've found this content useful. If you have feedback or suggestions on how this content could be improved, please leave a comment below.

Learn More

If you want to learn more about AWS Config service here are some links for more research: