DEV Community: Raphael Bottino

Get to know your AWS Managed Policies

Raphael Bottino — Tue, 04 Oct 2022 14:53:15 +0000

Understand what an AWS Managed Policy is and how a simple step can ensure you are using the appropriate one for your need

Have you ever googled AWS managed policies list? What about AWSLambdaExecute statements? Recently, once again, I found myself in a similar situation.

But let's start from the beginning.

What is an AWS Managed Policy?

From its documentation:

An AWS managed policy is a standalone policy that is created and administered by AWS. Standalone policy means that the policy has its own Amazon Resource Name (ARN) that includes the policy name. (…) AWS managed policies are designed to provide permissions for many common use cases. (…) AWS managed policies make it easier for you to assign appropriate permissions to users, groups, and roles than if you had to write the policies yourself.

In short, it is a special kind of IAM Policy that is curated and maintained by AWS and enables you to move faster, focusing more on your code and less about permission, leaving the latter to the pros at AWS.

But how do you know if the service you are working with has a managed policy that you can use for your benefit?

Service Specific Managed Policies

Reading the documentation, of course! Let's say the service in question here is AWS Lambda. A quick google search reveals the "Identity-based IAM policies for Lambda" page. There, as you can see below, three different managed policies are suggested:

Let's say you now need to use Amazon Polly so your awesome bot can have an Alexa-like voice. Again, a quick search will take you to its documentation, which lists two managed policies:

Let's move to something more complex and powerful, like theAWS Systems Manager, a service so comprehensive that it almost feels like multiple services in one. Googling will show you there are multiple SSM related AWS Managed Policies to use. What are the statements of AmazonSSMPatchAssociation for instance?

You don't need to exercise your Google-fu

If you know what managed policy you need more information on, you are good: an AWS CLI is all you need. And a bit of copying and paste.

First you run aws iam get-policy --policy-arn arn:aws:iam::aws:policy/AmazonSSMPatchAssociation. See the below:


aws iam get-policy --policy-arn arn:aws:iam::aws:policy/AmazonSSMPatchAssociation

{
    "Policy": {
        "PolicyName": "AmazonSSMPatchAssociation",
        "PolicyId": "ANPAZKAPJZG4EWLEL5ZX7",
        "Arn": "arn:aws:iam::aws:policy/AmazonSSMPatchAssociation",
        "Path": "/",
        "DefaultVersionId": "v1",
        "AttachmentCount": 1,
        "PermissionsBoundaryUsageCount": 0,
        "IsAttachable": true,
        "Description": "Provide access to child instances for patch association operation.",
        "CreateDate": "2020-05-13T16:00:42+00:00",
        "UpdateDate": "2020-05-13T16:00:42+00:00",
        "Tags": []
    }
}

Take note of the DefaultVersionId value, v1 in this example. Now, we run aws iam get-policy-version --policy-arn arn:aws:iam::aws:policy/AmazonSSMPatchAssociation --version-id v1:


aws iam get-policy-version --policy-arn arn:aws:iam::aws:policy/AmazonSSMPatchAssociation --version-id v1

{
    "PolicyVersion": {
        "Document": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Action": "ssm:DescribeEffectivePatchesForPatchBaseline",
                    "Resource": "arn:aws:ssm:*:*:patchbaseline/*"
                },
                {
                    "Effect": "Allow",
                    "Action": "ssm:GetPatchBaseline",
                    "Resource": "arn:aws:ssm:*:*:patchbaseline/*"
                },
                {
                    "Effect": "Allow",
                    "Action": "tag:GetResources",
                    "Resource": "*"
                },
                {
                    "Effect": "Allow",
                    "Action": "ssm:DescribePatchBaselines",
                    "Resource": "*"
                }
            ]
        },
        "VersionId": "v1",
        "IsDefaultVersion": true,
        "CreateDate": "2020-05-13T16:00:42+00:00"
    }
}

Now we have what we were looking for, the Managed Policy statements. With that information in hand, we can make an informed decision aboutthis Policy matches the use case requirements.

Pro Tip

If you do that enough, this can quickly become a tedious process. So let's fix that. Below you can find a Bash function that takes an AWS Managed Policy name as a parameter and outputs all the information that you might need.

get-managed-policy() { POLICY_NAME="$1"; aws iam get-policy-version --policy-arn arn:aws:iam::aws:policy/$POLICY_NAME --version-id $(aws iam get-policy --policy-arn arn:aws:iam::aws:policy/$POLICY_NAME | jq -r '.Policy.DefaultVersionId') ; }

get-managed-policy AmazonSSMPatchAssociation

{
    "PolicyVersion": {
        "Document": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Action": "ssm:DescribeEffectivePatchesForPatchBaseline",
                    "Resource": "arn:aws:ssm:*:*:patchbaseline/*"
                },
                {
                    "Effect": "Allow",
                    "Action": "ssm:GetPatchBaseline",
                    "Resource": "arn:aws:ssm:*:*:patchbaseline/*"
                },
                {
                    "Effect": "Allow",
                    "Action": "tag:GetResources",
                    "Resource": "*"
                },
                {
                    "Effect": "Allow",
                    "Action": "ssm:DescribePatchBaselines",
                    "Resource": "*"
                }
            ]
        },
        "VersionId": "v1",
        "IsDefaultVersion": true,
        "CreateDate": "2020-05-13T16:00:42+00:00"
    }
}

Another possible solution is to get access to all (currently) 973 AWS Managed Policies. The GitHub user Gene Wood was nice enough to write a gist with that list and the code he used to generate it. He also provided us with his code on how he generated it.

There is a problem, though. AWS is always releasing new services and features and this list was last updated almost 3 years ago. How can one have an always up-to-date list of all AWS Managed Policies and all of its statements?

Search No More

So I don't go over this pain again, and so others can also avoid it, I hacked together a simple website that, once a day, updates istself to make sure you have an accessible and updated list of all AWS Managed Policies right from your browser.

Introducing… awsmanagedpolicies.io!

awsmanagedpolicies.io is a simple-to-use, always up-to-date, accessible site that lists all of the AWS Managed Policies in a simple way, with a simple-but-it-works search bar to filter down the list

If you click any of the entries, it expands to show you the definition of said AWS Managed Policy:

If you just need a JSON file that always has the most recent list of AWS Managed Policies and its definitions you can bookmark this link instead!

Architecture

Of course, this website is 100% built on top of AWS and 100% Serverless! Its infrastructure was defined using CDK (TypeScript), and it contains, among other things, a lambda to fetch the latest AWS Managed Policies, an S3 Bucket to host the files, and a CloudFront distribution to serve the content to you.

As soon as I publish its code on GitHub and write an article on how was it to develop the site and how it works, I'll update this article with the links.

Conclusion

AWS Managed Policies are a great way to kick start your newest project. However, always make sure you are using the appropriate one. The best way to do it is verifying its statements via CLI or through the website awsmangedpolicies.io.

Full Disclosure

I decided to finally buy the domain, finalize the website, and write this article a few days ago. I started this project a year and half ago, give or take. Little did I know that today, there is an amazing solution to this problem, created by the AWS Hero Ian Mckay, called aws.permissions.cloud.

I highly recommend going to his site if you need more information than just a list of the AWS Managed Policies and their definitions, but also metrics like how many AWS Managed Policies are there, if a policy might expose a resource to the public, etc.

Kick Starting Your Cloud Career

Raphael Bottino — Mon, 28 Jun 2021 19:15:55 +0000

Originally published at: https://community.skycrafters.io/t/kick-starting-your-cloud-career/259

You might be new to the tech industry – or maybe you’ve been working in data centers for a while – but you decided it’s finally time to start you cloud career. What now?

Should you get certified?

There is a long debate in our forum about whether or not you should pursue certification as part of your cloud career roadmap. Some think that, in one hand, holding a certification doesn’t show content knowledge in itself, but in the other, it shows eagerness to learn. It’s true that certifications are no substitute to hands-on experience, but the goal of achieving one might be a great way to motivate yourself to learn a new skill.

Keep in mind, that a certification may not make your resume shine brighter than others, but it might be a requirement for the role. Watch out, though. Having too many certifications publicly, in an email signature or LinkedIn, might come off as bragging!

Exam Training

Once you decide to pursue a certification, you’ll have countless hours of study ahead of you. The question now is, should you do training in person or online? With some parts of the world slowly going back to business as usual (as much as possible), in person learning is now, once again, a possibility. Which one works best for you? This topic, also previously discussed by Skycrafters, is a really hot one! Let’s go over the pros and cons of each.

Online

Pros:

Being able to playback content faster or slower depending on familiarity with the topic
Easier, since it can be done from anywhere
Sometimes, free

Cons:

Easy to have your mind wandering to something else (after all, your Slack and phone are right there!)
The loneliness of not having someone to discuss the coursework with on a daily basis

In Person

Pros:

Closeness to other people to exchange ideas and thoughts
A live instructor to consult with in real time

Cons:

Harder to deal with boring topics/classes
Usually more expensive

So, which one is the best? This is a totally personal choice based on how you value each of the bullet points above. I’d personally pick in person training anytime, because I get easily distrac… Sorry, I was checking my phone. Where were we again?

Vendor lock-in in the Cloud

Cloud computing is great, there is no denying that. As someone that is learning a new technology, especially if you are aiming at a particular certification, it’s easy to get 100% focused on just one Cloud Service Provider for a while, like AWS for instance. If you ever need to switch and learn more about Azure, it’s not that hard to pivot from one to the other. After all, the overall concepts are so similar that even Azure itself provides a really nice guide on how their services compare to AWS’.

The dream of cloud-agnostic

However, the story isn’t as simple as providers like to paint. Vendor lock-in is a true challenge that the Skycrafters community has been discussing, and many consider a pipe dream. A dream because, a truly cloud-agnostic environment, would be able to run on any vendor environment, or even local. That allows the organization to pursue lower costs, whenever that is available from a competing vendor, but also a “get out of jail card” in case your current vendor becomes a competitor with a brand new service release. It’s a dream, however, because it can be expensive – in different ways – to run such a workload. If you try to escape using Amazon SQS, for instance, you might want to leverage open-source solutions like RabbitMQ. But now, you need to deploy and maintain a new stack of infra service that isn’t directly delivering value to your customers. Cloud computing is all about taking the most of the Shared Responsibility Model, and running your own infrastructure services isn’t the way to maximize it.

Kubernetes to the rescue?

Kubernetes is viewed as a way to minimize vendor lock in through its open architecture. It can take servers, no matter if running on AWS, Azure, GCP, on premises, etc., and transform them in computing capacity for this big cluster running spread across all of them. But it isn’t a bed of roses either, according to those that have experience on it. Running it yourself can be really painful, and exactly why most providers also offer their own flavor of Kubernetes-as-a-Service. But wouldn’t using it make you go back to the lock in stage? Don’t use it just because it’s a hot topic, as many do. If you ever learn and use it, make sure it makes sense to your challenge.

The goal here is to make clear you understand you shouldn’t create a vendor–lock-in situation for yourself. Make sure you don’t just understand how to use your provider of choice services, but also the reasoning behind it and its concepts.

Infrastructure as Code

Now that you dominated all the cloud knowledge you were seeking and learned all the cool stuff that your cloud provider of choice has to offer, you also need to learn that you rarely are going to use its dashboard to build anything for real. The dashboard is great for labs, tests, demos or to learn something new, but not for production environment. Production environment requires predictability, agility, consistency, minimization of risk and reproducibility. If you try a thousand times to create a simple S3 bucket in AWS via its dashboard, it’s almost guaranteed that you are going to make a mistake at least once and, even if you nailed it, it would take you a lot of time. Hence, Infrastructure as Code(IaC).

IaC is a way to describe your infrastructure as, you guessed it, code. As much as software is defined in lines of code, so is the infrastructure. You can write code that can define a thousand different S3 buckets and, as you execute it, you would reliable and quickly have a thousand buckets. No mistakes made.

To give you an better idea on what an IaC would look like, here’s a quick example:

Resources: 
  S3Bucket: 
    Type: 'AWS::S3::Bucket' 
    Properties: 
      BucketName: MY-REALLY-COOL-BUCKET

As our members previously discussed, there are a lot of different IaC flavors, some native, some open-source, some multi-cloud. To name a few that you might want to check out, we have:

CloudFormation – AWS Native and YAML/JSON based
Azure Resource Manager(ARM) – Azure native and it has its own DSL (Domain-Specific Language)
Terraform – Open-source, multi-cloud and it uses its own DSL
CDK – Newer AWS native offering that is open-source and you can code using your favorite programming language
Bicep– Newer Azure native offering that is open-source and has its own DSL
Pulumi – New open-source offering, that is multi-cloud and can code using your favorite programming language

And since IaC is easily replicable, you can take your time to build one really well crafted and documented template and reuse it across your projects, organization, and even publicly share with other members of the community! Which brings us to…

Best practices in the cloud

This is another great topic our members are discussing. Building confidently in the cloud can be challenging. Often, we use a technology and find out later that we could have been using it better. There are hundreds of different ways to build in the cloud using the providers’ services. And, despite the default configuration for many of them being “good enough”, “good enough” often times doesn’t cut it.

That’s exactly why many providers offer their set of best practices, usually called Well Architected Framework, or WAF for short. Taking AWS as example here, their WAF is divided in five pillars: Cost Optimization, Operational Excellence, Security, Performance and Reliability. Each pillar has their own set of white papers that explain thoroughly how to achieve the state-of-the-art usage of their services, while understanding the balance between the five pillars.

As it can take a while to build well-architected architectures for your projects, the combination of WAF with IaC is really powerful. Whenever you write your own IaC templates that build well-architected architectures, you can reuse them across your applications, saving time and bringing your environment to the forefront of what the cloud providers can offer.

Hybrid Cloud

Hybrid cloud is a really hot topic right next to multi-cloud that our community is debating. First, let’s take NIST’s definition for it:

The cloud infrastructure is a composition of two or more distinct cloud infrastructures (private, community, or public) that remain unique entities, but are bound together by standardized or proprietary technology that enables data and application portability (e.g., cloud bursting for load balancing between clouds).

Although NIST’s cloud computing definition is taken as the de facto way to describe it, the hybrid cloud approach is mostly used when there is a combination of one or more public providers and a private cloud to support an organization IT needs.

This is particularly interesting for organizations that have restrictions on how they process certain types of data, like banks. This allows them to leverage the public cloud to easily and cheaply scale as needed while maintaining the costumer’s data local to their data center.

It’s important to note, however, that being able to pull this kind of scenario off is really challenging, since it can be particularly hard to separate the data access through Identity Access Management across clouds. Also, dedicated links between the provider data center and your organization can be necessary because of the latency introduced by internet access, and they can be quite expensive.

Complement your studies with Podcasts

You probably can’t spend the entire day studying. At some point you are going to find yourself doing something manual and boring that doesn’t take much of your brain energy. Riding the subway or driving a car to the office (remember those?) is a good example, but it might also be cleaning your place or mowing your lawn. This is the perfect time to expand your cloud skills by listening to a good podcast. Our members have compiled a good set of podcasts that I’m happy to share with you here:

Screaming in the Cloud, by Corey Quinn
The Idealcast with Gene Kim
Talking Serverless, by Ryan Jones
The New Stack Makers, by The New Stack
Mik + One, by Dr. Mik Kersten
Girls In Tech podcast
Blinkist
Cloud Security Podcast

Keep in mind that not all of them are necessarily cloud related, but they might help you develop other skills and ares of expertise to help you in your cloud career. After all, technical ins’t binary and the cloud space isn’t just for those in the far–right spectrum of it.

Closing Notes

Cloud computing can get challenging really quickly, but it can also be exciting and fun. Especially, when you have a number of peers to work and innovate with. It doesn’t matter if you are a seasoned cloud practitioner, or if you are just starting out, Skycrafters can be a place for you to network, find answers quickly and bounce ideas off other members while learning in the process.

Skycrafters is home of great curated content, amazing open-source code to use or contribute to, and a safe place where I hope you can grow your cloud career and skills.

What are you waiting for? Skycrafters is 100% free, no gimmicks, and joining it can be a stepping stone for your cloud career and those around you.

Extra! Extra! Amazon AppFlow is Released!

Raphael Bottino — Thu, 30 Apr 2020 14:41:15 +0000

Have you heard about Amazon AppFlow? It’s a brand new service from AWS that allows you to easily integrate SaaS applications such as SalesForce and Marketo to AWS services, such as S3 or Snowflake.

Yet another day, yet another new AWS release. Even on challenging times with the current Covid-19 pandemic still slowing the global economy down, AWS shows that it is on full-throttle mode and released a new service last week called Amazon AppFlow.

What is it?

Pretty much paraphrasing the announcement, Amazon AppFlow is a fully managed integration service that enables you to securely transfer data between Software-as-a-Service (SaaS) applications and AWS services, in just a few clicks. As pretty much everything else on AWS, with AppFlow, you can run data flows at nearly any scale at the frequency you choose, paying just for flow run and data processed, with no upfront charges. For those that are security-aware (if you aren't, you should!), AppFlow automatically encrypts data in motion.

…and what does it mean?

It means 0-time invested to learn both the source's and destination's API. With a few clicks you can, for instance, backup all customer support cases from SalesForce to S3 on a weekly basis or daily push a list of new Leads from Marketo to AWS SnowFlake, allowing your team to quickly understand your leads behavior, all with no coding required.

But since I said that you should be security-aware, you might be thinking… "Can I leverage that for security purposes?" Yes! You can leverage this non-security related service to help you with your security.

Using it for Security

Trend Micro is the only Security vendor to be an AppFlow launch partner, which allows AWS and Trend Micro Cloud One customers to create AppFlow flows using Workload Security data as input, easily moving data from this security service to different destinations.

OK. AppFlow looks cool. The SalesForce and Marketo examples look cool. Having a security vendor like Trend Micro being a launch partner also looks cool. But how to use it?

Let's get our hands dirty

Of course, being the technical-curious person that I am, reading the release notes and examples are definitely not enough. I need to get my hands dirty. So, feel free to follow me on this journey.

Creating our First Flow

First, of course, let's hit the AppFlow dashboard.

If we click the bright orange "Create flow" button, we will be taken to the first step on creating our first flow. For this flow, I decided the name would be "CloudOneWorkloadSecurity-Computers" and I moved to the next step, without setting any of the optional settings.

On Step 2 we can see exactly where AppFlow shines. I picked Trend Micro as Source and all it requires to be able to fetch data from Cloud One is an API secret. Again, no coding required.

As soon as I add my API secret, AppFlow presented me with the different object options that it can retrieve from Cloud One. For launch, only "Computers" and "Policies" are available, as you can see below, but we should expect to see more options later down the road.

Then I picked "Amazon S3" as my destination, deciding on my bucket and a prefix to the objects.

Now we move to Step 3. Clicking on the drop-down "Choose source fields", we can decide on which fields we care about for this flow. I clicked first on "Map all fields directly", but because Cloud One is so thorough, I quickly realized it had way more information than I needed for this use case. So I selected only the 9 fields that I care about.

On the following step, I could pick to run the flow on demand or to set a schedule for it. I decided, for this example, to run it daily.

And that's it. The flow is ready to be used. And so I did.

In a little bit over 10 seconds, AppFlow fetched my Computers info from Cloud One Workload Security and dumped to a S3 bucket.

Clicking the "View data" link, it takes me straight to the bucket, where I can see the lonely file there. Downloading it shows me exactly what I expected, information taken straight from Cloud One.

Houston, we have a problem…

There is a problem with that, though… There isn't a ton of value on this flow on itself, plus, my hands didn't get that dirty. If you just wanted to know what AppFlow is and how to use it, the article ends here for you. Thanks for stopping by! If you, like myself, like to get your hands dirty, let's move to the next stage.

Working with the data

After the daily run of this flow, I want to work with the generated data — automatically, as soon as it hits the S3 bucket. The idea is to go trough the generated data, process it and write to another bucket. For this example, I decided to daily generate a JSON compatible array of computers that the current state is different from "active", which means they probably have some kind of connectivity issues with the Cloud One manager. The final result is something similar to the diagram below:

Before we go any further, it's important to note that the project — which has its code available on my GitHub — has its infrastructure built using AWS CDK (Typescript), while the Lambda code was built using JavaScript. If you are not familiar with CDK, I highly recommend the CDK Workshop documentation.

The code above describes the project infrastructure, generating a CloudFormation stack with a destination S3 bucket, a Lambda function and the proper permissions. Since I wanted to trigger this Lambda as soon as the source bucket received the data, I tried for a while to add this trigger to the code with no success; until I remembered, of course, that I wouldn't be able to do it — CloudFormation doesn't support adding event triggers to existing buckets.

After creating the infrastructure, I went ahead and coded the last missing piece: the Lambda itself.

The code is pretty straight forward. First, it downloads the newly added data to the Lambda execution environment. Then, it works the data. Since the original file has a JSON-described computer per line instead of an array of objects, I trimmed the file (to remove any white spaces from the end of it) and split it into an array of strings. Since each string represents an object, I mapped the array to return the objects that each string represents and, then, filtered out all objects where the state is active, since they are not relevant for us. Finally, all the non-active computers were written to the destination bucket.

After deploying the above stack, the last step is to manually connect the source bucket to it. Go to the bucket properties, click on Events and create a "All object create events" notification to it. Make sure to select the newly created Lambda to receive the notification. Now, for every AppFlow run, this lambda will also be triggered.

If you run the flow manually again to test the environment, you should see a new file on your new bucket, listing only the Cloud One computers that currently aren't on "active" state.
Resources:

[1] https://aws.amazon.com/new/

[2] https://aws.amazon.com/blogs/aws/new-announcing-amazon-appflow/

[3] https://docs.aws.amazon.com/appflow/latest/userguide/what-is-appflow.html

[4] https://blog.trendmicro.com/trend-micro-integrates-with-amazon-appflow/

[5] https://github.com/raphabot/AppFlowWorkloadSecurityDemo

[6] https://cdkworkshop.com

Originally posted at Medium