DEV Community: Animesh Bhadra 🎯

Doing Elastic Ips the Right Way

Animesh Bhadra 🎯 — Sun, 07 Nov 2021 16:30:06 +0000

Introduction¶

You are in the middle of understanding each of the steps in creating an EC2 instance. You have till now learned about Instance types, AMIs, CLIs and SDKs, and the pricing of AWS EC2 instance. One important step in this is to communicate with the AWS EC2 instance. Launching an AWS EC2 instance is of no use. There should be a medium to communicate with it. The communication may be external or internal.

Whichever type of communication you try to make, IPs play an important role. IP is like a unique address for your AWS EC2 instance, which can help you to connect with the instance. We have 3 types of IPs associated with an AWS EC2 instance.

Private IP
Public IP
Elastic IP

Let's jump right into learning what these different types of IPs are and what are its benefits.

Private IP¶

You might have already guessed by the name, this is a private IP. This is the default IP address associated with an AWS EC2 instance. This is an IP address which disassociated with an AWS EC2 instance based on these 2 parameters.

The VPC in which it is present.
The Subnet in within the VPC it is present.

AWS EC2 instance within the VPC can communicate with each other based on the private IP.

Each AWS EC2 instance, will have a private IP associate with it. This private IP is visible only within the VPC and not outside it.

If you want to connect to this AWS EC2 instance from your Laptop, it will not be possible as for this communication we need a Public IP.

Your laptop should have a Public IP for connecting to the AWS EC2 instance.

You should keep these things in mind while using private IP.

Private IP on an AWS EC2 instance does not change on reboot.
It also does not change when the AWS EC2 instance is in stopping state.
No two AWS EC2 instance, can have the same private IP in the same VPC.
You will need an AWS EC2 with private IP to launch a database system.
- This provides inherit protection from attach outside the AWS environment.

Public IP¶

AWS EC2 instance cannot function in isolation. There are certain use case which needs only private IPs, like for database machine.

A web server should be open to all. If you launch a web server in an AWS EC2 instance it should be accessible from outside. You will need a public IP on this AWS EC2 instance to achieve this.

Your AWS EC2 instance communicates over open Internet using a public IP. An AWS EC2 instance, can have both private and public IP associated with it.

When you stop an AWS EC2 instance, it releases the public IP associated with it. Public IP are a scarce resource in AWS, it's use should demand justification.

You can still keep the public IP address on the AWS EC2 instance if it reboots.

All AWS EC2 instance having a public IP does not connect to the Internet. You should enable the relevant security groups.

When you launch an AWS EC2 instance, in the default VPC, it has a Public IP. This is not the case of non default VPC.

Elastic IP¶

You might be wondering, Private IP and Public IP both solve a specific problem.

What is the need of Elastic IP?

If you want to access your AWS EC2 instance with a fixed IP every time, How do you do it? The AWS EC2 instance releases the public IP when it is in stopped state.

How do you assign a Static Public IP to your AWS EC2 Instance.

AWS Elastic IP belongs to an account and not to an Instance. Once you create an Elastic IP, you can assign it to any AWS EC2 instance.

The Elastic IP and the AWS EC2 has a symbiotic relationship. Once the AWS EC2 instance terminates, it frees the Elastic IP.

You can remove the Elastic IP attached to current AWS EC2 instance once the purpose finishes.

Cost of AWS Elastic IP¶

AWS Elastic IP has a weird pricing policy. It is free to use, till it's used.

If the elastic IP is not used, then it is chargeable.

These conditions will determine if your Elastic IP is in use.

AWS EC2 instance has an Elastic IP associated with itself.
The AWS EC2 instance is in running state.
Only one Elastic IP attached with this AWS EC2 instance.
The Elastic IP associated with an attached network interface.

AWS Elastic IP limits¶

AWS limits only 5 Elastic IP address per region. You can alter the limit from the console.

Is AWS Elastic IP free?

AWS elastic IP is free if it is in use. It's charged when not in use.

What is an Elastic IP in AWS?

It is a Static public IP associated with an instance.

In case of an instance failure, Elastic IP added to a new instance, without any impact on incoming traffic.

Public IP Vs Elastic IP¶

Public IP	Elastic IP
It is associated with the AWS EC2 Instance	It is associated with the AWS Account, and not the instance.
Public IP cannot be manually attached from one instance to another.	It can be removed from one instance and attached to another.
Public IP is released when the instance is stopped.	Elastic IP is not released, when the instance is stopped.
Public IP is free by default.	Elastic IP is charged if it is Idle.
An instance, launched in default VPC will have a public IP associated with it.	Elastic IP is manually assigned, it is not attached by default.
IPv4 and IPv6 support.	Only IPv6 support.

Static IP Vs Elastic IP¶

AWS Elastic IP is a static public IP for an AWS account.

There is a benefit in using Elastic IP

You can associate a new EC2 instance when old EC2 instance fails.
This happens through code, and no manual attention required.

Assign Public IP to an AWS EC2 Instance after launch¶

If you have an AWS EC2 instance, in running state having a private IP. How do you attach a public IP to this running instance. This is a million dollar question for you.

The only option to attach a public IP to an already running AWS EC2 instance is AWS Elastic IP. You can create a new Elastic IP. Once created you can associate this Elastic IP with the running instance. The running instance picks the new Elastic Public IP.

Conclusion¶

You have got a significant insight into the 3 different IP provided by AWS.

Private IP secures the AWS EC2 instance from outside traffic. Like, in database system, you do not need any outside traffic reaching it. AWS EC2 always have a private IP.

No communication inside a VPC is possible without these private IP. It's attached to the AWS EC2 instance throughout the life cycle of the instance. It's released only when the instance terminated.

When the AWS EC2 instance need communication with outside world, Public IP is the medium. Web Server needs public communication. When AWS EC2 instance launched in default VPC, its comes with Public IP.

It's assigned before creation or restart of the instance. The public IP is only associated with the AWS EC2 instance till instance is in running or reboot state. In any other state it release the Public IP.

Elastic IP associated with an AWS account and not AWS EC2 instance. The Elastic IP switched between instances. This only supports IPv4 unlike others which supports IPv6.

Elastic IP comes in helpful, when the EC2 instance fails. You can use Elastic IP reassigned to another AWS EC2. This is not a good architecture, but you should remember this is an option though not used.

Now the idea of AWS elastic IP, public IP, private IP might be clear. If you want to launch an EC2 instance. Checkout, the free tier EC2 instance launch article. See Step by Step guide to create an EC2 instance.

Reference¶

Save 72% of AWS EC2 cost with these pricing and tenancy options.

Animesh Bhadra 🎯 — Sun, 06 Jun 2021 05:35:22 +0000

Introduction¶

AWS as we know is pay as you go model. We pay for the amount of computing services we use. It is very good when we have limited requirement to compute. In a production environment, we need ways to reduce this cost. Pay as you go model does not always work in a production system. AWS provides 5 different pricing options for AWS EC2.

On-Demand
Savings Plans
Reserved Instance
Spot Instance.

You might be thinking where is the 5th? The 5th is called a Dedicated Host. You can run your above instance except Spot instance on these dedicated host. Dedicated Host is not actually a pricing model, but more of a Tenancy option.

You might already be getting confused. You had come to read this blog for saving some cost on your EC2 bills, but the different pricing and tenancy option is making it more confusing.

Do not worry, we will peel each layer of this pricing onion and figure out the best pricing solution for your requirement. Just to know a rough hierarchy if you are not interested in reading the whole blog.

On Demand pricing is the default and the costliest option.
Spot Instance is the cheapest option, almost 90% cheaper, but not suitable for production, dedicated instance requirement.
Reserved Instance and Savings Plans provide the maximum costing benefits if you commit to a certain period of usage.
Dedicated host is mostly used when you have existing licenses which you want to use on the cloud.

If you the above 4 lines have solved your initial pricing problem than thanks for reading. If you still have the patience to read more about the different pricing options and how it can save your cost by 72% compared to an on demand pricing read along.

AWS EC2 Pricing¶

You are already aware of the 5 different pricing options of AWS. There is also a free tier, which gives 750 hours of t2.micro on Linux or windows. This is a great way to test the water in the cloud world.

AWS EC2 pricing is also impacted by the amount of use an individual has on the cloud. There is no meaning of 72% reduced cost if you are learning AWS, and free tier will full fill your requirement. Let us first divide the type of user who will need these different options to save cost.

Hobby User.
- These users are like us, learning or trying AWS to see the services. These users should be happy with the free tier.
Amateur User
- These users are the intermediate user, who have graduated from the free tier. These users can use a lot of the services on AWS and can also have little complicated requirements.
- These users can mostly run On Demand EC2 instance.
Professional
- The professional is the real users of AWS, which includes companies, small start-up etc.
- These are people whose business rely on AWS.
- These people often need 2-3 types of AWS environment.
- Development - This is the AWS account which developer uses to develop the application.
- Staging - This is the AWS account which is almost similar to production environment, but lower scale.
- Production - The real user facing application runs on Production system.

You can understand with the above classification that AWS EC2 pricing impacts the 3rd type of user. If you fall into the first two categories the choice is pretty simple.

Now let's solve the pricing problem of the Professionals and also give insight to the Amateur User on On Demand pricing.

AWS EC2 On Demand Pricing¶

On-demand pricing is the default pricing model in AWS. This is the most easiest pricing model. You pay for the duration you use the EC2 instance. This is just like paying our electricity bills every month. The usage is broken into seconds.

On Demand pricing includes the cost of running AMIs on the EC2 instance. The lowest priced EC2 instance is t4g.nano with 1 vCPU, with On-Demand hourly rate at $0.0042. The max priced Ec2 instance is u-6tb1.112xlarge with 448 vCPUs, and On-Demand hourly rate of $54.60.

The usage duration is calculated between the time of termination or Stopping the instance. The partial hours are charged on per second basis, else charge on the full hourly basis.

Kindly remember the data transfer in and out of the AWS EC2 instance is also charged.

You might be thinking that there is no choice in AWS On-Demand pricing. There is no way to reduce the cost. There is one way. Generally the AWS Graviton is cheaper than the Intel Based EC2 and even the AMD based EC2 is cheaper than them. Pair a Linux AMI on these AWS Graviton or AMD based EC2 instance and you get you a significant cost savings.

If you use a m6g.large, which is an AWS Graviton based EC2 instance, you are charged $0.077. If you use a m6i.large in its place which is charged $0.096 per hour. This is 24% more.

What is an advantage of using the aws on-demand pricing model?

The major advantage of using AWS On-Demand pricing are these.

Off the shelf pricing, you do not have to use a lot of brain to figure out which is best for you.
Highly flexible, it ranges from $0.0042 to $54.60 hourly.
On-Demand pricing is the reason we have cloud computing today.

AWS EC2 Spot instance Pricing¶

You have seen the default pricing option on AWS. The AWS EC2 spot instance is another pricing model which will suit an individual developer the most. These Instances are available at 90% discount to On Demand pricing.

You might be already thinking, what is the gotcha for this drastically reduced pricing. There is no gotcha for this pricing. AWS has already dedicated data centers with huge computing power siting idle if not used. It is using EC2 Spot pricing to get some revenue of from idle instance.

Think of AWS EC2 Spot instance, as an End of Season Sale. You can use all the instances, with all the configurations of On demand, but at a 90 % lower price.

Little things might be bothering you right now, let's try to answer those questions.

What is the disadvantage of using an EC2 Spot instance?

There is no disadvantage, just remember that when AWS needs this capacity back, it will give a 2 minutes notice to wind up the current work being performed on the instances.

When would you use Spot Instances?

If you have a highly containerized application which can handle the 2 minutes notice for taking away the compute capacity, then you can use AWS EC2 Spot instance for production grade application.

Generally AWS EC2 Spot instance is used when you need more compute capacity, but at a lower price. These instances are very good for non mission critical application.

How can we save data for use once we lose the current pool of AWS EC2 spot instance?

We can just hibernate the AWS EC2 Spot instance and it will save the local data on the EC2 instance. Once the pricing or the capacity for AWS EC2 Spot instance is available, we can reactivate these hibernated instances.

How do you tell if an instance is a spot instance?

We can find the type of instance by selecting the Show/Hide Column on the AWS EC2 instance dashboard. Then we need to select the lifestyle option.

You should note that, once an AWS EC2 spot instance is available, there is not difference between it and an On demand instance. We can have all the ASG (Auto Scaling Groups) for these instances.

We can create an AWS EC2 instance, in two ways, the first is by selecting the below step AWS EC2 instance creation process.

Select the check box for requesting Spot instance.
Provide the maximum price you are willing to pay for a Spot Instance.
Lastly, mention if the request is persistent or not, if it is a persistent request, it will keep on trying to create the spot instance even after the request is full filled.

You can also check the current pricing of the AWS EC2 Spot instance here. As shown in the screen shot below.

You have now understood of two pricing mode, OnDemand and Spot instance, the issue is OnDemand is the instance you can rely on but Spot instance is something where you can run your non mission critical workloads. These workloads can survive a termination of the instance.

Though you can save 90% of the cost by using AWS EC2 Spot Instance, it may not be a perfect solution for mission critical jobs. Lets show you other pricing option which will save you cost for mission critical application.

AWS EC2 Savings Plans Pricing¶

You have learned about 2 flexible AWS pricing option, On demand and Spot. AWS provides deep discounts if you commit to a regular usage of compute power for a predefined usage period. There are two pricing options in this.

Savings Plans
Reserved Instance

Savings Plans is a newer edition of Reserved Instance so first learn this. We won't have any additional baggage of learning reserved instance. We can just learn what is the use of Reserved Instance. Saving Plans provide almost the same savings as reserved instance with an additional benefit of flexibility.

The flexibility offered by Savings plans is based on

Instance Family
Instance Size
Tenancy
or Region.

In savings plans you have to commit to a usage of predefined compute power, calculated on $/per hour basis for a fixed duration of 1 or 3 years.

Savings plans also offer you the option of paying in 3 modes.

All Upfront
Partial Upfront
No Upfront.

There are three AWS Savings Plan type.

Compute pricing plans
- This plan provides the best flexibility in terms of the variation of instance, size, tenancy and movement of the region.
- As it provides such flexibility the saving on this pricing plan is less, just 66% in place of the 72% promised.
- You can also migrate between fargate and lambda.
- This the best plan if you need all the flexibility.
EC2 Instance saving Plans
- If you are sure to restrict yourself to a type of instance, say M5 or C4 in a particular region.
- These provide the best savings of 72% in the savings plans.
- We can change the size of the instance from .xlarge to .2xlarge.
- We can even choose the different OS on these EC2 instance.
- We can change the tenancy from Host to, dedicated to, default.
SageMaker saving plans
- This plan offers 64% savings.
- This plan is applicable to all SageMaker Instance and provide all the flexibility of the Compute Pricing Plans.

Irrespective of which savings plans, you take, a few things you should always keep in mind.

Saving Plans do not provide capacity reservation.
Saving Plans provide discounted pricing based on the On Demand pricing, so we cannot cancel the request.

You can choose a savings plan by searching for "Saving Plans" in AWS console, and you will get a screen like below. Choose the type of plan you want as described earlier. You also need to select the term of 1 year or 3 years.

Once you select the Type of plan and the term, next is to choose the hourly pricing. You can take help of the AWS Cost Explorer. The AWS Cost Explorer will check for your existing usage and suggest an appropriate plan to purchase.

Next just select the payment option mentioned earlier. Choose a start date and done.

Can you sell AWS savings plans?

You cannot resell the AWS Savings plans like the Reserved Instance had a marketplace.

Can you cancel AWS savings plan?

Once purchased, you cannot cancel the AWS Savings plans.

You are now aware of AWS Savings Plans, which is also called Reserved Instance 2.0 or (RI 2.0). Now let take a step back and learn about the Reserved Instance.

AWS EC2 Reserved Instance Pricing¶

You have already learned about Savings Plans which is an improvement over Reserved Instances. You might be wondering what was in reserve instance, that led to creating savings plans. Reserved Instance and Savings plans looks almost the same.

You might still be wondering, will reserved instance exists down the like as savings plans are flexible. Let's try to find some of those answers and in the process learn about reserving instance.

The first thing to note is when you say reserved instance, it does not mean a physical instance reserved for you in AWS. It's just a terminology for a billing discount. This discount is applied to the On Demand prices.

The reserved pricing also based on 3 factors.

The first one and the very obvious one being instance type. You have to choose the same 4 attributes for an instance you chose in the savings plan.

Instance Type - m4.large, You have to select both instance family type and the size of the instance.
Region - Obviously, you have to tell the region where the instance would be needed, as EC2 is a regional service.
Tenancy - If you want to run it on the shared tenancy model or already have any dedicated instance.
Platform - Unix or Window.

The above choice is almost similar to the EC2 Instance saving Plans, the difference being, we selected the family of instance there, and the size of instances could be changed.

Once you are ready with your instance type, the next selection you have to do is for the Term commitment. There are same 2 choice like EC2 Instance saving Plans. The bigger the commitment you give, the bigger is the savings.

One - year
Three - year

The thing to note here is, the term does not get renewed automatically, we have to renew it else it will be charged on the on-demand rates once the term ends.

There are similar payment option like EC2 Instance saving Plans, and the names are self explanatory, The more you pay upfront the more you save.

All Upfront
Partial Upfront
No Upfront - You are required to have a billing history before you opt for this option.

The reserved instance also provides some flexibility depending on something called an offering class.

Standard
- It allows some attributes of the instance to be modified.
- These instances can't be exchanged.
- It can be bought and sold in a Reserved instance marketplace.
Convertible
- It also allows some instance attributes to be modified.
- These instances can be exchanged.
- It cannot be bought and sold in the marketplace.

You cannot cancel your purchase of Reserved Instances. Though depending on the type of offering class you have chosen, you can modify, exchange or sell the reserved instance.

There are also two different types of Reserved instance, you can purchase.

Regional Reserved Instance
- This does not provide capacity reservation.
Zonal Reserved Instance
- These provide capacity reservation on an Availability Zone.

Here is a snapshot to depict the type of savings you can get based on the reserved instance.

Can I cancel AWS Reserved Instance?

You cannot cancel your AWS Reserved Instance once you have purchased it. You can modify the instance, or trade them in the reserved instance market place.

You have learned about the Reserved Instance, and also learned that there is a capacity reservation in case of Zonal Reserved Instances, let us see how this capacity reservation helps us.

Can we reserve capacity when using Regional reserved instance, or AWS EC2 Savings Plans. These 2 does not offer any capacity reservation.

AWS EC2 Capacity Reservation¶

You have Zonal Capacity Reservation, which reserves capacity in a particular Availability Zone. This reservation provides a type of peace of mind, knowing that you can launch an EC2 instance at any given moment during high traffic as there is reservation.

AWS EC2 Savings plans and Regional Reserved Instance do not have this option, we can enable this by creating an independent capacity reservation. When you create a capacity reservation, you have to specify,

The Availability Zone (AZs) in which the capacity will be reserved.
The Number of instance
Instance attributes, like tenancy, instance type etc.

Please note that the capacity reservation will only be applicable only when you have the above matching instance running. If there is a mismatch the above capacity reservation will not be applied.

You have learned about all the pricing options available for an AWS instance, be it the default On Demand pricing, or highly discounted Spot instance used for non productive work, and immensely save cost using reserved instance or savings plans (RI 2.0).

There is also another factor for AWS instance, i.e. the tenancy, the physical host on which you run the AWS EC2 instance.

You cannot run an AWS EC2 instance, in a shared mode, which is the default because of compliance with laws in any case and in some case because of efficiency. Let's dive into the world of AWS EC2 tenancy.

AWS EC2 Tenancy¶

You can see in the below snapshot, that there are 3 types of tenancy available.

Shared Tenancy
Dedicated Instance
Dedicated Host

Tenancy is just like we rent a house when we move to a new city. How we move between houses is what is called tenancy.

AWS EC2 Shared Host¶

Share Host or Tenancy, it is more like few friend rents 2 or more houses in the city. The rent of the house is being paid, but which individual stays in which house is not predictable.

You can live in one house, but for a party, you can to the other house, if you sleep there, you will wake up in the same house, but if you leave the house there is an option you can land it either of the two houses.

The same is the case with AWS EC2 instance, when in the shared tenancy mode, the AWS EC2 instance can be launched on any physical machine. In case of reboot only the host will remain same else when you start or stop the instance can switch to a new host or physical machine.

AWS EC2 Dedicated Instance¶

The next one is a dedicated instance, it is more like a township of a big factory in a remote place. The township is created by the factory as people who work in the factory has to live.

You can live in one house in the township created by the factory and then in few years move to a bigger or a better house in the same township as you may seem suitable.

AWS EC2 dedicated instance is also a lot similar, when an instance is launched in this mode, AWS makes sure that it launches this instance, in the same physical host which has another instance of the same AWS Account.

When you stop and restart the instance it may not start the instance on the same instance, but it will make sure it starts the instance on a physical host where only your account instance are running.

_When would you use a dedicated instance?_Compliance is a big reason why you would want to launch a dedicated instance.

AWS EC2 Dedicated Host¶

When you are more stable in your life, with family, you need a house which you do not have to change. This is where you buy your own house.

The AWS EC2 dedicated host is also similar. A lot of time, especially in Banks and other financial organization they need surety that the application is running on a server which is physically separated from the other.

In addition to the compliance issue, there is also the case of software licenses, some costly licenses are per machine basis, if you have a physical server with AWS running only your instance, then you can move your licenses to these servers.

You can create your own dedicated host by selecting the proper option in the below snapshot. This we will discuss in details later.

What are the three main reasons for using a dedicated host?* The main reason for using dedicated host is compliance. Software licenses bought per machine basis is also a reason for this.

_What happens if the physical host we are using goes down?_If failure is detected on the physical host then AWS runs something called a Host recovery, which will basically recreate the same host instance setting. You can also use the AWS licenses Manager to migrate the licenses to this new machine.

Comparison¶

Regional Vs Zonal Reserved Instance¶

Regional Reserved Instance	Zonal Reserved Instance
Does not provide any capacity reservation.	Provides capacity reservation in a specific Availability Zone.
Reserved instance discount applies to any instance running in the region.	The discount is only specific to a Availability Zone
It provides flexibility in the same instance class, you can change the size of the instance.	It is very specific to the instance type and its corresponding size.
The purchase can be queued.	The purchase cannot be queued.
The limit of regional reserved instance to be launched is equal to the limit of the EC2 instance that can be launched in the region.	This provides addition limit from the regional limits.
This is just a billing discount.	Capacity reservation comes paired withe the discount.
Additional capacity reservation can be taken to make it more flexible.	No additional flexibility can be taken.

Standard Vs Convertible Reserved Instance¶

Standard Reserved Instance	Convertible Reserved Instance
You can modify only the Availability Zone, Scope, Networking Type and instance Size.	You can modify the instance family, OS, and tenancy in addition.
It cannot be exchanged.	It can be exchanged as long as the new instance is also convertible of higher or same pricing.
It can be bought and sold in the reserved instance marketplace.	It cannot be bought and sold in the reserved instance marketplace.
It provides the limited flexibility in configuration changes.	It provides the maximum flexibility in configuration changes.
Limited flexibility, hence highest discount.	Maximum Flexibility, hence limited discount

Compute Vs EC2 Savings Plans¶

Compute Savings Plans	EC2 Savings Plans
Provides the maximum flexibility.	Provides the minimum flexibility.
Provides the minimum discount.	Provides the maximum discount.
Discount can go up to 66%.	Discount can go up to 72%
Flexible across Instance family, AWS ECS, EKS using Fargate and even Lambda.	Flexible only across the instance size.
Can be compared to Convertible Reserved Instance.	Can be compared to Standard Reserved Instance.

Compute Savings Plans Vs Convertible Reserved Instance¶

Compute Savings Plans	Convertible Savings Plans
66% discount	66% discount
Does not provide capacity reservation.	Does not provide capacity reservation.
Same term or 1 or 3 years.	Same term or 1 or 3 years.
Applies the pricing discount automatically across instance family, tenancy or OS.	The benefits needs to be manually applied. across instance family, tenancy or OS.
Flexible across region.	Not flexible across region.

EC2 Savings Plans Vs Standard Reserved Instance¶

EC2 Savings Plans	Standard Savings Plans
72% discount	72% discount
Does not provide capacity reservation.	Does not provide capacity reservation.
Same term or 1 or 3 years.	Same term or 1 or 3 years.
Applies the pricing discount automatically across instance size, tenancy or OS.	The benefits needs to be manually applied. across instance size, tenancy or OS.

AWS Reserved Instances vs Savings Plan¶

Savings Plans	Reserved Instances
66% to 72% discount	66% to 72% discount
Discount is provided on On-Demand price based on the spending commitment.	Discount is provided on On-Demand price based on the utilisation commitment.
ECS, Fargate & Lambda are also covered in this discount.	None of these services are covered in RIs.
RDS instance are not covered under the Savings plans	RDS instance are covered under RIs
There is no marketplace for Savings Plans	There is a marketplace for Reserved Instance.
There is no capacity reservation.	There is a capacity reservation in Zonal RIs.

AWS Dedicated Instances vs Dedicated Host¶

Dedicated Instances	Dedicated Host
The Instance is not guaranteed to launch on same host.	The complete Host is dedicated.
The instance is always launched on host where same account instance are running.	The instance is always running on the same host.
You cannot use license which is per machine basis.	You can use license which are per machine basis.
It can be of any instance type.	The machine is based on instance type, so different size of same instance type can be launched.
There is no compute power wastage.	If the completed compute power of the host is not used, then the wastage is your responsibility.

Conclusion¶

AWS Pricing and tenancy is little confusing. There are just too many terms to go over. Let's keep it simple, Tenancy mean who owns the actual resources. Pricing means the discount provided for better commitment.

You should understand that in AWS Pricing options, except EC2 on-demand pricing and spot instance, the other two are just discount on the bill over the on-demand price.

On-demand is the default pricing options, and it is the costliest option. This can be used for production environment.

AWS Spot instance, are the unused instance, which AWS sells at a cheap discount. These are good for non productive work. AWS gives a 2 minute window to wind up the current activity on the instance before reclaiming.

Now coming to the discount schemes, both Reserved Instance and Savings Plans provide somewhere up to 66% to 72% discount over on-demand prices. This discount is based on commitment from us, to make use of the compute power for certain predefined duration, or agreeing to pay a certain amount per hour for using the same instance.

Savings plans are the newer version of discount pricing. It gives a better discount when you commit to a pay per hour a certain amount for compute usage. It also extends beyond just EC2 instance to AWS Fargate, ECS and AWS Lambda.

There is a one plan which is very flexible, Compute Savings Plans, this provide lots of choice in instance family, size, tenancy, etc.As this is a flexible option it provides limited savings.

If you are okay with a particular type of EC2 instance in a particular region, then EC2 Savings plans are a great plan, which can save you up to 72% from on-demand prices.

Savings plans are still not applicable to AWS RDS.

The last pricing option to discuss is the AWS Reserved Instances. This pricing option lowers the bill if we commit to a certain type or EC2 instance for a certain duration of time. The Reserved instance comes in 2 form like Savings Plans, Standard and convertible.

The difference between these 2 is that, the provide what degree of flexibility. More flexibility means less discount.

AWS has there type of tenancy. Shared, Dedicated Instance, and dedicated host.

Shared is the default tenancy mode, where an instance can come up on any number of physical hosts, with instance from other account also present on the same server.

Dedicated Instance means that the instance will only be launched in a physical server where it is already running the instance from the same account.

Dedicated hosting, means that the completed physical server is owned by you. You have to just books an instance type. Per software licenses loves such server.

Now the idea of AWS EC2 tenancy and pricing might be clear. If you want to launch an EC2 instance. Checkout, the free tier EC2 instance launch article. see Step by Step guide to create an EC2 instance.

Reference¶

Reinvent Videos
- AWS re:Invent 2019: [REPEAT 1] Dive deep on how to save with AWS Savings Plans (CMP210-R1)
- AWS re:Invent 2019: Save up to 90% and run production workloads on Spot Instances (CMP331-R1)
YouTube Videos
AWS Tenancy
AWS EC2 Pricing
- Amazon EC2 pricing
Savings Plans
- New – Savings Plans for AWS Compute Services
- AWS Savings Plans
Spot Instance
AWS Reserved Instances
- Reserved Instances
- Amazon EC2 Reserved Instances and Other AWS Reservation Models AWS Whitepaper
Capacity Reservation.
- On-Demand Capacity Reservations

The unconventional guide to AWS EC2 instance types.

Animesh Bhadra 🎯 — Mon, 10 May 2021 11:46:09 +0000

Introduction¶

AWS EC2 instance is the heart of AWS. They are the real server.

An AWS EC2 instance, has to support workload all across the boards, right from a web server, to FPGA and even HPC.

AWS EC2 instance supports all forms of computing needs. The hardware choice is diverse for such needs.

You have 350 types of different AWS EC2 instance, in the year 2021. Selecting the right AWS EC2 instance from these big piles will be difficult.

In this article you will learn how can you choose the correct AWS EC2 instance without breaking the bank. You will also learn about the various types of the AWS EC2 instance.

If you are studying for AWS Certification. You will also learn about the AWS EC2 instance naming convention. What does the AWS decides on the instance sizing. It will also help you with understanding the various instance type, and make its home in your memory. This is by far the best technique to remember the various names of the AWS EC2 instance.

Let's jump right in.

AWS EC2 Naming Convention.¶

The first thing you should learn is

How is the AWS EC2 instance, getting their name?

What does a name like M5d.xlarge means?

The below image may help you understand that.

If you look, these four component form the Instance Type.

Instance Size
Instance Family
Instance Generation
Additional Capability

AWS EC2 Instance Size¶

First, you should know the easy stuff, what does the xlarge means in the AWS EC2 naming convention?

The xlarge denotes the T-shirt size representation of the AWS EC2 instance. It defines the amount of

CPU
Memory
Storage
Network Performance.

An AWS EC2 instance possesses.

Now lets explore the M5d.

AWS EC2 Instance Family¶

The M means that AWS EC2 instance belongs to General purpose computing instance type. This is the Main Computing AWS EC2 instance.

You have five main categories of instance types.

General Purpose
Compute Optimized
Memory Optimized
Storage Optimized
Accelerated Computing

You also have sub classification with-in the five classifications.

AWS EC2 Instance Generation¶

The 5 in M5d represents the generation of the AWS EC2 instance. 5 is the current generation. The latest (current) generation of AWS EC2 instance is always better than a previous generation.

If you use the EC2 pricing calculator, you will find the pricing between an m4.large and m5.large is as below.

EC2 Instance Savings Plans rate for m5.large in the US East (Ohio) for 1 Year term and No Upfront is 0.06 USD

EC2 Instance Savings Plans rate for m4.large in the US East (Ohio) for 1 Year term and No Upfront is 0.062 USD

If you take the percentage, you have to pay 3.33% more for using an older generation AWS EC2. With this in mind, always choose the latest generation of hardware.

If the need is for a specific hardware, then choose the previous generation hardware.

AWS EC2 Additional capability¶

The d in M5d represents an additional capability.

Here is a table of additional capabilities.

Property	Representation
AMD	a
Graviton2	g
Local NVMe SSD	d
High networking(100Gbps)	n
Extra capacity	e

Now you are aware of the different component of the AWS EC2 name.

Every Instance size(xLarge) will have twice the number or CPU and Memory and storage resource from the previous size(large). How does this happen, jump forward to find it out.

Instance Sizing¶

The below image, helps you understand this concept. Each instance size (xlarge) will have twice the number of CPU and memory than previous size (large).

You should not believe the above image, lets talk data, use the AWS EC2 pricing calculator. It helped in creating the below table.

Instance Size	vCPUs	Memory (GiB)	Network Performance	1yr Std reserved hourly cost	Price increase
.large	2	8	upto 10 Gbps	0.06
.xlarge	4	16	upto 10 Gbps	0.121	101.67 %
.2xlarge	8	32	upto 10 Gbps	0.242	100.00%
.4xlarge	16	64	upto 10 Gbps	0.484	100.00%
.8xlarge	32	128	10 Gbps	0.968	100.00%

The table and the image representation completely match. xlarge has twice the number of vCPU and the memory compared to large. The pricing for 1 yr reserved hourly cost, is twice.

This poses an important question, while choosing the size. Will one go for 1 8xlarge instance or 8 xlarge instance.

Multiplying 0.121 * 8 = 0.968 tells that there is no difference between the two. You can pick any of the combinations.

The load on the service is not linear, it is a curve. This is not considered in the above multiplication.

If you remember the Bresenham's line drawing algorithm.It tells us, how to draw a line using square pixels.

The line can only be smooth if you have many dense pixels. The AWS EC2 instance, can match the uneven load if you have much granular AWS EC2 instance size.

It is always better to use a lower instance size. Use a Higher instance size if you have a specific need for it.

There are far to many instance classifications of AWS EC2, lets learn about them.

Instance Classification¶

The below image represent the relations of different AWS Instance Type. It has an alphabet written on a Green background. It will tell you the value of Instance Familyin AWS EC2 Naming Convention.

You will only focus on the naming of the classification and sub classification. The details of these classifications will come later.

There are five main categories of the AWS EC2 instance.

General Purpose
- General Purpose - T3/T4g Instance
- Development environment for application can use the T3/T4g instance.
- Transient or Temporary are a nice way to remember this sub classification.
- The short burst of CPU power which it will need is available.
- General Purpose - M5 Instance
- This is the main general purpose compute choice in AWS EC2.
- It is better to use this instance in place of the Brustable instance.
- These instances have predictable performance.
Compute Optimized - Optimized CPU
- The C Instance
- These instances are good for high compute power.
- These instances can serve high traffic web servers.
Memory Optimized - Optimized RAM
- Memory Optimized - R Instance
- They have higher capacity RAM, Like in case of r5.large, the base memory is 16 GiB, but in c4.large it is only 4 GiB.
- Memory Optimized - X Instance
- These are X tream RAM instance, the base x1.16xlarge is an instance with 976 GiB RAM.
- Memory Optimized - Z Instance
- The best CPU, with more memory.
- The z1d.large has a 4.0 GHz processor, with a memory starting at 16 GiB and going till 384 GiB.
- It even has a NVMe SSD attached, that is the reason of the d in the name.
- This is a complete instance for high performance
Storage Optimized - Optimized HDD/SSD
- Storage Optimized - D Instance
- The application with use high I/O performance.
- Storage Optimized - I Instance
- The instance has NVMe SSD with low latency.
- Storage Optimized - H Instance
- These are HDD based local storage, so higher throughput for operation.
- These are good for Big data operation.
Accelerated Computing - These are GPU and FPGA related
- Accelerated Computing - P Instance
- Parallel processing using the GPUs.
- Accelerated Computing - G Instance
- Graphics Rendering using the GPUs
- Accelerated Computing - F Instance
- The instance for programmable gate arrays.
- Accelerated Computing - Inf1 Instance
- The instance for AI/ML work load.

The instance body of work provides the root of above classification. Web Server can use the General Purpose instance. These instance are good for day to day computing.

General Purpose Instance¶

The most basic and all rounder AWS EC2 instances are the General Purpose Instances. They provide a perfect balance of computing, memory and networking resource. The below image shows the important points of both the sub class of the General Purpose Instance.

The four sub classification of the General Purpose Instance are.

General purpose - T3/T4g Instance
General purpose - M5 Instance
General Purpose - Mac
General purpose - Arm Instance

General Purpose - T3/T4g Instance¶

The T3/T4g have a baseline CPU performance of 2.5 GHz. These instance can burst to higher performance for shorter duration. The T3/T4g are burstable instance for this reason.

This burst is paid with CPU credits. When the instance is idle it leads to accumulation of CPU Credits.

EC2 Instance Savings Plans rate for t3.large in the US East (Ohio) for 1 Year term and No Upfront is 0.0522 USD

The T4g is the AWS Graviton2 Processor.

EC2 Instance Savings Plans rate for t4g.large in the US East (Ohio) for 1 Year term and No Upfront is 0.0421 USD

The ideal use case for these instances are micro-service, low-latency application, development environment.

You should be safe to not rely on the burstable CPU performance.

General Purpose - M5 Instance¶

These are, the more stable instance, in comparison to the T2/T4a Instance. They use a 3.1 GHz Intel Processor. These should be the first choice for anyone starting out on AWS. These instances provide a better baseline performance.

EC2 Instance Savings Plans rate for m5.large in the US East (Ohio) for 1 Year term and No Upfront is 0.06 USD

The best use case for M5 Instance are small and midsize databases, data processing tasks.

General Purpose - Mac Instance¶

AWS EC2 now provides macOS as an option for development. These are based on the Apple Mac Mini computer. It uses the Intel core I7 processor with 3.2 GHz (4.6 GHz Turbo) performance. There is only the mac1.metal option.

macOS products can use these instance for development, testing.

General Purpose - Arm Instance (A1)¶

The ARM based processor for AWS EC2 instance. It fully supports the ARM based development environment.

EC2 Instance Savings Plans rate for a1.large in the US East (Ohio) for 1 Year term and No Upfront is 0.0321 USD

Web server, micro-services, are some of the workload example for these instance.

If you have to pick one instance, then pick the M5 instance in this general category.

Use a T3/T4g only if you want to use the free tier service.

The other ARM and Mac instance, are very specific. You should use them till you do not have a specific need for these instance.

Lets move on the some very specific AWS EC2 computes instance.

Compute Optimized Instance¶

The server you need to use for higher compute power. They support 3.6 GHz to 3.9 GHz compute power.

EC2 Instance Savings Plans rate for c5.large in the US East (Ohio) for 1 Year term and No Upfront is 0.054 USD

The cost of a C5.large is cheaper than then General Purpose M instance. You need little higher compute power than use the C5 instance.

The general use case for Compute Optimized Instance are

high performance web servers
scientific modeling
gaming server.

Memory Optimized Instance¶

RAM has a direct impact on any compute operation. If you need higher RAM, then these are the instance you should use. They support 4.0 GHz compute frequency. The baseline instance, has 16 GiB RAM in them. This is more than the Compute Optimized and General Purpose Instance.

There are 3 types of sub classification in these memory optimized instances.

Memory Optimized - R instance
Memory Optimized - X instance
Memory Optimized - Z instance

Memory Optimized - R Instance¶

These are the generic higher RAM instance. They give more RAM per vCPU. 768 GiB being the highest RAM available.

EC2 Instance Savings Plans rate for r5.large in the US East (Ohio) for 1 Year term and No Upfront is 0.079 USD

Memory intensive applications can use the R instance to perfection.

Memory Optimized - X Instance¶

These instances support the large scale, enterprise class, in-memory application.

EC2 Instance Savings Plans rate for x1.16xlarge in the US East (Ohio) for 1 Year term and No Upfront is 4.11 USD

SAP HANA application can use the X Instance to perfection.

Memory Optimized - Z Instance¶

These are the instance which supports high compute capacity and high memory. The compute power is 4.0 GHz. They also support the NVMe SSD for low latency IO and higher RAM starting from 16 GiB to 384 GiB

EC2 Instance Savings Plans rate for z1d.large in the US East (Ohio) for 1 Year term and No Upfront is 0.117 USD

Relational databases can use the Z instance to perfection.

Storage Optimized Instance¶

These instance provides the variety in the Hard Disk or local storage option.

There are 3 sub classification of these instances.

Storage Optimized - D Instance
Storage Optimized - I Instance
Storage Optimized - H Instance

Storage Optimized - D Instance¶

These are the D or Dense instance. They provide 48 TB HDD instance storage.

EC2 Instance Savings Plans rate for d3.xlarge in the US East (Ohio) for 1 Year term and No Upfront is 0.315 USD

These instances are good for D istributed file systems like HDFS.

Storage Optimized - I Instance¶

High IO need the power of NVMe SSD. Since these instances are supporting NVMe SSD. You can get very low latency performance.

EC2 Instance Savings Plans rate for i3.large in the US East (Ohio) for 1 Year term and No Upfront is 0.107 USD

These instances are ideal for NoSQL databases like Cassandra, MongoDB, Redis.

Storage Optimized - H Instance¶

These instance support 16 TB of HDD based local storage. Since the storage is local, the latency is very low in these.

EC2 Instance Savings Plans rate for h1.2xlarge in the US East (Ohio) for 1 Year term and No Upfront is 0.318 USD

These instances are ideal for MapReduce work loads.

Accelerated Computing Instance¶

Till now all the instance type was using similar hardware. There was no special hardware used to improve performance. The Accelerated Computing instance changes that. It uses specific hardware for specific tasks. Like a GPU for both GPU intensive work or parallel processing.

It even uses hardware accelerators for FPGA and AWS Inferentia for AWS AI/ML work load.

There is four sub classification of the Accelerated Computing Instance.

Accelerated Computing - P Instance
Accelerated Computing - G Instance
Accelerated Computing - F Instance
Accelerated Computing - Inf1 Instance

Lets dig deeper into these.

Accelerated Computing - P Instance¶

The P instances are also referred to as Parallel Instance. These instances are best for Machine Learning and HPC. These instances have many network cards.

EC2 Instance Savings Plans rate for p4d.24xlarge in the US East (Ohio) for 1 Year term and No Upfront is 20.1754983 USD

The use case of these P Instances are Machine Learning, HPC, Computational Finance etc.

Accelerated Computing - G Instance¶

These instances are best used for Graphics Intensive workloads.

EC2 Instance Savings Plans rate for g3s.xlarge in the US East (Ohio) for 1 Year term and No Upfront is 0.551 USD

The use case of these instances are 3D Visualization, graphics-intensive remote workstation.

Accelerated Computing - F Instance¶

These instances are the equipped with field programmable gate arrays (FPGAs).

These instances are best for genomics research, financial analysis.

Accelerated Computing - Inf1 Instance¶

The Inf1 instance are best used for Machine learning inference application.

EC2 Instance Savings Plans rate for inf1.xlarge in the US East (Ohio) for 1 Year term and No Upfront is 0.232 USD

These instances are useful for Recommendation engine, forecasting etc.

How to choose an instance type¶

AWS provides a very good tool called AWS Instance Type Explorer. It helps to choose your required instance type.

You can select one of these four options.

Instance category
Hardware configuration
Accelerators
Additional Capabilities


			.

When you provide the requirement through these four options. This tool will suggest you some of the instance type which you can use. Like for the above combination the suggestion came as.

As you can see, it suggests M5d and M5dn but it did not suggest a size. If you get the category of the instance type like you have now. It is always better to start with the lowest size. Perform some testing with the on-demand instance and then decide on the size.

To reduce cost always go with the lower sized instance.

Conclusion¶

AWS EC2 has a great naming convention. Though this is not well document. The above naming convention comes from a few of the re-invent video and common sense. The more you look at the naming, the more confusing it looks at first. If you can separate the noise then you can observe the hidden pattern in the name of an AWS EC2 instance.

Once you have the grasp of the name.

The most important thing is to understand the four main classifications. Like, the various use of a server are.

Do memory heavy work
Do Compute heavy work
Do high IO work
Do GPU related work
Do generic work

The above five reasons, are the classification of the instance type. Be it Memory Optimized or Compute Optimized. Like I said you have to remove the noise to find the pattern.

Once you have the understanding of the main classification. The sub classification can come naturally. Like in Memory optimized instance, you may need

higher ram
higher optimized memory
finally a better CPU and memory work synchronization.

The AWS instance classification and sub classification are for a reason. Try your own understanding in finding the pattern. This will help you in remembering the details.

Finally, once you have decided on the instance. You should always choose the lower size instance to run experimentation. Based on the experimentation, decide on the actual instance size. As a rule of thumb always chose a lower instance size, it saves money.

Hope you are clear with the concept of AMI. If you want to launch an EC2 instance. Checkout, the free tier EC2 instance launch article. see Step by Step guide to create an EC2 instance.

Reference¶

How not knowing AWS CLI and SDKs makes you a Rookie.

Animesh Bhadra 🎯 — Sat, 01 May 2021 11:39:47 +0000

Introduction¶

AWS provides many ways to control its resources. One of the fastest and the most effective way is the AWS CLIs and SDKs. The best benefit of using CLI or SDKs it that you can automate complex workflow.

Depending on your expertise, you can use a bash script to control the AWS resources. You can use many programming languages, like .NET, Python, Go, etc. to control the AWS resources. You can use the Postman client to make HTTP API calls.

This article will provide an introduction to use the AWS CLI, SDKs and the HTTP APIs provided by AWS. Using these techniques, to control the AWS resources can save a lot of time.

Come and explore the work of AWS CLIs, SDKs and Postman Client.

Prerequisite¶

You can use pip and venv modules of Python for installing CLI and SDKs. pip is the Python package manager. venv is the virtual environment.

Python is cross platform. If you use Python, then the process becomes easier for Windows, Linux and MacOS.

There is a new version of AWS CLI V 2.x which do not support installation via pip. You will install only AWS CLI V 2.x via the package installer method mentioned below by AWS.

Install Python on Windows, Linux or macOS¶

There are many resources out there to install Python on these platforms. I cannot make any improvement over these installation instructions.

Please follow this RealPython article, to install Python on your respective operating system.

Execute the below command to confirm the installation is complete.

python --version

If you get any version information in return, you have installed Python.

You should now proceed to install the next important package called pip.

Install PiP on Window, Linux or macOS¶

Once you have Python installed, the next step would be to install the package manager called pip. It is the default installation on Python 2 version >= 2.7.9 and Python 3 version >= 3.4. You can check the installation by executing this command.

python -m pip --version

pip is already present, if you get the version information in response to the command. If not present you can use this documentation to install pip.

Installing PiP for Windows, Linux and macOS

Once you have pip installed, the next step is to create a virtual environment using the package venv.

Create Virtual Environment¶

You can have separate environments for your Python installation. This helps in not polluting the global package installation.

Create a directory name aws-cli in your local machine and follow along. Change to this aws-cli directory.

Create a virtual environment

python -m venv .env

If no error comes then there will be a .env inside aws-cli. If we get an error of package not found, please install the virtual environment package, using pip.

The .env is generally used to identify an environment folder, but you are free to use any name.

python -m pip install --user virtualenv

Once you have your .env folder created now we can proceed with the AWS CLI and SDK installation.

You are all set to install AWS CLI and SDK.

AWS CLI¶

AWS CLI is a tool to manage many services. If you choose the Amazon Linux AMI, the AWS CLI is pre-installed.

AWS CLI is an open source tool. AWS Management console provides the same functionality as AWS CLI. AWS CLI should have the new AWS IaaS administration features available. 180 days is the time limit for AWS CLI to have these new AWS IaaS features.

You get access to only the public APIs. While using AWS CLI, you do not have any special APIs to use, all the APIs are available in public.

AWS CLI comes in two versions, the most recent version is 2.x, it can be used in a production environment. 2.x version is not available as a Python package for installation through pip.

You have to use a separate installer to install the version 2.x. The support for AWS ClI v1 is still available.

There is an unofficial port of AWS ClI v2 on pypi called awscliv2.

If you want to install v2, jump to Install AWS CLI v2.

Install AWS CLI v1¶

Activate the virtual environment inside aws-cli folder. The below command works on Linux/macOS. It may be little different on Windows.

source .env/bin/activate

After activating the virtual environment, execute this command.

python -m pip install awscli

Once you have installed, it will show this at the end.

Successfully installed PyYAML-5.4.1 awscli-1.19.62 botocore-1.20.62 colorama-0.4.3 docutils-0.15.2 jmespath-0.10.0 pyasn1-0.4.8 python-dateutil-2.8.1 rsa-4.7.2 s3transfer-0.4.2 six-1.15.0 urllib3-1.26.4

If you see, while installing awscli it installs the above dependencies, one of that dependency is botocore. botocore is the base for even boto3.

You can verify the installation by running the command.

aws --version

If the version is successfully installed, you will get an output as aws-cli/1.19.62 Python/3.7.5 Darwin/18.6.0 botocore/1.20.62.

You have awscli v1 installed.

Install AWS CLI v2¶

For macOS:-

Download the latest AWS CLI v2 packages from AWS CLI pkg Download.

Install the package, by following the instructions. At the end, aws would be installed.

If you have both AWS CLI v1 and v2 installed, then it will pick the executable which is first in the PATH variable. Confirm the installation by opening a new terminal and executing.

aws --version

You should get an output as

aws-cli/2.2.1 Python/3.8.8 Darwin/18.6.0 exe/x86_64 prompt/off

The above output proves that we have the AWS CLI v2 installed. Now you should execute a sample command to see aws cli in action.

describeimages api using cli¶

Before you execute any aws command we should configure the aws environment. Please follow the instruction already mentioned. See AWS CLI configure AWS environment.

Once you have your AWS environment configured. Try running some operation on the AWS CLI.

Try the command mentioned in What is AMI? We will execute the describe-images command on an ec2.

aws ec2 describe-images --image-ids ami-0d758c1134823146a

If executed successfully we will get an output as

{
    "Images": [
        {
            "Architecture": "x86_64",
            "CreationDate": "2021-02-24T18:24:50.000Z",
            "ImageId": "ami-0d758c1134823146a",
            "ImageLocation": "099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20210223",
            "ImageType": "machine",
            "Public": true,
            "OwnerId": "099720109477",
            "PlatformDetails": "Linux/UNIX",
            "UsageOperation": "RunInstances",
            "State": "available",
            "BlockDeviceMappings": [
                {
                    "DeviceName": "/dev/sda1",
                    "Ebs": {
                        "DeleteOnTermination": true,
                        "SnapshotId": "snap-072d11ffd95664698",
                        "VolumeSize": 8,
                        "VolumeType": "gp2",
                        "Encrypted": false
                    }
                },
                {
                    "DeviceName": "/dev/sdb",
                    "VirtualName": "ephemeral0"
                },
                {
                    "DeviceName": "/dev/sdc",
                    "VirtualName": "ephemeral1"
                }
            ],
            "Description": "Canonical, Ubuntu, 20.04 LTS, amd64 focal image build on 2021-02-23",
            "EnaSupport": true,
            "Hypervisor": "xen",
            "Name": "ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20210223",
            "RootDeviceName": "/dev/sda1",
            "RootDeviceType": "ebs",
            "SriovNetSupport": "simple",
            "VirtualizationType": "hvm"
        }
    ]
}

The above command and output is same in v1 and v2. Congratulations on executing your first AWS CLI command successfully.

You should continue your command line journey with a little knowledge on AWS SDKs.

AWS SDK | Python | Boto3¶

SDKs take the complexity out of coding. They provide language specific APIs for AWS service. AWS SDK is available in these various languages.

JavaScript
Python
PHP
.NET
Ruby
Java
Go
Node.js
C++

You can use any of the SDK for controlling the AWS resources. You will learn to use the Python SDK called Boto3..

The Python Boto3 provides two key Python packages.

Botocore - Library providing low-level functionality shared with Python SDK and the AWS CLI
Boto3 - The Python SDK.

Installation of Boto3 is very easy. Activate the Python virtual environment, which we already did Install AWS CLI v1

Once you have activated the virtual environment, just execute this command.

python -m pip install boto3

Once the installation is done, it will give this message.

Successfully installed boto3-1.17.62

You can again execute the configure option, if not done as explained in AWS CLI configure AWS environment.

describe_images API using Python Boto3¶

Once installation is complete, we can execute the same describe_images to get the details of the AMI.

import boto3
import pprint

ec2 = boto3.client("ec2")

ImageIds = ["ami-0d758c1134823146a"]

response = ec2.describe_images(ImageIds=ImageIds)
pprint.pprint(response[""])

Once you execute the output would be.

{
   "Images":[
      {
         "Architecture":"x86_64",
         "BlockDeviceMappings":[
            {
               "DeviceName":"/dev/sda1",
               "Ebs":{
                  "DeleteOnTermination":true,
                  "Encrypted":false,
                  "SnapshotId":"snap-072d11ffd95664698",
                  "VolumeSize":8,
                  "VolumeType":"gp2"
               }
            },
            {
               "DeviceName":"/dev/sdb",
               "VirtualName":"ephemeral0"
            },
            {
               "DeviceName":"/dev/sdc",
               "VirtualName":"ephemeral1"
            }
         ],
         "CreationDate":"2021-02-24T18:24:50.000Z",
         "Description":"Canonical, Ubuntu, 20.04 LTS, amd64 focal image ""build on 2021-02-23",
         "EnaSupport":true,
         "Hypervisor":"xen",
         "ImageId":"ami-0d758c1134823146a",
         "ImageLocation":"099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20210223",
         "ImageType":"machine",
         "Name":"ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20210223",
         "OwnerId":"099720109477",
         "PlatformDetails":"Linux/UNIX",
         "Public":true,
         "RootDeviceName":"/dev/sda1",
         "RootDeviceType":"ebs",
         "SriovNetSupport":"simple",
         "State":"available",
         "UsageOperation":"RunInstances",
         "VirtualizationType":"hvm"
      }
   ],
   "ResponseMetadata":{
      "HTTPHeaders":{
         "cache-control":"no-cache, no-store",
         "content-length":"2184",
         "content-type":"text/xml;charset=UTF-8",
         "date":"Sun, 02 May 2021 13:42:28 GMT",
         "server":"AmazonEC2",
         "strict-transport-security":"max-age=31536000; ""includeSubDomains",
         "vary":"accept-encoding",
         "x-amzn-requestid":"bd357758-f81e-450d-bc7f-5764944d5186"
      },
      "HTTPStatusCode":200,
      "RequestId":"bd357758-f81e-450d-bc7f-5764944d5186",
      "RetryAttempts":0
   }
}

If you see the output, it is almost similar to the output when we executed the AWS CLI command for the same describe_images. see describeimages api using cli

Using Postman to execute AWS REST API¶

What is Postman?

As described by the company itself.

Postman is a collaboration platform for API development.

This tells us that we can use Postman, to test the following.

APIs
Automated testing on the APIs etc.

If you do not want to write code for getting some tasks done, we can use Postman, for that. Configuring Postman to achieve it is little complicated so please keep the focus.

You have understood about these 2 concepts in Postman to use it with AWS.

Environment
Collections

Configure Postman environment for AWS¶

As mentioned in Postman documentation, an environment is

A Postman environment is a set of variables, which you can use in Postman request.

You can use Postman environment to create a production or a staging environment. In our case we will create these 4 variables in the environment.

region - providing the AWS region, where to execute the Action
accountId - the 12 digit account ID
accessKey - The accessKey which is present when IAM console
secretAccessKey - The secret Access Key which we get once creating a new user, see AWS CLI configure AWS environment.

Create Postman environment for AWS¶

Let first create a Postman environment for AWS

You can see above you have to provide these 4 information in the environment, you can name your environment anything, I have named it aws-environment.

Once you have the environment, it's time for collection.

Create Postman collection for AWS¶

As mentioned in Postman documentation, a collection is

A Group of saved request which is organized into folders.

Create a new collection as shown below.

When you create a new collection, you should provide the Authorization details. Select AWS Signature as shown above.

You should provide all the AWS environment variable, we created in this collection.

You can reference all the AWS environment variables inside the Authorization panel of Postman collection by using {{accessKey}}, i.e. surrounding it inside {{ }}.

To verify if your variable are set properly, hover over the {{accessKey}}, if you get an Unresolved variable error like this,

This can happen, because the environment, we set is not accessible to this collection. Please check this drop down and choose the appropriate environment.

Once the environment is properly set, we can use the PostMan Collection to fire our request.

describeimages api using Postman¶

Once your Postman environment and collection is set, you can fire the same DescribeImages API to if the Postman is working.

As shown above, create a new request inside your existing collection which we create in the previous step.

You have to provide these Query Params.

Action - The DescribeImages
Version - This version should be 2016-11-15, which is the last document version, do not change it.
ImageId.1 - The AMI id you want to search.

If everything goes right, you should get a response in XML like below.

<?xml version="1.0" encoding="UTF-8"?>
<DescribeImagesResponse xmlns="http://ec2.amazonaws.com/doc/2016-11-15/">
    <requestId>961658cb-8af7-4d49-85d6-9012ac662616</requestId>
    <imagesSet>
        <item>
            <imageId>ami-0d758c1134823146a</imageId>
            <imageLocation>099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20210223</imageLocation>
            <imageState>available</imageState>
            <imageOwnerId>099720109477</imageOwnerId>
            <creationDate>2021-02-24T18:24:50.000Z</creationDate>
            <isPublic>true</isPublic>
            <architecture>x86_64</architecture>
            <imageType>machine</imageType>
            <sriovNetSupport>simple</sriovNetSupport>
            <name>ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20210223</name>
            <description>Canonical, Ubuntu, 20.04 LTS, amd64 focal image build on 2021-02-23</description>
            <rootDeviceType>ebs</rootDeviceType>
            <rootDeviceName>/dev/sda1</rootDeviceName>
            <blockDeviceMapping>
                <item>
                    <deviceName>/dev/sda1</deviceName>
                    <ebs>
                        <snapshotId>snap-072d11ffd95664698</snapshotId>
                        <volumeSize>8</volumeSize>
                        <deleteOnTermination>true</deleteOnTermination>
                        <volumeType>gp2</volumeType>
                        <encrypted>false</encrypted>
                    </ebs>
                </item>
                <item>
                    <deviceName>/dev/sdb</deviceName>
                    <virtualName>ephemeral0</virtualName>
                </item>
                <item>
                    <deviceName>/dev/sdc</deviceName>
                    <virtualName>ephemeral1</virtualName>
                </item>
            </blockDeviceMapping>
            <virtualizationType>hvm</virtualizationType>
            <hypervisor>xen</hypervisor>
            <enaSupport>true</enaSupport>
            <platformDetails>Linux/UNIX</platformDetails>
            <usageOperation>RunInstances</usageOperation>
        </item>
    </imagesSet>
</DescribeImagesResponse>

You can now experiment with the different HTTP APIs provided by AWS. Create an instance, destroy instance, everything which is possible using the public API.

From here on, the more you practice the more you can enhance the APIs. AWS provides multiple ways to control its resources. One of the fastest and the most effective way is the AWS CLIs and SDKs. The best benefit of using CLI or SDKs it that you can automate complex work flow.

Conclusion¶

AWS CLI, SDKs and HTTP APIs are three alternate ways to access and configure the AWS resources. AWS CLIs and SDKs requires a little bit of programming knowledge. It can be shell scripting or any programming language. HTTP Clients like Postman can use the HTTP APIs to provide the same result.

Python is a preferred choice when it comes to choosing the the AWS SDKs. The AWS CLIs core is also made with a Python Package. AWS has changed the way of installing AWS CLIs from pip to independent packages in Version 2.x.

In this article we have used the describeimage API to execute AWS CLIs, SDKs and HTTP CLient and get the same result in all the three possible ways.

You can proceed with any one of the technique from here on. It may depend on your previous experience. If you are knowledgeable in Shell scripting choose AWS CLI. If you are good in programming, choose the AWS SDKs. You can use the HTTP Client called Postman to fire the HTTP APIs if you have no programming experience.

Knowing AWS CLI, SDKs or HTTP API is one important step in the AWS Configuration. Checkout, the free tier EC2 instance launch article. see Step by Step guide to create an EC2 instance.

Reference¶

The complete beginner's guide to AWS AMIs

Animesh Bhadra 🎯 — Fri, 23 Apr 2021 07:16:43 +0000

Introduction¶

Have you ever pondered on the question

Why AMI (Amazon Machine Image) is not called Operating System?

If you revisit our previous article, where the first AWS EC2 instance in free tier was created, see Step by Step guide to create an EC2 instance,. The first selection was for AMI.

In this article you will find out, Why Amazon Machine Image (AMI) is not called operating system? This article is everything, you need to understand AWS AMIs.

You will also get answer's to these questions

Why are they referred to as AMI?

What does the AMI cost?

This article will enlighten you with the AWS AMIs inner working. It also explains that you pay for the AMI, when you get bill for your EC2 instance.

This will be a beginner friendly resource. For an experienced person, it will provide some details which you might have ignored.

Let's say Hello World to the Amazon AMIs.

What is AMI?¶

You should understand by now, AMI stands for Amazon Machine Image.

What is an Amazon Machine Image, the answer depends on the Storage supported.

An AMI includes these things depending on the Storage supported

Amazon Elastic Block Storage
- It provides the snapshot. This snapshot will include the operating system, or any other application required.
Instance Store Backed
- It is a template, providing
- Operating system
- Application server
- Application.

Amazon Elastic Block Storage and Instance Store, they both provide the same details.

You can extract the details of an AWS AMI, by the use of an API describe-images. To use this API, you might need the AMI Id.

In the previous post to create Free Tier AWS EC2 instance, see Step by Step guide to create an EC2 instance, you used the Ubuntu AMI.

It has a unique identifier call AMI ID, as highlighted in the image below.

You will use the AWS CLI, to execute this command

aws ec2 describe-images --image-ids ami-0d758c1134823146a

The output is

{
    "Images": [
        {
            "Architecture": "x86_64",
            "CreationDate": "2021-02-24T18:24:50.000Z",
            "ImageId": "ami-0d758c1134823146a",
            "ImageLocation": "099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20210223",
            "ImageType": "machine",
            "Public": true,
            "OwnerId": "099720109477",
            "PlatformDetails": "Linux/UNIX",
            "UsageOperation": "RunInstances",
            "State": "available",
            "BlockDeviceMappings": [
                {
                    "DeviceName": "/dev/sda1",
                    "Ebs": {
                        "DeleteOnTermination": true,
                        "SnapshotId": "snap-072d11ffd95664698",
                        "VolumeSize": 8,
                        "VolumeType": "gp2",
                        "Encrypted": false
                    }
                },
                {
                    "DeviceName": "/dev/sdb",
                    "VirtualName": "ephemeral0"
                },
                {
                    "DeviceName": "/dev/sdc",
                    "VirtualName": "ephemeral1"
                }
            ],
            "Description": "Canonical, Ubuntu, 20.04 LTS, amd64 focal image build on 2021-02-23",
            "EnaSupport": true,
            "Hypervisor": "xen",
            "Name": "ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20210223",
            "RootDeviceName": "/dev/sda1",
            "RootDeviceType": "ebs",
            "SriovNetSupport": "simple",
            "VirtualizationType": "hvm"
        }
    ]
}

This tells me all about the AMI, which family the AMI belongs, What is the Virtualization type. It also tells what is the root device type.

The above information is also present in the below screen.

As mentioned above, each AMIs has 3 information in itself, you will try to find these 3 information first.

Is the above AMI an EBS backed or Instance Store.
- "RootDeviceType": "ebs",
What is the operating system?
- "PlatformDetails": "Linux/UNIX",
- "Description": "Canonical, Ubuntu, 20.04 LTS, amd64 focal image build on 2021-02-23",
It will be created from this snapshot id "SnapshotId": "snap-072d11ffd95664698",
There is no additional application server or any other application installed.

The above AMI is a bare bone Ubuntu Image.

If you want, you can install your own Web Server Apache2, or any other application and then create an AMI from it.

The next natural progression should be to explore what is the use of AWS AMIs?

What is the use of AMIs?¶

If you observe the image below, you can understand 2 basic uses of the AMIs, Can you please guess the 2 use you can think of?

Before I can give you the answer to the above question, you should comprehend the above Image.

The first step is to create an EBS Snapshot, Once you have the EBS Snapshot, you should register the AMI.

These two steps are not required if you are choosing an existing AMI.

In Addition, the above image shows us the 3 use or life cycle of the AMI

Launch
- launches an AWS EC2 instance from AMI
Copy
- You can copy an existing AMI, for future use.
De-register
- You can de-register the AMI if not used.

If you have guessed any two use from the above image, please pat your back.

Once you are clear with the life cycle / use of an AWS AMIs, you should proceed towards seeing the types of AMIs.

Region, Operating System, Architecture, and Storage can define the AMI's classification. Let's us discuss them.

Types of AMI¶

Region (see Regions and Zones)¶

The region restricts the availability of a particular AMI.

If you check Ohio (us-east-2), the number of public AMIs are 69,431.

If you check for N.Virginia (us-east-1), the number of public AMIs are 140,650.

The above numbers can change in the future. The disparity between region for AMIs will still be present.

The AMI are region specific. You can create EC2 instance with the AMI present in that region.

If you want any other AMI from a different region you may have to copy those AMIs to your region.

Operating System & Architecture¶

You have already seen, AMIs are available for both Linux and Windows. There are different variants for both the operating system.

You also have both 32-bit and 64-bit architecture support in these.

Launch permissions¶

The AMI owner can specify the availability, by providing the launch permission. There are three types of launch permissions

public - Anyone on AWS can launch using this AMI
explicit - One account can grant explicit permission to another account.
implicit - Owner of the AMI has implicit permission to launch.

You will learn about public, explicit and implicit AMIs once you reach a state of creating AMIs. Till that time being, you can read this article on the same, see Share an AMI with specific AWS accounts.

Storage for the root device¶

The storage of the root device, creates two distinct classifications of AMI Types.

Backed by AWS EBS
- The root device would be an AWS Elastic Block Store
Backed by AWS Instance Store.
- The root device would be an instance store volume, created from a template.

Property	AWS EBS	AWS Instance Store
Boot time	less than 1 minute	less than 5 min
Size Limit	16 TiB	10 GB
Root device volume	EBS	Instance Store
Data Persistence	Root Volume data is deleted, Non root EBS is persisted	Data is persists only till life of the instance
Modification	Instance type, Kernel, Ram Disk, User data can be changed in stopped state	It is fixed.
Charges	Charged for instance use, EBS volume usage and Storing of AMI as EBS Snapshot	Instance usage and AMI storage on S3
AMI Creation	Single command	requires installation and use of AMI tool
Stopped State	Can be in stopped state	Cannot be in stopped state.

You might have this question in mind, Why should you use Instance Store, when EBS is better in all aspects?

Instance Store is the option when you need low latency. Read and write are faster because host has the volume mounted. Check out the result published here.

Instance store is ephemeral. This makes them a perfect candidate for temporary data which changes very often.

Linux AMI virtualization types¶

You might be thinking, What is Virtualization after all?

Virtualization is a type of abstraction. It allows many machines created from a single computer. It translates the virtual machine into the underlying hardware.

You might have used a Virtual Box to run Linux on your Windows PC, that is also a type of Virtualization.

Virtualization is at the heart for all AWS or any Cloud Provider. You will be running many different operating system on the same piece of hardware.

You might be thinking, How can you run different operating system on the same piece of hardware?

The answer is a technology called HyperVisor or Virtual Machine Monitor (VMM). It allows to host different virtual machine.

The different virtualization techniques used are

Full Software Virtualization
Hardware-assisted software virtualization or Hardware Virtual Machine (HVM)
Paravirtualization or paravitualized Machine (PV)
Hardware assisted software virtualization with Paravirtual drivers (PVHVM)
Component or resource virtualization

AWS supports two types of AMIs based on the virtualization techniques.

Paravirtual (PV)
Hardware virtual machine (HVM)

What are these Paravirtualization or Hardware Virtual Machine (HVM)? Come further the rabbit hole.

Hardware virtual machine (HVM)¶

CPU Chips with built in virtualization can support Hardware virtual machine (HVM).

The Hardware virtual machine is a type of full software virtualization. It is dependent on the hardware capability. Hardware Virtual Machine (HVM) is the future.

Paravirtualization¶

In this virtualization techniques, the guest OS uses the facilities provided by the host OS.

PV and HVM AMIs¶

Have you seen any Paravirtual (PV) or Hardware Virtual Machine (HVM) AMIs in AWS. Check the images below

HVM:-

Paravirtual

You can see that Paravirtual is a thing of the past. It is only available in community AMIs and not present in the Quick Start AMIs.

You have only HVM's as an option in Quick Start AMIs.

HVM Vs PV¶

Property	HVM	PV
Description	HVM AMIs are presented with a fully Virtualized set of hardware and boot by executing the master boot record of the root block device of the image.	PV AMIs boot with a special boot loader called PV-GRUB, which starts the boot cycle and then chain loads the kernel specified in the menu.
Hardware extension	Yes, can take advantage	No, cannot take any advantage
Instance Type	All current generation	Only certain generation, like C1, C3, HS1, M1, M3, M2 and T1
Region	All Region	Asia Pacific (Tokyo), Asia Pacific (Singapore), Asia Pacific (Sydney), Europe (Frankfurt), Europe (Ireland), South America (São Paulo), US East (N. Virginia), US West (N. California), and US West (Oregon)
How to find	Virtualization type of the AMI is set to hvm	virtualization type of the AMI is set to paravirtual,

You have understood the difference between the virtualization techniques. Its time to move to understand why would you pay for an AMI?

Shared and Paid AMIs¶

You can separate the AMIs based on the payment option.

As a Developers, you can share AMIs to the community, which you have created. Other developer's can pick these AMIs for modification. The Community AMIs are falling in this category.

Use of these shared AMIs in production environment without audit is a security risk.

The Paid AMIs are available from AWS Marketplace. It provides high quality licensed software configured in the AMIs. These are generally charged based on the hourly rate given by the owner.

You might be thinking, normal EC2 billing I can understand.

How does a component of that EC2 charges money?

Come with me to find..

Billing¶

You have executed the describe-images command in What is AMI? section. In the JSON output you got this two details.

{
    "PlatformDetails": "Linux/UNIX",
    "UsageOperation": "RunInstances",
}

These two parameters define the billing of the AMI, and also compatibility.

When launching a spot instance, always check if the spot instance supports the particular PlatformDetails.

In case of Reserved Instance, you should check if the operating system platform lits the AMI PlatformDetails.

The UsageOperation tells the lineitem/Operation in the actual billing of the AWS EC2 instance.

Conclusion¶

Cloud Computing is possible today because of advancement in virtualization techniques. Hardware virtual machine (HVM) has a special role in this.

AMI (Amazon machine image) has further enhanced the options.. Many instances can use these AMIs to launch identical copies. AMIs are the blueprint. AMIs have a definite life cycle of creating, register, copy, launch and de-register. If you are not creating your own AMI, then you have only last three life cycle states.

AMIs are different based on the region, you are launching. It is different based on the architecture and operating system. It differs in the way different people have access to launch. It is differentiated based on the root device type.

AWS supports only HVM and PV virtualization techniques. Between these, AWS is recommending the HVM and in future you may not even have a PV AMI.

The billing of an EC2 instance includes AMI cost. PlatformDetails & UsageOperation field of the AMI contributes toward this cost.

Hope you are clear with the concept of AMI. If you want to launch an EC2 instance. Checkout, the free tier EC2 instance launch article. see Step by Step guide to create an EC2 instance.

Reference¶

The Insider's Guide to AWS EC2

Animesh Bhadra 🎯 — Wed, 21 Apr 2021 05:18:54 +0000

Introduction¶

This is the shortest, easiest, and a beginner friendly step by step guide to launch and run a free tier AWS EC2 instance.

I am going to teach you everything, launch a free tier EC2 instance, connect to the EC2 instance using SSH or RDP. This post will teach you just enough to be dangerous.

If you ever wanted, just be able to launch a server (EC2) in the cloud and use that server, without worrying about the inner details. Even worried that it may cost you a fortune. This is the post you were waiting for.

In seven days, God Created the world.

You need only seven steps to create an AWS EC2 instance in cloud.

Let's say Hello World to the cloud.

What is AWS EC2 ?¶

EC2 stands for Elastic Compute Cloud, AWS's virtual server in the cloud.

In order to create an AWS EC2 instance, you should understand

What is AWS EC2?

Why to use EC2?

What is the benefit over your existing on premise server?

Cloud computing is all about collection of servers running on someone else premise. In case of AWS it is Amazon's premise.

AWS EC2 is that single server in the school of servers on the cloud. AWS EC2 is one of the ingredients in the cloud recipe.

AWS EC2 is that virtual server running in the cloud. It is also referred to as an instance sometimes. You can select the amount of memory, storage and compute power each of your AWS EC2 should have.

You can even commission and decommission AWS EC2 on demand. This is was makes it a preferred choice compared to the on premise server. Due to this facility, cloud can always manage the traffic coming to them. When the load is high you can have more EC2 instance, and less instance when the load is low.

Cloud computing is all about paying what you use. You never pay for anything which you are not using.

How do you create this AWS EC2 instance?

Lets jump right into it.

How to create EC2 instance in AWS ?¶

You have learned about AWS IAM before this, if not please read it. You have become comfortable with AWS Management console. You have still not seen anything happening in the cloud.

The purpose of this Blog, is to get your own private server running in the cloud. You will learn about all the inner details of the AWS EC2 instance, in future series of Blog.

In short you have still not said Hello, World! to cloud. You will learn it today. Login into AWS Management console and follow along. You are entering The Matrix.

EC2 DashBoard¶

In the AWS Management console, search for the service called EC2. It should launch an EC2 Dashboard just like this one.

You have to select the Launch Instance. There are two ways to launch instances,

Launch Instance
Launch Instance with a template.
- This is a basic configuration which can be saved as a template and be used to launch multiple instance with the same configuration.
- You will learn more about this in future, when you learn Auto Scaling Group.

You will have this screen next.

You can clearly see that creating an EC2 instance is a 7 step process. These steps are.

Choose AMI
Choose Instance Type
Configure Instance
Add Storage
Add Tags
Confirm Security Group.
Review

You will have to proceed with these 7 steps to create your AWS EC2 Instance.

Step 1: Choose an Amazon Machine Image (AMI)¶

An AMI or Amazon Machine Image, is a template which provide these information

Operating System
Application Server
Application

An AMI is just like kids Stencil, which they use draw same diagram multiple times. Hope you remember these...

There are multiple vendor for AMIs, but you will not focus on them here. This is a battle for another day.

Please note, all the AMI listed at Quick Start are not free, please select the check box Free tier only , to list only 16 AMIs out of 40 AMIs. See the below image for reference.

Broadly the AMI's are divided based on Operating System, Linux and Windows, and Linux is subdivided primarily based on its flavor.

You should go with Ubuntu Server 20.04 LTS (HVM), SSD Volume Type, Select 64-bit (x86) version and move to the next step to Choose the instance type.

Step 2: Choose an Instance Type¶

This step is analogous to choosing the hardware for your server. You have hardware which is optimized for database operation, some are optimized for Machine learning and AI application, some web application. You will understand the details on this in the very short future.

This screen looks like this.

The instance type tells us in detail about the family of hardware, the number of CPU cores it has, what type of instance storage it supports, Network performance, etc.You will learn about this, but as of now focus on the free tier instance type.

You have a t2.micro, which is eligible for free tier. You get 750 hrs of usage each month for the first year of joining. Please select this instance and proceed to the next step of, Configure Instance.

Step 3: Configure Instance Details¶

This is the step which will require an article on its own. Just look at the configuration you can do on this screen.

You can configure all the below option.

Number of instances you want to launch
Do you need spot instance, a type of instance which is very cheap and can be used for non production work
Do you want to use the default VPC, which will have an Internet gateway, making it easier for the instance to connect to the Internet.
Do you want it to launch in the default subnet of into your own subnet, also do you want to Auto assign public IP.
- Do not worry if you do not understand anything above 2 points, will make all of this information very easy to digest in a future Blog.
What placement group, capacity reservation to do, any join directory, or any specific IAM role to use.
How does the machine shutdown, hibernation needs to be enabled, any detailed Cloud watch monitoring to be done, please note this is a paid option so do not enable details cloud watch monitoring.
What type of tenancy will you choose, dedicated instance, dedicated host or shared instance.
Do you want your instance to burst from baseline performance in case of high load.
Do you want to mount an EFS file system on the creation time?
In certain advanced details, you can select to enable the Metadata information access, version, response hop limit etc.,
There is also an option to create a bootstrapping script.

For the matter of simplicity, do not change any option in this screen and just proceed to the next step to Add Storage.

Step 4: Add Storage¶

Every computer, be it server or desktop needs a hard disk. The hard disk can be magnetic or SSD. AWS provides all the options for hard drive. Depending on the type of instance chosen in the step two the option of Hard drive may change.

The screen will look like this.

If you see the option carefully, you can observe that there is also option to encrypt the data. Even you can attach additional storage (EBS) later. Select the default option and move to the next step, Add Tags.

Step 5: Add Tags¶

AWS gives you an option to add tags for everything. This will help in finding a particular instance and execute some script on those instances. The tags are case-sensitive.

The Add Tags screen looks like this.

Add a Name and value as Webserver tag. Proceed to next step a very important step to Configure the security group.

Step 6: Configure Security Group¶

Security is important in AWS, and as you learned in the Shared Responsibility model. The security of the instance is the user's responsibility.

The screen looks like this

You have to understand few things about security group. Security group is basically a firewall to the instance. The default is to deny all traffic. The port which you open is the only traffic allowed.

You may be aware, there are two needs to communicate with your AWS EC2 instance.

SSH to the AWS EC2 instance
- You should open port 22 for this.
- You can open it from anywhere in the world, denoted by 0.0.0.0/0 Source, in the above screen shot.
- You can even restrict the Source to be from just our public IP.
The AWS EC2 instance, should behave as a Web Server
- You can just open port 80 for this.

The default set in the above screen is allowing SSH to this particular AWS EC2 instance from anywhere in the world. This is good for our first AWS EC2 instance launch.

Select next and go to the last and final step of every AWS operation - Review.

Step 7: Review Instance Launch¶

Here you are in the EndGame. This screen looks like this.

This gives all the six steps you have done till now. This is the place you can review it. If you still want to change anything you can change it now. Press Launch if you have no changes to be done. It provides with a pop-up.

This Key Pair is necessary for SSH-ing to the Linux AWS EC2, and you can use this Key-Pair to generate RDP (Remote desktop) credentials in case of Windows AWS EC2. Save the eye pair (PEM file) in your local machine.

You are now ready to launch into Cloud, click Launch instance.

Once the AWS EC2 is created, you will be presented with this screen. This means the AWS EC2 is launching

Congratulation on launching your first AWS EC2 instance. Welcome to the Cloud.

What next? Login to AWS EC2 using ssh¶

You have launched your first AWS EC2 instance. What Next....

What do you do with it?

How do you run the application on this server?

You have to SSH to the AWS EC2 instance. To SSH into the AWS you need some information.

You should go to EC2 Dashboard > Instance, the screen will look like this.

If you see in the above screen, the AWS EC2 is still in Initializing phase. Please click the Connect button to get the connect information, which would look like this.

The above screen tells you all the information you need to connect.

To elaborate on the step, it is a two step process

Reduce the permission on the PEM key.
Connect to the AWS EC2 using the SSH command.

Reduce permission of PEM File¶

You should have the PEM file downloaded before, saved locally. Navigate to the directory where the PEM file is present. Execute this command to reduce the permission

chmod 400 <key_file>.pem

The chmod command above will only provide the read permission to only owner. Groups and Others have no permission on the files. Once you have reduced the permission move to the next step.

SSH connect to AWS EC2¶

This is the big step, where you actually connect to AWS EC2. The general structure of the command is,

ssh -i "key_file_name.pem" ubuntu@ec2-12-66-176-143.<region>.compute.amazonaws.com

Once you have the SSH executed your terminal should look like this.

You can execute some command on the SSH terminal, like for Ubuntu the most popular commands to execute is

sudo apt update && sudo apt upgrade -y

This is will update the software on the server.

Please think how you can launch an Apache2 Web Server in this AWS EC2 instance.

Terminate of launched AWS EC2¶

You have launched the AWS EC2 instance, in the free tier, it does not mean that you have infinite free usage. You should terminate the launched AWS EC2 instance. This will prevent you from unwanted cost.

Go to EC2 Dashboard > Instance, and click on Action, it will look like this.

Select the Instance State > Terminate. This will ask for confirmation like this,

Once selecting Yes, Terminate, it will start the termination process, and once done you should a screen like this.

With this, you have safely launched, executed and terminated an AWS EC2 instance. Congratulation on the big step.

Conclusion¶

When this article started, you may not have known what is an AWS EC2 instance, How to make it work. You may have never thought it could be so easy to create one.

You may still do not understand a lot of options which were selected in the above process, but you will understand each and every detail in the coming series of article.

Launching a new free-tier EC2 instance is a 7 Step process, and you can login to the instance by SSH in case of Linux OS and RDP for Windows.

Now you have your own piece in the cloud, do not forget to terminate the instance else, this piece of cloud will put a hole in your pocket.

Reference¶

Image by Comfreak from Pixabay
Photo by Anna Shvets from Pexels
Technology photo created by onlyyouqj - www.freepik.com
Photo by Wil Stewart on Unsplash

How to start using AWS Cognito

Animesh Bhadra 🎯 — Sat, 17 Apr 2021 08:46:23 +0000

Introduction¶

Does AWS provide any service which will offload my sign-up, login, user management responsibility?

What if I told you, AWS has a service which does all of the above, and also provide a hosted web UI which can used by you. It even provides data sync across devices. Everything is perfectly secure. I will help you understand everything you need to know about this service.

As a developer if you ever wished to focus only on the functionality or business logic of the application you are developing, and leaving the worries or sign-up, login, user management, data sync across devices in a safe and secure manner. Paying only based on the number of users per month. The AWS has answered your wish, and I will guide you through this.

As per AWS

You can focus on creating great app experiences instead of worrying about building, securing, and scaling a solution to handle user management, authentication, and sync across platforms and devices.

Let's jump right into the enticing world of AWS Cognito.

What Is Amazon Cognito?¶

The official definition from AWS:-

Amazon Cognito provides authentication, authorization, and user management for your web and mobile apps.

The most important concept in the above definition is authentication & authorization. This is provided using two components in AWS Cognito.

AWS Cognito User Pool
AWS Cognito Identity Pool

Originally AWS Cognito was used for mobile developers, who could use AWS Cognito for its authentication & authorization capabilities along with the user management.

AWS Lambda and ServerLess architecture have given a new dimension to use AWS Cognito. These developers can now offload user management of their application to an AWS Managed service.

AWS Cognito provides such developer with fully managed, scalable an cost-effective sign-up/sign-in service.

Before you jump into learning about User Pool and Identity Pool , you should have a fair understanding of the terms authentication & authorization. You may also need to understand federation.

Basics of Identity and Access Management (IAM)¶

There is a great article by Okta, which explains about IAM. Since you are here, I will summarize it.

Authentication¶

This is the first step in the security process of identity and access management.

Authentication is the act of validating that users are whom they claim to be.

The most common ways to authenticate user are:

User Name and password combination
OTPs
Biometrics
SSO (Social Sign On)

Authentication tells the application Who am I?.

Authorization¶

Authorization in a system security is the process of giving the user permission to access a specific resource or function.

Once a person is Authenticated, you have to provide him with relevant access. Even a Guest user, can be provided with minimum access.

You can divide your users, into these four categories.

Admin of the application
Authenticated User
Premium user (Paid)
Guest user

An Administrator can have a different view of the application than a normal authenticated user.

Even an authenticated user can be free user or a premium user, depending on the type, the view of your application may be different.

What type of experience you want to provide your user, decides the level of access.

Lets understand this concept used by an analogy in most of the company.

Most companies in pre-covid times used to give the RFID access card to its employee. Authentication means the process by which someone receives the RFID Access card. Once you receive your RFID access card, depending on the authorization of the employee, he may or may not have access to different parts of the office buildings.

Hope you are clear on these questions now

What is Authentication? --> This answers the question "Who am I?"
What is Authorization? --> This answers the question "What I can use?"

There is a third variable in this equation, which is called Federation. Lets understand What is federation?

Federation¶

The word federation, means a united, trusted relationship between two or more entities. To understand federation properly, you have to understand few other concepts.

Identity Federation
- It is a system of trust between two parties, to authenticate users and also convey the information required for giving authorization.
Identity provider
- The party in identity federation, which stores the user information, responsible for user authentication.
Service Provider
- The party in identity federation, which provides the service based on authentication and authorization provided by Identity provider.
Open Standards
- Identity federation is possible because of these open standards
- OIDC (OpenID Connect)
- SAML (Security assertion markup language) 2.0
- OAuth 2.0

When you book movie ticket online, you are authenticated by an online entity, who takes your money and gives you the ticket, when you go to the actual theater you are granted entry based on the ticket, you purchased online. In this case the online ticket vendor is the Identity provider , the theater is the service provider , and the bi-party arrangement is the Identity Federation.

Once the basic identity and access management is clear to you, let's move on to understand User Pool and identity pool.

User Pool¶

AWS Cognito User Pool , is a way to provide Authentication to user of an Application. It is represented as a user directory in Amazon Cognito.

The authentication mechanism provided by AWS Cognito User Pools is:-

Social Identity Providers
SAML Identity Providers
AWS Cognito User Pools, also provide authentication, or act as an identity provider.

In Federation, as explained, the Identity provider, stores the user information. When AWS Cognito User Pools are used as the identity provider, the user directory of AWS Cognito stores the user login details, else its store in the identity providers storage.

The user directory is accessible by an SDK. This can be used by applications to access user profile.

AWS Cognito User pool provide:-

Sign up and sign in service
A built-in customizable web-ui for user to register.
Social sign-in with social identity provider.
User directory management and user profile.
MFA.

Once a user is authenticated, the application receives a JWT (JavaScript Web token). The next step of authorizing uses this JWT.

Configure User Pool¶

When you select AWS Cognito in AWS console you get this screen, asking to choose .

Since you will configure the User Pool, lets choose the User Pool Option.

To Make life easier you can select the Review Defaults , and it could provide you with a good basic user pool.

You can also choose to configure each of these ten settings.

Customizing all the settings for user pool creation, would be beyond the scope of this Blog. Let's take these two approaches.

Create a user pool with the default option.
Add an App to enable the hosted WebUI.

Create a User Pool (Default)¶

Step 1 : Select the " Create a user pool".
Step 2 : It provides us with this screen.

Step 3 : Provide a name for Pool, and press the Review Defaults.
Step 4 : On pressing the Review Defaults , you get this Review screen.

The review pages, tells us these important information.

Pool Name
Email is a required attributes.
There is a password policies.
How the message's for AWS Cognito needs to be communicated.
MFA is enabled or not.
Tags are created or not
App Clients are registered or not.
Which are Triggers to configure. like pre sign-up, pre-authentication

If you carefully watch the Review page and the steps to create a user pool, they match.

Step 5 : Press the Create Pool , button and your User Pool is created.

Congrats on creating the default user pool. Now you should check out the Hosted UI provided by the AWS Cognito for sign-up and login.

Add an App to Enable the Hosted Web UI¶

AWS Cognito even goes a step ahead into offloading your user management work. It provides a user sign-in, login page as a hosted web. Let's see, how can you configure this.

You will use the default user pool created before. Once you have created a User Pool, you can edit a lot of attributes provided here.

To Use the hosted WebUi, you will focus on the App Integration property of the user pool.

Step 1: Select App Integration from Setting of User Pool.

To get the Web Hosted UI, you have to use this configuration, if you have your own domain, provide your custom domain, else use the AWS domain.

Step 2: Add Domain.

On choosing the Add Domain option, you get this screen.

Enter the domain, you wish, and keep a note of this, you will require it later.

Step 3 : Add App Client under General settings

Select the App Client under General Setting, so you can enter the app client attributes. The screen will look like this.

You should select the Add an app client option. The screen will look like this.

You should provide the name of the client, and de-select the option Generate client secret. This option can be used when you have a server side component to generate the client secret. Once the app client is created. We move to the Step 4.

Step 4 App Client Settings:

Select the App Client Setting, under App Integration. You will get a screen like this.

If you check the App Client details are already present, In the above screen, you have to select

Cognito User Pool as the enabled identity providers.
Since you are testing, provide " http://localhost" as the callback URLs, this is for validation.
Choose Implicit Grant , in Allowed OAuth Flows.
Select All the allowed OAuth Scope, you want.

Save the option.

Step 5 : Launch the WebHosted UI.

At the bottom of the previous screen, there is an option for Launch Hosted UI. Use this option. You should get a sign-up page like this.

Here is your simple web hosted a for Login and sign up. Though everything may not work. Just refer this as a guideline.

At this point you can use IAM roles for your application and this authentication to make your application function. But providing different level of authorization will still be the application's responsibility. If you want to hand over this part also then move to Identity Pool.

Identity Pool¶

AWS Cognito Identity pool does both Authentication and authorization, but in a different way.

The AWS Cognito Identity pool uses Federated identity for authenticating users. Different identity federation can be provided by

Social Identity Provider
SAML Identity Provider
OpenID Connect Provider
Amazon Cognito User Pool.

Kindly note, the AWS Identity pool is the service provider in the identity federation paradigm. It uses the identity providers to authenticate the user.

Once these identity providers do their magic (authentication), they inform the Identity pool by issuing a type of token. On receiving the token, identity pool will authorize users to a different level of access.

Identity pool will provide the user with different level of access by using IAM Roles.

Identity pool uses AWS STS, service to grant the users credentials to access AWS resources.

Configure Identity Pool¶

Configuring Identity Pool is a 2 Step Process.

Select Manage identity Pool from the screen when you created the user pool. You will be greeted with this screen.

Step 1 : Create Identity Pool

In this step you have to configure 3 things apart from providing a name to the identity pool.

Unauthenticated Identities
- AWS Cognitio provides support for Guest user. It generates a unique ID for each guest.
- In the future if they register, the complete session is saved into the user directory.
Authentication flow settings
- We can select a basic or an enhanced authentication flow.
Authentication providers
- We have All the Social Identity providers along with OpenID and SAML.
- We will use Cognito, which needs us to provide user pool id and App Client ID , which was created during the user pool.

Once you fill the required details, let's proceed to the most important step in AWS Cognito, providing Permissions.

Step 2 : Set Permissions

This is the screen, as you can guess, can provide two types of IAM Role to both Authenticated and Unauthenticated user. The IAM Role will be created here.

Here is the IAM policy statement for Authenticated Users in AWS Cognito Identity pool.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "mobileanalytics:PutEvents",
        "cognito-sync:*",
        "cognito-identity:*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

Here is the IAM policy statement for UnAuthenticated Users in AWS Cognito Identity pool.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "mobileanalytics:PutEvents",
        "cognito-sync:*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

Once you are ready, just select Allow. You will have your AWS Cognito Identity Pool created, just use this option for integrating with the SDK you want to use.

Getting Started with Amazon Cognito

Python Code for AWS Cognito¶

You might be thinking, how does it all come together. You have a user pool and an identity pool. You also created the Web Hosted UI. You might be thinking how do I use it together.

If you like video, to learn, visit the AWS Cognito Python tutorials by Paris Nakita Kejser. This is the only AWS Cognito's in Python video tutorial.

We will just pick two important flow from the above tutorials, as some changes need to be done in the code mentioned in the video.

Sign-Up using AWS Cognito, Python SDK Boto3¶

import os
import boto3
from dotenv import load_dotenv, find_dotenv

load_dotenv(find_dotenv())

# read the .env-sample, to load the environment variable.
dotenv_path = os.path.join(os.path.dirname( __file__ ), ".env-sample")
load_dotenv(dotenv_path)

username = "abc.xyz@gmail.com"
password = "#Abc1234"

client = boto3.client("cognito-idp", region_name="<region-name>")

print(os.getenv("COGNITO_USER_CLIENT_ID"))

# The below code, will do the sign-up
response = client.sign_up(
    ClientId=os.getenv("COGNITO_USER_CLIENT_ID"),
    Username=username,
    Password=password,
    UserAttributes=[{"Name": "email", "Value": username}],
)

There are certain prerequisites for this code to work.

Create a file called .env-sample, in the current directory where you have the above code. In this file you should provide the macro COGNITO_USER_CLIENT_ID, with the client ID from General Settings > App Client > App client id.

The above will be picked using the dotenv module.

When you execute the above code, you will get this back as a response,

{
   "UserConfirmed":false,
   "CodeDeliveryDetails":{
      "Destination":"a ***@g***.com",
      "DeliveryMedium":"EMAIL",
      "AttributeName":"email"
   },
   "UserSub":"123456-d094-44e0-942d-789012134",
   "ResponseMetadata":{
      "RequestId":"123-1842-4027-345-789abc09234",
      "HTTPStatusCode":200,
      "HTTPHeaders":{
         "date":"Mon, 19 Apr 2021 05:11:44 GMT",
         "content-type":"application/x-amz-json-1.1",
         "content-length":"175",
         "connection":"keep-alive",
         "x-amzn-requestid":"123-1842-4027-345-789abc09234"
      },
      "RetryAttempts":0
   }
}

If you check the General Setting > User and groups , the user will be unconfirmed, you will see this, and get a verification code in your email.

Confirmation Code using AWS Cognito, Python SDK Boto3¶

Now you are an unconfirmed user, you have got the confirmation code in the mail. Let's find a way to make you a confirm user. Here is the code to do that.

import os
import boto3
from dotenv import load_dotenv, find_dotenv

load_dotenv(find_dotenv())

dotenv_path = os.path.join(os.path.dirname( __file__ ), ".env-sample")
load_dotenv(dotenv_path)

username = "abc.xyz@gmail.com"

client = boto3.client("cognito-idp", region_name="<region-id>")

print(os.getenv("COGNITO_USER_CLIENT_ID"))
confirm_code = "112418"

# Below API sends the confirmation code.
response = client.confirm_sign_up(
    ClientId=os.getenv("COGNITO_USER_CLIENT_ID"),
    Username=username,
    ConfirmationCode=confirm_code,
)

print(response)

Post this the response is this.

{
   "ResponseMetadata":{
      "RequestId":"3412d123-c175-4571-91a5-12349c14a9bd",
      "HTTPStatusCode":200,
      "HTTPHeaders":{
         "date":"Mon, 19 Apr 2021 05:22:10 GMT",
         "content-type":"application/x-amz-json-1.1",
         "content-length":"2",
         "connection":"keep-alive",
         "x-amzn-requestid":"3412d123-c175-4571-91a5-12349c14a9bd"
      },
      "RetryAttempts":0
   }
}

If you again go and check in General Setting > User and groups , the user should be confirmed now.

You have successfully created your first app user.

Login and Getting User details using AWS Cognito¶

You have now successfully created a new user, and also confirmed the user. The next logical step will be to Login and get some user details from AWS Cognito.

This you can achieve in this manner.

import os
import boto3
from dotenv import load_dotenv, find_dotenv

load_dotenv(find_dotenv())

dotenv_path = os.path.join(os.path.dirname( __file__ ), ".env-sample")
load_dotenv(dotenv_path)

username = "abc.xyz@gmail.com"
password = "#Abc1234"

client = boto3.client("cognito-idp", region_name="ap-south-1")

print(os.getenv("COGNITO_USER_CLIENT_ID"))

# Initiating the Authentication, 
response = client.initiate_auth(
    ClientId=os.getenv("COGNITO_USER_CLIENT_ID"),
    AuthFlow="USER_PASSWORD_AUTH",
    AuthParameters={"USERNAME": username, "PASSWORD": password},
)

# From the JSON response you are accessing the AccessToken
print(response)
# Getting the user details.
access_token = response["AuthenticationResult"]["AccessToken"]

response = client.get_user(AccessToken=access_token)
print(response)

Please note sometime you may get this error

botocore.errorfactory.InvalidParameterException: An error occurred (InvalidParameterException) when calling the InitiateAuth operation: USER_PASSWORD_AUTH flow not enabled for this client

If you get this error, please check in General Settings > App Client > Auth Flow Configuration

You should have this option selected ALLOW_USER_PASSWORD_AUTH selected, or just for testing enable all the option like this.

You will get a JSON as a response of initiate_auth, you have to just pick the AccessToken from it and pass it to get_user. Once that is done, you will get this as a response.

{
   "Username":"abc.xyz@gmail.com",
   "UserAttributes":[
      {
         "Name":"sub",
         "Value":"1234eb31-d094-44e0-942d-50a1234a66b"
      },
      {
         "Name":"email_verified",
         "Value":"true"
      },
      {
         "Name":"email",
         "Value":"abc.xyz@gmail.com"
      }
   ],
   "ResponseMetadata":{
      "RequestId":"xxxxxxx-1231-4f1c-b881-dcf10c54e576",
      "HTTPStatusCode":200,
      "HTTPHeaders":{
         "date":"Mon, 19 Apr 2021 08:26:10 GMT",
         "content-type":"application/x-amz-json-1.1",
         "content-length":"213",
         "connection":"keep-alive",
         "x-amzn-requestid":"xxxxxxx-1231-4f1c-b881-dcf10c54e576"
      },
      "RetryAttempts":0
   }
}

This is enough to understand how AWS Cognito works, you can even follow the video for Forgot Password flow also.

AWS Cognito Sync¶

The official documentation of AWS Cognito Sync is.

Amazon Cognito Sync is an AWS service and client library that enables cross-device syncing of application-related user data.

Few use cases of the AWS Cognito Syncs are.

Synchronize the user profile data across the mobile devices and the web, without a back-end.
The application can cache data locally if there is no connectivity, and once you get connected the data are synced.
The AWS Identity pool is required to use the AWS Cognito Sync.

Conclusion¶

AWS Cognito is a service to use if you want to offload complete user management tasks to AWS. This will cost a little, but you can continue to focus on the good things of your application and ignore the mundane activity of user management.

You can rest assured that AWS will take responsibility of upgrading the service for latest security patches, and you are not exposed to such security flaws.

AWS Cognito User pools and Identity pools are the two brothers of AWS Cognito, shouldering the responsibility of authentication and authorization. Using IAM roles, you can provide very fine grained access to users.

Integration using SAML, OIDC also help in using 3rd party vendors as your Identity Providers. Even the Guest users can be assigned a unique ID which can later be saved to a user profile, if he registers.

Though you may never use the AWS Cognito provided Web Hosted UI, but it makes a great point in AWS service, how thought out their services are.

The AWS Boto3 SDK can use the AWS Cognito APIs to provide the complete functionality of User Management in your application with the most minimal amount of code.

This would have given you a fair understanding of AWS Cognito Service. Let me know if you tried the Python Code to login the user.

Reference¶

Info graphics¶

Authentication Vs Authorization¶

Authentication	Authorization
Validate the user	provide access to resources
Passwords, biometrics, OTP used for authentication.	Policy and rules are used to grant access.
First Step in IAM	Follows authentication, (exception - Guest user)
OIDC 2.0	OAuth 2.0
ID Token are used	Access tokens are used
User has some control	User have no control

AWS Cognito User Pool Vs Identity Pool¶

User Pool	Identity Pool
Authentication	Authorization
User Management	IAM Roles for Access
Sign-Up & Login	Fine Grain Access
Provide WebHosted UI	Complete back end access
User Pool is charged	Identity pool is free
No Guest user	Guest User

Using AWS Cognito User Pool With Identity Pool¶

When you have both AWS Cognito user pool and identity pool, they can function together with each other to provide access to user for AWS resources. These can be done in three steps.

The Application uses the AWS Cognito User Pool, to authenticate, and gets tokens in return.
The application exchanges this tokes with the AWS Cognito Identity Pool and received AWS credentials in returns.
Use the AWS Credentials, and use to access the AWS resources.

When you are using AWS Cognito User Pool With Identity Pool, the flow is explained above.

The application authenticates and get token from AWS Cognito User Pool as a JWT Token.
This JWT Token is then passed on to AWS Cognito Identity Pool, which returns an IAM Roles for the user.
Once the IAM role is assigned, the user can access any resources on AWS.

The AWS Inspector Blog of your dreams

Animesh Bhadra 🎯 — Sun, 11 Apr 2021 06:47:36 +0000

Introduction¶

You have learned about the AWS Shared responsibility model, Have you stopped and thought, how do you check if your VPC is accessible from outside network? Does my application compromises the EC2 in any regards?

You should remember that network accessibility and software security is the user's responsibility in AWS Shared responsibility model.

Being a beginner using the AWS, this responsibility may seem daunting. No worries, when AWS gives you a challenge, it also shows you the path.

You will learn how can we use an AWS Service to check the security vulnerability of your software. You will also learn how this service helps in checking the network accessibility of the VPC.

Let's dive into the inspected world of AWS Inspector.

What is AWS Inspector?¶

AWS inspector is a service provided by AWS, which helps you in two ways

Finding security vulnerabilities in your software.
Checking the network accessibility of the VPCs.

AWS inspector gives findings for the checks done, on which you can act on. You can use the findings and corrects the weakness in your application or the network.

AWS Inspector Agent¶

You should be thinking by now, okay, network assessment can be done via some external tools or service given sufficient permission via the IAM Roles.

How does AWS inspector do a security vulnerability test on an EC2 instance? Is some application is running beyond your knowledge in your own EC2 instance.

The answer to this is, When you enable the AWS Inspector to do a security vulnerability test on your application running on the EC2 instance, it asks permission to install an AWS inspector agent on the EC2 instance.

The AWS inspector agent does software telemetry for application and the OS running on the EC2 instance.

It provides various information about EC2 instance and the application running on it. Installation of this AWS inspector agent is optional.

If installed, AWS inspector agent monitors

Behavior of the EC2 instance.
checks the network file system
process activity
collects a lot of behavior and configuration data

Benefits of AWS Inspector¶

You have learned that AWS inspector does network assessments and security vulnerability checks on the EC2 instance, is this the only use of the AWS inspector?

No, you do get the other benefits of using the AWS inspector including

Automation
- You can integrate the security vulnerability, and network assessments automatically in your CI/CD pipeline.
- This gives your findings if any security or network related check are broken in the upgrade and could be corrected.
Application security
- The application security checks also can be automated, providing you with valuable information.
- AWS inspector vulnerability scanning when automated, helps in finding issues which can lead to hacking of your application.

Caution while using AWS Inspector.¶

You shouldn't relax knowing that AWS inspector does both network assessments and security vulnerability test. This should not give you a false sense of security that you will find all types of vulnerability by just running the AWS inspector.

AWS inspector helps in finding some of the security issues with your EC2 instance and application running on it.

AWS Inspector does not find issues in real time by log analysis like an AWS GuardDuty, or AWS Trusted Advisor, which even provides optimization techniques for your architecture.

The application running on the EC2 instance, and it's instance configuration itself poses complexity, which AWS inspector may not be configured for.

We as a user have a responsibility, which we should fulfill by running some complementary test like the AWS GuardDuty or AWS Trusted Advisor.

AWS inspector is part of the security and network monitoring not the heart of it.

AWS Inspector Pricing¶

Pricing is an important parameters while choosing a particular AWS Service, especially when we have 3rd party tools also competing

The AWS inspector pricing is based on these two dimensions.

Number of EC2 instance included in each assessment.
Number of rules invoked in each run.
- Host assessment
- Common vulnerability and exposures ( CVE )
  - CVE is a mission which identify, define and catalogs publicly disclosed CyberSecurity vulnerabilities.
- Center for Internet security ( CIS ) benchmarks
  - CIS provides more than 100 configuration guidelines across 25+ vendor product families to safeguard systems against today’s evolving cyber threats.
- Security best practices
- Runtime behavior analysis.
- Network assessment

For more detailed pricing report, visit the AWS official pricing page. The pricing on Free Tier is very easy, read along to find out.

Free tier¶

You do not have to bother about all the above complexities, accounts which have never run AWS inspector once, are eligible for

250 agent assessments with host rules package
250 instance assessments with the network reachability.

In the first 90 days. Other assessments will be billed at the normal price.

AWS Inspector service limits¶

AWS Inspector has a predefined service limits for different resource you can use. AWS inspector provides these four major categories for resources.

Number of instance running assessment.
Number of assessment running.
Number of various assessment template in every assessment.
Number of assessment targets

Getting started with Amazon Inspector¶

Let's dive into configuring AWS inspector, and how it can be used.

There are certain prerequisite for starting AWS inspector configuration.

Prerequisite for AWS Inspector¶

You much have at least one EC2 instance running. What will AWS inspector check if there is no EC2 instance running?
For Host assessment, you may need to install the AWS inspector agent on the EC2 instance.

Lets first see how to configure AWS Inspector.

You can see that AWS inspector uses a service-linked role , to describe the EC2 instance and network configuration.

You can see there are two types of AWS inspector setup we can do.

Network Assessment (Inspector Agent is not required.)
Host Assessment (Inspector Agent is required.)

The default option is the most easiest configuration to trigger AWS Inspector. Let us understand Network Assessment and Host Assessment.

Network Assessment¶

The checks performed by AWS inspector without the installation of an agents are

Network configuration analysis to checks which ports are reachable from outside of the VPC.

If you have the AWS inspector agents installed , it can provide you with additional information like,

The process whose ports are reachable from outside of the VPC.

Host Assessment¶

Host assessment requires the installation of the AWS inspector agent, so once it is installed we get this information

Common vulnerability and exposures (CVE)
- The host is checked towards the know CyberSecurity vulnerabilities.
Center for Internet security (CIS) benchmarks
Security best practices

Once you click on Run Once , the confirmation screen is displayed like this.

When we complete the test, we will receive a findings from the test. Before looking into findings, lets see there is an Advanced setup.

AWS Inspector | Advanced Setup¶

As shown in this screen, advance setup is a three step process.

Define an assessment target
Define an assessment template
Review.

Advance Setup | Define an assessment target¶

You might have guessed by now, an assessment, target is the AWS resources on which you can run the AWS inspector. As of now it is restricted by the operating system and region.

Network reachability test can be run on any EC2 instance without using the AWS inspector agent.

For running the assessment with an AWS inspector agent let us first check the supported Linux based operating systems.

64-bit x86 instance
- Amazon Linux 2
- Ubuntu
- Debian
- Red Hat Enterprise Linux
- CentOs
ARM instance
- Amazon Linux 2
- Red Hat Enterprise Linux
- Ubuntu

The supported Windows operating systems are

Windows Server 2019 Base
Windows Server 2016 Base
Windows Server 2012 R2
Windows Server 2012
Windows Server 2008 R2

The supported AWS regions are

US East (Ohio) us-east-2
US East (N. Virginia) us-east-1
US West (N. California) us-west-1
US West (Oregon) us-west-2
Asia Pacific (Mumbai) ap-south-1
Asia Pacific (Seoul) ap-northeast-2
Asia Pacific (Sydney) ap-southeast-2
Asia Pacific (Tokyo) ap-northeast-1
Europe (Frankfurt) eu-central-1
Europe (Ireland) eu-west-1
Europe (London) eu-west-2
Europe (Stockholm) eu-north-1
AWS GovCloud (US-East) gov-us-east-1
AWS GovCloud (US-West) gov-us-east-2

The first task of defining the assessment, target is to give it a name.

Then you have an option to run it on all EC2 instances in your account, or you can run it based on certain tags on the EC2 instance. Generally, we can run these assessments only on the production tagged system.

The AWS inspector agent is pre-installed on Amazon Linux AMIs. If you want to install on other AMIs manually you may have to uses AWS System Manager service. Best option is to use the install the agent automatically.

Advanced Setup | Define an assessment template¶

As shown below,

The first task is to Name the assessment templates. Once done, we have to define the rules packages to use. By default, there are 4 rules packages to select.

Common Vulnerabilities and Exposures-1.1
CIS Operating System Security Configuration Benchmarks-1.0
Network Reachability-1.1
Security Best Practices-1.0

Once you select which of these 4 rules you want to run, next selection to be done is the duration of the test.

In addition, you also also schedule the assessment to be recurring.

The findings of these assessments can also be fed to an SNS topic.

Advanced Setup | Review¶

As you might have seen many a times, the final step is the Review step.

Post all this you can just run the inspector assessment.

AWS Inspector Findings¶

Once you run any AWS inspector assessment, the result is called Findings. These are the potential security issues that AWS inspector has found during its assessment. Findings is not generated while the assessment is on going, it is only generated after the assessment is completed.

Individual findings from AWS Inspector cannot be deleted. You have to delete the completed assessment run.

Conclusion¶

AWS works in Shared responsibility model. In this model, as a user of AWS services you have certain responsibility. Two of them primarily are.

Check the network reachability of the VPCs.
Check the application's security running on the VPCs.

It would have been very difficult for you to accomplish this on your own. Just like for many other services, AWS has provided a managed service for this called the AWS Inspector.

AWS Inspector is a managed service which helps in finding security vulnerabilities in the application running on your EC2 instance and also checks if the instance's VPC is reachable from outside.

If we install an AWS inspector agent using the AWS system manager service, then we get additional telemetry about the application running on the EC2 instance.

You should not get a false sense of security thinking that AWS inspector will find all types of security vulnerabilities, It finds some, but it still does not find all the different possible vulnerabilities, it will be in your best interest to find some other alternatives to test complete security vulnerabilities.

AWS inspector has a limitation today on the type of resources it can evaluate, like it can evaluate the security vulnerabilities of an application on an EC2 instance running selective Linux and windows operating system. Though you can use its network reachability test for any type of hardware.

AWS inspector provides 250 agent and instance, assessment for the free tier, and then normal pricing.

Before running AWS inspector you should at least have 1 instance running and the agent should be installed.

You can use the most basic configuration to trigger the assessment which does both types of checks. The report or findings of the test, inform you about the vulnerabilities in different form.

AWS inspector is a great tool for doing the self evaluation of the application running on the EC2 instance, and also check the network reachability.

Info graphics¶

AWS Inspector Vs AWS GuardDuty¶

AWS Inspector	AWS GuardDuty
Finds if known threat exists.	Finds threat from different log source.
Static analysis from configuration and settings.	Dynamic analysis from multiple log source.
Scheduled timings.	Continuous monitoring
EC2 and VPC is monitored	Multiple services are monitored.. even S3
Available in 14 region	Available in 24 region
Free tier use of 90 days	Free tier use of 30 days
Pricing based on number of assessment.	Pricing based on volume of logs analysed.

AWS Inspector Vs AWS Trusted Advisor¶

AWS Inspector	AWS Trusted Advisor
Agent-based	Agent-less
No impact on performance	Improves performance by checking service limit
Free tier	Premium support
EC2 configuration	AWS account & administrations
No cost recommendations	Recommendations to optimize cost
Scheduled	Real time guidance
No impact on performance	Improves performance by checking service limit

Reference¶

Let me know if you run the AWS inspector assessment on your EC2 instance, and what are its findings 👇.

The Dummies Guide to AWS KMS.

Animesh Bhadra 🎯 — Fri, 02 Apr 2021 07:11:48 +0000

Introduction¶

This is the article you have been waiting for to gain knowledge on security, encryption, cipher text. All these topics puts a scare in a lot of us. You have always wished if somehow these topics just don't cross your road. Just like the black cat.

I am going to explain you these topics with interesting visuals, charts and info graphics. This will help you to hold a conversation with your colleague when such scary topics are being discussed.

In Short, if you want to score few marks of these topics in AWS Certification exams. If you want to have a conversation with your team mates next time, when these topics come up. This is the article you have been waiting for.

AWS provides a managed service which does a lot of heavy lifting for us. This service is like Rubeus Hagrid, the guide to novice Harry porter, though the woods of The Forbidden Forest.

This service provides a mechanism to encrypt and decrypt data, 2 of the most important task in encryption. It also helps us with envelope encryption. It can also generate the data keys for you.

This service is everything you wished for when you wanted to make your application secure and your data encrypted. This service also automatically integrates with lots of other AWS service, making it easier to encrypt the data.

Lets jump right in.. to the mystical world of AWS KMS.

What is encryption?¶

Before you get into the journey to understanding the AWS provided service. You should understand what is encryption in its simplest form.

As mentioned by Wikipedia, Encryption is

Encryption is the process of encoding information. This process converts the original representation of the information, known as plaintext , into an alternative form known as ciphertext.

You can consider it like this, there is a black box, into which we pass a plain text and get an encrypted text, which no one can understand. This helps in keeping the data secure, if we delete the plain text now, no one can decode the encrypted text back.

The process to get back the original plaintext differs on the basis of the encryption used.

There are two types of encryption used.

Symmetric Encryption
Asymmetric Encryption

The name derives from the fact that if they are using the same key to encrypt or decrypt the data.

Symmetric Encryption¶

In symmetric encryption, we use the same key to encrypt and decrypt the data.

Consider your home lock, it opens and closes with the same key. This is same as symmetric encryption.

Asymmetric Encryption¶

In Asymmetric Encryption, we have to encrypt the data with a public key which is known to everyone, but the decrypt happens with a specific private key known only the authorized person. In this case, both the keys are different.

Consider the bank locker, to open the locker, you need your key, and the bankers key. You cannot open with just one key. Asymmetric encryption is like the bank locker.

AWS KMS Definition¶

As per AWS, a KMS is

AWS Key Management Service (AWS KMS) is a managed service that makes it easy for you to create and control customer master keys (CMKs), the encryption keys used to encrypt your data.

This is the most apt definition for AWS KMS you will ever find. Trust me, I have searched. AWS KMS creates and controls the customer master keys (CMKs) only. For you to understand this is the master key which you have, and it can open any lock for you. This is the skeleton key for you.

If you notice carefully in the definition, it does not mention about the storing of keys, it just mentioned create and control. As you go further in this article you will understand why is that the case.

You might be getting an inkling, why is AWS KMS important after reading about the definition, encryption and its type. You can store a lot of data on AWS, using various services, namely AWS S3, RDS, EBS, etc. To keep the enterprise data on public platforms in a plain text form is not advisable.

So let's jump into the world of AWS KMS, by learning about some key concepts.

Customer Master Keys (CMKs)¶

Customer Master Keys (CMKs) are the core of AWS KMS. This is a logical representation of a master key. It is assumed that a CMKs have these metadata.

Key ID
Creation Date
Description
Key State
Key Materials

You cannot see all these metadata upfront. CMKs are used to encrypt or decrypt up to 4KB of data. As there are different types of encryption, we have corresponding types of CMKs.

Symmetric CMKs
- A 256 byte key, which is used for both encryption and decryption.
Asymmetric CMKs
- represent the RSA Key Pair.

The Symmetric CMKs keys, and the private key of Asymmetric CMKs never leave the AWS KMS unencrypted.

The CMKs provide APIs for various programming languages, like Boto3 for Python, which can use the AWS KMS Api's to get the task done, in place of the actual physical key materials which is not visible in AWS KMS.

As a user we only have to option to delete the CMKs but not modify any of the key metadata.

AWS KMS supports 3 different type of CMKs.

Customer Managed CMKs
AWS Managed CMKs
AWS Owned CMKs

AWS services manage to use the above 3 types of CMKs in different manner, some use only the Customer Managed CMKs, some use only AWS Managed CMKs or AWS Owned CMKs, and some gives the flexibility of all 3.

Customer Managed CMKs¶

These are the CMKs which are created, owned and managed by you. These are under your full control.

You can find the customer managed CMKs in the AWS Console. Just remember that Customer Managed CMKs has a monthly fee even in the free tier. A word of advice, after learning about customer managed CMKs, please delete the keys, not disable it.

AWS Managed CMKs¶

These are the CMKs which are automatically used by some AWS services when we decide to encrypt the data. These are completely in control of AWS, and you as a user have no control on these CMKs.

AWS Managed CMKs are free to use in free tier. You may have to pay for excess usage.

Data Keys¶

The Keys used to encrypt data, is called a Data Keys. CMKs are used to generate, encrypt and decrypt the data keys. Please note, we are using CMKs to generate, encrypt and decrypt the data keys and not the data itself.

AWS KMS does not take the responsibility of storing, managing or tracking your data keys.

Create a data key¶

When you use the AWS KMS APIs, for encryption, the generation of data keys happens automatically. You can explicitly create a data key by calling the API, GenerateDataKey.

The API returns 2 things.

Plain Text copy of the data key.
The data Key encrypted with the CMKs.

Encrypt a data key¶

AWS KMS does not have a mechanism for you to encrypt the data key, You can use the OpenSSL library to encrypt the data with Data key, or use the AWS Encryption SDKs.

The plain text data key generated above, can be used to encrypt the data using the OpenSSL Library or the encryption SDKs. The encrypted data can now be stored safely, but delete the plain text data key as soon as possible.

Decrypt a data key¶

Now you have stored the encrypted data, at some point you want to decrypt the data, You can use the decrypt API, which will decrypt the data key, and returns the plain text data key. This plain text data key can now be used to decrypt the data.

Envelope encryption¶

By definition, Envelope Encryption is,

Envelope Encryption is a practice where the data is encrypted with a key (data key), and then the key (data key) in turn is again encrypted.

To understand the above statement more clearly, you can take the analogy of when you leave your house for a long time, you lock your important document and valuable in a safety lock. You then keep the keys to this safety lockers inside another room or lock. Finally, you lock the front of your door and leave.

In above case the data (your valuable) & your key (safety lockers) are in the same place (your home), but you have the master key to the house (CMKs). This helps you leaving your house in peace.

In AWS KMS, Customer Master Keys (CMKs) are the master keys, which never leave AWS KMS unencrypted.

You can create a chain of such encryption cycle, and the reason why envelope encryption works is for this reason.

Encrypting Data is a slow process, but encrypting just the data key is a fast process.
Encryption of data keys, gives an added layer of security for you, and it helps you storing the encrypted data keys along with the data.
Different encryption algorithms can be used to encrypt multiple keys in each layer of the envelop.

Create A CMKs on AWS¶

You are now well versed with the major concepts in AWS, now lets dive into how to create a CMKs and its usage.

One thing you should note that a lot of old blogs mentions the AWS KMS as an option under AWS IAM, it is not the case now. AWS KMS is an independent service.

The first step in AWS KMS is to create a CMKs. Creation of CMKs is a five step process.

Configure Key ⚙️
Add Label 🏷️
Define key administrative 🚶
Define Key usage permissions 👨‍💻
Review 🔍

Lets get you a AWS KMS CMKs.

Create CMKs | Step 01 | Configure Key ⚙️¶

Here is the screen for your reference.

You learned about Symmetric encryption & Asymmetric encryption, in the topic above What is encryption?.

You have chosen one of those options for your own CMKs, lets keep is simple and choose Symmetric encryption, as you will use the same key to encrypt and decrypt data.

The default Advanced Option is selected as KMS, you can keep the same. The other 2 option are for giving your own Keys to encrypt the data or take it from CloudHSM.

Create CMKs | Step 02 | Add Label 🏷️¶

Let me take you through this option one at a time.

You have to give an Alias first, this is the name you will use to reference this key in your code, or for any other service. Give a good name and a description of the Key. You cannot go to the next step without giving the Alias name.

The next option is to give a Tag name, you can provide a key/value pair which you can easily reference in code or in service to identify the key in addition to the name you provided.

Create CMKs | Step 03 | Define key administrative 🚶¶

Once you are done with naming your keys, you have to find out off all the IAM user's or Roles who can be the Admin of these keys.

AWS KMS CMKs administrators are those people who can administer the keys, but they cannot use the key for cryptographic operation, which make it important for the next step.

Select the Administrator User or role, who will not be using it for cryptographic operation, and lets go to the next step.

Create CMKs | Step 04 | Define Key usage permissions 👨‍💻¶

You have the administrators of your AWS KMS CMKs, now lets get one of the IAM role or user be selected as the user of the key. These selections enable this user with two extra policies.

Policy to use the CMK directly.
Use the CMKs with the AWS Services.

Create CMKs | Step 05 | Review 🔍¶

The final step for you in most of the AWS configuration is Review. Check all the details, and see the key policy document. What ever we discussed in the last two steps will now make sense to you.

You analyze the key policy mentioned below, and understand what are the special permission given to the administrators and the user.

{
    "Id": "key-consolepolicy-3",
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Enable IAM User Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789012:root"
            },
            "Action": "kms:*",
            "Resource": "*"
        },
        {
            "Sid": "Allow access for Key Administrators",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789012:user/XYZ"
            },
            "Action": [
                "kms:Create*",
                "kms:Describe*",
                "kms:Enable*",
                "kms:List*",
                "kms:Put*",
                "kms:Update*",
                "kms:Revoke*",
                "kms:Disable*",
                "kms:Get*",
                "kms:Delete*",
                "kms:TagResource",
                "kms:UntagResource",
                "kms:ScheduleKeyDeletion",
                "kms:CancelKeyDeletion"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Allow use of the key",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789012:user/ABC"
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Allow attachment of persistent resources",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789012:user/ABC"
            },
            "Action": [
                "kms:CreateGrant",
                "kms:ListGrants",
                "kms:RevokeGrant"
            ],
            "Resource": "*",
            "Condition": {
                "Bool": {
                    "kms:GrantIsForAWSResource": "true"
                }
            }
        }
    ]
}

administrators
- The administrators is XYZ in the above policy, check carefully what you see.
- These administrators are given the Actions to most of the AWS KMS administrators an allow effect.
- This administrators do not have any encryption, decryption or generate key action associated with it.
user.
- The user is ABC in the above policy,
- The user as you see is granted the Action of encrypt, decrypt, generate and other key related uses with an allow effect.
- The user also has an addition allow effective on the AWS KMS Grants, which is mostly used by AWS Services when encrypting data at rest.

This makes you the owner of a CMK. Please keep this in mind, the AWS KMS CMKs are charged, so if you are finished learning delete the key immediately, and I would like to remind you, do not disable the key, delete the key.

AWS KMS API¶

When you have your CMKs configured, you obviously want to use it. You should wait a bit before using it. Let me explain to you a few of the AWS KMS APIs.

There are many APIs in AWS KMS, the full list of AWS KMS api's are mentioned here. We will not go through all of them, but the three most important once, which you can use in a fun manner in the next section.

GenerateDataKey - Returns a plain text and a cipher text version of a data key.
Encrypt - Encrypt the plain text using a CMKs
Decrypt - Decrypt the cipher text which was encrypted with the Encrypt API.

The name is self describing but still let me explain you the little fundamental about the APIs.

GenerateDataKey¶

When you are using an SDKs, you generally do not need to use the GenerateDataKey, because you can use the Key Id generated above.

This API generates a symmetric data key for you, which you can use for client side encryption.

It generates two keys for you.

Plain text copy of the data key
encrypted copy of the data key with the CMKs.

Once the client side encryption of data is done and stored, the plain text copy of the data key can be deleted and the encrypted copy of the data key can be kept along with the encrypted data.

GenerateDataKey always returns a unique data key on each call.

There are different variations of the same API, which you can consider to use.

GenerateDataKeyWithoutPlaintext - Which will only generate the encrypted copy of the data key.
GenerateDataKeyPair - Generates the Asymmetric data key.

Encrypt¶

Once you have the Key id generated, you can use the encrypt API to encrypt your data. This API just encrypts the plain text data into ciphertext using the CMKs we generated.

This API has two primary use cases handled.

Encrypt small amount of data, like a database password.
It can be used to move encrypted data from one Region to another.

Decrypt¶

The name itself tells you, it will decrypt the data we have encrypted till now. We do not need to pass the Key Id in the API, if we are symmetric encryption, which is our case anyway.

AWS KMS API in action with Boto3¶

Finally the code which uses the Boto3 SDK to do all the good things for you.

import boto3
kms = boto3.client('kms')
key_id = 'alias/test' # this should be present in the KMS
database_password = 'Lorem-ipsum-dolor-sit-amet.'
result = kms.encrypt(KeyId=key_id, Plaintext=database_password) # result will now have these fields
# ChiphertextBlob - encrypted data
encrypted_password = result['ChiphertextBlob']
decrypt_result = kms.decrypt(ChiphertextBlob=encrypted_password) # will have the password decrypted
decrypt_result['Plaintext']

key_id = 'alias/test'
- this is the key id you created the label., you have to add alias/ before the actual label.
result = kms.encrypt(KeyId=key_id, Plaintext=database_password)
- You will call the encrypt() API, passing the key id, and the text which has to be encrypted.
- This API is only used to encrypt small amount of data.
encrypted_password = result['ChiphertextBlob']
- You can retrieve the encrypted password which is mapped with a key value of ChiphertextBlob in the result.
decrypt_result = kms.decrypt(ChiphertextBlob=encrypted_password)
- You can decrypt, the encrypted_password using the decrypt api.
- You were using symmetric encryption, this is the reason you are not passing the key id.
decrypt_result['Plaintext']
- You can retrieve the plain text password back from the key value of Plaintext

The above is a very simplistic example of the use of Boto3 for encrypt & decrypt APIs. You can explore further once you have completed the AWS certification. For now the understanding of these three important APIs is enough.

Conclusion¶

You might have always wondered about cloud computing, How does the data in the cloud is secured? Though there are multiple service are involved in AWS to provide that data security.

You came to know about one of the building blocks in that data security infrastructure.

You understood that encryption is the way of encoding information with a key so that you can transmit data securely. You also found that you can use the same or a different key for encrypting and decrypting.

AWS KMS is a service which provides a way to create Customer master Keys (CMKs), which can be made by you, or AWS or owned by AWS. This key is the master keys, which you can use the data key or small plain text data.

When encrypting large chunks of data, we cannot use the CMKs, but we need something called data keys, which can be generated using the CMKs we created. You can use these data keys to encrypt the data, and deleted the plain text data key. You can store the encrypted data keys with the data.

AWS KMS provide different APIs to for you to generate the data key. AWS KMS does not give the option to encrypt and decrypt the data, You have to use the OpenSSL library, or the AWS Encryption SDKs. AWS KMS does not own the responsibility of storing the data key, you are the owner to store it.

In encryption you also understood envelope encryption, where you can use multiple keys to encrypt the data. You can use the combination of symmetric and asymmetric encryption to encrypt the data.

You followed the 5 step process to create an AWS KMS CMKs, You also found out about the three AWS KMS API, GenerateDatakey, Encrypt and Decrypt. You also used the AWS Boto3 SDK for Python to create a sample understanding of the process to encrypt and decrypt data.

Overall you have now a fair understanding of the AWS KMS. You can further enhance the reading of AWS KMS, by following the various articles provided in reference.

Reference¶

Let me know if you used the AWS Boto3 SDK to encrypt and decrypt data. How was your experience, if you could not encrypt or decrypt the data, please give the error in the comment and I will try to help you.

Who really uses AWS IAM API keys?

Animesh Bhadra 🎯 — Mon, 29 Mar 2021 05:19:58 +0000

Introduction¶

Developer community uses the cloud technologies the most. In your conquest to learn about AWS, you have been focusing on configuring thing using the AWS console. As a developer, you may not find the use of AWS console efficient to do the tasks.

Sounds like Neo's trapped in the matrix. It's you, but you didn't find Morphaeus. You have no knowledge of the existence of the RED pill to show you the truth.

Soon you can access AWS Cloud with tools like CLIs, SDKs or HTTP APIs. These are the tools you completely understand. Even if you don't, sit tight we will make it easy to flow along.

You will first create a user with programmatic access, then progress to configure the developer's machine to connect to the cloud. After this, you will get information from the cloud using programmatic access.

Stay seated and enjoy this journey of programmatic access to the promised Earth of the Cloud.

IAM API Keys¶

You have already found out that there are three ways to connect to AWS. The AWS console is the most basic way to access. AWS CLIs and SDKs provide a much better way to access, as this can be controlled programmatically.

To revise we will list does the various ways to access AWS.

AWS Console
AWS SDKs
AWS CLIs
- Windows PowerShell

You can use any of the above methods to access AWS, all of them use the AWS APIs in the back-end. The way of access to AWS does not change the features of AWS.

You might think, AWS Console is easy to use, there is a web interface, you type in the username and password, which allows access to AWS.

How will you use the AWS using these programmatic ways.

IAM API Keys to the rescue¶

You need to have access to IAM API keys to enable programmatic access to AWS. This API Keys is tied up with an IAM user, so you have to create an IAM user and enable the IAM API Keys.

So let's dive into the world of creating and using the IAM API Keys.

How to create IAM API Keys?¶

The first step is to build an IAM user. You have already learned the creation of IAM User here.

IAM User creation is a five step process. We have to follow all the steps with a minor change in Step 1.

You can see below, we have to just select the option, Programmatic Access.

You can also enable this user to have an AWS Console access by enabling the option AWS Management console access.

You have to follow the remaining steps from the IAM user creation article.

In the step five, Success, you have to take some specific action.

You might see in step five, your screen will look little different than the last time you created the user. We have two new field

Access Key ID
Secret access key

You will get these two fields only when you enable the IAM user with programmatic access.

Please download the security credentials in CSV format, and keep it safe.

Previous time you may have seen this screen. In this the user did not have the programmatic access.

Now you have a user with IAM API keys enabled.

Properties of IAM API Keys¶

You should keep in mind few important points about IAM API Keys.

The above user creation step is the only time you will see both Access Key ID and Secret access key together.
Access Key ID is only visible in IAM User's security credentials.
It is advised to key the security credentials downloaded in CSV format.
Once lost, you cannot recreate the Secret access key corresponding to the Access Key ID.
You have to deactivate the old Access Key ID and create a new pair of Access Key ID & Secret access key.
You might have already figured this out, since the IAM API keys are tied to an IAM User, we cannot have it associated with IAM Roles.
The above combination should never be stored in an AWS EC2 instance, you should use IAM Roles for this.

How to use the IAM API Keys?¶

You can use these IAM API Keys in two major ways.

AWS SDKs
AWS CLIs

We will go through both these steps.

Using IAM API Keys with AWS CLI.¶

We can install the Python based AWS CLI with a simple command if you have pip package manager.

pip install awscli

To use the AWS CLI you need to configure the AWS environment by running this command.

aws configure

If you get four questions as an output than it proves that the AWS CLI is installed.

The above command will ask four questions which you can provide the details from the above Access Key ID & Secret access key you created for the new IAM user.

AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]: json

Let me try to explain you the details about these 4 options.

AWS Access Key ID [None] : fill the value you got for Access Key ID
AWS Secret Access Key [None]: fill the value you got for the Secret access key.
Default region name [None]: You should provide the region on which the AWS CLIs or SDKs should execute.
Default output format [None]: JSON is generally the preferred option.

The above execution creates 2 file in ~/.aws folder.

-rw------- 1 user staff 116B Dec 24 23:43 credentials
-rw------- 1 user staff 44B Dec 24 23:43 config

The credentials file contains this, you gave the Access Key ID & the Secret access key above.

[default]
aws_access_key_id = AKIAIOSFODNN7EXAMPLE
aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

The config file has this.

[default]
region = us-west-2
output = json

With this you can execute AWS CLI specific commands. Here is a reference of such commands.

Once you have the CLI installed and configured, using the SDK is much easier.

Using IAM API Keys with AWS SDKs.¶

AWS support SDKs in many languages.

C++
Go
Java
JavaScript
.NET
Node.js
PHP
Python
Ruby

You will use the Python SDK for this example. The Python3 AWS SDK is called Boto3

You should complete the AWS CLI installation before proceeding.

Boto3 installation | QuickStart¶

Python has a very good package manager called pip, and we can install boto3 with a simple command

pip install boto3

Post this you need to add a few details to some configuration files if you have still not installed the AWS CLI. Create the credentials and config file in the aws directory as shown above.

Here is a sample code, which just list all the buckets on AWS S3. Even if you do not understand AWS S3 no worries. The below code will not give any errors if you do not have any S3 buckets. No error in the below code signifies that the AWS SDKs and CLIs are working in conjunction with each other.

import boto3

s3 = boto3.resource("s3")
for bucket in s3.buckets.all():
    print(bucket.name)

Conclusion¶

AWS Console if a great way to use AWS, but it becomes difficult to use only AWS Console for all the tasks. There are times when you may need to do the job more than once. Manually performing these repeated tasks is very difficult and subject to errors.

AWS offers two additional access channels, AWS CLIs and SDK. These accesses require a user enabled with programmatic access. The same user can also have the console access with programmatic access.

After you have created the user, you should note that you receive an Access Key ID and a Secret Access Key. These pairs of key are only available during user creation. This is the reason we should download this information in the CSV format, as this information is required multiple time for access.

You now have all the raw materials to connect to AWS programmatically. The next step is to download the Python AWS CLI package and configure it. Once configured, you will see a folder in the home directory ~/.aws. This folder now has all the sufficient information to connect to AWS.

The CLI is a powerful tool to use, if you still want the SDK, you can download the BOTO3 Python SDKs and use the sample code to access all the bucket in the S3 if you have created. If the above code executes without any error you should assume that the SDKs is installed.

You can now use any of the methods to access AWS. Let me know your experience of accessing the AWS using CLIs or SDKS, was it easy or hard.

Info Graphics¶

Reference¶

Doing AWS STS the right way.

Animesh Bhadra 🎯 — Sat, 20 Mar 2021 10:50:58 +0000

Introduction¶

You have seen in the previous topic on IAM Roles, some users and resource can assume a role, moreover an IAM Roles are like a hat which anyone can wear and gets its power. One important part of this should bother you, how does AWS authenticate such users, if the user is a genuine or not.

AWS STS of Security token service plays an important part in enabling IAM Roles. When you are using a cross account resource or any federated users, you can also use AWS STS to provide temporary user credentials.

AWS STS though can be used to support mobile application using AWS resources, but it is advised to use AWS cognitio, which will be discussed in the future.

You will learn what is AWS STS, what are its benefits, when to use it. You will also learn to use a specific Action/API called assumerole to get access to an AWS resource for an AWS cross account.

AWS STS¶

AWS STS (Security token service) as the name suggest, provides a security token for accessing a AWS resources. You may think AWS STS as the provider of temporary access.

AWS STS has these specific properties when assigning temporary access

It can range from few minutes to a few hours.
Once the AWS STS provided temporary token expires, it cannot be reused at any point.
You can invoke AWS STS only through AWS SDKs or AWS CLIs.

Benefits of AWS STS¶

AWS STS solves a very specific problem for you, when you want someone to temporarily access your AWS resource without having concerns of revoking the permission.

AWS STS provides a way to

You should not embedded long term AWS security credentials into an application.
You should not create extra IAM identities, using IAM roles with AWS STS is enough to satisfy the temporary access requirement.
You do not have to worry about deactivating the AWS STS credentials, 36 hours is the maximum you can set the AWS STS expiry time depending on the API invoked.

When to use AWS STS¶

You have now understood what is AWS STS, also what are the benefits of AWS STS. You might also have guessed the use cases for using AWS STS. Here if a breakdown for this.

In Hybrid Cloud setup, where you have to give access to the non AWS account holder. These methods are generally used for giving access to 3rd party
- SAML 2.0 Identity federation.
- Web Identity Federation. (Facebook, Github, etc.)
Cross Account roles, when you have to give your developer account a temporary access to your production account.
IAM roles for AWS services.

AWS STS Actions¶

You should learn about these five common AWS STS Actions.

AssumeRole : This is used for getting cross account access.
AssumeRoleWithWebIdentity : This is using any 3rd party web IDP like Google or Facebook.
AssumeRoleWithSAML : This is for hybrid cloud, where you have an entity with SAML 2.0
GetFederationToken : This is used by the AWS root account or any IAM user.
GetSessionToken : This is used by the AWS root account or any IAM user.

Here is a comparison for you on the above APIs.

AWS STS | AssumeRole Action¶

You have the basic understanding of the different Action provided by AWS STS. Let's now try to use AssumeRole API to understand how this works.

Here is what you are going to try, or what we call a problem definition.

You will have a user with no permission on the AWS Account. Now create an IAM Role, with AmazonS3FullAccess permission. Once you have the Role, edit the trust relationship to give ARN of the user which does not have any permission. Now using AWS Boto3 SDK you will make the user connect to AWS.

A pictorial representation of the step to help you understand.

AWS STS | Create Role¶

You have to follow all the steps mentioned in the article, IAM Roles. Once change would be this time we should select the Another AWS account option. You may need to give the 12 digit account number.

The above steps are the same we will use for cross account access. The steps will not change.

AWS STS | Change the trust relationship¶

When you create the role like it is mentioned in the previous step, by default it will always point to the account root user, you have to change it to the ARN of the user you want to do a AssumeRole. Here is how you can do it.

Select the role.
Click on Trust Relationship options, and select the Edit trust relationship button.
This will open a JSON Editor and edit the JSON for this particular user shown below, esp. the Principal, AWS option.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:user/Test"
      },
      "Action": "sts:AssumeRole",
      "Condition": {}
    }
  ]
}

You might be thinking, if we have to edit this option every time we have to assign to a new user, then how is this scalable?

The answer is, most of the time we will add a particular group with the IAM role attached and the required user is added or removed from the group to control the access.

AWS STS | BOTO3 code to AssumeRole¶

Now you have to write using the BOTO3 SDK provide for Python, the sample code to AssumeRole. Here is the sample code.

import boto3
import pprint
from boto3.session import Session

# Below is the ARN of the role.
arn = "arn:aws:iam::123456789012:role/account-s3-full-access"
session_name = "example-role"
client = boto3.client("sts")
account_id = client.get_caller_identity()["Account"]
print(account_id)

# Assume role takes the roles ARN and a sample session name
response = client.assume_role(RoleArn=arn, RoleSessionName=session_name)

pprint.pprint(response)

# Create an S3 resource that can access the account with the temporary credentials.
temp_credentials = response["Credentials"]

# Access the S3 as a resource passing the temporary credentials received from STS.
s3_resource = boto3.resource(
    "s3",
    aws_access_key_id=temp_credentials["AccessKeyId"],
    aws_secret_access_key=temp_credentials["SecretAccessKey"],
    aws_session_token=temp_credentials["SessionToken"],
)
print(f"Listing buckets for the assumed role's account:")
for bucket in s3_resource.buckets.all():
    print(bucket.name)

Once you run the above code, you should be getting the list of S3 buckets in your account. Though the user did not have access initially.

AWS STS | AssumeRole | Return Parameters¶

You should be thinking what is returned by STS, here is the JSON response returned by calling assume_role BOTO3 API.

{
   "AssumedRoleUser":{
      "Arn":"arn:aws:sts::123456789012:assumed-role/acc-s3-full-access/example-role",
      "AssumedRoleId":"AROAUXRIFYXT7BG3ENQGE:example-role"
   },
   "Credentials":{
      "AccessKeyId":"ASIAUXRIFYXTUVPIQWFL",
      "Expiration":datetime.datetime(2021,3,20,9,24,27,"tzinfo=tzutc())",
      "SecretAccessKey":"BsXYZCGNuemA8wevm6CnYVfZtNgdGaoOCJ4VwXnf",
      "SessionToken":"FwoGZXIvYXdzEEoaDFAVerghnmasN971Z76yKwAfJgq3tccU72Gj6Xl28zJwJIUS/UEEMtwYmxUDsplTKg0if/keQ9z1BdoPFdLsmDtUiWDnfvIkICUbCeVk+DKI4c9LtdIAXmhpssg4IAMncYFsmh+ylOdbbcud134TOkDkCtuZMkfKuUbIMG3lTq10k93DsiUFAoH5pqyLAa9IyqHUbKUxwwde0UAcUU1lNFMO/sTZI8kAIQNM4cpGMxdyPsYZaX5M1IGWqr2gPNLqLtKLvi1oIGMi33r+lP9GWX5W+Ich1MHUAfUfhgqIjXHjpmDQY5S0e/WOTBwrPLoorgXQlHMak="
   },
   "ResponseMetadata":{
      "HTTPHeaders":{
         "content-length":"1057",
         "content-type":"text/xml",
         "date":"Sat, 20 Mar 2021 08:24:27 GMT",
         "x-amzn-requestid":"72f38158-1d90-4619-a431-5a2fcf460a31"
      },
      "HTTPStatusCode":200,
      "RequestId":"72f38158-1d90-4619-a431-5a2fcf460a31",
      "RetryAttempts":0
   }
}

You should really be concerned about the Credentials parameters which is returned. It basically provides 4 information.

AccessKeyId : The access key id, which is always required for programmatic access.
Expiration : Generally it is 15 min, but can vary depending on the type of API being called.
SecretAccessKey : The secret access key, which is also generated only once.
SessionToken : As the name suggests, a unique way of identifying the session.

Conclusion¶

You might have now understood that IAM Roles and AWS STS have a symbiotic relationship. AWS STS is required when you need to provide these range of access.

The cross account use case, ex. the developer account my need a temporary access to the production system.
The Hybrid cloud use case, ex. On premise user, authenticated using SAML may need access to AWS resources.
In Hybrid cloud use case and cross account use case, ex. Authenticating the user using the web identity providers.
Sometimes IAM Services may need permission to another service, ex EC2 wants to write to a S3 bucket.

The above access is provided to you using the AWS STS because.

AWS STS provides short term credentials, which lives from a few minutes to some hours.
We should not be bothered to revoke the access as you cannot reuse the expired access.
AWS STS can be provided using the AWS SDKs or CLIs.

The real benefits of AWS STS are,

No need to embed long term credentials to the application.
No need to create multiple identities for each access request.
No need to revoke the access, as it expires automatically.

The various actions provided by AWS STS are

AssumeRole
AssumeRoleWithWebIdentity
AssumeRoleWithSAML
GetFederationToken
GetSessionToken

and we discussed AssumeRole Action uses the cross account example, where you created an IAM role, edited the trust relationship and then the user assuming the role, by using the Boto3 SDKs. By doing this the IAM user from another account could access AWS resources for a short period of time.

The AssumeRole returns these parameters when a call is made it

AccessKeyId
Expiration
SecretAccessKey
SessionToken

You might recognize it returns all the parameters created for IAM user having programmatic access, i.e. AccessKeyId and SecretAccessKey. In addition, we get the SessionToken and an expiration time.

Please provide you feedback if you have any other use case for AWS STS.

Reference¶

A foolproof guide to AWS IAM Roles

Animesh Bhadra 🎯 — Thu, 18 Mar 2021 04:19:34 +0000

Introduction¶

The FaceLess Men in GoT (Games of Thrones), hope you remember this character, or Raven/Mystique in X-Men. They both share a common power, what would that be?

The power to change identity as they wish to accomplish the task at hand. This is what IAM Roles do in a very broad sense.

You can also think of IAM roles as the Invisibility Cloak in Harry Potter, who ever acquires it becomes invisible.

IAM Roles are just like a hat, which anyone within AWS can wear, and get the powers presented by the hat, and loses the powers as soon as the hat is removed. The hat does not discriminate between real users or hardware like EC2, anyone can wear this hat.

You will also see how IAM Roles can be used, how it is created and attached. You will also see how IAM user is a different concept than IAM Role.

IAM Roles¶

An IAM role is an identity that you can create having very specific permissions. The only difference being you are not associating this policy to any particular user or services at the time of creation. This IAM role can be assumed by anyone who needs it, be it a user or an AWS resource.

You might be thinking, which problem does IAM role solve which a normal IAM user or a new IAM policy cannot solve. So lets understand why we need the use of IAM Role.

IAM Roles for user¶

When you are creating a user in your own account, you cannot give them all the permission as explained in IAM Introduction, the principle of least privileges. You cannot have one or multiple users will super user permissions, you cannot have Batman in every city, only Gotham.

In addition, we can also have non AWS accounts, like cross account want to access your AWS services, or the federated user, who may also need access to the AWS services in case of Hybrid Cloud.

You have to consider many other use-case along with the above two, for understanding we cannot keep on creating IAM Users to address such diverse range of permission request.

IAM Roles for AWS Services¶

You might have already read about IAM Policy, AWS resources cannot have IAM policy attach directly. There is another good policy, AWS credentials should not be stored on EC2 instances.

If you consider the above two guidelines, it will not be possible for any AWS services to communicate with other AWS services / resources to complete the work.

IAM Roles to the rescue¶

When you consider both the limitation of IAM User and AWS services, you can think we need some type of magic hat, which gives the sufficient power to both User or Service to execute the task it is elevated for. This is where magic happens with IAM Roles.

You can use IAM Roles for:-

One AWS service have to access another AWS service.
- Example: Application running on EC2 may need to access S3.
When you have a Hybrid Cloud implementation
- User from On Premise may want to access AWS Cloud infrastructure.

EC2 can assume a role only at the time of creation, but once it has a role attached, you can modify it using APIs, CLIs or console at any time. You can attach only 1 role to EC2 at any given time.

Benefits of IAM Role¶

You may need people from Hybrid cloud implementation to access the AWS account, you may need IAM Roles to be attached once these users are authorized to use SAML or Active directory.
You may have production and development account in AWS, and your development team may be required to be elevated to access a production account to fix a bug. This can be done using IAM Roles.
You have an application running on an EC2 instance, It needs access to S3, this can be achieved using IAM Role.

How to create an AWS IAM Role¶

AWS IAM Role creation is a three step process.

Step 1 | Select the trusted entity¶

The first step guides you choose which services or identities can assume the role, your options are

AWS Services
- The EC2 instance, lambda etc which needs access to other services.
Another AWS account
- This is a cross account access, like a developer taking a production access etc.
Web Identity
- Using OIDC to validate user, like the mobile phone apps.
SAML 2.0 Federation
- Using office resources as authentication parameters, like using service Active Directory.

You can select AWS Services for this experimentation, You can attach this to an EC2 instance, so that it can access the S3.

Step 2 | Attach permission policy¶

You can now select policy required to be attached, this step is same as the explained in IAM Policy. You have basically chosen from one of these 2 type

Custom policy, created using the visual editor or JSON editor.
AWS managed Policy.

You can select the AmazonS3ReadOnlyAccess for attaching to EC2 instance as agreed in Step 1.

Step 3 | Review¶

You have to provide these details to complete the IAM Role Creation process.

Role Name
- An explicit name, which can be used to attached to the EC2 instance.
Role description
- A description about the role.
Trusted Entities
- This is the service selected from Step 1.
Policies
- The policies attached, in this case AmazonS3ReadOnlyAccess

How does the IAM roles looks after creation?¶

You can see above, this is how an IAM Role looks like after creation. The policy document looks exactly same as an IAM Policy. This you can also assume as a baton in a relay race, the service or the user can run with the permission as soon as it gets the policy in the form of an IAM Role.

IAM Roles Vs IAM Users¶

You should see this video tutorial on YouTube. This video explains much better, also the flow chart shown above should also clear your doubts

You can see the logic is elementary,

If you have a non living things, like EC2 etc., it gets an IAM Roles. No question asked.
If it is a living thing, like a person, we have to ask two questions.
- If the permission is temporary then he gets an IAM Roles, else he is an IAM User.

Hope this clears your mind about the difference between two confusing terms.

IAM Roles Vs IAM Policy¶

IAM Roles and IAM Policy both have a JSON document identifying the rule. The only difference is the mention of Trusted Entities.

Conclusion¶

You might now be relieved by understanding the concept of IAM roles, as discussed we use IAM Roles for mainly three purposes.

One AWS service using another service, like EC2 instance, wants to read from S3.
In the hybrid environment, a non AWS user might need access to AWS resources temporarily.
Cross account access, where a developer may need access to production account.

You should also think of an IAM Roles as a hat, which a person or services wears and it magically gets the permission and when it drops the hat, it comes back to original state.

You also should think about EC2 instance, it should not store the IAM User credentials, they should be using the IAM Roles instead. The IAM Roles can be attached to an EC2 instance, during creation, and can be changed afterwards, but not attached after creation. You can attach only one IAM role to an EC2 instance and not multiple.

You learned that IAM Role creation is a 3 step process.

Select the trusted entity
Attach permission policy
Review

The IAM Role has been just like IAM Policy document, a JSON document having statements.

You can be confused with IAM Role and IAM User, we can simplify it, saying, if it is a resource we use an IAM Roles. If a physical user needs temporary access, it uses IAM Roles, otherwise it is an IAM User.

You can comment and let me know if the IAM user is different than IAM Role.

Reference¶

Photo by Ales Nesetril on Unsplash
Photo by Laura Thonne on Unsplash
Photo by Ali Kokab on Unsplash
Photo by Almos Bechtold on Unsplash
AWS IAM Overview - It’s Surprisingly Simple - Users vs Roles