DEV Community: aziz.amghar

All you need to know about EC2 instance

aziz.amghar — Wed, 23 Jun 2021 12:50:27 +0000

EC2 instance is an amazon virtual machine, there are many EC2 instance types, find below the main ones:

- R: apps that need a lot of RAM -in memory caches
- C: apps that need good CPU – compute / databases
- M: apps that are balanced (think medium) – general / web app
- I: apps that need good local I/O (instance storage) – databases.
- G: apps that need a GPU – video rendering / machine learning.
- T2/T3: burstable instances (up to a capacity)
- T2/T3: unlimited burst

Pricing models:

On Demand: pay per hour or second with no commitment, it is low cost, flexible and used for short term, dev/testing and you have a predictible price.
Reserved: you get a significant discount (1y-3y), it is used for apps that have steady state and that require capacity:
- Convertible reserved instances: long workloads with flexible instances.
- Scheduled reserved instances, example: every Thu between 3 and 6 pm.
Spot: you bid whatever price you want for instance, it is used for apps that have flexible start & end times.
Dedicated instances: no other customers will share your hardware, billing is based on instance.
Dedicated hosts: you book an entire physical server, for regulatory (no multi-tenant support), great for licensing,

Important: for prod environnement
Termination protection is turned off by default, you must turn it on.

EBS: 5 different types of EBS storage:

General purpose SSD
Provisionned iops SSD
Throughput optimized HDD
Cold HDD (lowest )
EBS magnetic

AMI Types (EBS vs instance store):

For EBS volumes: the root device for an instance launched from the AMI is an EBS volume created from an EBS snapshot.
For Instance store volumes: the root device for instance launched from the AMI is an instance store volume created from a template stored in S3.
Instance store volume is an Ephemeral storage: if stopped, you will lose all your data.
Instance store is physically attached to the machine (EBS is a network drive).
Instance Store Pros:
- Better I/O perofrmance
- Good for buffer / cache / scratch data / temporary content.
Instance Store Cons:
- On stop or termination, the instance store is lost.
- You cant resize the instance store
- Backups must be operated by the user.
EBS RAID Options (do it on OS not AWS):
- RAID0: increase performance (lots of iops), but more risk (if 1 disk fail, data lost).
- RAID1: increase fault tolerance (mirroring).
- RAID5 (not recommanded for EBS)
- RAID6 (not recommanded for EBS)

Elastic Network Interface vs Enhanced Networking vs Elastic Fabric Adapter:

ENI: elastic network interface, virtual network card
- For basic networking, create a management network, use a network appliance in your VPC..
- Low cost
EN: Enhanced networking: uses single root I/O virtualization, there is 2 types:
- ENA elastic network adapter: 100Gbps
- VF virtual function: 10Gbps
Elastic Fabric Adapter: for machine learning or high performing computing (HPC) or OS bypass.

EBS encryption:

Data in transit between an instance and an encrypted volume is also encrypted
Encryption is supported on all Amazon EBS volume types

EFS: Elastic File System

It grows automatically, great for file server or sharing data between EC2 instances
You pay for what you use
EFS support NFSv4 protocol
You only pay for the storage you use
Can scale up to the petabytes
Can support 1000s of concurrent NFS connections
Data is stored across multiple AZs within a region
Read After write consistency

Amazon FSx for Windows and for Lustre:

FSx for windows: built on windows server, use SMB and supports AD users, DFS (Distributed File System).. . Centralized storage for windows, Sharepoint, IIS or other native app for microsoft:
- Can be accessed from your on-premise infrastructure
- Can be configured to be MultiAZ (HA)
- Data is backed up daily to S3
EFS: is linux only, use it if you need distribution..
FSx for Lustre: for linux, if you process large data sets, millions of IOPS, machine learning, High Performance Computing (HPC), video processing, electronic design automation, financial modeling.

EC2 Placement Groups:

Clustered placement group: within a single Availability Zone
- Pros: Low Network Latency / High network throughput
- Cons: if the rack fails, all instances fails at the same time.
- Use case: Big data job that needs to complete fast.
Spread placement group: individual instances placed in different hardware (rack..), for single instance
- Pros:
  - Can span across AZs
  - Reduce risk of simultaneous failure
  - EC2 instances are on different physical hardware
- Cons:
  - Limited to 7 instances per AZ per placement group
- Use case:
  - App that needs to maximize high availability.
  - Critical apps where each instance must be isolated from failure from each other
- Partitioned Placement Group: think multiple instances in partition group.
  - Up to 7 partitions per AZ
  - Up to 100s of EC2 instances
  - The instances in a partition do not share racks with the instances in the other partitions.
  - A partition failure can affect many EC2 but won’t affect other partitions.
  - EC2 instances get access to the partition: information as metadata
  - Use case: HDFS, HBase, Cassandra, Kafka

EC2 Hibernate.

The in-memory (RAM) state preserved.
The instance boot is much faster (os is not stopped/restarted)
Under the hood:the RAM state is written to a file in the root EBS volume
The root EBS volume must be encrypted
Supported instance families: C, M and R.
Instance RAM size must be less than 150GB
Available for OnDemand and Reserved Instances.
An instance cannot be hibernated more than 60days
Use cases:
- long running processing
- saving the RAM state
- services that take time to initialize.

EC2 Best practices
Security

Manage access to AWS resources using IAM roles.
Implement the least permissive rules for your security group (Firewall).
Patch, update and secure regularly the operating system and applications on your EC2 instance.

Storage

Use separate Amazon EBS volumes for the operating system and data.
Ensure that the data volume persists after instance termination.
Encrypt EBS volumes and snapshots.

Resource management

Use instance metadata and custom resource tags to track and identify your AWS resources.
View your current limits for Amazon EC2 and plan in advance the request of any limit increases.

Backup and recovery

Backup periodically your EBS volumes using snapshots.
Create an Amazon Machine Image (AMI) from your instance to save the configuration as a template if needed for future installation.
Deploy critical components of applications across multiple AZs.
Monitor and respond to events.
Test regularly the process of recovering your instances and Amazon EBS volumes if they fail.

15 Things that you must know about AWS S3 (Simple Storage Service)

aziz.amghar — Mon, 21 Jun 2021 21:28:32 +0000

1. S3 is a secure and scalable storage service

You can store securely your files (called objects) to S3, the object size can be up to 5 TB.

2. Objects Attributes:
S3 objects can have:

Key (name of the object)
Value (data)
Version ID.
Metadata (data about data you are storing)
Subresources: Access control list, torrents.

3. S3 Naming convention:
There are some rules that you must respect in order to name your S3 objects:

No uppercase nor underscore
3-63 characters long
Not an IP and it must start lowercase letter or number
S3 is a universal namespace, so it’s unique.

4. S3 has the following features:
Tiered storage available
Lifecycle management
Versionning
Encryption
MFA Delete (multi factor auth): can be only configured in CLI mode.
Secure data using ACL (Access Control List) and bucket policies.
Signed URLs: URLs that are valid only for a limited time (ex: premium video service for logged in users)

5. S3 storage classes:

S3 standard: 99.99% availability, 99.99999999% durability, it is the default storage class.
S3 IA (infrequently Accessed)
S3 one zone - IA
S3 Intelligent Tiering
S3 Glacier (for data archiving, 99.999999999% durability of archives )
S3 Glacier Deep Archive (retrieve data in 12hours)

S3 Pricing Tiers:
You pay per:

Storage
Requests and data retrieval
Data transfer

Most expensive: S3 standard, then followed by:

S3 IA
then S3 Intelligent Tiering
then S3 one zone IA
then S3 glacier
and finally S3 glacier deep archive.

6. S3 Encryption:

Two types of encryption:

Encryption in Transit: SSL/TLS
Encryption at Rest (server side), there are three types of server side encryption:
- S3 managed keys -SSE -S3,
- AWS Key Management Service,
- Server side encryption with customer provided keys SSE-C
Then there is client side encryption

8. S3 Security:

User based: IAM policies.
Resource based, that can be managed in three ways:
Bucket policies, used to:
- Grant public access to the bucket
- Force a bucket to be encrypted at upload
- Grant access to another account (Cross Account)
Object ACL,
Bucket ACL.

9. S3 CORS:

If you request data from another S3 bucket, you need to enable CORS.
Cross Origin Resource Sharing allows you to limit the number of websites that can request your files in S3, thus limit your costs.

10. Consistency Model

Read after write consistency for PUTS of new objects:
- As soon as an object is written, we can retrieve it, ex: PUT 200 -> GET 200)
- This is true, except if we did a GET before to see if the object existed (ex: GET 404 -> PUT 200 -> GET 404) – eventually consistent
Eventual Consistency for DELETES and PUTS of existing objects
- If we read an object after updating, we might get the older version (ex: PUT 200 -> PUT 200 -> GET 200 (might be older version))
- If we delete an object, we might still be able to retrieve it for a short time (ex: DELETE 200 -> GET 200)

11. S3 Access Logs:

For audit purpose
Any request made to S3, from any account, authorized or denied will be logged into another S3 bucket
That data ca be analyzed using data analysis tools like Athena.

12. S3 pre-signed URLs:

Can generate pre-signed URLs using SDK or CLI
For download (easy, can use the CLI)
For uploads (harder, must use the SDK)
Valid for a default of 3600s, can change timeout with –expires in {TIME_BY_SECONDS] argument
Users given a pre-signed URL inherit the permissions of the person who generated the URL for GET / PUT. Examples:
Allow only logged in users to download a premium video on your S3 bucket
Allow an ever changing list of users to download files by generating URLs dynamically
Allow temporarily a user to upload a file to precise location in our bucket

13. S3 Performance:

Baseline Performance:
- S3 scale automatically to high request rates, latency 100-200ms
- Your app ca achieve at least 3500 PUT/COPY/POST/DELETE and 5500 GET/HEAD requests per second per prefix in a bucket.
KMS Limitation:
- If you use SSE-KMS, you may be imapcted by the KMS limits
- When you upload, it call the GenerateDataKey KMS API
- When you download, it calls the Decrypt KMS API
- Count towards the KMS quota per second (5500, 10000, 3000 req/s based on region)
- You cant request a quota increase for KMS
Multi Part upload:
- Recommended for files > 100MB, must use for files > 5GB
- Can help parallelize uploads (divied in parts and speed up transfers)
S3 Transfer Acceleration (upload only)
- Increase transfer speed by transferring file to an AWS edge location which will forward the data to the S3 bucket in the target region using AWS backbone.
- Compatible with multipart upload
- Check this url for S3 Acceleration speed: https://s3-accelerate-speedtest.s3-accelerate.amazonaws.com/
S3 Byte range Fetches
- Parallelize GETs by requesting specific byte ranges
- Better resilience in case of failures
- Can be used to speed up downloads
- Can be used to retrieve only partial data (for example the head of a file)

14. Select & Glacier Select:

Retreive less data using SQL by performing server side filtering
Can filter by rows & columns (simple SQL statements, server side filtering)
Less network transfer, less CPU cost client side.

15. Object & Glacier Vault Lock:

Do you know any other functionnality of S3 that I didn't mention, please feel free to post it in the comment.