DEV Community: Lam Wing Chun

AWS Relational Database Service (RDS) Overview

Lam Wing Chun — Sun, 09 Jul 2023 12:01:49 +0000

Outline

Introduction & Deployment
RDS Connection and SQL Query
Backup and Restore
High Availablity & Disaster Recovery

Introduction & Deployment

Amazon Relational Database Service (Amazon RDS) is a managed database service While RDS can be deployed on EC2 or Outposts (the service that allow you to run AWS services on Premises Private Cloud).

When deploying RDS, we need to specify the EC2 instance specification such as vCPUs and RAM.

In addition, RDS supports different database engine types and verison, which includes:

MariaDB
Microsoft SQL Server
MySQL
Oracle
PostgreSQL

After creating the RDS, you can check the connection details such as username and password as well as the Endpoint and port

RDS Connection and SQL Query

We can connect RDS for PostgreSQL via pgAdmin, psql, or programming language.

Connecting RDS for PostgreSQL via Python
Create and List the table within the database

Remark:

when you connect the RDS via public internet, you need to allow public access in RDS and configure the security group for allowed port and source IP.

import psycopg2
engine = psycopg2.connect(
    database="postgres",
    user="postgres",
    password="HvNnP1UnYVtNZS15ofUf",
    host="database-1.cjylreq9cdvs.ap-east-1.rds.amazonaws.com",
    port='5432'
)
cursor = engine.cursor()
cursor.execute('''
CREATE TABLE IF NOT EXISTS Persons (
    PersonID int,
    LastName varchar(255),
    FirstName varchar(255),
    Address varchar(255),
    City varchar(255)
);
               ''')
cursor.execute('''
select relname from pg_class where relkind='r' and relname !~ '^(pg_|sql_)';
''')
print("The RDS postgres database contains the following table:")
print(cursor.fetchall())

Backup and Restore

By default, RDS is enabled with automated backups and backup retention period (e.g. 7 days). If you have some significant change on RDS Database, you can manually take the snapshot for restoration purpose.

When you select the snapshot, you can restore the database back to that specific time of status.

High Availablity & Disaster Recovery

You can create the read replicas in another availablity zone for better availablity while the read replicas only allow read database data. Once the primary RDS instance is down, you can promote the read replicas to be the primary RDS.

Option	Recovery time objective (RTO)	Recovery point objective (RPO)	Cost	Scope
Automated backups	Good	Better	Low	Single Region
Manual snapshots	Better	Good	Medium	Cross-Region
Read replicas	Best	Best	High	Cross-Region

Reference:
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html#Welcome.Concepts.on-prem
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ConnectToPostgreSQLInstance.html
https://aws.amazon.com/tw/rds/features/backup/
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithAutomatedBackups.html
https://aws.amazon.com/tw/blogs/database/implementing-a-disaster-recovery-strategy-with-amazon-rds/

AWS Storage Comparison: EBS vs S3 vs EFS vs FSx

Lam Wing Chun — Sun, 09 Jul 2023 03:25:49 +0000

Introduction

In this tutorial, we are going to understand the common types of storage options on AWS, which include Simple Storage Service (S3), Elastic Block Store (EBS), Elastic File System (EFS), and Amazon FSx.

Firstly, when we deploy EC2, we need a disk storage for storing the EC2 Operating system file and temp cache file to be able to start running the EC2 instance and we can call this type of disk as a OS Disk. Depending on user practice, we can expend the OS disk size or attach the additional disks for more disk storage.

While the file in the disk storage can be accessed by the attached EC2, it may be inconvenient for other EC2 to access. In this case, we may prefer to use S3 bucket or Elastic File System (EFS) for better file sharing purpose.

For the Elastic File System (EFS), it will support cross-VPC EC2 or single VPC EC2 file drive mounting as Network Attached Storage (NAS). For the cross-VPC mounting, it requires the VPC peering.

In some case, using S3 as file sharing can be less troublesome. However, EFS outpreform S3 in term of file access speed and user firendliness.

Since Elastic File System (EFS) only support Linux-based file system, if we want to mount the NAS to Windows-based EC2, we need to use Amazon FSx for Windows File Server.

Simple Storage Service (S3)

During the creation of S3 bucket, you are only required to select the region to deploy the bucket

When you upload the file to S3, you can determine which storage class the file belongs to.

Storage class	Designed for	Availability Zones	Min storage duration	Min billable object size	Monitoring and auto-tiering fees	Retrieval fees
Standard	Frequently accessed data (more than once a month) with milliseconds access	≥ 3	-	-	-	-
Intelligent-Tiering	Data with changing or unknown access patterns	≥ 3	-	-	Per-object fees apply for objects >= 128 KB	-
Standard-IA	Infrequently accessed data (once a month) with milliseconds access	≥ 3	30 days	128 KB	-	Per-GB fees apply
One Zone-IA	Recreatable, infrequently accessed data (once a month) stored in a single Availability Zone with milliseconds access	1	30 days	128 KB	-	Per-GB fees apply
Glacier Instant Retrieval	Long-lived archive data accessed once a quarter with instant retrieval in milliseconds	≥ 3	90 days	128 KB	-	Per-GB fees apply
Glacier Flexible Retrieval (formerly Glacier)	Long-lived archive data accessed once a year with retrieval of minutes to hours	≥ 3	90 days	-	-	Per-GB fees apply
Glacier Deep Archive	Long-lived archive data accessed less than once a year with retrieval of hours	≥ 3	180 days	-	-	Per-GB fees apply
Reduced redundancy	Noncritical, frequently accessed data with milliseconds access (not recommended as S3 Standard is more cost effective)	≥ 3	-	-	-	Per-GB fees apply

Elastic Block Store (EBS)

In EC2 console, you can create EBS Volume for attaching to EC2. When you create it, you can select the different Volume types and provisioned storage size

Or specify the EBS Volume when creating EC2 instance.

Solid state drive (SSD) volumes	Volume type	Durability	Use cases	Volume size	Max IOPS per volume (16 KiB I/O)	Max throughput per volume	Amazon EBS Multi-attach	Boot volume
General Purpose SSD volumes	gp3	99.8% - 99.9% durability (0.1% - 0.2% annual failure rate)	Transactional workloads Virtual desktops Medium-sized, single-instance databases Low-latency interactive applications Boot volumes Development and test environments	1 GiB - 16 TiB	16,000	1,000 MiB/s	Not supported	Supported
General Purpose SSD volumes	gp2	99.8% - 99.9% durability (0.1% - 0.2% annual failure rate)	Transactional workloads Virtual desktops Medium-sized, single-instance databases Low-latency interactive applications Boot volumes Development and test environments	2 GiB - 16 TiB	16,000	250 MiB/s *	Not supported	Supported
Provisioned IOPS SSD volumes	io2 Block Express ‡	99.999% durability (0.001% annual failure rate)	Workloads that require: Sub-millisecond latency Sustained IOPS performance More than 64,000 IOPS or 1,000 MiB/s of throughput	4 GiB - 64 TiB	256,000	4,000 MiB/s	Supported	Supported
Provisioned IOPS SSD volumes	io2	99.999% durability (0.001% annual failure rate)	Workloads that require sustained IOPS performance or more than 16,000 IOPS I/O-intensive database workloads	4 GiB - 16 TiB	64,000 †	1,000 MiB/s †	Supported	Supported
Provisioned IOPS SSD volumes	io1	99.8% - 99.9% durability (0.1% - 0.2% annual failure rate)	Workloads that require sustained IOPS performance or more than 16,001 IOPS I/O-intensive database workloads	5 GiB - 16 TiB	64,001 †	1,001 MiB/s †	Supported	Supported

Hard Disk Drive (HDD) volumes	Volume type	Durability	Use cases	Volume size	Max IOPS per volume (1 MiB I/O)	Max throughput per volume	Amazon EBS Multi-attach	Boot volume
Throughput Optimized HDD volumes	st1	99.8% - 99.9% durability (0.1% - 0.2% annual failure rate)	Big data Data warehouses Log processing	125 GiB - 16 TiB	500	500 MiB/s	Not supported	Not supported
Cold HDD volumes	sc1	99.8% - 99.9% durability (0.1% - 0.2% annual failure rate)	Throughput-oriented storage for data that is infrequently accessed Scenarios where the lowest storage cost is important	125 GiB - 16 TiB	250	250 MiB/s	Not supported	Not supported

Previous generation volumes	Volume type	Use cases	Volume size	Max IOPS per volume	Max throughput per volume	Boot volume
Magnetic	standard	Workloads where data is infrequently accessed	1 GiB-1 TiB	40–200	40–90 MiB/s	Supported

Elastic File System (EFS)

When creating EFS, you need to specify the VPC for deployment.

After the EFS deployment, you can change the performance setting to elastic or provisioned.

you can run the following commands provided by EFS on Linux EC2 to mount the Network Attached Storage (NAS). Unfortunately, Amazon EFS does not support mounting from Amazon EC2 Windows instances.

Using the EFS mount helper:

sudo mount -t efs -o tls fs-04de9a52fae17a3e2:/ efs

Using the NFS client:

sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-04de9a52fae17a3e2.efs.ap-east-1.amazonaws.com:/ efs

Storage class	Designed for	Durability (designed for)	Availability	Availability zones	Other considerations
EFS Standard	Frequently accessed data requiring the highest durability and availability.	99.999999999% (11 9's)	99.99%	>=3	None
EFS Standard–Infrequent Access (IA)	Long lived, infrequently accessed data requiring the highest durability and availability.	99.999999999% (11 9's)	99.99%	>=3	Per GB retrieval fees apply.
EFS One Zone	Frequently accessed data that doesn’t require highest levels of durability and availability.	99.999999999% (11 9's)*	99.90%	1	Not resilient to the loss of the Availability Zone.
EFS One Zone-IA	Long lived, infrequently accessed data that doesn’t require highest levels of durability and availability.	99.999999999% (11 9's)*	99.90%	1	Not resilient to the loss of the Availability Zone. Per GB retrieval fees apply.

Amazon FSx

you need to joins the Windows EC2 instance to your AWS Managed Microsoft AD directory

After joining EC2 to the domain/directory, you can run the following commands to mount the NAS to windows EC2

you can replace "Z:" with any other available drive letter

//using the default DNS name
net use Z: \\\share
//using DNS aliases
net use Z: \\alias\share

[Step-by-Step] How to import/export EC2 Image from/to S3 bucket

Lam Wing Chun — Fri, 07 Jul 2023 07:25:10 +0000

Introduction

While there are plenty of Virtual Machine Image available in EC2 and AWS Marketplace, sometime there is a need to set up the custom virtual machine image such as server migration and autoscaling template.

In such case, when the source images are from other platforms such as VMware vCenter, Hyper-V, and other cloud platforms, we need an approach to upload the virtual machine image. At the same time, we need to export the virtual machine image from EC2 and download it to our on-premises environment for backup and other reasons. In this tutorial, we are going to import/export EC2 Image from/to S3 bucket.

For Importing EC2 Image from S3 bucket, we first upload the source virtual machine image to S3 bucket. After that, we will use the EC2 Image Builder to create the custom virtual machine image and register it as AMI for EC2 to deploy.

For Exporting EC2 Image to S3 bucket, when we have made some configuration changes in EC2 and we want to export it, we will use the AWS CLI export-image command to export the image to S3 bucket and we can download it once it is ready.

Remark:

When downloading files from S3 bucket to on-premises, there will be the data transfer charge based on the amount of data. [First 100GB of data transfer out to the internet free each month] https://aws.amazon.com/s3/pricing/ https://aws.amazon.com/blogs/architecture/overview-of-data-transfer-costs-for-common-architectures/
AMI Operating Systems List for AWS EC2: https://aws.amazon.com/marketplace/b/c3bc6a75-0c3a-46ce-8fdd-498b6fd88577?ref_=hmpg_categories_c3bc6a75-0c3a-46ce-8fdd-498b6fd88577&category=c3bc6a75-0c3a-46ce-8fdd-498b6fd88577

Step Summary

Step 1: Create IAM Role for importing and Exporting EC2 Image
Step 2: Create S3 bucket for storing the EC2 Image File.

For Importing EC2 Image from S3 bucket:

Step 3: Uploading the Image to S3
Step 4: Create AMI Image via EC2 Image Builder

For Exporting EC2 Image to S3 bucket:

Step 5: Start the EC2 image exporting task via AWS CloudShell
Step 6: Downloading the Image from S3

Step Details

Step 1: Create IAM Role for importing and Exporting EC2 Image

Go to IAM

Go to Roles

Press Create Role

Select Custom trust policy

Copy the following trust policy to the policy editor and Press Next



{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Principal": { "Service": "vmie.amazonaws.com" },
         "Action": "sts:AssumeRole",
         "Condition": {
            "StringEquals":{
               "sts:Externalid": "vmimport"
            }
         }
      }
   ]
}

Press Create Policy

Copy the following IAM policy to the policy editor
Press Next



{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect": "Allow",
         "Action": [
            "s3:GetBucketLocation",
            "s3:GetObject",
            "s3:ListBucket",
            "s3:PutObject",
            "s3:GetBucketAcl"
         ],
         "Resource": "*"
      },
      {
         "Effect": "Allow",
         "Action": [
            "ec2:ModifySnapshotAttribute",
            "ec2:CopySnapshot",
            "ec2:RegisterImage",
            "ec2:Describe*"
         ],
         "Resource": "*"
      }
   ]
}

Name the IAM policy and Press Create Policy

Select the created IAM Policy and Press Next

Name the IAM Role and Press Create Role

Now, you have created the IAM Role for importing and Exporting EC2 Image

Step 2: Create S3 bucket for storing the EC2 Image File.

Go to S3

Press Create bucket

Name the S3 bucket and select the region

Disable Block all public access and acknowledge it

Press Create bucket

Now, you have created the S3 bucket for storing the EC2 Image File

For Importing EC2 Image from S3 bucket:

Step 3: Uploading the Image to S3

In this tutorial, Ubuntu 18.04 LTS (Bionic Beaver) VMware VMDK Image File is selected as the EC2 Image file

Donwload Link: https://cloud-images.ubuntu.com/bionic/current/
Go to the created S3 bucket

Press Upload

Press Add files

Select the downloaded VMDK file

Press Upload

Now, you have uploaded the image file to the S3 bucket

Step 4: Create AMI Image via EC2 Image Builder

Go to EC2 Image Builder

Go to Images

Press Import Image

Name the EC2 Image and Version

Select the nearest base image operating system

Browse S3 to select the uploaded Image file

Select the created IAM Role and Press Import Image

Once the import image task has finished, we can see the EC2 image in AMI.

Go to AMIs

In this page, you can see the created EC2 Image and you can deploy the EC2 Image by pressing Launch instance from AMI

For Exporting EC2 Image to S3 bucket:

Step 5: Start the EC2 image exporting task via AWS CloudShell

After changing or deploying some application and configuration in the EC2, we can export it to S3 to back up the current application and configuration status of the virtual machine image.

Select the EC2 Instance and Press Create Image

Name the AMI Image and Press Create Image

Now. you have created the new AMI Image from EC2

In order to export the AMI Image to S3, we are going to use the AWS Cloudshell
Go to AWS Cloudshell

Copy the IAM Role Name (role-name), the AMI id from the created AMI image (image-id) and the S3 Name (s3-export-location) as follows:



aws ec2 export-image \
--role-name EC2ImageImportExportRole \
--image-id ami-0dbeabfbb124c3ae5 \
--disk-image-format VMDK \
--s3-export-location S3Bucket=ec2imagebucketstorage123,\
S3Prefix=exports/

Paste it in AWS Cloudshell

While we can monitor the exporting task by using the following command (export-image-task-ids)
(export-image-task-ids) can be found in ExportImageTaskId Value in the above JSON response when starting the export task



aws ec2 describe-export-image-tasks \
--export-image-task-ids export-ami-0510f25de97082fe0

Step 6: Downloading the Image from S3

Once the EC2 Image has exported, you can find it in the S3 Bucket

Go to exports/

Select the exported EC2 Image and Press Download

Reference:
https://docs.aws.amazon.com/vm-import/latest/userguide/vmimport-image-import.html
https://docs.aws.amazon.com/vm-import/latest/userguide/vmexport_image.html
https://docs.aws.amazon.com/cloudshell/latest/userguide/supported-aws-regions.html
https://docs.aws.amazon.com/cli/latest/reference/ec2/export-image.html

[Step-by-Step] How to start & stop AWS EC2 automatically using Lambda & EventBridge

Lam Wing Chun — Mon, 03 Jul 2023 16:21:43 +0000

Introduction

In this tutorial, we are going to automate starting and stopping EC2 via Lambda & EventBridge at regular intervals (e.g.running EC2 from 8:00 am to 8:00 pm). While the EC2 is stopped, we can save the compute cost during this time.

In order to let Lambda be able to start and stop the EC2 instance, we first need to assign the required IAM permission for lambda in Step1.

Once it is done, we can create two lambda functions, which the first one is for starting the EC2 and the second one is for stopping the EC2. After that, we can create the EventBridge Schedule to trigger the lambda function at the desired times.

Step Summary:

Step 1: Create IAM role for Lambda
Step 2: Create 2 Lambda function for starting and stopping EC2
Step 3: Create 2 EventBridge Schedule to invoke 2 Lambda functions

Step Details:

Step 1: Create IAM role for Lambda:

Go to IAM

Go to Role

Press Create Role

Select AWS Service and Lambda and press Next

Press Create Policy

Press JSON

Copy the following IAM Policy and paste in JSON Editor

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "ec2:Start*",
        "ec2:Stop*"
      ],
      "Resource": "*"
    }, 
    {
      "Effect": "Allow",
      "Action": [
          "ses:SendEmail",
          "ses:SendRawEmail"
      ],
      "Resource": "*"
    }
  ]
}

Go Back to Visual and Press Next

Name the IAM Policy and Press Create Policy

Select the created IAM Policy and Press Next

Name the IAM Role and Press Create Role

Now, you have created the IAM Role for Lambda

Step 2: Create 2 Lambda function for starting and stopping EC2

Go to Lambda

Press Create Function

Select Author from scratch and Name the Lamdba function
Select Python 3.9 runtime and Select IAM Role for Lambda
Press Create function

Modify the following Python Code and paste in Code source Editor

AutoStartEC2
Replace 'ap-east-1' to the designated Region for your situation
Replace 'i-0f7f94a56f47e7221' to the EC2 ID
you can add more instances in this array if you want
E.g. ['i-0f7f94a56f47e7221','i-0uweq7w7d7293','i-0wwq78e78we781']

import boto3
/////////////////////////////////////////////////////
region = 'ap-east-1'
instances = ['i-0f7f94a56f47e7221']
/////////////////////////////////////////////////////
ec2 = boto3.client('ec2', region_name=region)
def lambda_handler(event, context):
    ec2.start_instances(InstanceIds=instances)
    print('started your instances: ' + str(instances))

Press Deploy

Now, you have created the lambda function for starting EC2

Repeat the above steps and refer to the following code to create the lambda function for stopping EC2

AutoStopEC2

import boto3
/////////////////////////////////////////////////////
region = 'ap-east-1'
instances = ['i-0f7f94a56f47e7221']
/////////////////////////////////////////////////////
ec2 = boto3.client('ec2', region_name=region)
def lambda_handler(event, context):
    ec2.stop_instances(InstanceIds=instances)
    print('stopped your instances: ' + str(instances))

Step 3: Create 2 EventBridge Schedule to invoke 2 Lambda functions

Go to EventBridge

Select EventBridge Schedule and Press Create schedule

Name the EventBridge Schedule

Select Recurring schedule and Cron-based schedule

Define the cron expression for the schedule
cron(minutes hours day-of-month month day-of-week year)

For our cron expression, it will run at 8:00 am from Monday to Firday every week every year

Select Off in Flexible time window

Select the Timezone and Press Next

Select Lambda

Select the Lambda Function and Press Next

Press Next

Press Create schedule

Now, you have created the EventBridge Schedule for triggering lambda to start EC2

Repeat the above steps and Change the cron expression from 8 to 20 and select the AutoStopEC2 lambda function to create EventBridge Schedule for triggering lambda to stop EC2

Reference:
https://docs.aws.amazon.com/scheduler/latest/UserGuide/schedule-types.html?icmpid=docs_console_unmapped#cron-based
https://aws.amazon.com/lambda/pricing/?nc1=h_ls
https://repost.aws/knowledge-center/start-stop-lambda-eventbridge
https://www.amazonaws.cn/en/eventbridge/pricing/#:~:text=With%20Amazon%20EventBridge%2C%20you%20pay%20only%20for%20events,price%20is%20%C2%A5%206.75%20per%201%20million%20events.

hi

Lam Wing Chun — Fri, 01 Jul 2022 13:39:11 +0000