DEV Community: Nurul Ramadhona

An Easy Way to Migrate Emails to the Amazon Workmail

Nurul Ramadhona — Mon, 27 Mar 2023 03:40:33 +0000

Have you decided to move your resources to AWS? Resources are not only about storage or anything to build your application, right? Something you use to communicate with your customers or business partners is also an important thing. Amazon Workmail is the answer for our email service.

I've created some posts in this series but those are for something built from scratch. Then, I thought what if we want to migrate our emails from the current hosting provider to AWS? So in this section, I'll show you the easy way that AWS already provided at no cost. Actually, there are two migration specialists we can use but here I choose one of them which doesn't require us to install anything. It's web-based so we can easily access it anywhere. We will use audriga.

Before we migrate, we need to do a few things "carefully" because the state of our domain or the email itself is in use. So, these are the steps I created to do before we migrate all emails.

Announce your migration schedule a few days before you do that to all email users and tell them the next steps they should take after the migration process has been done, such as changing the default password, etc.
Start to create the Amazon Workmail resources, consisting of an organization, some users along with the emails as the target and an administrator (I'll tell you what it's used for later).
I suggest you do the migration at a time when the users are not actively using their emails like at midnight for example as we usually do for maintenance.
Set the required DNS records provided by AWS except the MX record. But if there is no more email transaction at that time, you can go ahead with all records.
Start migrating the emails.

Since I don't have email hosted somewhere, I'll do two types of migration. First is manual migration from Outlook to Workmail. Then, the second one is batch migration between two Workmail organizations (here we will also see how to migrate either from or to Workmail).

Now we are ready to go!

Create The Amazon Workmail Resources

In this step, we just need to create an organization along with registering the external domain. Here I use the domain dhona.xyz.

$ aws workmail create-organization --alias dhona --domains DomainName=dhona.xyz --region us-east-1

Note*: it's not mandatory to use an external domain since AWS gives us a domain alias for each organization subdomain.awsapps.com.

Set The Required DNS Records

Please set the DNS records generated by Workmail and make sure all are verified.

$ aws workmail get-mail-domain --domain-name dhona.xyz --organization-id m-44968df215c443dea726cd731821614a --region us-east-1
DkimVerificationStatus: PENDING
IsDefault: false
IsTestDomain: false
OwnershipVerificationStatus: PENDING
Records:
(the record will be shown here)

Once we set the DNS properly, it should be successfully verified.

Start The Migration Process

1. Manual Migration

This method we can use to migrate a single email or only for a few users (small quantities). So in the beginning, we will create one email user as the target of Outlook's email.

$ aws workmail create-user --organization-id m-44968df215c443dea726cd731821614a --name dhonaxyz --display-name "Nurul Ramadhona" --password $password --region us-east-1
$ aws workmail register-to-work-mail --organization-id m-44968df215c443dea726cd731821614a --entity-id bdb219b2-c7ed-4c0e-8e04-293b5bd69127 --email dhonaxyz@dhona.xyz --region us-east-1
$ aws workmail describe-user --user-id bdb219b2-c7ed-4c0e-8e04-293b5bd69127 --organization-id m-44968df215c443dea726cd731821614a --region us-east-1
DisplayName: Nurul Ramadhona
Email: dhonaxyz@dhona.xyz
EnabledDate: '2023-03-26T12:52:35.822000+07:00'
Name: dhonaxyz
State: ENABLED
UserId: bdb219b2-c7ed-4c0e-8e04-293b5bd69127
UserRole: USER

As I mentioned above, we will use audriga. Here are the steps on how to use it:

Open the link => https://app.workmail.audriga.com/?client=workmail

Select the provider (source and target).

Enter the email account details (source and target).

Because we migrate in user mode, we should enter the email and password manually. Make sure both passed the validation checks.

Start the migration (we can leave the screen because we will get an email notification once the migration has been configured, started and completed).

Check if the emails exist on the target email (the source email currently has two emails, each one email on Inbox and Sent Items).

2. Batch migration

We have successfully migrated a single account. That's good, right? But what if we have a large number of users? Should we migrate all one by one?

Don't worry! We can use a template file (usually in .csv) for uploading the users' details. This is a common thing for managing email services.

Since we will do batch migration. Please create some target users along with the emails on the Workmail. Make sure all users are created and enabled. You can use the following Ansible playbook I created:

- name: workmail-users
  hosts: localhost
  connection: local
  gather_facts: no

  tasks:
    - name: create users
      command: aws workmail create-user --organization-id your-org-id --name "{{ item.username }}" --display-name "{{ item.fullname }}" --password "{{ item.pass }}" --region your-choosen-region
      loop: 
         - { username: "user1", pass: "passwordup2U!", fullname: "User 1"}
         - { username: "user2", pass: "passwordup2U!", fullname: "User 2"}
         - { username: "user3", pass: "passwordup2U!", fullname: "User 3"}
         - { username: "user4", pass: "passwordup2U!", fullname: "User 4"}
         - { username: "user5", pass: "passwordup2U!", fullname: "User 5"}
      tags: [create]

    - name: list users
      shell: "aws workmail list-users --organization-id your-org-id --region your-choosen-region --query 'Users[?Name==`{{ item.username }}`].Id' >> id-list.txt"
      loop: 
         - { username: "user1" }
         - { username: "user2" }
         - { username: "user3" }
         - { username: "user4" }
         - { username: "user5" }
      tags: [list]

    - name: list users id
      shell: 'cat id-list.txt'
      register: list_id
      tags: [list]

    - debug:
        var: list_id.stdout_lines
      tags: [list]

    - name: enable users
      command: aws workmail register-to-work-mail --organization-id your-org-id --entity-id "{{ item.userid }}" --email "{{ item.email }}" --region your-choosen-region
      loop: 
         - { userid: "user1id", email: "user1@your.domain" }
         - { userid: "user2id", email: "user2@your.domain" }
         - { userid: "user3id", email: "user3@your.domain" }
         - { userid: "user4id", email: "user4@your.domain" }
         - { userid: "user5id", email: "user5@your.domain" }
      tags: [enable]

Note*: Please enter the value with your own user's details as well as the number of users. Then, run the 'enable' tag separately as we need the entity-id values after the 'create' and 'list' tags.

Here I create 5 users as example:

We also need to enable migration permission and choose an administrator. By using administrator, we can migrate all emails without providing each user's password. Yes, we only use the credential of the administrator as it has access to all users.

Then, because I migrate between two Workmail organizations. I'll create one more organization as I do for the target organization. I'll migrate from nurul.awsapps.com to dhona.xyz which both are hosted on Workmail. But if you currently have emails hosted somewhere, you don't need to do this.

Now, we are ready to migrate the emails! The steps are similar to the manual migration above, so here I'll mention the difference between them:

Select the provider (source and target).

Because I'll do migration between two Workmail organizations, I choose the same source and target provider. In case the source (your current email service) is hosted somewhere, please choose to add the missing provider or server and enter the details needed.

Configure the account by choosing to add multiple accounts, then upload the .csv file. Here's the example:

Start the migration.

Check the email.

Here before I did migration, I sent a test email to all source emails. As we can see above there are 5 accounts that have been migrated and each contains 1 email. Then, I'll log in to one of those five users to see if the email exists.

That's it! It's very easy, right? AWS has provide us the easy way and the self-service as well, so we can do it independently anytime we need to migrate to Amazon Workmail.

Alright! Last but not least, don't forget to follow me for more content! Thank you!

Clean Up AWS Resources - Amazon Workmail

Nurul Ramadhona — Sat, 25 Mar 2023 14:23:00 +0000

So far, we have done many things to explore how Amazon Workmail works so we can host our email services there. In this section, I'll show you how to delete any resources we created within Amazon Workmail through AWS CLI. Please don't misunderstand! I'm not here to ask you to stop using AWS services but I'm here to show you how to delete it in case you may want to create a new resource or replace an existing one.

If you have followed me on the second post of this series, here's the cleanup process. It has its own flow to delete the resources, so please follow the following instructions without missing a single step.

Deregister User & Group From Workmail

It will disassociate a user/group, mark it as no longer used and change the state to disabled. Here we will create a simple bash script for the repetitive tasks.

Before that, let's gather all the information needed (ID of user/group to be deleted)!

$ aws workmail list-groups --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --region us-east-1 | grep Id
  Id: bcefb7d0-1f5a-45e4-8ef4-853a74823e86
$ aws workmail list-users --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --region us-east-1 | grep Id
  Id: 3815a14e-e0d1-4d31-b998-bb290589191c
  Id: 4b1d1dd0-4c9a-451a-83de-4145063999f0
  Id: 510f7b96-800d-47e2-a869-c3c47af4e9ea
  Id: a036c622-4d14-4075-b2a4-9a17975cbc83

Then, we will copy-paste the ID:

$ cat delete-emails.sh 
#! /bin/sh
aws workmail deregister-from-work-mail --entity-id bcefb7d0-1f5a-45e4-8ef4-853a74823e86 --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --region us-east-1 
aws workmail deregister-from-work-mail --entity-id 3815a14e-e0d1-4d31-b998-bb290589191c --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --region us-east-1
aws workmail deregister-from-work-mail --entity-id 4b1d1dd0-4c9a-451a-83de-4145063999f0 --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --region us-east-1
aws workmail deregister-from-work-mail --entity-id 510f7b96-800d-47e2-a869-c3c47af4e9ea --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --region us-east-1
aws workmail deregister-from-work-mail --entity-id a036c622-4d14-4075-b2a4-9a17975cbc83 --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --region us-east-1
$ chmod +x delete-emails.sh 
$ ./delete-emails.sh

Now all users and groups are disabled and ready to be deleted:

$ aws workmail list-groups --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --region us-east-1 | grep State
  State: DISABLED
$ aws workmail list-users --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --region us-east-1 | grep State
  State: DISABLED
  State: DISABLED
  State: DISABLED
  State: DISABLED

Delete User & Group

$ cat delete-usergroup.sh 
#! /bin/sh
aws workmail delete-group --group-id bcefb7d0-1f5a-45e4-8ef4-853a74823e86 --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --region us-east-1
aws workmail delete-user --user-id 3815a14e-e0d1-4d31-b998-bb290589191c --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --region us-east-1
aws workmail delete-user --user-id 4b1d1dd0-4c9a-451a-83de-4145063999f0 --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --region us-east-1
aws workmail delete-user --user-id 510f7b96-800d-47e2-a869-c3c47af4e9ea --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --region us-east-1
aws workmail delete-user --user-id a036c622-4d14-4075-b2a4-9a17975cbc83 --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --region us-east-1
$ chmod +x delete-usergroup.sh 
$ ./delete-usergroup.sh

Check the state one more time!

$ aws workmail list-groups --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --region us-east-1
Groups:
- DisabledDate: '2023-03-23T21:06:45.687000+07:00'
  EnabledDate: '2023-03-23T20:40:45.076000+07:00'
  Id: bcefb7d0-1f5a-45e4-8ef4-853a74823e86
  Name: developers
  State: DELETED
$ aws workmail list-users --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --region us-east-1 | grep State
  State: DELETED
  State: DELETED
  State: DELETED
  State: DELETED

Deregister External Domain

If we use an external domain as the default one, please change it first at least to the domain alias provided by Workmail.

$ aws workmail update-default-mail-domain --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --domain-name dhona.awsapps.com --region us-east-1
$ aws workmail deregister-mail-domain --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --domain-name dhona.xyz --region us-east-1

Delete The Organization

This is the last step and make sure you have deleted all things above. If you find an error, please repeat those tasks because you may skip some steps.

$ aws workmail delete-organization --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --delete-directory --region us-east-1
OrganizationId: m-fb75a642ab0f4745b33b54f729f6af01
State: Deleting

Alright! That's it for now! I think we can do more things with Amazon Workmail so I won't say that this is the last part of this series. Please look forward to it and don't forget to follow this blog! Thank you!

How to Build Reputation of Email Hosted on Amazon Workmail

Nurul Ramadhona — Sat, 25 Mar 2023 14:22:41 +0000

In the previous post, we already and successfully hosted our emails on Amazon Workmail. We also tested sending/receiving email within the same domain and it works well, no problem so far. Now, let's see how is it going outside! Because a company doesn't only communicate between departments (internal) but also with external like business partners for example. Then, because this is just a practice and I don't have a company to work with (other domains). So I'll show you how our email goes to famous email providers, I mean it's like commonly everyone has an account at least on one of those providers.

By the way, I'm sorry if you will see lots of images on this post since nothing to do with the configurations through AWS CLI :)

To check how it works, we will send emails to Google Mail, Yahoo and Microsoft from two different domains. One from Workmail alias dhona.awsapps.com and one external domain dhona.xyz.

1. GMAIL

First Email Results: Email from dhona.awsapps.com is delivered to Inbox and email from dhona.xyz is marked as SPAM.

Second Email Results: Both are delivered to Inbox.

2. YAHOO

First Email Results: Both are delivered to Inbox.

Second Email Results: Both are delivered to Inbox as the first email.

3. OUTLOOK

First Email Results: Both are delivered to Junk.

Second Email Results: Both are still delivered to Junk as the first email.

Explanation

As we can see above any emails are not delivered to Inbox on the first try, even the second one as well. Don't worry! It usually happens to a new email domain although all authentications (SPF, DKIM, DMARC) checks are passed. To prevent being domain with a "bad" reputation, we can do the following actions:

Ask the receiver to "Mark as not SPAM" for your email that went to JUNK.
Keep your privacy as secure as you can, it means not using a weak password, clicking an unknown link, and many more to prevent being used to send spam, phishing, or other kinds of issues.
If you are sure with all securities and since the emails passed the authentication checks. Ask your partners to whitelist your emails or domain or even the IP address.
Don't send bulk emails at one time!

After we do all the above, let's check our email one more time!

Third emails are all successfully delivered to Inbox.

Additional Thing

To check how our domain reputation is, we can use some online tools. I'll show you one of them which is MXToolbox. It helps us to know if our domain is listed somewhere.

There are some DNS Blocklists that can be a reason for your email to be marked as Spam or even failed to deliver based on our domain or even IP address reputation. The ones that have the big impacts are SORBS and Barracuda. So make sure your domain is not listed anywhere for smooth email transactions. If already listed, try to delist or request removal and start to build a good reputation!

By the way, feel free to comment if you have different perspectives or other things to do regarding this topic. I'll appreciate that, so we can learn together.

Host Your Own Business Email Using Amazon Workmail

Nurul Ramadhona — Sat, 25 Mar 2023 14:22:26 +0000

Continuing the previous post where I told you how important email is and why we choose Workmail as our email service. Then, here we will get started on how to set up email hosting using Amazon Workmail. I'll do all the steps using AWS CLI, so make sure you have installed it and set the credential.

Are you comfortable with the Console? Please go ahead with it, no pressure at all :)

When I created this series, Amazon Workmail was only available in 3 regions. In this case, I randomly choose N. Virginia (us-east-1) and you can choose which one you wanna use.

More about Amazon Workmail, click here!

So, what are we gonna do?

Create Organization
Create User
Register Domain
Update Default Domain
Create User Alias
Send & Receive Email
Update Mailbox Quota
Create Group
Associate User To The Group
Setup Email On Mobile Email Client App

I think those 10 steps are enough for us to get started with Amazon Workmail!

1. Create Organization

Here we will start using the free domain provided which is awsapps.com. So please decide on the alias first, then AWS will do the rest to set up your email hosting such as domain verification along with the webmail client link provided.

For example, I use dhona as an alias, so AWS will verify dhona.awsapps.com and use dhona.awsapps.com/mail to access the webmail client.

$ aws workmail create-organization --alias dhona --region us-east-1
OrganizationId: m-fb75a642ab0f4745b33b54f729f6af01
$ aws workmail describe-organization --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --region us-east-1
ARN: arn:aws:workmail:us-east-1:0123456789:organization/m-fb75a642ab0f4745b33b54f729f6af01
Alias: dhona
CompletedDate: '2023-03-23T18:20:03.872000+07:00'
DefaultMailDomain: dhona.awsapps.com
DirectoryId: d-9067aebc88
DirectoryType: IdentityPoolDirectory
OrganizationId: m-fb75a642ab0f4745b33b54f729f6af01
State: Active

Note*: please save the organization-id as we always use it for the next configurations.

2. Create Users

When we're creating a user, we don't need to choose the domain that will be used directly. So it will be just a user, not an email user but the state is still disabled.

$ aws workmail create-user --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --name dhona --display-name "Nurul Ramadhona" --password $password --region us-east-1
UserId: 510f7b96-800d-47e2-a869-c3c47af4e9ea
$ aws workmail describe-user --user-id 510f7b96-800d-47e2-a869-c3c47af4e9ea --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --region us-east-1
DisplayName: Nurul Ramadhona
Name: dhona
State: DISABLED
UserId: 510f7b96-800d-47e2-a869-c3c47af4e9ea
UserRole: USER

To enable the user, we will need to choose a "temporary" primary email address. Why is it temporary? Because we can change it anytime we want (I'll tell you more about it in the fifth step). By doing this, you will create an email address for that user.

$ aws workmail register-to-work-mail --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --entity-id 510f7b96-800d-47e2-a869-c3c47af4e9ea --email dhona@dhona.awsapps.com --region us-east-1
$ aws workmail describe-user --user-id 510f7b96-800d-47e2-a869-c3c47af4e9ea --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --region us-east-1
DisplayName: Nurul Ramadhona
Email: dhona@dhona.awsapps.com
EnabledDate: '2023-03-23T19:00:40.432000+07:00'
Name: dhona
State: ENABLED
UserId: 510f7b96-800d-47e2-a869-c3c47af4e9ea
UserRole: USER

Note*: entity-id is an ID of either a user or group. Please adjust it based on your condition, you're managing a user or group. Don't get confused!

Right after the user is enabled, we can log in to the webmail client and start sending/receiving emails.

3. Register Domain To The Organization

Within an organization, we can have more than one domain to be used. Since we got a free alias dhona.awsapps.com, here I'll add my external domain dhona.xyz (you can skip this step if you don't have one and jump to the sixth step). We can use the domain either from Amazon Route53 or external domain.

$ aws workmail register-mail-domain --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --domain-name dhona.xyz --region us-east-1

When we use a custom domain, we should add some required DNS records generated by Workmail. Each record has its own purpose.

$ aws workmail get-mail-domain --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --domain-name dhona.xyz --region us-east-1
DkimVerificationStatus: PENDING
IsDefault: false
IsTestDomain: false
OwnershipVerificationStatus: PENDING
Records:
(the required DNS records will be shown here)

Mail Exchange (MX): used to direct where the mail server of that domain is placed.

For example: Value: 10 inbound-smtp.us-east-1.amazonaws.com.

It shows the mail server address by using the domain along with the priority which is 10. The domain can consist of more than one IP address in case we have multiple servers. Anyway, if the domain is in use somewhere (you already hosted your email service using that domain), I suggest you wait till the migration process is done.

Sender Policy Framework (SPF): used to list all addresses that are allowed to send email using the domain.

For example: Value: v=spf1 include:amazonses.com ~all

It means all emails that don't come from amazonses.com should be marked as insecure or spam.

DomainKeys Identified Mail (DKIM): used by the receiver to verify emails using the key signed through cryptographic authentication. The hostname is marked with ._domainkey.

For example: Value: abcdefghijklmnopqrstuvwxyz.dkim.amazonses.com.

Domain-based Message Authentication, Reporting and Conformance (DMARC): used to decide the action if the authentication is failed. This is just an additional record after SPF and DKIM.

For example: Value: v=DMARC1;p=quarantine;pct=100;fo=1

It means the receiver should quarantine all emails that don't pass the authentication checks and generate a report to the sender when the emails fail to deliver.

Those 3 things are parts of email security. You can get to know more about them and you can also make them custom as you need. In this case, I'll follow what Workmail has generated for me. If we already set the DNS records properly, the domain will be successfully verified and ready to be used.

4. Update Default Domain

Since now we have two domains, we are free to choose which one will be used as the default domain.

$ aws workmail update-default-mail-domain --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --domain-name dhona.xyz --region us-east-1
$ aws workmail describe-organization --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --region us-east-1
ARN: arn:aws:workmail:us-east-1:0123456789:organization/m-fb75a642ab0f4745b33b54f729f6af01
Alias: dhona
CompletedDate: '2023-03-23T18:20:03.872000+07:00'
DefaultMailDomain: dhona.xyz
DirectoryId: d-9067aebc88
DirectoryType: IdentityPoolDirectory
OrganizationId: m-fb75a642ab0f4745b33b54f729f6af01
State: Active
$ aws workmail list-mail-domains --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --region us-east-1
MailDomains:
- DefaultDomain: false
  DomainName: dhona.awsapps.com
- DefaultDomain: true
  DomainName: dhona.xyz

5. Create User Alias

Alias in this section is different from alias of the organization. When we create a user, the user can have multiple email addresses using different domains registered on Workmail. One acts as the primary email address and the rest as aliases. It's a good choice if you have more than one domain but you want to use the same username. All emails sent to an alias will be directed to and received by the primary email address.

$ aws workmail create-alias --entity-id 510f7b96-800d-47e2-a869-c3c47af4e9ea --alias dhona@dhona.xyz --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --region us-east-1

As I said above, we can change our primary email address anytime we want. So I'll make dhona@dhona.xyz as primary email of the user named dhona.

$ aws workmail update-primary-email-address --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --entity-id 510f7b96-800d-47e2-a869-c3c47af4e9ea --email dhona@dhona.xyz --region us-east-1
$ aws workmail list-users --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --region us-east-1
Users:
- DisplayName: Nurul Ramadhona
  Email: dhona@dhona.xyz
  EnabledDate: '2023-03-23T19:00:40.432000+07:00'
  Id: 510f7b96-800d-47e2-a869-c3c47af4e9ea
  Name: dhona
  State: ENABLED
  UserRole: USER

6. Send & Receive Email

For testing the email, we will create one more user and we will see if it works along with SPF, DKIM and DMARC checks.

$ aws workmail create-user --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --name nurul --display-name "Nurul Ramadhona" --password $password --region us-east-1
UserId: 4b1d1dd0-4c9a-451a-83de-4145063999f0
$ aws workmail register-to-work-mail --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --entity-id 4b1d1dd0-4c9a-451a-83de-4145063999f0 --email nurul@dhona.xyz --region us-east-1
$ aws workmail describe-user --user-id 4b1d1dd0-4c9a-451a-83de-4145063999f0 --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --region us-east-1
DisplayName: Nurul Ramadhona
Email: nurul@dhona.xyz
EnabledDate: '2023-03-23T20:03:58.821000+07:00'
Name: nurul
State: ENABLED
UserId: 4b1d1dd0-4c9a-451a-83de-4145063999f0
UserRole: USER

Here is the result:

It's a test email within the same domain, passed all authentication.

7. Update Mailbox Quota

With Workmail, we are free to customize the quota of each mailbox. By default, each user gets 50GB and we are allowed to increase or decrease as we need.

$ aws workmail get-mailbox-details --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --user-id 4b1d1dd0-4c9a-451a-83de-4145063999f0 --region us-east-1
MailboxQuota: 51200
MailboxSize: 0.010987281799316406
$ aws workmail update-mailbox-quota --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --user-id 4b1d1dd0-4c9a-451a-83de-4145063999f0 --mailbox-quota 12800 --region us-east-1
$ aws workmail get-mailbox-details --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --user-id 4b1d1dd0-4c9a-451a-83de-4145063999f0 --region us-east-1
MailboxQuota: 12800
MailboxSize: 0.02144145965576172

8. Create Group

A group always be the best practice for managing users. Let's say we have a company that consists of many departments such as HR, Marketing, Developers, etc. It will be easier to spread information to all members of each department so no one will lose any updates.

$ aws workmail create-group --name developers --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --region us-east-1
GroupId: bcefb7d0-1f5a-45e4-8ef4-853a74823e86
$ aws workmail register-to-work-mail --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --entity-id bcefb7d0-1f5a-45e4-8ef4-853a74823e86 --email developers@dhona.xyz --region us-east-1
$ aws workmail list-groups --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --region us-east-1
Groups:
- Email: developers@dhona.xyz
  EnabledDate: '2023-03-23T20:40:45.076000+07:00'
  Id: bcefb7d0-1f5a-45e4-8ef4-853a74823e86
  Name: developers
  State: ENABLED

9. Associate User To The Group

Now, we will add some users to the Developers group and use developers@dhona.xyz as the email address (should not already used by other groups/users).

$ aws workmail associate-member-to-group --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --group-id bcefb7d0-1f5a-45e4-8ef4-853a74823e86 --member-id 3815a14e-e0d1-4d31-b998-bb290589191c --region us-east-1
$ aws workmail associate-member-to-group --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --group-id bcefb7d0-1f5a-45e4-8ef4-853a74823e86 --member-id 510f7b96-800d-47e2-a869-c3c47af4e9ea --region us-east-1
$ aws workmail list-group-members --organization-id m-fb75a642ab0f4745b33b54f729f6af01 --group-id bcefb7d0-1f5a-45e4-8ef4-853a74823e86 --region us-east-1
Members:
- EnabledDate: '2023-03-23T18:37:51.155000+07:00'
  Id: 3815a14e-e0d1-4d31-b998-bb290589191c
  Name: admin
  State: ENABLED
  Type: USER
- EnabledDate: '2023-03-23T19:00:40.432000+07:00'
  Id: 510f7b96-800d-47e2-a869-c3c47af4e9ea
  Name: dhona
  State: ENABLED
  Type: USER

I'll log in to one of the members.

10. Setup Email On Mobile Email Client App

The webmail client provided by Workmail is not the only email client we can use. As with other email services, we can set up our email on mobile too by using the Microsoft Exchange option. Here is how it goes:

Access to the email also brings other features such as Calendar into our mobile app and we can use it as well as the email itself. Here's an example of how I create a reminder on the Calendar from the webmail client and mobile. Both will be synchronized automatically.

Create Reminder from Webmail Client

Create Reminder from Mobile

What's next?

We reach at the end of this post but it's not the end of this series. So please keep this state because we will use it again in the next post.

An Intro to Email Hosting World with Amazon Workmail

Nurul Ramadhona — Sat, 25 Mar 2023 08:45:06 +0000

Hi everyone! Today I wanted to create a series, my third series to be exact. In this case, I wanted to talk about email. Yes, email is one of the most important things for us. I think nowadays everyone has at least one email since anything is starting to go digital. Each has various purposes, such as:

Personal: banking, digital IDs, etc.
Business: e-commerce, customer support, communication with partners, etc.
Many more

For personal purposes, you can create free email from some popular email providers. Why? Because you may not need a very big size of mailbox for some things like creating an account, verifying something, saving banking transactions or online shopping history. I think there are many email providers that give us pretty enough quota along with any other features such as calendar, notes, etc.

BUT

When it comes to business, email is a very important thing and it's like being a part of a company's identity. You know, I've faced a lot of customer email issues and here I will share one of them which is a big deal and tell you in short.

Domain Spoofing

Yes, a client almost lost their money because of the "attacker" using almost similar domain. For example, if the official domain is aws.com then the "attacker" is using avvs.com. So, it's very important to have a high level of accuracy. Other people may know what's our plan or our other business without realizing it. Then, they'll easily attack you since they already know what you have done, are doing, and will do. So please be careful, especially for social engineering! This is just one of other many email issues.

It's an unusual thing for a business or government company to use free email. I mean when we're looking for any information about a company, we will think about their official website and contact (email) in addition to phone number. So they should have their own official domain as a trusted identity.

Now let's talk about email services! When I think about AWS, Amazon SES came in my mind. Yes, I used to think that it was the only one :) but I wasn't completely wrong, right? As we all know, SES is a Simple Email Service. When we search AWS email services, SES is on top for that keyword. But for daily transactions like sending and receiving emails, I prefer to use Amazon Workmail. I also don't see that using EC2 as a mail server is the best practice, considering that the associated ports are not allowed to be opened as well.

Amazon SES vs Workmail

I won't compare them to get the best service but I'm looking for the most suitable service depending on our needs. SES is best for sending emails to those who subscribe to your apps like transactional or marketing emails, but it doesn't include POP or IMAP and I think that's enough for the reason why we don't use SES for email hosting. Since we are able to do that with Workmail, it's suitable for our case.

Why Do We Choose Workmail and What Are The Benefits?

Since I'm new to this service, I'll just mention some points:

Only $4 for a 50GB mailbox per month.
Get a free domain alias subdomain.awsapps.com (in case you haven't bought domain) but if you have, you can use your own domain and all is under your control.
Webmail client is provided along with a calendar and contact as additional features.
Integrated with Microsoft Exchange so you can set up email clients on your mobile and of course along with the calendar.
For many more, click here!

Alright! I think it's enough for an introduction. In the next posts of this series, we will do more things with Amazon Workmail. I'll try my best to make it easy to follow so you can run your own email hosting for your business on AWS. Let's get started!

Setting Up Amazon EC2 for Windows Server with Ansible

Nurul Ramadhona — Wed, 15 Feb 2023 11:18:27 +0000

Hi everyone! I come back and this is my first post in 2023. I hope you are well, healthy, and still excited to keep learning.

So, this post was created because someone asked me about his user_data script for the Windows instance that didn't work. He's following my blog post here and there you can see my conversations with him as well. I think the case is an interesting and rare topic to be discussed, then here it is!

To be honest, I've never launched a Windows EC2 instance as long as I learned AWS but at this time I do because I have to reproduce someone's case as I mentioned above. The reason actually doesn't sound cool :) but I hope it will be useful for anyone else who is in the same condition.

Goal: Remote directly once the instance is running.

Here are some steps we will do:

Create Key Pair
Create Security Group
Create Instance
Retrieve Password
Remote

As usual, if you are familiar with this blog. Before we do ansible tasks, you have to prepare some prerequisites:

AWS CLI and set at least one credential;
Ansible;
Ansible collection for AWS by running ansible-galaxy collection install collection_name. There are 2 collections you can use, amazon.aws and community.aws.

Ready? Let's get started!

1. Create Key Pair

I usually import a key pair for Linux instance but at this time I do something different. This is also the only requirement so we can retrieve the password for the Windows instance. Please note, that we only can use rsa as a key type for Windows instances.

    - name: create rsa key pair
      amazon.aws.ec2_key:
        name: Administrator
        key_type: rsa
      register: key

Then, let's save the file! We never know that we may need it again for future tasks. We can use it for any other Windows instances.

    - name: download private key
      copy: content="{{ key.key.private_key }}" dest="administrator.pem" mode=0600

2. Create Security Group

We will create a custom SG that allows RDP port which is 3389.

    - name: create security group
      amazon.aws.ec2_group:
        name: rdp
        description: allow remote windows
        vpc_id: vpc-xxxx
        region: ap-southeast-3
        rules:
          - proto: tcp
            ports: 3389
            cidr_ip: 0.0.0.0/0
      register: sgroup

3. Create Instance

Here we will directly create one instance using amazon.aws.ec2_instance module. Please check this blog post below for more.

    - name: create instance
      amazon.aws.ec2_instance:
        name: windows
        vpc_subnet_id: subnet-xxxx
        image_id: ami-019fd4e0ba82e7e28
        instance_type: t3.micro
        key_name: "{{ key.key.name }}"
        security_group: "{{ sgroup.group_id }}"
        state: present
        volumes:
          - device_name: /dev/sda1
            ebs:
              volume_size: 30
              volume_type: gp2
              delete_on_termination: true
        user_data: "{{ lookup('file','script_file_name') }}"
        wait: true
      register: instance

Note: user_data is optional. In case you wanna use it with PowerShell. Don't forget to put your script between powershell tag. It should seem like below:

<powershell>
$put_your_script_here
</powershell>

4. Retrieve Password

Here we ask default password for the default user of the Windows server which is the administrator.

    - name: get the Administrator password
      community.aws.ec2_win_password:
        instance_id: "{{ instance.instances[0].instance_id }}"
        region: ap-southeast-3
        key_file: administrator.pem
        wait: true
      register: password

    - name: show password
      debug:
        msg: "{{ password.win_password }}"

Let's run the playbook!

Here we run all tasks at once (in one YAML file) and refer the resources to each other as they are newly created (marked with register). All can be customized based on your need, either using variables, tags, or anything. For existing resources, you can directly define each resource's name as the value.

$ ansible-playbook -i host.yml windows.yml

PLAY [windows] *****************************************************************

TASK [create rsa key pair] *****************************************************
changed: [127.0.0.1]

TASK [download private key] ****************************************************
changed: [127.0.0.1]

TASK [create security group] ***************************************************
changed: [127.0.0.1]

TASK [create instance] *********************************************************
changed: [127.0.0.1]

TASK [get the Administrator password] ******************************************
ok: [127.0.0.1]

TASK [show password] ***********************************************************
ok: [127.0.0.1] => {
    "msg": "baHNB1OIx8BL8;15$&*76xp.BNrp63DF"
}

PLAY RECAP *********************************************************************
127.0.0.1                  : ok=6    changed=4    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

5. Remote

Once all is ready, we can remote it using the RDP client. If you are a Linux user, you can use an RDP client for Linux such as Remmina.

That's all for now. Let me know if you have any questions or even corrections. Last but not least, don't forget to follow this blog! Thank you!

References:

More Automation for Your AWS Resources, More Coffee Time for You!

Nurul Ramadhona — Sun, 30 Oct 2022 05:34:08 +0000

Hi everyone! It's been a while since my latest post (2 months ago) but today I have an interesting topic to discuss. Actually, I already talked about this topic at the AWS UG Indonesia event but the session was in Indonesian (here's the video) so I thought about turning it into a blog post in English.

Alright! You may see that mostly I talked about Ansible for AWS on this blog, but at this time I'll talk on the other side about where we can combine them. They can actually be "friends" and shouldn't always be comparisons. These works can help you to do more automation for your AWS resources which means that you will have more time to enjoy your coffee :) I mean everybody loves it, right?

Prerequisites:

AWS CLI and set at least one credential;
Ansible;
Ansible collection for AWS by running ansible-galaxy collection install amazon.aws and ansible-galaxy collection install community.aws.

1. Dynamic Inventory For EC2 Instances

Yes, we usually manage our server inventory manually which we input the IP addresses or hostnames manually like below:

ec2:
  hosts:
    108.136.226.235:
    108.136.226.232:
    108.136.226.238:

But Ansible has a plugin that enables us to manage our EC2 instances and meets our needs. It's very easy to do, simple and you will get what you want in a short time.

To use this plugin, below are the steps:

Create a new YAML file and input the code below:

plugin: aws_ec2
regions:
- ap-southeast-3
filters:
availability-zone: ap-southeast-3b

It will enable us to get the EC2 details and create a group for our instances based on the filter we use. For example above, I need to manage EC2 instances in the Jakarta region and I added a filter AZ because I just need to manage the instances in the AZ B. Any filter we can use is just the same as we use the EC2 filter by AWS CLI (if we can use the filter using AWS CLI EC2, then we can use it too on this plugin).

After creating the file, run the command:

$ ansible-inventory -i aws_ec2.yml --list

Here's the sample output:

"aws_ec2": {
  "hosts": [
    "ec2-108-136-56-235.ap-southeast-3.compute.amazonaws.com"
  ]
}

If we just wanna see the grouping, run the command:

$ ansible-inventory -i aws_ec2.yml --graph

Here's the sample output:

@all
  |--@aws_ec2:
  |  |--ec2-108-136-56-235.ap-southeast-3.compute.amazonaws.com
  |--@ungrouped:

In case we wanna run some command or playbook to our instances for example to install an application on them, we can mention the inventory just like we use Ansible as usual. Below is a sample by running ansible ad-hoc and we will see the output when it's done.

$ ansible -i aws_ec2.yml -u ubuntu -m shell -a "sudo apt install apache2 -y"

Now we all see, how easy is for us to manage our EC2 instances. More about this plugin, you can find it here.

2. Running AWS CloudFormation using Ansible

In this concept, we automate the AWS IaC tool. As we all know, CloudFormation gives us an easy way for infrastructure provisioning on AWS into a file, just by pushing a template. That sounds great, right? But do you know that Ansible has a module for that?

Yup! Ansible has and we can get some benefits by combining them. As I mentioned before, they shouldn't always be comparisons but it will be better to run them together.

What can we do with them?

Create multiple stacks at once

In case we wanna create some stacks with the same resources along with their own values, we can do that with Ansible. We don't need to copy-paste the same template just for the same resources. All we need to do is just turn it into a variable. Let's see how it works!

Here's the sample task that we can use. The key is the template_body argument which allows us to use the jinja2 template. In this case, I wanna create a bucket in two stacks and each stack has its own stack name and the bucket name as well. So in the stack1, I wanna create a bucket named nuruls3 and for the stack2, the bucket name is dhonas3.

    - name: create cloudformation stacks
      amazon.aws.cloudformation:
        stack_name: "{{ item.name }}"
        state: present
        template_body: "{{ lookup('template', 'cf-template.yml.j2') }}"
      loop:
        - { name: stack1, bucket: nuruls3 }
        - { name: stack2, bucket: dhonas3 }

What does the template look like? It's just as simple as this. We turn the bucket name which the data type is string into a variable.

Resources:
  S3Bucket:
    Type: 'AWS::S3::Bucket'
    Properties:
      BucketName: "{{ item.bucket }}"

So when we wanna define different values for different stacks using one (same) template, we just need to change the variables under the loop action.

Here's the output:

TASK [create a cloudformation stack] *******************************************
changed: [localhost] => (item={'name': 'stack1', 'bucket': 'nuruls3'})
changed: [localhost] => (item={'name': 'stack2', 'bucket': 'dhonas3'})

and the result:

$ aws s3 ls
2022-10-30 11:08:19 dhonas3
2022-10-30 11:07:51 nuruls3

Note: We can also update and delete as well as the creation. More about this module, click here!

Create a stack that contains resources in different accounts (in case you have more than one).

This is a sample from the Ansible documentation:

- name: Create a stack set with instances in two accounts
  community.aws.cloudformation_stack_set:
    name: my-stack
    description: Test stack in two accounts
    state: present
    template_url: https://s3.amazonaws.com/my-bucket/cloudformation.template
    accounts: [1234567890, 2345678901]
    regions:
    - us-east-1

On this step, I'm not able to reproduce it because I just have one account and I think I don't need to create a new one just to do this :) it's enough for me to have one because it's just for personal use but in case you need to use it because you have more than one account, please let me give you some tips. For me, instead of using template_url which you have to upload on the S3 bucket, you can use template_body. Why? Because you will be able to do something more flexible. For example, you can use the jinja2 template where you can freely customize the parameters.

3. Running Ansible from AWS Systems Manager

SSM is a great thing to use to manage our EC2 instances and the good thing is we can run the Ansible playbook from SSM.

All we need to do is just to attach IAM role on the EC2 instances you want to manage that allows them to communicate with SSM or we can say to make the SSM-Agent installed on the EC2 instances work. Yup! For the latest AMI, SSM-Agent has already been installed by default so we don't need to install it again (check the full information here).

One more thing we need to do is install Ansible on our EC2 instances because it will be running on your instances which acts as an Ansible host, that's why it should be installed. (This action can be done by SSM as well as by SSM RunCommand but in this case we use Shell).

Here we will use RunCommand with the document RunAnsiblePlaybook from Systems Manager and AFAIK it's free to do RunCommand so it can be one of your alternatives to run Ansible instead of using your localhost.

Here's a sample of the playbook. You can directly write it into the column provided or use the URL where you store your playbook.

In case you wanna run it using AWS CLI, below is the sample:

$ aws ssm send-command --document-name "AWS-RunAnsiblePlaybook" --document-version "1" --targets '[{"Key":"InstanceIds","Values":[your-instances-ids]}]' --parameters '{"playbook":["hosts: localhost\ntasks:\n - name: install web server\n apt: name=apache2"],"playbookurl":[""],"extravars":["SSM=True"],"check":["False"],"timeoutSeconds":["3600"]}' --timeout-seconds 600 --max-concurrency "50" --max-errors "0" --region ap-southeast-3

Alright! Those are three cool things we can do with Ansible to automate our AWS resources. I hope they can help you extend your time enjoying the coffee :) instead of spending more time working for your system (just let them work for you).

That's it for now! Thank you for coming and I'm looking forward to your feedback. Follow me to get notified when my new post is published!

Trying New RDS Feature: Connectivity with EC2 in 1-Click?

Nurul Ramadhona — Thu, 25 Aug 2022 03:33:00 +0000

Have you noticed that RDS has a new feature? Yup! 3 days ago AWS announced that now RDS supports setting up connectivity between RDS and EC2 only in 1 click. We don't need to prepare VPC, subnets, security groups, etc. Wow! That sounds great, right? But some "T&Cs" are applied to make this feature work properly. How does it work?

In case you want more information about new features, you can visit this link.

By the way, I did all of these following practices via Console because I still can't find the CLI option for this feature maybe because it just launched (CMIIW and tell me in the comment if you know any information about this). I'm sorry if you will see a bunch of images below :)

Key

The minimum network prefix (CIDR) per subnet is /24 to create the RDS DB instance in the same VPC as the EC2 instance.

For example in this case I'll be creating RDS in the Jakarta region (ap-southeast-3) which has 3 AZs.

Based on my experience in "trying" this feature 2 days ago, I found some stuff you may need to know. Here we will play some scenarios, so we can be more understanding about this feature.

1. Using Custom VPC With CIDR /22

Here I use /22 because it's enough to create a subnet with /24 for each AZ (total 3). /22 can be divided into 4 pieces of /24.

Below is the diagram when I only created 1 public subnet in 1 AZ for EC2 instance before creating RDS and 2 more AZs don't have subnet yet. From this condition, 3 idle /24 is available and enough to be used by RDS to create one subnet with /24 in each AZ.

Note: In this case, I used the Free Tier template which Multi-AZ is disabled because it's just for personal testing purposes. That's why the automated backup is created in the same AZ and a stand-by instance is not created. You can use Dev/Test or Production to make those options enabled.

What did RDS create?

1 subnet in each AZ (total 3 subnets for 3 AZs)

2 security groups (allow 3306 from EC2 to RDS) and both are linked to each other

1 RDS instance and its automated backup successfully created

Testing

2. Using Default VPC

In this case, the connectivity should be successfully established because it has a CIDR that big enough to create a new subnet /24 in each AZ for RDS which is /16.

3. Using Custom VPC With CIDR Smaller Than /22

The result is a failure. Why? Because /23 only can be divided into 2 /24. See the following diagram!

In this case, the RDS DB instance is still created but it's using default VPC which is in a different subnet from EC2.

2 new security groups are also created by RDS for the new VPC but only the security group for EC2 is properly attached because RDS is using default subnet group of the default VPC which is a specific rule that is not configured to allow incoming traffic from EC2 port 3306 to RDS.

Q: Then, how to make it work?

A: We can't do that because we can't change VPC of both RDS and EC2

but

If you followed my scenario number 2, it created an EC2 instance and RDS in the same VPC which is the default VPC and it already has 2 security groups created by RDS. So in this case, all we can do is:

"Move" EC2 instance to the default VPC where RDS is placed. You can create a new AMI from the existing EC2 instance or create a new instance. At this time, I'll use the EC2 instance that was created in scenario number 2 which is placed in default VPC and the security group created by RDS is already attached to the EC2 instance.
Change the current RDS DB subnet group to other subnet groups in the same VPC. In this case, it's the default VPC that was created in scenario number 2.

Alright, we are done with all the scenarios!

Clean Up

Scenario 1 (Custom VPC)

Delete the RDS DB instance
Delete the RDS DB subnet group
Terminate the EC2 instance
Delete the VPC

Scenario 2 (Default VPC)

Delete the RDS DB instance
Delete the RDS DB subnet group
Terminate the EC2 instance
Delete security groups created by RDS (please delete rules that associated with each other before deleting them)
Delete the route table for subnets created by RDS
Delete the subnets created by RDS

Scenario 3 (same as Scenario 1)

That's it for now! Thank you for coming and I'm looking forward to your feedback. Follow me to get notified when my new post is published!

AWS Auto Scaling Features to Optimize Cost and Ensure Availability

Nurul Ramadhona — Thu, 18 Aug 2022 07:24:00 +0000

Cloud Computing may be a solution for your on-premises problem but not everyone knows the cloud, right? Let's say you find a solution for your application that can be done using the cloud but a solution to "persuade" your boss may seem a bit harder :)

If you agree with me, please comment "Yes" so I can hear your voice :) Just kidding!

Are you the one who is confused with so many questions?

How about the cost? (we can't deny that it always be number one)
What are the benefits?
Many more ...

The following solutions can be your "weapon" when it comes to the cloud, but how?

Always Available

One of the biggest benefits of using the cloud is the provider is always able to meet our needs. Let's say you need a VM right now! You can directly access your AWS console and create an EC2 instance. It will be available and running as soon as possible after you create it. One problem is solved.

Then, what if we need more features? Let's say you want to use LightSail to run your WordPress instantly but it seems like it's not available in your region (for example Jakarta)! You can choose the nearest region where the service is already activated (for example Singapore). The second problem is solved and you can access your WordPress dashboard now.

Or maybe you want to have a backup for your application when there is a problem in one area? You can activate multi-AZ or region, third problem is solved.

But what if our product/application becomes viral and accessed from various countries? You can use CloudFront as CDN.

That's how cloud availability solves our problem "in general", I can't say it's the best AWS solution since AWS has so many services. So when we need the resources as soon as possible, we don't need to buy and look for the best brand for our devices.

Ease of Scaling

Let's say we're newbies in the cloud and we're just moving from on-premises culture. So EC2 will bring you easily to get to know about cloud environment. We all know we can create as many EC2 instances as we need (20 per region but can be increased by request) but one thing that really important and we can't forget about is cost. It can be so high when you don't know how to optimize based on your need (especially when you use on-demand instances).

Alright! There are so many ways we can scale our applications and optimize the cost, but here I will take two services that can be a basis for how we do it with AWS.

In this post, I'll create AWS Auto Scaling Group and Application Load Balancer using CloudFormation. All operations will be done through CLI, so make sure you have installed AWS CLI and set up the credentials. Note: You can do this through the web console if you want.

Network details (you can change based on your need):

VPC CIDR: 172.16.0.0/16
Subnet 1: 172.16.1.0/28
Subnet 2: 172.16.15.0/28 
Subnet 3: 172.16.30.0/28

(Sorry for a little typo, subnet 2 should be 172.16.1.16/28 and subnet 3 should be 172.16.1.32/28 for /28 usage but it's okay because the above networks still work)

Here's the whole content of the CloudFormation template named cfn-asg.yaml! I'll tell you the main points later.

AWSTemplateFormatVersion: 2010-09-09

Parameters:
  AmazonLinux2LatestAmiId:
    Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
    Default: /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2
  EnvironmentName:
    Type: String
    Default: DhonaASGTemplate
  SpotInstanceType1:
    Type: String
    Default: t3.large
  SpotInstanceType2:
    Type: String
    Default: t3.medium
  SpotInstanceType3:
    Type: String
    Default: t3.small
  SpotInstanceType4:
    Type: String
    Default: t3.micro
  SpotInstanceType5:
    Type: String
    Default: t3.nano

Resources:
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 172.16.0.0/16
      EnableDnsHostnames: true
      Tags:
        - Key: Name
          Value:
            Fn::Sub: ${EnvironmentName}
  InternetGateway:
    Type: AWS::EC2::InternetGateway
    Properties:
      Tags:
        - Key: Name
          Value:
            Fn::Sub: ${EnvironmentName}
  InternetGatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      InternetGatewayId:
        Ref: InternetGateway
      VpcId:
        Ref: VPC
  PublicRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId:
        Ref: VPC
      Tags:
        - Key: Name
          Value:
            Fn::Sub: ${EnvironmentName} Public Routes
  DefaultPublicRoute:
    Type: AWS::EC2::Route
    DependsOn: InternetGatewayAttachment
    Properties:
      RouteTableId:
        Ref: PublicRouteTable
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId:
        Ref: InternetGateway
  PublicSubnet1RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId:
        Ref: PublicRouteTable
      SubnetId:
        Ref: PublicSubnet1
  PublicSubnet1:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId:
        Ref: VPC
      AvailabilityZone: ap-southeast-3a
      CidrBlock: 172.16.1.0/28
      MapPublicIpOnLaunch: true
      Tags:
        - Key: Name
          Value:
            Fn::Sub: ${EnvironmentName} Public Subnet (AZ1)
  PublicSubnet2RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId:
        Ref: PublicRouteTable
      SubnetId:
        Ref: PublicSubnet2
  PublicSubnet2:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId:
        Ref: VPC
      AvailabilityZone: ap-southeast-3b
      CidrBlock: 172.16.15.0/28
      MapPublicIpOnLaunch: true
      Tags:
        - Key: Name
          Value:
            Fn::Sub: ${EnvironmentName} Public Subnet (AZ2)
  PublicSubnet3RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId:
        Ref: PublicRouteTable
      SubnetId:
        Ref: PublicSubnet3
  PublicSubnet3:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId:
        Ref: VPC
      AvailabilityZone: ap-southeast-3c
      CidrBlock: 172.16.30.0/28
      MapPublicIpOnLaunch: true
      Tags:
        - Key: Name
          Value:
            Fn::Sub: ${EnvironmentName} Public Subnet (AZ3)

  DhonaWebAppAutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    DependsOn: InternetGatewayAttachment
    Properties:
      CapacityRebalance: true
      HealthCheckType: ELB
      HealthCheckGracePeriod: 300
      MinSize: "3"
      MaxSize: "5"
      DesiredCapacity: "3"
      VPCZoneIdentifier:
        - Ref: PublicSubnet1
        - Ref: PublicSubnet2
        - Ref: PublicSubnet3
      MixedInstancesPolicy:
        InstancesDistribution:
          OnDemandAllocationStrategy: prioritized
          OnDemandPercentageAboveBaseCapacity: 30
          SpotAllocationStrategy: capacity-optimized
          SpotMaxPrice: 0.003
        LaunchTemplate:
          LaunchTemplateSpecification:
            LaunchTemplateId:
              Ref: DhonaWebAppLaunchTemplate
            Version:
              Fn::GetAtt:
                - DhonaWebAppLaunchTemplate
                - LatestVersionNumber
          Overrides:
            - InstanceType:
                Ref: SpotInstanceType1
            - InstanceType:
                Ref: SpotInstanceType2
            - InstanceType:
                Ref: SpotInstanceType3
            - InstanceType:
                Ref: SpotInstanceType4
            - InstanceType:
                Ref: SpotInstanceType5
      TargetGroupARNs:
        - Ref: DhonaWebAppTargetGroup
      Tags:
        - Key: Name
          Value:
            Fn::Sub: ${EnvironmentName}
          PropagateAtLaunch: true
  DhonaWebAppLaunchTemplate:
    Type: AWS::EC2::LaunchTemplate
    DependsOn: InternetGatewayAttachment
    Properties:
      LaunchTemplateData:
        ImageId:
          Ref: AmazonLinux2LatestAmiId
        SecurityGroupIds:
          - Ref: DhonaWebAppEC2SecurityGroup
        UserData:
          Fn::Base64: >
            #!/bin/bash

            yum -y install httpd

            echo "hello world! 

            My instance-id is $(curl -s http://169.254.169.254/latest/meta-data/instance-id)

            My instance type is $(curl -s http://169.254.169.254/latest/meta-data/instance-type)

            I'm on Availability Zone $(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone)" > /var/www/html/index.html

            service httpd start
  DhonaWebAppELBSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Security group for ELB
      SecurityGroupIngress:
        - CidrIp: 0.0.0.0/0
          IpProtocol: tcp
          FromPort: 80
          ToPort: 80
      VpcId:
        Ref: VPC
  DhonaWebAppEC2SecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Security group for EC2 ASG
      SecurityGroupIngress:
        - SourceSecurityGroupId:
            Ref: DhonaWebAppELBSecurityGroup
          IpProtocol: tcp
          FromPort: 80
          ToPort: 80
      VpcId:
        Ref: VPC
  DhonaWebAppLoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    DependsOn: InternetGatewayAttachment
    Properties:
      SecurityGroups:
        - Ref: DhonaWebAppELBSecurityGroup
      Subnets:
        - Ref: PublicSubnet1
        - Ref: PublicSubnet2
        - Ref: PublicSubnet3
  DhonaWebAppTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      Port: "80"
      Protocol: HTTP
      VpcId:
        Ref: VPC
  ALBListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      LoadBalancerArn:
        Ref: DhonaWebAppLoadBalancer
      Port: 80
      Protocol: HTTP
      DefaultActions:
        - Type: forward
          TargetGroupArn:
            Ref: DhonaWebAppTargetGroup

Outputs:
  URL:
    Value: !Join
      - ''
      - - 'http://'
        - !GetAtt
          - DhonaWebAppLoadBalancer
          - DNSName

Let's create the CloudFormation stack!

$ aws cloudformation create-stack --stack-name cfn-asg --template-body file://cfn-asg.yaml
StackId: arn:aws:cloudformation:ap-southeast-3:0123456789:stack/cfn-asg/438781a0-1e7e-11ed-b1ba-06ff26b14cdc
$ aws cloudformation describe-stacks --stack-name cfn-asg | grep "OutputValue\|StackStatus"
    OutputValue: http://cfn-a-Dhona-1NM3JA9AKJ0BE-1240836435.ap-southeast-3.elb.amazonaws.com
  StackStatus: CREATE_COMPLETE

Main Points

1. Use Mixed Instances (On-Demand + Spot) To Optimize Cost

      MixedInstancesPolicy:
        InstancesDistribution:
          OnDemandAllocationStrategy: prioritized
          OnDemandPercentageAboveBaseCapacity: 30
          SpotAllocationStrategy: capacity-optimized
          SpotMaxPrice: 0.003

If your workload can be interrupted (the process can be repeated when it fails), you can use spot instances. Why? Because you will get a big discount instead of using On-Demand. Here's the comparison:

As you can see, it's almost 70% compared to the On-Demand instance but one thing you should know is it can be interrupted when your bid price is lower than the actual price. That's why you should know which workloads are suitable for this instance type.

Then here I use 5 instance types based on priority:

          Overrides:
            - InstanceType:
                Ref: SpotInstanceType1 (t3.large)
            - InstanceType:
                Ref: SpotInstanceType2 (t3.medium)
            - InstanceType:
                Ref: SpotInstanceType3 (t3.small)
            - InstanceType:
                Ref: SpotInstanceType4 (t3.micro)
            - InstanceType:
                Ref: SpotInstanceType5 (t3.nano)

Pricing:

From the details above, I choose to use:

On-demand instance type based on top priority (t3.large) with percentage 30 which means 30% on-demand and 70% for spot instances. So if I use desired and minimum capacity 3, 1 on-demand instance and 2 spot instances should be running.
Spot instances with a maximum price is 0.003 and based on all instance types I used, t3.nano meets the criteria.

$ aws ec2 describe-instances --filters Name=instance-state-name,Values=running --query 'Reservations[].Instances[].{ID:InstanceId}'
- ID: i-02323c2a2be9759f2
- ID: i-004b63ceec33e2bb5
- ID: i-0b11803cfd9e623bb

From the settings, the first creation of this architecture is going to be:

2. Activate Capacity Rebalance To Ensure Availability

It helps you to create a new spot instance based on the EC2 instance rebalance recommendation two minutes before your current spot instance is interrupted.

      CapacityRebalance: true

For more information, click here!

3. Use Launch Template Instead of Launch Configuration

For more information, click here.

Alright! Those are all the main points. Let's play some scenarios so we can understand them better!

1. Change Maximum Spot Price (Increase)

On the first change, we will increase the maximum spot price and terminate one spot instance to see which instance type will be chosen.

Note: You can make a change to the same template directly. Here I create a new template to show you the difference between them.

$ diff cfn-asg.yaml cfn-asg-2.yaml 
145c145
<           SpotMaxPrice: 0.003
---
>           SpotMaxPrice: 0.01

Update the stack:

$ aws cloudformation update-stack --stack-name cfn-asg --template-body file://cfn-asg-2.yaml
StackId: arn:aws:cloudformation:ap-southeast-3:0123456789:stack/cfn-asg/438781a0-1e7e-11ed-b1ba-06ff26b14cdc
$ aws cloudformation describe-stacks --stack-name cfn-asg | grep "OutputValue\|StackStatus"
    OutputValue: http://cfn-a-Dhona-1NM3JA9AKJ0BE-1240836435.ap-southeast-3.elb.amazonaws.com
  StackStatus: UPDATE_COMPLETE

Terminate one spot instance:

$ aws ec2 describe-spot-instance-requests --query "SpotInstanceRequests[*].{ID:InstanceId}"
- ID: i-0b11803cfd9e623bb
- ID: i-02323c2a2be9759f2
$ aws ec2 terminate-instances --instance-ids i-02323c2a2be9759f2
TerminatingInstances:
- CurrentState:
    Code: 32
    Name: shutting-down
  InstanceId: i-02323c2a2be9759f2
  PreviousState:
    Code: 16
    Name: running

Here's the output when it's considered as UNHEALTHY and replaced with the new instance:

- AutoScalingGroupName: cfn-asg-DhonaWebAppAutoScalingGroup-NQBH0P5D9DNF
  AvailabilityZone: ap-southeast-3a
  HealthStatus: HEALTHY
  InstanceId: i-00c3754aed2929ad9
  InstanceType: t3.small
  LaunchTemplate:
    LaunchTemplateId: lt-0ab4b34b94589f554
    LaunchTemplateName: DhonaWebAppLaunchTemplate_6Nl7i61kH5qJ
    Version: '1'
  LifecycleState: InService
  ProtectedFromScaleIn: false
- AutoScalingGroupName: cfn-asg-DhonaWebAppAutoScalingGroup-NQBH0P5D9DNF
  AvailabilityZone: ap-southeast-3a
  HealthStatus: UNHEALTHY
  InstanceId: i-02323c2a2be9759f2
  InstanceType: t3.nano
  LaunchTemplate:
    LaunchTemplateId: lt-0ab4b34b94589f554
    LaunchTemplateName: DhonaWebAppLaunchTemplate_6Nl7i61kH5qJ
    Version: '1'
  LifecycleState: Terminating
  ProtectedFromScaleIn: false

The instance type is t3.small based on the best match with the maximum spot price from the priority.

But what if we terminate the on-demand instance? Will the instance type be changed? The answer is No because the maximum spot price doesn't affect the on-demand setting.

$ aws ec2 terminate-instances --instance-ids i-004b63ceec33e2bb5
TerminatingInstances:
- CurrentState:
    Code: 32
    Name: shutting-down
  InstanceId: i-004b63ceec33e2bb5
  PreviousState:
    Code: 16
    Name: running

2. Change The On-Demand Allocation Strategy

By changing it to the lowest-price, it will ignore the priority and choose the lowest price of all available instance types. In this case, it's t3.nano.

$ diff cfn-asg-2.yaml cfn-asg-3.yaml
142c142
<           OnDemandAllocationStrategy: prioritized
---
>           OnDemandAllocationStrategy: lowest-price

Update the stack:

$ aws cloudformation update-stack --stack-name cfn-asg --template-body file://cfn-asg-3.yaml
StackId: arn:aws:cloudformation:ap-southeast-3:0123456789:stack/cfn-asg/438781a0-1e7e-11ed-b1ba-06ff26b14cdc

Terminate the on-demand instance:

$ aws ec2 terminate-instances --instance-ids i-018469de2746c9432
TerminatingInstances:
- CurrentState:
    Code: 32
    Name: shutting-down
  InstanceId: i-018469de2746c9432
  PreviousState:
    Code: 16
    Name: running

As we expected, the new on-demand instance type is t3.nano.

- AutoScalingGroupName: cfn-asg-DhonaWebAppAutoScalingGroup-NQBH0P5D9DNF
  AvailabilityZone: ap-southeast-3b
  HealthStatus: HEALTHY
  InstanceId: i-0f3ab2d7fa671a597
  InstanceType: t3.nano
  LaunchTemplate:
    LaunchTemplateId: lt-0ab4b34b94589f554
    LaunchTemplateName: DhonaWebAppLaunchTemplate_6Nl7i61kH5qJ
    Version: '1'
  LifecycleState: InService
  ProtectedFromScaleIn: false

3. Make a Change To Launch Template (Create Version)

We have set to use the latest version of the launch template when there is a change, right? So now we will try to prove it.

        LaunchTemplate:
          LaunchTemplateSpecification:
            LaunchTemplateId:
              Ref: DhonaWebAppLaunchTemplate
            Version:
              Fn::GetAtt:
                - DhonaWebAppLaunchTemplate
                - LatestVersionNumber

$ diff cfn-asg-3.yaml cfn-asg-4.yaml
193c193,195
<             I'm on Availability Zone $(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone)" > /var/www/html/index.html
---
>             I'm on Availability Zone $(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone)
> 
>             This is additional line for updating launch template" > /var/www/html/index.html

Update the stack:

$ aws cloudformation update-stack --stack-name cfn-asg --template-body file://cfn-asg-4.yaml
StackId: arn:aws:cloudformation:ap-southeast-3:0123456789:stack/cfn-asg/438781a0-1e7e-11ed-b1ba-06ff26b14cdc

Then, let's terminate one spot instance again!

$ aws ec2 describe-spot-instance-requests --filters Name=state,Values=active --query "SpotInstanceRequests[*].{ID:InstanceId}"
- ID: i-00c3754aed2929ad9
- ID: i-0b11803cfd9e623bb
$ aws ec2 terminate-instances --instance-ids i-0b11803cfd9e623bb
TerminatingInstances:
- CurrentState:
    Code: 32
    Name: shutting-down
  InstanceId: i-0b11803cfd9e623bb
  PreviousState:
    Code: 16
    Name: running

We're still using the same Auto Scaling Group with the right capacity but in a different version of the launch template. In this case, it's the latest. So that's why we are suggested to use a launch template instead of a launch configuration. By using a launch configuration, we have to create a new one when we need to make a change.

Now, the latest architecture is:

Here's the output!

Alright! Those are all the changes we have made to the architecture. Please note that all the changes will be applied to the new instances, not to the current running instances. So for example when one instance is interrupted, the changes will be applied to the new instance created.

Cleanup

If you already followed all the instructions above, you can remove all of them only by deleting the stack. That's why we use CloudFormation, just to make it simple :)

$ aws cloudformation delete-stack --stack-name cfn-asg
$ aws autoscaling describe-auto-scaling-groups
AutoScalingGroups: []
$ aws ec2 describe-spot-instance-requests --filters Name=state,Values=active
SpotInstanceRequests: []
$ aws ec2 describe-instances --filters Name=instance-state-name,Values=running
Reservations: []
$ aws cloudformation describe-stacks --stack-name cfn-asg

An error occurred (ValidationError) when calling the DescribeStacks operation: Stack with id cfn-asg does not exist
$

In this post, I just show you some features that you can use with Auto Scaling. This is not the all features Auto Scaling has. You can mix it with Reserved Instance, use dynamic scaling, and many more. Here I'm not doing it all because this is just a "personal" demo, I don't need too big resources for very high workloads or long-term subscriptions.

That's it for now! Thank you for coming and I'm looking forward to your feedback. Follow me to get notified when my new post is published!

2 Easy Steps to Host WordPress Like a Magic Using Amazon Lightsail

Nurul Ramadhona — Wed, 20 Jul 2022 05:04:55 +0000

Amazon Lightsail is the easiest way to help us get started with AWS very quickly for many purposes such as building websites and many more. There are a lot of services that can be launched with Lightsail very quickly with low and predictable monthly prices. It can be an instance (VPS), storage, container, managed database, and so on. For more information about Amazon Lightsail, click here!

Alright, here we will only do 2 things:

1. Import Key Pair

Please note that the key pair defined in Lightsail is different or not associated with other AWS services like EC2 or IAM User (for CodeCommit).

$ aws lightsail import-key-pair --key-pair-name bitnami --public-key-base64 "ssh-rsa pubkey-goes-here" --region ap-southeast-1
{
    "operation": {
        "id": "4568c408-0349-4788-a922-e9fa6c3591a7",
        "resourceName": "bitnami",
        "resourceType": "KeyPair",
        "createdAt": 1653553305.861,
        "location": {
            "availabilityZone": "all",
            "regionName": "ap-southeast-1"
        },
        "isTerminal": true,
        "operationType": "ImportKeyPair",
        "status": "Succeeded",
        "statusChangedAt": 1653553306.012
    }
}

2. Create Instance

Before we create an instance, there are some parameters that we need to define. Those are BluePrint ID and Bundle ID. If you still don't know, you can get the list with the following commands:

BluePrint ID details for WordPress:

$ aws lightsail get-blueprints --region ap-southeast-1 --query 'blueprints[?contains(blueprintId, `wordpress`) == `true`]'
[
    {
        "blueprintId": "wordpress",
        "name": "WordPress",
        "group": "wordpress",
        "type": "app",
        "description": "Bitnami, the leaders in application packaging, and Automattic, the experts behind WordPress, have teamed up to offer this official WordPress image. This image is a pre-configured, ready-to-run image for running WordPress on Amazon Lightsail. WordPress is the world's most popular content management platform. Whether it's for an enterprise or small business website, or a personal or corporate blog, content authors can easily create content using its new Gutenberg editor, and developers can extend the base platform with additional features. Popular plugins like Jetpack, Akismet, All in One SEO Pack, WP Mail, Google Analytics for WordPress, and Amazon Polly are all pre-installed in this image. Let's Encrypt SSL certificates are supported through an auto-configuration script.",
        "isActive": true,
        "minPower": 0,
        "version": "5.9.3-8",
        "versionCode": "1",
        "productUrl": "https://aws.amazon.com/marketplace/pp/B00NN8Y43U",
        "licenseUrl": "https://d7umqicpi7263.cloudfront.net/eula/product/7d426cb7-9522-4dd7-a56b-55dd8cc1c8d0/588fd495-6492-4610-b3e8-d15ce864454c.txt",
        "platform": "LINUX_UNIX"
    },
    {
        "blueprintId": "wordpress_multisite",
        "name": "WordPress Multisite",
        "group": "wordpress_multisite",
        "type": "app",
        "description": "WordPress Multisite is ideal for organizations such as universities, corporations, and agencies that need to enable many people to host and manage their own websites while giving overall control to a central administrator. These websites can all have unique domain names and layouts while sharing assets such as themes and plugins. Popular plugins like Jetpack, Akismet, All in One SEO Pack, WP Mail, Google Analytics for WordPress, and Amazon Polly are all pre-installed in this image. Let's Encrypt SSL certificates are supported through an auto-configuration script. This image is certified by Bitnami as secure, up-to-date, and packaged using industry best practices.",
        "isActive": true,
        "minPower": 0,
        "version": "5.9.3-8",
        "versionCode": "1",
        "productUrl": "https://aws.amazon.com/marketplace/pp/B00NN8XE6S",
        "licenseUrl": "https://d7umqicpi7263.cloudfront.net/eula/product/2f1d4d67-324b-41d7-8af9-b7860d269c6d/6338ef47-28a8-482d-b57a-4859280a021e.txt",
        "platform": "LINUX_UNIX"
    }
]

Bundle ID for the cheapest one ($3.5 for nano 2.0):

$ aws lightsail get-bundles --query 'bundles[0]' --region ap-southeast-1
{
    "price": 3.5,
    "cpuCount": 1,
    "diskSizeInGb": 20,
    "bundleId": "nano_2_0",
    "instanceType": "nano",
    "isActive": true,
    "name": "Nano",
    "power": 300,
    "ramSizeInGb": 0.5,
    "transferPerMonthInGb": 1024,
    "supportedPlatforms": [
        "LINUX_UNIX"
    ]
}

Now, let's create the instance!

$ aws lightsail create-instances --instance-names wp-instance --availability-zone ap-southeast-1a --blueprint-id wordpress --bundle-id nano_2_0 --key-pair-name bitnami --region ap-southeast-1
{
    "operations": [
        {
            "id": "b4324f47-a520-4424-b99c-ad026c50546b",
            "resourceName": "wp-instance",
            "resourceType": "Instance",
            "createdAt": 1653555007.628,
            "location": {
                "availabilityZone": "ap-southeast-1a",
                "regionName": "ap-southeast-1"
            },
            "isTerminal": false,
            "operationType": "CreateInstance",
            "status": "Started",
            "statusChangedAt": 1653555007.628
        }
    ]
}
$ aws lightsail get-instances --region ap-southeast-1 --query 'instances[].[name, location, blueprintId, bundleId, privateIpAddress, publicIpAddress]'
[
    [
        "wp-instance",
        {
            "availabilityZone": "ap-southeast-1a",
            "regionName": "ap-southeast-1"
        },
        "wordpress",
        "nano_2_0",
        "172.26.13.11",
        "18.141.24.138"
    ]
]

Let's access it through SSH and web browser!

$ ssh bitnami@18.141.24.138
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
       ___ _ _                   _
      | _ |_) |_ _ _  __ _ _ __ (_)
      | _ \ |  _| ' \/ _` | '  \| |
      |___/_|\__|_|_|\__,_|_|_|_|_|

  *** Welcome to the WordPress packaged by Bitnami 5.9.3-8         ***
  *** Documentation:  https://docs.bitnami.com/aws/apps/wordpress/ ***
  ***                 https://docs.bitnami.com/aws/                ***
  *** Bitnami Forums: https://community.bitnami.com/               ***
bitnami@ip-172-26-13-11:~$ cat bitnami_application_password 
KPInbPj1giRB

Because we have the password now and the default username is user, let's go through dashboard!

It's a wrap! In case you want to delete them, just run delete command delete-instance & delete-key-pair!

That's it! Thank you for coming and I'm looking forward to your feedback. Follow me to get notified when my new post is published!

Best Way for Giving Permission to AWS Services

Nurul Ramadhona — Wed, 20 Jul 2022 04:57:39 +0000

For AWS IAM service, we must keep at least privileged access. It's the best practice in using IAM for security purposes. For IAM user, attaching policy at a group level is the best practice. For "specific" AWS services, IAM role is the best way to give permission for the source and IAM policy for the destination.

For example:

Use IAM role to give permission for EC2 instances, let's say for accessing S3 such as listing or creating a bucket.
Use IAM policy to only allow access from any sources, let's say for static website purposes we allow public access to specific buckets.

Here I'll you the first option which is IAM role!

1. Create EC2 instance

Here I'll create an EC2 instance through CLI with Amazon Linux 2 as AMI and leave the rest to use default as it is. Before that, I'll also import the key pair.

$ aws ec2 import-key-pair --key-name "ec2-user" --public-key-material fileb://home/nurulramadhona/.ssh/id_rsa.pub
$ aws ec2 run-instances --image-id ami-021fb2b73ff1efc96 --count 1 --instance-type t3.micro --key-name ec2-user
$ aws ec2 describe-instances --query 'Reservations[].Instances[].{PublicIP:PublicIpAddress, ID:InstanceId}'
[
    {
        "PublicIP": "108.136.45.150",
        "ID": "i-0f3df2b1eb51bc6a1"
    }
]

2. Create IAM role & attach policy

Trust document:

$ cat ec2-role.json 
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

$ aws iam create-role --role-name ec2-role --assume-role-policy-document file://ec2-role.json
$ aws iam attach-role-policy --role-name ec2-role --policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess

3. Create instance profile & add role

An instance profile is a container for an IAM role that you can use to pass role information to an EC2 instance when the instance starts. Please note that we only can have one role per instance profile.

$ aws iam create-instance-profile --instance-profile-name ec2-profile
$ aws iam add-role-to-instance-profile --role-name ec2-role --instance-profile-name ec2-profile

4. Associate instance profile to EC2 instances

aws ec2 associate-iam-instance-profile --instance-id i-0f3df2b1eb51bc6a1 --iam-instance-profile Name=ec2-profile
aws ec2 describe-iam-instance-profile-associations

Let's check! Before:

$ ssh ec2-user@108.136.45.150 

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/
4 package(s) needed for security, out of 5 available
Run "sudo yum update" to apply all updates.
[ec2-user@ip-172-31-0-125 ~]$ aws s3 ls
Unable to locate credentials. You can configure credentials by running "aws configure".

After:

[ec2-user@ip-172-31-0-125 ~]$ aws s3 ls
[ec2-user@ip-172-31-0-125 ~]$ aws s3 mb s3://bucket-ec2-role
make_bucket failed: s3://bucket-ec2-role An error occurred (AccessDenied) when calling the CreateBucket operation: Access Denied

5. Least Privilege Access

As we can see above, we can list a bucket (currently empty) but can't create a bucket with an error Access Denied. If we really need it, we can attach one more policy to the IAM role. This is what I mean by giving permission as needed. So, let's try!

$ aws iam attach-role-policy --role-name ec2-role --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess

[ec2-user@ip-172-31-0-125 ~]$ aws s3 mb s3://bucket-ec2-role
make_bucket: bucket-ec2-role
[ec2-user@ip-172-31-0-125 ~]$ aws s3 ls
2022-05-26 02:43:47 bucket-ec2-role

Then, we also can detach if the policy is no longer needed.

$ aws iam detach-role-policy --role-name ec2-role --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess

[ec2-user@ip-172-31-0-125 ~]$ aws s3 mb s3://bucket-ec2-role2
make_bucket failed: s3://bucket-ec2-role2 An error occurred (AccessDenied) when calling the CreateBucket operation: Access Denied

Additional: In case you want to delete the IAM role, make sure we have:

Remove the role from the instance profile before deleting the instance profile.
Delete instance profile.
Detach all policies from the role.

That's it! Thank you for coming and I'm looking forward to your feedback. Follow me to get notified when my new post is published!

GitHub Vibes on AWS: What Makes CodeCommit Special?

Nurul Ramadhona — Sat, 16 Jul 2022 05:55:00 +0000

AWS CodeCommit is a secure, highly scalable, managed source control service that hosts private Git repositories. Simply means we can use GitHub on AWS with any special features such as easy scale and many more. One of the biggest benefits is it can be integrated with other AWS services. Before we talk about it, let me show you how to set up or use it first.

How To Use CodeCommit

1. Setup Credential

Just like we use GitHub, we need to set up credentials to be able to access our repositories but in this case, we will integrate it using IAM either HTTPS or SSH. Here I'll use SSH by uploading my public key to one of my IAM User credentials.

$ aws iam upload-ssh-public-key --user-name dhona --ssh-public-key-body file:///home/nurulramadhona/.ssh/id_rsa.pub

Create config file under .ssh directory:

$ chmod 600 .ssh/config
$ cat .ssh/config 
Host git-codecommit.*.amazonaws.com
  User (ssh-key-id-goes-here)
  IdentityFile /home/nurulramadhona/.ssh/id_rsa

2. Create & Clone Repository

Please specify the "other" region if it's not available yet in your region.

$ aws codecommit create-repository --repository-name ops-io --region ap-southeast-1
$ git clone ssh://git-codecommit.ap-southeast-1.amazonaws.com/v1/repos/ops-io
Cloning into 'ops-io'...
warning: You appear to have cloned an empty repository.

3. Make Any Changes

GitHub commands:

$ cd ops-io/
~/ops-io$ cat test.txt
this is first file
~/ops-io$ git add -A
~/ops-io$ git commit -m "upload first file"
[master (root-commit) 1406bd0] upload first file
 1 file changed, 1 insertion(+)
 create mode 100644 test.txt
~/ops-io$ git push origin master
Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Writing objects: 100% (3/3), 236 bytes | 236.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0)
To ssh://git-codecommit.ap-southeast-1.amazonaws.com/v1/repos/ops-io
 * [new branch]      master -> master

CodeCommit commands:

~/ops-io$ echo "this is second file" > file.txt
~/ops-io$ aws codecommit get-branch --repository-name ops-io --branch-name master --region ap-southeast-1
{
    "branch": {
        "branchName": "master",
        "commitId": "1406bd0566858448e486011edc364ca3a77d0b0d"
    }
}
~/ops-io$ aws codecommit create-commit --repository-name ops-io --branch-name master --parent-commit-id 1406bd0566858448e486011edc364ca3a77d0b0d --put-files "filePath=file.txt,fileContent='upload 2nd file'" --region ap-southeast-1
{
    "commitId": "307e9fb75d0a7ccc490aacbf28aba957342a8c3b",
    "treeId": "918e7c9702c916bc7b67f7645432e048cdfe94a4",
    "filesAdded": [
        {
            "absolutePath": "file.txt",
            "blobId": "6e29f4102c6b829414b53053cd65c1b123d28d29",
            "fileMode": "NORMAL"
        }
    ],
    "filesUpdated": [],
    "filesDeleted": []
}

4. List Repositories & Branches

$ aws codecommit get-branch --repository-name ops-io --branch-name master --region ap-southeast-1
{
    "branch": {
        "branchName": "master",
        "commitId": "1406bd0566858448e486011edc364ca3a77d0b0d"
    }
}
$ aws codecommit get-folder --repository-name ops-io --folder-path "" --region ap-southeast-1
{
    "commitId": "307e9fb75d0a7ccc490aacbf28aba957342a8c3b",
    "folderPath": "",
    "treeId": "918e7c9702c916bc7b67f7645432e048cdfe94a4",
    "subFolders": [],
    "files": [
        {
            "blobId": "6e29f4102c6b829414b53053cd65c1b123d28d29",
            "absolutePath": "file.txt",
            "relativePath": "file.txt",
            "fileMode": "NORMAL"
        },
        {
            "blobId": "ce553855c343e93616e1e937cd288a9824f0d761",
            "absolutePath": "test.txt",
            "relativePath": "test.txt",
            "fileMode": "NORMAL"
        }
    ],
    "symbolicLinks": [],
    "subModules": []
}

Special Task: Copy S3 Object To CodeCommit

As I mentioned before, one of the biggest benefits we host services on AWS is that they can be integrated with one another. Here I'll give you a simple task of how we can do it.

I'll go with CloudFormation to create a new repository named S3 and copy objects (zip format) from the S3 bucket to the branch bucket.

Before that, I'll create a new bucket named repository-ops-io with versioning enabled and upload the zip file to that bucket.

1. Create Bucket

$ aws s3 mb s3://repository-ops-io --region ap-southeast-1
make_bucket: repository-ops-io
$ aws s3api put-bucket-versioning --bucket repository-ops-io --versioning-configuration MFADelete=Disabled,Status=Enabled --region ap-southeast-1
$ aws s3api get-bucket-versioning --bucket repository-ops-io --region ap-southeast-1
{
    "Status": "Enabled",
    "MFADelete": "Disabled"
}

2. Upload Object

$ touch s3.txt
$ zip s3.zip s3.txt
  adding: s3.txt (stored 0%)
$ ls
s3.zip
$ aws s3 cp s3.zip s3://repository-ops-io --region ap-southeast-1
upload: ./s3.zip to s3://repository-ops-io/s3.zip
$ aws s3api list-object-versions --bucket repository-ops-io --query 'Versions[].[Key, VersionId]' --region ap-southeast-1
[
    [
        "s3.zip",
        "Km7s2m1Jd2FlcvoIi9v7n.KUenzjHZ5R"
    ]
]

3. Create Stack I give the template name ops-io.yaml.

AWSTemplateFormatVersion: 2010-09-09
Resources:
  Repository:
    Type: 'AWS::CodeCommit::Repository'
    Properties:
      RepositoryName: s3
      RepositoryDescription: S3 to Codecommit
      Code:
        BranchName: bucket
        S3:
          Bucket: repository-ops-io
          Key: s3.zip
          ObjectVersion: Km7s2m1Jd2FlcvoIi9v7n.KUenzjHZ5R

$ aws cloudformation create-stack --stack-name ops-io --template-body file://ops-io.yaml --region ap-southeast-1 --region ap-southeast-1
{
    "StackId": "arn:aws:cloudformation:ap-southeast-1:0123456789:stack/ops-io/ae15bb10-dcaf-11ec-b865-0a5e031cf822"
}

Let's check!

$ aws codecommit list-repositories --region ap-southeast-1
{
    "repositories": [
        {
            "repositoryName": "ops-io",
            "repositoryId": "706334d5-c480-4b69-9395-512e0aff3a4c"
        },
        {
            "repositoryName": "s3",
            "repositoryId": "71719f7e-5e45-4b25-aa82-3247bfd297a3"
        }
    ]
}
$ aws codecommit list-branches --repository-name s3 --region ap-southeast-1
{
    "branches": [
        "bucket"
    ]
}
$ aws codecommit get-folder --repository-name s3 --folder-path "" --region ap-southeast-1
{
    "commitId": "1c69f63e89e579a7e41b27c2b9a97e2f90344f26",
    "folderPath": "",
    "treeId": "3210de697af8657277ba1d8ec5509438f4521482",
    "subFolders": [],
    "files": [
        {
            "blobId": "e69de29bb2d1d6434b8b29ae775ad8c2e48c5391",
            "absolutePath": "s3.txt",
            "relativePath": "s3.txt",
            "fileMode": "NORMAL"
        }
    ],
    "symbolicLinks": [],
    "subModules": []
}

Additional:

To delete all resources created above, please do the following steps:

Delete the S3 object with the version mentioned
Delete the S3 bucket
Delete the CloudFormation stack
Delete the CodeCommit repository

That's it! Thank you for coming and I'm looking forward to your feedback. Follow me to get notified when my new post is published!