DEV Community: Bansikumar Mendapara

AWS Elastic WordPress Architecture Evolution

Bansikumar Mendapara — Tue, 06 Oct 2020 15:06:11 +0000

In this blog, I am going to evolve the architecture of a popular WordPress web application. I did this demo as a part of the AWS certification course by Adrian Cantrill. You can find this video course here. Even the free version of this demo is available here.

Overview

Here our goal is to learn how to design architecture for any given scenario. We are going to see the architectural evolution of the WordPress web application. We will start by manually creating a single instance, running application and database. In the end, We will create highly available, scalable and resilient architecture.

Steps

I will go through all the steps of this demo that I followed.

Setup WordPress manually
Automate setup using Launch Template
Database migration to RDS
EFS for local storage
Elasticity using ASG and ALB
Upgrade RDS to Aurora cluster [OPTIONAL]
Cleanup

If you skip step 6, you will able to complete this demo almost in the free tier. If you have any doubt, then I have attached a more detailed instruction link with each step.

Step 1 - Setup WordPress manually [GitHub]

For complete isolation from our current VPC in our AWS account, We will set up a new VPC and perform the whole demo in that. So, in the end, We can easily clean our resources and do not incur any unnecessary charges. You can find one-click deployment here.

After this CloudFormation stack moves into CREATE_COMPLETE status, you have to launch the EC2 instance. To store configuration information, We will use SSM parameter store and create various parameters like DBUser, DBName, DBEndpoint, DBPassword, DBRootPassword.

Now, We will connect to EC2 instance and install database and WordPress. For that, We have to install a web server and database server. We will set up a database and download WordPress. After that, We have to configure a wp-config.php file. We will change permission for file system and create a WordPress user. Next, We will create a database user and configure permissions.

At the end of this step, We can test WordPress by creating a post. Till now everything is manual and We have so many limitations.

Step 2 - Automate setup using Launch Template [GitHub]

In step 1, We did all set up manually. To automate this process, We will create a launch template for this. Whatever configuration We did in the first step, We will do this in user data. Therefore when We launch the instance using this template, We will directly get our WordPress web application ready to use.

By doing this, this will remove the overhead of configuring an EC2 instance whenever we launch it. Even doing this using user data is less time-consuming.

Step 3 - Database migration to RDS [GitHub]

We will split out database functionality from EC2 instance by launching the RDS DB instance. To migrate WordPress data, We will take a backup of the database from EC2 instance and restore into RDS. To accommodate this database migration, We have to change one SSM parameter DBEndpoint and update launch template with necessary command.

By doing this, it allows the database to scale independently of the application.

Step 4 - EFS for local storage [GitHub]

Currently, any media for WordPress posts are stored locally. Now, We will move this data to EFS, which will allow all instances access this shared data. After successful creation of EFS, We will add EFS id to SSM parameter store. To access this EFS, We have to modify user data entered in Launch template.

Now, our application media and UI store are not local. It will also not create any problem in scaling.

Step 5 - Elasticity using ASG and ALB [GitHub]

To add elasticity in our WordPress application, We will add ASG and ALB. We need to add the DNS of ALB in SSM parameter store. To adopt changes, We have to change user data of launch template. We can add a scaling policy for ASG to scale in and out dynamically.

After completion of this step, We have removed direct connection of customer and instance. Now, there is no dependency between the database and instance.

Step 6 - Upgrade RDS to Aurora cluster [OPTIONAL] [GitHub]

In this step, We will move RDS to 3AZ aurora cluster to provide high levels of resilience for the WordPress application. You can skip this part if you do not want to incur any charges. To upgrade our database, firstly We will create a snapshot of RDS instance. After that, We will migrate this snapshot Aurora DB cluster. For resilience, We can create additional readers. We will change the DBEndpoint SSM parameter with the Aurora database endpoint. To apply these changes, We have to refresh our instances using ASG.

Finally, We can create a highly available, scalable and resilient architecture of WordPress web application.

Step 7 - Cleanup [GitHub]

To do clean up, We have to delete load balancer, target group, auto-scaling group, EFS, RDS, launch template, RDS snapshot and CloudFormation template. By doing this, you will delete all resources created during this demo.

Conclusion

I learned how to create this kind of architecture by doing step by step evolution. Thank you, Adrian Cantrill, for this demo. I highly recommend his course on learn.cantrill.io.

Event-Driven Python on AWS

Bansikumar Mendapara — Sun, 27 Sep 2020 17:31:53 +0000

Around 15 days ago, A Cloud Guru launched #CloudGuruChallenge - Event-Driven Python on AWS. It might sound like a simple task, but I faced a few challenges and learned so many things. This blog will completely guide through this challenge.

Overview of Challenge

Forrest Brazeal is the creator of this challenge. The main goal of this challenge is to automate an ETL processing pipeline for COVID-19 data using Python and Cloud services. More about this challenge, you can find here.

Challenge Steps

Now, I would like to go through all the steps of this challenge that I followed.

Transformation
Load transformed data into the database
Notify customers when the database gets any updates
Error handling
Trigger function once a daily
Infrastructure as Code
CI/CD pipeline and Source control
Quicksight dashboard

You can find a full working code in my GitHub repository.

Step 1 - Transformation

One of the requirements of this challenge was to create a separate python module for the transformation task. Therefore, I have created a separate python code for the transformation of data.

Firstly, I filtered both data according to the country name because here we want only COVID-19 data of US only. Next, I dropped unnecessary columns if present and converted date in python DateTime data type. Lastly, I wanted to join both data according to report date and to make that happen, I made some changes in data frame indexes. This function is completely abstracted as it does not care where data is stored and don't know about database also.

def transform(dfNYT,dfJH):
    dfJH = dfJH[dfJH['Country/Region']=='US'].drop(columns='Country/Region')
    dfJH.columns = ['date','recovered']
    dfNYT['date'] = pd.to_datetime(dfNYT['date'],format='%Y-%m-%d')
    dfJH['date'] = pd.to_datetime(dfJH['date'],format='%Y-%m-%d')
    dfNYT.set_index('date', inplace=True)
    dfJH.set_index('date',inplace=True)
    dfJH['recovered'] = dfJH['recovered'].astype('int64')
    dfFinal = dfNYT.join(dfJH, how='inner')
    dfFinal.reset_index(inplace=True)
    return dfFinal

Step 2 - Load transformed data into the database

To fulfil this task, I used the Lambda function and the RDS PostgreSQL database. PostgreSQL has a database adapter for python called psycopg2. It allows us to connect to the PostgreSQL database and perform various SQL queries.

I have used Lambda environment variables to store necessary values like URL of both CSV files, database information(endpoint, port, username, password, region) etc. To make code more structured, I have divided this task into small subtasks and created a function for them.

Database connection - connect to database
First-time data insertion - insert all data at once
Daily data insertion - only insert new data available

I want to minimize manual interaction with AWS Management Console. Therefore, firstly I checked if there is any table called etl available in the database. If it is not available, then the lambda function will create the table first and do first-time data insertion. If etl table is already available then it will check for daily data insertion.

For optimal data insertion, I tried several methods like executemany(), mogrify() and many others. At the end, I went with the below approach.

for i in dfFinal.index:
    row = (dfFinal.loc[i,'date'], int(dfFinal.loc[i,'cases']),int(dfFinal.loc[i,'deaths']),int(dfFinal.loc[i,'recovered']))
    data.append(row)
records = ','.join(['%s'] * len(data))
query = "insert into etl (reportdate,cases,deaths,recovered) values{}".format(records)

Step 3 - Notify customers when the database gets any updates

To do this, I have used SNS service of AWS. I created a function in lambda to publish the message to SNS whenever there is any update in the database.

def notify(text):
    try:
        sns = boto3.client('sns')
        sns.publish(TopicArn = snsARN, Message = text)
    except Exception as e:
        print("Not able to send SMS due to {}".format(e))
        exit(1)

Step 4 - Error handling

For a seamless workflow, error handling is important. My code is able to manage various errors. Like,

If there is any unexpected or malfunction input in data, then transformation function will raise an exception and will notify.
If if there is any issue while connecting to the database, then database_connection() function will notify with reason.
If there is any problem while table creation or data insertion, then a notification will be sent with the raised exception.

If we are doing daily insertion then there is no need to insert whole data again. It is better to insert only new data. For that, I have checked the maximum(last) report date in the table and insert subsequent data. It will also notify customers with the number of updated rows.

Step 5 - Trigger function once a daily

As these CSV files get updated once a day. So there is no need to check for updates in data continuously. That's why I configure event rule which invokes lambda function once a day. You can find a tutorial for the same here.

Step 6 - Infrastructure as Code

It was one of the most challenging parts for me as I didn't play around CloudFormation much. I went from creating one by one element in YAML. In the end, I didn't find that much tough task as AWS has very well-oriented and well-described documentation for each type of AWS service. Even I end up creating RDS instance by CloudFormation and that's why in lambda function I checked first if the table exists or not. Overall, this part gave me more idea about IaC. You can check my CloudFormation template here.

Step 7 - CI/CD pipeline and Source control

Even though this part was optional, I wanted to do it because I do not want to do a manual update in AWS whenever I change my python code or infrastructure. For source control, I used GitHub as it is very handy. To configure CI/CD pipeline, I went for GitHub actions. Now, whenever I push a change to my GitHub repository, GitHub actions will run the pipeline and perform the necessary update.

Step 8 - Quicksight dashboard

The final task was to generate virtualization of US case counts, recoveries, death ratio, daily increase in cases etc.

Major Challenges and Learning

Throughout this project, I faced so many challenges but even learned from them.

To do a transformation and database connectivity, I used pandas and psycopg2 module of python. This module is not readily available in lambda. I have to add as layers in lambda. In starting, I faced an issue with that, but at the end, I found a curated list of awesome AWS Lambda Layers here which made my task easy.
As I wanted to create a table by code only, I was not able to make a way for this in lambda. But after trying a few things, I got this.
I do not have much experience with CloudFormation. So, I learned first by going through various AWS documentation and was able to make it at the end.
CI/CD pipeline is new for me, but I found it really interesting. I configured pipeline with GitHub actions still I want to explore AWS CodePipeline for this.

Conclusion

Overall, I enjoyed this challenge and learned so many things. In this journey, so many people helped me. I would like to thank Forrest Brazeal for creating this amazing challenge. I really appreciate any feedback for this challenge or blog. Thank you.

GitHub Repository