DEV Community: Emmanuel Awotunde

Building More Resilient Backend Infrastructures

Emmanuel Awotunde — Thu, 29 Oct 2020 05:48:47 +0000

When building for the web, the backend infrastructure is usually the part that needs the most security and resilience. Over the years, the web has grown very large. There are billions of people surfing the web today.

Some perspective

To put it in perspective, Facebook has over 2.7 billion monthly active users as of the second quarter of 2020 (1), Instagram has over 1 billion monthly active users (2), Twitter has over 330 million (3). Compare this to the current population of the world which is about 7.8 billion as of October 2020 (4).

So when building software for the web, you need to put this into consideration. Compared to the late '90s and early 2000s, whatever we are building today has the potential of being used by billions of people. Millions of people could be using our software simultaneously. To ensure we are building what can serve this high population, a lot of things have to be considered when building the infrastructure of the backend. No doubt, the frontend and other parts of a software matter just as much but in this article, my focus is on the backend.

I have spent more than 2 years writing a backend for software that serve tens of thousands of users, I have experienced a lot of challenges and I can only imagine the challenges I'd face when the users of those software grow into hundreds of thousands, millions and billions. One thing I have realized is that in software, experience teaches you a lot of things you would not learn from training or tutorials. I thought to compile challenges that you can face in building for a growing population of users.

Why I wrote this

I got this idea while talking with one of my friends, Nicholas, an engineer at Paystack. At that time, I was confronting a challenge I encountered with an implementation I wrote for a particular part of the codebase I manage at my company. I spoke with him and he was able to profer a viable solution.

Whenever I encounter a major bug and it's taking more than 10 minutes or more to figure out, I decide to stop using the conventional ways of fixing bugs. If a bug is taking me too long to fix, it most likely means that I don't have enough knowledge in the domain of that that bug is showing up. For example, if I encounter a bug that has to do with reading files in Node.js and it's taking me more than 10 minutes to figure out then I need to back out from attempting to fix the bug. What I do is read a few articles or videos of working with files in Node.js. I also reach out to people who I know are experts in the domain. I could figure out what's wrong from this or just having the knowledge will streamline my search for how to fix the bug. This has long term benefits. This is just by the way.

Back to the point of this article. How do we build a resilient backend? This isn't dependent on the programming languages or frameworks you use. Writing code is an art, the more creative the artist is the better the work of art. In building a resilient backend, there a number of things I have found out from experience.

Whatever can go wrong will go wrong

This is a very relevant idea to have when building backend infrastructures. You should note things that as they come to your mind and ensure to come back to them. While coding, you may have thoughts that some implementation you wrote may be problematic or may break in certain situations. It's better to either stop and consider all the ways things can go wrong and handle them properly or comment in the code that you have some concerns so you or anyone who looks through the code could attempt to fix it.

Single point of failure

The larger your backend infrastructure, the more attention you need to give to it. You need to constantly monitor all the moving parts and ensure they are smooth. If you notice unusual traffic in certain areas, you should note them and attend to them. You should look out for bottlenecks and single points of failure. A single point of failure is a part of your infrastructure that if it fails, the whole backend infrastructure will not be able to function. You should either reduce as many single points of failure as possible or work very hard to secure and ensure they are very reliable.

Automation

As your backend infrastructure grows bigger, you need to automate a lot of things you previously did manually. Some examples:

running database migration
running builds
code linting and formatting
backups
deployment
and many more

Monitoring

Monitoring how well your backend is performing will give you insights on how to improve it. You should use robust monitoring tools depending on how large your backend is. Tools like Prometheus, Grafana, etc.

Security

Take security very seriously on the first day. There are many areas to consider security

Code - perform validation before processing data, add lots of checks to prevent malicious activities. You should also implement rate limiters to prevent DDoS attacks
Secrets - you should protect all your secret keys
Server & cluster - ensure you implement things like firewalls and other server related security checks
Load balancer - apply the same security checks for your load balancers
Database - ensure no one else can access your database. It's advisable to only allow incoming request from your servers alone to prevent brute force attacks
Domain - when purchasing a domain, some domain providers offer extra security for your domain

You should consult experts in the security domain to help you tighten your loose ends.

Routine checks

As an extension of security, you should perform routine checks on your servers to ensure everything is stable. You should have a schedule for this, it could be daily, weekly, monthly, quarterly or anyone that works for you and your team. This could help you to catch errors on time. It keeps you informed and you can know how frequently certain errors show up and how you can prevent them. You can also write different scripts to do these checks so you can even plug it into a cron job and it notifies you when it finds anything fishy.

Code quality

Your codebase is being used by two sets of people. The customers it's serving and the developers that are working on it. A code that works is good for customers, a code that's readable is good for developers. Your code quality will help you iterate faster and build new features faster. You'd be able to have more developers work on the codebase because it will be easier to read and understand.

A lot of things go into ensuring high code quality. Things like documenting, commenting where needed, having good workflows (such as git flow), having a checklist before allowing code merge into the main branch.

Automated tests

You should write tests to cover for as many scenarios as possible. This will make you more confident in your code and reduce the number of manual tests you have to carry out. You should also add a test step to your continuous integration so no feature is shipped without passing its tests. This is extremely useful when refactoring implementations, you run the test before refactoring, you run the test after refactoring. This will help you know if you broke the code. Of course, you may not be able to write an automated test for everything, there are times you have to test manually.

Common database tricks

I have used SQL databases more than I have used NoSQL databases. Most of the common tricks in this list are for SQL databases.

Transactions - there are certain flows in your application that needs to run a number of database writes that must all happen in a sequence and if one fails, all should fail. You can use database transactions to achieve this
Racing conditions - a racing condition occurs when two database queries try to access and update the same row simultaneously. This causes computational inconsistencies. You can fix this problem using specific select queries. You can check this stackoverflow question to learn more.
Pagination - when fetching data from a user-generated table, it's best to paginate because attempting to fetch all at once will be problematic as the table grows very large.
Others - there are a lot more issues to consider when using databases. It's better to consider and implement them from day one.

Timezone issues

I like the following articles on working with timezones. You should check them out.

Failover, fallbacks, retries and rollbacks

To keep backend servers operational all the time, you need to implement failover in the case of a service outage. You should have a backup plan for when your primary servers crash or experience outage.

Fallbacks are similar too but this can be applied to the backend code and third-party integrations e.g payment. If your application depends on a third party service then you should handle cases when the third party service will be offline gracefully and provide a form of fallback action that should occur instead.

Having retries is also another good way of ensuring reliability. For services that need to be available all the time, if there's a slight issue with the service then the connection should be retried a number of times and timeout to notify you that there's a serious issue with the service. If it's a less important service and retries are an expensive operation then you can implement exponential back off. Exponential backoff is an algorithm that uses feedback to multiplicatively decrease the rate of some process, in order to gradually find an acceptable rate. You can check out this article on Google Cloud.

Rollbacks, a good deployment process should include a rollback option. When you mistakenly ship a bug into production, instead of attempting to quickly fix the bug and redeploying, you should be able to rollback your deployment to the older version.

Queues

When handling requests from users, there are some computations/processes that may take very long and the request may timeout. Such computations/processes can be handled by Queues. Queues can solve many different problems in an elegant way, from smoothing out processing peaks to creating robust communication channels between microservices or offloading heavy work from one server to many smaller workers, etc.

Distributed systems

This article doesn't cover design patterns for distributed systems. You can read about them in these links:

Keeping up to date

I sometimes read about how companies managing large scale applications solve difficult problems. This helps me foresee some problems or challenges I may face in the future and learn about them before time. Two blogs come to mind.

Please note that you don't have to solve the problems they solve the same way or take their own solution as final, you should just learn from their solution and that can help you think of better solutions. You should also know how large their application is and compare that to the problem before you start assuming your own application will face the same problems soon.

There are definitely more challenges I haven't faced yet but these are very common ones you will face when building for a growing population of users. Remember that most challenges will be specific for your situation so you should seek to understand your situation well and apply the right solution to it.

References

(1) https://www.statista.com/statistics/264810/number-of-monthly-active-facebook-users-worldwide

(2) https://backlinko.com/instagram-users

(3) https://ng.oberlo.com/blog/twitter-statistics

(4) https://www.worldometers.info/world-population

My Experience With Rebuilding a Product from Ground Up

Emmanuel Awotunde — Tue, 20 Oct 2020 06:33:14 +0000

I joined my present company (Pettysave) mid-2019 after concluding my undergraduate studies. In my first few weeks, I didn't feel like I was in a tech company. The business product was initially built by an outsourcing agency. In my opinion, they didn't do a good job with the product. Of course, I wasn't present during the process of developing the product so I did not know how they got there. One thing was sure, the product was very problematic and needed an upgrade.

Problems With Outsourcing

Outsourcing the development of your business product is not bad but before doing it, you should definitely know the pros and cons. It is usually an easy and cheap short-term option.
Apart from the problems I personally experienced with the outsourcing company that built the initial version of Pettysave, I also did a little research and found that it's an often discussed issue globally. Instead of listing the problems, you can just read these really good articles. They explain what I'd have written here:

Initial Worries

When I joined the team, I had only one engineering teammate. My teammate was a frontend engineer while I was a backend engineer. I remember being very concerned with the product and how we could make it better. Pettysave management hired two of us to take over the product from the outsourcing agency. After several discussions with the outsourcing agency and my teammate, we decided we'd rebuild the product and rewrite the entire codebase.

The initial version of the Pettysave product was built with:

ASP.NET (Onion Architecture) - For the website
Microsoft SQL Server - Database

I had no experience with the .NET stack and generally did not like it anyway. It was a no-brainer for me to start learning what I didn't like. I spent a week or more doing research on which stack we were going to use. I read a lot of articles and talked to friends.
After much research, I had two options for the backend stack: Node.js or Python. The frontend stack was decided on time: Vue.js. After looking at my current skill-set, I decided to go with Node.js.
This eventually became the new stack for Pettysave:

Vue.js - Frontend
Node.js (Adonis.js) - Backend
MySQL - Database
Redis - In-memory Database

After deciding the new stack, the next piece of the puzzle was how we were going to migrate the database. This is a very important step that should be done early in the process because it determines if you can move forward or not. We were migrating from Microsoft SQL to MySQL. Luckily for us, they are fairly similar but they have lots of differences regardless. I spent time researching the best approach to solve this. The structure of the old database wasn't the best so I had to make modifications to make it more normalized and to track more information.

Migrating the database

Because this step was very important, I had to be very careful. I had a few concerns:

Changing the structure of the database without destroying old data
Migrating the passwords of users i.e rehashing to fit new hashing algorithm in version 2
Generating new data to fit new database structure i.e populating new tables or columns created in version 2

I eventually found a tool that could migrate a database in MSSQL to MySQL: MySQL Workbench. You can read this article to know how it works: How to Migrate from MSSQL to MySQL by Sebastian Insausti. After several trials, I was able to convert the database to MySQL. The next step was to transform the MySQL database into the new data structure for version 2.

This proved to be a very hard task because I had to write a script for it and test it several times. This took weeks, no kidding. I didn't spend the whole day working only on the script, I was still writing the backend for version 2 alongside. I got frustrated a lot during this process and even scared sometimes.

Writing the Backend

As I said earlier, I had to rewrite the whole backend infrastructure from the ground up. It was fairly easy because I learnt from the current version of the product and improved on it. I used Adonis.js, a Node.js framework. Adonis.js is a really simple framework to use, it was modelled after Laravel. It had most of the things I needed to write a savings & investment backend:

Routing
Database ORM
Middleware
Mailing
Validator
Authentication
Encryption & Hashing
File Upload
Redis
Events
Testing

I enjoyed most of the time I spent writing the backend code.

Handover

When I started working on the backend, Pettysave management had requested for the codebase of version 1 from the outsourcing agency. We got the codebase weeks later. So I was able to extract more information from the code. I saw lots of mistakes they made and thought of ways to correct them in version 2.

Apart from the codebase, the outsourcing agency had our domain name, hosting, database and some API keys. So we started the process of retrieving all these. The hardest one to retrieve was the domain name because it was bought in their name so they had to do domain transfer, it took a while to do this. We didn't need the hosting because we weren't going to deploy into a Windows environment, we discarded all old API keys they had access to prevent any funny thing that may occur in the future.

Once we got everything, we were more confident about the launch and to continue the maintenance of the platform.

Test & Launch

After about 3 months of working on version 2 with my team members, we were set to launch. Others were set to launch, I wasn't. I was yet to finish the database migration script. I focused more on the script and spent much time on SQL. I learnt so much about SQL in that period. When I completed the script and tested it, we were now ready for launch. We had previously shifted the launch date like 4 times. We eventually settled on January 17th 2020. I had deployed a test version of the product so we could be testing internally. That had been going for weeks, this process is very important. It's easy to know if a product was tested before launch. A tested product is more usable than a non-tested one. It's that simple because, during the test, people simulate how users will use it and give feedback on how to improve the usability of the product.

On the launch day, of course, I was still making a lot of edits to the backend and working on the deployment. I had chosen AWS for hosting. So I made use of Docker and deployed to AWS Elastic Container Service (ECS). I wrote an article about deploying to ECS. Version 2 of Pettysave went live at about 11 am. Poof! that was it.

Launching Mobile Apps

Version 2 of the website was at app.pettysave.com. We used Cordova to bundle the web application into a mobile app. We submitted the apk for the Android version to Play Store later that day. Releasing on iOS was a longer process. We had to first apply to the Developer account of App Store to get approval. That took about a week before we were able to submit the app on App Store.

Problems After Launch

Once we launched, we initially experienced a lot of problems with the servers. I had provisioned a very small EC2 (Elastic Compute Cloud) instance so the server kept crashing several times day. I had to stay up all night to ensure it was stable.

Due to the changes in the data structures on the database, there was some information I had to generate. Some customers complained that their data wasn't consistent with the old version, I had to do checks to ensure they were all corrected.

Alongside the API server, we had a task runner (cron job) that was responsible for debiting customers when it's their turn to save. Initially, I had written a simple loop to get all the debit cards and made API calls to our payment processor to charge the card. This was a very bad idea because one error stopped the whole loop and that was occurring every day after launch. I had to do some research and speak with my friends on how to solve this, I eventually used a queuing system to solve it (I'd write an article about later).

The deployment was manual at launch, I had built the docker images on my PC and push to AWS ECS (Elastic Container Registry). After a week or more, I set up continuous delivery on Bitbucket so that whenever the master branch was updated, a build is triggered and deployed without any manual process. This saved me a lot of time and headache. I explained how I did this in this article.

After Launch

One thing was clear once we launched, we had to be ready to answer our customers' questions on complaints. We were ready, we had installed an instant chat service directly on the app. We received lots of complaints about the new changes. Many customers were happy with them, some weren't but it was a generally positive response. We also set up our support mail [support@pettysave.com] which I handled. I handled the complaints from customers for several months. This made me have a deeper understanding of what to build and what's important to customers.

It's been 9 months since then and the progress we've made has been fantastic. We acquired more users in the first 3 months of version 2 than the first one year of version 1. That's more than 100% increase in user growth. Revenue also went up and is still going up.

Lessons I Learnt

I really loved the experience of working a product from the ground up. The joy of building something that adds value to people is really great. Rebuilding a product gives you flexibility and allows you to correct old mistakes and make new and better approaches to solving old problems. The problem the product was solving didn't change, the product did.

You cannot optimize the code you haven't written - until you write the code before you can optimize it
Focus on the most important aspects of the product - there are many other aspects that are not so important and could distract you
Listen to your customers, your customers tell you their problem - it's your job to figure out the solutions to their problems
Prepare for failure - when building a product, you should know that things go wrong so be prepared for this
Iterate fast - receive feedback from customers and use it to build and iterate new features
Measure progress and success - before improving a product, you should have success metrics you'd use to determine if your new version is successful
Chill - there will always be work to do, don't overwork yourself, have a plan and include rest in your plan
Celebrate - celebrate your wins, after releasing new features you should celebrate as it grants you the reward for your work

Pettysave is an online savings and investment platform that helps people achieve their financial goals, you can sign up with my referral link to receive a NGN250 bonus when you start saving. Sign up here: app.pettysave.com/s/EMMA16522

A Simple Road Map to Learn New Frameworks

Emmanuel Awotunde — Mon, 27 Jul 2020 23:58:57 +0000

It's 2020, there are lots of things to learn as a software developer. Your ability to learn fast sets you apart.

One of the best things about modern software development is the availability of free tools and frameworks. For example, as a frontend developer there's a lot of frontend frameworks you can use: React, Vue, Angular, Emberjs, Backbonejs, and many more.
This could be challenging because you have to select what to use. A developer in 2020 would have used at least 2 frameworks one time or another.

Picking up a new framework is good for the learning experience and expanding your perspective. It doesn't really matter which framework it is. I see it as starting a new adventure. I have used a lot of frameworks over time. In this article, I will explain some methods I used to learn fast and also talk about some nuances that surround the frameworks' ecosystems.

Getting the Overview

A simple overview of the framework helps create a foundation or a sort of guide. Most frameworks have a website and give a short explanation of the framework. A good explanation will include a particular problem the framework aims to solve. Knowing this will help you see the point of using the framework.

An example is React. On React's landing page, you easily see what the framework is about.

Watching videos about the framework also helps kick start your learning. Good videos will expand on the problem the framework solves and if there are other frameworks solving the problem, the video explains the approach the framework takes and why it may be better than the others.

Code

As much as understanding the overview is very important, nothing is more important than code. The best way to learn code is to write code. When learning a new framework, you want to get to code as soon as possible because it's a different ball game entirely. Some frameworks have a good overview experience (OX, lol) but when it comes to code it's terrible.

If the framework has tutorials on their documentation then it's probably the best place to start. It'll be good to just follow the tutorial, that's writing the code as explained in the tutorial. If the framework's documentation lacks tutorials then you may have to do a little search on the internet for tutorials.

Depending on your experience level, this phase should take a few hours to days to get used to the basic parts of the framework.

Project

Why learn a framework if you won't use it for building something?

Learning a framework should be done for a purpose. A framework is a tool for building something so while in the processing of learning a framework this should be taken most seriously. To explore the deep parts of a framework, you need to have a project that makes use of those parts of the framework.

One way to learn really fast in this phase is to join an existing project with someone or a group of people who are experienced with the framework. This makes it easier for you to explore the framework. You can easily reach out to the experienced people on the project if you have questions or you're experiencing some blocker.

If you are working on a personal project, you can still consult people with experience in the framework. Experienced people have a wealth of knowledge beyond the surface level, they can explain some methodologies used in the framework that'll make your learning experience very easy. They will also point out some nuances you may not be aware of (and sometimes that is not in the framework's documentation).

To quickly evaluate how comfortable you will be with a framework, you can re-implement a former project with this new framework. Of course, this will take time if you are re-implementing a large project. If you do not have small projects, you can strip down features from a large project and implement only a few parts of it. Doing this allows you to compare your experience with this framework compared to other frameworks you have used.

Tinkering

At this phase, you are already comfortable with the framework to some extent. This is where you can flex your developer muscles and play around the framework. There are a number of things you can do here:

Experiment with config: Frameworks usually come with configurations that have defaults. To really know how these configs work, you can change the defaults to different allowed values (You can attempt extreme values to see how the framework reacts). Doing this helps you understand better how the configuration affects the overall working of the framework.

Look into the source code: A framework's source code is the single source of truth. You can learn a lot by studying the source code. Sometimes, documentation and tutorials are out-dated but the source code is never outdated. You should take this as practice. While studying a framework's source code, you may find some inefficiency and you can offer to contribute by opening an issue on Github or submitting a pull request.

Play with the CLI: If a framework comes with a Command Line Interface (CLI), then you should definitely play with the commands it comes with.

Community

No framework is perfect

If you use a framework long enough, you will find out it doesn't provide everything you want. Lots of frameworks have a community around them and this is where you find other people who have similar ideas, issues, or problems while using the framework. It's really beneficial to be active in the community because you get to meet really cool people and learn from them. You can even get to interact with the maintainers of the framework which is really cool!

A good community will have a set of guidelines that will keep things organized. The members will also be welcoming and will always offer to help when raise questions or have issues. A good community may also showcase large projects that are using the framework, this gives anyone new to the framework some level of confidence in the framework.

I hope this article will help you to get faster at learning new frameworks or technologies.

Happy coding!

Easy Deployment Setup With Bitbucket and AWS ECS

Emmanuel Awotunde — Mon, 13 Jul 2020 17:57:00 +0000

Deployment is one of the most important parts of product development and launch. The better the deployment process is the faster it is. There are many deployment tools that make things easier today.

In this article, I will take you through a deployment setup with Bitbucket and AWS Elastic Container Service (ECS).

Requirements

Basic knowledge of git version control
Basic knowledge of Docker
An AWS account
A Bitbucket account

Bitbucket is a web-based version control repository hosting service owned by Atlassian. They have built-in Continuous Integration and Delivery/Deployment (CI/CD).

We'll deploy a simple express app. Let's generate an express app



npx express-generator

If you don't have npx, you can install express-generator globally



npm install -g express-generator
express

Now we have our express app, let's create a docker file that we would use for deployment



touch Dockerfile

Copy this and paste in your Dockerfile



FROM node:8-alpine
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE 3000
CMD [ "npm", "start" ]

Setup AWS ECS

Navigate to Repositories under Amazon Elastic Container Registry (ECR).

AWS ECR is a container registry for docker. It's similar to Docker Hub. Container registries are used to store and distribute docker images.

Create a repository by clicking the 'Create repository' button then give it a name like my-express-app.

Now we have our repository URI: XXXXXXXXXXX.region.amazonaws.com/my-express-app

Pushing to AWS ECR

We will build our docker image and push it to our newly created repository on ECR.
For this article, we'll use docker build. If you have a more complex project setup, you may want to consider docker-compose

cd into the folder where the Dockerfile is and run



docker build -t my-express-app .

If the command ran well, you should have this:

To push the image to ECR, you have to first install aws cli tool.

Run aws2 configure to enter your AWS access ID and secret access key. Once you're done, run this command:



$(aws2 ecr get-login --no-include-email --region us-east-1)

Then push the docker image. For the --region option, put your own region.



docker tag my-express-app:latest XXXXXXXXXXX.region.amazonaws.com/my-express-app:latest
docker push XXXXXXXXXXX.region.amazonaws.com/my-express-app:latest

Setup ECS Cluster

Navigate to the Clusters page. Click Create Cluster.

We will use EC2 Linux + Networking.

You can use the following configuration to create the cluster:

Give your cluster a name.
Create an empty cluster: Unchecked
Provisioning Model: On-Demand Instance
EC2 instance type: t2.nano (You can use whatever will fit your application needs)
Number of instances: 1
EC2 Ami Id: Amazon Linux 2 AMI [ami-08b26b905b0d17561]
Root EBS Volume Size (GiB): 30
Key pair: You can create anyone and attach it. This is the key you can use to ssh into the ec2 instance.
Networking: You can create a new VPC or use an existing one.
- Auto-assign public IP: Use subnet settings
- Security group: Create a one that opens up port 80
CloudWatch Container Insights: Unchecked

Create an Application Load Balancer

An Application Load Balancer allows us to dynamically assign applications to routes using pattern matching. This will be used in the ECS Service

To create one, navigate to the load balancers tab on EC2 and click on Create Load Balancer

Give it a name, fill the Availability Zones, and add a security group that exposes port 80 and 443 (if you want to enable SSL).

In Step 4: Configure Routing, you'd create a target group. You can skip Step 5.

Setup ECS Task Definition & Service

A task definition specifies the container information for our application. Navigate to Task Definitions and click on Create new Task Definition.

Select EC2 as launch type compatibility

You can use the following configuration to create the task definition:

Task Definition Name: my-express-app
Task Role: Leave empty
Network Mode: default
Task memory (MiB): 128 (You can use whatever size that fits your application needs)
Task CPU (unit): 128 (You can use whatever size that fits your application needs)
Container: Click on Add container
- Container name: my-express-app
- Image: XXXXXXXXXXX.region.amazonaws.com/my-express-app:latest
- Port mappings: Host - 0, Container port: 3000
- Configure the remaining to fit your application needs
- You can enable Log configuration so you can view the application logs on CloudWatch.

Once you're done, click on Create. Next, we'll create a Service.
A service lets you specify how many copies of your task definition to run and maintain in a cluster.

Click the Actions dropdown and select Create Service

You can use the following configuration to create the service:

Launch type: EC2
Task Definition: my-express-app
Cluster: my-express-app (or whatever cluster you created)
Service name: MyExpressAppService
Service type: Replica
Number of tasks: 1 (Or whatever number fits your application needs).
Minimum healthy percent: 100
Maximum percent: 200
Deployment type: Rolling update
Placement Templates: AZ Balanced Spread
Load balancing: Application Load Balancer
- Load balancer name: Select the load balancer we created earlier
- Container to load balance: Click on Add to load balancer
- Select the target group we created earlier or create a new one here
Use default in the remaining steps or configure to your application needs

Once the service is created, it'll run the task in the cluster.

Setup CD with Bitbucket

We are going to create a new repository on Bitbucket. You can name it anything. I named mine my-express-app.

Automated CI/CD comes built-in with Bitbucket. To enable it, navigate to Repository Settings > Pipelines > Settings and switch it on.

Bitbucket pipeline file

Bitbucket uses a file named: bitbucket-pipelines.yml for automating CI/CD. Create the file.



definitions:
  services:
    push-image: &push-image
      name: Build and Push Docker Image
      image: atlassian/pipelines-awscli
      caches:
        - docker
      services:
        - docker
      script:
        - export BUILD_ID=$BITBUCKET_BRANCH_$BITBUCKET_COMMIT_$BITBUCKET_BUILD_NUMBER
        - export DOCKER_URI=$DOCKER_IMAGE_URL:latest
        # Login to docker registry on AWS
        - eval $(aws ecr get-login --no-include-email)
        # Build image
        - docker build -t $DOCKER_URI .
        # Push image to private registry
        - docker push $DOCKER_URI

    deploy-to-ecs: &deploy-to-ecs
      name: Deploy to ECS
      image: atlassian/pipelines-awscli
      script:
        - pipe: atlassian/aws-ecs-deploy:1.1.0
          variables:
            AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
            AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
            AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION
            CLUSTER_NAME: 'my-express-app'
            SERVICE_NAME: 'MyExpressAppService'
            TASK_DEFINITION: "task-definition.json"

pipelines:
  branches:
    master:
      - step: *push-image
      - step: *deploy-to-ecs

You'd notice TASK_DEFINITION. This is needed to update the ECS service for deployment. So let's create one:



touch task-definition.json

Paste this in the task definition file



{
  "containerDefinitions": [
    {
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/my-express-app",
          "awslogs-region": "us-east-1", // use your own region
          "awslogs-stream-prefix": "ecs"
        }
      },
      "portMappings": [
        {
          "hostPort": 0,
          "protocol": "tcp",
          "containerPort": 3000
        }
      ],
      "cpu": 0,
      "image": "XXXXXXXXXXX.region.amazonaws.com/my-express-app:latest",
      "essential": true,
      "name": "my-express-app"
    }
  ],
  "memory": "128",
  "family": "my-express-app",
  "requiresCompatibilities": ["EC2"],
  "cpu": "128"
}

Now we need to put the following environment variables on our bitbucket repository:

DOCKER_IMAGE_URL
AWS_DEFAULT_REGION
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

Navigate to Repository Settings > Repository variables >

Once you add them, commit and push to bitbucket to trigger the pipeline and the deployment process is run.

The following things occur:

The docker image is built and pushed to the AWS ECR repository you created
The task definition is updated with the task-definition.json file
The service (MyExpressAppService) is updated
The service automatically registers a new instance of the task and begins draining the old task running
The new task is dynamically registered on the Application Load Balancer

You can check the full bitbucket repository here

Debugging Adonis With Chrome DevTools

Emmanuel Awotunde — Wed, 22 Apr 2020 22:34:05 +0000

Debugging Node.js can be very difficult. Most times when something goes wrong it would take a while to get to the root of the problem because the error message may not be descriptive enough or
there's a variable that you expect to have a value but it's null or undefined.

The first thing that comes to mind when debugging in Node.js is usually to use console.log. This usually will get you a solution but in the long run, it wastes time.
We've all been in a situation where we had to use console.log multiple times because once was not enough. In a bid to quickly solve the problem we spend more time logging at different
points of the application.

But it does not have to be that way. There's a much simpler and more effective way of debugging in Node.js. It's by using a very simple keyword:

debugger;

As simple as this is, many people do not know about it. The keyword is more popular for Javascript on the browser because it doesn't need anything more than just typing it anywhere in your code.
In Node.js it's a little different, you have to do more than just typing the keyword. To enable debugging in Node.js, in addition to adding debugger; in your code, you have to run the application using:

node inspect <file_name.js>

There are other ways to debug Node.js aside from node-inspect, some are:

Chrome DevTools: I'd expand more on this later in the article.
Visual Studio Code: It's the most effective if you already use VS Code.
Visual Studio
Some JetBrains IDEs
Eclipse IDE

Debugging with Chrome DevTools

We'll be using AdonisJs (a Node.js web framework) as an example. To enable debugging mode on Adonis run:

adonis serve --dev --debug

This debugger makes use of Chrome DevTools so head over to Chrome (If you have it installed). You should see the Node.js icon when you enable the DevTools. Clicking it will open a new window for debugging.

Whenever the execution of your application encounters the keyword debugger, it'll automatically pause execution and open the Chrome debugger window. You can do a lot of things in this window:

See the value of scope and global variables
View the call stack: allows you to see all the functions that were executed up until the current function being executed
Resume execution
Step over to the next function call
Step into the next function call
Step out of the current function
Pause execution where exceptions are thrown
and many more

Voila! That's it. With this, you can easily debug your AdonisJs application and not have to console.log everywhere😂.

You can check Node.js docs for more information on debuggers.

How to add custom validation rules to AdonisJs Validator

Emmanuel Awotunde — Thu, 09 Apr 2020 05:58:04 +0000

If you are reading this article, I assume you must have used AdonisJs or you must have heard about the framework.
AdonisJs is a NodeJs framework that was inspired by Laravel. The framework was modeled after Laravel.

Like most robust frameworks, AdonisJs comes with a validator module that helps you in validating data (usually user input). However, the default validator does not come with every possible rule, you sometimes have to implement your own. The good news is that it is very easy to write custom rules and add them to the validator. In this article, I will walk you through how to do that.

First, we have to setup our application - let's call it MCU Character Checker. To start, install adonis globally with:

npm i -g @adonisjs/cli

Create a new application with:

adonis new mcu-character-checker

Check installation page for more information.
Here is what our folder structure will look like:

├── app
│   ├── Middleware
│   └── Models
├── config
├── database
│   └── migrations
├── public
├── resources
│   └── views
└── start

Start the application server:

cd mcu-character-checker && adonis serve --dev

You can open the application on localhost:3333, AdonisJs serves on port 3333 by default. You should have this page in the image below if everything ran correctly.

Now, let's create a page that collects data. We'll edit start/routes.js.

Replace the index route with this line:

Route.get("/", "CharacterController.showPage");

We have added our / route, now we have to create our CharacterController to handle this route. We'll do this by running

adonis make:controller CharacterController --type http

Great! Now we have our CharacterController, let's create our showPage method to handle the route:

async showPage({ view }) {
  return view.render("check-character");
}

Create the view for the route:

adonis make:view check-character

This will create resources/views/check-character.edge. Add the lines below to the file.

<!DOCTYPE html>
<html lang="en">
<head>
    <!-- Standard Meta -->
    <meta charset="utf-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=5.0">
    <title>
      Check MCU Characters
    </title>
    <style>
      .container {
        margin:auto
      }
      .form {
        text-align: center;
        margin-top: 20vh;
      }
      .form input, .form button {
        width: 120px;
        height: 30px;
        padding: 0;
        font-size: 15px;
        border-color: black;
      }
    </style>
  </head>
<body>

  <div class="container">
    <div class="form">
      <h1> Check for an MCU Character</h1>
      <form action="" method="POST">
        {{ csrfField() }}
        <input type="text" name="character" required>
        <button type="submit">Click to Check</button>
      </form>
    </div>
  </div>

</body>
</html>

The page should look like this;

Our goal is to validate that a character (provided in the form) exists in the MCU. Obviously, this kind of validation may not be easy to achieve with the rules provided on AdonisJs validator so we will have to create a custom rule.

The rule will check a list of names and validate that the provided name is for an MCU character. To create a custom rule we need to first install Adonis Validator. Run this command to install it:

adonis install @adonisjs/validator

Then, register the validator provider inside the start/app.js file:

const providers = [
  '@adonisjs/validator/providers/ValidatorProvider'
]

To extend the validator, we'll implement our validation function and add it to start/hooks.js. The file isn't part of our current setup, so we'll create it then add these lines to the file:

const { ioc } = require("@adonisjs/fold");
const { hooks } = require("@adonisjs/ignitor");

Adonis provides different hooks for different points in the application lifecycle. Adonis currently has 5 hooks namely: providersRegistered, providersBooted, preloading, httpServer and aceCommand. For our use case, we'll use providersRegistered. Next, we implement our validation function and extend the Validator. Our validation function will have the following parameters, data, field, message, args, get.

data is an object containing the input fields and their values. In our case, it will look like this { character: 'Iron Man' }.

field is the particular input field that is being validated which would be character in our case.

message is the error message that should be returned if the validation fails.

args is a list of extra values passed in the validation rule.

get is a function that is used to get the value of a key in an object (i.e data). To get the value of character, we'll use get(data, field).

Our validation function which will be in start/hooks.js will look like this:

const mcuCharacter = async (data, field, message, args, get) => {
  // get the character value
  const value = get(data, field);

  if (!value) {
    /**
     * skip validation if value is not defined. `required` rule
     * should take care of it.
     */
    return;
  }

  // validate the character
  const shortListOfMCUCharacters = [
    "iron man",
    "hulk",
    "spider-man",
    "loki",
    "thor",
    "odin",
    "black widow",
    "nick fury",
    "gamora",
    "black panther",
    "doctor strange",
    "groot",
    "ant-man",
    "captain america",
    "hawkeye",
    "wasp",
    "star-lord",
    "shuri",
    "valkrie",
    "dormammu"
  ];

  // if the character is not in the list, throw the validation error message
  if (!shortListOfMCUCharacters.includes(value.toLowerCase()))
    throw message ||
      `Nah, ${value} isn't part of the list as far as I am concerned`;
};

We'll extend the Validator with this new validation function.

Validator.extend("mcuCharacter", mcuCharacter);

Note: the Validator.extend's first argument is a string, it represents how you would use this rule. It must be in camel case, Ours is mcuCharacter which would be mc_character when using it as a rule.

We'll place this extension in the providersRegistered hook. So our start/hooks.js will look like this:

hooks.after.providersRegistered(() => {
  const Validator = ioc.use("Validator");
  Validator.extend("mcuCharacter", mcuCharacter);
});

Note: the Validator in imported inside the providersRegistered hook before it's in this hook that providers are registered so the Validator cannot be accessed outside of this.

Now that we have created our custom validation rule mc_character, let's go ahead and use it.

Ways to use Adonis Validator

Route Level Validation: This validates the request payload before it proceeds to the Controller. Check here.
Middleware Level Validation: Here, you perform the validation in a middleware.
Validation in the Controller: You perform the validation in the controller.

You can use any method you like or the one that suits your code structure. I use Route Level Validation when I'm expecting a high number of input fields, I use Middleware Level Validation whenever I need to access the auth object
and I used Validation in the Controller when it's one or two validations I want to perform.

In our example, we'll use Validation in the Controller. Import the Validator in CharacterController:

const { validate } = use('Validator')

We'll add a new route to receive the character in start/routes.js:

Route.post("/", "CharacterController.checkCharacter");

Create the checkCharacter method in CharacterController to handle the request:

async checkCharacter({ request, view }) {
  return view.render("check-character");
}

Let's validate the character. Add these lines to the checkCharacter method

// create the rules for validation
const rules = {
  character: "required|mcu_character"
};
const validation = await validate(input, rules);
let message = `${input.character} is part of the MCU, yay!`;

if (validation.fails()) {
  message = `Nah, ${input.character} isn't part of the list as far as I am concerned`;
}

return view.render("check-character", { message });

We'll modify our check-character.edge template and add the message data. Add this below the submit button:

<p>{{ message }}</p>

Our application is now validating MCU characters!

You can check the full source code here

This post is also published on my website