DEV Community: Ashwani Pandey

EC2: The heart of AWS

Ashwani Pandey — Fri, 18 Sep 2020 09:39:00 +0000

Amazon web services have so many services to offer it often feels like finding a needle in a haystack.
EC2 is one of the most popular offerings of AWS.

Elastic Cloud Compute is the scalable compute capacity provided by AWS. It is your hardware on the cloud which is on-demand and ready to be provisioned with a lot of options to choose from. Imagine having the power to provide an unlimited fleet of Instance with a lot of computing capacity and no upfront payment.

EC2 mainly consists of capabilities such as Renting virtual machines, storing data on virtual devices using EBS, distributing the load across machines using ELB, and scaling services using the auto-scaling groups.

Renting virtual machines:

Virtual machines are the heart of cloud computing if you ignore serverless for a moment. Nearly every application running on the cloud uses a virtual machine as their host.

AWS provides a variety of virtual machines to rent with different sizes and memory or compute capacity or based on the payment options as well. The instances sizes vary from t2.nano being the smallest to i3en.metal with various combinations of memory-optimized, compute-optimized, storage-optimized, and General purpose.

It has so many options that one can easily get confused while choosing the right instance for their type of workload. But having so many options to choose from and having categorized instances based on the type of workload they can optimally handle can also provide some help while choosing the instance type.

Pricing of virtual machines also varies from no upfront payment to reserved instances which also provides options to its customer on whether they want to a commitment or not. The pay as you go model can help some to save money if you have no idea about the workload you are going to have. While reserving instances or Host for a period of time can also save some money for customers who know their workload and can commit for a period of time to have stability. There are many more options such as spot instances that are super cheap but can be taken away if the bid goes higher.

Storing data:

Elastic Block Storage is a network drive for AWS, It acts as a raw disk for storage in EC2. It is like a detachable drive on the cloud. You can use EBS volumes and attach them to any instance and start using them instantly. In the case of instance failure, the data stored in EBS is secured and can be detached from the instance and used somewhere else.

As EBS acts as an independent entity it can be detached from one instance and can be attached to another. EBS volumes are locked to an Availability zone and have a provisioned capacity. EBS volumes come in 4 types i.e.

GP2 (SSD)
IO1 (SSD)
ST1 (HDD)
SC1 (HDD)

Instance Store also provides temporary block storage for instances. These are different from EBS as they are physically attached to the Host instance, this comes with its pros and cons.

Pros being:

As they are physically attached to the host instance it has better I/O performance.
Instance store can be used as a Buffer or a cache.
The data stored in the Instance store is persisted during reboots.

Cons being:

On stop or termination, the instance store is lost along with the data stored in it.
You cannot resize the instance store.
If you need to back up the data in the instance store we manually need to back it up there is no automated process. In short Instance store is a physical disk with very high ops that cannot be increased in size and has a risk of data loss if hardware fails.

EFS (Elastic File Storage)
Amazon EFS is a network file system that is managed by AWS. It provides scalable file storage. It is said to be infinitely scalable and has so many advantages over EBS and Instance Store. We can configure multiple instances to have a common file system. It also works in multiple Availability Zones, which makes it easy for instances from different Availability Zones to connect to the file system and work from the same data source.

EFS uses the NFS v4.1 protocol. It is a highly available, scalable, and Expensive service i.e 3 times more expensive from GP2 of EBS. We can use security groups to control access to EFS. We can have encryption at rest using the KMS service.

NOTE: It is only compatible with Linux based AMI and not windows.

Distributing load:

EC2 service provides a load balancer for distributing load between multiple instances in AWS. Using a load balancer helps us to manage the incoming request and balance it across a fleet of instances. This helps us ensure that our application's downtime is minimized as much as possible.

We can have multiple downstream instances to balance your load while exposing a single point of access or DNS. This helps us manage failure, Load balancer regularly health checks the instances, and if an instance fails it stops sending traffic to those instances and triggers an alarm. It also provides SSL termination.

There are three types of load balancer in AWS i.e.

Classic load balancer (V1)
Application load balancer (V2)
Network load balancer (V2)

We can also set up a load balancer internally. So that we can load balance internally between the instances. for example, we can have a web tier and an application tier, the web tier sends a request to the application tier. We can set up an internal load balancer to balance requests coming from the web tier into the application tier.

We can also set up stickiness so that the request from one user only goes to a single instance for a period of time to ensure consistency.

Scaling Services:

Selecting the perfect instance for the workload is one of the most difficult tasks for a solutions architect. Even though there is a range of options to choose from, we still can't predict the correct amount of instances required to balance our workload. AWS's Auto-scaling does a commendable job in balancing the number of instances.

Just imagine your server having more workload on Wednesdays and less on Sundays, how will you commission servers to match the workload? No worries, we just need to add our instances into an Auto-Scaling group and define policies on how you want to scale your instances. If you have a predictive workload you can set up a policy saying that increases the number of instances to 5 on Wednesdays or if you have an unpredictable workload you can set up policies based on various parameters such as CPU or memory usage. Say if my instances have more than 80% of CPU utilization increase the number of instances by one.

We can also set up an scale in policy that will ensure that we don't over provision instances that are of no use. So we can set a policy that says remove instances if the CPU utilization is below 20%. In this way, you can automate the process of scaling in and out instances with no manual intervention. Auto-Scaling ensures high availability and when used along with load balancer can help us provide quality service to our customers.

Conclusion:

EC2 is one of the most popular offerings of AWS and there is much more to discuss and learn in EC2.

Prototyping Javascript { }

Ashwani Pandey — Mon, 24 Aug 2020 08:53:42 +0000

Managing memory while writing code is one of the major qualities a developer can possess. Execution environment executes javascript code in two stages, i.e Creation and Hoisting.

Execution context: Creation and Hoisting

Execution context creates a couple of things before actually executing the code. Firstly it creates a Global Object and the outer environment and then sets up memory space for variables and functions which is called Hoisting. Before the code is executed memory is allocated so that the variables exist in memory.
Functions are written along with the code but that's not the case with variables instead a placeholder called as undefined is assigned to the variables and later in the execution phase where the code is executed line by line, the variables are assigned to their respective values. This helps in Dynamic typing or Coercion of javascript, wherein the type of variable is defined during the run time.
So to summarize all variables are initialized with undefined but functions are allocated with memory and therefore can be called even before it is defined. In the case of variables, we will get an undefined value.

 function person(firstname, lastname){
           return "Hello "+this.firstname+" "+this.lastname
      } 
}

In the above example, we have a function that takes in two arguments i.e. first and last name, and returns a greeting. Our javascript objects constitute of various functions like this, and these functions are allocated with memory during the hoisting phase of execution. Mind you, the more number of functions that are written in the object definition, the more memory is allocated to the object, and each time its instance is created.

Function constructors

Function constructors are normal functions that are used to construct objects. The this variable points a new empty object and that object is returned from the function automatically.
Creating a function constructor for the Person object.

function person(firstname, lastname){
    this.first = firstname;
    this.last = lastname;
}

let employee1 = new person("John" , "Doe");
let employee2 = new person("Jane", "Doe");

Now that we extend the properties of the person object we can add new variables on the fly. for eg:

employee1.designation = "Developer"
employee2.designation = "Tester"

Prototyping

Prototyping an object is a method that can be used to add member functions to the object prototype which will make it available to all its extended objects but will save memory as the method is only available in the prototype and not copied to every object.
This helps us to create base objects of sorts and extend their functionality without actually allocating the memory for functions.
for eg:

Person.prototype.getFullName = function(){
    return this.firstname+" "+this.lastname;
}
Person.prototype.greet = function(){
    return "Hello "+this.firstname+" "+this.lastname;
}

This above example will add two methods to the prototype available for the objects.

Javascript leverages this functionality to provide various functions on inbuilt data structures and types. If we closely watch the object definition of an array we can see the functions that javascript provides

In the object definition, we have proto which consists of various functions that a developer can use. When we define an array the functions are not allocated with memory, we can still use the methods.

Built-in function constructors

We can have our own methods which can be added to the prototype of the built-in function constructor. for eg

String.prototype.isLengthLessThan = function(boundary){
    return this.length < boundary;
}

The above method adds a function called isLengthLessThan() to the prototype of string.

Various javascript frameworks such as JQuery leverages these functionalities in jQuery.fn.init to write code that allocates minimum memory and provides tons of functionality to users.

Conclusion

Prototype objects is a way to go for creating objects with tons of functionalities with minimal memory allocation. There are a lot more things we can achieve using prototyping.

Github Actions: Power of CICD

Ashwani Pandey — Mon, 17 Aug 2020 08:27:39 +0000

Github has been improving continuously to provide a better experience to its users. The recent update of Nov 19 Github's actions was introduced. By Introducing Github actions developers now have the power of CI/CD and version control in the same dashboard.

CI/CD is the heart of automation. Github now provides a one-stop solution for maintaining, building, documenting, testing, and deployment of code. It is so convenient to have the code and CI/CD pipelines in the same place. I remember changing tabs to see if the build has passed in Travis CI and having to maintain a separate dashboard. Which with the introductions of travisci.com has made it even more difficult to track. My old repository builds are not even automatically imported from .org to .com. And there were other issues like ghost repositories build whose name has been changed or deleted.

Github's actions is also a better solution than Jenkins's pipeline. I would prefer an automated and freely available hosted server for my CI/CD pipeline than a fully user-managed installation on my local computer.

So what is Github action

Github is more than CI/CD it is a general-purpose workflow system for building and testing or doing pretty much anything you want to do with your code after you push it to the repository.

We can automate and execute different workflows right from our repository with Github actions.
Being an opensource project you can find pretty much everything as an already implemented repository either by Github or some other third party repository. If we want to deploy to AWS there is a workflow for that. If we want to publish to GitHub pages there is a workflow for that. We can even run a CRON job, for instance, so if you want to update your static webpages every two hours? there is an implementation for that too.

What is a workflow?

So the workflow is what makes all of this possible. It is the string that holds all the beads together. We can compare a workflow with Spring batch, It made up of different jobs which in turn has actions. We can choose to create a workflow of our own or can use an already existing workflow provided by Github or any other third party repository.
A single workflow can consist of various different jobs that can be executed parallelly or one after the other based on the configuration we define. So if we want to deploy our application after a successful build we can have the deploy be dependent on the build job.

#Build.yml

name: Build using Github Workflow   # Specifying the name of the workflow

on:                                 # This acts as the trigger 
  push:                             # and lets GitHub know when to trigger the job
    branches:
      - master
  pull_request:
    branches:
      - master

jobs:                               # This lets us define the various 
  build:                            # jobs we need to execute for a successful build
    runs-on: ubuntu-latest          # Specifing the image to run our code on
    strategy:
      matrix:
        node-version: [10.x, 12.x, 14.x] # Specifying the runtime
    steps:                          # Steps are actions that are to be performed
      - uses: actions/checkout@v2
      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v1
        with:
          node-version: ${{ matrix.node-version }}
      - run: npm install --production
      - run: npm ci
      - run: npm run build --if-present
      - run: npm test

Above is one of a typical workflow file that defines how we want to build our application. For Github to automatically detect and start building our application we need to put our configuration file in .github/workflows directory

So what can be done using Github Workflow

Github workflow provides the developer with an arsenal of events to have a trigger on. We can trigger a job on not just push or pull requests but pretty much anything event in the Github ecosystem. This level of flexibility provides us with tremendous potentials. We can have a job to treat a new contributor, we can check for stale issues and pull request, We can build using macOS, Windows, and Linux simultaneously as parallel build jobs.

The UI of Github actions is a piece of art, we have live streaming logs that are color-coded and have emoji support as well, yes you heard it right 'emoji' support. We can search in the entire logs for errors and keywords. Suppose you have an error in the logs, now if you want to share that error you just need to copy an already provided URL for the exact line of log containing the error. Now no more "line number 12 of blah blah", just copy the URL and share it to friends, peers and it will take you to the exact line where the error is.

We can also generate badges saying build passed or failed and use it in our readme file to let other contributors know.

A good thing about these workflows is that they are nothing but repositories that can be referenced from anywhere, which means no pre-configuration is required to use actions present somewhere in the whole Github ecosystem. We just need to reference it and done we can extend the functionality.

Now if you want to deploy your application on cloud, Github has support for almost every cloud out there and actions to take help from. We can always create our own actions from scratch as well. Now what about my credentials, don't worry now every Github repository comes with a secret store where you can secretly store your credentials and reference it in the workflows. I mean what else do we need. By the way, these builds are absolutely free for all open source projects.

Conclusion

Having said that, there is so much more that Github's actions have to offer. And the amount of extensibility it provides is commendable.

Docker in a nutshell

Ashwani Pandey — Fri, 14 Aug 2020 05:45:01 +0000

Docker has made the life of a Software engineer easy. The whole "It was working on my machine" problem has a one-stop solution. Docker has introduced a way of standardizing the isolation of software into a container. This isolation helps developers to define dependencies and have predictable behavior of the developed software.

For many developers today Docker is a defacto standard to build a containerized application. But what makes Docker so good at what it does. Docker uses virtualization to accomplish the isolation of software from other processes.

Let's see how OS runs on the system:

The kernel is the central module of an Operating System. It is the part of the operating system that loads first and remains in the main memory. Processes running on the system communicate with the kernel that in turn talks with the hardware.

Processes talks using System calls. A System call commonly knowns as syscalls is a programmatic way in which a process or a computer program requests a service from the kernel of the operating system on which it is executed. Using Namespaces we can segment out harddrive to have different versions or isolated resources per process or a group of processes.

Docker leverages these techniques to isolate the software from the underlying hardware. Docker works by making images of the software in a container with the required amount of dependencies and hardware, required to run the software not more or less. Images are basically a snapshot of the filesystem, a Startup command, and the required dependencies in a closed and concealed container providing developers to expect predictable behavior of the environment the software is running in.

One can create a Docker image using a Dockerfile. Dockerfile is a set of instructions that leads to the creation of a Docker image and later on, can be used to create any number of Docker containers to run the application. As the set of instructions remains the same after the creation of the image it's behavior remains unchanged.
Building a Doker image is fairly simple if one knows what is required for the piece of code to run perfectly. One can follow the steps and can create Docker images with ease.

What is a Docker image

Docker image is more like a blueprint for creating containers of sorts. We can compare it with java classes only with memory allocated to the images on the hard drive. These images act as a template or basis to create a docker container which actually runs the application.

How to create a Docker Image:

#Dockerfile 

FROM node:alpine     # Getting the base image

WORKDIR "/app".      # Specifying the working Directory

COPY package.json .  # Copying the required files

RUN npm install      # Installing the Dependencies/ Running some configuration

COPY . .             # Copying the rest of the code

CMD ["npm", "start"] # Specifying the Startup command

Getting the base image

It is the most important step of the docker image creation process to define the base image. This helps us to leverage the benefits of using an already built image of sorts. In the above example, I have used a Node image to start with. The node image is pulled from a public repository called the Docker Hub. By using a node image I don't have to do the manual work of installing node into my image it is preconfigured by the node image provider.

The FROM command takes in the name of an image and the tag if we want to specify. By default, if not specified the docker daemon pulls up the latest version of the image available. We can specify different tags that are available to us on the Dockerhub. The syntax is:

FROM <image_name>:<image_tag>

Alpine Linux is a Linux distribution built around musl lib and busybox. The size of an Alpine image is about 5MB and has access to a package repository making it a great image base for utilities. We could have used an Alpine image as a base image and then installed node into it, but it would increase the number of lines we. need to write in the docker file.

Specifying the working Directory

Specifying a working directory is such a handy tool to avoid conflict between the filesystem snapshots file names and user-specified file names. A working directory can be specified at any point in time in the Dockerfile, after which every command we specify for example COPY or RUN or CMD will be executed only in the specified working directory.
Suppose we have a file named lib in our application which is very common and if we copy the code into the root directory of our docker image the lib folder will override the filesystem's lib folder creating conflict and unexpected behavior. It is recommended to specify a working directory to separate application files and fS files.
We can specify a working directory using this syntax:

WORKDIR /app  #Or any folder name you want to give

Copying the required files

While copying the files into the docker image we can directly copy the whole project into the image or we can copy only the files required to install the dependencies into the image FS.
We might not understand the signification of this step first time, but when we rebuild the image docker uses the cache to rebuild the image and only executes the steps after something is modified.
We generally don't change the dependencies and therefore it is not required to install the dependencies every time we rebuild the image. Although if we copy the entire project it will force docker to reinstall dependencies as the code we write changes.
It is wise to copy files like package.json beforehand and install the dependencies as they change less frequently.

We can use the COPY command to copy the files

COPY <file in the system> <copy location>

"." is a wildcard that can be used to copy all the files in the current directory.

Running configuration

Using this step we can certain commands in the docker image such as installing dependencies or getting something from the internet. This step helps us to do anything in the image which we would generally do in our local machine to make sure that our program runs properly.

We can make use of RUN command to run the configuration or any other command such as npm install.

RUN npm install

Specifying Startup command

This step is a crucial step for the creation of a docker image as it specifies the default command which is to be used while creating and starting the container. The Startup command is what docker runs when we create a container out of a Docker image and try to run it. We can specify the startup command using CMD command.

CMD npm start

This can be anything such as java -jar myjar.jar if we want to execute a jar as our primary command

Command to create a docker image from a Dockerfile:

$ docker build -t <tag> <build-context>

We can create a docker image using the above syntax in the terminal. The -t flag helps us to specify tags to the created image for later use. We can start the container using the tags we specify while building the image.

Command to run a Docker image

$ docker run <image_id>

$ docker run <tag>

When we build an image in docker we get back an image id which can be used to fire up a container. we can also use the tag we used while building the image. If the Docker daemon doesn't find an image with the specified tag, It will look into docker hub repo for the tagged image. If an image is found on the Dockerhub docker will pull that image and start a container from it.

Conclusion

This is just a gist of what Docker has to provide. This blog is just for getting started with Docker. There is a lot more that we can do by leveraging the features and tools provided by Docker.

Spring JPA: Under the covers

Ashwani Pandey — Mon, 10 Aug 2020 03:29:20 +0000

Spring Data JPA

In the early days, the DAO layer used to consist or still has a lot of boilerplate code which makes it cumbersome to implement. Spring Data JPA, part of the spring-data framework, helps to reduce this boilerplate code and makes it easy for a developer to focus on what's really important.

Reduced boilerplate means reduced code and artifacts to define and maintain. Spring JPA takes this to another level wherein you can replace the DAO layer with a configuration in a property file. The level of abstractions JPA provides helps the developer to only have an interface artifact to maintain.

To start working with Spring JPA a DAO interface must extend JPARepository. Just my extending this interface the developer has tons of methods already implemented and ready to use.

public interface FooRepository extends JpaRepository<Foo, Long> { 
}

But how does Spring JPA work!

There are many ways you can leverage the power of Spring JPA, We will discuss Query creation here.

There are two ways to implement query creation:

Automatic Custom Queries
Manual Custom Queries

1. Automatic Custom Queries:

When we extend the JPARepository Spring scans every method in the interface and tries to parse it to generate queries. It stripes the prefixes such as find...By, read...By, count...By, query...By, get...By, from the method, and starts parsing the rest of it.

public interface FooRepository extends JpaRepository<Foo, Long> { 

    public Optional<Foo> findByName(String name); 

}

The first 'By' acts as a delimiter to indicate the start of the query. We can add conditions on the entity properties and concatenate them with 'And' or 'Or'. we can also add a Distinct clause to set a distinct flag.

public interface FooRepository extends JpaRepository<Foo, Long> { 

    public Optional<Foo> findByFirstnameAndLastname(String firstname, String lastname); 

    public List<Foo> findByFirstnameOrLastname(String firstname, String lastname);

    public List<Foo> findDistinctByLastname(String lastname);
}

" Expressions are usually property traversal combined with operators that can be concatenated. We can combine properties expression with 'And' and 'Or'. There are other operators such as 'Between', 'LessThan', 'GreaterThan', 'Like' for property expression." :- Spring Docs

The above line defines the automatic query creation perfectly.

Property Expression

Property expression can only refer to a direct property of the managed entity. At query creation time, you already make sure that the parsed property is a property of managed domain class

2. Manual custom Queries

If we still wish to write some custom query to refine our result and to achieve the desired set of records which cannot be done using the Automatic query creation we are free to write custom JPQL queries.

We can define the custom queries using the @Query annotation.

public interface FooRepository extends JpaRepository<Foo, Long> { 

    @Query("SELECT f FROM Foo f WHERE LOWER(f.name) = LOWER(:name)")
    Foo retrieveByName(@Param("name") String name);

}

Once we use the @Query annotation the method name doesn't
matter as the method name won't be parsed.

Conclusion

Thanks to the Spring Data JPA team we can implement the DAO layer effortlessly and focus on what matters. This blog is just a gist of what Spring Data JPA has to offer, there is a lot more we can do with the JPA implementation