DEV Community: Ranvir Singh

How does Django validate passwords?

Ranvir Singh — Mon, 21 Sep 2020 11:21:34 +0000

This blog was first posted on my blog.

A few days ago, I was working on one of my old Django projects. It was running an old version of Django and I wanted to keep it updated with the latest changes of the framework.

So to check the admin site, I tried to create the superuser after connection to the local database.

python manage.py createsuperuser

When I passed a password similar to the username it failed saying, The password is too similar to the username.

This error triggered me to check the working behind this password validation feature.

The first thing that I looked into was the manage.py file itself which in turn was importing and executing a method called, execute_from_command_line.

I traced it back and found a package commands containing everything that I wanted to know. This directory had two files.

1. createsuperuser.py
2. changepassword.py

The changepassword command

Since I had never used/ heard about the changepassword command, I thought of trying it first and to my great pleasure, it worked. You have to pass the username as the first argument.

python manage.py changepassword username

Sometimes you find gold when you read the code, right? 😍

Now let's get back to the business and look into the createsuperuser command class in more detail.

Fetching the correct database

If you have been using Django for some time, you would know that Django allows you to change a lot of things depending upon the settings you define.

This also includes using some random model as your base User model. This is the first thing that the superuser creation __init__ constructor method checks for.

Creating the superuser without interaction

You can use a version of the command that allows you to create the superuser without any interaction.

python manage.py createsuperuser --username ranvir --email abc@abc.com --no-input

Although the user created using this process will have no password. We can create the password either using the changepassword command or the admin panel.

Required fields and interactive mode

For the default User model, email is the only required field but you can change that by changing your REQUIRED_FIELD setting as well.

In the interactive mode( which is the default mode as well), the first thing that the prompt asks you to fill, is the username.

Django tries to smartly suggest the current system username as the default username. (Just Wow)

It won't suggest the system username if it is already taken. (That's AI for me 😂)

After the username, you have to fill in the required field which is the email field for the default User model.

Finally, you have to fill in the password field.

The validate password method

Sorry for keeping you waiting this long before jumping onto the real reason behind the post.

The validatepassword is the function that is used to validate the password provided by the user.

Again, we can configure all these validators as well, if these different password validation doesn't work for you, go forward and remove the classes from your settings file.

These are the default validators.

AUTH_PASSWORD_VALIDATORS = [
    {
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
    },
]

If we use the default validators, then the password,

Should not be similar to username, first_name, last_name and email. It also checks for the similarity using SequenceMatcher. It should be less than 0.7 similar which you can customize. (Told you, it's AI)
Should be greater than 8 characters.
Should not be in the list of common passwords. The list of common passwords is in the file, common-passwords.txt.gz. It contains a list of around 20000 common passwords which you should not use.
Should not contain all numeric characters.

I would suggest keeping the basic configuration intact for the password handling. You can use your own validations on top of it as well.

So, that's it for this time. Till next time.

Achieving asynchronous behavior using asyncio in Python

Ranvir Singh — Tue, 01 Sep 2020 06:23:03 +0000

This post was first published on my blog.

A lot has changed since I last published my post on handling long-running async tasks in Python using celery. Last time we used it to run a few async tasks to fetch data from some service which took around 1-5 minutes.

Although I will still use celery for these types of long-running tasks, there is a subset of tasks that are better handled inside the execution thread itself. Today we will discuss a way of handling such operations.

To the people who know me, I switched to a new role which allowed me to write Node.js over a year ago and I have been working on it since then. During my time working and building APIs using it, I have started falling for the way in which Node handles Async behavior and long-running tasks.

The idea is simple, if the task is waiting for some IO, we can run some other task in the mean-time. Only the part that needs it, will wait for it.

I wanted to use the same concept in Python. Fortunately, in Python version 3.4, they have introduced asyncio for the same purpose. In this post, we are going to talk a little about it and try to understand its importance.

By the end of the post, you will be able to understand the importance of async functions and will be able to start using them into your codebase.

Prerequisite

I am using Python 3.8 for this tutorial and you might want to use the same to run and try out the examples. You can try them online as well.

Basic async function in Python

This is how we write a basic async function in Python.

import asyncio

async def func1(a):
    print(f"started func 1: {a + 1}")
    await asyncio.sleep(1)
    return a + 1

asyncio.run(func1(1))

Response

started func 1: 2
2

You have to start by importing the asyncio module.

Async functions are defined just like a normal function, you just have to add the keyword, async. If you want to wait for the execution of some coroutine, you have to use the keyword, await.

Finally, you can run the function using asyncio.run method.

Anytime you use the await keyword inside the async function, the thread will not move forward until we get a response from the called function.

This simple implementation will not help you understand the advantage of using async functions. In the next section, we will talk about the time taken for the execution and there you will really start to see the difference.

Time taken for the full execution

To understand the benefits of using async functions, we will compare the time taken to run the same code in async and sync behavior.

Let's first write the async code.

import asyncio
import time
start = time.time()

async def async_sleep():
    await asyncio.sleep(1)

async def func1(a):
    print(f"started func 1: {a + 1}")
    await async_sleep()
    return a + 1

async def func2(a):
    print(f"started func 2: {a + 2}")
    await async_sleep()
    return a + 2

async def func3(a):
    print(f"started func 3: {a + 3}")
    await async_sleep()
    return a + 3

async def func4(a):
    print(f"started func 4: {a + 4}")
    await async_sleep()
    return a + 4

async def main():
    tasks = (func1(1), func2(1), func3(1), func4(1))
    await asyncio.gather(*tasks)
    print(f"Completed after: {time.time() - start}")

asyncio.run(main())

I have defined 4 different functions doing almost the same thing. Let's assume there is a long-running function called async_sleep which is doing some IO operation. For example: Fetching data from the database.

asyncio.gather() is used to run the tasks passed to it concurrently.

The response after running this code is as follows.

started func 1: 2
started func 2: 3
started func 3: 4
started func 4: 5
Completed after: 1.1852593421936035

The function completes the execution somewhere between 1-2 seconds for every execution. (Ran it 5 times to check).

Now let's try to run the sync version of the same code.

import time
start = time.time()

def sync_sleep():
    time.sleep(1)

def func1(a):
    print(f"started func 1: {a + 1}")
    sync_sleep()
    return a + 1

def func2(a):
    print(f"started func 2: {a + 2}")
    sync_sleep()
    return a + 2

def func3(a):
    print(f"started func 3: {a + 3}")
    sync_sleep()
    return a + 3

def func4(a):
    print(f"started func 4: {a + 4}")
    sync_sleep()
    return a + 4

def main():
    func1(1)
    func2(1)
    func3(1)
    func4(1)
    print(f"Completed after: {time.time() - start}")

main()

The response that we got after executing this was:

started func 1: 2
started func 2: 3
started func 3: 4
started func 4: 5
Completed after: 4.168870687484741

This saved us ~3 seconds for executing 4 functions. Now if we wanted to run 10000 such functions, we would save ourselves ~2000 seconds/ i.e. 35-40 minutes.

That's awesome, right.

Even if we are just running 2-3 such functions in each iteration these small savings can be very important to provide better user experience to your customers.

Now that we have understood the advantage of using async functions, we must know a few more things about it.

Order of execution for async functions

We should understand that while running the functions in async mode, we don't really know about the order in which functions are executed. Let's understand the concept with an example.

The functions will return as soon as the execution is completed.

We can mock the different execution time by sleeping for a random time between 0 and 1 using, random.randint(0, 10) * 0.1.

Here is the full code.

import asyncio
import time
import random
start = time.time()

def random_sleep_time():
    return random.randint(0, 10) * 0.1

async def async_sleep():
    await asyncio.sleep(random_sleep_time())

async def func1(a):
    await async_sleep()
    print(f"completed func {a}: {a + 1}")
    return a + 1

async def main():
    tasks = [func1(a) for a in range(0, 5)]
    await asyncio.gather(*tasks)
    print(f"Completed after: {time.time() - start}")

asyncio.run(main())

The response to the following code is,

completed func 0: 1
completed func 3: 4
completed func 1: 2
completed func 2: 3
completed func 4: 5
Completed after: 0.9764895439147949

Practical Usage

After the release of Python 3.8, which moved the asyncio.run() method to stable API, you can start using it without any problem.

The only issue with moving from a synchronous approach to an async one, is to change the way of thinking about the problems. You have to change the way you think about every little detail. Once you reach that point of thinking in terms of async functions, believe me, you will love it.

How does it work under the hood?

Let's check the type of the async function.

type(main())

# <class 'coroutine'>

So, the async function is of type coroutine.

This GeeksForGeeks article does a very good job of explaining coroutines.

To sum up what they wrote in the post, subroutines are the set of instructions that have one entry point and are executed serially.

On the other hand, Coroutines can suspend or give away the control to other Coroutines, allowing multiple tasks to run at the same time.

Python uses this concept to allow async behavior.

Handling timeout in async functions

It is necessary to have a timeout for waiting for the task to get completed. We are not supposed to wait forever for it to complete. asyncio also provided the ability to add a timeout to the async function so that you can skip the execution before its completion.

A practical application of this can be a case when you are calling a third party API in your application and the third party itself is down. In that case, you don't want to wait for a long time.

You can use the timeout method to solve your problem. See the code for example:

async def async_sleep():
    await asyncio.sleep(2)
    print('Execution completed')

async def main():
    try:
        await asyncio.wait_for(async_sleep(), timeout=1.0)
    except asyncio.TimeoutError:
        print('Timeout error')

asyncio.run(main())

The async function will raise TimeoutError if the coroutine doesn't return before the given timeout is passed.

Response after executing the above code is

Timeout error

There are other awesome methods in the asyncio module.

You can check them at python's official website.

Conclusion

If you want to provide a better and faster experience to your users, you can start using the asyncio module in your application. This will definitely help you find cases in which you can reduce the execution time of your API or whatever else that you are doing.

Just forcing yourself to write in this way for a few months can yield big returns in the future. Would love to hear your thoughts, please reach out using the comments below.

Spring Boot Continuous Integration pipeline using CircleCI

Ranvir Singh — Sun, 31 May 2020 09:36:49 +0000

This post was first published on my blog. Please read, Spring Boot Continuous Integration pipeline using CircleCI.

It is very important to automate the process of running the tests before deploying your code. CircleCI is a platform which can you to create CI pipelines for your project.

If you haven't yet set your test environment, you can do so by following my last post on Spring boot integration tests.

CircleCI will try to run the project according to the configuration specified in a special yml file.

Login to CircleCI and turn on the test for the repository

You can easily log in to Circle CI using your GitHub or Bitbucket account. After that you can turn on the CI pipeline by clicking on Set up Project button for that repository.

In the next step it will ask you about the language of the project. You can choose any and according to your choice it will create a branch with .circleci/config.yml file in the root directory.

After a lot of tweaks to the config file that CircleCI created for me, I ended up with this.

.cirlceci file

version: 2.1

jobs:
  build:
    docker:
      - image: circleci/openjdk:8-jdk
      - image: circleci/mysql:5.7
        environment:
          MYSQL_ROOT_PASSWORD: rootpw
          MYSQL_DATABASE: test_db
          MYSQL_USER: user
          MYSQL_PASSWORD: passw0rd
    steps:
      - checkout
      - run:
          name: Waiting for MySQL to be ready
          command: |
            for i in `seq 1 10`;
            do
              nc -z 127.0.0.1 3306 && echo Success && exit 0
              echo -n .
              sleep 1
            done
            echo Failed waiting for MySQL && exit 1 
      - run:
          name: Install MySQL CLI; Import dummy data; run an example query
          command: |
            sudo apt-get update
            sudo apt-get install mysql*
            mysql -h 127.0.0.1 -u root -prootpw --execute="use test_db;"
      - run:
          mvn test

Here we are using two docker images, Java 8 and MySQL and according to the way circle CI is set up both the images are built on the separate containers. So, we have to wait for MySQL to start before running the project.

With - checkout, we are getting the latest code from the given branch. Then we are waiting for the MySQL to be ready. Eventually, we are running a few SQL commands to connect to the database. Lastly, we are trying to run the tests.

Just push this part of the code and it will run the tests for you.

Test changes that I had to do.

The earlier written tests are good for local, and they work great when trying it on the local. But while running the tests on the CI, we have to test a little more.

Strategy for testing

In the single Test case I am going to test two APIs, one for creating the A School and then checking if the entry was created or not.

Changes required in configuration

Since on CI, we have to create the tables again and again we have to change the following in the application-test.properties file. Also, since we have changed the database name, we can change that configuration as well.

spring.datasource.url=jdbc:mysql://localhost:3306/test_db
spring.jpa.hibernate.ddl-auto=create-drop

Changes required in Test file.

According to the strategy change, we will change the way we write our test as well. This is how our updated test file looks like.

package io.singh1114.springboottut;

import org.json.JSONException;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.skyscreamer.jsonassert.JSONAssert;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.boot.test.web.client.TestRestTemplate;
import org.springframework.boot.web.server.LocalServerPort;
import org.springframework.http.*;
import org.springframework.test.context.ActiveProfiles;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

@ActiveProfiles("test")
@RunWith(SpringJUnit4ClassRunner.class)
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
public class SchoolControllerTest {

    @LocalServerPort
    private int port;

    TestRestTemplate restTemplate = new TestRestTemplate();

    @Test
    public void testGetSchoolData () throws JSONException {
        HttpHeaders postHeaders = new HttpHeaders();
        postHeaders.setContentType(MediaType.APPLICATION_JSON);

        String requestJson = "{\"name\":\"School 1\", \"principle\":\"Mr. Charles\", \"address\":\"California\"}";
        HttpEntity<String> schoolPostEntity = new HttpEntity<String>(requestJson, postHeaders);

        restTemplate.exchange(
                createURLWithPort("/school"),
                HttpMethod.POST, schoolPostEntity, String.class);

        HttpHeaders headers = new HttpHeaders();
        HttpEntity<String> entity = new HttpEntity<String>(null, headers);

        ResponseEntity<String> response = restTemplate.exchange(
                createURLWithPort("/schools"),
                HttpMethod.GET, entity, String.class);

        String expected = "[{\"id\":1,\"name\":\"School 1\",\"principle\":\"Mr. Charles\",\"address\":\"California\"}]";
        JSONAssert.assertEquals(expected, response.getBody(), false);
    }

    private String createURLWithPort(String uri) {
        return "http://localhost:" + port + uri;
    }
}

Firstly, we are sending a POST request to the /school API to add data to the school database and finally we are sending a GET request to get the data for /schools API.

We can confirm that the things are working fine by running the tests in debug mode and stopping once the POST request is made and checking in the database at the same time.

Once the test is done, you can run the same SQL command to see that the table was deleted.

Finally, you can push the code to see the tests running on CircleCI.

You can learn more about the changes in the following Pull Requests.

Circle CI setup PR

Test changes PR

I hope you liked the post. Please share your views in the comment section below. Also, please Subscribe if you want to read more such posts.

Introduction to KNN | K-nearest neighbor algorithm using Examples

Ranvir Singh — Thu, 02 Apr 2020 18:37:04 +0000

This post was first posted on my blog. Please read K-nearest neighbor algorithm with Sklearn and upvote on Reddit and hackernews.

KNN also known as K-nearest neighbor is a supervised and pattern classification learning algorithm which helps us to find which class the new input(test value) belongs to when k nearest neighbors are chosen and distance is calculated between them.

It attempts to estimate the conditional distribution of Y given X, and classify a given observation(test value) to the class with highest estimated probability.

It first identifies the k points in the training data that are closest to the test value and calculates the distance between all those categories. The test value will belong to the category whose distance is the least.

Probability of classification of test value in KNN

It calculates the probability of test value to be in class j using this function

P_r(Y=j|X=x_o) = \frac{1}{K}\sum_{i\epsilon N_o}I(y_i = j)

Ways to calculate the distance in KNN

The distance can be calculated using different ways which include these methods,

Euclidean Method
Manhattan Method
Minkowski Method
etc...

For more information on distance metrics which can be used, please read this post on KNN.

You can use any method from the list by passing metric parameter to the KNN object. Here is an answer on Stack Overflow which will help. You can even use some random distance metric.

Also read this answer as well if you want to use your own method for distance calculation.

The process of KNN with Example

Let's consider that we have a dataset containing heights and weights of dogs and horses marked properly. We will create a plot using weight and height of all the entries.

Now whenever a new entry comes in from the test dataset, we will choose a value of k.

For the sake of this example, let's assume that we choose 4 as the value of k. We will find the distance of the nearest four values and the one having the least distance will have more probability and is assumed as the winner.

KneighborsClassifier: KNN Python Example

GitHub Repo: KNN GitHub Repo

Data source used: GitHub of Data Source

In K-nearest neighbor algorithm most of the time you don't really know about the meaning of the input parameters or the classification classes available.

In the case of interviews, this is done to hide the real customer data from the potential employee.


# Import everything
import pandas as pd
import numpy as np

import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline

# Create a DataFrame
df = pd.read_csv('KNN_Project_Data')

# Print the head of the data.
df.head()

KNN algorithm Head of dataframe

The head of the data clearly says that we have a few variables and a target class that contain different classes for given parameters.

Why normalize/ standardize the variables for KNN

As we can already see that the data in the data frame is not standardized, if we don't normalize the data the outcome will be fairly different and we won't be able to get the correct results.

This happens because some feature has a good amount of deviation in them (values range from 1-1000). This will lead to a very bad plot producing a lot of defects in the model.

For more info on normalization, check this answer on stack exchange.

Sklearn provides a very simple way to standardize your data.

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
scaler.fit(df.drop('TARGET CLASS', axis=1))
sc_transform = scaler.transform(df.drop('TARGET CLASS', axis=1))
sc_df = pd.DataFrame(sc_transform)

# Now you can safely use sc_df as your input features.
sc_df.head()

Test/Train split using sklearn

We can simply split the data using sklearn.

from sklearn.model_selection import train_test_split

X = sc_transform
y = df['TARGET CLASS']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

Using KNN and finding an optimal k value

Choosing a good value of k can be a daunting task. We are going to automate this task using Python. We were able to find a good value of k which can minimize the error rate in the model.

# Initialize an array that stores the error rates.
from sklearn.neighbors import KNeighborsClassifier

error_rates = []

for a in range(1, 40):
    k = a
    knn = KNeighborsClassifier(n_neighbors=k)
    knn.fit(X_train, y_train)
    preds = knn.predict(X_test)
    error_rates.append(np.mean(y_test - preds))

plt.figure(figsize=(10, 7))
plt.plot(range(1,40),error_rates,color='blue', linestyle='dashed', marker='o',
         markerfacecolor='red', markersize=10)
plt.title('Error Rate vs. K Value')
plt.xlabel('K')
plt.ylabel('Error Rate')

Seeing the graph, we can see that k=30 gives a very optimal value of error rate.

k = 30
knn = KNeighborsClassifier(n_neighbors=k)
knn.fit(X_train, y_train)
preds = knn.predict(X_test)

Evaluating the KNN model

Read the following post to learn more about evaluating a machine learning model.

from sklearn.metrics import confusion_matrix, classification_report

print(confusion_matrix(y_test, preds))
print(classification_report(y_test, preds))

Benefits of using KNN algorithm

KNN algorithm is widely used for different kinds of learnings because of its uncomplicated and easy to apply nature.
There are only two metrics to provide in the algorithm. value of k and distance metric.
Work with any number of classes, not just binary classifiers.
It is fairly easy to add new data to algorithm.

Disadvantages of KNN algorithm

The cost of predicting the k nearest neighbors is very high.
Doesn't work as expected when working with a big number of features/parameters.
Hard to work with categorical features.

A good read that benchmarks various options present in sklearn for Knn

Hope you liked the post. Feel free to share any issues or any questions that you have in the comments on the original article.

Using Google Optimize to show different versions of the same website

Ranvir Singh — Thu, 26 Mar 2020 20:59:13 +0000

This post was first written on my blog. Read why not to use Google Optimizer and How to use if you want to?.

How would you like to show a modified version of the same webpage to diverse users differently?

What we want to achieve in the post is to change the structure of the page dynamically without you making any change to your code( False promise).

Back Story (You can skip if you want)

I recently launched another version of my blog using Gatsby.

So the idea behind creating another version of the blog was to slowly move everything to the new version without really affecting the traffic over my Jekyll blog.

And why I want to move is already there in the my blog using Gatsby post.

Also, I wanted to check which version of the website users like the most.

As the main blog contains most of the posts and does get a fair amount of traffic, I wanted to move a few users from the old website to the new one and check if the performance was improved even a little.

Before jumping into any conclusion, I wanted to validate my theory with numbers.

Performance metric I was looking to improve

The key metric that I was looking to improve was the bounce rate. One of the pages for which I was getting maximum page views was having a high value of bounce rate( 88%).

I settled on the following strategy.

Let,
   x = Bounce rate of the page on Jekyll blog
   y = Bounce rate of the page on React blog

If, y < x - 0.25x:
   move this page to React.

Else:
   No need to move.

This means if the bounce rate of the new website comes out to somewhere around 65%, I will move the page to the new location with applied redirects.

But still, there was a problem, as I was not running my own nginx, I was not able to apply any of the routes forwarding techniques available in nginx.

My only option was to choose another service, I searched online and came across Google optmize.

It was fairly easy to understand and you can get started by using your existing Google account.

Once you are in, it will ask simple things to complete the setup.

The first thing that you can do is to Create an Experience. Google Optimize provides 4 options to choose from.

Before heading in to choose which experience you want to add, you will have to add the Google Optimize chrome extension which will allow you to make changes to the website.

The one that I wanted to use was the Redirect/ Split URL test. You can choose the one that suits your situation.

Read this post for more details to choose from all the options.

A-B Test

A/B testing is a type of testing where you create two or more versions of a single webpage and show modified versions to each user.

Google Optimize allows you to split traffic.

Splitting the traffic means that each user who lands on the targeted webpage will either one of the versions that you have specified according to the traffic split that you have defined.

If you have selected 50-50 split, nearly half of the users will see one result and another half will see the other.

If you try to access the targeted web page, you will always see the same version for some time, no matter how hard you try. It takes some time to start triggering the next version and once that is live, you will see that version for some time( Most probably the traffic is split on the basis of time).

Optimize makes it fairly easy to create two or more versions. It loads the targeted webpage inside the given optimize environment so that you can apply CSS and make required changes on the fly.

Optimize will then store all the versions and wait for the users to land on the targeted page.

Redirect Test

Redirect/Split test was the one that I wanted to choose. As I have already specified what this means, I will skip explaining it in more detail.

You will know when you want to use it when you have two versions of the same webpage on separate URLs.

Multivariate tests

A multivariate test is a type of A/B testing where you makes 2 or more combination of changes deployed together.

I haven't really understood the real use case of this test.

If you have used it before, do let me know.

Personalize Test

Personalization Test will show different versions of your webpage to different people depending on their personality.

You can create separate versions as we do in A/B testing and target different versions with different demographics, devices and many other options.

Turning the test ON

You can turn the test on by adding the optimize key to your Google Analytics string.

    ...
    gtag('config', 'UA-*****-*', { 'optimize_id': 'G**-******'});

Once you are live with your optimize_id change. You will see the traffic being diverted and data available in the Optimize account.

Although it takes some time to compute the real data and come up with proper statistics.

You can keep running a redirect test for 3 months.

And after you get the correct result back you can decide what you want to do with your web page.

Some numbers that will stop you from using Google Optimize

I am a big fan of checking Google Page Speed statistics after every addition of dependency in my code.

And as Google Optimize is an added dependency, you might want to check the changes in the behavior of your site.

The changes that I saw were pretty drastic.

**Old website page speed Score without optimize tag**

- Mobile score: 71, web score: 97

**New website page speed scores**

- Mobile score: 86, web score: 99

**Old website Score with optimize tag**

- Mobile score: 55, web score: 95

You can already see that there is a whole lot of difference in the page speed before and after the addition of the Google Optimize tag.

I will leave the decision making step to you.

You can make your own judgment whether you want to use Google Optimize or not. For me most of the visitors are Desktop, so I can totally ignore the mobile visitor and keep it that way. Still, after checking those results, it took the optimize tag away and went back to running my dumb blog.

If not Google Optimizer, what?

I still haven't discovered the solution yet. I will update this when I get a real answer. Also, let me know guys if you get to any.

Anyway if you can afford to have hire a developer and you have the control over your server, you can implement such things with great ease on the server level.

Create react based blog using Netlify CMS and Gatsby

Ranvir Singh — Sun, 22 Mar 2020 19:53:38 +0000

This post was originally shared on my blog. Read create a react blog using gatsby.

Some weeks ago, my colleague shared the Gatsby blog that he created for his personal blog.

He has also written a good post on JavaScript Proxies that you will enjoy reading.

Whenever I see a new website, I immediately head over to Google's pagespeed insight to find out the speed of the page. I was astonished by the speed that the page was able to achieve.

Honestly speaking I was never able to reach to those numbers on the Jekyll blog.

So, I started working on my personal Gatsby blog. This post will contain all the details which I happen to do to make this blog to the stage it is currently at.

Also, currently I don't have the concept of private posts on this blog, I am unable to keep it private when it is clearly not finished. You have to bear with for this.

Final result

The final blog after this series is going to be present on this URL.

Prerequisites

I don't think there are any prerequisites required to start a blog using Gatsby and Netlify CMS, It is fairly easy to set up if you know how to make changes and use basic GitHub.

The basic things required for this specific version of post are:

Domain Name

I already had a domain name which I was already using. It will still work if you don't have it though. 🙏

GitHub Account

To follow this tutorial, you will need to have a GitHub account where the code for the blog can reside. There are other ways to keep the code, but I think GH provides the easiest way to maintain your code.

Netlify Account

You will also need account on Netlify so that you can deploy and

The rest of the stuff we are going to figure out during the course of the post.

Let's begin

Choose a theme

You can choose from a list of themes that are available. Most of them has deploy on Netlify button which will deploy it directly on Netlify and can save the code to your Netlify Account. Next time you make any change to your code, netlify will automatically detect the change and deploy the change.

I choose the basic gatsby-theme-blog. Just click on the deploy button and choose Connect to GitHub the code and it will host the code on GitHub. Finally, choose the name of the Repository that you want to contain the code in.

Making the changes

Once your site is deployed on something.netlify.com, you can clone the code locally and start making the changes.

git clone https://github.com/your_repo/repo_name

For more information on various git commands check this post.

The first thing to change if the gatsby-config.js file. Change siteMetadata as follows

  siteMetadata: {
    title: `Tutorial blog`,
    author: `Ranvir`,
    description: `Basic description of the blog`,
  ...

Push the code and you will see the changes being deployed in your Netlify account. https://app.netlify.com/sites/your_site_id/deploys.

git push origin master

After some time the changes will be available on the current website.

Adding Netlify CMS

Adding Netlify CMS will help you to quickly make changes to your posts and deploy them with a click of a button. All your posts are at a single place.

You can create and edit your posts easily using the Netlify CMS.

For adding Netlify CMS and allowing editing using GitHub Login, you will have to add a static directory to the root of your code.

Add this to the static/admin/config.yml file.

backend:
  name: github
  repo: singh1114/tutorial
  branch: master

media_folder: static/img
public_folder: /img

collections:
  - name: "blog"
    label: "Blog"
    folder: "content/posts"
    create: true
    slug: "{{slug}}"
    editor:
     preview: false
    fields:
      - { label: "Title", name: "title", widget: "string" }
      - { label: "Publish Date", name: "date", widget: "datetime" }
      - { label: "Image", name: "image", widget: "string", required: false }
      - { label: "Tags", name: "tags", widget: "list", required: false }
      - { label: "Description", name: "description", widget: "string" }
      - { label: "Body", name: "body", widget: "markdown" }

You can change the fields using widget guide according to your needs.

After deploying this you can access the website's admin panel on https://web.netlify.com/admin.

For turning on the GitHub login on your website's admin page, so that only you can edit the posts, you will have to turn on the OAuth access using your Netlify account.

Just go to the settings tab > Access Control > OAuth >Install Provider. For installing, you will need auth credentials which you can generate using the GitHub Application page

Add your netlify address as the home page URL and https://api.netlify.com/auth/done as callback.

Once done, you will be able to Login to the admin panel and make changes to the posts directly.

Adding Redirects

As I already told you that I had a domain name and I wanted the website to be hosted on that https://blog.ranvir.xyz and not on something.netlify.com. For this we will need to set up CNAME record in your domain name provider.

You can find more about what record to set with simple google search.

In my case, where I was using a subdomain, I had to use blog as a host and something.netlify.com as value.

After that is done you can redirect, something.netlify.com to blog.ranvir.xyz by creating a redirect file.

Create static/_redirects file and add the following content to it.

# Redirect default Netlify subdomain to primary domain
https://something.netlify.com/* http://blog.ranvir.xyz/:splat 301!

This will create a 301 permanent redirect.

Adding Google Analytics

Adding analytics to your website can be very useful, you can check how many people are looking at your website at every given moment.

You just have to install a basic npm package and add the Google Analytics tracking ID to your code.

npm install --save gatsby-plugin-google-analytics

Now change the code in the config file.

module.exports = {
  plugins: [
    ...
    {
      resolve: `gatsby-plugin-google-analytics`,
      options: {
        trackingId: 'UA-*******-*'
      }
    },
    ...

You can take the tracking ID from the Google Analytics dashboard.

How to use FactoryBoy to create model instances in Django for testing

Ranvir Singh — Sat, 11 Jan 2020 21:57:17 +0000

Original post: Using FactoryBoy to create model objects automatically for testing.

Testing is one of the basic and most important parts of the development process that is needed to be done every time you write or change any part of the code. It gives you the authority confidence in the code that you are writing.

I have already discussed testing in one of my earlier posts.

Writing unit tests for Django models

In this post, I will talk about the way in which we can write tests for models.

For testing models, we need to create sample objects. 😱

It's pretty hard to write the same code again and again to create the same objects.

Today I am going to talk about using FactoryBoy for testing Django application.

The name FactoryBoy is inspired by the Factory Girl testing tool for Ruby on Rails. Without Factory boy, you will have to write a lot of code for the creation of test cases.

For writing tests, you have to spend a lot of time writing the setup method for the tests.

Without the use of factories, this is how your tests will look like:

from django.test import TestCase
from my_app.models import Person

class DataBaseStore(TestCase):
    def setUp(self):
        # Creating all the models   
        Person.objects.create(gender='male', age=20, name='ranvir')  

    def test_something(self):
        # test all the things that you want to.
        pass

On the other hand, it is always nice to separate your setting up method from the testing method. For this sometimes we have to write a long code at the beginning which helps us in the future by allowing us to write smaller code.

We can write factories anywhere but it's always a good idea to write it in a separate file named factories.py inside the root directory of your app.

Write following in the factories.py file inside your root directory.

from factory.django import DjangoModelFactory  
from my_app.models import Person  

class PersonFactory(DjangoModelFactory):  
    class Meta:  
        model = Person  
    name = 'ranvir'  
    age = 20  
    gender = 'male'

Now we can simply call these factories during the test generation and create as many instances of models as we want to. This is a pretty simple case but we really can do a lot more. We can create instances of varies types by making use of FuzzyText and Sequence.

With the use of FuzzyText, you can generate random text making testing more awesome.

Applications of FactoryBoy

Getting random values

We can get random values for the purpose we specify to the Faker method. The values created are far more realistic than any other random methods that we use otherwise.

from factory import Faker

class PersonFactory(DjangoModelFactory):  
    class Meta:  
        model = Person
    name = Faker('name')
    age = 20
    gender = 'male'

Here is a list of available providers by faker library which can be used.

Reproducing random values

Sometimes our tests fail for some random value and not for all of them. So, factory have introduced a concept using which we can create the same value again and again.

from factory.random import reseed_random

class PersonFactory(DjangoModelFactory):  
    class Meta:  
        model = Person
    name = reseed_random('Bill Clinton')
    age = 20
    gender = 'male'

Using dependent values of other fields

There are some cases in which you want to use the values which are dependent on the random values of the other fields. For example, full name can be the combination of first and last name.

from factory import Faker, LazyAttribute

class PersonFactory(DjangoModelFactory):  
    class Meta:  
        model = Person
    first_name = Faker('first_name')
    second_name = Faker('last_name')
    full_name = factory.LazyAttribute(lambda a: '{0} {1}'.format(a.first_name, a.last_name))

Using sequences for unique fields.

With random function, there is a chance that after some time the values will start repeating, and if your test suite has grown out of that threshold size you will have to use sequences in your unique fields.

from factory import Sequence

class PersonFactory(DjangoModelFactory):  
    class Meta:  
        model = Person
    email = Sequence(lambda n: 'person{0}@ranvir.xyz'.format(n))

Foreign Keys

If your field is a foreign key, you can directly use that whole factory altogether.

from factory import Sequence

class PersonFactory(DjangoModelFactory):  
    class Meta:  
        model = Person
    email = Sequence(lambda n: 'person{0}@ranvir.xyz'.format(n))

class SocietyFactory(DjangoModelFactory):  
    class Meta:  
        model = Person
    society_number = Sequence(lambda n: 'Society {0}'.format(n))
    person = factory.SubFactory(PersonFactory)

Testing is also considered as the sign of a good programmer. There are a ton of benefits of testing which were discussed in the post.

I hope you guys liked the idea of the post. We really can do a lot of stuff using this library. Do share your views regarding this post if you have used factories in the past or want to use it in your project. 😇

HTML presentation framework reveal.js and why I am a big fan?

Ranvir Singh — Tue, 31 Dec 2019 11:54:10 +0000

This post was first posted on my blog.

Reveal.js is an HTML presentation framework that can help you to create awesome presentations for your talk or simple knowledge transfer session in your class.

I have created a number of presentations using reveal.js.

Here are the links:

Prologue (How I came to know about reveal.js)

So, during my time at college, I had to deliver a presentation to a group of students and teachers from a special student cell regarding one of the emerging technologies of those times.

My mentor at that time suggested me to use reveal for creating the presentation. I was skeptical in the beginning but as soon as I got the grip of it, I loved it.

I went home and thought that it would be a difficult task but in the end, I ended up doing it just in two hours.

Setting up reveal.js

All you need to run reveal powered presentation in your local system is to have javascript up and running and if you reading this post, you definitely have it running.

All you have to do is to go to reveal.js GitHub page and download the latest version of the code on the website.

Here is the link for reveal.js GitHub repository.

The downloaded file would be a zip file. If in windows, open the file simply by double-clicking it.

If in Linux use these commands in the terminal to open the file. First of all download the unzipper by using this command.

sudo apt-get install unzip

Then use this code to unzip the file by giving the name of the destination folder.

unzip file.zip -d destination_folder

Otherwise, if you prefer, you can fork the code from the https://revealjs.com/

After unzip open the folder and you will find an index.html file inside.

Creating presentations using reveal.js

Open this file in your text editor. Inside the body, tags delete every section.

Now create each section using the beginning and ending section tag. Each section is equivalent to one slide.

Here is the reference of one equivalent slide.

You can do a whole lot of things using reveal.js presentation creator. You can create a style of your own if you want to.

Epilogue (Delivering the reveal.js presentation)

So, there were somewhere around 100-200 students and teachers in the room where I was delivering the presentation. The stage was at a good height and I was going to show a good number of charts in my presentation.

Before jumping on to the stage, I hosted the presentation on to GitHub pages website and created a short link using one of the services out there. Finally, I shared the link with everyone in the room.

Everyone opened the link on their phones and were able to look at the chart more clearly.

What do you use to create presentations for talks?

Ranvir Singh — Thu, 19 Dec 2019 23:20:15 +0000

I personally prefer to use google slides or reveal.js for creating my presentations for any type of talk.

I have seen people using a combination of vim and emacs for delivering their presentations.

Just wanted to know what is your choice for the task.

How to break out of SDE-I role

Ranvir Singh — Tue, 17 Dec 2019 20:01:55 +0000

A few days ago, I posted a new blog on my personal blog.

Breaking out of Software Developer-I role as a web developer. Please head over to the post if you want to read the detailed description.

Here are some of the key points that I want to share here as well. Experienced Devs, please share your checklist as well will add it there.

Learn Database designing
Learn some DevOps
Learn how to debug
Learn to write tests
Learn about scaling issues
Learn a little bit about security
Learn about caching
Learn to review code properly
Maintain your personal websites
Help out new joinings and business guys
Ask for the promotion, when you feel like you are ready to step up.

How to link to your old posts like a pro in GitHub Pages blog

Ranvir Singh — Sun, 01 Dec 2019 09:16:21 +0000

This post was originally shared on my blog. Please read it there for a better experience.

Linking to your old posts is very good for the SEO of your blog and help you not to write the same things again and again. It also helps you to interconnect to your old posts increasing the traffic to your website.

After writing a good number of posts on dev.to, I started liking the options they provide to share the old posts or other dev.to posts in a well-formated way.

I wanted to create the same experience on my personal blog as well.

As I use GitHub Pages ( Jekyll) to build my blogging website, I was sure that it won't be a huge task to do this.

I have a lot of separate widgets on my blog. When someone clicks on any of the posts, all those widgets come together and are shown to the user as a single unit.

Some examples of these widgets are:

Social sharing buttons

The social sharing buttons on my blog pages are included as a separate widget. This helps your readers to share your posts on social media.

BTW, please share this post on any of the social media and help us get more readers.

Read time widget

The read time for any post on my blog is again a separate widget. I use it to show the amount of time it will take to read the post.

Now that we know how widgets work in Jekyll blog, let's try to create something similar to what dev.to have.

Let me know if you want to know more about them in the comments on my blog. I will share the details.

Final result after creating

Let me a link to the social sharing post that I created some time back to show you how the final result will look like.

How we created amazing old post linking widget for our Jekyll blog

The first step was to create the include file so that we keep the structure of our code clean. The only requirement for this is that we need to pass some parameters so that we can find out which post is being linked to.

The only thing which is unique for my blog is the URL. Although I know the URLs can change but this is the only way I can accomplish this task.

Creating the include file

Add the following code to _includes/linked_post.html file.

We loop through all the posts on the site and check which post has the same URL as passed to the include statement which I will share very soon.

Finally, we have just shown the heading and along with the date of the post. We have also added the read time widget and post categories on my blog but for that, we will need to include two more widgets so, for simplicity, I have omitted it in the above code.

The code above would be sufficient to help you create something of your own. Still, if you are stuck somewhere, do let me know in the comments below.

CSS for the widget

I have added the CSS separately in the style.scss file. Here is the CSS code for this. You can add it anywhere.

// old code.
.linked_post_div {
  border: 1px solid #ccc;
  border-radius: 5px;
  padding: 15px;
}

.linked_post_div h1 {
  margin-top: 0px;
}

How to use this widget.

All you have to do is something like this.

Nice right, I hope it helps you to create links to your old posts. Please share this post on social media and let me know in the comments below.

Also, if you found something wrong do let me know. Thanks for reading.

Integration testing of APIs in Spring-Boot( Java) with Junit

Ranvir Singh — Sun, 24 Nov 2019 19:39:20 +0000

This post was initially shared on my blog. Read the post testing in spring boot with JUnit for beginners

Shipping our product without testing is like selling a car without turning on the engine. We have to run a few tests again and again while the development of the product and if those tests are not automated, we will have a hard time testing our application.

I have written a post on the importance of writing tests against your codebase.

This post is in continuation of the last post we shared related to Spring boot in which we shared on how to write Rest APIs in Spring Boot.. Please go through that post if you want to learn on how to create rest APIs using Spring Boot.

In this post, we are going to discuss how we can test our spring boot application from start to end without any mocking.

First of all, we have to add the test libraries to the pom.xml, so that maven can install them for us.

Here is the code to add testing libraries to pom.xml

        ...
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
        </dependency>
        <dependency>
        ...

Using separate config for testing purposes

We don't want to add the data to the same database as of the production one. So, we will start by creating a separate configuration for handling the test cases. Also, its always a good practice to have separate configurations anyway.

Create a new resource file, src/main/resources/application-test.properties and add the following data to it.

Note: We will have to create the database in our local system to make it work.

Now that we have a separate configuration file, we have to export it as an App configuration so that we can use it in the tests itself.

Add the following code to the file, src/main/java/io/singh1114/springboottut/AppConfigurationTest.java

Writing Controller test for Spring Boot API.

Now since we are going to test the application from end to end, let's create a controller test.

Add the following code to src/test/java/io/singh1114/springboottut/SchoolControllerTest.java

Let's go through the file and understand this.

@ActiveProfiles, is used to let the framework know that we are going to use the given profile. As we have already declared the test Profile in the Configuration file, we can use it here directly.

@RunWith and @SpringBootTest is used to start a simple server to carry out the tests using the SpringBootFramework.

Running on a random port

With SpringBootTest.WebEnvironment.RANDOM_PORT we are telling the framework to launch the application on a Random PORT.

Removing any change made to the database after test completion.

@Transactional will help us to clean the database once tests are completed. We don't want to keep adding stuff to the database and want to clean it after every test run.

Creating the API endpoint

    @LocalServerPort
    private int port;

    private String createURLWithPort(String uri) {
        return "http://localhost:" + port + uri;
    }

These are used to create the URL for a given endpoint to hit.

Creating the School Object

    @Autowired
    private SchoolRepository repository;
    ...
        School testSchool = new School(1, "First Location", "Mr. Ranvir", "California");
        repository.save(testSchool);
    ...

This code is used to create the object in the database.

Sending the request and getting the response

    TestRestTemplate restTemplate = new TestRestTemplate();

    HttpHeaders headers = new HttpHeaders();
    ...
        HttpEntity<String> entity = new HttpEntity<String>(null, headers);

        ResponseEntity<String> response = restTemplate.exchange(
                createURLWithPort("/schools"),
                HttpMethod.GET, entity, String.class);
    ...

Checking if the response is equal to the object created

JSONAssert.assertEquals(expected, response.getBody(), false);

Here we are checking if the body of the response is equal to the expected response.

Run the tests

Use can use the following command to run the tests.

mvn test

I hope you liked the post. Please share your views in the comment section below.