DEV Community: Danial Malik

How do you profile a node application?

Danial Malik — Mon, 22 Jul 2019 16:41:49 +0000

I have a forum application, implemented in node and express, integrated with another project and is causing some serious latency issues for some routes and operations. I want to figure out the pieces of codes which are the bottlenecks for these slow operations and want to do some profiling. Is there any tool that could help me with that?
Right now I have written a middleware that calculates the total time for a request but It would be better if I could get something like a stack-trace along with the request URL, something like what django-silk provides for django applications.

A Brief Guide to Clean Code: Functions

Danial Malik — Sun, 21 Jul 2019 08:40:46 +0000

Hi guys,

This is my second post in the series "Clean Code". In my first post I talked about how to name variables in a "clean" way. Now I am going to talk about "Clean Functions" and just like last time, these points are derived from the book "Clean Code" by Robert C. Martin.

Below are some key points that should be kept in mind when writing functions. So let's start.

Functions should be small !!!

This is the first rule for writing functions. The second rule for writing functions is that they should be even smaller. Small functions are easy to understand and their intention is clear.

Below is a sample scrapy code written in python.

def get_product(html_response):
   title = html_response.css('.title').html()
   price = html_response.css('.price').html()
   try:
        price = int(price)
   except ValueError:
        price = None

   sizes_html = list(html_response.css('.product .size').html())
   sizes = [parse_size(html_size) for html_size in sizes_html]
   # ...and so on

we could divide same logic into smaller functions like this:


def get_product(html_response):
   title = get_title(html_response)
   price = get_price(html_response)
   sizes = get_product_sizes(html_response)
   # ...and so on

def get_title(html_response):
   return html_response.css('.title').html()

def get_price(html_response):
   price = html_response.css('.price').html()
   try:
        price = int(price)
   except ValueError:
        price = None
   return price

def get_product_sizes(html_response):
   sizes_html = list(html_response.css('.product .size').html())
   sizes = [parse_size(html_size) for html_size in sizes_html]
   return sizes

In this way it is easier to debug and refactor the code when their is some change in target website.

Block and indenting

The rule of keeping the functions small also implies that we keep the code blocks inside if, else and while statements minimal, preferably up to one line and that one line should be a function call. Similarly, we should avoid nesting such structures inside functions by moving the nested logic into another function.

Do one thing! (Single Responsibility Principle)

FUNCTIONS SHOULD DO ONE THING. THEY SHOULD DO IT WELL. THEY SHOULD DO IT ONLY.

One function must do only one thing. If a function is doing more than one thing, the function should be split up into more functions.
This can be illustrated with the help of an example below:

def generate_weather_report(date):
   data = weather_client.get_date(date)
   if not data:
       logger.error('Error!')
   else:
       report_data = {}
       # logic to get required data into report_data dict.
       store_into_db(report_data)

This function is clearly doing more than one tasks.

Get data from web.
validate data
generate data
store data into db

This could be refactored as following code:

data = get_weather_data(date)
is_valid = validate_data(data)
if is_valid:
   report_data = generate_weather_report(data)
   store_into_db(data)
else:
   logger.error('Error!')

def get_weather_data(date):
   return  weather_client.get_date(date)

def generate_weather_report(date):
       report_data = {}
       # logic to get required data into report_data dict.
       return report_data

In this way our function generate_weather_report is doing only one thing.

Sections

Similarly different logical sections in a function such as initializations, declarations, validations, and the actual logic should be divided into functions as well.

Switch statements

Please note that there are no switch statements in python. The example below is for only explanation purposes.

It is very hard to make small switch statements as the purpose of switch statements is to do N things.
Though in some cases switch statements can be avoided and SHOULD be avoided.
For example:

def month_index_to_string(index):
   switch index:
       case 1:
           return "January"
       case 2:
           return "February"
       default:
           return "INVALID"
       ...

can be written as:

MONTHS = {
   1: "January",
   2: "February",
   ...
}

def month_index_to_string(index):
   try:
       return MONTHS[index]
   except KeyError:
       return "INVALID"

Use descriptive names

The intention of the function should be clear from its name and its implementation must not deviate from its name. Do not afraid to use long function names. Ideally function names should start with a verb e.g. get_employees_data, delete_employee etc.
You can find more details on function names in my first post.

Function Arguments

Ideally, a function should have zero arguments (niladic functions). Next comes one (monadic), then comes two (dyadic) and finally three (triadic) arguments functions which should be avoided whenever possible. More than three-argument (polyadic) functions must not be used if possible.
Several arguments could mean that function is doing more than one thing. It is also hard from a testing point of view to test the functions with many arguments as it would be hard (or nearly impossible) to test all combinations of arguments.
If you have to pass a lot of data to the function, try sending that data through an object.

def evaluate_person_salary(person_id, hours_worked, hourly_rate, overtime_rate)
   #logic

This is absolutely evil. It should be written as:

# person_id, hours_worked, hourly_rate, overtime_rate are data members of the person_data object.
def evaluate_person_salary(person_data, hours_worked):
   #logic

If in some scenarios you cannot group the arguments into a single object, you are doing something wrong and that function should be split up.

Flag Argument

A flag argument is horrible and should never be used. Passing a boolean into a function means that function is doing more than one tasks which is a clear violation of the SRP principle.

Arguments Lists

Sometimes we want to pass a variable number of arguments to a function e.g. "{}:{}-{}".format(date, person, message).
If these arguments are treated identically then these are considered to be a single argument (List in this case).

Have No Side Effect

One function must do only one thing with no other side effects. Sometimes we think that our function is doing only one thing but actually it is doing more than one thing and those other things are its side effects. consider this example below:

def check_password(username, password):
   if authenticate_user(username, password):
       return True
   else:
       return False
...
def authenticate_user(username, password):
   if User.objects.get(username=username).check_password(password):
       login(username)
       return True
   return False

Here it seems that check_password is doing only one thing, checking the password, but when we see at the definition of authenticate_user we come to know that it has logged in the user also.

The Stepdown Rule

The author's stepdown rule is very helpful for writing easily readable code. According to this rule, you should write functions in such a way that function calls should be followed by their definitions in the code below.

In this way, while reading a code file, when you see a function call you don't have to scroll up to find its definition because you will find its definition down below while reading through the code at a constant pace.
This approach is illustrated below in next section's example.

DON'T REPEAT YOURSELF

This is one of the most important principles of programming. You should never repeat yourself. If you have written a class function that is needed elsewhere too, move it out of class and make it a utility/helper function. If the same function is needed in two or more classes, make a parent class with that function and inherit other classes from it.

I had to use this approach quite recently when I was writing some test cases for some project. The setup process for the test cases was mostly the same so I decided to make a base class with the setUp function and then inherited other test cases from it. In this way, I didn't have to repeat myself for all the test cases.
An abstract code of the setUp function is described below. You will notice the stepdown approach too. You don't have to move up and down again and again to understand the code.

class SampleTestsBaseClass(TestCase):
   """Base class for all test cases of "Sample" module."""

   def setUp(self):
       """Setup all test data required for testing any of the test cases. """
       super(SampleTestsBaseClass, self).setUp()
       self.topic = self._create_topic()
       self.course = self._create_course()
       self.team = self._create_team(self.course.id, topic_id=self.course.teams_topic['id']))
       self.user = self._create_user()
       self.team_membership = self._create_team_membership(self.team, self.user)

   def _create_topic(self):
       """ Return a topic dict. """
       topic = {u'name': u'Topic', u'description': u'The best topic!', u'id': u'0', 'url': 'example.com/topic/0'}
       return topic

   def _create_course(self, **kwargs):
       """ Create and return a course with test data """
       course = CourseFactory.create(kwargs)
       return course

   def _create_team(self, course_id, topic_id):
       """ Create a CourseTeam for provided course_id and topic_id
       """
       team = CourseTeamFactory.create(
           course_id=course_id,
           topic_id=topic_id,
           name='Test Team',
           description='Testing Testing Testing...'
       )
       return team

   def _create_user(self, username, password, **kwargs):
       user = UserFactory.create(username=username, password=password, **kwargs)
       return user

   def _create_team_membership(self, team, user):
       """Create a team membership object that is responsible for joining of a user in a team.
       """
       membership = CourseTeamMembershipFactory.create(team=team, user=user)
       return membership

# Now inherit the test cases from the above base class.
class Module1TestCase(SampleTestsBaseClass):
   ...

class Module2TestCase(SampleTestsBaseClass):
   ...

Now I can inherit all other related test cases with this BaseClass and won't have to repeat the setUp code.

Conclusion

Functions are the basic building blocks of any system. These are verbs of the system while classes are the nouns. This post was about writing the functions in such a way that those are easy to read, to understand, to maintain and to refactor. Functions should be implemented while keeping the DRY (Don't Repeat Yourself) principle in mind. By following the above principles and practices your functions will be descriptive, clean and beautifully organized.

A Brief Guide to Clean Code: Naming Conventions

Danial Malik — Wed, 19 Jun 2019 06:54:56 +0000

Hi guys,

Recently I started reading Clean Code by Robert C. Martin. So far I think this is a great book and every programmer should read it at least once or at the very least the starting few chapters as in those chapters the author tries to highlight the very basic and fundamental coding practices which should not be ignored ever.

In this post, I will try to summarize the second chapter of the book in my own words and will try to keep things as simple as possible. So let's start :)

The problem

"Naming" is a very important step of any process. We put a lot of thought into naming our children because we understand its importance. But when we do the same for our code, we don't have enough time for it. We either think that we already know the best naming practices or we just don't think that it's worth our time. Bad variable names might seem okay in the start but when the code base is getting bigger with many more developers working on it, it just gets so much difficult to manage and coordinate our efforts. Our code is our creation, therefore, we ought to name it properly at the very least.

What to do?

Intention of the variable should be clear from its name.

If you have to write a comment to make the purpose of the variable clear, rethink the name.
In the old times, we had smaller (very smaller) RAMs and thus had to think very carefully while giving variables big names. But that's not an issue now, is it?. Then why give variables vague names like a, n1 etc.

consider this piece of code

    getData(plist):
        ep = [] // expired person accounts
        for p in plist:
            if p.expiry_date >= datetime.now()
            ep.push(p)
        return ep

and compare with

    get_expired_persons_list(persons_list):
        expired_persons_list = []
        for person in persons_list:
            if person.expiry_date >= datetime.now()
            expired_persons_list.push(person)
        return expired_persons_list

Do not use variable names which could misguide other developers.

An example would be naming a collection of Person objects persons_list when it is not actually a List. This name could misguide other developers towards thinking that it is a List type object. Therefore naming it persons_collection would be a better choice.

Variable names should be meaningfully distinct

We often tend to name variables in a way that we could only get rid of compiler errors due to same variable names. An example would be to name variables PersonData and PersonData2. In this example, the purpose of the variables is not clear.
Similarly if you have written a Person class with methods like get_person(), get_person_info(), it would not be easy for other developers to call the correct method for the desired purpose. A developer can easily get confused about the functionality of any of these methods.

Variable names should be pronounceable.

Our brain can remember the variable names easily if they are pronounceable and we can avoid the unnecessary hassle of searching for the variable name in previous code to reuse it. An example would be using names like this:

class Person(object):
    ...
    get_age_year_month_days()
    get_last_modified_year_month_days()
    ...

instead of

class Model01(object):
    ...
    get_ageiymd() // get age in year, month, days
    get_lmdiymd() // get last modified in year, month, days
    ...

Searchable variables and constants should be used.

Suppose you have a working project. There is a requirement for every user to be at least 18 years old. You have implemented this in the following way:

def create_person(data):
    if data['age']['year'] < 18:
        return None

similarly, you have used this number 18 in other places where this was required in any way.

Now the requirement is changed and the user must be at least 15 years old to create an account. What would you do to now? Search for every occurrence of number 18 and decide if this is the number used for our purpose and then change it everywhere.

Or you could have simply used a constant MIN_AGE_REQ=18 or MINIMUM_AGE_REQUIREMENT=18(which is more meaningful and searchable). Then you would have to simply change the value of this constant to satisfy the new requirement.

Class names should always be nouns.

Class names and objects should always be nouns like Person, Car, Record and so on. Avoid generic and language-specific technical names like Info, Manager and Controller.

Method names should have a verb

Method names should have verb or verb phrase names (preferably in the start) like get, set and delete.

Decide a standard for names serving similar purposes.

Say your project has lots of classes in it with many developers working on them. You decide to use the following convention for setters and getters:

class PersonUpdateForm(object):
    ...
    get_age()
    set_age()
    ...

while another developer might do things like this:

class UpdatePersonForm(object):
    ...
    retrive_age()
    update_age()
    ...

This causes inconsistency in the project and difficult to search for desired objects and methods.
Decide ahead of starting the development that what kind of conventions are you going to follow and have every developer follow those.

Use context with variable names wherever is necessary.

If you have a variable named state inside a class MemberAddress then it is clear that the state belongs to a MemberAddress object. but outside the class, if the variable state is used then it may not be clear for other developers that the variable state belongs to what object. In that scenario using the variable name, member_address_state would be more meaningful.

Conclusion

We should never ignore the importance of naming practices as bad variable names would become a big hurdle in future development plans. Changing the codebase at that point is unnecessary overhead and can be avoided in the very beginning by following good practices.

This is all from my side for now. I will write soon about the later chapters of the book.

Happy coding!