DEV Community: Dollar Dhingra

Benchmarking Python JSON serializers - json vs ujson vs orjson

Dollar Dhingra — Wed, 25 May 2022 17:12:15 +0000

Introduction

If you work with a large datasets in json inside your python code, then you might want to try using 3rd party libraries like ujsonand orjson which are replacements to python’s json library.

As per their documentation

ujson (UltraJSON) is an ultra fast JSON encoder and decoder written in pure C with bindings for Python 3.7+.
orjson is a fast, correct JSON library for Python. It is the fastest python library for json encoding & decoding. It serializes dataclass, datetime, numpy, and UUID instances natively.

Benchmarking

I did a basic benchmark comparing json, ujson and orjson. The benchmarking results were interesting.

import time
import json
import orjson
import ujson

def benchmark(name, dumps, loads):
    start = time.time()
    for i in range(3000000):
        result = dumps(m)
        loads(result)
    print(name, time.time() - start)

if __name__ == " __main__":
    m = {
        "timestamp": 1556283673.1523004,
        "task_uuid": "0ed1a1c3-050c-4fb9-9426-a7e72d0acfc7",
        "task_level": [1, 2, 1],
        "action_status": "started",
        "action_type": "main",
        "key": "value",
        "another_key": 123,
        "and_another": ["a", "b"],
    }

    benchmark("Python", json.dumps, json.loads)
    benchmark("ujson", ujson.dumps, ujson.loads)

    # orjson only outputs bytes, but often we need unicode:
    benchmark("orjson", lambda s: str(orjson.dumps(s), "utf-8"), orjson.loads)

# OUTPUT:
# Python 12.502133846282959
# ujson 4.428200960159302
# orjson 2.3136467933654785

ujson is 3 times faster than the standard json library

orjson is over 6 times faster than the standard json library

Conclusion

For most cases, you would want to go with python’s standard json library which removes dependencies on other libraries. On other hand you could try out ujsonwhich is simple replacement for python’s json library. If you want more speed and also want dataclass, datetime, numpy, and UUID instances and you are ready to deal with more complex code, then you can try your hands on orjson

As a lead engineer/manager, what steps can you take for your team to provide better estimations for their tasks?

Dollar Dhingra — Thu, 21 Apr 2022 15:38:53 +0000

As a lead engineer/manager, what steps can you take for your team to provide better estimations for their JIRA stories or tasks before starting the development work?

2021 year in review

Dollar Dhingra — Sun, 20 Mar 2022 11:36:53 +0000

Yes I know, I am late at publishing this. Somehow I missed it in my drafts.

Looking back at your achievements at the end of every year is a great way to boost your confidence and tackle the imposter syndrome :) So, here we go!

1st prize at Remote Hackathon - Participated in the company’s internal remote hackathon with 6 other team members. Checkout more details here
Started writing technical blogs on topics related to python programming and software engineering. I also self-hosted the blog here with my first ever article. Since then I try to write at least 1 blog article each month. This year itself my posts had 20k views.
Delivered talks on following topics at my workplace
- Introduction to ELasticsearch
- Clean Code
- Data Structures in Python
Created an open source telegram bot for covid : covid19-india-resources, repo link
Created a panel for checking vaccine booking slots. Booked 30+ slots for family and friends using this panel, Repo link
Read the following books:
Awesome movies that I watched:
- A beautiful mind
- Parasite
Accepted a job offer from Tata 1mg as SDE-3. We are hiring at Tata 1mg. Checkout the jobs and let me know if you are interested to work at India’s leading healthcare brand.

Bus Factor in Software Engineering

Dollar Dhingra — Tue, 25 Jan 2022 16:41:35 +0000

1. Introduction

Imagine that John Doe works on a project along with 7-10 folks. John started working on this project from the beginning and therefore knows a lot more than everyone else in the project.

These leads to some problems such as:

John is the go-to person for each detail that the team needs. John has a lot of responsibility and workload on his shoulders, making his progress sometimes slow.
Whenever John falls sick and goes on unplanned leave, he is bugged by his team to keep things moving forward in the project.
The organisation finds it hard to replace John and has therefore retained him twice in the past year by raising his compensation twice.

In short, the project has a single point of failure -> John Doe

2. What is Bus Factor?

As per Wikipedia, The bus factor is a measurement of the risk resulting from information and capabilities not being shared among team members, derived from the phrase “in case they get hit by a bus”. It is also known as the bread truck scenario, bus problem, beer truck scenario, lottery factor, truck factor, bus/truck number, or lorry factor.

In simple words, what is the probability of the project failing if John Doe gets hit by a bus and is not able to return to work?

Higher the bus factor higher the risk of failure.

3. Reducing/Preventing the Bus Factor in Software Development

3.1 Documenting

Creating and maintaining documentation can take a good amount of effort from all the stakeholders. But in the long run, it is worth documenting. Documenting includes Product Requirement Documents(PRDs), High & Low-level design documents, changelogs, READMEs, complete ticket descriptions on ticketing tools, Root Cause Analysis documents in case of production fires, etc

3.2 Code Reviews

If done effectively, code reviews can act as an excellent tool to reduce the bus factor, given that everyone in the team gets a chance to review the code. The Engineering manager/team lead should make sure that there are relative code review guidelines set up as per the team so that it is effective enough.

3.3 Meetings

Yes, I know meetings sometimes slow us down and hinder our productivity. Also, some meetings can just be replaced by an email. But, meetings (especially if you are working remotely) can actively reduce the bus factor if done correctly. Meetings like standup, sprint planning/grooming, design discussions, sprint retros etc bring everyone together as a team. Imagine you reading a minute of meeting document vs actually brainstorming in the meeting. Meetings do not replace the documentation. Instead, the documentation complements the meeting process.

3.4 Team Process

More team processes can improve the knowledge distribution in the team like:

Pair programming
Office hours
On-call rotations
Engineering demos
Knowledge sharing sessions

3.5 Positive Work culture

A positive culture means employees are happy in the organisation and a better work environment is created. Following techniques can be applied to improve the work culture and reduce the bus factor:

3.5.1 Mandatory Paid Annual Vacation

Most of the organisations provide paid leave to their employees. The organisations should also encourage people to use these leaves and go on a vacation(for a week at least). This will enable other employees to take care of work in the absence of an employee.

3.5.2 Short Notice/Termination Period

A short notice period means, the organisation making sure that there is less dependency on an employee.

Conclusion

The first step is to talk about Bus factor in your team, even if you are a junior member in the team. Maybe, some of the above methods like meetings, code reviews, documenting etc. are already being used in your organisation for sharing knowledge. You can consider optimising those methods to reduce/prevent the bus factor in your team. For eg, you can consider including the junior members of your team in meetings and code reviews. The members with a centralized knowledge can work on spreading their knowledge in the team so that they can spend their time and energy on something new for the team.

What do you think of Bus factor in your team? Do you have more methods to prevent high bus factor? Feel free to give your feedback!

How to install older Ruby versions(2.x) on Mac M1 without using arch x86_64

Dollar Dhingra — Thu, 20 Jan 2022 16:53:20 +0000

Introduction

After spending hours of searching different solutions on the internet for installing older version of ruby (2.5.x, 2.6.x etc ) on Mac M1, I finally figured out the solution and decided to document it by writing a blog.

Pre-Requirements

Make sure you have following installed on your Mac M1

Update your Mac OS
Homebrew

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

rbenv (Should work for rvm also)

brew install rbenv

Make sure that you have latest xcode command line tools

xcode-select --install

if the above fails, then:

sudo rm -rf /Library/Developer/CommandLineTools
xcode-select --install

Steps to Reproduce The Problem

The problem occurs if you try to install an older version with Ruby.

rbenv install 2.6.2

Error:

- BUILD FAILED (macOS 12.1 using ruby-build 20210707)
- Inspect or clean up the working tree at `/var/folders/.....`

Solution

Try installing the ruby version with this variable RUBY_CFLAGS="-Wno-error=implicit-function-declaration"

RUBY_CFLAGS="-Wno-error=implicit-function-declaration" rbenv install 2.6.2

This should work fine!

Other Installations

After this, install other dependencies normally using brew. For eg: postgresql I installed postgresql@10 which was required for the project.

brew install postgresql@10

The end of success clearly states that you have to do some entries in your bash/zsh file

If you need to have postgresql@10 first in your PATH, run:
  echo 'export PATH="/opt/homebrew/opt/postgresql@10/bin:$PATH"' >> ~/.zshrc

For compilers to find postgresql@10 you may need to set:
  export LDFLAGS="-L/opt/homebrew/opt/postgresql@10/lib"
  export CPPFLAGS="-I/opt/homebrew/opt/postgresql@10/include"

For pkg-config to find postgresql@10 you may need to set:
  export PKG_CONFIG_PATH="/opt/homebrew/opt/postgresql@10/lib/pkgconfig"

To restart postgresql@10 after an upgrade:
  brew services restart postgresql@10
Or, if you don't want/need a background service you can just run:
  /opt/homebrew/opt/postgresql@10/bin/postgres -D /opt/homebrew/var/postgresql@10

Do these changes and then go to your project and set the ruby version for the project:

rbenv local 2.6.2

Check if the ruby version is correct:

ruby -v # should output whatever version you had set in the previous step.

The next step is to install the bundle

bundle install

During the installation, I got an error while installing http-parser. The error message looked like this:

Caused by:
LoadError: cannot load such file -- 2.6/ffi_c

Caused by:
LoadError: cannot load such file -- ffi-compiler/compile_task

(See full trace by running task with --trace)

rake failed, exit code 1

Gem files will remain installed in /Users/Dollar/.rbenv/versions/2.6.2/lib/ruby/gems/2.6.0/gems/http-parser-1.2.3 for inspection.
Results logged to /Users/Dollar/.rbenv/versions/2.6.2/lib/ruby/gems/2.6.0/extensions/-darwin-21/2.6.0/http-parser-1.2.3/gem_make.out

An error occurred while installing http-parser (1.2.3), and Bundler cannot continue.
Make sure that `gem install http-parser -v '1.2.3' --source 'https://rubygems.org/'` succeeds before bundling.

In Gemfile:
  elastic-apm was resolved to 3.15.1, which depends on
    http was resolved to 4.4.1, which depends on
      http-parser

The error message clearly states that the reason

LoadError: cannot load such file -- ffi-compiler/compile_task

I rectified the problem through installing ffi compiler for ruby platform using the following command:

gem install ffi --platform=ruby

Now, try running

bundle install

Conclusion

Things should work now! Feel free to drop a suggestion or any feedback if this process can be further improved.

Best Practices For Writing Clean Pythonic Code

Dollar Dhingra — Sat, 11 Dec 2021 09:24:32 +0000

Introduction

This article is a collection of best practices for more idiomatic python code especially if you are new to python.

Contribute

Feel free to contribute to this list, and I will thank you by including a link to your profile with the snippet!

1. Catching Exceptions

This is a sure-shot way to get your code into trouble

# very bad practice
try:
    do_something()
except:
    pass

# A slightly better way(included logging), but still a bad practice:
try:
    do_something()
except Exception:
    logging.exception() # although, logging is a good practice

# When we don't use `Exception` we will also be catching events like system exit. 
# So, using Exception means that we are only catching Exceptions.

# A better way:
try:
    do_something()
except ValueError:
    logging.exception()
    # some code to handle your exception gracefully if required

Here we have used a specific type of Exception i.e. ValueError.

2. Name Casing


# using camelCase is not a convention in python
def isEmpty(sampleArr):
    ...

# instead use snake_case in python
def is_empty(sample_arr):
    ...

In python, snake_case is preferred for variables, functions and method names. However, for classes, PascalCase is used.

3. Chained comparison operators

There are multiple ways of comparing in Python:

# Don't do this:
if 0 < x and x < 10:
    print('x is greater than 0 but less than 10')


# Instead, do this:
if 0 < x < 10:
    print('x is greater than 0 but less than 10')

4. Mutable default arguments


# wrong way!
# a sure shot way to get some unintended bugs in your code
def add_fruit(fruit, box=[]):
    box.append(fruit)
    return box


# correct way!
# recommended way for handling mutable default arguments:
def add_fruit(fruit, box=None):
    if box is None:
        box = []

    box.append(fruit)
    return box

Read more about mutable default arguments here

5. String Formatting


# Avoid using it
# %-formatting
name = "James Bond"
profession = "Secret Agent"
print("Hello, %s. You are a %s." % (name, profession))

# slightly better
# str.format()
print("Hello, {}. You are a {}.".format(name, profession))

# Short, crisp and faster!
# f-strings
print(f"Hello, {name}. You are a {profession}.")

The f in f-strings may as well stand for "fast." f-strings are faster than both %-formatting and str.format(). (Source)

6. Top-level script environment

Executes only if it is run as a script and not as a module

# Filename: run.py

if __name__ == '__main__':
    print('Hello from script!')

$ python run.py
$ Hello from script!

Hello from script! will not be printed if the module is imported into any other module.

7. Conditional expressions

if x < 10:
    return 1
else:
    return 2

Can be reduced to this one:

return 1 if x < 10 else 2

8. Iterating over an iterator

You don’t necessarily need to iterate over the indices of the elements in an iterator if you don’t need them. You can iterate directly over the elements.
This makes your code more pythonic.

list_of_fruits = ["apple", "pear", "orange"]

# bad practice
for i in range(len(list_of_fruits)):
    fruit = list_of_fruits[i]
    process_fruit(fruit)

# good practice
for fruit in list_of_fruits:
    process_fruit(fruit)

9. Indexing/Counting during iteration

# Don't do this:
index = 0
for value in collection:
    print(index, value)
    index += 1


# Nor this:
for index in range(len(collection)):
    value = collection[index]
    print(index, value)

# Definitely don't do this:
index = 0
while index < len(collection):
    value = collection[index]
    print(index, value)
    index += 1

# Instead, use `enumerate()`
for index, value in enumerate(collection):
    print(index, value)

10. Using context managers

Python provides context managers that manage the overhead of initializing and clearing up the resources and let
you focus on the implementation. For example in the case of reading a file, you don't need to be concerned
to close the file manually.

d = {"foo": 1}

# bad practice 
f = open("./data.csv", "wb")
f.write("some data")

v = d["bar"] # KeyError
# f.close() never executes which leads to memory issues

f.close()

# good practice
with open("./data.csv", "wb") as f:
    f.write("some data")
    v = d["bar"]
# python still executes f.close() even if the KeyError exception occurs

11. Using set for searching instead of a list

s = set(['s', 'p', 'a', 'm'])
l = ['s', 'p', 'a', 'm']

# ok for small no. of elements
def lookup_list(l):
return 's' in l # O(n)


# better for large no. of elements
def lookup_set(s):
return 's' in s # O(1)

Sets are implemented using hash in python, which makes searching of element faster(O(1)) as compared to searching in a
list(O(n)).

12. using * while importing a module

Imports should always be specific. Importing * from a module is a very bad practice that pollutes the namespace.

# bad practice 
from math import *
x = ceil(x)

# good practice 
from math import ceil   
x = ceil(x) # we know where ceil comes from

13. using items() for iterating a dictionary


d = {
     "name": "Aarya", 
     "age": 13
 }

# Dont do this
for key in d:
    print(f"{key} = {d[key]}")

# Instead do this
for key,val in d.items():
    print(f"{key} = {val}")

Sources

Anti Pattern - Mutable Default Arguments

Dollar Dhingra — Sun, 31 Oct 2021 17:34:56 +0000

Introduction

In this article, you will understand what mutable default arguments are, why you should avoid using them and the workaround.

Code Example With A Mutable Default Argument

Consider the following code example below.

def add_fruit(fruit, box=[]):
    box.append(fruit)
    return box

Let's understand step by step what is happening:

We are creating a function to add fruits(str) in a box(list)
There is a add_fruit function which is responsible for adding the fruit
This function takes 2 arguments: fruit and box
Attention! : The second argument here is a mutable default argument.

So, what is mutable default argument?

An argument in a function with default value as mutable.

In short, Python has both mutable and immutable types. The difference is:

mutables can be modified
immutables can't be modified.

For eg: Tuple is an immutable type. If we define a tuple like this:

weekends = ('saturday', 'sunday',)

weekends[0] = 'Monday' # TypeError: 'tuple' object does not support item assignment

An immutable type canot be modified.

You Might Expect

let's modify our code and create a couple of boxes, i.e. red box and yellow box

def add_fruit(fruit, box=[]):
    box.append(fruit)
    return box

red_box = add_fruit("apple")
print(f"red box: {red_box}")

yellow_box = add_fruit("mango")
print(f"yellow box: {yellow_box}")

Expected Output

red box: ["apple"]

yellow box: ["mango"]

Actually Output

Actually, you get the following output:

red box: ["apple"]

yellow box: ["apple", "mango"]

Wait? What? We never added apple in the yellow box.

What Exactly Happened?

A new list is created once when the function is defined, and the same list is used in each successive call.

Python’s default arguments are evaluated once when the function is defined. This means that if you use a mutable default argument and mutate it, you will and have mutated that object for all future calls to the function as well.

We will get the same result for other mutable types also(For eg: dict).

What Should Be Done?

If your function needs to have a default argument for a mutable type, then default it with None and also add a check for the same.
Let's modify our add_fruit function:

def add_fruit(fruit, box=None):
    if box is None:
        box = []

    box.append(fruit)
    return box

red_box = add_fruit("apple")
print(f"red box: {red_box}")

yellow_box = add_fruit("mango")
print(f"yellow box: {yellow_box}")

This extra check can saves hours of debugging!

Conclusion

It's always a best practice to not use mutable default arguments. Instead, try adding an extra comparison check with
None to handle the default arguments which are mutable.

Code Quality Tools in Python

Dollar Dhingra — Sat, 09 Oct 2021 16:04:52 +0000

What is code quality?

Before we talk about code quality tools, let's see what code quality is. We can agree that code is of high quality if

It does what it is supposed to do

If the code written is not doing what is supposed to do, then it doesn't meet the very basic requirements, and we can say that the code quality is low.

It does not contain defects or problems

Let's say that your code does what it is supposed to do, but struggles to perform well on edge cases. Let's take an example of a mobile app that you start using for communicating with friends and family. You can send & receive messages easily from your friends through this app. One fine day, you want to share a photo with a group and your app crashes. Certainly, there is some issue with the code that has not been tested properly.

It is easy to read, maintain, and extend

Finally, let's consider that a code does what it is supposed to do, and it does not contain any defects or problems. But, the code is not easy to read, maintain and extend.

Robert C. Martin in his book Clean Code says:

Indeed, the ratio of time spent reading versus writing is well over 10 to 1. We are constantly reading old code as
part of the effort to write new code. Because this ratio is so high, we want the reading of the code to be easy, even if it makes the writing harder.

Therefore, code readability is a very important factor in code quality.

Style guide: following the conventions

Great codebases look like they were written by an individual, when they were worked on by a team.

Consistency code is easier to read & therefore maintain by a team.
A style guide serves the purpose of defining a consistent way to write your code.
The style guide contains conventions to be followed for writing a code. PEP8 is the official style guide for Python that is recommended to use for writing any python code. Let's consider a small example. In the python code below, the operators tend to get scattered across different columns on the screen, and each operator has moved away from its operand and onto the previous line. Here, the eye has to do extra work to tell which items are added and which are subtracted:

Not Recommended:

## No: operators sit far away from their operands
income = (gross_wages +
          taxable_interest +
          (dividends - qualified_dividends) -
          ira_deduction -
          student_loan_interest)

To solve this readability problem, mathematicians and their publishers follow the opposite convention which is more
readable

Recommended:

## Yes: easy to match operators with operands
income = (gross_wages
          + taxable_interest
          + (dividends - qualified_dividends)
          - ira_deduction
          - student_loan_interest)

Another official recommendation from Python is PEP-257 Docstring Conventions. A docstring is a string literal that occurs as the first statement in a module, function, class, or method definition.

def complex(real=0.0, imag=0.0):
    """Form a complex number.

    Keyword arguments:
    real -- the real part (default 0.0)
    imag -- the imaginary part (default 0.0)
    """
    if imag == 0.0 and real == 0.0:
        return complex_zero
    ...

There are tools capable of generating documentation directly from the code if the docstrings are consistent in the code.

Some organisations like Google have their own style guides.
Checkout google's python style guide.

How to measure code quality?

There are different ways to measure code quality in general. This article talks about some basic tools
provided in Python to measure the code quality:

Linters
Formatters

Linters

Linters analyze code to detect various categories of issues:

Logical Issue
- Code errors
- Code with potentially unintended results
- Dangerous code patterns
Stylistic Issue
- Code not conforming to defined conventions/style guide

Popular python linters

Pylint: Features like checking coding standards, error detection, refactoring help and more.
pycodestyle: for checking some of the style conventions in PEP8
Flake8: a combination of following linters: PyFlakes, pycodestyle, Ned Batchelder’s McCabe script
Pylama: composed of the following linters and other tools for analyzing code:
pycodestyle
pydocstyle
PyFlakes
Mccabe
Pylint
Radon
gjslint

Every linter has its advantages and disadvantages and should be picked up based on the project you are working on.
If a linter doesn't give out any warning for any lint, that doesn't mean that code is always correct, a linter is just a
static tool for code analysis. It cannot check if the job that is supposed to be done is done or not.

Formatters

Formatters automatically format your code based on a style guide. Some popular python formatters are:

Black

as per the black's documentation, Black is "The uncompromising Python code formatter".
It is my personal favourite because it has minimal configuration and is fast enough. Black is used by some very popular open-source projects, such as pytest, tox, Pyramid, Django Channels, Poetry, and so on. Example usage:

unique.py

def is_unique(
               s
               ):
    s = list(s
                )
    s.sort()


    for i in range(len(s) - 1):
        if s[i] == s[i + 1]:
            return 0
    else:
        return 1


if __name__ == "__main__":
    print(
          is_unique(input())
         )

$ black unique.py produces the following

def is_unique(s):
    s = list(s)
    s.sort()

    for i in range(len(s) - 1):
        if s[i] == s[i + 1]:
            return 0
    else:
        return 1


if __name__ == "__main__":
    print(is_unique(input()))

YAPF

YAPF (Yet Another Python Formatter) is Google's official python formatter which
follows google's style guide. The documentation is easy to
understand the installation and configuration for this formatter.

autopep8

autopep8 is an unofficial, yet popular, tool that automatically formates
Python code to conform to PEP 8. It uses pycodestyle, Python’s official PEP-8 violation checker tool, to determine what parts of the code need to be formatted.

isort

isort is a Python utility/library to sort imports alphabetically, and
automatically separated into sections and by type.

Before isort:

from my_lib import Object

import os

from my_lib import Object3

from my_lib import Object2

import sys

from third_party import lib15, lib1, lib2, lib3, lib4, lib5, lib6, lib7, lib8, lib9, lib10, lib11, lib12, lib13, lib14

import sys

from __future__ import absolute_import

from third_party import lib3

print("Hey")
print("yo")

After isort:

from __future__ import absolute_import

import os
import sys

from third_party import (lib1, lib2, lib3, lib4, lib5, lib6, lib7, lib8,
                         lib9, lib10, lib11, lib12, lib13, lib14, lib15)

from my_lib import Object, Object2, Object3

print("Hey")
print("yo")

Where to use the code quality tools?

You can use these code quality tools:

In your IDE/editor, when you are writing your code
on committing the code
merging the code or while running the tests

IDE/editor

Most modern IDEs and editors have inbuilt support for linters. Some editors like VS Code have great support for linters. You can also run a specific formatter on running the auto-format command in VS Code. Check more about it
here.

For IDE like Pycharm, there are plugins available for running lints and formatters. For example, check out the pylint plugin and
black formatter plugin

On Committing the code: Hooks

Like many other Version Control Systems, Git has a way to fire off custom scripts when certain important actions occur through git-hooks. You can use these scripts to run the
lints and formatters to block any new code that doesn’t meet quality standards or to automatically format the commit.
A useful pre-commit (written in python) is a multi-language package manager for
pre-commit hooks. You specify a list of hooks you want and pre-commit manages the installation and execution of any hook written in any language before every commit

When running tests: Continuous Integration

You can also place linters & formatters directly into whatever system you may use for continuous integration.
The linters can be set up to fail the build if the code doesn’t meet quality standards. A CI(continuous integration) is a practice of automating the integration of code changes from multiple contributors into a single software project. This means that as soon as a developer merges their code in the main branch, the CI pipeline(for eg: git workflow) can perform some automatic operations like linting before the code is deployed.

Conclusion

A code is of high quality if it does what it is supposed to do, it does not contain defects or problems and
it is easy to read, maintain, and extend. Style guides like A style guide like PEP8 and google's style guide help serve the purpose of defining a consistent way to write your code. The different tools for measuring and improving the code quality are linters & formatters.

Linters analyze code to detect various categories of issues like logistical issue and stylistic issues. Some popular linters are Pylint, pycodestyle, [Flake8 (https://pypi.org/project/flake8/) and Pylama.

Formatters automatically format your code based on a style guide. Some popular formatters are Black, YAPF, autopep8 and [isort (https://github.com/PyCQA/isort).

Finally, You can use these code quality tools using:

IDE/editors - when you are writing your code
pre-commit hooks - on committing the code using
continuous integration - while running the tests

CoWIN Vaccine Availability - My first open source project

Dollar Dhingra — Mon, 17 May 2021 12:02:15 +0000

Introduction

CoWIN is a vaccination booking platform for booking vaccination for Indian citizens. This is the largest vaccination drive ever. Anyone can simple go and book the vaccination slots as per the availability in the respective state and districts. In this post I have created another platform just for checking the vaccine availability.

But what's the need for creating a different platform for checking the availability when you already have the official platform?

Problem

I had a hard time finding the vaccination slots (for 18-45 age group). Some of the problems I(and other people I think) faced:

Most of the times the slots were not available and when they were available, they were booked in seconds.
The User experience of the original CoWIN portal is not that great. Firstly you need to login through your phone number(by an OTP which sometimes refuses to arrive on your phone). You are automatically logged out after 15-20 mins.
There are some features that are missing, but can be really helpful in finding the slots. For eg: Ability to search across multiple districts.
A filter for hiding the unavailable slots from the search result would have been really helpful.

Some people have also written scripts for notifying whenever the slots are available. The idea is really great but there are couple of problems with the scripts approach:
As soon as you get the notification that slots are available in your district, you open up your browser ->login into the CoWIN portal -> select the beneficiaries that you want to get vaccinated -> select the state & district/enter pin code and click search -> BOOM! the slots have already been booked!

Solution

I created a simple HTML page which calls the public CoWIN API in JS. Checkout the live demo here.

Note: This website can only be used for checking the available slots. For booking, the official site must be used.

This helps in booking as

There is no need for entering phone number/otp again and again
The results are visible for multiple districts at once.

How I have booked more than 20 slots till now

I have booked more than 20 slots(for friends and family) with the following approach:

There is usually a 1-2 hour window when the slots are open. Usually between 6-10pm on weekdays and 12-4pm on weekends. These timings may change so keep casually looking for the slots whenever you have your phone in hand through the cowinquick site.
Let's say that you find available slots in your district or a nearby district. Note the name of the center. Let's say that the name of the center is "BLK Hospital Site 1".
In one tab, login into the cowin portal and select the city and distric/pin code. In the other tab you can open the cowinquick site.
Now check if "BLK Hospital Site 1" is available. If not, then come back to the cowinquick site and search again. There is high probability that "BLK Hospital Site 2" will come up next in few minutes.
Keep searching after every 5-10 seconds. There is a chance that you are searching in the time window where the slots are being updated. You will come to know that the slots are being updated if you keep searching for 5-10 mins(clicking search after every 10-20 seconds). Either of following will happen:
- You will see that for some centers the slots are being open and booked in 15-20 seconds. Try booking that center in those 15 seconds. The advantage you have on cowinquick site is that you can see the results for multiple districts at once & you don't need to select 18+ again on every search.
- There maybe a situation when you don't see any centers updating slots. You can then try the same after some time.

Beginner's guide to abstract base class in Python

Dollar Dhingra — Fri, 02 Apr 2021 14:33:48 +0000

What is an abstract base class?

In simple words, an abstract base class provides a blueprint for other classes. Abstract base classes don't contain the implementation. Instead, they provide an interface and make sure that the derived classes are properly implemented.

Why do we need abstract base classes?

Imagine that you are creating a game in which you have different animals. For defining animals you can have an abstract class called Animal. A Dog/Cat/Duck are all the classes that are derived from the base class Animal.

Animal Base Class

class Animal:
    def get_age():
        ## return age of the animal

    def is_dead():
        ## return the colour of the animal

Every animal will have both of these methods. So, we want every animal to implement these methods whenever we are inheriting an animal class from the Animal class. We cannot have an animal that doesn't support any of these methods. [Rule 1]
Whenever we see an animal, we call it a dog/cat/duck/tiger. These are all kinds of animals. That is, you never look at something purple and furry and say "that is an animal and there is a no more specific way of defining it". The point is, that you can never see an animal walking around that isn't more specifically something else (duck, pig, etc.) [Rule 2]

Rules

By using these 2 rules we can make our code more maintainable and programmer-friendly

Rule 1

Subclasses inherited from a specific base class must implement all the methods and properties defined in the abstract base class.

Rule 2

Abstract base classes cannot be instantiated. They are inherited by the other subclasses.

Implementation in Python

Let's try to implement these animal classes in Python with the rules we talked about.

class Animal:
    def get_age(self):
        # return the animal's age
        raise NotImplementedError()

    def is_dead(self):
        # check if animal is dead or alive
        raise NotImplementedError()


class Dog(Animal):
    def bark(self):
        print("whoof whoof!!")

    def get_age(self):
        print("5 years")

In the above program, we are creating a Dog class that is inheriting all the methods from the Animal class. Let's see if rule 1 is followed.

Rule 1: Subclasses inherited from a specific base class must implement all the methods and properties defined in the abstract base class.

Let's create an object of the Dog class

bulldog = Dog()

bulldog.get_age() # 5 years

bulldog.is_dead() # NotImplementedError

The subclass Dog does not implement the is_dead method of the Animal base class and when we try to call this method from the bulldog object we get an error. So this works as expected. But we can run the program successfully if we don't call this method without even a warning for breaking this rule. Ideally, all the methods of the base class should be implemented by the subclass.

** The above program does not follow Rule 1.**

Rule 2: Abstract base classes cannot be instantiated. They are inherited by the other subclasses.
Let's check if Rule 2 is followed. Let's try to instantiate the base class.

animal = Animal()
animal.get_age() # NotImplementedError

We get the error if we try to call the base class' method. But, we can instantiate the base class.
So, the above program doesn't follow Rule 2

`abc` module to the rescue

With Python's abc module, we can do better and make sure that both the rules are followed.
Let's update the above program

from abc import ABC, abstractmethod


class Animal(ABC):
    @abstractmethod
    def get_age(self):
        pass

    @abstractmethod
    def is_dead(self):
        pass


class Dog(Animal):
    def bark(self):
        print("whoof whoof!!")

    def get_age(self):
        print("5 years")


    # we forgot to declare "is_dead" again

In the above program, we have imported the helper class ABC from the module abc through which we have implemented the base class. Also, we have used the decorator @abstractmethod for all the methods in our base class.

Let's try to instantiate our subclass: Dog

bulldog = Dog()

TypeError: Can't instantiate abstract class Dog with abstract methods is_dead

This time we get a TypeError specifying that the abstract method is not implemented.

This follows Rule 1 successfully.

Let's try to instantiate our base class: Animal

animal = Animal()

TypeError: Can't instantiate abstract class Animal with abstract methods get_age, is_dead

This time we get a TypeError specifying that the abstract class cannot be instantiated.

This follows Rule 2 successfully.

Conclusion

Using ABCs in your code will make your class hierarchies more robust and more readily maintainable. Some of the key takeaways in this article are:

Abstract e Classes(ABCs) ensure that derived classes implement the particular methods from the base class at instantiation time.
Using ABCs can help in writing bug-free class hierarchies which are easy to read and maintain.

DEV Community: Dollar Dhingra

Benchmarking Python JSON serializers - json vs ujson vs orjson

Introduction

Benchmarking

Conclusion

As a lead engineer/manager, what steps can you take for your team to provide better estimations for their tasks?

2021 year in review

Bus Factor in Software Engineering

1. Introduction

2. What is Bus Factor?

3. Reducing/Preventing the Bus Factor in Software Development

3.1 Documenting

3.2 Code Reviews

3.3 Meetings

3.4 Team Process

3.5 Positive Work culture

3.5.1 Mandatory Paid Annual Vacation

3.5.2 Short Notice/Termination Period

Conclusion

How to install older Ruby versions(2.x) on Mac M1 without using arch x86_64

Introduction

Pre-Requirements

Steps to Reproduce The Problem

Solution

Other Installations

Conclusion

Best Practices For Writing Clean Pythonic Code

Introduction

Contribute

1. Catching Exceptions

2. Name Casing

3. Chained comparison operators

4. Mutable default arguments

5. String Formatting

6. Top-level script environment

7. Conditional expressions

8. Iterating over an iterator

9. Indexing/Counting during iteration

10. Using context managers

11. Using set for searching instead of a list

12. using * while importing a module

13. using items() for iterating a dictionary

Sources

More Articles You May Like

Anti Pattern - Mutable Default Arguments

Introduction

Code Example With A Mutable Default Argument

So, what is mutable default argument?

You Might Expect

Expected Output

Actually Output

What Exactly Happened?

What Should Be Done?

Conclusion

Code Quality Tools in Python

What is code quality?

It does what it is supposed to do

It does not contain defects or problems

It is easy to read, maintain, and extend

Style guide: following the conventions

How to measure code quality?

Linters

Popular python linters

Formatters

Black

YAPF

autopep8

isort

Where to use the code quality tools?

IDE/editor

On Committing the code: Hooks

When running tests: Continuous Integration

Conclusion

Similar Posts by me

CoWIN Vaccine Availability - My first open source project

Introduction

Problem

Solution

How I have booked more than 20 slots till now

Beginner's guide to abstract base class in Python

`abc` module to the rescue