DEV Community: chanduthedev

Book a Cook

chanduthedev — Thu, 31 Oct 2024 04:09:36 +0000

These days I am finding lot of requests in our gated community for cook for a day, only morning session etc. I think it will be good idea to have an mobile app for this.

Decorators in python

chanduthedev — Sun, 11 Apr 2021 09:46:20 +0000

Before learning decorators in python, lets check what is decorator in general.

As the name itself says, decorator means, decorating or wrapping. If we say, decorating a table means making the table more beautiful by adding/placing other things like flowers, lights etc. So decorating basically means adding/modify extra things to make things looks differently.

A similar way, in python, decorators means decorating or wrapping functions. We can add some more functionality before/after executing the actual function.

Let's see this with example.

Decorator example with no arguments:

# Normal function
def add_num():
    return 2 + 3

# Decorator function
def decorator_func(fun):
    def wrapper():
        res_str = f'Sum of 2 + 3 is '
        re = fun()
        return res_str + str(re)
    return wrapper

Normal function call:

add_num()

Output:
5

Decorator function call:

dec_result = decorator_func(add_num)
print(dec_result)

Output:
Sum of 2 + 3 is 5

In Decorator function, we added an extra string value to make the result more clear. In this way decorators are very useful..

Instead of calling decorator function directly, we can use annotations by using @ symbol like below

@decorator_func
def add_num():
    return 2 + 3

Normal function call with decorator annotation:

dec_annot_call = add_num()
print(dec_annot_call)

Outout:
Sum of 2 + 3 is 5

Decorator example with arguments:

To pass arguments to the decorator function, we need to use keyword arg params *args and *kwargs like shown below.

# Decorator function with arguments
def decorator_func(fun):
    def wrapper(*args, **kwargs):
        res_str = f'Sum of {args[0]} + {args[1]} is '
        re = fun(*args, **kwargs)
        return res_str + str(re)
    return wrapper

# Decorator annotations for normal function with args
@decorator_func
def add_num(arg1, arg2):
    return arg1 + arg2

# Calling with args
dec_annot_call = add_num(8, 9)
print(dec_annot_call)

Output:
Sum of 8 + 9 is 17

What is first class object in python?

chanduthedev — Sat, 10 Apr 2021 05:55:43 +0000

In python, functions are first class objects in python.

Advantages of first class objects:

We can treat functions like other normal objects like int, string, list etc
We can pass functions as another function arguments, return a function in another function and assign a function to another variable
Define a function within a function (Inner functions)
This first class object plays a key role in python for decorator design pattern.

Examples:

# Simple function for adding two numbers
def add_num(num1, num2):
    return num1 + num2

# Passing function as a first argument
def add_values(fun, arg1, arg2):
    result = fun(arg1, arg2)
    return result

Assigning function name to the variable:

fun = add_num

Passing function as a parameter

two_num_sum = add_values(fun, 5, 8))

Category type in pandas

chanduthedev — Fri, 09 Apr 2021 13:16:25 +0000

Python pandas library supports a data type called Category. When working with pandas dataframe, using Category will help in many ways. Let's see about Category datatype.

What is Category data type in pandas?

Category is a datatype which can be used when we have a fixed number of string values like
- Months(Jan, Feb)
- Country Names(India, Singapore)
- Size(Small, Medium, Large)
In a simple way is using a sequence of integer values for the strings(Jan - 1, Feb - 2 etc)
Categories are similar to ENUM data types in other programming languages like C/C++, Java.

Advantages of using Category:

Saving lot of memory by reducing the size
Increasing processing speed

How to use Category in pandas dataframe:

- While reading the CSV file:

We can convert column from object to category while reading the file like below

filename = "~/Downloads/US_Accidents_Dec20.csv"
# Converting into category data type while reading CSV file
us_accidents_dec20_cat = pd.read_csv(filename, dtype = {'State' : 'category', 'City' : 'category'})

- Converting column into category type:

We can convert the column on the fly like below

# Loading csv file into data frame
filename = "~/Downloads/US_Accidents_Dec20.csv"
us_accidents_dec20_cat = pd.read_csv(filename,)

# Normal column access
us_accidents_dec20['State']

# Converting to category data type
us_accidents_dec20['State'].astype('category')

Memory comparison between Object vs Category data types:

Normal object column:

us_accidents_dec20['State'].memory_usage(deep=True) / 1e6

Result:
249.720047

Category column:

us_accidents_dec20['State'].astype('category').memory_usage(deep=True) / 1e6

Result:
4.23684

We can clearly observe storage space reduced from 249 to 4 which is a very huge difference.

Converting to Category data type will certainly help improve processing speed and space with a large set of data.

Happy Learning!!

P.S: Used Accidents' data of December 2020 from The USA, You can get this data from kaggle.

Python List vs NumPy Array

chanduthedev — Sat, 09 Jan 2021 07:05:14 +0000

I will try to explain why NumPy is more powerful and efficient compare with Python List.

First we will see basic definition of the list vs Array programming terminology

Lets see basic data structures arrays and lists

Arrays: are used to store homogeneous data (same data type) of fixed size storing in sequential order in memory
Lists are used to store data of growing in size and storing this data in available place anywhere(not sequential) in the memory.

Almost same logic applies to Python List and NumPy Array. You can see in the below picture, how list and array stores data in memory.

Python List:

Able to store different data types in the same list
Storing each item in random location in the memory
Good for the scenario where list can grow dynamically
Inbuilt data type.
It has more inbuilt functions
Appending will take less time in O(1) time

NumPy Array:

Can store only one data type in an array at any time
Storing each item is sequential which makes array more effective in processing.
Good for the scenario where the items are fixed size and same data time
Need to install external library NumPy
No extra functions, so it will not more memory store.
Appending elements will take more time in O(N) time

Happy Learning!!!

Error when trying to delete docker image

chanduthedev — Sat, 02 Jan 2021 03:59:38 +0000

When I try to remove docker image using docker rmi image-id, many times I got below error.
Error response from daemon: conflict: unable to delete e3c37a7601a1 (must be forced) - image is referenced in multiple repositories

Which means image was used by some other process.
But When I use docker ps to get docker process, got empty container list.

Follow below three steps to delete docker image if you get this error.

Step1. Use below command to show all container list.

docker ps -a

Step2. User below command to remove container

docker rm container-id

Step3. Use below command to remove docker image.
docker rmi image-id

Another way to delete docker unused containers.

This will delete all unused containers, networks, images (both dangling and unreferenced).

docker system prune

Be cautious while using this command as it will delete all containers.

Is removing Duplicate code always good?

chanduthedev — Sat, 19 Dec 2020 01:01:06 +0000

While refactoring code, If I see duplicate code, I will do below

Make a function with common code snippet
Remove all the similar duplicate code
Calling common method whenever required.

My opinion on duplicate code.

Need to update the code in all the places when some changes need in that code snippet
Need to test all the flow which involves this code snippet
There is a definite chance to miss to modify in all places (Major problem)

Removing duplicate code is good for maintainability as we will have less code(and less bugs ;-))

But Removing duplicate code is not always good.
As per the design principle keep it simple, stupid(KISS),
unnecessary complexity should be avoided. All functions kept simple rather than made complicated.

Let's take below examples with sample code.

Validating date format (startdate and enddate)
Validating username (admin and customer)

These code samples are only for example purpose, so writing very basic validation code.

1. Validating date format:

See duplicate code in the two methods validate_start_date and validate_end_date.
In this case you can remove duplicate code and make a common method as validate_date().

Reason for making a common method are.

validation format is same (yyyy-mm-dd) for both methods

2. Validating username:

for admin and customer users, even though code is almost same in both methods, validate_admin_username() and validate_customer_username(), It's NOT advised to replace into common method validate_usernmae() like validating date for below reasons.

Simplicity is the higher priority than maintainability
Remove dependency between admin and customer - As per KISS design principle , not making complicated
Different Feature enhancement may come in future for admin and customers

If we keep separate methods for admin and customer, later it will be very easy and simple for adding features separately for admin and customer.

In conclusion, It is absolutely fine to have duplicate code until it matches KISS principle.

Happy Learning!!!

TOGAF 9.2 components

chanduthedev — Sun, 29 Nov 2020 04:00:11 +0000

For clearing TOGAF certification, there are two key topics.

Architecture Development Method(ADM)
TOGAF Components.

I have published article on TOGAF ADM, You can refer here if not read earlier.
If you memorize these two topics, I am sure you will clear TOGAF exam very easily.
You can refer in the below picture all about TOGAF components in detail.

Get the original image from here.

TOGAF 9.2 Architecture Development Method(ADM) in one page

chanduthedev — Sun, 22 Nov 2020 05:26:31 +0000

I have completed my TOGAF 9.2 certification part1 exam with 80% score. For TOGAF, Architecture Development Method(ADM) is the key architecture framework. After reading TOGAF documentation, I prepared completed ADM process in one page. I hope this will help others to complete TOGAF9.2 certification.
When I prepare this page, TOGAF version was 9.2 and this one page prepared basing on TOGAF 9.2 version.

Get the original image from here.

Try to remember everything in the page and do not skip anything form this page. This will cover ~12-15 questions out of 40 questions.

Happy Learning!!

docker learning

chanduthedev — Wed, 12 Aug 2020 15:23:08 +0000

Recently I started working on a project which includes both NodeJS and Python application. We planned to Deodorization build process for our project. During Deodorization I faced multiple issues which I eventually solved. I thought of writing a post on my docker experience to help others. Below are the some of my observations/issues I found during my journey

Using multiple FROM in docker file
Speed up docker docker builder by NOT copy all files at the same time
Using RUN stead of CMD
Use docker exec to run commands on docker container
Showing docker containers (running docker images)
Connect to the docker container docker exec -it CONTAINERID /bin/bash
How to access host application from docker container
Expose Port numbers

Refer chanduthedev for more details.

Happy Learning!!!

ElasticSearch error message 'The [dims] property must be specified for field'

chanduthedev — Tue, 21 Jul 2020 13:56:14 +0000

When I try to insert dense_vector values to ElasticSearch(ES), I got below error. I will explain how I resolved this error.

Used PUT method to create a index in ES with mappings as body

PUT http://localhost:9200/my_index

Created successfully and got below response.
{ "acknowledged": true, "shards_acknowledged": true, "index": "my_index" }

But when I try to insert values, got below error for the POST request.
POST http://localhost:9200/my_index/_creat

Request Body:

{
"name": "chanduthdev",
"vector" : [ 8.56176019e-02, 6.78501204e-02]
}

Response :

{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "The [dims] property must be specified for field [vector]."
}
],
"type": "mapper_parsing_exception",
"reason": "The [dims] property must be specified for field [vector]."
},
"status": 400
}

Solution:

To resolve this error, replace _creat with _doc in the url for the POST method.
POST http://localhost:9200/my_index/_doc

Response:

{
"_index": "my_index1",
"_type": "_doc",
"_id": "3VbDcHMBBskIYHrmDycX",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"_seq_no": 0,
"_primary_term": 1
}

Happy Learning!!!

Difference between CLI and GUI

chanduthedev — Mon, 09 Dec 2019 06:38:00 +0000

One of the key thing I observed was that there are NO hidden actions performed when using CLI where as in GUI, there might be some extra hiden actions will be performed which we are not aware off.

When using CLI, "we know what exactly we are doing" whereas in GUI, "we dont know what extra things happening in the background". One of the best example of this was using Sourcetree tool to commit code changes git repository.

Using CLI, to commit changes in git, there are two CLI commands, one is git add to add files and second one is git commit to add commit message.

When using GUI with Sourcetree, We just need to provide the commit message after selecting files.

GUI has the advantage of providing ease and simplicity to the customer. But as a developer or technical person, we should know what we are doing. My Suggestion is always prefer CLI and you never go to GUI until and unless you know what exactly what GUI tool is doing.

Happy Learning!!!