DEV Community: Prashant Sharma

Metaclasses in Python

Prashant Sharma — Sat, 22 Aug 2020 13:09:18 +0000

In most programming languages, classes are just pieces of code that define the rules for an object, but in Python, as you must hear that everything is an object: it turns out that this is true of classes themselves. classes are actually first-class objects, they can be created at runtime, passed as parameters, returned from functions, and assigned to variables.

let's look at the below example -

class Tutorial:
    pass

print(Tutorial())

# Output - 
# <__main__.Tutorial object at 0x7fd92c1500f0>

as we can see the instance of Tutorial class tells us that this is an object of the main Tutorial object. at some location.

Now just print the class itself -

print(Tutorial)

# Output - 
# <class '__main__.Tutorial'>

The reason we're able to do this because the Tutorial class is an object, just like any other object. When you use the class keyword, Python creates this object automatically. It's an instance of a metaclass - type.

A metaclass is the class of a class; it defines how a class behaves.

Now for making it easier, let go into depth, Maybe you have come across the type keyword in Python? Which used to finding the Type of an Object -

print(type(1))

# Output - 
# <class 'int'>

print(type('Hey'))

# Output - 
# <class 'str'>

As expected 1 is the type of int class, Hey is the type of str class, let's find out the Type of our Class -

print(type(Tutorial))

# Output - 
# <class 'type'>

This time, we get a printout that Tutorial is of type of class type. But what about the type itself? What is the type's type?

print(type(type))

# Output - 
# <class 'type'>

the type of type is class type, you maybe find this weird. Thus we find that type is also its own metaclass!

Understanding How Metaclasses Work

type is a built-in metaclass in python. it is used to construct classes just like a class is used to construct the objects. So Whenever we create any class then the default metaclass(type) gets called and giving us an option to use it as an object.
That means every class in python is also an object of type, Therefore We can use type directly to make a class, without any class syntax. The type() function can be used to directly define classes by using the following three arguments -

type(<name>, <bases>, <dct>)

name - This is the internal representation of the class. This is the name of the class.
bases - This specifies anything that we inherit from a superclass or a parent class. This is a tuple of the parent class.
dct - This specifies a namespace dictionary containing definitions of class's methods and variables.

To make it more clear -

Test = type('Test', (), {})

so the above line of code is completely equivalent to this below code -

class Test:
    pass

there's absolutely nothing different about them.

Hence, if we want to modify the behavior of classes, we will need to write our own custom metaclass.

To create our own custom metaclass, we first have to inherit the default metaclass type, and implement the metaclass's __new__ method and/or __init__ method.

__new__: This dunder method is usually overridden type's __new__, to modify some properties of the class to be created,
before calling the original __new__ which creates the class.
__init__: This method is called when you want to control the initialization after the instance/object has been created.

class MyMeta(type):
    def __new__(self, name, bases, attr):
        print(attr)
        return type(name, bases, attr)

So here I have defined a simple metaclass MyMeta, and we're just going to print out the attributes so we can see how they look like. after that, I have defined another class Sample and it's having the metaclass MyMeta, with name and age variable -

class Sample(metaclass=MyMeta):
    name = 'bob'
    age = 24

Now without creating an instance this stills runs as you can see the output below -

{'__module__': '__main__', '__qualname__': 'Sample', 'name': 'bob', 'age': 24}

So how this class Sample has been created -

Interpreter sees metaclass=MyMeta defined in Sample, So now the interpreter has information that default metaclass type must not be used to create Sample class, Instead, MyMeta must be used to create Sample class.

So, the interpreter makes a call to MyMeta to create class Sample, When MyMeta is got called, __new__ of MyMeta is invoked and it prints out the attributes and using type its construct the instance of MyMeta which is Sample and returns to us the object.

This metaclass only overrides object creation. All other aspects of class and object behavior are still handled by type.

Now We've covered enough theory to understand what metaclasses are and how to write custom metaclasses. Now let's look a simple real case scenario -

let's suppose we have a requirement that all attributes of your class should be in upper case, There are multiple ways to achieve this functionality, but here we are going to achieve this using metaclass at the module level, So if in namespace dictionary(attributes) if a key doesn't start with a double underscore we need to change it to be uppercase -

class MyMeta(type):
    def __new__(self, name, bases, atts):

        print(f'current_attributes - {atts}\n')
        new_atts = {}

        for key, val in atts.items():
            if key.startswith('__'):
                new_atts[key] = val
            else:
                new_atts[key.upper()] = val

        print(f'modified_attributes - {new_atts}')        
        return type(name, bases, new_atts)


class Sample(metaclass=MyMeta):
    x = 'bob'
    y = 24

    def say_hi(self):
        print('hii')

Output -

current_attributes - {'__module__': '__main__', '__qualname__': 'Sample', 'x': 'bob', 'y': 24, 'say_hi': <function Sample.say_hi at 0x7fd92c10d048>}

modified_attributes - {'__module__': '__main__', '__qualname__': 'Sample', 'X': 'bob', 'Y': 24, 'SAY_HI': <function Sample.say_hi at 0x7fd92c10d048>}

As you can see above we have printed the current attributes and created a dictionary that we used to represent our modified attributes.

Here we are just checking if the key starts with a double underscore then just adds the proper value otherwise we are adding the uppercase attribute with that corresponding value.

Now to make sure that this is working let's try, I have created an instance of Sample, and just print one of the old attributes.

s = Sample()
s.x

Output -

AttributeError Traceback (most recent call last)
<ipython-input-18-87b2922593a9> in <module>
      1 s = Sample()
----> 2 s.x

AttributeError: 'Sample' object has no attribute 'x'

As expected we have got an error AttributeError, which is totally fine because we have just modified the construction of the object, let's try to access by modified attributes -

print(s.X)

s.SAY_HI()

Output -

bob
hii

This is why they call it magic because with this kind of hook into the creation of classes you can really enforce quite a bit of constraint on how classes are created

So for example, if you want every single class in a specific module to never be allowed to use a certain attribute or to follow a specific pattern you could set metaclasses for those specific modules.

Conclusion

The purpose of metaclasses isn't to replace the class/object distinction with metaclass/class - it's to change the behavior of class definitions (and thus their instances) in some way.
A reasonable pattern of metaclass use is doing something once when a class is defined rather than repeatedly whenever the same class is instantiated. When multiple classes share the same special behavior, repeating metaclass=X is obviously better than repeating the special purpose code and/or introducing ad-hoc shared superclasses.

Reference

https://stackoverflow.com/questions/100003/what-are-metaclasses-in-python

https://stackoverflow.com/questions/392160/what-are-some-concrete-use-cases-for-metaclasses/393368

Monkey Patching In Python

Prashant Sharma — Fri, 19 Jun 2020 22:47:47 +0000

What is Monkey patching

In Python, the term monkey patch only refers to dynamic modifications of a class or module at runtime, which means monkey patch is a piece of Python code that extends or modifies other code at runtime.

Monkey patching can only be done in dynamic languages, of which python is a good example. In Monkey patching, we reopen the existing classes or methods in class at runtime and alters the behavior, which should be used cautiously, or you should use it only when you really need to. As Python is a dynamic programming language, Classes are mutable so you can reopen them and modify or even replace them.

It is often used to replace or extends a method on the module or class level with a custom implementation.

Let's Implementation monkey patching -

class MonkeyPatch:
    def __init__(self, num):
        self.num = num

    def addition(self, other):
        return (self.num + other)

obj = MonkeyPatch(10)
obj.addition(20)

Output -

As you can see below, now only two methods __init__ and addition and are available for the above class object. (You can use dir(obj) also to get the members of the object)

import inspect 

inspect.getmembers(obj, predicate=inspect.ismethod)

Output -

[('__init__',
  <bound method MonkeyPatch.__init__ of <__main__.MonkeyPatch object at 0x7f32495d7c50>>),
 ('addition',
  <bound method MonkeyPatch.addition of <__main__.MonkeyPatch object at 0x7f32495d7c50>>)]

In the above piece code, we have already defined a MonkeyPatch class with an addition method, and later on, we are adding a new method to the MonkeyPatch class. Suppose the new method is as follows.

def subtraction(self, num2):
    return self.num - num2

Now how we add the above method to the MonkeyPatch class, simply just place subtraction function into MonkeyPatch class with an assignment statement, as below -

MonkeyPatch.subtraction = subtraction

the newly created subtraction function would be available for all existing instances as well as a new instance also, Let's do some practical now -

import inspect

inspect.getmembers(obj, predicate=inspect.ismethod)

Output -

[('__init__',
  <bound method MonkeyPatch.__init__ of <__main__.MonkeyPatch object at 0x7f28186b8c88>>),
 ('addition',
  <bound method MonkeyPatch.addition of <__main__.MonkeyPatch object at 0x7f28186b8c88>>),
 ('subtraction',
  <bound method subtraction of <__main__.MonkeyPatch object at 0x7f28186b8c88>>)]

Did you notice for the existing object obj which we have defined earlier contained the newly created function, let's check it out if the newly created function will work for the existing instance -

>>> obj.subtraction(1)               # Working as expected
9
>>> obj_1 = MonkeyPatch(10)          # create some new object
>>> obj_1.subtraction(2)
8

obj.subtraction work as expected, but Keep in mind if a method already exists with the same name then this will change the behavior of the existing method.

Things to keep in mind

The best thing is not to monkey patch. You can define child classes for the ones you want to alter. Still, if monkey patch needed then follow these rules -

Use if you have a really good reason(like - temporary critical hotfix)
Write proper documentation describing the reason for monkey patch
Documentation should contain the information about the removal of the monkey patch and what to watch for. Lots of monkey patches are temporary, so they should be easy to remove.
Try to make monkey patch as transparent as possible also place monkey patch code in separate files

Conclusion

Now we have learned how to do a monkey patch in Python. However, it has its own drawbacks and should be used carefully. Well, this often means the bad architecture of your application, and is not a good design decision, because it creates a discrepancy between the original source code on disk and the observed behavior and can be very confusing when troubleshooting.

Reference

https://en.wikipedia.org/wiki/Monkey_patch
https://stackoverflow.com/questions/5626193/what-is-monkey-patching

SpeedUp Python List and Dictionary

Prashant Sharma — Sun, 03 May 2020 17:22:02 +0000

Today, we will be discussing the optimization technique in Python. In this article, you will get to know to speed up your code by avoiding the re-evaluation inside a list and dictionary.

Here I have written the decorator function to calculate the execution time of a function.

import functools
import time

def timeit(func):
    @functools.wraps(func)
    def newfunc(*args, **kwargs):
        startTime = time.time()
        func(*args, **kwargs)
        elapsedTime = time.time() - startTime
        print('function - {}, took {} ms to complete'.format(func.__name__, int(elapsedTime * 1000)))
    return newfunc

let's move to the actual function

Avoid Re-evaluation in Lists

Evaluating nums.append inside the loop

@timeit
def append_inside_loop(limit):
    nums = []
    for num in limit:
        nums.append(num)

append_inside_loop(list(range(1, 9999999)))

In the above function nums.append function references that are re-evaluated each time through the loop. After execution, The total time taken by the above function

o/p - function - append_inside_loop, took 529 ms to complete

Evaluating nums.append outside the loop

@timeit
def append_outside_loop(limit):
    nums = []
    append = nums.append
    for num in limit:
        append(num)

append_outside_loop(list(range(1, 9999999)))

In the above function, I evaluate nums.append outside the loop and used append inside the loop as a variable. Total time is taken by the above function

o/p - function - append_outside_loop, took 328 ms to complete

As you can see when I have evaluated the append = nums.append outside the for loop as a local variable, it took less time and speed-up the code by 201 ms.

The same technique we can apply to the dictionary case also, look at the below example

Avoid Re-evaluation in Dictionary

Evaluating data.get each time inside the loop

@timeit
def inside_evaluation(limit):
    data = {}
    for num in limit:
        data[num] = data.get(num, 0) + 1

inside_evaluation(list(range(1, 9999999)))

Total Time taken by the above function -

o/p - function - inside_evaluation, took 1400 ms to complete

Evaluating data.get outside the loop

@timeit
def outside_evaluation(limit):
    data = {}
    get = data.get
    for num in limit:
        data[num] = get(num, 0) + 1


outside_evaluation(list(range(1, 9999999)))

Total time taken by the above function -

o/p - function - outside_evaluation, took 1189 ms to complete

As you can see we have speed-up the code here by 211 ms.

I hope you like the explanation of the optimization technique in Python for the list and dictionary. Still, if any doubt or improvement regarding it, ask in the comment section. Also, don't forget to share your optimization technique.

Mutable Default Arguments in Python

Prashant Sharma — Thu, 19 Mar 2020 21:16:59 +0000

This article was originally shared on my blog.

Objects of built-in types like (int, float, bool, str, tuple, Unicode) are immutable. Objects of built-in types like (list, set, dict) are mutable.
A mutable object can change its state or contents and immutable objects cannot.

This is a very simple definition of a Mutable and Immutable object in Python. Now coming to the interesting part which is Mutable Default Object in function definitions.

I have written a very simple function as below -

def foobar(element, data=[]):
    data.append(element)
    return data

In the above function, I set data to a list(Mutable object) as a default argument. Now let's execute this function -

>>> print(foobar(12))
[12]

The output is as expected, Now execute multiple times -

>>> print(foobar(22))
[12, 22]
>>> print(foobar(32))
[12, 22, 32]

What is going on here? As you can see, the first time, the function returns exactly what we expected. On the second and all subsequent passes the same list is used in each call.

Why is this happening

Python’s default arguments are evaluated once when the function is defined, not each time the function is called. When Python encounters it, the first thing it will do is compile it in order to create a code object for this function. While this compilation step is done, Python evaluates and then stores the default arguments in the function object itself.

So, let's do some introspection, to clear the confusions, I have taken a few lines of code as below-

Function Before Execution -

def foo(l=[]):
    l.append(10)

After function executes this definition, You can check the default attribute for this function. by using __defaults__.

About __defaults__
A tuple containing default argument values for those arguments that have defaults, or None if no arguments have a default value.

>>> foo.__defaults__
([],)

The result an empty list as the only entry in __defaults__

Function After Execution-

Let's now execute this function and check the defaults.

>>> foo()
>>> foo.__defaults__
([10],)

Astonished? The value inside the object changes, Consecutive calls to the function will now simply append to that embedded list object:

Let's execute the function multiple times:

>>> foo()
>>> foo()
>>> foo()

>>> foo.__defaults__
([10, 10, 10, 10],)

The reason why this is happening because default argument values are stored in the function object, not in the code object (because they represent values calculated at run-time).

The Solution

Now, the question is how to code foobar in order to get the behavior that we expected.

Fortunately, the solution is straightforward. The mutable objects used as defaults are replaced by None, and then the arguments are tested for None.

def foobar(element, data=None):
    if data is None:
        data = []
    data.append(element)
    return data

>>> foobar(12)
[12]

So, by replacing mutable default arguments with None solved our problem.

Good Use of Mutable Default Argument

However, Mutable Default Argument have some good use-case also as following-

1.Binding local variable to the current value of the outer variable in a callback
2.Cache/Memoization

I hope you like the explanation of Mutable Default Arguments in Python. Still, if any doubt or improvement regarding it, ask in the comment section.

References:
https://docs.python.org/3/reference/datamodel.html#the-standard-type-hierarchy

Assignment vs Shallow Copy vs Deep Copy in Python

Prashant Sharma — Mon, 03 Feb 2020 05:33:26 +0000

This article was originally shared on my blog.

Today, we will be discussing the copy in Python. There are three ways we can do it. In this article, you will get to know what each operation does and how they are different.

1 - Assignment Operator (=)
2 - Shallow Copy
3 - Deep copy

Assignment Operator (=)

>>> a = [1, 2, 3, 4, 5]
>>> b = a

In the above example of the assignment operator, It does not make a copy of the Python objects instead it copying a memory address (or pointer) from a to b, (b=a). Which means both a & b pointing to the same memory address.

Here we can use the id() method to get the address of the object in memory and check if both lists are pointing the same memory.

>>> id(a) == id(b)
True

>>> print('id of a - {}, id of b - {}'.format(id(a), id(b)))
id of a - 140665942562048, id of b - 140665942562048

So, here if you would edit the new list, It’ll get updated in the original list also -

>>> b.append(6)
>>> a
[1, 2, 3, 4, 5, 6]
>>> b
[1, 2, 3, 4, 5, 6]

Because there is only one instance of that list in the memory.

Shallow Copy

A shallow copy constructs a new compound object and then (to the extent possible) inserts references into it to the objects found in the original.

we have 3 different ways to create a shallow copy -

nums = [1, 2, 3, 4, 5]      

>>> import copy

>>> m1 = copy.copy(nums)       # make a shallow copy by using copy module
>>> m2 = list(nums)    # make a shallow copy by using the factory function
>>> m3 = nums[:]       # make a shallow copy by using the slice operator

Here all the above lists contains the same values as the original list -

>>> print(nums == m1 == m2 == m3)
True

but the memory address of each is different -

>>> print('nums_id - {}, m1_id - {}, m2_id - {}, m3_id = {}'.format(id(nums), id(m1), id(m2), id(m3)))
nums_id - 140665942650624, m1_id - 140665942758976, m2_id - 140665942759056, m3_id = 140665942692000

This means that this time, each list's object has its own, independent memory address.

Now move to the more interesting part. If the original list is a compound object (e.g. a list of lists), then after shallow copy new list elements are still referencing the original elements.
So, if you modify the mutable elements like lists, the changes will be reflected on the original elements. Let's look at the below example to get a better understanding -

>>> import copy

>>> a = [[1, 2], [3, 4]]            
>>> b = copy.copy(a)

>>> id(a) == id(b)
False

>>> b[0].append(5)
>>>
>>> b
[[1, 2, 5], [3, 4]]
>>> a
[[1, 2, 5], [3, 4]]      # changes reflected in original list also

As you see in the above example while we are modifying the internal list elements in the new list, it’s getting updated in the original list also, because a[0] and b[0] are still pointing to the same memory address(original list).

>>> print('a[0] - {} , b[0] - {}'.format(id(a[0]), id(b[0])))       
a[0] - 140399422977280 , b[0] - 140399422977280
>>>

>>> id(a[0]) == id(b[0])
True
>>> id(a[1]) == id(b[1])
True

So the new list b has its own memory address but its elements do not. because in shallow copy instead of copying the list's elements to the new object, It simply copies the references to their memory addresses, Therefore while we are making changes to the original object it's reflecting in copied objects and vice versa.

This is a characteristic of shallow copy.

Deep copy

A deep copy constructs a new compound object and then, recursively, inserts copies into it of the objects found in the original.

Creating a deep copy is slower because you are making new copies for everything. In this rather than just copy the address of the compound objects, It Simply makes a full copy of all the list's elements (simple and compound objects) of the original list and allocates different memory address for the new list and then assigns them the copied elements.

To achieve the deep copy we have to import the copy module. And use copy.deepcopy().

>>> import copy
>>> a = [[1, 2, 3], [4, 5, 6]]  
>>> b = copy.deepcopy(a)                     

>>> id(a) == id(b)
False

>>> id(a[0]) == id(b[0])      # memory address is different      
False

>>> a[0].append(8)            # modify the list                                                                                                                                            


>>> a
[[1, 2, 3, 8], [4, 5, 6]]
>>> b
[[1, 2, 3], [4, 5, 6]]        # New list’s elements didn’t get affected

As you see above the original list does not get affected.

The difference between shallow and deep copying is only relevant for compound objects (objects that contain other objects, like lists or class instances).

Hope you like the explanation of Assignment Operator, Shallow Copy and Deep Copy. Still, if any doubt or improvement regarding Copy in Python, ask in the comment section.

References:
https://docs.python.org/2/library/copy.html

Pickling and Unpickling in python Explained

Prashant Sharma — Tue, 21 Jan 2020 20:21:49 +0000

This article was originally shared on my blog

Pickling allows you to serialize and de-serializing Python object structures. In short, Pickling is a way to convert a python object into a character stream so that this character stream contains all the information necessary to reconstruct the object in another python script.

To successfully reconstruct the object, The Pickled byte stream contains instructions to the unpicker to reconstruct the original object structure along with instruction operands, which help in populating the object structure.

Pickle protocols:

There are currently 6 different protocols that can be used for Pickling. The higher the protocol used, the more recent the version of Python needed to read the Pickle produced. To know more about Pickle Protocol.

Pickle has two main methods. The first one is a dump, which dumps an object to a file object and the second one is load, which loads an object from a file object.

pickle.dump(object, file_obj, protocol)

This function takes three arguments -

Python object to serialize.
File object in which the serialized python object must be stored.
Protocol (If a protocol is not specified, protocol 0 is used. If the protocol is specified as a negative value or HIGHEST_PROTOCOL, the highest protocol version available will be used.)

So let's continue to practical:

First of all, we have to import it through the following command:

import pickle

Below is the example of pickling and unpickling -

Pickling

import pickle

def pickle_data():
    data = {
                'name': 'Prashant',
                'profession': 'Software Engineer',
                'country': 'India'
        }
    filename = 'PersonalInfo'
    outfile = open(filename, 'wb')
    pickle.dump(data,outfile)
    outfile.close()

pickle_data()

To Open the file we used open function, The first argument should be the name of your file and the second argument is wb, wb refers the writing in binary mode. This means that the data will be written in the form of byte objects.

Unpickling

import pickle

def unpickling_data():
    file = open(filename,'rb')
    new_data = pickle.load(file)
    file.close()
    return new_data


print(unpickling_data())

Here r stands for reading mode and the b stands for binary mode. You'll be reading a binary file.
The output of the Unpickling function is the same as we pickled.

{'name': 'Prashant', 'profession': 'Software Engineer', 'country': 'India'}

What can be pickled?

The following data types can be pickled:

Booleans,
Integers,
Floats,
Complex numbers,
(normal and Unicode) Strings,
Tuples,
Lists,
Sets, and Dictionaries that contain pickable objects.
Built-in functions defined at the top level of a module
Classes that are defined at the top level of a module

Common UseCase of Pickling

To save a program's state data to disk so that it can carry on where it left off when restarted (persistence)
Sending Python data over a TCP connection in a multi-core or distributed system (marshalling)
Storing Python objects in a database

Dangers of Pickling

The documentation of the Pickle module states:

The Pickle module is not secure against erroneous or maliciously constructed data. Never unpickle data received from an untrusted or unauthenticated source.

Safe implementation of Pickle

The Pickle should never be used between unknown parties.
Ensure the parties exchanging Pickle have an encrypted network connection.
In case of unsecured connection, any alteration in Pickle can be verified by using a cryptographic signature. Pickle can be signed before the transmission, and its signature can be verified before loading it on the receiver side.

I hope that you now have a fair understanding of Pickling/Unpickling in python.

If you have any suggestions on your mind, please let me know in the comments.

Reference

https://docs.python.org/3.8/library/pickle.html

https://stackoverflow.com/questions/7501947/understanding-pickling-in-python

Global Interpreter Lock (GIL) in Python

Prashant Sharma — Wed, 27 Nov 2019 19:30:52 +0000

This post was initially shared on my blog.

Any operation which is executed in the interpreter, GIL ensures that the interpreter is held by a single thread at a particular instant of time. it means that only one thread can be in a state of execution at any point in time.

Let's take an example -

Single-Threaded

import time
from threading import Thread

COUNT = 50000000

def countdown(num):
    while num > 0:
        num -= 1

start = time.time()
countdown(COUNT)
end = time.time()

print('Time is taken by single_threaded - {} Sec.'.format(end - start))

Output -

Time is taken by single_threaded - 1.9384803771972656 Sec.

Multi-Threaded

import time
from threading import Thread

COUNT = 50000000

def countdown(num):
    while num > 0:
        num -= 1

t1 = Thread(target=countdown, args=(COUNT//2,))
t2 = Thread(target=countdown, args=(COUNT//2,))

start = time.time()
t1.start()
t2.start()
t1.join()
t2.join()
end = time.time()

print('Time is taken by multi_threaded - {} Sec.'.format(end - start))

Output -

Time is taken by multi_threaded - 1.905367784500122 Sec.

As you can see both the example single-threaded and multi-threaded take almost the same time to finish the execution because GIL restricts CPU to only work with a single thread.
So even if your system could be have multiple cores/processors. And multiple cores allow multiple threads to execute simultaneously. But since the interpreter is held by a single thread. So Python threads cannot take advantage of multiple cores also.

How two threads perform both CPU and `I/O` operations during execution -

01. The python interpreter creates a process and spawns the threads.
02. When Thread 1 starts execution, it'll acquire the GIL and lock it.
03. Thread 2 has to wait for GIL to be released by Thread 1.
04. In case if Thread 1 is waiting for IO from the client, it releases the GIL and Thread 2 will acquire it and start the execution.
05. Now suppose If Thread 1 gets the IO, Then Thread 1 has to wait for GIL to be release by Thread 2.

A thread waiting for the GIL will do a timed wait on the GIL, with a preset interval that can be modified with sys.setswitchinterval.

Note that Python's GIL is only really an issue for CPython, the reference implementation. Other implementations like Jython and IronPython don't have a GIL, because the platforms they are built on Java for Jython, .NET for IronPython, they handle dynamic memory management differently, and so can safely run the Python code in multiple threads at the same time.

Why CPython use GIL -

Python is not get executed directly, it gets compiled into Python bytecode and bytecode is executed.

Python uses automatic memory management via garbage collection, implemented with a technique called reference counting.

Python internally manages a data structure containing all object reference count, and when an object reference count reaches 0 it removes the object and frees its allocated memory.
However, race conditions in multithreaded programming made it so that the object reference count could be updated incorrectly, making it so that objects could be erroneously freed or never freed at all. One way to solve this problem is with more granular locking like locks are set on every shared object, but this would create issues such as increased overhead due to a lot of locks acquire/release requests, as well as increase the possibility of deadlock.

The Python developers instead chose to solve this problem by placing a lock around the entire interpreter, making each thread acquire this lock when it runs Python bytecode. This avoids a lot of the performance issues around excessive locking but effectively serializes bytecode execution.

Why use Multithreading when GIL exists -

Suppose a Web application that receives requests from clients, does some query to the database and return the response to the client. While Waiting for IO to complete may take 90% (or more than that) of the time the request is processed. When a single-threaded application is waiting for IO and it just not using the core and the core is available for execution. So such application has a room for other threads to execute even on a single core.

So in this kind of case when one thread waiting for IO, it releases GIL and another thread can continue execution.

Benefits -

It is faster in the single-threaded case.
It is faster in the multi-threaded case for IO-bound programs.
It is faster in the multi-threaded case for CPU-bound programs that do their compute-intensive work in C libraries.

Quote from the Python threading documentation
In CPython, due to the Global Interpreter Lock, only one thread can execute Python code at once (even though certain performance-oriented libraries might overcome this limitation). If you want your application to make better use of the computational resources of multi-core machines, you are advised to use multiprocessing or concurrent.futures.ProcessPoolExecutor.
However, threading is still an appropriate model if you want to run multiple I/O-bound tasks simultaneously.

The GIL is a problem if, and only if, you are doing CPU-intensive work in pure Python. Many Python libraries solve this issue by using C extensions to bypass the GIL.

Reference

https://rohanvarma.me/GIL/

I hope that you now have a fair understanding of the Global Interpreter Lock in python.

If you have any suggestions on your mind, please let me know in the comments.

Garbage Collection in Python

Prashant Sharma — Sat, 23 Nov 2019 06:45:10 +0000

In languages like C or C++, the programmer is responsible for dynamic allocation and deallocation of memory on the heap. But in python programmer does not have to preallocate or deallocate memory.

Python uses following garbage collection algorithms for memory management -

Reference counting
Cycle-detecting algorithm (Circular References)

Reference counting

Reference counting is a simple procedure where referenced objects are deallocated when there is no reference to them in a program.
In short, When the reference count becomes 0, the object is deallocated(frees its allocated memory).

Let's have a look at below example -

def calculate_sum(num1, num2):
    total = num1 + num2
    print(total)

In the above example, We have three local references num1, num2 and total. Here total is different from num1 and num2 because it only has reference inside the block thus its reference count is 1, num1 and num2 referenced outside the block so maybe their reference count more than one.

So, here when the function has finished execution the referenced count of total reduced to 0. Since it tracked by the garbage collector.

The garbage collector finds out that the total is no longer referenced(reference count field reaches 0) and frees its allocated memory.

The Variables, which are declared outside of functions, such variables do not get destroyed even after function has finished execution.

We can do the manual deletion also using the del statement. del statement removes a variable and its reference. When the reference count reaches 0, it will be collected by the garbage collector.

The reference counting algorithm has some issues also, such as circular references -

Circular References

A reference cycle occurs when one or more objects are referencing each other.

As you can see in the above image, the list object is pointing to itself, and object1 and object2 are pointing to each other. The reference count for such objects is always at least 1.

Let's go to the practical -

import gc

gc.set_debug(gc.DEBUG_SAVEALL) 

lst = []
lst.append(lst)

lst_address = id(lst)

del lst

object_1 = {}
object_2 = {}
object_1['obj2'] = object_2
object_2['obj1'] = object_1

obj_address = id(object_1)

del object_1, object_2

In the above example, the del statement removes the variable and its reference to the objects.

Let's check-it deleted variables using gc.collect, gc.collect saves to gc.garbage instead of deleting.

>>> gc.collect()
3

When we delete a variable, we only delete the __main__ reference. Now we don’t have access to lst, object_1 and object_2 at all, but these variables still have 1 reference, it means reference count is 1, the reference count algorithm will not collect it.

Check the reference count as below-

import sys
print(sys.getrefcount(obj_address))
print(sys.getrefcount(lst_address))

2
2
# 1 from the variable and 1 from getrefcount

Multiply this number by 1 Million objects and you may have absolutely a serious memory leak issue.

For this kind of Reference cycle, Python has another algorithm specially dedicated to discovering and destroying circular references. It is also the only controllable part of Python’s GC

Summary

Python has 2 Garbage Collection algorithms. One for dealing with reference count, When the reference count reaches 0, it removes the object and frees its allocated memory. The other is the Cycle-detecting algorithm which discovers and destroys circular references.

I hope that you now have a fair understanding of the Garbage Collection algorithms in python.

If you have any suggestions on your mind, please let me know in the comments.

Reference

https://rushter.com/blog/python-garbage-collector/

Securely Transfer Files from/to a Remote Server using SCP

Prashant Sharma — Wed, 20 Nov 2019 18:20:52 +0000

The SCP ( Secure Copy Protocol ) is a network protocol, based on the BSD RCP protocol, which supports file transfers between hosts on a network. SCP uses Secure Shell (SSH) for data transfer and uses the same mechanisms for authentication.

Using SCP you can copy file/directory :

From your local machine to a remote system.
From a remote system to your local system.
From one remote system to another remote system from your local system

While transferring data using SCP, files and password is encrypted, so that anyone snooping on the traffic doesn’t get anything sensitive.

Things to keep in mind before the start -

The scp command relies on ssh for transferring the data, so it requires an ssh key or password to authenticate on the remote systems.
To be able to copy file/directory you must have at least read permissions on the source file and write permission on the target system.

Syntax

The syntax for the scp command is:

scp [options] username@source_host:directory/filename  /where/to/put

In the below examples I Recursively copying entire directories -

Examples

From remote to local -

scp -r username@ipaddress:/directory/to/send /local/where/to/put

From local to remote -

scp -r /local/directory/to/send username@ipaddress:/where/to/put

Copying between two remote hosts -

scp -r username@ipaddress1:/file/to/send username@ipaddress2:/where/to/put

You can use scp with the following options according to your requirements.

Options

scp –P port : Generally 22 as a default port of scp. You can also specify a specific port.
scp –p : An estimated time and the connection speed will appear on the screen.
scp –q : Disable progress meter and warning.
scp –r : Recursively copy entire directories.
scp –v : Print debug information into the screen. It can help you debugging connection, authentication and configuration problems.
scp -c : By default SCP using “AES-128” to encrypt files. If you want to change to another cipher to encrypt it, you can use “-c”.

I hope that you now have understood how to make the best use of scp command to securely transfer files between the systems.

If you have any suggestions on your mind, please let me know in the comments. And if you know any other awesome scp command, do share with us.

Increase maximum recursion depth in python using Context Manager

Prashant Sharma — Tue, 19 Nov 2019 17:30:11 +0000

A context manager is an object that defines the runtime context to be established when executing a with statement. The context manager handles the entry into, and the exit from, the desired runtime context for the execution of the block of code.

Let's have a look at below example, suppose I want to calculate the Fibonacci of a number -

def fib_cal(fib_num, memo):
    if memo[fib_num] is not None:
        return memo[fib_num]
    elif fib_num == 1 or fib_num == 2:
        result = 1
    else:
        result = fib_cal(fib_num-1, memo) + fib_cal(fib_num-2, memo)
    memo[fib_num] = result
    return result


def get_fibonacci(fib_num):
    memo = [None] * (fib_num+1)
    return fib_cal(fib_num, memo)

print(fibonacci(100))

Output- 354224848179261915075

Let suppose if my number if bigger like - 3000

print(fibonacci(3000))

In the above case, it’ll throw an error-

RecursionError: maximum recursion depth exceeded in comparison

because the maximum recursion limit of the os has been exceeded. you can check recursion limit as follow -

import sys
sys.getrecursionlimit()

So, In this kind of situation, we can use context manager which will allow us to allocate and release resources precisely when we want to.

import sys

class RecursionLimit:
    def __init__(self, limit):
        self.limit = limit
        self.cur_limit = sys.getrecursionlimit()

    def __enter__(self):
        sys.setrecursionlimit(self.limit)

    def __exit__(self, exc_type, exc_value, exc_traceback):
        sys.setrecursionlimit(self.cur_limit)


MAX_LIMIT = 10000

with RecursionLimit(MAX_LIMIT):
    print(fibonacci(3000))

Output- 4106158863079712603335683787192671052201251086373692524........6000

The __enter__() returns the resource that needs to be managed and the __exit__() does not return anything but performs the cleanup operations.

Context-managers can be used for other purposes also like- simple file I/O, like opening and closing sockets, implementing set-up and tear-down functionality during testing.

I hope you enjoyed reading this article, If you have any suggestions on your mind, please let me know in the comments.

When to use continue, pass and break in python.

Prashant Sharma — Mon, 18 Nov 2019 20:53:43 +0000

continue

continue forces the loop to start at the next iteration. In short, continue will jump back to the top of the loop

a = [0, 1, 2]
for element in a:
    if not element:
        continue
    print(element)

output-

1
2

UseCase: The continue statement can be used to skip the current iteration only. Loop does not terminate but continues on with the next iteration.

pass

while pass means there is no code to execute here, it passes execution to the next statement. So pass will continue processing.

a = [0, 1, 2]
for element in a:
     if not element:
         pass
     print(element)

output-

0
1
2

UseCase: You can use pass statement when you have created a method but have not implemented, yet.

break

The break statement provides you with the opportunity to exit out of a loop when a given condition is triggered.

a = [0, 1, 2, 3, 4, 5]
for element in a:
    if element == 3:
        break
    print(element)

output-

0
1
2

UseCase: The break statement can use in a program to break out of a loop for the matching condition.

How to Return JSON-Encoded Response for non-dict object

Prashant Sharma — Mon, 18 Nov 2019 20:22:51 +0000

An HttpResponse subclass that helps to create a JSON-encoded response. Its default Content-Type header is set to application/JSON.

The first parameter, data, should be a dict instance. If non-dict object is passed as the first argument, a TypeError will be raised.

Example dict instance:


from django.http import JsonResponse

def user(req):
    response = {
        'id': 1,
        'name': 'Tom',
        'status': Active
    }
    return JsonResponse(response)

Example non-dict object:

return JsonResponse([1, 2, 3, 4])

TypeError: In order to allow non-dict objects to be serialized set the safe parameter to False

By default, the JsonResponse’s first parameter, data, should be a dict instance. To pass any other JSON-serializable object you must set the safe parameter to False.

return JsonResponse([1, 2, 3, 4], safe=False)

The safe boolean parameter defaults to True. If it’s set to False, any object can be passed for serialization (otherwise only dict instances are allowed).

DEV Community: Prashant Sharma

Metaclasses in Python

Understanding How Metaclasses Work

Conclusion

Reference

Monkey Patching In Python

What is Monkey patching

Things to keep in mind

Conclusion

Reference

SpeedUp Python List and Dictionary

Avoid Re-evaluation in Lists

Avoid Re-evaluation in Dictionary

Mutable Default Arguments in Python

Why is this happening

The Solution

Good Use of Mutable Default Argument

Assignment vs Shallow Copy vs Deep Copy in Python

Assignment Operator (=)

Shallow Copy

Deep copy

Pickling and Unpickling in python Explained

Pickle protocols:

Pickling

Unpickling

What can be pickled?

Common UseCase of Pickling

Dangers of Pickling

Safe implementation of Pickle

Reference

Global Interpreter Lock (GIL) in Python

How two threads perform both CPU and I/O operations during execution -

Why CPython use GIL -

Why use Multithreading when GIL exists -

Benefits -

Reference

Garbage Collection in Python

Reference counting

Circular References

Summary

Reference

Securely Transfer Files from/to a Remote Server using SCP

Syntax

Examples

Options

Increase maximum recursion depth in python using Context Manager

When to use continue, pass and break in python.

continue

pass

break

How to Return JSON-Encoded Response for non-dict object

Example dict instance:

Example non-dict object:

How two threads perform both CPU and `I/O` operations during execution -