DEV Community: Pradhvan Bisht

Context Managers in Python

Pradhvan Bisht — Sat, 20 Jul 2019 17:59:00 +0000

Context Managers in Python

Context Managers in Python helps the user to manage the resources efficiently in a program i.e opening or closing a resource or locking or unlocking a resource. Context Managers come handy when we have to manage resource before and after an event.

The most common context manager that every python programmer uses very frequently for managing files is a With as statement.

with open('MyContextManager.txt') as f:
    f.write('Using Context Manager is FUN !')

The above code snippet has mainly two advantage:

Helps to close resources automatically and effectively. This might be a small code block, so it might a bit obvious that the file was open and closed properly but what if when the scope of the function increases? This is why context managers really come into the picture.
Makes the code readable and complex logic simple. The above code can also be written as:

file = open('MyContextManager.txt','W')
try:
    file.write('Not using a Context Manager.')
finally:
    file.close()

Here we manage the opening and closing of a file manually with a try-finally statement.

Python's standard library comes with a module, contextlib. This contains utilities for working with context managers and the with statement.

Writing Your Own Context Manager

So why would someone want to write there own Context Managers?

Because, Context Managers are the best at managing resources before and after an event. Thus you don't have to worry about the allocation/de-allocation or Locking/Unlocking or Opening/Closing.

Also, they make the code simple and more readable.

Writing your context manager can be done in two ways either create your own class or use the Contextlib module to create a Context Manager decorator.

Let's first look at how we can create a simple Context Manager class. A Context Manager class consists of two main methods enter and exit. If you're familiar with testing you can compare these two methods with the setup and teardown.

Just like in every class in Python, the init method is optional. But in the case of Context Managers,we use init only if we're using a with as statement. init has to be passed the name which you want to associate with as in the with as statement.

Now let's take a look at a simple Game of Thrones inspired ContextManager which creates a dict of the house symbols.

```PythonJust like every classJust like every class in Python, the init method is optional only if you're using a with * statement. If you're using *with as statement it has to be passed the name which you want to associate with as in the with as statement. in Python, the init method is optional only if you're using a with * statement. If you're using *with as statement it has to be passed the name which you want to associate with as in the with as statement.
class ThronesContextManager:
def init(self, HouseSymbol):
self.HouseSymbol = HouseSymbol

def __enter__(self):
    print("Enter: {}".format(self.HouseSymbol)")
    return self.HouseSymbol

def __exit__(self, *exc):
    print("Exit: {}".format(self.HouseSymbol))

with ThronesContextManager({"House Stark": "Wolf"}) as HouseSymbol:
HouseSymbol["Targaryen"] = "Three Headed Dragon"

"""
---Output---
Enter: {'House Stark': 'Wolf'}
Exit: {'House Stark': 'Wolf', 'Targaryen': 'Three Headed Dragon'}
"""





* The init method takes in the dict associated with the *as* in the *with-as* statement. It creates an instance of the class and assigns it to the dict. Much similar to any normal Python Class.
* The *enter* method is called by the *with* and is passed the dict. It returns the value which is associated with the dict(HouseSymbol).
*  The *exit* takes in the exception(*exc) these are of mainly three types exc: exception, exc_type: exception type and exc_tb: exception_traceback. 
*  If for some reason you want the program to ignore the exception you can also return True to just ignore the exception.


Now taking a look at the above code example we can say that any Context Manager is one which as two methods an *enter* method and an *exit* method. 

Before moving forward to *contextmanager decorator* let's break down the code snippit we saw in the starting of the post and see how it works behind the hood. 

Since we know how context managers work it won't be difficult to the observer what's happening when we call *with as* statement while opening a file.




```Python
with open('MyContextManager.txt') as f:
    f.write('Using Context Manager is FUN !')

With calls the enter method of the File class.
The enter method opens the file and returns it.
The opened file handle is passed to f.
We write to the file using .write().
The with statement calls the exit method.
The exit checks for exceptions, if no exception is found it closes the file.

The easier way to write a context manager is by using the Contextlib module and creating a context manager decorator.

The good thing about using the @contextmanager is that it builds the enter and exit method for you so automatically.Thus we can transform a generator function into a contextmanager decorator.

Let's re-write the ThronesContextManager again but with a @ThronesContextManager.

from contextlib import contextmanager

@contextmanager
def ThronesContextManager(data):
    print("Enter: {}".format(data))
    yield data 
    print("Exit: {}".format(data))

with ThronesContextManager({"House Stark": "Wolf"}) as HouseSymbol:
    HouseSymbol["Targaryen"] = "Three Headed Dragon"

"""
---Output---
Enter: {'House Stark': 'Wolf'}
Exit: {'House Stark': 'Wolf', 'Targaryen': 'Three Headed Dragon'}
"""

PyRandom

Here are some things interesting things I found about Contextmanagers. These were found when I was researching for this blog post thus I am adding this to the section PyRandom. I would be constanly updating this section as I learn more about Contex Managers.

Context Managers do not create a separate new scope in the program i.e variables defined inside the withas block will be available after the block is executed.

with open('MyContextManager.txt') as f:
    # Variable defined inside the Context Manager
    VariableName = f.read()
print(VariableName)

When using the multiple ContextManager in a withas statement the flow of enter and exit statement becomes LIFO(Last In First Out) i.e the enter method that is called last will have it's exit method called first.

import contextlib

@contextlib.contextmanager
def make_context(name):
    print ('entering:', name)
    yield name
    print ('exiting :', name)

with make_context('A') as A, make_context('B') as B, make_context('C') as C:
    print ('inside with statement:', A, B, C)

"""
---OUTPUT---
entering: A
entering: B
entering: C
inside with statement: A B C
exiting : C
exiting : B
exiting : A
"""

What now ?

Since we covered all the basic stuff on Context Managers, we can start digging deeper and learn to use Context Managers in a more real-life use case. So here are some things that I would like you to read more about it.

Learn how to handle exceptions in/with Context Managers.
Try to find out real project use cases where using a Context Manager would be best suited.
Find out the role of init and enter in the Context Manager.

Still can't get enough ?

The reason behind the blog is that I recently picked a Python problem which goes something like this

Write a Context Manager named Suppress which suppresses the exception of a given type/types i.e if the given exception type is raised, that exception should be caught and muted in a sense.

Code Example:

>>> x = 0
>>> with suppress(ValueError):
...     x = int('hello')
...
>>> x
0
>>> with suppress(ValueError, TypeError):
...     x = int(None)
...
>>> x
0

Since you read this far I am assuming you are also just starting to learn about this topic. Let's put to the put to the test what you have learned so far and get some hands-on experience of writing your Context Manager. Try to solve this problem.

I am still solving the problem and once it's done I would link my solution here.

Happy Coding 😄

Memory Managment in Python – Part 2

Pradhvan Bisht — Wed, 19 Dec 2018 12:28:42 +0000

In the last part, we checked out how variables are stored in Python and how Python handles memory management with reference counts and garbage collector. If you haven't checked it out and want to here is the link.

In this part, we will dig deep a little on how reference counting works and how it can increase or decrease taking in account the different cases. So let's start where we left off, every object in python has three things

Type
Reference Count
Value

Reference Count is a value showing how much an object has been referred(pointed) too by other names(variables). Reference counting helps the garbage collector in freeing up space so the program can run efficiently. Though we can increase or decrease the value of the reference count and can check the value with the inbuilt function called getrefcount().

Let's take a small code snippet:

import sys

a = []

# Two reference one from the variable and one from the getrefcount() function

print(sys.getrefcount())

2

Though the examples look great and everything seems to be working but I did kinda trick you, first thing is that not all reference count values start from 0 so if you do the same example with a different value of the output may be different. Reference count values are calculated on two factors number of times the object is used in the bytecode and the number of time it’s been referenced to this includes your previous programs too.

Let’s look into another example:


import sys

a = 100

print(sys.getrefcount(a))

4

b = 100

print(sys.getrefcount(b))

5

c = 100

print(sys.getrefcount(c))

6

When more variables reference to the same value the reference count increase. But this is not the case when we take into example the case of container objects like lists and constants.

import sys

ex_list = [a,b,c,d]

print(sys.getrefcount(a))

8

print(sys.getrefcount(b))

9

print(sys.getrefcount(c))

10

print(sys.getrefcount(d))

11

del ex_list

print(sys.getrefcount(a))

7

print(sys.getrefcount(b))

8

print(sys.getrefcount(c))

9

print(sys.getrefcount(d))

10

# Same thing goes with constants

print(sys.getrefcount(10))

12

const = 10

print(sys.getrefcount(10))

13

const = const + 10

print(sys.getrefcount(10))

12

As we saw container objects here list refer to other objects referring to the value thus when we delete them the reference link is deleted thus objects inside the list decrease the reference count by one. The same happens with constants too, when the variable they get referenced to is incremented the reference count is decremented.

By now you must have realized that del does not actually delete the object on the contrary it removes that variable(name) as a reference to that object and decrease the reference count by one.

All the example we saw are kinda similar considering the fact that they are in the global namespace but what happens when there is function what happens to the reference count then, let’s find out through this code snippet.

import sys

num = 100

print(sys.getrefcount(num))

4

def ytf(number):

print(sys.getrefcount(num))

ytf(num)

6

print(sys.getrefcount(num))

4

We saw that when ytf() got into scope the reference count increased while the reference count decreased when the function got out of scope. Keeping this in mind that we should be careful of using large or complex objects in the global namespace because an object in a global namespace don’t go out of scope unless we decrease the value of the reference count thus a large object would consume more memory making the program less efficient.

That’s all for this part, in the next part we would look closely into the garbage collector and how it functions inside a python program in freeing up memory.

Memory Managment in Python – Part 1

Pradhvan Bisht — Sun, 16 Dec 2018 14:37:28 +0000

Stumbling upon a Python code snippet from a GitHub repo I did come to realize that in Python variables don’t actually store values they are assigned too, variables actually store the location to the value. Unlike in C++/C which actually creates a space of a fixed size and assigns it to a variable created which we usually call a bucket/room while explaining variables to a beginner “what variables are?” in programming.

Thought python variables are a bit different they work like keys which points to a particular room in the hotel(memory space) so whenever we make an assignment to available we are not creating rooms rather creating keys to that room which is freed/overwritten by the python’s garbage collector automatically. (More on the topic of garbage collector later). So the point is whenever we do something like

>>>a = 10

>>>b = 10

>>>id(a)

94268504788576

>>>id(b)

94268504788576

We are optimizing here and creating two same keys which points to the same room in the hotel (memory) thus they would have the same id but this kind of optimization works only with the range of integers from -5 to 256 if you exceed the range, variables would point to two different storages thus will have different id().

Just don’t get confused why we did not use “==” instead used id(), == checks whether the value pointed by the variables are same or not inside the object and id() checks if it uses the same object or not because every object has a unique identity which can be checked by the id() function.

Coming back to the code snippet from the GitHub repo given below and applying the same knowledge of integers to strings.


>>>a = "wtf"

>>>b = "wtf"

>>>id(a),id(b)

(139942771029080, 139942771029080)

>>>a = "wtf!"

>>>b = "wtf!"

>>>id(a),id(b)

(139942771029192, 139942771029136)

>>>a = "hello world this is a string"

>>>a = "hello world this is a string"

>>>id(a),id(b)

(139942770977328, 139942770977408)

The same kind of optimization happens here too when the strings are small they refer to the same object in the memory rather creating a new one thus saving memory, this kind is called interning. But when the object becomes bigger or contains ASCII letters, digits or underscore they are not interned.

This shows abstraction at it’s best and python is very good at it, it does all the heavy lifting job for you of allocating/deallocating memory for you and lets you focus on the other parts of the program. Until you really want to know what’s happening and I assume you do that’s why you are reading this blog 😛

Though this explanation was already available on the repo. I wanted to know more about how memory management happens internally so I stumbled about a talk of “ Memory Managment in Python – The basics” by nnja. So yeah people with nicks like ninja are great with Python and Fortnite, hahaha! (I could not resist myself from posting this joke and just to clear things out ninja is one of the top Fortnite players)

Thus if you see technically Python does not have variable but ‘names’ which refers to other objects or names and Python likes to keep a count of all the references called as reference count of all the object’s references. So if the reference count of an object decreases to zero this means no reference is made to that object which as seen by the Garbage collector of python as free space thus the object is deleted and space is free to use.

We as a programmer can increase or decrease the reference count of an object as a python object stores three things:

* Type: int,string,float
* Reference Count
* Value

Seeing the first code snippet of the blog where two names a and b in which both names points to the same object with the value 10 with the reference count of 1 and type as int.

That’s all for this part, will cover the same topics in a bit detail in the next part. Still confused in some of the things so keeping this plain and simple for future me to look back when I am lost and can’t even notice the simple things 😛 and for someone who just wants to know this briefly.