Comparing C++ RAII and Python Context Managers

I was reading up on the practice in C++ called Resource Acquisition Is Initialization or RAII, after hearing about it on a podcast. I mostly spend my time programming in Python and this made me think about context managers and the with statement in Python. Both RAII and context managers are methods for making sure a resource is released when you leave the current scope.

A while ago I watched the talk Python as C++’s Limiting Case by Brandon Rhodes, where he compares Python to C++. I recommend you to watch the talk, as it's an interesting way of exploring Python. I would summarize the premise as the following, using the same comparison as the author:
As the circle is the limiting case of a polygon when the number of edges goes to infinity, so is Python the limiting case of C++ when taking the best ideas of C++ to their extremes.

With that as a background, I will add another example of how Python indeed might be the limiting case of C++. As mentioned in the introduction, the C++ community has found RAII as a good method for making sure resources are released. But before getting into RAII, what is a resource? It can be many different things, but in this context, it is anything that has to be acquired and released. In C++, this can be allocating memory on the heap, something we Pythonistas doesn't have to think about. However, it can also be a file, socket, database-session or mutex-lock. All of these and many more examples exist in both Python and C++.

Now, what is RAII? It is a practice where the resource is bound to the lifetime of the object. This is done by defining a class where the resource is acquired in the constructor and released in the destructor. When you create an instance of that class the resource will be bound to that instance until it is destroyed. If you do not manually/dynamically allocate the new instance, it will be destroyed upon leaving the current scope. This makes it really handy, as in C++ you can just enclose a piece of code in curly-brackets to create a new scope. However, for this to work, there is one final condition for an RAII class. The destructor is not allowed to raise an error, as it has to finish to guarantee the resource is released. In the introduction, I linked this page on RAII at cppreference.com, which has a description of the practice. They also have a good example, comparing RAII vs non-RAII C++ code for locks, so if you would like an example I recommend you check it out. However, for this post, the most important part to understand about RAII is that:

• RAII allows C++ developers to make sure a resource is released when leaving the current scope. When the program leaves the scope no matter what, even if it is through an early return or raising an exception, the resource is released.

Sidenote: cppreference is my favorite resource for C++ documentation and a good place to start if you are interested in diving deeper into RAII.

Whether you are a seasoned Pythonista or are relatively new to the language, if you ever worked with files you probably have written something like this:

with open("my_file.txt", "r") as my_file:
    my_data = my_file.read()

This is the humble context manager, a pythonic way of handling resources. Similar to RAII, context managers bind the resource to a scope, releasing it when leaving the scope. In this case, as long as you are inside the context manager the file is open and as soon as you leave it is closed.

Creating your own context manager is quite easy as well. Either, you do it by creating a class which defines the __enter__ and __exit__ methods or using the contextmanager decorator. I will use the first method for this example as it makes the comparison to RAII clearer. Then, we can create a self-printing tempfile context manager (which of course is extremely useful) like this:

from tempfile import TemporaryFile

class PrintingTempFileContext:
    def __enter__(self):
        print("<Opening File>")
        self.file = TemporaryFile(mode="w+t")
        return self.file

    def __exit__(self, exception_type, exception_value, traceback):
        print(f"<Exception info: {exception_type=} - {exception_value=} - {traceback=}>")
        self.file.seek(0)
        print(self.file.read())
        self.file.close()
        print("<File closed>")

Running this code:

with PrintingTempFileContext() as tempfile:
    tempfile.write("Hello DEV!")

Output:

<Opening File>
<Exception info: exception_type=None - exception_value=None - traceback=None>
Hello DEV!
<File closed>

Now, what would happen if we raised an error inside the context manager? Let us try to run this:

with PrintingTempFileContext() as tempfile:
    tempfile.write("Hello DEV!")
    raise RuntimeError

Output:

<Opening File>
<Exception info: exception_type=<class 'RuntimeError'> - exception_value=RuntimeError() - traceback=<traceback object at 0x000002041C5FE9C0>>
Hello DEV!
<File closed>
Traceback (most recent call last):
  File ".../print_tempfile_context.py", line 19, in <module>
    raise RuntimeError
RuntimeError

The __exit__ method is still called and the file is closed. So no matter how we exit the scope of the context manager, the clean up will get run. This sounds an awful lot like using an RAII-class inside a scope, doesn't it? I said I chose this method of implementing the context manager as it is easier to compare to RAII. This is because the __enter__ method and the constructor of an RAII-class does the same type of work. They both bind the resource we want to acquire, in this example an instance of TemporaryFile. Similarly, the __exit__ method does the same work as the destructor of an RAII-class. Therefore, it is also important that the __exit__ method doesn't raise an error. Similar to the destructor of an RAII-class this could result in the resource not being released upon leaving the scope.

To conclude, I want to answer how this relates to the talk, Python as C++’s Limiting Case. My hope is that I have shown how context managers and RAII are very similar in how they solve the problem of managing resources. Furthermore, context managers build upon the practice of RAII by having dedicated syntax, the with-statement. Therefore, I argue that context managers are the limiting case of RAII. As giving a practice its own syntax is among the clearest ways a programing language has to endorse a practice. So this adds one more argument, to those given by Brandon Rhodes, to why Python is the Limiting Case of C++.

Comments

[thitkhotauhp9x]

What are the issues if I use init and del to simulate RAII in Python?

[Fredrik Sjöstrand]

The main reason to use the with statement is that it is more "pythonic". It is a great tool that is designed for resource management and is generally accepted as the defacto solution. But, I know that isn't a satsifying answer so let's look at what makes __del__ bad for resource management.

This qoute from this part of the python docs

It is not guaranteed that __del__() methods are called for objects that still exist when the interpreter exits.

highligts the, according to me largest issue. It might just not be called at all. Depending on what clean up is required for the resource, that is a big problem.

Scopes also works quite differently in Python, for example we don't have "statement scop" like C++ have for if, for, etc. This makes it harder to scope it using something other than a function-scopes. Also the del keyword works like this:

Note del x doesn’t directly call x.__del__() — the former decrements the reference count for x by one, and the latter is only called when x’s reference count reaches zero.

This is from the same part of the docs. Both the scoping and how del works might cause a resource to be allocated longer than we intended.

So, that is my take on this. I hope my thought process is, at least, sort of clear. Also, if someone else want to chime in, don't be shy 😁

[Swastik Baranwal]

Amazing explaination!

[Fredrik Sjöstrand]

Thank you! 😁