Lucy Linder

Posted on Jan 9 • Originally published at blog.derlin.ch

Diving Deeper into Python Exceptions

#python #learning #development #coding

I have been coding in Python for a long time, yet I am puzzled by how little I knew about Exceptions. This post is about some of my recent findings on this topic.

Content

A little story
Exception chaining (and the magic of __context__ )
Bare except vs except Exception
Raising shorthands
Annotating exceptions (3.11+)
What about UserWarning?
Bonus

A little story

I had an interesting use case at work lately: some external library code was "swallowing" another exception, and I needed to get back the message of the initial one somehow, without touching the library itself.

The library code looked something like this:

class TokenBackend:
   def decode(self, token: str) -> dict:
      # Decode a JWT token,
      # It may fail for many reasons, detailed in the
      # TokenError message
      if not token:
         raise TokenError("Empty token")
      if len(token.split(".")):
         raise TokenError("A JWT must have 3 parts")
      if ...:
         raise TokenError("Invalid signature")
      # ...

class Token:

   def __init__(self, raw_token):
      # ...
      try:
         self.backend.decode(token) # <- calls TokenBackend.decode
      except TokenError:
         # Here, the library swallows the exception,
         # we LOSE the detailed error message!
         raise DecodeError("Token could not be decoded")

I had no clue how to do this except to override the Token class in my codebase. When I asked my boss about this, he looked at me and said: "just use __context__". Huh? Never heard of it. I started digging, and long story short: he was right. This was the perfect solution.

Those small discoveries happened a lot lately, and I wanted to share them. If this intrigues you, keep reading!

Exception chaining (and the magic of `context` )

So, what is this __context__??

Formalised in PEP 3134 (I love PEPs), exceptions in Python 3 have three dunder attributes that provide information about the context in which they were raised: __cause__, __context__ and __suppress_context__. To understand, let's first make sense of implicit and explicit exception chaining.

An exception chain starts when a new exception is raised during the handling of another, for example from an except clause:

try:
    open("foo.bar")
except OSError:
    raise RuntimeError("oops")

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
FileNotFoundError: [Errno 2] No such file or directory: 'foo.bar'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
RuntimeError: oops

This is called an implicit chain (hence the "During handling ..."). To make it explicit and clearly state an exception is the cause of another, one can use the special raise ... from :

try:
    open("foo.bar")
except OSError as e:
    raise RuntimeError("oops") from e

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
FileNotFoundError: [Errno 2] No such file or directory: 'foo.bar'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
RuntimeError: oops

As you can see, we now have another log message: "was the direct cause of". Of course, chains can be longer than two.

To go back to the context attributes of an exception:

__suppress_context__ is false by default.
When raising a new exception while another exception is already being handled (in except, finally or with), the new exception’s __context__ attribute is automatically set to the handled exception.
When using raise ... from , the supplied exception will additionally be saved in the __cause__ attribute of the raised exception, and __suppress_context__ will be set to true.

The default traceback uses those attributes to display stacktraces in the following way:

if __cause__ is present, always show it
if __cause__ is None, show the __context__ only if __suppress_context__ is false.

To ensure you followed, what does this valid Python code prints?

try:
   1/0
except ZeroDivisionError:
   raise RuntimeError("zero!") from None

It only shows the RuntimeError, because the from will set the __cause__ (to None) and the __suppress_context__ (to true).

Now, this doesn't completely swallow the original exception. Even when using a from, the initial ZeroDivisionError is still in the __context__, just ignored when printing the stacktrace.

Back to the problem in the introduction, I simply catch the exception e raised by the library (in Token.__init__), and then use e.__context__.args[0] to get the initial exception message.

Bare except vs except Exception

I learned this one from the ruff rule bare-except (E722). When you don't care about which exception is raised, you may be tempted to use a bare except (but should NOT):

try:
   do_something()
except: # <- no exception class is called bare except
   print("oops")

A bare except catches BaseException

BaseException is the common base class of all exceptions. One of its subclasses, Exception, is the base class of all the non-fatal exceptions. Exceptions which are not subclasses of Exception are not typically handled, because they are used to indicate that the program should terminate.

This except thus catches Exception, but also KeyboardInterrupt, SystemExit, and other fatal errors, making it hard to interrupt the program (e.g., with Ctrl-C) and potentially disguising other problems or leaving the program in an unexpected state.

So instead, always specify an exception type, or simply Exception if you are in doubt:

try:
   do_something()
except Exception: # now we are good
   print("oops")

Raising shorthands

When re-raising inside an except clause, you don't need to pass an argument to raise, as it re-raises the caught exception by default.

try:
   x / y
except ZeroDivisionError: # no need to add "as e"
   log.error("we got a zero here")
   raise

Similarly, when raising an exception with no argument, no need for parentheses. If an exception class is passed to raise, it will be implicitly instantiated by calling its constructor with no arguments.

Hence the following is perfectly valid and more concise (see ruff rule unnecessary-paren-on-raise-exception (RSE102)):

raise ValueError

Annotating exceptions (3.11+)

Since Python 3.11, it is possible to attach notes to exceptions, effectively enriching their context. This is a very interesting feature, that could replace re-raising an exception with a different message.

try:
  try:
    raise ValueError
  except Exception as e:
    e.add_note("This was raised as an example")
    raise
except Exception as e:
  e.add_note("Great article btw!")
  raise

Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
ValueError
This was raised as an example
Great article btw!

Those notes are saved in the __notes__ attribute (list of strings).

What about `UserWarning`?

If you look at the Python documentation, you will see some strange built-in exceptions such as UserWarning, DeprecationWarning, etc. They all inherit from Warning (which itself inherits from Exception) but they are NOT meant to be raised. Instead, they are used as warning categories.

In short, Warning exceptions are to be used with the warnings module. There is much more to it, but let's look at a simple example:

import warnings

def foo():
    # Explicit category
    warnings.warn("Don't use me anymore!", DeprecationWarning)
    # Implicit category
    warnings.warn("bar") # <- default to UserWarning

What is nice about warning is that users have complete control over what is reported, thanks to the warning filter:

foo()
# <stdin>:3: DeprecationWarning: Don't use me anymore!
# <stdin>:5: UserWarning: bar
foo() # second time, no more warnings printed
---
warnings.simplefilter("ignore")
foo() # -> nothing printed
---
warnings.simplefilter("error")
foo() # -> raises!
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
#   File "<stdin>", line 3, in foo
# DeprecationWarning: Don't use me anymore!

For all available filters, see The Warnings Filter.

So, why use exceptions for that? You guessed it, it simplifies turning warnings into exceptions (error filter): one just has to raise it.

Bonus

This article is already way too long, so here is a bullet list of other interesting subjects and picks:

Python 3.11 introduced ExceptionGroup, a nice way to pack multiple exceptions into one. The new syntax except* allows filtering groups efficiently. Find out more in the documentation.
Python supports try-except-else-finally, although I never found a good use case for the else (also supported in for loops). The else block is executed after the try block, but before the finally block (if the except block doesn't run). Exceptions raised inside the else block are not caught by the except. See Handling exceptions for more info.
The NotImplementedError (not to confuse with the constant NotImplemented) signals a missing implementation that should come one day. If the feature will never be implemented, raise a TypeError instead.
Exceptions store their __init__ arguments in the args attribute.
I am always tempted to name my custom exception classes with the Exception suffix (a remnant of Java perhaps?). However, PEP 8 clearly states we should use the suffix Error for exception class names.
It is possible to catch multiple exceptions using parentheses: except (FooException, BarException)
Never return in a finally: this will override whatever return you may have inside the try or the except.