What to Except When You're Excepting: Python error handling dos & don'ts

Exceptions in Python are a great part of the language. They're easy to use, easy to subclass, and pretty fast. Unfortunately, convenience can lead to sloppy thinking and sloppier programming. In my 5+ years of Python programming in various startups, I've encountered a lot of bad exception handling, so here are some recommendations for dos and don'ts.

Do think about your exceptions

Towards the bottom of the post I linked to above, the author provides evidence that try/catch can be slower than if statements when the errors are frequent. His somewhat flippant advice is "to look into more reliable infrastructure," which is helpful in some contexts. I've found, however, that the convenience of Python exceptions, coupled with the dynamically-typed nature of the language itself, leads to developers either taking exceptions for granted, over-using try/catch, or both.

One advantage of statically-typed languages like Haskell and Rust is that they force the programmer to think more deliberately about errors, and provide them with the language constructs to do so in the form of Maybe/Either (in Haskell) or Option/Result (in Rust.)

[For those unfamiliar, Maybe/Option are ways to avoid null, aka undefined aka None aka Hoare's 'billion-dollar mistake'; in Haskell, a value of type Maybe a can either be Just a (e.g. Just "a friend"), or Nothing. Similarly, a value of type Either a b can be either (get it?) Left a (e.g. Left "for dead") or Right b (e.g. Right 1). Both languages use one half of their Either-like types for "success" values, and the other for "error" values; it's common to see things like Result<Int,String> in Rust or Either String Int in Haskell. (I know why Haskell orders it the way it does (ask me in the comments if you care), I don't know why Rust switched the order.)]

Both of these languages have "real" exceptions, but most Haskell and Rust programmers eschew them in favor of Maybe/Either/Option/Result. Why? Because using these types allows you, and people reading your code, to differentiate between errors and exceptions. Errors are anticipatable, semi-routine, unfortunate but not disastrous. Exceptions are exceptional—they shouldn't happen, their very presence represents some serious derivation from the usual course of events.

Even in Python, which doesn't have anything like Maybe/Either, it's still worth thinking about which of your "exceptions" are actually exceptional, and which are merely errors. Viz:

try:
    middle_name = user['middle_name']
except KeyError:
    middle_name = ''


try:
    birthday = user['birthday']
except KeyError:
    birthday = ''

We're treating two pieces of missing information the same way, but they're actually completely different. Lots of people don't have middle names; everybody has a birthday. (Of course, some users may be wary of divulging such information to strangers on the internet, but let's assume you've earned their trust.) It's much clearer to treat different cases differently:

middle_name = user.get('middle_name', '')
birthday = user['birthday']

Now we're treating errors and exceptions with the relative alarm they each deserve.

Do handle specific errors appropriately

This is an extension of the previous point. You can use multiple catches per try, and if there's cause to, you should.

# instead of this...
try:
    do_whatever()
except SomeException as err:
    if isinstance(err, FooException):
        handle_foo_exception(err)
    elif isinstance(err, BarException):
        handle_bar_exception(err)

# ...do this
try:
    do_whatever()
except FooException as err:
    handle_foo_exception(err)
except BarException as err:
    handle_bar_exception(err)

Do write your own exception subclasses

You can facilitate and extend the pattern above by writing and throwing your own exception subclasses. It's easy and aids in clarity.

# instead of this...
def check_birthday(user):
    if not user.birthday:
        raise ValueError('User birthday missing!')
    elif user.birthday.year > 2002:
        raise ValueError('User is too young!')
    elif user.birthday.year < 1900:
        raise ValueError('User is probably dead!')

try:
    check_birthday(user)
except ValueError as err:
    msg = str(err)
    if 'missing' in msg:
        redirect_to_birthday_input()
    elif 'young' in msg:
        redirect_to_site_for_teens()
    elif 'old' in msg:
        handle_probable_fraud()

# ...do this
class UserBirthdayException(ValueError):
    pass

class UserBirthdayMissing(UserBirthdayException):
    pass

class InvalidUserBirthday(UserBirthdayException):
    def __init__(self, problem, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.problem = problem


def check_birthday(user):
    if not user.birthday:
        raise UserBirthdayMissing
    elif user.birthday.year > 2002:
        raise InvalidUserBirthday(problem='young')
    elif user.birthday.year < 1900:
        raise InvalidUserBirthday(problem='old')

try:
    check_birthday(user)
except UserBirthdayMissing:
    redirect_to_birthday_input()
except InvalidUserBirthday as err:
    if err.problem == 'young':
        redirect_to_site_for_teens()
    else:
        handle_probable_fraud()

Sure, it's more boilerplate, but the added clarity and control is worth it. It's especially worth considering if you're doing a lot of validation, or find yourself repeatedly handling a set of errors from an API—write some custom error classes and a response handler function, and your days of grepping through your codebase to find all the places you need to change your error handling because some third-party has switched from colons to tabs in their API responses are over.

Do Make use of exception class hierarchies

Like everything else in Python, exceptions are objects, and they have their own classes and inheritance. catch will handle subclasses of errors appropriately.

class AError(Exception):
    pass

class BError(AError):
    pass

class CError(AError):
   pass


try:
    raise BError
except CError:
    print('Caught CError')  # won't happen
except AError:
    print('Caught AError')  # will happen

Familiarize yourself with Python's builtin exceptions, and don't be shy about using inheritance when defining your own error classes.

Do keep it tight

Quick, tell me which of these functions could throw SomeException:

def f(x):
    try:
        a = do_something(x)
        b = do_something_else(a)
        c = lastly_do(b)
        log_or_whatever(c)
        return c
    except SomeException as err:
        log_exception(err)
        return None

Give up? Me too. How about here:

def f2(x):
    a = do_something(x)
    try:
        b = do_something_else(a)
    except SomeException as err:
        log_exception(err)
        return None
    else:
        c = lastly_do(b)
        log_or_whatever(c)
        return c

Better, no? Keep the scope of your try/catch blocks as narrow as possible. Yes, nested try/catch/else blocks can get annoying, but why not just refactor each of the potential culprits into separate functions?

Don't over-generalize

This should go without saying—by me, because your tooling (pep8/flake8/pylint, etc.) should be saying it at you instead. But I'm saying it anyway, because it bears repeating (and because I've seen—and written!—far too many noqa/pylint: disable=bare-except comments.)

A common excuse I've seen for catch Exception is ignorance—there's a 3rd-party API with bad documentation, or some complicated code with multiple points of failure. Unfortunately, as with the law, ignorance of potential exceptions is not a valid excuse. Do as much research as you can, add logging, and then refactor to catch the actual exceptions you want to catch.

And whatever you do, DO NOT catch BaseException. Trust me.

Lastly,

Don't be afraid to let it ride

Computers are incredibly complex; the world is a terrifying, unpredictable place; chaos is the only constant. Stuff (as it were) will happen. You don't have to catch everything; you don't even have to try. Think about what what handling every exception would even mean: if your DB can't be reached, or a config file doesn't exist, or Jeff Bezos decides overnight to get out of cloud storage and into self storage, what's the point of carrying on like your user can go about their business as normal? Five 9s is the goal, but Potemkin-oriented programming serves no-one. Print that TB, return that 500, move on.

Comments

[Duy K. Bui]

Thanks for the info. I started learning Python and found this a good source of tips for proper error handling. I have a question:

def f2(x):
    a = do_something(x)
    try:
        b = do_something_else(a)
    except SomeException as err:
        log_exception(err)
        return None
    else:
        c = lastly_do(b)
        log_or_whatever(c)
        return c

Why do I need the else? Can I simply write it like this?

def f2(x):
    a = do_something(x)
    try:
        b = do_something_else(a)
    except SomeException as err:
        log_exception(err)
        return None
    c = lastly_do(b)
    log_or_whatever(c)
    return c

[sam]

technically you don't, in this case, because of the return. but if you don't have a return you still need the else, and i'd argue it's clearer/more informative either way.

[Duy K. Bui]

So I will end up having nested try? As in

try:
    foo
except FooException:
    print "foo"
else:
    try:
        bar
    except BarException:
        print "bar"
    else:
        try:
            # ... so on so forth

?

[sam]

potentially! but like i said, it might be worth refactoring those cases into separate functions

[Duy K. Bui]

So

def wrap_foo:
    try:
        foo
    catch FooException:
        print "foo"

def wrap_bar:
    # same as above

def main:
    wrap_foo
    wrap_bar

Am I getting there?

[sam]

yeah, depending on what you're trying to do—if everything's decoupled and bar doesn't depend on the success of foo, then this approach is the way to go