Python exceptions considered an anti-pattern

Originally published in my blog: https://sobolevn.me/2019/01/simple-dependent-types-in-python

What are exceptions? Judging by their name it is an entity representing some exceptional situation that happens inside your program.

You might be wondering how do exceptions are an anti-pattern and how does this relate to typing at all? Well, let's find out!

Problems with exceptions

First, we have to prove that exceptions have drawbacks. Well, it is usually hard to find "issues" in things you use every day because they start to look like "features" to you at some point.

Let's have a fresh look.

Exceptions are hard to notice

There are two types of exceptions: "explicit" that are created with raise keyword right inside the code you are reading and "wrapped" that are wrapped inside some other functions/classes/methods that you are using.

The problem is: it is really hard to notice all this "wrapped" exceptions.
I will illustrate my point with this pure function:

def divide(first: float, second: float) -> float:
     return first / second

All it does is dividing two numbers. Always returning float. It is type safe and can be used like so:

result = divide(1, 0)
print('x / y = ', result)

Wait, did you get it? print will never be actually executed. Because 1 / 0 is an impossible operation and ZeroDivisionError will be raised. So, despite your code is type safe it is not safe to be used.

You still need to have a solid experience to spot these potential problems in a perfectly readable and typed code. Almost everything in python can fail with different types of exceptions: division, function calls, int, str, generators, iterables in for loops, attribute access, key access, even raise something() itself may fail. I am not even covering IO operations here. And checked exceptions won't be supported in the nearest future.

Restoring normal behavior in-place is impossible

Hey, but we always have except cases just for this kind of situations. Let's just handle ZeroDivisionError and we will be safe!

def divide(first: float, second: float) -> float:
     try:
         return first / second
     except ZeroDivisionError:
         return 0.0

Now we are safe! But why do we return 0? Why not 1? Why not None? And while None in most cases is as bad (or even worse) than the exceptions, turns out we should heavily rely on business logic and use-cases of this function.

What exactly do we divide? Arbitrary numbers? Some specific units? Money? Not all cases can be covered and easily restored. And sometimes when we will reuse this function for different use-cases we will find out that it requires different restore logic.

So, the sad conclusion is: all problems must be resolved individually depending on a specific usage context. There's no silver bullet to resolve all ZeroDivisionErrors once and for all. And again, I am not even covering complex IO flows with retry policies and expotential timeouts.

Maybe we should not even handle exceptions in-place at all? Maybe we should throw it further in the execution flow and someone will later handle it somehow.

Execution flow is unclear

Ok, now we will hope that someone else will catch this exception and possibly handle it. For example, the system might notify the user to change the input, because we can not divide by 0. Which is clearly not a responsibility of the divide function.

Now we just need to check where this exception is actually caught. By the way, how can we tell where exactly it will be handled? Can we navigate to this point in the code? Turns out, we can not do that.

There's no way to tell which line of code will be executed after the exception is thrown. Different exception types might be handled by different except cases, some exceptions may be suppressed. And you might also accidentally break your program in random spots by introducing new except cases in a different module. And remember that almost any line can raise.

We have two independent flows in our app: regular flow that goes from top to bottom and exceptional one that goes however it wants. How can we consciously read code like this?

Only with a debugger turned on. With "catch all exceptions" policy enabled.

Exceptions are just like notorious goto statements that torn the fabric of our programs.

Exceptions are not exceptional

Let's look at another example, a typical code to access remote HTTP API:

import requests

def fetch_user_profile(user_id: int) -> 'UserProfile':
    """Fetches UserProfile dict from foreign API."""
    response = requests.get('/api/users/{0}'.format(user_id))
    response.raise_for_status()
    return response.json()

Literally, everything in this example can go wrong. Here's an incomplete list of all possible errors that might occur:

1. Your network might be down, so request won't happen at all
2. The server might be down
3. The server might be too busy and you will face a timeout
4. The server might require an authentication
5. API endpoint might not exist
6. The user might not exist
7. You might not have enough permissions to view it
8. The server might fail with an internal error while processing your request
9. The server might return an invalid or corrupted response
10. The server might return invalid json, so the parsing will fail

And the list goes on and on! There are so maybe potential problems with these three lines of code, that it is easier to say that it only accidentally works. And normally it fails with the exception.

How to be safe?

Now we got that exceptions are harmful to your code. Let's learn how to get read off them. There are different patterns to write the exception-free code:

1. Write except Exception: pass everywhere. That's as bad as you can imagine. Don't do it.
2. Return None. That's evil too! You either will end up with if something is not None: on almost every line and global pollution of your logic by type-checking conditionals, or will suffer from TypeError every day. Not a pleasant choice.
3. Write special-case classes. For example, you will have User base class with multiple error-subclasses like UserNotFound(User) and MissingUser(User). It might be used for some specific situations, like AnonymousUser in django, but it is not possible to wrap all your possible errors in special-case classes. It will require too much work from a developer. And over-complicate your domain model.
4. You can use container values, that wraps actual success or error value into a thin wrapper with utility methods to work with this value. That's exactly why we have created @dry-python/returns project. So you can make your functions return something meaningful, typed, and safe.

Let's start with the same number dividing example, which returns 0 when the error happens. Maybe instead we can indicate that the result was not successful without any explicit numerical value?

from returns.result import Result, Success, Failure

def divide(first: float, second: float) -> Result[float, ZeroDivisionError]:
    try:
        return Success(first / second)
    except ZeroDivisionError as exc:
        return Failure(exc)

Now we wrap our values in one of two wrappers: Success or Failure. These two classes inherit from Result base class. And we can specify types of wrapped values in a function return annotation, for example Result[float, ZeroDivisionError] returns either Success[float] or Failure[ZeroDivisionError].

What does it mean to us? It means, that exceptions are not exceptional, they represent expectable problems. But, we also wrap them in Failure to solve the second problem: spotting potential exceptions is hard.

1 + divide(1, 0)
# => mypy error: Unsupported operand types for + ("int" and "Result[float, ZeroDivisionError]")

Now you can easily spot them! The rule is: if you see a Result it means that this function can throw an exception. And you even know its type in advance.

Moreover, returns library is fully typed and PEP561 compatible. It means that mypy will warn you if you try to return something that violates declared type contract.

from returns.result import Result, Success, Failure

def divide(first: float, second: float) -> Result[float, ZeroDivisionError]:
    try:
        return Success('Done')
        # => error: incompatible type "str"; expected "float"
    except ZeroDivisionError as exc:
        return Failure(0)
        # => error: incompatible type "int"; expected "ZeroDivisionError"

How to work with wrapped values?

There are two methods two work with these wrapped values:

• map works with functions that return regular values
• bind works with functions that return other containers

Success(4).bind(lambda number: Success(number / 2))
# => Success(2)

Success(4).map(lambda number: number + 1)
# => Success(5)

The thing is: you will be safe from failed scenarios. Since .bind and .map will not execute for Failure containers:

Failure(4).bind(lambda number: Success(number / 2))
# => Failure(4)

Failure(4).map(lambda number: number / 2)
# => Failure(4)

Now you can just concentrate on correct execution flow and be sure that failed state won't break your program in random places.

And you can always take care of a failed state and even fix it and return to the right track if you want to.

Failure(4).rescue(lambda number: Success(number + 1))
# => Success(5)

Failure(4).fix(lambda number: number / 2)
# => Success(2)

It means that "all problems must be resolved individually" practice is the only way to go and "execution flow is now clear". Enjoy your railway programming!

But how to unwrap values from containers?

Yes, indeed, you really need raw values when dealing with functions that actually accept these raw values. You can use .unwrap() or .value_or() methods:

Success(1).unwrap()
# => 1

Success(0).value_or(None)
# => 0

Failure(0).value_or(None)
# => None

Failure(1).unwrap()
# => Raises UnwrapFailedError()

Wait, what? You have promised to save me from exceptions and now you are telling me that all my .unwrap() calls can result in one more exception!

How not to care about these UnwrapFailedErrors?

Ok, let's see how to live with these new exceptions. Consider this example: we need to validate the user's input, then create two models in a database. And every step might fail with the exception, so we have wrapped all methods into the Result wrapper:

from returns.result import Result, Success, Failure

class CreateAccountAndUser(object):
    """Creates new Account-User pair."""

    # TODO: we need to create a pipeline of these methods somehow...

    def _validate_user(
        self, username: str, email: str,
    ) -> Result['UserSchema', str]:
        """Returns an UserSchema for valid input, otherwise a Failure."""

    def _create_account(
        self, user_schema: 'UserSchema',
    ) -> Result['Account', str]:
        """Creates an Account for valid UserSchema's. Or returns a Failure."""

    def _create_user(
        self, account: 'Account',
    ) -> Result['User', str]:
        """Create an User instance. If user already exists returns Failure."""

First of all, you can not unwrap any values while writing your own business logic:

class CreateAccountAndUser(object):
    """Creates new Account-User pair."""

    def __call__(self, username: str, email: str) -> Result['User', str]:
        """Can return a Success(user) or Failure(str_reason)."""
        return self._validate_user(
            username, email,
        ).bind(
            self._create_account,
        ).bind(
            self._create_user,
        )

   # ...

And this will work without any problems. It won't raise any exceptions, because .unwrap() is not used. But, is it easy to read code like this? No, it is not. What alternative can we provide? @pipeline!

from result.functions import pipeline

class CreateAccountAndUser(object):
    """Creates new Account-User pair."""

    @pipeline
    def __call__(self, username: str, email: str) -> Result['User', str]:
        """Can return a Success(user) or Failure(str_reason)."""
        user_schema = self._validate_user(username, email).unwrap()
        account = self._create_account(user_schema).unwrap()
        return self._create_user(account)

   # ...

Now it is perfectly readable. That's how .unwrap() and @pipeline synergy works: whenever any .unwrap() method will fail on Failure[str] instance @pipeline decorator will catch it and return Failure[str] as a result value. That's how we can eliminate all the exceptions from our code and make it truly type-safe.

Wrapping all together

Now, let's solve this requests example with all the new tools we have. Remember, that each line could raise an exception? And there's no way to make them return Result container. But you can use @safe decorator to wrap unsafe functions and make them safe. These two examples are identical:

from returns.functions import safe

@safe
def divide(first: float, second: float) -> float:
     return first / second


# is the same as:

def divide(first: float, second: float) -> Result[float, ZeroDivisionError]:
    try:
        return Success(first / second)
    except ZeroDivisionError as exc:
        return Failure(exc)

And we can see that the first one with @safe is way more readable and simple.

That's the last thing we needed to solve our requests problem. That's how our result code will look like in the end:

import requests
from returns.functions import pipeline, safe
from returns.result import Result

class FetchUserProfile(object):
    """Single responsibility callable object that fetches user profile."""

    #: You can later use dependency injection to replace `requests`
    #: with any other http library (or even a custom service).
    _http = requests

    @pipeline
    def __call__(self, user_id: int) -> Result['UserProfile', Exception]:
        """Fetches UserProfile dict from foreign API."""
        response = self._make_request(user_id).unwrap()
        return self._parse_json(response)

    @safe
    def _make_request(self, user_id: int) -> requests.Response:
        response = self._http.get('/api/users/{0}'.format(user_id))
        response.raise_for_status()
        return response

    @safe
    def _parse_json(self, response: requests.Response) -> 'UserProfile':
        return response.json()

Things to recap:

1. We use @safe for all methods that can raise an exception, it will change the return type of the function to Result[OldReturnType, Exception]
2. We use Result as a container for wrapping values and errors in a simple abstraction
3. We use .unwrap() to unwrap raw value from the container
4. We use @pipeline to make sequences of .unwrap calls readable

This is a perfectly readable and safe way to do the exact same thing as we previously did with the unsafe function. It eliminates all the problems we had with exceptions:

1. "Exceptions are hard to notice". Now, they are wrapped with a typed Result container, which makes them crystal clear.
2. "Restoring normal behavior in-place is impossible". We now can safely delegate the restoration process to the caller. We provide .fix() and .rescue() methods for this specific use-case.
3. "Execution flow is unclear". Now it is the same as a regular business flow. From top to bottom.
4. "Exceptions are not exceptional". And we know it! We expect things to go wrong and are ready for it.

Use-cases and limitations

Obviously, you can not write all your code this way. It is just too safe for the most situations and incompatible with other libraries/frameworks. But, you should definitely write the most important parts of your business logic as I have shown above. It will increase the maintainability and correctness of your system.

dry-python / returns

Make your functions return something meaningful, typed, and safe!

---

---

Make your functions return something meaningful, typed, and safe!

Features

• Brings functional programming to Python land
• Provides a bunch of primitives to write declarative business logic
• Enforces better architecture
• Fully typed with annotations and checked with mypy, PEP561 compatible
• Adds emulated Higher Kinded Types support
• Has a bunch of helpers for better composition
• Pythonic and pleasant to write and to read 🐍
• Support functions and coroutines, framework agnostic
• Easy to start: has lots of docs, tests, and tutorials

Installation

pip install returns

You are also required to configure mypy correctly and install our plugin to fix this existing issue:

# In setup.cfg or mypy.ini:
[mypy]
plugins =
  returns.contrib.mypy.returns_plugin

We also recommend to use the same mypy settings we use.

Make sure you know how to get started, check out our docs!

Contents

• Maybe container that allows you to write None-free code
• RequiresContext container that allows…

View on GitHub

Comments

[Jonathan Higgs]

I'm torn by this. While I do like the railway programming model a lot, and the the idea of returning additional information makes a lot of sense in cases of potential failure (C# for example has bool TryGet<TKey, TValue>(TKey key, out TValue value) to deal with lookup failure without exceptions), I feel that Exceptions are highly useful and don't really suffer from the issues you assign to them

They aren't difficult to notice, when they happen I get a big stack trace dump and my IDE helpfully breaks when they are thrown. I'd say they are much more difficult to know ahead of time, but the cause of this is that python is impossible to statically analyse, it is a language problem and not an Exception problem. Other languages have tried noting functions are noexcep or listing all the possible exceptions that might come from a function call, but ultimately those either didn't solve the know-ability problem or created too much overhead to be useful.

Program flow is as unclear as without using them. There is always the issue of not knowing how a function is called, and therefore where the program might continue on an error. The only defense we have against that is to write clear and specific code, SOLID etc.

Comparisons to goto are extremely unjustified. It is just as easy to say that function calls, and even if statements are just structured gotos. The exact same reasoning could be applied to those but we still use them because it would be impossible to do anything without them. Since a raw goto can make it very difficult to follow a program, we have specific sub-cases where they make sense and restrict their use as much as possible to those sub-cases. It is absolutely possible to make a program difficult to follow with misuse of exceptions, it is equally possible to make it difficult to follow with poorly designed functions

Exceptions are a control-flow mechanism. Not all functions can produce a result when the accept variable input. Sometimes remote resources aren't available. Sometimes cosmic rays hit a register and flip a bit leading to undesired behavior. The places where some undesired state is hit is probably not the place that needs to decide how to continue with execution. Exceptions allow us to control this unexpected state and make that decision at the appropriate point in the control flow

Again, I really like the railway notion for high level program flow, but down in the weeds of making things happen that syntax is far too obscure and much less readable

[Nikita Sobolev]

Wow, thanks for this reply. It is awesome.

I agree that it is easy to notice exceptions when they happen. It is hard to notice them ahead of time. There are a lot of different attempts to solve it, including checked exceptions in java.

Program flow is as unclear as without using them

I cannot agree. The main difference is that Result is just a regular value. It returns to the same place where the function was originally called. While exceptions can jump to any other layer of your call stack. And it will depend on the execution context. That's what I have called "two execution flows" in the article.

Comparisons to goto are extremely unjustified

Well, it seems rather similar to me:

try:
    print(1 / 0)
except ZeroDivisionError:
    do_something()

It is clear that except ZeroDivisionError is going to catch this exception.
Now, we will move this print(1 / 0) into a new function called print_value():

def print_value():
    print(1 / 0)

But, this except ZeroDivisionError will still execute. And it looks like goto mark to me.

I really like the railway notion for high level program flow

That's exactly the case! That's why I have this "Limitations" section. I prefer my business-logic to be safe and typed, while my application might throw exceptions that are common to the python world. Frameworks will handle them.

[Michael Morehouse]

But, this except ZeroDivisionError will still execute. And it looks like goto mark to me.

But that's your apparent misunderstanding... it doesn't behave at all like a goto ... like not even within the same ballpark.

This:

try:
    return n / d
except ZeroDivisionError:
    return foo()

Is fundamentally no different than:

if not d:
    return foo()
return n / d

And adding in a deeper level of function nesting doesn't change that ... this:

def div(n, d):
    return n / d

try:
    return div(n, d)
except ZeroDivisionError:
    return foo()

Is now fundamentally no different than:

if not d:
    return foo()
return div(n, d)

The exception handler is effectively just a convenient way of writing the boilerplate for whatever test(s) would be required in that if to satisfy any given exception state, with the added benefit of deferring the test until an actual exception state has been encountered and needs to be addressed (or allowed to bubble further up the stack).

If you're writing the exception handler there's no magic (and certainly nothing that resembles a goto; you're reacting to a specific exception that has happened below you and the traceback tells you precisely where that came from ... if your reaction is to call a function then you know precisely where the control flow goes to and also that it will always return to you, even if it returns to you in the form of another exception (like for instance a SystemExit you've wrapped in the response function). A goto is a completely arbitrary escape from normal control flow, an exception is not, it's entirely predictable, though I'll happily admit it's not always easy to predict.

If no exception handling is involved then the behaviour is very predictable: where the exception is raised the stack frame exits and bubbles up a level. In each successively higher frame in which it's not handled it's effectively the same as catching it and immediately re-raising it, and so it continues to bubble ... when it hits the surface the interpreter catches it, prints the traceback, and shuts down in an orderly fashion with a non-zero returncode.

I don't have an issue with you wanting to make error handling a bit more sane for yourself, especially if you're going to try to enforce type constraints, but comparison to goto is just incorrect.

[fakuivan]

Exceptions totally do behave like gotos, a clear example is how you can use exceptions to "short circuit" things like map or functions that take in other functions. Here's an example:

def find_first_zero_pos(ints: list[int]) -> int | None:
    class FoundZero(Exception):
        pass
    i = 0
    def inc_or_rise_if_zero(num: int):
        if num == 0:
            raise FoundZero()
        nonlocal i
        i += 1

    try:
        for _ in map(inc_or_rise_if_zero, ints):
            pass
    except FoundZero:
        return i
    return None

In this case, execution flow is returned to find_first_zero_pos from map at will. You would not be able to do this with a language that does not support exceptions using only pure code (a function that shuts down the computer doesn't count lol). The with statement exists in part to be able to handle this "execution could end at any moment" situation by providing a way to do cleanup even when an exception is thrown.

The problem with goto is that usually execution flow always goes down into a function and comes up out of it, aka your call stack is actually a stack. gotos allow you to never return from a function, jumping to another section of code while your call stack is not a stack anymore. Python even has a way to type this, it's the NoReturn return type for functions, which is automatically assigned by static types analizers to functions that raise unconditionally.

[rhymes]

I think the context in this discussion is everything. I'll explain:

And checked exceptions won't be supported in the nearest future.

I feel like this is a pro in the context of Python though, not a con. Guido Van Rossum talks about it in the thread you linked. Java's checked exception aren't great for usability as he hints at. In theory they are a safe idea, in practice they encouraged suppression, rethrowing, API fatigue over the years. A "simple feature" that really polarized Java programmers :D

On the other side, Go designers have basically decided to force programmers to handle errors all the time (though you can still supress them) but the ergonomics is still not perfect (and they are thinking of making adjustments for Go 2.0 using a scope-wide central error handler)

There's no way to tell which line of code will be executed after the exception is thrown.

I'm not sure I follow. The stack for unhandled exceptions tells you where you were, the rules for handled exceptions are quite clear:

try:
  1 / 0
  print("Hello") # this is never going to be executed
except ZeroDivisionError as e:
  # do something with the error
  pass

print("World")

This is going to print only World. Where is the unclear part?

How can we consciously read code like this?

This sentence is a little odd, again in context, because Python programmers have been doing it since 1991. It's not yesterday :D I'm not saying the status quo is perfect, but sentences like "how can we deal with it" hide the fact that a lot of people have been doing it fine for decades.

Exceptions are just like notorious goto statements that torn the fabric of our programs.

Maybe, but not exactly. Goto was a bad idea because it let you jump to any place in the code at any time. Exception don't do that. Also, the "goto(ish)" (context again matters here) aspect of exceptions really depend on you as the developer. If you don't actively used them as a control flow mechanism but only to handle errors, then they are error handlers.

Now we got that exceptions are harmful to your code.

It's still your opinion though, I'm reading your post and I disagree :-D.

it is not possible to wrap all your possible errors in special-case classes. It will require too much work from a developer. And over-complicate your domain model.

Mmm again, people have been wrapping their errors in custom exceptions for decades, why are you suddenly saying it can't be done? I'm not saying it's the best strategy, I dispute your "cants" and "harmfuls" and "wrongs". By the way creating custom exceptions is so standard practice that it's even suggested in the tutorial.

In general: I'm not saying exceptions are the best ever method of handling error situations and I understand my bias as an enthusiastic Python developer (though my list of stuff that I don't like about Python has some items, my pet peeve with exceptions was recently fixed) but the way you wrote it sounds like this: "this pretty fundamental and central feature of Python is trash, I wish it were different. Here how you should do it, my way is better".

Going back to the idea co context: if I was going around saying "goroutines in Go are bad" I might be right in a context (it takes a few lines of Go to create a leak in the abstraction :D) but I would also ask myself if Go is the right choice for me. There's nothing wrong with a good rant (which this sounds like) or not liking features but at the end of the day we have only one life. Instead of trying make a language conspicuously deviate from its definining principles (aka the reason we're not all using the same language), why not just use another? We don't have to be "mono language".

The idea of wrapping everything in what basically are option types (if I'm not mistaken, I'm not strong on type theory and polymorphic typing) seems to veer quite a bit away from "Python's zen" ethos which are quite explicit on the idea that errors shouldn't be silent (which you're basically doing by discarding raise in favour of an option type) and to favor simplicity above all (which is probably less evident by looking at the seemingly innocous decorators in the last example)

I don't see the idea of having to unwrap values and exception constantly as an improvement to be honest :D The only advantage I see here is the compile time check, but it worsen the readability of the code and the mental gymnastic any Python developer has to do (considering that your proposal has not been built in in Python since day one), even with the decorator based shortcuts.

That's what I meant by citing context and language design in the beginning. Your idea is not bad per se, it's just probably not the right one in the context of Python's design. It reminds me a little of the argument in favor of removing the GIL :D

"Exceptions are hard to notice". Now, they are wrapped with a typed Result container, which makes them crystal clear.

But it's not actually true, is it? Let's look at your example:

def __call__(self, user_id: int) -> Result['UserProfile', Exception]

Here you're telling the reader that the method can raise an exception. What new information I have than I didn't know before? Knowing that any method can return exceptions in Python?

Also, this method is telling me that I basically have to prepare for any type of exception, which makes the advantage of the declaration moot if we put aside mypy and the tools for a second: what's the gained advantage of this instead of using a comment/documentation that says: this method can raise a HTTP error or a parsing error?

Tools are important but my brain the first time it read this code said: "well, thanks for nothing, Exception is the root class, anything can throw an exception".

If Python was a statically typed language I would probably have a different opinion about all this but it isn't, and the types can be ignored (and are ignored by default) which by being optional only leave the constant of the programmer reading this code in 5 years and being told something they already know. Again, context is everything here.

The TLDR; of all this long response is that the role of a language designer is to find tradeoffs and in the context of Python exceptions work. Stepping out of them would be changing the language, which could be perfectly fine in theory, but are we really sure?

I'm so glad I'm not a language designer, it's a tough job, and now I'm starting to understand why Guido semi retired ;-)

[Nikita Sobolev]

@rhymes wow! That's a very detailed and deep opinion. Thanks for bringing this up.

There's no way to tell which line of code will be executed after the exception is thrown.

Let me give you another example. Imagine that you have a big library / framework / project. And you are just debugging it. And then your execution flow jumps to some random place in the code with an exception thrown somewhere inside the method you were jumping over. "What the hell happened?" is what I usually have in my mind at this moment.

And that's what make me feel like we have two executional flows in the app. Regular and exception flows. The problem that I see in this separation that we have to wrap our minds against these two fundamentally different flows. First is controlled with regular execution stack and the second one is gotoish (as you called it). And it is controlled with except cases. The problem is that you can break this second flow by just placing some new awkward and accidental except case somewhere in between.

And, while I can perfectly live with the exceptions, I start to notice that my business logic suffer from it when it is growing in size. And having an extra way to work with different logical error in your program is a good thing.

it is not possible to wrap all your possible errors in special-case classes

I was talking about special classes like AnonymousUser is django. Not custom exception classes.

seems to veer quite a bit away from "Python's zen" which are quite explicit on the idea that errors shouldn't be silent

Unless explicitly silenced!

Jokes aside, we are not making them silent with wrapping into container values, just making them wrapped. Later you can use them anyway you want. You can reraise them if that's how it should be done.

Here you're telling the reader that the method can raise an exception

Yes, exactly! All functions may raise. But, some functions are most likely to do it: when dealing with IO, permissions, logic decisions, etc. That's what we indicate here. And, yes, my example with Result['UserProfile', Exception] should be rewritten as Result['UserProfile', ExactExceptionType].

When exceptions become the important part of your logic API - you need to use Result type. That's a rule that I have created for myself. Not talking about just python, but also elixir and typescript.

Thanks for taking a part in my rant, I have really enjoyed reading your response ;)

[rhymes]

Let me give you another example. Imagine that you have a big library / framework / project. And you are just debugging it. And then your execution flow jumps to some random place in the code with an exception thrown somewhere inside the method you were jumping over. "What the hell happened?" is what I usually have in my mind at this moment.

I know what you're talking about, I've been there but to me is more an issue of abstraction than of exception handling. Some apps have too many deps :D BTW how is that super different from calling a function?

import template_library
def method_to_do_something():
  a = "Hello world"
  result = template_library.render(a=a)
  # do something else...
  return result

in this example I jump from method_to_do_something to template_library.render and back everytime I debug. And then if render calls yet another library the thing goes on. Sometimes you spend minutes stepping in libraries because the dependency tree is long (Ruby with open classes, heavy use of metaprogramming and DSLs sometimes is a nightmare to debug)

If you think about it, in a way, functions are labeled gotos with an address to go back to ;-)

and, yes, my example with Result['UserProfile', Exception] should be rewritten as Result['UserProfile', ExactExceptionType].

ok, that's make a little more sense :D

That's a rule that I have created for myself. Not talking about just python, but also elixir and typescript

Got it. Programmers are strange animals, when they find a tool or an idea they like they want to apply to everything :D.

Thanks for taking a part in my rant, I have really enjoyed reading your response ;)

ahhaha :D

[edA‑qa mort‑ora‑y]

"Judging by their name it is an entity representing some exceptional situation that happens inside your program."

This is, and has always been false. Exceptions are a mechanism for propagating errors. There is no reason for them to be exceptional.

[Nikita Sobolev]

Consider "exceptional situation that happens inside your program" as errors. Or some unreachable state. It does not make any difference.

[Yoandy Rodriguez Martinez]

The word "antipattern" sure get's thrown a lot around here lately.

[Nikita Sobolev]

Because we have a lot of them! That's a good thing that people share their opinions about how to write and how not to write good code.

Do you like the concept of Result type?

[Yoandy Rodriguez Martinez]

Maybe 😉

[Ryan Palo]

That's interesting. So far, from what I've seen, raising Exceptions seems like a valid, Pythonic way to handle edge cases.

I'm interested to find out your opinion on how I'd do things normally.

For the division function, it seems like a totally valid way to go to let it raise its own ZeroDivisionError. Looking in log files or seeing that come up in the terminal would be as good a message as any to communicate what's happening. You would see that a function called divide is raising a ZeroDivisionError, and you would know exactly what the problem is and that you need to track down where a zero is getting sent to your divide function.

And leaving it up to client code to handle the propagated ZeroDivisionError is a pretty standard idiom in Python. If you really feel the need to catch the error, you could maybe raise your own signaling the bad input value?

def divide(first: float, second: float) -> float:
    if second == 0:
        raise ValueError("Zero is an invalid divisor.")

    return first / second

In fact, Python's documentation itself recommends this idiom in the bisect module docs. If a binary search doesn't find the value it's looking for, it doesn't return None, it raises a ValueError. It's up to the client to handle that exception.

---

All of that to say, I think the wrapped value idea is neat, and a useful idiom from languages like Go and Rust. I can see that it would definitely have some use cases in Python where it would be the best. I agree that you wouldn't want to lean on Exceptions for standard control flow for the most part.

But using exceptions to indicate a state other than the happy path seems like it's definitely not an anti-pattern in Python, and in a lot of cases, it's the Pythonic way to go.

Does that make sense at all?

[Nikita Sobolev]

I like to separate two kind of layers I have in my app: logic and application.

Logic is almost independent, it is my choice how to work with it. It allows a huge portion of freedom.
On the other hand it requires to be readable. Because there are no other-projects-like-this-one. So, that's why it is important for me to make exceptions crystal clear. Other people does not know if my FetchUser class is going to raise or not. And I do want to make this contract explicit.

On the lower, application, level I am using the existing stuff. Like django, celery, scrapy. And I need to respect their APIs. And, for example, django uses a lot of exceptions to control the execution flow. So, I prefer to raise on this level. And check that every exception is meaningful with tests.

That's how I see the big picture. Hope that you will find a proper use-cases for returns!

[Rémy 🤖]

That's a feeling that I had when I started programming and many years along the way: you feel like you should eventually catch all exceptions.

But that's not true. I embrace and expect exceptions. That's the runtime automatically checking that all possible inputs are correct and telling me when I was wrong. It's the luxury of getting explicit bugs neatly organized in Sentry reports.

If you ask me, all that sugarcoating is a sweet and seductive idea, but it's a bad idea.

[Mišo]

I was thinking about exceptions recently too. But I don't consider them an anti-pattern only the support in IDEs and type systems is not very good. And somehow I see it also in your example with pipeline/safe decorators. Where is the difference? Because I see a lot in the comments that you have problem with the second exceptional flow but doesn't have your solution the same problem? And isn't it even worse because you don't a see a big stacktrace with the info about the exception? Let me describe what I see.

@pipeline
def pipeline(...):
    one().unwrap()
    two().unwrap()
    return three()

@pipeline
def parent_pipeline():
    pipeline().unwrap()
    ...

@pipeline
def grand_pipeline():
    parent_pipeline().unwrap()
    ...

So when the function two returns error the flow jumps to grand_pipeline exactly the same. And without the decorator with the proper exceptions the situation is also the same but with extra boilerplate of checking the result and returning upwards ala Golang. Rust has nice syntax sugar like two()? but still.

If I rewrite the code to use exceptions I see the same but without extra @safe and .unwrap() boilerplate. So where is the difference? If you read my article you will see I agree with that exceptions are hard to notice but I believe they can be fixed. By typesystem. Or even maybe by IDEs. What do you think about that? Still consider them anti-pattern even if they were upgraded?

I am still researching how the exceptions work under the hood and how stack unwinding affects the flow to complete my article but the idea there should be clear.

[Nikita Sobolev]

Well, @pipeline actually had multiple problems. And it is now removed from dry-python.
It is actually imperative compared to declarative function composition. Why so? Because @pipeline is an partial implementation of do-notation, which is imperative by definition.

But. The difference is still visible.

1. Now exceptions can be visible in the funtion's signatire (let's use the same example from your post, even knowing that @pipeline is removed):

@pipeline
def grand_pipeline() -> Result[ValueType, ExceptionType]:
    parent_pipeline().unwrap()

Here it is! We know that something happens! And we can express it with types.

1. We have a special feature to enable stacktraces for your Results in development. See returns.readthedocs.io/en/latest/p...

[mt3o]

Dear author, for me it seems you spend not enough time with languages not having exceptions, without sophisticated libraries, without sophisticated tools (like dynatrace or even debugger attached) trying to work with exception-rich cases like doing IO on filesystem or HTTP calls, using language lacking exceptions or just making bad use of them. In thinking of plain old pre y2k C or even PHP<5. If you had - you'd value exceptions, even more exceptions with properly designed type hierarchy and understood why Java enforces checked exceptions unless directly specified otherwise.
For single task of writing for to disk - there are so many different types of potential errors! Disk not found. Lacking permissions. File not existent. Directory non existent. Disk full. Lock being put on the file. Drive missing during the file write. And you have to check for those on (almost) each single operating on that file. And pass the state to the upper layer, without any strict convention (hello PHP, I'm watching at you). And with networking IO you have even more cases to handle, including such bizarre scenarios as timeouts in the middle of read.
Introduction of checked exceptions opened possibility to return something more than just value of declared type, but also complex information about errors. It should be embraced not suppressed.

[Nikita Sobolev]

Well, indeed I have not that much experience with C or php@4 (while I still maintain one insanely large legacy php@4 project), but I have spent quite a lot of time with other languages without exceptions: like rust or elixir (which technically has exceptions, but their use is limited and noted with special ! functions).

And I value python's way of dealing with errors. I just want to provide an alternative that will solve problems that do exist.

Now it is your choice either to use exceptions or to use returns when you need to as explicit as possible.

[Mihail Malo]

Hahaha, look at the beautiful Result/Either datatype viciously penetrating this Python of yours.
Maybe in a couple of years, something like this will become accepted in JS as well!
One can only hope.

[Sergey Kislyakov]

Failure(4).map(lambda number: number / 2)
# => Failure(2)

Looks like .map got executed for a Failure. Is it a bug?

Also, does PyCharm handle @safe decorator properly? I mean, if I have a function something() -> User, do I get auto-completions for User and not [User, WhateverExcetpion]?

[Nikita Sobolev]

Sorry, that's a typo! And thanks for noticing. I have updated this line:

Failure(4).map(lambda number: number / 2)
# => Failure(4)

PyCharm resolves types correctly, since we are shipping PEP561 annotations.

[Jeff Doak]

Nikita, thank you so much for your article. I ran into it looking for discussion on the topic. We use Marshmallow for validation and until version 3 it had a very simple mechanism of returning (valid_data, errors). In version 3 they decided to use exception upon errors instead. While it was easy to work around and wrap the call, It was a much better design in the original approach. I liked the design so much that I've used it myself in several designs.
I believe you are correct and I hope more developers come around to this way if thinking.
Code that is doing large sequences of data manipulation does not want to be halted or even thrown off course due to a single error. It wants to note it, store some details, and move on.
Exceptions are great in IO, memory, resource issues, etc. In my opinion they should be avoided in designs such as Marshmallow where errors are expected.

[Nikita Sobolev]

Thanks a lot, Jeff!

[Mr. Vegetable]

In my experience, exceptions in Python mostly make things MORE clear, not less, since they are used as ordinary control flow, like for or if, and you just expect them. Our brain models a lot of things in terms of "typical case + few special cases" and so, for me personally, exceptions make it much more easier to convert ideas into code and understand the ideas behind the code.

Exceptions are not errors or problems or unexpected thing. They are just cases that don't arise most often. So for me it's logically wrong to map them to "Failure". And the problem of finding "where the exception is caught" does not seem to become significantly easier when converted to "where do we check that the return result is Failure".

[Sergey Kislyakov]

That looks interesting and I'd probably try it in some of my pet projects. Looks a bit Rust'y, though.