Rust's Option type... in Python

Cover Image (C) Tai Kedzierski

How many times have you written/seen code like this:

data = get_data()
print(data.strip())

Looks reasonable right?

How about if data == None?

It is so easy to assume thingness and forget to handle nullness. So frequent. In Python, Java, JavaScript, and many other languages, this is prone to happening.

Then some languages, like Rust, have a semantic way of dealing with this... and I pondered, what if we tried to intorduce this to Python? How would it look?

Just for a laugh, and in between waiting for integration tests to finish running, I threw this together: a quick Python implementation to mimic Rust's Option type

In Rust, the Option enumeration type is used with the match operator to force handling the concept of null-ness, without treating it as a first-order value, unlike in languages that have a null representation.

Does that sound weird? Let me digress a moment

Nullity as a typed value, not a stand-in

In languages where there can be a None or null placeholder-concept, it works like this:

functionA() returns a value, say of type String (or other, it is immaterial). If functionA() returns null, it effectively says it returned null instead of a String. Trying to treat this null as String (by calling its methods, concatenating it, etc) leads to errors.

In Rust, this cannot happen.

In Rust if functionA() says returns a String, it always returns a valid string. If we want to express that a non-valid value can be returned, then functionA() must flip: it returns an Option instead.

We must then handle the Option enumeration type - check if it is the variant known as None , or check if it is the variant known as Some.

We can then .unwrap() the result explicitly, or through pattern matching implicitly:

# Panics if we get a None variant
let value = functionA().unwrap();

# vs

match functionA() {
    None => println!("Fail!"),
    Some(value) => println!("Success: {}", value),
}

In Python

So how does this experiment look in Python ?

Well, we're not replacing the native None type. We'll use a Null class (a regular class), subclassed to Option. I didn't include a Some type, but it would be simple enough - just class Some(Option): pass and it's done. Everything is in fact implemented on Option

We are then able to wrap this around any return result we care to add to our script to make it return an Option.

def some_func() -> Option:
    some_value = None

    # ... implementation ...

    if valid:
        return Option("some value")

    elif not_valid:
        # Explicit null-return
        return Null

    # or even
    return Option(some_value)
    # (will behave like Null if some_value is still None)

res = some_func()
if res.is_null():
    ... # handle it
value = res.unwrap()

This forces whoever uses some_function() to consider that nullity can come of it, before gaining access to the value itself:

# Without a check, this may "panic" (borrowing from rust's terminology)
#  the program exits systematically with an error message
value = some_func().unwrap("failed to get data from some_func")

# On occurrence of nullity, a default value is substituted instead.
value = some_func().or_default("default data")

This makes for explicit force-handling of None , where before it could easily pass under the review radar silently.

An interesting side-effect is that managing default-substitution is now done outside of some_func's logic; the design of some_func can stay unencumbered with specifically dealing with alternate eventualities, just as the caller code can declare the default value without multiple lines of if/else checking.

We also don't have a match keyword to use, but this doesn't impede the usability in any significant way - merely having to acknowledge unwrapping brings gains, and the or_raise() method allows some additional flow control.

Should this even be used in Python ?

I am actually tempted to use it in my projects... as a way of forcing awareness of null possibility.

It ensures that nullness cannot be ignored/forgotten by the caller, promoting more conscientious coding, and helping avoid some common pitfalls.

It is succinct enough to allow minimal changes in code flow, but explicit so that it cannot be ignored.

But it's very un-idiomatic, literally "trying to write Rust style code in Python."

And yet.

🪙 A penny for you thoughts?

Comments

[xtofl]

My penny?

Mypy has caught many forgotten None-checks in my code. The Pythonic way seems to be to not bother at runtime, but type-hint, and check statically:

from typing import Optional
def maybe_fruit(i: int) -> Optional[str]:
  return "5" if i == 5 else None

len(maybe_fruit(5))

This will fail to type check:

$ mypy m.py 
m.py:6: error: Argument 1 to "len" has incompatible
   type "str | None"; expected "Sized"  [arg-type]
Found 1 error in 1 file (checked 1 source file)

Note: it does seem that mypy is not fool proof, though.

Another, run-time checked, approach is to implement the Maybe monad. Some articles exist, and some libraries do. E.g. skeptric.com/python-maybe-monad/.

[Tai Kedzierski]

Static analysis is one of the ways to try and ferret things out ; it probably does rely on it identifying correctly that a function can in fact return None, which I presume is not fully trivial...

The technique I explored is more "forceful"...

It's an interesting Mybe-monad article you link ; it also links through to a Wikipedia section which... essentially describes the Rust Option itself....

en.wikipedia.org/wiki/Monad_(funct...

[Ellis Breen]

It still requires the end user to handle the Null special case, rather than the monadic approach where the monad handles it:

def maybe_fmap(f: Callable[a, b]) -> Callable[Optional[a], Optional[b]]:
  return lambda(x): f(x) if x is not None else None

This Option solution just moves the problem - the end user can just forget to check for Nullity and then:

def unwrap(self, message=_NULL_MESSAGE):
    if self.value is None:
        raise Panic(message)``

Is this any better than the AttributeError or a TypeError that'd be raised if attempting to reference/operate on None?

This just increases the boilerplate a user has to use in order to remind them to check null, while doing nothing to force them to. The maybe monad approach saves both typing and enforces correctness, and is composable to boot.

[Tai Kedzierski]

Yes - the manifestation of the problem is moved as you point out ;

The point of my musing is not the attempt of making a fool-proof type-safety , but more of a "how's this for signposting?"

Let's say the dev is a user of my library, and I've done the wrapping and this moved-problem type, etc.

What was

data = thelib.some_call()
print(data.strip() )

necessarily becomes

data = thelib.some_call().unwrap() # and by virtue of typing this, alarm bells should have started ringing
print(data.strip() )

The point is, forcing the unwrapping makes the consuming dev explicitly think of handling null, instead of forgetting.

Instead of

# TICKET-123456: fix the None-crash
data = thelib.some_call()
if data is None:
  print("fail")
else:
  print(data.strip() )

it becomes

#data = thelib.some_call().unwrap^[^[^[^[^[^[^[^[^[luckily I got that before shipping
data = thelib.some_call()
if data.is_none():
    print("fail")
else:
    print(data.unwrap().strip())

It's a kludge, of course, don't get me wrong.

And no, I haven't put it in my projects after all ;-)

[Ellis Breen]

I mean, maybe it's better than nothing, though I feel like it gives you a false sense of security. It's akin to a security-by-obfuscation measure. To be fair, annoying the developer in order to make foolish things hard is sometimes necessary as an API or language programmer. I've just been rather frustrated that many of the 'Optional' types (such as Java's Option) still offer a get (equivalent to your unwrap) at the end of the day, allowing you to entirely bypass the purpose of the endeavour, much like one could with the Option class you describe.

Glad you haven't put it in your projects after all.. But it's a nice exercise in trying to replicate Rust's exacting, beautiful compiler and standard library. Obviously Rust and its wonderful ADT support does the heavy lifting in enforcing correctness here.

[Michal Mazurek]

I like the or_default()

But you could do:

value = some_func() or "default value"

The other option I think would be more pythonic, by just raising an exception instead of returning None.

def some_func(...):
      # ... something produces value
      if value is None:
            raise DoesNotExists("Some meaningful message")
      return value

This whole Result handling in rust comes from lack of exceptions, python do have them, so why not use them?

[Tai Kedzierski]

Indeed, this is completely unpythonic, but for a reason:

In most languages (python being one of a plethora), it is an extremely common mistake to just assume we have a value, until it blows up at runtime with a rogue None.

Hence the "billion dollar mistake," where so many developers forget to handle the None case.

By making it a type, and gating it inside an Option, the idea is to enforce "you cannot do anything with this unless you fully acknowledge that you could have nullity and you have explicitly chosen how to take action"

So instead of null-ness being maybe-or-maybe-not a possibility:

# oops, might be None, dunno, I thought we guaranteed something?
# whatevs, should be fine, didn't see it ...
data = get_data()

the function explicitly says "hey, handle the case, or else!"

# I explicitly acknowledge that I am not handling None
#       because FML YOLO
data = get_data().unwrap()

So really, yes, there are idiomatic pythonic wasy to handle None. Adding the type enforces "You cannot mistakenly fail to handle it"

[Michal Mazurek]

Yeah, I agree I saw many cases where None was not handled properly. But if we require a specific thing to be returned only from a function, then we as well can ask for handling of None or writing a function in a way that None will never be returned.

On the other hand what floats your boat, if it works it's good!