DEV Community: Ignacio Vergara Kausel

Dynamic generation of informative `Enum`s in Python

Ignacio Vergara Kausel — Sat, 27 Mar 2021 23:04:05 +0000

Since I started learning some Rust and getting to know its enums, I've been very into using Enums in Python. And while enums in both languages are very different, they're close enough for most of my current use cases. Moreover, within Python, Enums are used in frameworks like Typer (CLI) and FastAPI (API), where they are used to provide validation to input within a set of possibilities.

In Python, an Enum without explicit member values is created as follows

from enum import Enum, auto
class Color(Enum):
    RED = auto()
    BLUE = auto()
    GREEN = auto()

list(Color)
# [<Color.RED: 1>, <Color.BLUE: 2>, <Color.GREEN: 3>]

or alternatively, using a functional API

from enum import Enum
Animal = Enum('Animal', 'ANT BEE CAT DOG', module=__name__)
list(Animal)
# [<Animal.ANT: 1>, <Animal.BEE: 2>, <Animal.CAT: 3>, <Animal.DOG: 4>]

Great! The problem is that when having Enums that interface with external users, the integer values of the members are of little use and very uninformative.

That's why I tend to prefer to use a different way, where the values are strings.

class Animal(str, Enum):
    ANT = 'ant'
    BEE = 'bee'
    CAT = 'cat'
    DOG = 'dog'
list(Animal)
# [<Animal.ANT: 'ant'>,
#  <Animal.BEE: 'bee'>,
#  <Animal.CAT: 'cat'>,
#  <Animal.DOG: 'dog'>]

Now, attending to the real problem at hand hinted at by the title. I had a rather long set of about 15 possible elements, and I didn't want to manually create all the elements of the Enum myself in a way that the values are informative. That's the exact thing we try to avoid, repetitive manual work.
We already saw the functional API work with an iterable, so basically, the problem is already solved. However, it does in a way that the member values are uninformative.

The solution is to provide a dictionary instead of a list (or a string in the example). Then, the (keys, value) pairs of the dictionary become the member name and value of the Enum as shown below. Going from a list (or an iterable) to a dictionary is straightforwardly achieved with a dictionary comprehension.

Animal = Enum('Animal', {el:el.lower() for el in 'ANT BEE CAT DOG'.split(" ")}, type=str)
list(Animal)
# [<Animal.ANT: 'ant'>,
#  <Animal.BEE: 'bee'>,
#  <Animal.CAT: 'cat'>,
#  <Animal.DOG: 'dog'>]

Which accomplishes what I was going for.

My Journey

Ignacio Vergara Kausel — Wed, 01 Jul 2020 21:36:58 +0000

My journey

I thought it would be interesting, at least for me, to tell my journey with computers and in particular programming. It's a good exercise in retro- and introspection.

It was early on in life that I got to be able to use a computer, probably I was around 6 years old. My first memories were a computer my father brought every now and then from his office to work at home (early "laptops" were cumbersome to transport and even more expensive than a desktop computer). I don't remember many details of it, but with my brother, we used to do very rudimentary drawings in WordStar or WordPerfect. Here I did learn the basics of how to move around the command line in DOS.

Skipping forward some years, I was in 5th grade when the school I attended offered a programming extracurricular. It was certainly intended for older students as it was imparted in the high school section and somewhat late in the afternoon. Anyhow, I went anyways as I had interest and curiosity. I was most likely the youngest student there. Our teacher was a German man (not too surprising considering I attended a German school) of advanced age (I went to a German school), which also did some astronomy extracurricular and probably was the physics teacher. In that class, we used Pascal or TurboPascal and learned about loops, conditionals, variables, and functions. Lessons mostly consisted of writing on the computer what the instructor wrote in the blackboard while the instructor explained the different elements. I fondly remember how it didn't have a hard drive; thus, we had two floppy disks, one for the OS and IDE/compiler and the other for my programs.

Since then, I have always been interested in programming but never put much time practicing it or doing anything of any worth. During my late teenage years, I read books on C, C++, Java, Perl, and eventually, Python. Only the last two ones I did give them a real practical try. In Perl, I did work during the southern hemisphere summer of 2001/2002 in a small company producing a web interface to assist in ISO certifications. Back then, it was a pretty fancy approach to using web technologies.

Probably on that summer or perhaps a bit later, I got introduced to Python by a friend (the same friend who brought me to work on that summer). Which I then used to write a small application to review the performance of salespeople over time to assess the impact of the investment in training.

Once in university, I had to take Introduction to Programming, which that semester was given in C#, and had exposure to Maple and Matlab in some mathematics lectures (Calculus and Linear algebra respectively). Some years later, I took a lecture on numerical programming, exemplified with C/C++. However, we didn't get too much practical experience since it was mostly reviewing strategies and algorithms in scientific computing. Still, it was the closest I had to a Computer Science class, and I remember really liking that class; I was very impressed with all the little tricks used to get better performance, accuracy, or stability. Much later, I took but never finished a lecture on simulating processes (my memory of that lecture is rather vague) where the exercises were given in Java.

For years I was curious and explored Python, and I used it for any small little thing during my Physics studies (2004-2010), and in various research activities, I got involved with. This exposed me to the nascent Python scientific stack (numpy, scipy, matplotlib), and I saw it's evolution and how it became more mature over time. At some point, I took a Coursera class on building modern web applications, which was in Ruby using the Ruby on Rails framework. During my last years researching physics (2013-2016), I even attempted to develop an application/library to perform some fitting and modeling of a particular type of experimental data in the field I was involved to provide an open-source alternative to an excellent but fully closed software tool. Until then, that had been my most ambitious project, but it didn't go very far... maybe someday I'll continue.

Somehow I kinda always knew that I was going to end up in technology/programming sooner than later. Eventually, I left the academic world and got my first job as a software developer. My first task was to implement an optimization routine based on Evolutionary Strategies to try and find better parameters in a scoring process. Given my physics background (I had taken a lecture during my masters called something like Physical principles of evolutionary processes), I felt very comfortable doing that. Of course I did this in Python!
On the job, I took on some internal applications written in Go and started being curious about Rust. And although I think Rust would be great for some pieces of the application we develop, I believe that due to internal knowledge, that won't happen.

Now, Rust is another beast. Quite exciting language, and given my lack of formal training, it has been quite an illuminating language to learn. Again, I haven't really done anything with it, but I think just learning and reading about it has made me a better programmer (for some more thoughts on this read here).

Until now, this journey has been rather shallow and exploring in some way or another, a few languages. I'm excited to see how this journey continues into the future, hoping that Rust becomes a more relevant part. Excited to see how Python and its ecosystem keeps evolving. And wishing to be able sooner than later to contribute to open source projects beyond doing translations (Debian), or cleaning/improving/documenting tests (pandas).

Links of the Week (4)

Ignacio Vergara Kausel — Mon, 27 Jan 2020 16:08:47 +0000

https://www.youtube.com/watch?v=pBuS7EUPnQA Entertaining talk about usability and the oddities of character encodings.
https://fasterthanli.me/blog/2020/a-half-hour-to-learn-rust/ A summary on Rust syntax.
http://tomasp.net/blog/types-and-math.aspx/ I have almost zero knowledge on functional programming languages, and also no formal CS education, but a Physics one. I find it extremely interesting how theoretical CS could get.

Links of the week (3)

Ignacio Vergara Kausel — Mon, 20 Jan 2020 08:42:31 +0000

https://locusmag.com/2020/01/cory-doctorow-inaction-is-a-form-of-action/ A long piece on technology, policy, and democracy.
https://blog.letsenhance.io/all/2019/07/09/computational-photographyfrom-selfies-to-black-holes/ Another long piece, on computational photography. After reading this, I started to respect cellphone cameras/photography and their improvement over time.
https://github.com/cosmicpython/book#table-of-contents An interesting book (in the open) on application design in Python.

Link of the Week (2)

Ignacio Vergara Kausel — Mon, 13 Jan 2020 08:45:41 +0000

https://basecamp.com/guides/how-we-communicate I've been a fan of Basecamp (the company) for a while. This is a good read on how to do communication within a company, particularly if it's a remote-first one.
https://www.wired.com/story/the-style-maven-astrophysicists-of-silicon-valley/ As as physicist, I enjoyed this story about how a scientist can repurpose themselves outside academia. Long-time ago I did hear from an established economist in an academic institution, that he preferred to work with physicists than mathematicians because the former is more practical (result-oriented) than the latter who tend to focus more on generalizing results. Moreover, I'd say that in general astrophysicists tend to be much better prepared to jump into data analytics than "plain" physicists.
https://fasterthanli.me/blog/2019/reading-files-the-hard-way/ A three part series exploring how to open files in depth.

Links of the Week (1)

Ignacio Vergara Kausel — Sat, 04 Jan 2020 21:54:39 +0000

I decided to start sharing, in a weekly fashion, a few noteworthy links that I've found over the week. Hopefully will manage to be consistent and publish once a week.

https://evertpot.com/h2-parallelism/
An interesting exploration comparing the performance of HTTP 1.1 and 2.0 with different strategies for requesting data.
http://reasonablypolymorphic.com/blog/protos-are-wrong/
A lengthy critique of protobuffers, mostly centered on its type system. In my current job, we use them extensively, from before my time there, but in my experience, I haven't suffered from the criticisms. Unfortunately, the author doesn't point to any better solution.
https://julien.danjou.info/properly-managing-your-gitignore/
Great explanation on how to use .gitignore files, the role of the local versus the global one.
http://cliffle.com/p/dangerust/
A great and detailed tutorial on how to port a highly optimized C program into Rust.

Boolean arguments to functions in Python

Ignacio Vergara Kausel — Mon, 12 Aug 2019 11:20:00 +0000

Recently I did read the following short piece from M. Fowler regarding the use of boolean flags in function signatures. I’ll wait while you read it… done? Good!

As a general rule, I completely agree with him. They can be confusing, and should, in general, be avoided (easier said than done). But the article got me to reflect on this advice in the context of Python.

The example shown in the article would translate to Python (with type annotations and following PEP8) as follows

class Concert:def book(self, customer: Customer, is_premium: bool):
    ...

This definition does indeed has all the downsides mentioned in the article. But what if we use the keyword argument-only feature of Python?

class Concert:def book(self, customer: Customer, *, is_premium: bool = False):
    ...

In this case, by adding the * in the signature, one is forced to mention the name of the argument when calling the function. Thus, a call like the following one

a_book = concert.book(customer, True)

will raise a TypeError exception. Then, the correct way to call the function would be

a_book = concert.book(customer, is_premium=True)

There is no ambiguity as to what the flag points to, completely transparent and explicit.

However, since in most cases a boolean flag implies some branching in the code of the function, this approach still has some downsides that could be avoided by getting rid of such boolean flag.

Ultimately, it’s a matter of judgment as with everything. If the function has basically one of such flags that have a very restricted function, it’s still manageable.

Picking up from other languages

Ignacio Vergara Kausel — Mon, 12 Aug 2019 11:20:00 +0000

After many years of doing Python, first as a hobbyist and then in a professional setting, I’ve had to do some amount of Go programming. Also, I’ve been trying to explore other languages, primarily Rust, but have been touching some C while following some projects ([1] and [2]).

There is value in learning from other languages for the sake of itself, one can end up deepening knowledge and getting enlightenment through it. I remember having that feeling while doing some Ruby stuff in a MOOC years ago. In this article, I want to explore some of the learnings I’ve had lately. For a comprehensive view on this topic and how it reflects mainly in Python this article is a great and insightful read.

Typing

At the beginning of my Go tenure, it was slightly strange to be in the realm of a statically typed language enforced at compile time. Over the years, I had had some experience with compiled languages (Pascal, C, C#, Java), but never in any serious way. However, I quickly started enjoying the clarity that static types can provide, mostly through what the type checking enables in an IDEs like Jetbrain’s Goland.

While working with Python alongside Go, I found myself missing the clarity and helping “hand” that the static typing offers and enables. I was aware of mypy and the whole effort to add some type annotation into Python, but I was observant on it. This experience with Go drove me to start using mypy. In general, I’ve found it to be a great tool, and I’m trying to annotate as much as possible. It was a great help when doing a large scale refactoring/recycling of code, and in other instances, it has helped me to find some latent bugs “hidden” in the code. On the other hand, sometimes it can be slightly tricky to annotate arguments and variables accurately. Personally, I haven’t found the typing system in Python to be wrong or particularly bad as some people in the community have expressed, most likely because the things I’m doing are rather straightforward and don’t use really any metaprogramming or abuse the dynamic nature of Python (too much).

from Go import

Additionally, from Go, I did pick up some of the implicit and explicit conventions of the broader community. The most prominent ones that come to mind being- Avoid the use of frivolous else statements (link)- Use gofmt to format code as a standard- Conditional assignments (link)

The first point in the list is rather easy to implement while writing Python code. Although try-except blocks make the implementation of this idea to be slightly more indented. But one can still aim to use such a style pattern when possible.

For the second point, there are some tools like autopep8, yapf, and black. I personally favor the last one because it is intentionally designed to have very few options that the user can control. You might like or not the decisions, like with gofmt, but it keeps a high degree of consistency across different code bases and liberates you of trying (and failing) to follow a particular style. And I certainly prefer not having to worry and have some uniformity than pleasing my personal sense of code aesthetics. After some months of using blackened code, it’s hard for me to read messily styled code.

Now, regarding the last one, I’m waiting for Python 3.8 where the expression assignment operator is being included. A simple example from PEP-572

# Python 3.7
env_base = os.environ.get("PYTHONUSERBASE", None)
if env_base:
    return env_base

# Python 3.8
if env_base := os.environ.get("PYTHONUSERBASE", None):
    return env_base

This one surprised me because it’s not something I ever thought that I could need, but once I got to use it in Go, then I kept finding places where I’d have used it in Python.

from _ import

Since this post is not called something like “taking Go teachings to Python”, then it is time to introduce another language. As mentioned in the opening of this article, in my reduced spare time, I’ve been exploring Rust as it seems like a very complementary tool to have. I’ll not explain why Rust is cool and worthy, but I’ll mention that it has something extremely interesting that touches the topic of type annotations in Python.

Rust has a Result<T, Err> and Option<T> “types” that are used as return “values” of functions. They encode different possibilities of returned values.

In the first case, Result<T, Err>, a composite “object” is returned that either holds a value of type T or an error. Somehow, in my opinion, it feels rather similar to what Go 1.X does for error handling.

// Go
x, err := double_number("10")
if err {
    fmt.Println("Error: ", err)
} else {
    fmt.Println("Doubled number: ", x)
}

// Rust
match double_number("10") {
    Ok(n) => println!("Doubled number: {:?}", n),
    Err(err) => println!("Error: {:?}", err),
}

This doesn’t have a direct equivalent in Python since it has exceptions instead of error values/types, but one could certainly make a similar pattern than the one in Go by catching exceptions and returning some value instead.

The second, Option<T>, encodes a value of type T or something akin to None in Python. Taking the example from the documentation (playground)

fn divide(numerator: f64, denominator: f64) -> Option<f64> {
    if denominator == 0.0 {
        None
    } else {
        Some(numerator / denominator)
    }
}

// The return value of the function is an option
let result = divide(2.0, 3.0);

// Pattern match to retrieve the value
match result {
    // The division was valid
    Some(x) => println!("Result: {}", x),
    // The division was invalid
    None    => println!("Cannot divide by 0"),
}

After studying Rust a bit, I started to use the Optional type annotation in Python more often to explicitly signal that a function/method returns a value or None. Certainly returning or receiving a None in Python is something rather frequent, but formalizing it in a type makes it very explicit and clear. Taking the previous Rust example ported to Python, it’d look like the following

from typing import Optional

def divide(numerator: float, denominator: float) -> Optional[float]:
    if denominator == 0.0:
        return None
    return numerator / denominator

result = divide(2.0, 3.0)

if result:
    print(f"Result: {result}")
else:
    print("Cannot divide by 0.")

Probably not the best example, but it shows how to apply this same pattern in a properly annotated form. Annotating and then running mypy will force you to cover your bases, thus reducing some silly bugs in the code.

Quite interestingly, I found this post showing some sort of implementation of these two ideas in Python. It’s worthy of checking it out, a very interesting experiment! I hope to try it out at some point.

Furthermore, the Optional type can also be used for argument annotations for arguments that might have a particular value of a type or None, as it could be when they have None as a default.

from typing import Optional

def fib(n: int) -> int:  # declare n to be int
    a, b = 0, 1
    for _ in range(n):
        b, a = a + b, b
    return a

def cal(n: Optional[int]) -> None:
    print(fib(n))

If mypy is run over this code, it’ll complain at the print(fib(n)) line saying Argument 1 to "fib" has incompatible type "Optional[int]"; expected "int". It’s clear, at that point n could be some integer or None. This forces us to assure that n is actually a number beforehand and thus avoiding some ill-defined call and a highly likely runtime error. Thus, the annotation error in the previous code is fixed by modifying in a verbose mode the cal function as follows

def cal(n: Optional[int]) -> None:
    if n:
        print(fib(n))
    else:
        print("No number provided!")

In my opinion, mypy has been a great tool when working on a moderately sized project. Others of the like of Dropbox and Instagram have found it invaluable in their efforts to move from Python 2 to Python 3.

Closing remarks

Go and explore other languages! It will most likely provide great insight into your preferred language and your own programming style. There is no need to be an expert, but already, the challenge of understanding a new language provides plenty of teachings.

And regarding Python itself, use black and annotate code, particularly the code that gets into production and is being produced by a team.

Deeper into dataclasses

Ignacio Vergara Kausel — Tue, 06 Aug 2019 07:31:51 +0000

Usually, examples using the dataclasses module in Python are rather simple in the use of its features. That by itself is completely fine, but sometimes the implementation can be very tedious and cumbersome. However, the dataclasses module offers ways to be smarter, which are rarely talked about. With this article, I want to change that. Thus, this article doesn't cover topics like when and how to use them. There is plenty of material on the Internet to learn about that.

The following code represents a 3D point which can be added to another point and multiplied by a number element-wise. (Note: this makes this object more akin to a vector in my view). An extra feature is that it also supports iteration and unpacking via the __iter__ method, making the point and a number commutative.

# Baseline solution
from dataclasses import astuple, dataclass


@dataclass
class Point:
    x: float
    y: float
    z: float

    def __add__(self, other):
        x1, y1, z1 = self
        x2, y2, z2 = other
        return Point(x1+x2, y1+y2, z1+z2)

    def __sub__(self, other):
        x1, y1, z1 = self
        x2, y2, z2 = other
        return Point(x1-x2, y1-y2, z1-z2)

    def __mul__(self, scalar):
        x, y, z = self
        return Point(scalar*x, scalar*y, scalar*z)

    def __rmul__(self, scalar):
        return self.__mul__(scalar)

    def __iter__(self):
        return iter(astuple(self))

This is a good implementation, but even when using the fact that the point can be unpacked, it is quite tedious and repetitive. Typing x1, y1, z1 = self for every method is less than ideal. Also, what if we want to also have points in 2D, 4D or 6D? Well, that's straightforward but error-prone. We have to add/delete attributes/fields definitions and all references to them in the relevant methods. In this particular case, that would be six lines modified. The first part is the easy and the second one (very) annoying. We could do slightly better and only having to care for the first part if we want to address other dimensions.

Using `dataclasses` introspection

Let's be slightly smarter and use more of the tools available in the dataclasses module. In particular, the function fields() which exposes the fields defined in a dataclass. Thus, instead of having to name each coordinate of the point in the different operation methods, we can iterate over them.

# Introspection based solution
from dataclasses import astuple, dataclass, fields


@dataclass
class Point:
    x: float
    y: float
    z: float

    def __add__(self, other):
        return Point(*(getattr(self, dim.name)+getattr(other, dim.name) for dim in fields(self)))

    def __sub__(self, other):
        return Point(*(getattr(self, dim.name)-getattr(other, dim.name) for dim in fields(self)))

    def __mul__(self, other):
        return Point(*(getattr(self, dim.name)*other for dim in fields(self)))

    def __rmul__(self, other):
        return self.__mul__(other)

    def __iter__(self):
        return iter(astuple(self))

To understand how it works, I will focus on the __add__ method. There is quite a bit to unpack. At the core is the call to fields() function, which returns a tuple of 3 field objects, a field object being how a dataclass represents an attribute. You can (and should) go and check it out in a REPL, this can be done on the class itself or an instance of it. Since we have a tuple, we can iterate over it and access the name of each attribute.

>>> for field in fields(Point):
...     print(field.name)
...
x
y
z

Then, the next step is to use this to get the values associated with each attribute from the instance as follows

>>> p = Point(1,2,3)
>>>list(getattr(p, field.name) for field in fields(p))
[1,2,3]

Now we can do the operation between the two point instances that are being operated, and we unpack the generator expression into the arguments of the Point initialization

Point(*(getattr(self, dim.name)+getattr(other, dim.name) for dim in fields(self))

The last step is to put this as the return value of the __add__ method and then to implement the same strategy for the other methods. With this, we achieved the first step in removing the annoyance of having to touch every method if we change the number of dimensions of the point class. As a bonus, we also got to reduce the total amount of lines of code involved slightly.

An alternative solution

I have to agree that using this level of introspection of a dataclass might be a bit cumbersome, particularly by the use of the getattr function. This would be the solution if we weren't implementing the __iter__ method. But since we're supporting the iterator protocol, we can do something that perhaps is smarter. Thus, instead of having to iterate over the defined fields, we can iterate over the point object itself!

# Iterator based solution
import operator

from dataclasses import astuple, dataclass, fields


@dataclass
class Point:
    x: float
    y: float
    z: float

    def __iter__(self):
        return iter(astuple(self))

    def __add__(self, other):
        return Point(*(operator.add(*pair) for pair in zip(self,other)))

    def __sub__(self, other):
        return Point(*(operator.sub(*pair) for pair in zip(self,other)))

    def __mul__(self, other):
        return Point(*(operator.mul(*pair) for pair in zip(self,other)))

    def __rmul__(self, other):
        return self.__mul__(other)

To highlight the pivotal piece that the __iter__ method plays in this solution, I moved it to the top. For the rest, the code should be pretty much self-explanatory. I'd say that the use of the functions defined in the operator module also makes the code clearer.

Code quality metrics

Lately, I've also been tangentially interested in code quality metrics. I have a hypothesis regarding the standard metrics to quantify code quality, namely, that they are not well-tailored for dynamic languages like Python. In particular with features like decorators.

Let's explore some statistics for the three solutions covered before plus a solution without using dataclasses (classic) which is not shown but easy to get from the naive dataclass based solution.

	SLOC	MI	Rank
Classic	24	53.41	A
Baseline	21	54.28	A
Introspection	16	60.22	A
Iterator	17	100	A

Without diving too deep, the maintainability index increases as we move through the different implementations. This result was something that I intuitively expected. What it's shocking is the value of 100 for the iterator based solution. This warrants a deeper dive into this later on, as it seems unlikely it's a bug in the radon library and right now I'm too ignorant about this topic to be able to have an idea why this is the case.

Somehow I would have expected a more significant step between the classic and baseline dataclass solutions. But given that the methods we do implement are the same and the ones we didn't have to write (__init__, __eq__, __repr__) are rather simple it is understandable that they don't differ much.

Performance

Using %timeit on my laptop, I ran a quick benchmark of the addition of two points

>>> p1 = Point3D(1,2,3)
>>> p2 = Point3D(4,5,6)
>>> %timeit p1+p2

for the three solutions implemented in this article, with the following results

	mean (µs)	std
Baseline	33	5.18 µs
Introspection	6.04	450 ns
Iterator	30.4	3.34 µs

We can say that the introspection based solution is considerably more performant than the other two. Would be interesting to understand better where the win (loses) for the introspection (iterator) based solution come from.

In any case, this example shows something interesting, we can increase the maintainability index while also increasing the performance. This fact is not necessarily a given, as usually performance comes at the cost of readability.

What about nD points?

But the situation could still be improved. Let's say you want to be able to have 2D and 3D point living along. We could copy and paste the whole definition, give each class a different name, and be sure to have the correct number of attributes. But that'd be extremely silly and we could (and should) use inheritance (see the previous post which explores abstract base classes and dataclasses) and reuse all the code for the operations since we just made them independent of the dimension the point lives in. Since we're exploring dataclasses, I'll return to the first solution.

from abc import ABC

from dataclasses import astuple, dataclass, fields


@dataclass
class BasePoint(ABC):

    def __add__(self, other):
        return self.__class__(*(getattr(self, dim.name)+getattr(other, dim.name) for dim in fields(self)))

    def __sub__(self, other):
        return self.__class__(*(getattr(self, dim.name)-getattr(other, dim.name) for dim in fields(self)))

    def __mul__(self, other):
        return self.__class__(*(getattr(self, dim.name)*other for dim in fields(self)))

    def __rmul__(self, other):
        return self.__mul__(other)

    def __iter__(self):
        return iter(astuple(self))


@dataclass
class Point2D(BasePoint):
    x: float
    y: float


@dataclass
class Point3D(BasePoint):
    x: float
    y: float
    z: float

Excellent, we have now points in any dimension we want with little effort!

A factory of points

But is there a way to make even less work than this? The dataclasses module has a nifty function called make_dataclass which, as its name says, makes dataclasses based on its arguments.

We can try and create a point in 1D

>>> Point1D = make_dataclass('Point1D', [('x',float)], bases=(BasePoint,))
>>> Point1D(1)
Point1D(x=1)

Compared to defining the class in a normal way this doesn't seem to be a big win. But what if we want to create an exotic point in 5 dimensions? Well, first we create a list of tuples with the field names and types and then we use make_dataclass with it.

>>> dims = 5
>>> fields_definition = ((f'x{i}', float) for i in range(dims))
>>> Point5D = make_dataclass('Point5D', fields_definition, bases=(BasePoint,))
>>> Point5D(*range(5))
Point5D(x0=0, x1=1, x2=2, x3=3, x4=4)

I move from the naming xyz to x{i} in a more "mathematical" notation which for computers also works much better.

This sets the stage up to create a whole family of points. For this we create a function which will create them (a Factory)

def PointFactory(dim):
     fields_definition = ((f'x{i}', float) for i in range(dim))
     return make_dataclass(f'Point{dims}D', fields_definition, bases=(BasePoint,))

Making use of this factory a series of classes representing points in different dimensions can be easily created

>>> point_classes = [PointFactory(dim) for dim in range(5)]
>>> point_classes
[<class 'abc.Point0D'>, <class 'abc.Point1D'>, <class 'abc.Point2D'>, <class 'abc.Point3D'>, <class 'abc.Point4D'>]
>>> point_classes[3](1,2,3)
Point3D(x0=1, x1=2, x2=3)

Conclusion

Besides the boilerplate reduction that the dataclasses module provides, it offers some powerful tooling to work with them. As an example, I showed how to create a factory of n-dimensional points.

Moreover, I discovered that using the introspection machinery of dataclasses leads to a higher performant code. Not that this was a goal of this article, but it's always nice to get a boost. Keep in mind that introspection, in this case, might be a slightly wrong term, as it would make people believe it should be less performant, particularly those coming from Go.

After seeing the performance of each implementation, the question arises if the iterator based could be improved by being smarter. At least my first exploratory attempt by moving from import operator to from operator import add, mul, sub did not show any change. Perhaps this would be a good exercise for the reader ;)

Acknowledgments

I want to thank Nour Faroua for her contribution leading to simplification on the code, and to Bryan Reynaert for his thorough review and input improving organization, explanations, and language of the article.

Having fun with dataclasses and abstract base classes

Ignacio Vergara Kausel — Mon, 05 Aug 2019 12:16:07 +0000

Python is well known for the little boilerplate needed to get something to work. But even Python can get a bit cumbersome when a whole bunch of relatively trivial methods have to be defined to get the desired behavior of a class.
In this article we're going to explore how to combine dataclases with the abc and collections.abc modules of the standard library in Python. I'll assume that you know/understand what abc, collections.abc and dataclases. With the last two one could get a lot of behavior for free!

If you don't know about abstract base classes then I strongly recommend to check articles like this and this, for abc, and abc.collections, respectivelly. Likewise, if you don't know why dataclasses are interesting and about their advantages, you should check this other article. Or if you prefer some more visual guide check this talk. Take the time to learn these tools, it'll be worth it. Personally, since I discovered the abc and collections.abc modules, I've been trying to use them every time I can.

When I saw the inclusion of the dataclass module in the standard library of Python 3.7, I told myself I wanted to use it. Being able to reduce even more the boilerplate in Python seemed like a great idea. Of course I could have already been using attrs for basically the same effect, but when I tried it it didn't feel natural. That was most likely due to a lack of experience on my part.

Thus, it's very obvious that I'd end up mixing abstract classes, abstract collections, and dataclasses eventually at some point. Unfortunatelly, I haven't taken the time to refactor the code at work to this effect. To ammend that, I decided to explore the combination of abc+dataclasses and abc.collections+dataclasses with a toy example to see how straightforward, or not, the combination works. And since I didn't find any article mixing these two concepts (not that I looked too much around), I decided to write about it.

Now, getting our fingers dirty. First let's use pipenv to create an environment (as you should do to keep some environment hygene). It can feel slightly an overkill to do so, but I find it easier than trying to create a virtual environment from scratch. So, we initialize a Python 3.7 environment as follows pipenv --python 3.7.

To guide this experiment, we'll write a simple test. You could for sure skip this and manually play with the code in the REPL of choice, which I'd recommend in any case in this case to freely explore and discover your use case, but having tests makes the process easier. Install pytest to run the tests by executing pipenv install pytest. Note that I'm not using a separate dev environment for this, as this is just an experimentation environment. Now, we can activate the virtual environment by using pipenv shell.

The simplest test I can come up with, is that the abstract class Base should raise a TypeError exception if you try to instantiate it directly.

# test_demo.py
import pytest
from demo import Base

def test_abstract_base_class():
    with pytest.raises(TypeError):
        Base()

Before executing pytest, since it'll fail, we can quickly write an implementation based on dataclasses and abc. As such, the class is decorated with @dataclass and inherits from abc.ABC. Furthermore, it'll define one field a of type str with a __post_init__ and a process method, the last one defined as abstract.

# demo.py
import abc
from dataclasses import dataclass

@dataclass
class Base(abc.ABC):
    a: str

    def __post_init__(self):
      self.a = self.a.upper()

    @abc.abstractmethod
    def process(self) -> str:
        pass

So far so good. Since the __post_init__ method is not an abstract one, it'll be executed in each class that inherits from Base.

Now it's time to create a class that implements the abstract class. As it is described in the reference, for inheritance in dataclasses to work, both classes have to be decorated.
In this case, the implementation will define another field, b of type str, reimplement the __post_init__ method, and implement the abstract method process.

# demo.py
@dataclass
class Implementation(Base):
    b: str

    def __post_init__(self):
        super().__post_init__()
        self.b = self.b.lower()

    def process(self) -> str:
        return f"{self.a} {self.b}"

We're reimplementing the __post_init__ method just to show that we could cover more sophisticated use cases easily. This forces us to call super().__post_init__() to get the post initialization of the base class. Now we can cover the behavior of this new class in a test as follows.

# test_demo.py
from demo import Implementation

def test_implementation():
    implemented_instance = Implementation("Pythonic", "Musings")
    assert isinstance(implemented_instance, Base)
    assert implemented_instance.process() == "PYTHONIC musings"

In this test, the first assert is to make sure that the Implemented class is really an instance of Base. We could have just trusted Python, but since this article has an educational purpose, better to be explicit about our expectations.

One great advantage of dataclasses is that you're forced to do type annotation. So we can see what happens if we run a type checker like mypy on it. Executing pipenv install mypy and then mypy . we get the following

demo.py:7: error: Only concrete class can be given where "Type[Base]" is expected
...

Only one error! That's worse than what we'd expected. Searching a bit, I found the following issue discussing this situation. Otherwise all checks up fine, and we're not interested in mypy's edge cases so we could ignore it, or silence it in case you're running mypy as part of a CI.

At last we reach now to combining dataclasses and the collections.abc module. This combination is great since both modules provide ways to reduce boilerplate while also making intent very clear. To keep it simple it'll be a straight container with a field c of type List and a custom method capitalize.

# demo.py
import collections.abc
from typing import List

@dataclass
class Derived(collections.abc.Container):
    c: List[str]

    def __contains__(self, value):
        return value in self.c

    def capitalize(self) -> List[str]:
        return list([e.upper() for e in self.c])

Here we get the full power of combining two boilerplate reducing approaches.
This example, by no means shows all the potential of the collections.abc since we chose the simplest collection possible. But it's only here to show that the combination with dataclasses works. I really recommend using the collections.abs module as it will allows you to encapsulate a lot, and leads to a better design.

To test it, we can go with the following code

# test_demo.py
from demo import Derived

def test_derived():
    instance = Derived(["Hi", "Bye"])

    for element in ["Hi", "Bye"]:
        assert element in instance

    assert instance.capitalize() == ["HI", "BYE"]

The first assertion checks if our method works as intended. The second group of assertions, those inside the for loop, has only a pedagogical objective since we are ultimately testing that the collections.abs is working. Although sometimes, such a test is valid to comply with specification. Anyhow, will not go into epistemology of tests in here. Here again we see that the mix of collections.abs and dataclasses just works.

Our implementation of the Derived class is very rough. As an exercise, try to turn the capitalize method into a classmethod that takes an instance of Derived and returns an instance of the class with the capitalized elements. This would improve the ergonomics, since is not too coherent to return a list in such an implicit way. Bonus points for proper type annotation! (Hint: check PEP 563.)

With this we conclude the article. Not surprisingly, the dataclasses module work extremely well together with abc and collections.abs. Certainly, after this exploration, I'll start using these combinations into the future and going through older code to make use of it.

Teaser: there is another thing in Python I enjoy using, the '@property' decorator. Personally, I also wonder how that mixes with dataclasses, luckily someone already told their story here. Spoiler alert, has a happy ending ;). Although while playing with it I've found some edge cases where things stop working nicely. I hope to explore it in more detail and show some alternative/solution.

Exploring Rust

Ignacio Vergara Kausel — Fri, 15 Mar 2019 10:00:00 +0000

For a while I’ve been interested in Rust, reading articles and books, and trying to do some small exercises in it, but in general still rather superficial. Now, after reading the following the second part of a series of articles on *Scientific computing in Rust, I decided to use that example to explore a bit more and challenge myself. The article mentions very early the following

But what happens when someone else comes up with a new numerical type with a legitimate concept of addition and multiplication (e.g. complex numbers)?

to motivate exploring generics, but the scenario of complex numbers was not explored. Thus, I wondered how to use the example function used in it, namely generic_scalar_product, with a more complex type., e.g., complex numbers. This means to implement the num_traits::Zero trait into a type implementing a complex number since the function has such trait as type bound for its arguments.

Complex number

(Un)Fortunately, this case is easy by using the related crate num_complex that indeed implements the Complex type with the needed trait num_traits::Zero. Thus, taking the code from the article and making use of the num_complex crate we get the following (playground)

extern crate num_traits;
extern crate num_complex;

use num_complex::Complex64;

fn generic_scalar_product<T>(v: &[T], w: &[T]) -> T
where
    T: num_traits::Zero + std::ops::Mul<Output = T> + Copy
{
    let length = v.len(); 
    assert_eq!(length, w.len()); 

    // We initialise `sum` to a "zero" of type T
    // using the `zero` method provided by the `Zero` trait
    let mut sum = T::zero(); 
    for index in 0..length {
        sum = sum + v[index] * w[index];
    }
    sum
}

fn main() {
    // Complex
    let v: Vec<Complex64> = vec![Complex64 { re: 1.0, im: 0.0 }, Complex64 { re: 1.0, im: 0.0 }, Complex64 { re: 1.0, im: 0.0 }];
    let w: Vec<Complex64> = vec![Complex64 { re: 1.0, im: 0.0 }, Complex64 { re: 1.0, im: 0.0 }, Complex64 { re: 1.0, im: 0.0 }];
    assert_eq!(generic_scalar_product(&v, &w), Complex64 { re: 3.0, im: 0.0 });
}

So, this was not much of a challenge… still a nice excercise/warmup for a beginner (like myself) in Rust to gain some confidence. But then, I was already commited to explore some more. This leads then to the real challenge of how to use this generic_scalar_product with a type of our own making. We could take for example quaternions or some other thing.

(Local) Custom type

We could take two paths to develop this example: * Build piece by piece doing pair programming with the rust compiler implementing all needed traits to our new type until we stop having compilation errors. * Write our type already implementing commonly expected traits like Add and Mul as if it were a “mature” piece of code we already have and we want to use with the generic_scalar_product function.

I’ll take the second option, and then set out to extend it to make it compatible with generic_scalar_product.

Our new type will be NAlgebra, which will be implemented as two underlaying f64 types. This is to make our task easier by not needing to take care for generics at this point, but it’d be a good next exercise.

use std::ops::Add;
use std::ops::Mul;

#[derive(Debug)] // debug to be allowed to be printed
struct NAlgebra {
    a: f64,
    b: f64,
}

impl NAlgebra {
    fn new(x: f64, y: f64) -> NAlgebra {
        NAlgebra { a: x, b: y }
    }
}

impl Add for NAlgebra {
    type Output = NAlgebra;

    fn add(self, other: NAlgebra) -> NAlgebra {
        NAlgebra {
            a: self.a + other.a,
            b: self.b + other.b,
        }
    }
}

impl Mul for NAlgebra {
    type Output = Self;

    fn mul(self, rhs: Self) -> Self {
        let a = -self.a * rhs.a;
        let b = -self.b * rhs.b;
        NAlgebra::new(a, b)
    }
}

fn main() {
    let v: Vec<NAlgebra> = vec![NAlgebra { a: 1.0, b: 0.0 }, NAlgebra { a: 1.0, b: 0.0 }, NAlgebra { a: 1.0, b: 0.0 }];
    let w: Vec<NAlgebra> = vec![NAlgebra { a: 1.0, b: 0.0 }, NAlgebra { a: 1.0, b: 0.0 }, NAlgebra { a: 1.0, b: 0.0 }];
    assert_eq!(generic_scalar_product(&v, &w), NAlgebra { a: -3.0, b: 0.0 });
}

It works, in the extent of our test and for the purposes of our example. Now the interesting part starts. We need to implement the relevant trait for our generic_scalar_product function, namely num_traits::Zero. We can check the documentation of the Zero trait, and we see the following

pub trait Zero: Sized + Add<Self, Output = Self> {
    fn zero() -> Self;
    fn is_zero(&self) -> bool;
}

Thus, the methods to be implemented are fn zero() -> Self and fn is_zero(&self) -> bool. This can be easily achieved as follow (for some help/inspiration see the code in num_complex crate for the implementation onto Complex)

// implementing the Zero trait for our custom type
impl num_traits::Zero for NAlgebra {
    fn zero() -> Self {
        NAlgebra::new(Zero::zero(), Zero::zero())
    }
    fn is_zero(&self) -> bool {
        self.a.is_zero() && self.b.is_zero()
    }
}

Here we’re assuming that the fields a and b in NAlgebra are supported by the num_traits crate, which is the case since our underlaying data is f64. Thus, the whole code looks as follows (playground)

extern crate num_traits;

use num_traits::Zero;  // importing trait Zero into local scope

use std::ops::{Add,Mul};

#[derive(Debug, PartialEq, Copy, Clone)]
struct NAlgebra {
    a: f64,
    b: f64,
}

impl NAlgebra {
    // Another static method, taking two arguments:
    fn new(x: f64, y: f64) -> NAlgebra {
        NAlgebra { a: x, b: y }
    }
}

impl Add for NAlgebra {
    type Output = NAlgebra;

    fn add(self, other: NAlgebra) -> NAlgebra {
        NAlgebra {
            a: self.a + other.a,
            b: self.b + other.b,
        }
    }
}

impl Mul for NAlgebra {
    type Output = Self;

    fn mul(self, rhs: Self) -> Self {
        let a = self.a * rhs.a;
        let b = self.b * rhs.b;
        NAlgebra::new(a, b)
    }
}

impl Zero for NAlgebra {
    fn zero() -> Self {
        NAlgebra::new(Zero::zero(), Zero::zero())
    }
    fn is_zero(&self) -> bool {
        self.a.is_zero() && self.b.is_zero()
    }
}

fn generic_scalar_product<T>(v: &[T], w: &[T]) -> T
where
    T: num_traits::Zero + std::ops::Mul<Output = T> + Copy
{
    let length = v.len(); 
    assert_eq!(length, w.len()); 

    // We initialise `sum` to a "zero" of type T
    // using the `zero` method provided by the `Zero` trait
    let mut sum = T::zero(); 
    for index in 0..length {
        sum = sum + v[index] * w[index];
    }
    sum
}

fn main() {
    // NAlgebra
    let v: Vec<NAlgebra> = vec![NAlgebra { a: 1.0, b: 0.0 }, NAlgebra { a: 1.0, b: 0.0 }, NAlgebra { a: 1.0, b: 0.0 }];
    let w: Vec<NAlgebra> = vec![NAlgebra { a: 1.0, b: 0.0 }, NAlgebra { a: 1.0, b: 0.0 }, NAlgebra { a: 1.0, b: 0.0 }];
    assert_eq!(generic_scalar_product(&v, &w), NAlgebra { a: 3.0, b: 0.0 });
}

Besides having to implement the Zero trait onto NAlgebra, the struct had algo to be extended with the Clone+Copy annotations due to the sum = sum + v[index] * w[index]; line. This was nicely provided by the compiler, and while I understand why Copy is needed I’m not sure why Copy and Clone have to be specified. This is something I’ve encountered before too in other examples and should give some more thought.

There are quite some more ways to keep working on this little example, to extract more learning nuggets. I’ll list a few that come to mind: 1. Generalizing the definition of NAlgebra to support a generic underlaying type would be an interesting next step. 2. Taking an external crate like quaternion and extending it from outside to be able to be used in generic_scalar_product. 3. Improve the structure of this example and put all the NAlgebra stuff in its own module within the same crate. 4. Use Rust’s testing capabilities to produce proper testing of NAlgebra and its later added behavior.

I’d say that the first two have some overlap, while the last two ones could be tackled together.

So, go on… explore and play with these ideas!

Bonus

At the end of this excursion, there was one thing that kinda bothered me. The return value of generic_scalar_product is T. As an example is fine, and avoids burdening the begginer reader with error handling, but I believe that showing proper error handling early on is extremely important. This means, that the return of that function should be something more like Result<T,Err> with the assert_eq!(length, w.len()); replaced with an early return with the error. To make it easy, the error returned will be just a string thus the type will be Result<T,String>

Quickly attending to this last quibble (playground)

extern crate num_traits;

fn generic_scalar_product<T>(v: &[T], w: &[T]) -> Result<T,String>
where
    T: num_traits::Zero + std::ops::Mul<Output = T> + Copy
{
    let length = v.len(); 

    if length != w.len() {
        return Err("The vectors have different lengths!".to_string());
    }

    // We initialise `sum` to a "zero" of type T
    // using the `zero` method provided by the `Zero` trait
    let mut sum = T::zero(); 
    for index in 0..length {
        sum = sum + v[index] * w[index];
    }
    Ok(sum)
}

fn main() {
    // Unsigned integers
    let x: Vec<u32> = vec![1, 1, 1];
    let y: Vec<u32> = vec![1, 0, 1];
    assert_eq!(generic_scalar_product(&x, &y).unwrap(), 2);
}

And sure, the unwrap() at the end might seem less than ideal, but the main() function is acting as a test suite and we’re handcrafting the inputs. In this case, to me, it seems completely fine to just unwrap the result.