DEV Community: Fredrik Sjöstrand

Writing a Text Editor in 7 Minutes using Textual

Fredrik Sjöstrand — Thu, 05 Jun 2025 21:44:32 +0000

I was trying to prepare a lightning talk at work and thought about a small experiment I did with Textual. I played around with writing a client that interacts with Outlook through the win32 API. Trying to open the Python file in Vim, I noticed I didn't have Vim installed 😱. That's when the question hit me: Could you live-code a text editor during my lightning talk, in 10 minutes or less? Spoiler: yes, yes you can. You just need the right tools.

A few things before we get started. The full code that I will walk through in this post is available in this gist. The dependencies for this little application are available here. In the post, I will use ... in the code to signify the code we have already written. This is to keep things a bit neater. Now, let's dive in!

Before getting into the main part, I will add some scaffolding for the application. First, we will add all the imports for the application. Then we create a class Editor that inherits from the App class from Textual. This is all that is required to create an empty application. I also want to solve argument parsing right off the bat. For this, we will use ArgumentParser from argparse in the standard library. Here we take a single argument, folder, the folder the editor should open. We run some basic input validation, like checking that the folder exists. We also assume that the user wants to open the parent folder if we have a file in the path.

from argparse import ArgumentParser
from pathlib import Path

from textual.app import App
from textual.widgets import TextArea, DirectoryTree
from textual.containers import Horizontal

class Editor(App):
    pass

if __name__ == "__main__":
    parser = ArgumentParser("My Editor")
    parser.add_argument("folder", type=Path)

    args = parser.parse_args()
    folder: Path = args.folder

    if not folder.exists():
        raise FileNotFoundError(f"No folder found for path: {folder}")
    if not folder.is_dir():
        folder = folder.parent

    app = Editor(folder)
    app.run()

As we expect to pass a folder to the Editor class, we have to implement the __init__ method.

class Editor(App):
    def __init__(self, folder):
        super().__init__()
        self.folder = folder
        self.file = None

We take the folder as an input argument, storing it as a field. We also create a field for file that we will use to keep track of the current file open in the editor. Note that it´s essential to do the super().__init__() call here so textual can wire up the application properly.

Running the application, we still see just an empty screen, which is quite boring. So, let´s start getting some widgets on the screen.

class Editor(App):
    ...
    def compose(self):
        yield Horizontal(
            DirectoryTree(self.folder),
            TextArea("", id="editor")
        )

Now we get a screen with two columns. To the left, it shows the file and folder structure of the folder we asked it to open. To the right, it shows an empty text area. You can write in the text area, however, you can't open any file. Let's fix that next.

Textual uses the naming of methods to hook into the event system. This has the general syntax of _on_<object_type>_<event>. Here we want to hook into:

an event on the DirectoryTree -> _on_directory_tree_<event>
and trigger when a file is selected -> _on_directory_tree_file_selected

class Editor(App):
    ...
    def _on_directory_tree_file_selected(self, event):
        path: Path = event.path
        if not path.is_file():
            return

        self.file = path

        text_editor = self.query_one("#editor")
        text_editor.text = self.file.read_text()
        text_editor.language = "python" if self.file.suffix == ".py" else None

In this method, we check that the path is indeed a file, returning early otherwise. We make sure to assign this path to self.file, we will need this later. To open the file, we need to get a reference to the editor text area. We can query for it using the ID we defined earlier, prefixing it with a #. Now we copy the text from the file into the text area. To add some flair, we also add syntax highlighting from Python files. Note that we need an extra dependency for syntax highlighting, using pip install textual[syntax] installs this for you.

Yaaaay, now we can edit the file in our editor! We are done!

... but wait, how do I save the file ???

To do this in Textual, we will use keyboard bindings and actions. Let's define bindings for saving a file as ctrl+s. And, while we're here, let's beat Vim in quitability by binding ctrl+q to quit the editor 😉. We declare binding by assigning a list of tuples to a class-level variable named BINDING. The first value is the keyboard shortcut, and the second value, "save_file" and "quit", will be mapped to actions. The action quit is already defined in Textual, but save_file isn't. Instead, we define our own method named action_save_file that Textual will, by using the name, hook up to the binding.

class Editor:
    BINDINGS = [
        ("ctrl+s", "save_file"),
        ("ctrl+q", "quit"),
    ]
    ...

    def action_save_file(self):
        if self.file is None:
            return
        editor = self.query_one("#editor")
        self.file.write_text(editor.text)

We return early if no file is open. Otherwise, we get the text area and write the text content back into the original file. It's for this step that we need self.file, to know where to store the new content.

Below is a screenshot of the editor we just coded up. It’s not the most impresive editor in the world, but it’s pretty damn good for something you can write in 7 miuntes.

I got a bit carried away and wrote an extended version of the editor; the code is available here, and a preview is shown below. This adds some extra elements to the UI and a command palette, allowing you to open files in the directory. The command palette is a powerful feature built into Textual that allows for searchable, dynamically generated commands. In the improved editor, I used this to generate a list of commands to open files in the directory. I will not go into the details here, but in the gist for the improved editor, you can see how this was done.

Finally, now that you know how it's done, can you beat my time and write it in less than 7 minutes? Or maybe extend the application in some interesting way? Maybe you have some other tool that would work well as a terminal application? Either way, I would recommend that you experiment a bit with textual. It’s a powerful library and really, really fun to play with.

Tips & Tricks for Writing a Workaround

Fredrik Sjöstrand — Mon, 01 Aug 2022 09:38:05 +0000

Most of you have probably written a workaround for some bug in a web API or a third-party package. If you haven't, trust me, it's just a matter of time. The question then becomes, how can I make sure the fix isn't pre-maturely removed or overstays its welcome? I aim to give you some tools I have found useful when writing workarounds. And if you have any good tips, please add them in the comments!

Our Example

First off, I want to set up a fictive example to compare against. This is a class of a third-party library, finder_lib, I am using:

class Finder:
    def __init__ (self, pattern):
        self.pattern = pattern

    def find(self, text):
        if self.ignore_case:
            text= text.lower()
        return text.find(self.pattern)

Here we have a class that takes a pattern when instantiating the object. You can then use the finder object to find that pattern in multiple strings. However, we have a bug! When calling the find method it will crash because self.ignore_case isn't defined. This value should be set when creating the object, but we can't as the __init__ method doesn't take it as a parameter. Yes, this is a very simple case but it is based on a workaround I had to write once. While that class was more complex, the bug is structurally the same. It also had a required field that wasn't accepted in the __init__ method.

Python is a very dynamic language that allows us to easily work around this issue:

from finder_lib import Finder

finder = Finder("hello")
finder.ignore_case = True
finder.find("Hello World")

Now we have a bug and a way to work around it. Time to get into the main part of this post, how to manage this workaround during its lifetime.

Extract the Workaround

Making sure the workaround is written only once goes a long way. It makes it sooo much easier to remove when the day comes. In this case, I would extract it to a function:

from finder_lib import Finder

def create_finder(pattern, ignore_case=False):
    finder = Finder(pattern)
    finder.ignore_case = ignore_case
    return finder

finder = create_finder("hello", True)
finder.find("Hello World")

Remember that we have a lot of tools in the Python arsenal. If the workaround is complex we may use modules or classes to break it up. If it has set-up and tear-down, maybe we should use a context manager. Maybe a decorator can be used to make sure it always runs before or after the required functions. In some cases, even monkey patching might be the best option. What tools to use will depend on the nature of the workaround. Use your best judgment and you will be fine.

Comments

For workarounds, code comments are a very useful tool. I don't think the most avid opponents of code comments would argue against their use here. You see, workarounds often require a lot more context to be understood. They are, almost by definition, breaking the ordinary programming structures, hierarchies, or flows of our application code.

Let's start with an example of how I write my code comments for workarounds and then I'll break it down.

from finder_lib import Finder

def create_finder(pattern, ignore_case=False):
    finder = Finder(pattern)
    # FIXME: Workaround, missing ignore_case parameter in __init__ [2020-06-16]
    # Implemented for version 2.1.12 of finder_lib.
    # This is a known issue documented at https://github.com/finderlibteam/finderlib/issues/1234
    finder.ignore_case = ignore_case
    return finder

finder = create_finder("hello", True)
finder.find("Hello World")

It's so common to use prefixes such as "fixme" or "todo" in code comments that many editors have built-in support or plug-ins for finding them. In VS Code FIXME is highlighted (I think by default) and I use the plug-in Todo Tree to get a list of all these comments in the codebase.

One distinct part of these comments is that I always add the date in brackets at the end of the first line. You could argue that source control will show when the workaround was written. You would be correct, however, I find that the date pulls my attention to the comment. Especially if the date was a long time ago. My thought is that others will react to old dates the same way. What happened here two years ago? Hopefully this will trigger them to look into if the workaround can be removed. You see, my goal is to get the workaround expelled from code as soon as possible. For the same reason, I think it is a good idea to add information like the library version. If I know we are on 3.4.0, a fix for 2.1.12 might very well be out of date.

If there isn't an issue in the library's issue tracker, you should preferably create one. When you have found the issue or reported it, I would recommend adding the link in the code comments. This makes it easier to check in on the progress next time you scroll past this piece of code.

Some final notes on comments. In cases where the workaround is more complicated than this example, add more context. Maybe describe in more detail what the issue is and how the workaround aims to solve it. If there is a planned or expected version for the bugfix, write down this version as well. Context is everything when you try to understand strange code.

Sidenote: You might be screaming at me for not using doc-strings here. In general, I would add a doc-string as well with some of this information. However, I would keep the comments as they are more likely to be picked up by todo-finding tools.

Testing

A helpful method for detecting if a workaround has served its purpose is to add a test that will fail when the fix can be removed. This might not always be possible, but if you can it's nice to have. For our example a test could look like this:

from finder_lib import Finder

def test_Finder_workaround_can_be_removed():
    error_was_raised = False
    try:
        Finder("hello", ignore_case=True)
    except TypeError:
        error_was_raised = True
    msg = "Finder accepts ignore_case, remove workaround in mymodule.create_finder"
    assert error_was_raised, msg

This test will be detected by pytest, a common third-party testing framework, and will pass as long as Finder doesn't accept ignore_case. We detect this by looking for a TypeError, as this exception is raised when passing an unsupported parameter to a function or class in Python. As long as the error has been raised the workaround is still valid.

Summary

When you have written a workaround your main goal is that it should have as little impact on the application code as possible. This means that we want the impact on the code to be small, have the workaround removed as soon as possible, and for everyone to understand why the code looks strange. We can do this by extracting the workaround, for example into a function, minimizing its footprint in the application code. We should also take extra care to document it thoroughly. The documentation should make sure everyone understands when it was created, why it is needed, and when it can be removed. Finally, in some cases, we can also write a test that fails as soon as the workaround can be removed. Having a test makes it automatic, which is great.

Final notes

I hope you found this post helpful and can use some of these techniques for your next workaround. If you have any tips of your own, please add a comment below, I would like to hear how you have approached this in your own work. Happy coding!😄

A Personal .gitignore - Even for a Single Repository

Fredrik Sjöstrand — Tue, 07 Jun 2022 18:27:17 +0000

I think I might be a bit messy, but maybe so are you? Either way, I quite often find myself with stuff in a repository that I don't want to delete or commit. Then the question becomes, should I add this to the .gitignore file? Quite often I feel like i shouldn't, as no one else working on the project will benefit from it. If everyone is adding their personal stuff to the file, it will get messy quite fast. What I want is a personal .gitignore. Does this exist? You bet it does!

Solution 1 - Configuring a Global .gitignore

One solution I have seen suggested is to use a global .gitignore. You configure this by setting the path in your global git config like this:

$ git config --global core.excludesFile ~/.gitignore

Here I use a file named .gitignore in my home folder, but this could be any path.

My main issue with this solution is that this applies to all repositories. This can be useful if you for example always use the same name for your python virtual environments. However, quite often I find myself with some files that are relevant for a single project, making a global solution the wrong choice.

Solution 2 - Configuring a Local .gitignore

The observant reader might have noticed we used the --global flag with the command. May we not just change this to --local in the previous command? Yes, yes we can! This allows us to set the configuration for another .gitignore file for the single repository. To not have a naming collision we will need to name it something else, e.g. .mygitignore. The command looks like this:

$ git config --local core.excludesFile .mygitignore

This will look for a file named .mygitignore in the current repository. Now you just need to create the file and add what you want to ignore to it. Here I would also suggest adding the .mygitignore file to itself. It will still be used but won't show up when you do e.g. git status

Sidenote: The local configuration file is stored in the .git/config in your repository if you ever need to change something.

Solution 3 - .git/info/exclude

You can also cut the middle man from the previous solution. Git already has a local .gitignore type file inside it that you can use.
This is the file .git/info/exclude. It uses the same syntax as your bog-standard .gitignore and is automatically used by .git, no need for any configuration. It doesn't get committed and is specific to the repository.

Closing Remarks

Of course, you should be careful that this doesn't result in duplicate work for your teammates. If everyone has a notebooks folder or names their virtual environments .venv, it is probably better to just add it to the .gitignore.

I'm torn between the second and third solutions. With the third solution, I like that it's standardized in git and doesn't require configuration. However, it hiding out in the .git folder makes it harder to remember when returning to an old project. In general, I never look in the .git folder unless I have to. So, I will go with the second solution. But, pick your poison as they say.

Exploring Python: Using and Abusing the Iterator Protocol

Fredrik Sjöstrand — Mon, 03 May 2021 14:53:45 +0000

The concept of iterators isn't in any way specific to Python. In the most general terms, it is an object that is used to iterate (loop) over a sequence of items. However, different programming languages implement them differently or not at all. In Python every for-loop uses an iterator, differing from many other languages. In this post, I will cover how iterators are defined in Python, and on the way, we will take a look at iterables and what I call nextables.

Iterables

An iterable is an object that can be iterated over, for example, a list. In python, for an object to be an iterable it has to implement the __iter__ method. This dunder method takes only the object instance, self, as input and has to return an iterator object. You can use the built-in function iter to get the iterator of an iterable object.

Note that an iterable isn't necessarily an iterator as it doesn't actually perform the iteration. You could have a separate iterator object that is returned from the iterable class, rather than the class handling its own iteration. But, more on this later.

Iterators

Let's move on to the actual iterators, the workhorse of iteration (especially in Python). Iterators are an abstraction layer that encapsulates the knowledge of how to take items from some sequence. I am intentionally very general here as the "sequence" can be anything from lists and files to data streams from a database or remote service. What's cool about iterators is that the code using the iterator doesn't even have to know what source is used. Instead, it can focus on one thing only and that is, "what should I do with each element?".

Non-iterator Iteration

To better understand the benefits of iterators, let's take a short look at iteration without iterators. An example of non-iterator iteration is the classic C-style for-loop. This style exists in not only C but also in e.g. C++, go, and javascript.

An example of this in javascript:

let numbers = [1,2,3,4,5]
for(let i=0; i < numbers.length; i++){
    // Extract element
    const current_number = numbers[i]
    // Perform action on element
    const squared = current_number ** 2
    console.log(squared)
}

Here we see how this type of for-loop has to have both the extraction of and action for each element bundled.

All Python for-loops uses Iterators

In Python, we don't have C-style for-loops. Instead, Pythons for-loops are what in other languages might be called "for each"-loops. A type of loop that uses iterators. This means that every for-loop you write in Python has to use an iterator.

First let's look at the syntactically closet equivalent of the previous example:

numbers = [1,2,3,4,5]
for i in range(len(numbers)):
    number = numbers[i]
    squared = number**2
    print(squared)

Yes, I know I should just have looped over numbers, but I wanted to push it a bit closer to the javascript for-loop. Here we need to perform the extraction ourselves, so we are not using an iterator over numbers. Rather, we are creating a range that goes over the indices of numbers (an iterator). While this is relatively close to the javascript example, it still works at a higher abstraction level.

If you look closely at the javascript example you can see how we actually tell the loop when to end i < numbers.length and also how to increment i++. So to get Python code closer to this abstraction level, we would actually need to write something like this:

numbers = [1,2,3,4,5]
i = 0
while i < len(numbers):
    number = numbers[i]
    squared = number**2
    print(squared)
    i += 1

Here we initialize i, define what condition should be met to exit the loop, and how to increment. We need to very explicitly manage the value of i, something not required when using range.

Python Iterator Protocol

In the Python documentation, an iterator is defined as a class that implements __next__ and __iter__. By this definition, an iterator is also an iterable (it implements __iter__). It is also what I will call nextable, as it implements the __next__ method. No, nextable is not a commonly used term and that's because you might as well make it into an iterator. You see the __iter__ method is trivial to implement for iterators. It is actually written out explicitly in the iterator definition what the method should do:

class MyABCIterator:
    ...
    def __iter__(self):
        return self

That's it, the method just returns a reference to the iterator itself. So if you copypaste this __iter__ code into your nextable, you have an iterator. I named the class MyABCIterator as I will build this into an iterator that iterates over the alphabet.

Now let's make it into an iterator by making it nextable as well. The __next__ method should return the next object in the sequence. It also has to raise StopIteration when we have reached the end of the sequence, often referred to as having "exhausted the iterator". In this case, when we have reached the end of the alphabet.

I will use a handy import, ascii_lowercase, from the string module in Pythons standard library. This is a string of all lowercase characters in the english alphabet.

>>> from string import ascii_lowercase
>>> ascii_lowercase
'abcdefghijklmnopqrstuvwxyz'

Okay, now let's look at the code for our class, then I will explain the implementation.

from string import ascii_lowercase
class MyABCIterator:
    def __init__(self):
        self.index = 0
    def __next__(self):
        if self.index >= len(ascii_lowercase):
            raise StopIteration()
        char = ascii_lowercase[self.index]
        self.index +=1
        return char
    def __iter__(self):
        return self

To know which character to return each time __next__ is called, we will need an index. Therefore, we add a __init__ to our class, where we initialize self.index to zero. Then, for each call to __next__ we will first check if we reached the end of the alphabet. If the index is outside the length of the string, we raise StopIteration as dictated in the Python documentation. Next, we extract the current character before incrementing self.index. Otherwise, we would start with b instead of a. Finally, we increment the index and return the character previously extracted.

Now, let's try it out in a for-loop:

>>> for char in MyABCIterator():
...    print(char)
a   
b   
c   
d   
e   
...

I truncated the output as the alphabet isn't that interesting now, is it? This iterator, as you might expect, is utterly useless. We could just as well iterated over ascii_lowercase directly. But, I hope you learned something about iterators at least.

A final note on the iterator protocol. Note how the for-loop does all the work in using the protocol. It automatically gets the iterator using __iter__ and repeatedly iterates over it using __next__. This follows Pythons pattern of dunder methods not being directly used, but rather a way for us to hook into Pythons syntax or top-level function.

Nextables

Now that we have properly implemented an iterator, let's experiment a bit. The first question I'd like to ask is:

What does Python thinks about looping over a nextable?

For this, I removed the __iter__ method from the previous example, resulting in:

class MyABCNextable:
    def __init__(self):
        self.index = 0
    def __next__(self):
        if self.index >= len(ascii_lowercase):
            raise StopIteration()
        char = ascii_lowercase[self.index]
        self.index +=1
        return char

Trying to create a for-loop with this object gives us: TypeError: 'MyABCNextable' object is not iterable. Which, you know isn't that surprising. The interpreter can't find a __iter__ to call and can, therefore, not create an iterator.

Python Separates the Iterator from the Sequence

I started playing around with the built-in sequences and made a fun little discovery. In Python, the sequences aren't iterators themselves. Rather each has a corresponding iterator-class responsible for the iteration. Let's look at range as an example.

>>> range_0_to_9 = range(10)
>>> type(range_0_to_9)
<class 'range'>

range() gives us a an object of type range. Now let's see what happen when we try to use next on this object.

>>> next(range_0_to_9)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'range' object is not an iterator

So if the range object isn't an iterator, what do we get from iter?

>>> range_0_to_9_iterator = iter(range_0_to_9)
>>> type(range_0_to_9_iterator)
<class 'range_iterator'>

Just to verify, we can use next with the range_iterator.

>>> next(range_0_to_9_iterator)
0
>>> next(range_0_to_9_iterator)
1

Here we get back a range_iterator object that is actually responsible for doing the iteration. This seems to be the case for all built-in types; including lists, dictionaries, and sets. It seems that in CPython, the developers have chosen to separate the iteration of an object from the object itself.

Creating a Separate Iterable and Nextable

Armed with this new knowledge of separating the iterable from the iterator, I got a new idea:

Can I return a plain nextable from the iterable?

The iterable is really easy:

class MyABCIterable:
    def __iter__(self):
        return MyABCNextable()

This is just a wrapper around our nextable from the nextables example. Then we write the loop:

for char in MyABCIterable():
    print(char)

It works! So, while Python documentation says that the __iter__ method must return an iterator (implementing both __iter__ and __next__), this isn't demanded by the for-loop.

However, this setup is fragile. Going back to our properly implemented iterator, this code will work:

my_abc = iter(MyABCIterator())
for char in my_abc:
    print(char)

However, with our iterable + nextable combo, it doesn't:

my_abc = iter(MyABCIterable())
for char in my_abc:
    print(char)

It raises TypeError: 'MyABCNextable' object is not iterable. This is the reason the iterator protocol is defined the way it is. It makes it possible to pass around the iterator and still loop over it. For example, if you could first create the iterator and then pass it off to another function. Our hacky solution wouldn't work for this.

Final Notes

I just want to end with some key takeaways from this post. The first one I think most of you know, all for-loops in Python uses iterators!. We also took a look at how iterators allow us to separate the code doing the iteration from the code working with each item. I also hope you learned a bit more about how iterators work in Python and what the iterator protocol is.

The final takeaway is a bit more abstract, it actually doesn't have anything to do with iterators or Python at all. Rather, it is how code might seem to work well until you stumble on the case where it breaks. If I would have called it a day after solving just trying the iterable + nextable combination in a for-loop, I wouldn't have discovered how it breaks when passing around the iterator.

That's it for now and I hope you enjoyed this deeper look at Pythons iterator protocol!

Edit (2022-01-19):

There was some discussion about how CPython just require next for the object returned from iter for e.g. for-loops to work. So the question was if the definition of an iterator should just include next or not. The python steering council decided that the protocol should still be defiend as next and iter, but that there is a CPython implementation detail that it is inconsistant with validating this (reference comment on github).

So it seems everything in the post is still correct, but that there will be an addition to the documentation noting this implementation detail of CPython. So, really just implement iter for your iterators. Swapping to other Python run-times might cause issues otherwise.

Different Barmodes Simultaneously in Plotly Subplots (Python)

Fredrik Sjöstrand — Thu, 04 Feb 2021 20:18:24 +0000

My first and most read blog post, by an order of magnitude, is this one about plotly:

Stacked and Grouped Bar Charts Using Plotly (Python)

Fredrik Sjöstrand ・ Jan 8 '20 ・ 4 min read

#python #visualization #plotly

This is probably why I recently received an email asking about how you can have two different barmodes simultaneously in Plotly subplots. The first one should be stacked and the second one should be grouped. As always I started off by searching around to see if someone else had solved this problem. However, the only post I could find was an answer on the plotly community forum saying it can't be done. Spoiler alert, it can be done. Let's get into it!

Sidenote: If you just want to dive into the code, you can find the entire example as one large block at the end.

The data

I am using the same data as for my previous blog post:

data = {
    "original":[15, 23, 32, 10, 23],
    "model_1": [4,   8, 18,  6,  0],
    "model_2": [11, 18, 18,  0,  20],
    "labels": [
        "feature",
        "question",
        "bug",
        "documentation",
        "maintenance"
    ]
}

You can think of this as a table/dataframe with four columns: original, model_1, model_2 and label. The content of the lists is the rows.

Plot

Based on the results from my initial research there is no straightforward way for doing this. However, a natural starting point is looking at how to do subplots to start with. First of all, let's look at what we need from plotly:

from plotly.subplots import make_subplots
from plotly import graph_objects as go

graph_objects, shortened to go, is the core library of plotly. For this example, all we will use is the Bar class, which represents a bar chart. I think all of you already figured out that we will use makes_subplots to make, well, subplots.

A standard subplot to start

Here we go, a standard subplot with two bar charts to start off.

fig = make_subplots(rows=2, cols=1)

# Subplot 1
fig.add_trace(
    go.Bar(
        name="Original",
        x=data["labels"],
        y=data["original"],
    ),
    row=1,
    col=1,
)

# Subplot 2
fig.add_trace(
    go.Bar(
        name="Model 1",
        x=data["labels"],
        y=data["model_1"],
    ),
    row=2,
    col=1,
)

fig.show()

Not a lot of interesting things going on here. We create a subplot with two rows and one column. We then add two bar charts, one to the first row and one to the second row. The result looks like this:

Multiple bar charts in each subplot

Okay, now we have a solid starting point. The make_subplots function returns a single figure object, to which we add our traces. When we add the traces we tell plotly which of the subplots the trace should be added to. Let's expand our solutions to contain multiple bar charts in each of the subplots.

fig = make_subplots(rows=2, cols=1)

# Subplot 1
fig.add_trace(
    go.Bar(
        name="Original",
        x=data["labels"],
        y=data["original"],
    ),
    row=1,
    col=1,
)
fig.add_trace(
    go.Bar(
        name="Model 1",
        x=data["labels"],
        y=data["model_1"],
    ),
    row=1,
    col=1,

)

# Subplot 2
fig.add_trace(
    go.Bar(
        name="Original",
        x=data["labels"],
        y=data["original"],
    ),
    row=2,
    col=1,
)
fig.add_trace(
    go.Bar(
        name="Model 1",
        x=data["labels"],
        y=data["model_1"],
    ),
    row=2,
    col=1,
)

fig.show()

To each of the subplots, we add two bar charts, both containing the same data. The result looks like this.

The Solution

Now we can get to the final step of the solution. Styling the independent bar charts. Usually, we would have used something like this to change to a stacked bar chart:

fig.update_layout({"barmode":"stack"})

However, this automatically targets both subplots and there is no way of changing that. The solution then is to use the same trick as in my previous plotly post and manually create stacked and grouped bar charts. This works as this changes the Bar object directly and isn't a style on the Figure.

fig = make_subplots(rows=2, cols=1)

# Subplot 1 - Stacked
fig.add_trace(
    go.Bar(
        name="Original",
        x=data["labels"],
        y=data["original"],
        offsetgroup=0,
    ),
    row=1,
    col=1,
)
fig.add_trace(
    go.Bar(
        name="Model 1",
        x=data["labels"],
        y=data["model_1"],
        offsetgroup=0,
        base=data["original"]
    ),
    row=1,
    col=1,

)

# Subplot 2 - Grouped
fig.add_trace(
    go.Bar(
        name="Original",
        x=data["labels"],
        y=data["original"],
        offsetgroup=0,
    ),
    row=2,
    col=1,
)
fig.add_trace(
    go.Bar(
        name="Model 1",
        x=data["labels"],
        y=data["model_1"],
        offsetgroup=1,
    ),
    row=2,
    col=1,
)

fig.show()

When creating the stacked version we set the offsetgroup to zero for both Bar objects. This places them in the same position along the x-axis. However, the second will only be rendered in front of the first one, instead of above. Therefore, we set the base parameter for the second Bar object to be the list of values of the other one. This shifts the y-position of each bar in "Model 1", based on the height of the corresponding bar in "Original", creating a stacked bar chart. To create the grouped subplot, all we have to do is to assign different values to the offsetgroup of the Bar objects.

Sidenote: You can read more about how this stacking and grouping works in my previous post.

Now, for the final reveal.

Entier example

Here is all code for the entire finished example. As long as you have plotly installed you can just copy-paste this into e.g. a Jupyter notebook and get the final plot.

from plotly.subplots import make_subplots
from plotly import graph_objects as go

data = {
    "original":[15, 23, 32, 10, 23],
    "model_1": [4,   8, 18,  6,  0],
    "model_2": [11, 18, 18,  0,  20],
    "labels": [
        "feature",
        "question",
        "bug",
        "documentation",
        "maintenance"
    ]
}


fig = make_subplots(rows=2, cols=1)

# Subplot 1 - Stacked
fig.add_trace(
    go.Bar(
        name="Original",
        x=data["labels"],
        y=data["original"],
        offsetgroup=0,
    ),
    row=1,
    col=1,
)
fig.add_trace(
    go.Bar(
        name="Model 1",
        x=data["labels"],
        y=data["model_1"],
        offsetgroup=0,
        base=data["original"]
    ),
    row=1,
    col=1,

)

# Subplot 2 - Grouped
fig.add_trace(
    go.Bar(
        name="Original",
        x=data["labels"],
        y=data["original"],
        offsetgroup=0,
    ),
    row=2,
    col=1,
)
fig.add_trace(
    go.Bar(
        name="Model 1",
        x=data["labels"],
        y=data["model_1"],
        offsetgroup=1,
    ),
    row=2,
    col=1,
)

fig.show()

requirements.txt in your requirements.txt

Fredrik Sjöstrand — Thu, 19 Nov 2020 19:24:21 +0000

Are you managing multiple requirements files for your Python project? Are you copy-pasting rows from one file to the other when changing versions? Maybe you have one requirement file for development and one for production. Or maybe a separate file for that docker image. Fear no more! Here's how to do it.

Starting point

Let's jump straight into an example. In this application, I have two requirements files, one for production install, and one for development.

requirements.txt:

requests>=2.25.0
pandas>=1.1.0

requirements-dev.txt:

requests>=2.25.0
pandas>=1.1.0
pytest>=6.1.2
responses==0.12.1

As you can see we have duplicated the requests and pandas rows. This can lead to trouble when updating dependencies. Forgetting to update requirements.txt could, in the worst-case scenario, lead to downtime in production.

There must be a better way? -- Raymond Hettinger

Yes, there are at least two better ways!

Multiple -r Flags when Installing

You can actually use multiple files when running the pip install command. First, let's edit the files.
requirements.txt:

requests>=2.25.0
pandas>=1.1.0

requirements-dev.txt:

pytest>=6.1.2
responses==0.12.1

Now each dependency is just defined once! To install the full development environment we would run:

$ pip install -r requirements.txt -r requirements-dev.txt

While this works and is better than before, it still doesn't really feel right. This is will probably trip people up when joining the project. Let's look at the final and best way to solve this!

requirements.txt in requirements-dev.txt

When installing with pip we use the -r flag to refer to a requirements file. We can do this inside another requirements file as well.
requirements.txt:

requests>=2.25.0
pandas>=1.1.0

requirements-dev.txt:

-r requirements.txt
pytest>=6.1.2
responses==0.12.1

Now you can install all development dependencies using only: pip install -r requirements-dev.txt. To install the production environment you still use pip install -r requirements.txt.

Pip Install a Git Repository

Fredrik Sjöstrand — Thu, 22 Oct 2020 10:31:06 +0000

If you would like to install the latest version of flask, you would probably use pip install flask. While this gives you the latest released version, it doesn't contain the latest commit to the repository. At least not most of the time. Sooo, what if I want the latest commit? Maybe some new bugfix or feature has been added which I need for my project. Let's look at how you can get the latest commit made to flask or any other open-source package.

You could clone the repository and install it using pip locally:

$ git clone https://github.com/pallets/flask.git
$ pip install ./flask

However, this method creates a dependency on local files, and creating a requirements.txt you could share is impossible. Using pip freeze it would look like this for flask:

Flask @ file:///C:/Users/Fronkan/my_project/flask

Fear not, there is a better way of doing this! You can use pip to install directly from a git repository. To install flask the shortest version of the command is pip install git+https://github.com/pallets/flask.git.
This will install whatever is on the default branch of the project. Freezing this gives us:

Flask @ git+https://github.com/pallets/flask.git@bdf7083cfdeef0e0bdd0bf6d4c23b26c92b52d95

Notice how pip has stored the specific commit hash used when installing. This means you can share this with others and they will, for sure, get the same version as you!

As I said, that is the short version of the command. This corresponds to pip install git+https://github.com/pallets/flask.git@master, as master is the default branch of the flask repository. The reason I show you this is because we can change master to any other branch in the repository. Actually, we can use either a branch, commit hash, tag name, or git ref. This gives you a lot of control over what to install from the repository.

If you follow the pip documentation you would run the command as: pip install git+https://github.com/pallets/flask.git@master#egg=flask. The #egg=flask part tells pip which package it is installing before downloading and parsing the metadata and is used for the dependency logic. This seems to be the, by pip, recommended way of doing things.

Pip also supports other version control systems like Mercurial and SVM. If you use these instead of git or just want to read up on pips support for version control systems in general, check out the docs right here.

This isn't something you will use every day, I have used it once. I needed a version of a package available on the master branch which wasn't yet released, like with this flask example. However, you could also use this to install packages that aren't available on PyPI at all, for example, an internal project at work. I hope you found this as interesting as I did and if you see any other uses for this, please leave a comment!😄

A First Look at Python Protocols (PEP 544)

Fredrik Sjöstrand — Mon, 05 Oct 2020 05:48:42 +0000

I really like the type hints added to python 3.5, PEP 484, but somehow I totally missed PEP 544 - Protocols: Structural subtyping (static duck typing). Protocols were added to python 3.8 which, while it is the current stable version of Python, is almost a year old. We are very much closing in on Python 3.9, the release is planned for the fifth of October. In this post, I will take a first look at protocols and how they work with linting using Pylance in VS Code.

The title of the PEP says static duck typing, which is actually a really good description of protocols. They allow you to define an interface/template which an object has to fulfill, and if does it is said to be an instance of the protocol. Protocols are just checked by type-checking tools and aren't enforced at run-time.

sidenote: For others like me who have been playing around with TypesScript (TS), protocols are really similar to the interfaces of TS.

Type Checking Protocols in VS Code

I think the easiest way to understand protocols is to write some code, so let's dive in. First, we define a protocol Printable:

from abc import abstractmethod
from typing import Protocol
class Printable(Protocol):
    @abstractmethod
    def print(self) -> None:
        raise NotImplementedError

As you can see this is a class which inherits from the class Protocol. It has a single abstract method print, which takes no arguments and returns None. Note that in python None is returned if a function doesn't have a return statement.

Let's define a class that fulfills this protocol and a class that doesn't.

class MyPrintable:
    def print(self) -> None:
        print("Hello DEV!")

class NonPrintable:
    pass

I am using Visual Studio Code with the Pylance plug-in. With Pylance you can use the setting python.analysis.typeCheckingMode. This can be either off, basic or strict; I am currently running using basic. The screenshots in this post use these settings.

Now let's define a function that takes a Printable and see how the editor reacts to the two classes just created.

def simple_print(printable: Printable):
    printable.print()

If we pass an instance of MyPrintable and then NonPrintable to the function this is what shows up in the editor.

NonPrintable is underlined and if hovered we can see this error message:

Both Mypy and Pylance complains about how NonPrintable doesn't fulfill the Printable protocol. This also means that the MyPrintable object is correctly identified as fulfilling the protocol, no inheritance needed.

sidenote: I am really enjoying Pylance, look at that error message! It is just fantastic!

Like I mentioned earlier, at runtime this information isn't used. However, the program would, of course, crash when we try to call the print-method on NonPrintable as it doesn't exist.

Runtime Type Checking

In python, we commonly use the isinstance function to verify if an object is an instance of a class. Does this work for protocols as well? I guess we have to try it out and to do so, I wrote this:

def my_print(printable: Printable):
    if isinstance(printable, Printable):
        print("It is a printable!")
        printable.print()
    else:
        print("boooo, not a printable")

my_print(MyPrintable())
my_print(NonPrintable())

Now let's see what Pylance thinks about this.

Well, that doesn't look very good.

Okay, so a protocol doesn't work in the isinstance check. Let's look at the other error message as well.

Interesting, so to check an instance against the protocol using isinstance, we need to decorate our protocol with @runtime_checkable. Note, that you would be informed by this if you run the code as well. It would report this error at runtime:

TypeError: Instance and class checks can only be used with @runtime_checkable protocols

According to the error all we need to do is add the decorator to the class, like this:

@runtime_checkable 
class Printable(Protocol):
    @abstractmethod
    def print(self) -> None:
        raise NotImplementedError

Now running the same code will show this output:

It is a printable!
Hello DEV!
boooo, not a printable

Final Notes

PEP 544 gives us protocols that allow us to define what requirements a function or class has for a parameter. Even more interesting, it allows us to do this without using inheritance while still working with isinstance.

This was just a first look at protocols and there is a lot more to explore. For example, they support defining instance and class variables. Protocols can also be generic, extended, and merged. This adds a really interesting tool for writing type-checked Python and it will be fun to experiment with it in a future project.

Game Jam Post Mortem - Supply Run

Fredrik Sjöstrand — Sun, 06 Sep 2020 19:06:48 +0000

During two weeks I developed a game for the Game Jam, Kodsnacks Tvåveckorssylt - Fjärde Omgången. This was the fourth run of quite a small Swedish Game Jam from the community around the podcast Kodsnack. One of the core ideas with these jams is:

a game jam for busy people

Therefore, it goes over two weeks with the intent that each participant spends about 1-3 hours a day. This is in contrast to some other jams where you e.g. just have 48 hours. The general spirit of the jam is also very casual, making it a really nice experience.

The Game Jam was held in ... the end of May. MAY!?!? That's over three months ago. Time flies in a pandemic I suppose, anyway, about time to get this post published.

The Theme

The theme for the jam was isolation/loneliness. I started off brainstorming ideas with my fiancee. I must say the thinking about different game ideas is my favorite part of every game jam. In the end, I went with an idea inspired by the ongoing pandemic.

The Game

My submission to this game jam was Supply Run. A game about going to the store in a pandemic. You have to collect everything you need from the store and get out without getting infected. If you are interested in trying the game, it runs directly in the browser on itch: https://fronkan.itch.io/supplyrun.

Now a short Q&A.

How do I avoid getting infected?

Avoid other people. Unlike in real life, they have a helpful circle around them showing how far away you should be.

What if I can't? Like if a shopper corners me.

Hope the person you just bumped into wasn't infected ¯\(ツ)/¯

Where did you find that great music?

Thank you, I made it my self 😄! This was the first time I actually made sure to have time to create music for the game. Also one of the first times I created music at all. I don't feel it perfectly fits the theme and pace of the game. However, that doesn't really matter, I am really happy with producing even a passable music track for the game.

The core-loop is to avoid getting to close to the other shoppers while getting all pick-ups. While it would have been fun with proper AI for the shoppers, I didn't feel comfortable enough with the tools to take on such a large task. Instead, I opted for a simpler solution where each shopper has a fixed path through the store. The paths are made using splines and at the beginning of the game, how far along the path a shopper starts is randomized. This has the effect of people starting at different positions in the store. Each shopper follows their path until they reach the end of the store, at which point they start back at the entrance where the spline begins. From the player's perspective, it seems like there is a constant flow of people in and out of the store. Finally, each time you start the level some shoppers are randomly selected to be infected.

One of the best moments I had while playing, was when I was cornered in the store. I knew I was screwed and would get inside the infection range of one of the shoppers. The infection circle touches me... and nothing happens. This was the type of experience and feeling I hoped for with the game.

The Game Engine

I tried out the game engine Phaser for the first time. I have been learning some front end development the weeks leading up to the jam and thought it would be a nice challenge. It was at least challenging that much I can tell you 😅.

Phaser seems to have had almost a Python 2 to Python 3 style break when going from version 2 to 3. Hmmm, I see a pattern here. There are a lot of Phaser 2 tutorials and it seemed very hard to apply them in version 3 which I was using.

My other problem with Phaser is probably just a personality issue. I really couldn't find a good model for structuring my program. With some added stress when closing in on the deadline, the code became somewhat of a mess. If you want to give it a look, I published the code in this github-repo. I feel like the models of Unity and Löve, engines I used for other jams, have matched my mental model much better. I have recently played a bit with Godot as well, again without the same headache as with Phaser.

I opted for using typescript instead of pure javascript. Before the jam, I set up the build pipeline using webpack. I went with webpack as I at least looked at it before and I found a good tutorial. It was still infuriating to get everything to work and I am glad I got the basic setup working before the jam started.

I probably won't use Phaser a lot in the future. It really felt like I was fighting the engine at times. Also, it wasn't as fun using as other engines I have tried. If you haven't tried Löve, I really recommend you do. My experience is that creating games in Löve feels fun and easy. What I did like with Phaser, the same as with Löve, is that it feels lightweight. When opening the webpage the Phaser game loads really quickly without a loading screen like with a Unity game.

Final notes

Creating music was fun and the addition of music and/or sounds really adds to a game. I aim to create music for future game jams, hopefully learning a bit about creating music that fits the theme and pace of the game. In general, I will try to work more with sounds for game jams. I am also really happy with the pixel art for the pick-ups while I am less pleased with the environment, player, and NPCs. In the end, this just came down to not having the time.

Finally, I truly recommend joining game jams. It is a lot of fun especially when there is a good community around it. A special thanks to kodsnack for creating a great community!

TIL: Python Recursion Limit When Accessing Elements in Recursive List

Fredrik Sjöstrand — Wed, 05 Aug 2020 19:21:51 +0000

Earlier today I found this post by Ryan Westlund:

Article No Longer Available

It reminded me that in Python you add a reference to a list to itself, like this:

>>> my_list = [1]
>>> my_list.append(my_list)
>>> my_list
[1, [...]]
>>> my_list[1]
[1,[...]]
>>> my_list[1][1][1]
[1, [...]]

Sidenote: this is one of the cases where reference counting breaks down and why Python needs a garbage collector (GC). It is the GCs job is to clean up circular references like these. Now, back to the story!

Here we have recursion and in Python, we have a maximum recursion limit. If you have ever forgotten the base case in a recursive function you probably have seen this error:

RecursionError: maximum recursion depth exceeded during compilation

which is triggered when the limit is reached.

How does accessing the elements work? You can give your own classes the []-operator by implementing __getitem__(self, key). So we can probably think about doing this my_list[1][1] as recursive calls to __getitem__:

list.__getitem__(
    list.__getitem__(my_list, 1),
    1
)

It, therefore, seems reasonable that this would trigger the max recursion error if repeated enough times. What is the default maximum recursion limits?

>>> import sys
>>> sys.getrecursionlimit()
1000

Wow, that is more than the times I am willing to write [1] to test my hypothesis. Luckily for us, or at least for me, we can change the recursion limit to let's say 5 using sys.setrecursionlimit.

>>> sys.setrecursionlimit(5)
>>> my_list[1][1][1][1][1]
[1, [...]]

It must be an off by one error, let's do 6 access instead.

>>> my_list[1][1][1][1][1][1]
[1, [...]]

What!?!?

>>> my_list[1][1][1][1][1][1][1][1][1][1][1][1][1][1]
RecursionError: maximum recursion depth exceeded during compilation

Success!!! So, I found 14 access is what is needed for the maximum recursion error to trigger. Interestingly, I only needed 8 when using exec:

>>> exec("my_list" + "[1]"*8)
RecursionError: maximum recursion depth exceeded during compilation

I'm actually still not sure what causes this behavior. It is demonstrably true that accessing the elements can cause a RecursionError, however, I do not understand why it took 14 access with a recursion limit of 5. Might it be some sort of optimization going on here? Even more interesting is that using exec takes 8 accesses, more than the limit but less than the other method. If you have any idea on what might be causing the behavior, please leave a comment below. It would be really interesting to hear your thoughts on this one!

Don't Try This at Work: Exception Driven Fizz Buzz V2

Fredrik Sjöstrand — Sat, 01 Aug 2020 14:32:50 +0000

So I have decided I will embrace some of my, sometimes, let's call them "less practical ideas". Therefore, I will do this as a series, Don't try this at work or DTTAW for short. My previous and first post of the series was:

Don't Try This at Work: Exception Driven Fizz Buzz

Fredrik Sjöstrand ・ Jun 27 '20 ・ 3 min read

#python

However, I figured I would take on the challenge of removing all if-statements from the code. This was surprisingly easy. The solution:

import sys
import inspect
import traceback

def fizzbuzz(val):
    assert val % 15 != 0
    fizz(val)

def fizz(val):
    assert val % 3 != 0
    buzz(val)

def buzz(val):
    assert val % 5 != 0

def exception_driven_fizzbuzz(val):
    try:
        fizzbuzz(val)
        print(val)
    except AssertionError as e:
        tb = e.__traceback__
        while tb.tb_next:
            tb = tb.tb_next
        print(tb.tb_frame.f_code.co_name)


for val in range(1, 16):
    exception_driven_fizzbuzz(val)

The only difference from last time is that we have swapped the if-statements for assert-statements. In doing so we had to flip the logic checks. Instead of detecting that value is divisible by 3 for the fizz function we assert that it is not divisible by 3. Then if it is divisible by 3 an AssertionError is raised. As we are using AssertionError instead of ValueErrors we also had to modify the error type in the except-statement, but that is it!

Small caveat here, the change from if-statement to asserts actually introduces a bug. This bug is triggered only if you run the script using the -O flag: python -O fizzbuzz_v2.py. If you do, it will just print the numbers instead. The reason is that the -O flag runs the script with optimizations, which includes removing all assert-statements. No assert-statements gives no errors and results in no fizzbuzzing.

I hope this post was entertaining and can inspire you to try some unorthodox coding practices (at home of course). It is a fun way to explore the language where you might find some interesting and actually useful insights.

Don't Try This at Work: Exception Driven Fizz Buzz

Fredrik Sjöstrand — Sat, 27 Jun 2020 12:58:14 +0000

So, I was catching up on some of the talks from PyCon 2020. First, I watched Steven Lott - Type Hints: Putting more Buzz in your Fizz and then Elizaveta Shashkova - The Hidden Power of the Python Runtime. I think both talks were really interesting, however, together they inspired me to write this abomination of Fizz buzz implementation. If you haven't heard about Fizz buzz, see this Wikipedia page.

Background

So from Elizavetas talk, I learned a bit about how Python stack frames are easily accessible at runtime. Most importantly for this post, when you capture an exception using the except-keyword you can get the current stack frame from the traceback in the exception.

try:
    raise ValueError()
except ValueError as e:
    tb = e.__traceback__
    current_stack_frame = tb.tb_frame

After experimenting a bit I also found that you can use tb.tb_next, to get the next traceback object in the chain. So, if you iterate until there is no tb_next then you have reached the bottom of the trace. This is where the error was raised.

Implementation

Now let's get to the actual implementation. First, we will define three functions, fizz, buzz and fizzbuzz:

def fizzbuzz(val):
    if val % 3 == 0 and val % 5 == 0: 
        raise ValueError()
    fizz(val)

def fizz(val):
    if val % 3 == 0: 
        raise ValueError()
    buzz(val)

def buzz(val):
    if val % 5 == 0: 
        raise ValueError()

fizzbuzz is the only function we will call from another part of the code. Therefore, we can think of that as our entry point to this call stack. In fizzbuzz we do our first check, to see if the value is both divisible by 5 and 3 in which case we should print fizzbuzz. We will get to how we print the values in just a little bit. However, if the value didn't fulfill the condition we call fizz. Here again, the fizz-condition is checked, and if the condition fails it calls buzz which tests the buzz-condition.

Now let's see how we write this to the console:

def exception_driven_fizzbuzz(val):
    try:
        fizzbuzz(val)
        print(val)
    except ValueError as e:
        tb = e.__traceback__
        while tb.tb_next:
            tb = tb.tb_next
        print(tb.tb_frame.f_code.co_name)

Given some value, we call fizzbuzz. This triggers the call stack we just looked at. The idea is that the function which raises the exception should have its name written to the console. This is what we do in the except-block of this function. If none of the functions in the call stack raised an error the print-function after the fizzbuzz call will run, writing just the number to the console. As no error then has been raised the except-block isn't evaluated. However, if one the functions do raise an error we will iterate until we reach the last traceback. This traceback corresponds to the function which raised the error. Therefore, we extract the stack frame from this traceback and access it's code-object (f_code) which contains a name, f_name. f_name is the name of the function or module of the code object. In this case this will be either: fizz; buzz or fizzbuzz, depending on which function raised the exception.

Let's take an example to see how this is evaluated. We call the function with val=3. This doesn't trigger an error in fizzbuzz, but it does trigger one in fizz. The program goes into the except-block, without evaluating the print in the try-block. The first value of tb is the traceback in this function exception_driven_fizzbuzz. We take one lap in the loop, tb now correspond to fizzbuzz. Then we take a final lap in the loop and tb is now fizz. From tb we then extract the name fizz and write it to the console. If, we do the same for val=2, this will trigger no error in our call stack. Therefore, just the try-block will be executed, but this time it reaches the print-function.

Now we just need to slap a loop around this and our fizzbuzz is done:

for val in range(1, 16):
    exception_driven_fizzbuzz(val)

output:

1
2
fizz
4
buzz
fizz
7
8
fizz
buzz
11
fizz
13
14
fizzbuzz

Words of warning

I think this is a fun way to challenge yourself and learn new things about the language. However, I wouldn't do this on my next job interview if I were you.