DEV Community: Giovanni Barillari

How I made a binary version of Poetry package manager

Giovanni Barillari — Sat, 08 Jan 2022 18:45:19 +0000

Almost a year ago, I started producing binary executables of Poetry, a package manager for Python. Here is a technical deep-dive on how why and how I made it.

Poetry package manager

Started by Sébastien Eustace, Poetry quickly began – for good reasons – a popular package and dependency management tool in the Python scene, thanks to the PEP 518 and the standardised pyproject.toml manifest.

python-poetry / poetry

Python packaging and dependency management made easy

Poetry really helps out thousands of developers out there into managing their Python projects, from their dependencies to the publication.

And, as many projects out there within the same context, Poetry is written in Python. Which really makes sense: it has to deal with Python code and dependencies, and working with Poetry clearly means you already have a Python environment fully working.

But – there's always a but – this also means Poetry itself is hardly tightened to the Python environment you have on your machine, a VM or the docker container executing your code. And while this not necessarily an issue by itself, it still can produce issues in some cases.

The issue

And here it also comes the rationale behind the idea of producing binary versions of Poetry.

As I said a few lines back, Poetry depends itself on the Python environment you setup on your machine. Of course the team behind Poetry made everything correct under a Python packaging perspective: when you install Poetry through the installation script a virtual environment is created to isolate poetry from the rest of your projects and packages. But here is the thing: a Python virtual environment is still linked to the python executable and library in which the environment was built. And why is this an issue? Because on some platforms, upgrading the python distribution will just break the virtual environments produced from that distribution.

For instance, on MacOS, the typical way of handling Python installations is by Homebrew. But what happens when brew upgrade your Python version to your poetry installation? It will probably break. And you will need to reinstall Poetry.

Also, any further change in how you manage your Python installation(s) might affect your Poetry instance.

So how do we solve this? We need a way to fully-isolate our Poetry environment from the system one.

The PyOxidizer project

Now, since Poetry is a CLI, and thus all the interactions you can have with it won't require access to its code, we just need a way to produce some executable which contains Poetry code, all of its dependencies, and a Python interpreter.

Here is where the PyOxidizer project by Gregory Szorc comes super-handy. To quote the project description:

PyOxidizer is a utility for producing binaries that embed Python [...] PyOxidizer is capable of producing a single file executable - with a copy of Python and all its dependencies statically linked and all resources (like .pyc files) embedded in the executable. You can copy a single executable file to another machine and run a Python application contained within. It just works.

indygreg / PyOxidizer

A modern Python application packaging and distribution tool

But how does it work? Once you installed PyOxidizer on your system, everything starts with a Starlark file:

def make_exe():
    dist = default_python_distribution(python_version="3.9")

    policy = dist.make_python_packaging_policy()
    policy.resources_location_fallback = "filesystem-relative:lib"

    config = dist.make_python_interpreter_config()
    config.module_search_paths = ["$ORIGIN/lib"]
    config.run_module = "poetry.console.application"

    exe = dist.to_python_executable(
        name="poetry",
        packaging_policy=policy,
        config=config,
    )
    exe.add_python_resources(exe.pip_install(["./poetry"]))

def make_install(exe):
    files = FileManifest()
    files.add_python_resource(".", exe)
    return files

register_target("exe", make_exe)
register_target("install", make_install, depends=["exe"], default=True)

resolve_targets()

With this content in pyoxidizer.bzl and the poetry source code folder, you can invoke pyoxidizer build --release and a binary file will be produced under the build folder.

Is that so? Well.. actually no.

Making code "oxidizable"

There are few caveats in using the PyOxidizer project, mainly because the importer is not the standard one provided with Python, but a custom one provided by the PyOxidizer project. This makes perfectly sense: normally importing something in your python code means looking for a .py file in the filesystem, but since we're producing a binary executable, the contents of your Python files won't actually reside in the filesystem.

Now, the main issue here is that there's a long history in the Python community about the usage of the __file__ variable to access resources or to dynamically import packages: this variable won't hold any value in PyOxidizer, as a path for the matching Python source file won't exists.

Considering this – and the compliance of the oxidized importer you generally have a couple of options in PyOxidizer:

patch the code to convert all the __file__ instances into something else, like the importlib.resources library
force PyOxidizer to store those contents as files and fall-backs to the original Python importer

The difference between the two approaches is quite obvious: while the latter requires less effort, it also means your executable will still need some other files around it when you distribute it. And which also means those files will be editable by anyone.

And this is also why I chose to patch Poetry code and its dependencies.

Patching everything

If you look at the patches in my repository, we can make some observations:

there's an importlib_metadata patch, even if the embedded Python version is 3.9 and includes importlib.metadata module. This is beacuse the importlib.metadata implementation in the PyOxidizer project does not implement the full Distribution interface, and some dependencies of Poetry make calls to that interface
there's a patch for the requests package. This is because the requests package loads the SSL certificates from certifi using paths instead of a resource loader
the virtualenv patch is HUGE. This was needed because Poetry interacts quite a lot with the virtualenv package, and this is designed to work a lot with the file-system
the poetry-core patch is mainly required to comply with external dependencies which expect some resource files to be present in the file-system

While if we look at the poetry patch file, we can see the majority of code is included to change some behaviour of Poetry in treating Python environments. Specifically speaking, the environment handling in Poetry expects to have a standard "system" Python distribution, which is actually the one used to install Poetry in the standard way.

Now, since Poetry uses this environment as one of the possible implementations to build projects virtualenvs, I had to change the logic to always look to "external" Python distributions, as the "system" one will be the embedded distribution in the final executable, which – for obvious reasons – cannot be used to produce new environments.

Put everything together

In the end, the steps involved into producing the final binary are:

forking the code in need of patches
apply the patches
tell PyOxidizer to install the patched code instead of the standard packages available on pypi
adding some magic to treat special resources that need to be on the file-system no-matter-what

Then is just a matter of writing the Github workflow to produce the binaries for all the different platforms.

gi0baro / poetry-bin

Poetry binary builds

And adding some bonus points like:

an Homebrew formula
some Docker images with poetry included
a Github action to setup the binary version of Poetry

Final considerations

Looking back at everything involved in the process, I just want to leave some final thoughts.

First of all, the Poetry project was definitely a game-changer in the Python dependency management scene, so kudos to Sébastien and all the team for their efforts!

Secondly, the PyOxidizer project is – at least in my opinion – THE way of producing binaries from Python code. Comparing this project to all the available solutions out there leaves me with no doubts: the design, the principles, the correctness and the documentation of PyOxidizer are simply on a different level compared to the competition. Kudos to Gregory!

Then, let's move to the real question: should you compile your Python code into binaries? And – even more importantly – is it worth the hassle?

Well, it depends, of course.

If you already have a Python CLI project, you can stop requiring a Python environment and distribute it with PyOxidizer. This also gives you the advantage of avoiding compatibility code, as the only Python version you need to support is the one you embed with your project.

In case you're starting a new project and you're comfortable with Python, rather than a compiled language like Rust, and you can write code which will respect PyOxidizer requirements from line zero, sure, why not? This is what I did with noir.

But in cases where you have dozens of dependencies and/or code you can't control, like in my case, I just can't recommend it. Too many things out of your control can produce unwanted bugs, and today is still quite hard to test how your oxidized-only code will work. And in case you still want to make it, I suggest you to open up pull-requests to the involved open source projects to eliminate at least the __file__ usage, so the next person trying to do this won't need to re-walk the entire patch process.

And remember: making binaries also means you target specific architectures and platforms, so the Python code that worked on all the systems with an interpreter mightn't be compiled on all the same architectures and platforms – which is the point of interpreted languages vs compiled ones, isn't it?

In the end, I can only suggest you to precisely define your use-case and your audience. Answering the question "should I compile this?" will be easy then.

Cheers,
/G

Coding on an iPad: is it worth?

Giovanni Barillari — Wed, 25 Dec 2019 17:40:06 +0000

At the end of the year I faced the new experience of living without a laptop, occurrence which, as a software developer, never happened before.

While I spend my time at home that’s definitely not an issue: I have my desktop with dual monitors and a pleasant chair that suit my needs almost perfectly. But especially during the holidays, as I am travelling home to see my parents and old friends, that’s not an option and I decided to give a try into coding on my iPad Pro 11”, bought somewhen around one year ago.

Before proceeding further, I want to give you a bit of context about my coding needs. At the moment I mostly work with full-stack web projects, focused mainly on Python language for the backend and VueJS for the frontend. Considering this context, I usually need:

several versions of the Python language
virtual environments
node and npm
databases
docker and related utilities
git

Consequentially, even if today on the Apple Store there are plenty of nice applications for coding in a vast variety of languages and manage git repositories, everything else is – time of writing – completely missing, probably due to OS limitations.

Using something like Pythonista would let me write Python code and even run it, but it would be impossible to use everything else one of my projects needs, like a PostgreSQL instance or Redis.

This let me with only one real viable option: a remote IDE, deployed somewhere, option that would let me use everything I need relying just on the internet connection. Considering the few free options out there, code server – a VSCode server implementation – caught my attention. The idea is quite simple in reality: you deploy a virtual machine with the Linux distro you prefer, install the package, and use it on the machine you prefer through the browser.

Since I have quite a long history of DevOps and Systems management, it took me just a couple of hours getting everything run safely and configured for my needs, even with themes and extensions for VSCode; it also requires just a few bucks per month for the virtual machine.

So, after a few weeks, how was it?

Let me start answering with the code server project.
The experience is almost the same you have with a real instance of VSCode, with just a few exceptions and mainly regarding the extensions: in order to install an extension you need to run a specific command from the shell, and some of them seem just to be not compatible with this specific environment. But everything else, from the shortcuts, to the arrangement, to the terminal, are there, exactly the same.

Regarding the iPad experience the evaluation is quite complex in comparison.

First of all, I felt the VSCode interface is not really designed for touch inputs. The menus and small icon buttons are quite hard to click with the finger: you have to be quite precise to avoid erroneous taps on near elements. Even scrolling the code with the finger is a bit awkward, since the responsiveness of the input it doesn’t feel super natural. But the Magic Mouse was not an option here, since the iPad doesn’t support (yet?) its gestures, and consequentially you cannot scroll using the mouse :-/

Then it comes the keyboard. Now, I’m not sure I completely understood what is wrong with keyboard bindings in iPadOS or Safari, but keyboard shortcuts, that generally work out of the box at the beginning, start to get weird as soon as you start switching between different panels, or even different apps. For instance, if I resume the editor ‘cause I was in the browser checking something, and I opened one time at least the terminal, hitting cmd + F in order to search something in the code gets you to the terminal search. And it seems not to be possible to instruct the editor to get back over the code, as the only working thing seems to close the editor completely and reopen it. Also, the actual implementation in iPad OS add a small bar in the very low section of the screen while you’re typing, but since the editor is in full-screen mode, it gets a bit weird since it covers the last lines of the code, even if you’re writing on those lines.

Now, the two issues I just described get you mad few times during the day, but the most annoying thing in coding over an iPad for web development is the absence of a valid browser inspector and a console. Now, I even tried a couple of paid apps from the Store, but in general they work pretty bad and it’s almost impossible to use those tools in the same way you do on a Mac or a PC. And, trust me on this, trying to design Vue components without an inspector or a Javascript console is the perfect way to get completely mad and crazy.

Except for these points, I have to admit that I managed to run everything I usually have on a proper computer: starting containers using docker-compose, testing, merging changes with git, use the shell and all the cli tools I need; everything was possible with this solution.

Also, coding over an iPad with 11” screen is the most portable setup you might ever have: even coding during a train rail or a flight is super comfortable compared to a super light MacBook Air. And, bonus point: it doesn’t heat you while you’re compiling stuff, which is, by definition, super cool :)

Putting together all of this, I came up with these ratings for this setup:

Code server 4/5
Mouse usability 1/5
Keyboard usability 3/5
OS functionalities 2/5
Portability 5/5

And to answer the question from the title: is it worth?

I think that, when it comes to web development, the main issue preventing you from having a proper environment is the lack of daily tools like a decent inspector or a fully functional javascript console in the browser. I mean, it might be a solution for travelling, but I think is quite hard today to use this setup as a computer replacement.

On the other side, if your development is mostly backend oriented or involves solutions not requiring constant checks over UI components, then yes, for sure it can be a productive every day solution for you.

And, if you don’t need any database or utilities, you can even use one of the good editors now available on the Store and skip also the configuration of a remote editor.

In order to make it completely functional for my needs, it would need a bit of fixes in coder server regarding input handlings over an iPad, and a solution for the inspector.

But hey, 2020 is almost here, and I hope new options will came in the next months :)