Python: Compile standalone executable with nuitka

#python

So nuitka compile python code into an executable. I have always impress with Go ability to generate single binary and even cross compile for different OS. I wish we can have the same thing with python.

As always, let's start with just a simple script first:-

import requests

resp = requests.get("https://httpbin.org/get")
print(resp.content)

Actually the first I try is with just a simple print("hello world") but that just too simple. I want to see if nuitka can handle more significant code like the requests library above. To compile the code:-

python3 -mnuitka --follow-imports hello.py

It will generate a file called hello.bin in the same directory you run the command above. Execute the file (./hello.bin) and it works! But if you copy the file to different system (I compiled it on ubuntu 18.04 and try to run it on my laptop running Manjaro), I got this error:-

./hello.bin: error while loading shared libraries: libpython3.6m.so.1.0: cannot open shared object file: No such file or directory

So it still need the same python libraries that it get compiled with. And since my manjaro system using python 3.7, hence the error. Fortunately there's second option:-

python3 -mnuitka --follow-imports --standalone hello.py

This time it will generate a folder instead named hello.dist. Inside the folder you'll see various .so files and a file named hello. This time when I copied the folder hello.dist from Ubuntu 18.04 to my Manjaro's laptop, the command works!

Unfortunately the reverse didn't work though. When compiling on my Manjaro and try to run it on Ubuntu 18.04:-

./hello: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /home/kamal/hello.dist/libpython3.7m.so.1.0)
./hello: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /home/kamal/hello.dist/libpython3.7m.so.1.0)

This is because manjaro is using more recent glibc than ubuntu 18.04. We can check this with ldd:-

ldd --version
ldd (GNU libc) 2.30

And the only workaround for this is to compile is on the oldest system you want to support. I try this with ubuntu 14.04, it work for simple script but then failed with missing ssl certs when compiling my tgcli script.

So a standalone distribution of your python program kind of work. For my tgcli, the installation look like:-

tar xzf tgcli.dist-glibc-2.27.tar.gz
sudo mv tgcli.dist /usr/local/
sudo ln -s /usr/local/tgcli.dist/tgcli /usr/local/bin/tgcli

For more fun story on getting standalone python program, can also read this sharing from syclladb.

Top comments (6)

Kidus Adugna • Jan 10 '20

I tried this on Windows with nuitka --standalone --recurse-all --recurse-stdlib --plugin-enable=tk-inter script.py, hoping it would reduce the size of the bundle compared to Pyinstaller but the opposite happened (51.7MB vs 20.8MB). Is there anything I don't know?

Kamal Mustafa • Jan 10 '20

Have you checked this?

nuitka.net/doc/user-manual.html#wi...

Kidus Adugna • Jan 11 '20

That worked, removed a good chunk and reduced it to about 28MB, its more than Pyinstaller's 20MB but still. Thanks!

Kamal Mustafa • Jan 10 '20

I've just bumped into this old HN discussion and someone posted GvR comment on nuitka:-

This project looks completely misguided. The talk focused on the trivialities of mapping Python to C++ rather than on the interesting problems to be encountered when trying to optimize Python while maintaining its extremely dynamic semantics. Also the benchmarking effort is laughable; pystone is not to be taken seriously (only exercises a tiny part of the language) and pybench does microbenchmarks, which are optimized away. You should try the "real-world" benchmarks from the PyPy and Unladen Swallow projects. And what is the size of the generated code? (E.g. how big would the binary for the entire standard library be?) In your blog, please use less boring subjects than "version x.y.z released". ~ Guido van Rossum

news.ycombinator.com/item?id=8771925