DEV Community: Chris

Portable Python Bundles on Windows

Chris — Sun, 26 Jan 2025 12:06:44 +0000

Packaging Python applications and environments on MS Windows for other users so that they're "ready to run" on any machine can be a tricky task. This blog post describes my personal solution to the problem: Something that I like to call Python Bundles for Windows, a construct similar to virtual environments, but portable between machines.

Python Bundles stand somewhat at the intersection of the values and tradeoffs provided by virtual envs, regular Python installations and standalone executables created by tools like pyinstaller or py2exe.

No new tool is required to create such a bundle. It's just a loose and lightweight convention for a folder structure and some wrapper scripts that you can easily create manually. Or automate their creation in scripts or CI jobs.

1. The problem

Let's assume we want to package and ship a Python application or environment to our users in a self-contained and "ready to run" way.

We probably don't know what version of Python our users have installed, if any at all. And we definitely don't want to tamper with their potentially existing Python installations, which includes not having them to ask to install an additional version of Python. In other words: Our package should be all our users need to run our application or work with our Python environment.

1.1 The problem with venvs

After some brainstorming we might come up with the idea to simply create a virtual env (python -m venv venv_dir), install everything into the venv and then zip and distribute the venv folder to our users. But then we realize that venvs are based on hardcoded paths and a venv folder can't get easily relocated to a different path than the one it was created in. And that our venv in addition depends on the base Python installation that was used to create it (with exactly that Python version under exactly that path). So we'd need to tell our users exactly where to place their copy of our venv. And that they have to install a specific version of Python under a specific path. Not what we want.

1.2 The problem with packaging up regular Python installations

Instead of a venv we could just install all of our requirements into a regular Python installation and zip and distribute its folder (i.e. c:\Program Files\Python 3.13.1). That would work mostly. The Python installation dir on Windows can get generally relocated to a different path (not so on Unix due to the static prefix path - but that's a different topic).

However, there's one big flaw: There are script executables (.exe files in the .\Scripts\ dir, usually created by pip when a package isn't just a library but also provides scripts as entrypoints. One such executable is pip.exe itself). And these script executables depend on the path to the Python installation that is hardcoded "in themselves". Trying to relocate the Python installation dir results in these .exe files to no longer be working.

1.3 The problem with tools like `PyInstaller` and `py2exe`

Tools like PyInstaller or py2exe can bundle a Python application and all its dependencies into a single package. The user can run the packaged app without installing a Python interpreter or any modules.

This perfectly solves our distribution needs. However, the tradeoff is that we're not distributing a full Python environment, but a custom, minified bundle format. This might be the right tool for packaging an application. But it isn't suitable if we, for example, want to send our users a "starterkit" Python environment that can be used in an IDE and get extended with additional pip installs, etc. We're looking for a more generic solution.

2. My recipe for a solution: Meet Python Bundles!

Let's start with creating a bundle. It's quick and easy.

(You can find the completed example bundle from this article here on GitHub).

2.1 Creating a bundle from scratch

We'll start off in Powershell, creating a folder for our bundle:
(Feel free to name it however you like, here we'll just go with bundle as the name).

mkdir bundle
cd bundle

Then, we'll add a Python installation in <bundle>\python3. In this example, I'll download and add Python 3.13.1 from its nuget package , which is officially maintained by the CPython project and can be used as a "portable" copy of Python. (In the nuget zip file, the Python installation is in the tools dir. That's all we need here).

curl.exe -L "https://www.nuget.org/api/v2/package/python/3.13.1" -o python3.zip
Expand-Archive .\python3.zip -DestinationPath extracted_nuget
move .\extracted_nuget\tools python3
rm -R extracted_nuget
rm .\python3.zip

Now our bundle looks like this:

bundle
└───python3
    ├───python3.exe
    ├───Lib/
    ├───...

Let's also add a Scripts dir:

mkdir python3\Scripts

Currently we don't have pip enabled yet, so let's do that now.

python3\python.exe -m ensurepip

To work around the problem that pip creates .exe files in the Scripts dir that depend on a hardcoded Python installation path, we are going to use a wrapper script for pip, that monkey-patches the behaviour of pip.

Let's create a file <bundle_root>\python_wrapper\scripts\pip.py:

#!/usr/bin/python
import sys
import os

if __name__ == "__main__":
    from pip._vendor.distlib.scripts import ScriptMaker
    ScriptMaker.executable = r"python.exe"

    from pip._internal.cli.main import main
    sys.exit(main())

How does it work? Whenever, we install a package via our pip.py wrapper script (i.e. by running python3\python.exe pip_wrapper\scripts\pip.py <some-package>), any new .exe file generated in python3\Scripts\ will now just point to and use whatever python.exe is found via the PATH environment variable during runtime (instead of using a hardcoded, absolute path to an executable like c:\program files\Python 3.13.1\python.exe).

This is, of course risky and has implications. It is now our job to ensure that the python.exe in the PATH variable is the correct one when someone executes such a "patched" Scripts\*.exe file. That's why our bundle needs to get activated by the user, similar to a virtual env. We'll get to that in a moment.

(For more infos on the idea of such a pip wrapper, see here)

Now, wouldn't it be nice to also have a pip.exe for our pip wrapper, so that we later can just use the pip command (instead of having to do python pip.py)? Let's create one. It needs to be portable too, of course, that's why we'll create it in a similar way.

To do so, let's create a <bundle>\pip_wrapper\bin folder and create the .exe file there.

mkdir pip_wrapper\bin

Then let's start a Python shell (REPL) with python3\python.exe and execute the following code to create the pip.exe.

from pip._vendor.distlib.scripts import ScriptMaker
maker = ScriptMaker("pip_wrapper/scripts", "pip_wrapper/bin")
maker.executable = r"python.exe"
maker.make("pip.py")

Our folder structure should now look like this:

bundle
└───python3
    ├───python3.exe
    ├───Scripts/
    ├───Lib/
    ├───...
└───pip_wrapper
    ├───scripts/
        ├───pip.py
    ├───bin/
        ├───pip.exe

Now, we'll need a way to activate our Python bundle in a current cmd.exe or powershell shell session.

For this we're going to create venv-style activate scripts for both shell types in the root of our bundle:

<bundle>\activate.cmd :

@echo off
set PATH=%~dp0pip_wrapper\bin\;%~dp0python3\Scripts\;%~dp0python3\;%PATH%

<bundle>\activate.ps1 :

$ScriptDir = (Split-Path -Parent $MyInvocation.MyCommand.Definition)
$Env:PATH = "$ScriptDir\pip_wrapper\bin;$ScriptDir\python3\Scripts;$ScriptDir\python3;$Env:PATH"

Now our basic bundle folder structure is complete.

3. Usage

So how can we use our newly created Bundle?

Let's open either a cmd.exe shell or a powershell session and activate the bundle.

3.1 Activate in a `cmd.exe` shell:

To activate the bundle:

call <bundle>\activate.cmd

For example:

call c:\workspace\activate.cmd

We can now verify that our active python, pythonw and pip executables are those from the bundle:

$ where python
c:\workspace\bundle\python3\python.exe
$ where pip
c:\workspace\bundle\pip_wrapper\bin\pip.exe

3.2 Activate in a Powershell instance:

Open a Powershell instance and activate the bundle using the dot source method:

. <bundle>\activate.ps1

For example:

. c:\workspace\bundle\activate.ps1

(If you receive a Powershell error message that script execution is disallowed for your user, you may need to allow it once with Set-ExecutionPolicy RemoteSigned -Scope CurrentUser).

Afterwards you can verify that the currently active python.exe, pip.exe, etc are indeed the ones from our bundle:

$ Get-Command python | Format-List -Property Path
C:\workspace\bundle2\python3\python.exe

$ Get-Command pip | Format-List -Property Path
C:\workspace\bundle2\pip_wrapper\bin\pip.exe

3.3 How to use a bundle after its activation:

We can now use the bundle's Python environment from the shell that we have activated it in, similar to an activated venv. (In the following I'm assuming we're in Powershell).

For example we could use pip to install Jupyter Notebook:

$ pip install notebook

This creates a <bundle>\python3\Scripts\jupyter.exe that is already in the PATH of our activated environment:

$ Get-Command jupyter | Format-List -Property Path
C:\workspace\bundle2\python3\Scripts\jupyter.exe

So, to start Jupyter Notebook, we could run this command directly from our Powershell instance:

jupyter notebook

4. You said our bundle is portable?

It is! You can rename or copy the <bundle> dir to any path you like - it will not break. Try it out and make a copy of it under a new name.

This means you can zip the <bundle> folder and send it to other users. All they need to do is to place it somewhere on their Windows filesystem, activate it in a shell and they're ready to go.

It's like a venv - but self-contained and portable / path-independent. Mission accomplished. 🎉

5. FAQ & Bonus section

We’ve reached the end of this blog post. If you’re still not tired to read on, here’s some bonus stuff.

5.1 Q: Can I still start `Scripts\.exe` files directly (i.e. double-click on them)?*

A: No, other than with a regular Python installation or venv .exe files from the Script dir can only get started from an activated shell. If you need a replacement for this limitation, I'd recommend to create wrapper scripts for them. Here's an example for a wrapper script for jupyter.exe, that passes through all arguments to the jupyter processes and passes back its exit code to the shell.

You can double click on it to launch it or use it in a cmd shell with or without arguments.

<bundle>\jupyter.cmd:

@echo off
setlocal
call "%~dp0activate.cmd"
"jupyter.exe" %*
exit /B %errorlevel%

5.2 Q: What are some good example use cases for bundles?

5.2.1 Example1:

I have created a Jupyter Notebook, that requires a specific set of libraries for its contents to work (like numpy, pandas, etc). I want to share this notebook with someone and make it low effort for them to open it on their Windows machine. Creating a bundle allows me to tell them: "Here's a zip file, extract it anywhere you like and in the extracted folder you'll find a notebook.cmd wrapper script. Just double-click it to start the Jupyter WebUI."

(I've put exactly this example as a Demo on GitHub here).

5.2.2 Example2:

I want to create a Windows installer for a Python application. The entire runtime and dependencies should be part of my application bundle to make things easy and safe for the user. The default installation path by the installer is c:\program files\myapp, but the installer also lets the user optionally select a custom path. Packaging my app as a bundle gives me portability and path independence. I can automate the creation of the bundle as a build artifact in the CI pipeline of my application project.

5.3 Q: What about Unix?

Linux, MacOS, etc are out of scope of this solution. Bundles only work on Windows. On Unix systems you usually need to build Python for a specific prefix path (i.e. /opt/myapp/python) and the resulting Python binaries aren't moveable to other paths. You also need to build against various dependencies, which differ between distros. This all makes it more difficult.

An easy way for application/environment bundles on Linux are Docker/OCI containers. The official python:3 and python:3-slim base images are a great starting point.

Alternatively you could create distro specific bundles that need to get placed under a specific path (i.e. /opt/myapp). You can use most patterns of our Windows bundles, but you'd need to build your own copy of Python from source for your custom prefix path with something like:

./configure --prefix=/opt/myapp/python3
make -j `nproc`
make install

5.4 Can I use bundles with IDEs and editors?

Many IDEs and editors have first class support for venvs , but are of course unaware of bundles. So some extra steps might be necessary.

For some IDEs, it's sufficient to just start the IDE from an activated shell. For some other IDEs you might need to modify the IDE's Python runner/debug configuration profile, so that the correct Python executable is being used and the PATH environment variable gets prepended with the same three paths that our activate scripts set:

<bundle>\pip_wrapper\bin
<bundle>\python3\scripts\
<bundle>\python3\

If you'd like to see a more detailed follow up blog post how to do this in vscode, Pycharm, etc please leave a comment.

5.5 Q: Are bundles the right tool for every job? Should I stop using venvs?

Certainly not. I for myself am using venvs as the default in my day to day work, for multiple reasons - the major one being to save disk space. Each bundle contains a full Python installation and is therefore much larger and slower to create than a venv.

Here's a quick, incomplete comparison of some of the aspects of some of the many options, just to visualize the idea.

5.5.1 comparison


	Bundle	Virtual Env	Python Installation	Pyinstaller
Path independent and self-contained?	yes	no (path to Python installation is hardcoded in venv)	no (`.\scripts\*.exe` files will break)	yes
Can have multiple instances on the same system	yes	yes	not without issues (concept is one Python installation per Python version per user or system)	yes
Disk Usage	large (contains a full Python installation)	small (depends on a Python installation)	large	medium
Needs to get activated	yes	yes	no	no
Single executable	no	no	no	yes
Can be used like a regular Python installation (REPL, pip, scripts, etc)	yes	yes	yes	no
Can be used with IDEs?	yes, but you might need to configure environment variables in the IDE’s run/debug profile	yes	yes	no

How to 100% CPU

Chris — Wed, 08 Jan 2025 20:41:58 +0000

I've been working with many sysadmins over the years and one question comes up from time to time: "I quickly need to create some dummy CPU load on this machine, what cpu stress tool should I install?"

If our need is very basic (i.e. we just want to see 100% CPU load on one or multiple cores), maybe we should consider building our own.

The One-Liner

All we need is to put this line of C code in a file, build it with gcc -o stressme stressme.c (or on Windows cl stressme.c) and run it with ./stressme (or stressme.exe).

int main() {while (1) {}}

And while the program runs, we'll see 100% CPU load on one core. For multiple cores, we could start the program multiple times.

Multi-Threaded

Or we could use threads, here's a variant that uses four POSIX threads:

#include <pthread.h>
#include <unistd.h>

#define NUM_THREADS 4

void *loop(void *arg) {
    while (1) {}
}

int main() {
    pthread_t threads[NUM_THREADS];
    for (int i = 0; i < NUM_THREADS; i++)
        pthread_create(&threads[i], 0, loop, 0);
    pause();
}

(To build it, add the -pthread flag: gcc -o multistress multistress.c -pthread)

Why does this work?

We're running an infinite loop. When we look at the simplest infinite loop in assembly, it becomes clear the CPU is busy doing something: Executing a jmp instruction that "jumps to itself", as fast as possible. So the CPU is busy running instructions. It's not doing nothing like with a pause/sleep.

global _start

_start:
    jmp _start

If we were on an older operating system with a cooperative multitasking scheduler, such an infinite loop would probably make our system unresponsive. On today's preemptive multitasking systems, infinite loops cause the program to consume all available processor time, but can still be terminated.

Migrating a large Dropbox account to Onedrive

Chris — Sat, 02 Sep 2023 12:12:36 +0000

I recently migrated my ~ 1TB Dropbox account (single user paid plan) to Microsoft OneDrive (consumer version). Here's my "lessons learned".

I prepaid for Microsoft Office 365 Home for several years with discounted gift cards bought during a Black Friday sale. It turns out you can prepay Office 365 Home for a maximum of 4 years. When you try to redeem more gift cards you'll receive a cryptic error message ("MaxStackingDurationExceeded") - and you'll need to wait at least 1 year before you can redeem the next gift card. (I believe the gift cards have no specific expiration date, but am not entirely sure).
The easiest way to migrate data from Dropbox to OneDrive (consumer) is to use Microsoft's https://mover.io service. Mover was a data cloud migration startup that Microsoft bought a couple of years ago and integrated their tech into its Enterprise offerings. For consumers who want to migrate their data from another cloud storage to OneDrive (consumer) Microsoft still operates the original mover.io service - and it's free. (More info in the Microsoft docs: https://support.microsoft.com/en-us/office/transfer-files-to-your-personal-onedrive-with-the-mover-transfer-wizard-7dbda93c-71e6-483f-8914-ad445554cd31)

you can login on mover.io with your Microsoft (consumer) login and connect your Dropbox account as source and your OneDrive account as destination, pick a source and target folder (i.e. the root folder on both sides) and start a data transfer job.
the data transfer runs on the mover.io servers (not via your laptop and your own Internet connection). In my case it took around 14 hours to transfer ~ 1 TB data. In the end you get access to a detailed log that can be filtered for skipped and error'ed files.
in my case I had a number of files that hadn't been copied due to these problems:
- file too large (OneDrive only allows files up to 10 GB. I had some ISOs, VM images, etc that were larger)
- empty files. (unlike Dropbox, OneDrive doesn't support 0 byte files. In my case I had hundreds of .gitkeep, __init__.py, etc files. Fixing these was a manual process (i.e. I put in comments or dummy content).
- path too long (OneDrive doesn't support long paths)
mover.io also seems to support other cloud storage providers as source like Google Drive, Box and other OneDrive accounts. (However, I haven't tried connecting them).

That's it. Happy migrating!