Deploying Python projects to air-gapped systems

Motivation

A lot of times in the real world we have to deploy Python projects behind a company firewall, restricted systems or even an air-gapped system. This is most certainly a big hassle for most people.
I've found a pretty good setup which allows us to develop using modern tooling like poetry, but also deploy to any system without the need to pull any packages from the internet.

Project setup

The project is setup as follows

airgap/
├── airgap/
│   ├── __init__.py
│   └── main.py
├── poetry.lock
├── pyproject.toml
├── README.md

and we're pulling some dependencies into our project (I've used arrow which is an awesome date and time manipulation library):

[tool.poetry]
name = "airgap"
version = "0.1.0"
description = "A project running on an air-gapped system."
authors = ["..."]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.10"
arrow = "^1.3.0"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

We can have as many dependencies as we want, but for demonstration purposes I'll keep it short.

Preparation before distribution

We'll need to generate a requirements.txt file for our next steps. Let's use poetry to do that:

poetry export -f requirements.txt -o requirements.txt

Command breakdown:

• export the command used by poetry to convert the pyproject.toml file
• -f: the format to use, only contstraints.txt and requirements.txt supported
• -o: the name of the output file

After running you'll get a requirements.txt similar to this one (except the hashes which I've omitted for brevity):

arrow==1.3.0 ; python_version >= "3.10" and python_version < "4.0"
python-dateutil==2.8.2 ; python_version >= "3.10" and python_version < "4.0"
six==1.16.0 ; python_version >= "3.10" and python_version < "4.0"
types-python-dateutil==2.8.19.14 ; python_version >= "3.10" and python_version < "4.0"

Now comes the funky part. We'll create wheels from these dependencies and store them locally. We'll use pip to get the wheels:

pip wheel --no-deps --wheel-dir ./wheels -r requirements.txt

Command breakdown:

• wheel: the pip command used to generate wheels docs
• --no-deps: do not install dependencies (we've already got them all in the requirements.txt)
• --wheel-dir: the directory to store the wheels (doesn't have to exist, it will be created for you)
• -r: which requirements file to use

Our requirements.txt includes hashes, which means that pip wheel will imply the --require-hashes option. In turn this will verify the packages when installing.

Now our directory wheels will hold four packages (the ones from the requirements.txt). The final part is our own package source. We'll use poetry to build the wheel and place it in the wheels directory.

poetry build && mv dist/*.whl ./wheels

Command breakdown:

• build: the poetry command which builds our project
• mv: move files
• dist/*.whl: source all files ending in .whl in the dist/ directory
• ./wheels: the destination for the sourced files

At this point we're done with the preparation. Just zip the ./wheels directory and distribute it to your air-gapped server. We'll just create a zip of the files:

zip -r airgap-dist-0.1.0.zip ./wheels/*

Command breakdown:

• -r: compress the archive
• iargap-dist-0.1.0.zip: name of the compressed archive
• ./wheels/*: the sources to put in the archive

Deployment

Get your zip to your air-gapped system, unpack it and install. The best-practice I stick to is to use the /opt directory to deploy custom software. Let's put it there:

mkdir /opt/airgap
unzip airgap-dist-0.1.0.zip -d /opt/airgap

Note: you might have to run the mkdir command using sudo. It's very dependant on your setup, user, OS version etc.

Just as an example you might need to do this:

sudo mkdir /opt/airgap
sudo chown youruser:yourgroup /opt/airgap
sudo chmod 755 /opt/airgap

Now in general I wouldn't advocate to install any project in the global Python site, but since we're striving for simplicity we'll do that. The next step is very easy, install our wheels:

pip install --no-cache /opt/airgap/wheels/*

Bonus

Just in case you want to use venv you can do the following:

python -m venv /opt/airgap/venv

Command breakdown:

• -m venv: tell Python to use the module venv (should be installed by default)
• /opt/airgap/venv: the location where to create the virtual environment, note that the directory name venv is just a convention, but you can use any name you like.

Activate the newly created virtual environment:

source /opt/airgap/venv/bin/activate

And then do the install:

# verify you're using the venv pip
which pip
$ /opt/airgap/venv/bin/pip

# install your packages
pip install --no-cache /opt/airgap/wheels/*

Note: to exit the virtual environment you can run deactivate in your terminal. It's a command automatically available as soon as you activate a virtual environment

Conclusion

We've seen how to extract the dependencies from our new pyproject.toml file, build wheels and zip them up for distribution. Consequently how we can deploy our project to our air-gapped system. Happy coding!

Comments

[Allen Kennedy]

I feel like you got 90% of the way then fell on your face.
How do I run the program after all this?

[borisuu]

It depends what kind of application you're deploying.

A simple example would be to activate the venv and run your main in the background

source /opt/airgap/venv/bin/activate
# run once
python -m airgap.main

# run in the background
python -m airgap.main &

That's assuming you've got a if __name__ == '__main__' block in your main.py file.

You'll notice that I didn't discuss the internals of the application at all, as the variations are endless. If you have a more specific use-case, I'd be glad to take a look.