DEV Community: Chris

Publishing to a private Python repository with Poetry

Chris — Thu, 03 Feb 2022 11:02:01 +0000

In this article, we will look at Poetry, a tool for dependency management and packaging in Python, and how to publish a package built in Poetry to a private Python package manager hosted in Packagr

What is Poetry?

Poetry describes itself as “Python packaging and dependency management made easy”. It combines the functionality of the standard python packaging tools setuptools and twine, allowing you to build and publish your Python package in a single command, while also managing your python package dependencies.

By default, Poetry publishes built packages to the default public Python package repository, PyPI. However, it is also possible to point poetry at your own private Python package server instead.

Provisioning a private Package repo

There are several open-source solutions for provisioning your own, personal PyPI server — for example, this can be done using pypiserver or devpi. However, configuring these services takes time and effort, and it costs money to deploy them. Instead, we’ll use Packagr, a cloud-hosted python package server that allows you to provision your own private Python package repository. It also supports NPM packages and even Docker registries and is very easy to set up. You can get started by creating a free trial account on Packagr — when you’ve created your account, you’ll see this:

When your account is created, Packagr automatically generates a private repository URL for you. To find it click on Python packages in the left-hand sidebar, then look for your unique Repository URL towards the top of the page:

The URL will be in the format https://api.packagr.app/abcdef. Make a note of your repository URL before continuing — you will need it later.

Installing Poetry

Use this one-liner to install Poetry on your local machine:

curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -

You may also need to configure your shell with the following command to complete the installation:

source $HOME/.poetry/env

Check your installation worked with the following command:

poetry -v

Creating a package in Poetry

Now that poetry is installed you can create a poetry project like this:

poetry new private-package

This will create a subfolder called private-package, containing a package skeleton.

And that’s it! We have created an empty Python package. We’re going to leave it blank for the sake of this simple tutorial.

Pointing Poetry at your private repository

We can now configure Poetry to publish our private-package package to our Packagr private repository by using the below command, replacing the URL with our unique Packagr repo URL:

poetry config repositories.packagr https://api.packagr.app/abcdef

Next, we can use the below command to store our Packagr credentials locally, meaning that we don’t have to enter them every time we want to publish the package:

poetry config http-basic.packagr packagr-username@example.com password

Publishing the package to our private repository

We’re now ready to publish! To do so, simply use this command:

poetry publish -r packagr

If all goes well, your private Python package will be uploaded successfully. You can confirm this by refreshing your Packagr UI:

Installing a private package with Poetry

Now that we have published our Python package to a private repository, we can install it in other poetry projects. To do this, you’ll need to add this section to your pyproject.toml (inserting your private repository URL):

[[tool.poetry.source]]
name = "packagr"
url = "https://api.packagr.app/abcdef"
secondary = true

The secondary = true bit is not essential here - however, if you have a project which has a lot of dependencies in PyPI as well, then resolving packages will take much less time.

You can now do the following to install python packages locally:

poetry add private-package --source packagr

Installing private Python packages in Docker images

Chris — Mon, 29 Jun 2020 19:04:36 +0000

In this tutorial, we’ll look at how to install a Python package stored in a private repo in a Docker image

The problem

Let’s say you have a set of common Python utilities that you use across a number of different projects.

For example, let’s say you have some scripts that generate random user data for the purpose of testing various applications, that you share across multiple projects

Copying these scripts across multiple code bases is a nightmare from the perspective of maintenance, but these utilities may contain proprietary code that cannot be put somewhere public. So where can you put them?

The solution

One option is to package your utilities into a Python package, store the package in a private Python repository, then have the Docker image that contains your projects install these utilities from your repository. That way, you can easily control versioning of these utilities across all projects that have a dependency on them.

In this tutorial, we’ll look out how to create a Python package, store it securely in a private repository (in this case, Packagr.app) and write a docker image that authenticates your private Python repository, then pulls and installs your package.

For good measure, we’ll also store our built Docker image in Packagr.app

A practical example

For the purposes of this tutorial, we are going to build a simple tool that generates random user data, which we can use to populate APIs, perform tests, etc. Your utilities might do something totally different, but you can use the same approach to create a distributable package to do whatever you need, and store it privately.

Creating your project

Let’s start by creating our package. This tutorial assumes that you already have python3 and virtualenv installed. Let’s create a folder called data-generator and a virtual environment:

# create a directory and cd into it
mkdir data-generator && cd data-generator

# create a virtual environment, and activate it
python3 -m venv env
source env/bin/activate

Next, let’s create a subdirectory called package that will store our code

mkdir package

Let’s install Faker, a Python environment that generates random data

pip install Faker

And let’s create a file called process.py in the package folder, which generates a random user every time it is called:

# process.py
from faker import Faker
import json
fake = Faker()

def generate_person():
    print(
        json.dumps(dict(
            name=fake.name(),
            email=fake.email(),
            password=fake.password(),
            address=fake.address()
        ))
    )

The above function simply outputs a dummy person in json format, that looks like this:

{
  "name": "Samuel Mendez",
  "email": "michelle71@hotmail.com",
  "password": "*hMRsQBbK1",
  "address": "78287 Morgan Summit\nPhillipsstad, WV 38051"
}

Next, lets create a folder called bin in the root directory, and add a file called make-person. This is a simple script that will let you call your function from the shell

#!/usr/bin/env python
from package import process
process.generate_person()

Finally, lets create a file called setup.py in the root data-generator directory.

from setuptools import setup

setup(
    name='data-generator', # the name of the package
    version='1.0',
    packages=['package'], # contains our actual code
    author='chris',
    author_email='chris@packagr.app',
    description='a random person generator',
    scripts=['bin/make-person'], # the launcher script
    install_requires=['faker==2.0.2'] # our external dependencies
)

Our folder structure should now look something like this:

/data-generator
    /package
        process.py
    /bin
        make-person
    setup.py

We’re now ready to build our package! Enter the following lines into the command line:

# install wheel (to build packages in the bdist_wheel format)
pip install wheel
# create the package
python setup.py bdist_wheel

The process will run, and a few additional folders will be created. The one we’re most interested in is the dist folder, which contains our built package:

/dist
   data_generator-1.0-py3-none-any.whl

You can check that our package works as expected by installing and running it as follows:

# install the local package
pip install dist/*
# call the script
make-person
{"name": "Alexandra Nelson", "email": "sgarcia@yahoo.com", "password": "*c706Hvc+H", "address": "625 Powers Orchard\nNorth Bonnietown, IN 04475"}

Uploading the package to our private repository

Now that we’ve built the package, we can upload it to our private repository. If you don’t already have a private repository to upload to, you can create one for free at Packagr.app by following the simple sign up process. Once you’ve logged into your account, click on Create new package in the left hand menu to see your private repository URL — make a note of this, as you’ll need it later.

Now that you’ve got a private repository, and a package, we can upload files to it. We do this using a tool called twine

# install twine
pip install twine
# upload your built package to your repository (update the URL as necessary)
twine upload  --repository-url https://api.packagr.app/63cdQSDO/ dist/*

You’ll be prompted to enter credentials — just use the username and password you signed up for Packagr with. Once you’ve done that successfully, you’ll see your new package in your Packagr dashboard.

Installing your private package from inside a Docker container

Now that our package is securely stored in our repository, let’s create a docker image capable of installing it. It’s important to remember that you should never store raw credentials in your Dockerfile, or anywhere else in your code for that matter. So, you should pass your credentials as environmental variables at build time. With that in mind, let’s start by creating a file called requirements.txt in an empty folder, remembering to change the repository URL to your own :

--extra-index-url https://api.packagr.app/63cdQSDO/
data-generator==1.0.0
faker==2.0.2

A requirements file is just a list of packages for pip to install. In this case, we are adding our private package, data-generator and the public dependency, faker. Adding the --extra-index-url tells pip to look in our private repo, in addition to the public pypi.org one.

Next, lets create our Dockerfile

FROM python:3.7-alpine
ARG USER
ARG PASS
RUN echo "machine api.packagr.app \
         "    login ${USER} \
         "    password ${PASS}" > /root/.netrc
RUN chown root ~/.netrc
RUN chmod 0600 ~/.netrc
COPY requirements.txt /requirements.txt
RUN pip install -r requirements.txt
CMD make-person

Let’s take a detailed look at this, one line at a time:

FROM python:3.7-alpine tells Docker the base image to use — in this case we are using the lightweight alpine python distro, mostly to save time/space
ARG USER and ARG PASS defines variables that we will provide at build time — specifically, our Packagr username and password
RUN echo... creates a file called .netrc which tells the image to use the Packagr username and password when connecting to api.packagr.app. The 2 following lines set the permissions of this file correctly
COPY requirements... copies the requirements file to the Docker image
RUN pip... installs the dependencies in our requirements file.
CMD make-person just calls the script defined in our setup.py at run time

Now that we have our Dockerfile, let’s build the Docker image, substituting in your Packagr username and password:

docker build -t dg-image --build-arg USER=chris@packagr.app --build-arg PASS=changeme .

If all goes well, your docker image should build correctly! We can now upload it to our docker registry. Go back to the Packagr interface, click on Docker registry, and make a note of your Docker registry URL - it should be similar to your Python package repo URL, but with docker.packagr.app instead of the domain, and with the hash id in lowercase characters.

The first step is to login to docker — as usual, you’ll need to update the URL, username and password to your own

docker login docker.packagr.app/63cdqsdo -u me@example.com -p changeme

Next, you’ll need to tag the image you just build (again, change the URL)

docker tag dg-image docker.packagr.app/63cdqsdo/dg-image:latest

Finally, you can push your tags:

docker push docker.packagr.app/63cdqsdo/dg-image:latest

You should now see your image in your docker registry in Packagr. You can now pull this docker image from any other machine you are logged into with this command:

docker pull docker.packagr.app/63cdqsdo/dg-image:latest