DEV Community

Kenji Koshikawa
Kenji Koshikawa

Posted on

Manage private Python packages using Artifact Registry (Google Cloud)

#googlecloud #artifactregistry #python

Lately, I build a system over microservices on Google Cloud. I often use Python with App Engine, Cloud Functions and Cloud Run.

There are some private libraries in common across each microservice.
Referring to here official documents(App Engine, Cloud Functions), we had deployed each microservice using the pip command pip install -t to copy dependencies into a local folder.

After a couple of week since introducing the above pip command operation when deploying, I realized that it was hard to control a system quality, because the above pip command was difficult to specify a dependency version each library in our team environments. So we could not feel free to update some libraries which had been used by many microservices.

In this article, I introduce how manage private Python packages using Artifact Registry and solve those quality control issues.

All source code which I wrote about in this article has been uploaded below for your reference.
https://github.com/koshilife/artifact-registry-python-repo-example

Set up tools and a virtual environment

The following are versions of gcloud, Python and direnv when I tried.

$ gcloud app update
$ gcloud --version
Google Cloud SDK 374.0.0
alpha 2022.02.22
beta 2022.02.22
core 2022.02.22
gsutil 5.6
kubectl 1.21.9

$ python3 --version
Python 3.9.10

$ direnv --version
2.28.0
Enter fullscreen mode Exit fullscreen mode

I made a directory for an example app and set up a Python virtual environment. You need twine which is for publishing Python and keyrings.google-artifactregistry-auth which handles authentication with Artifact Registry repositories. The official guide is here for your reference.

$ mkdir hello-artifact-registry
$ cd hello-artifact-registry

$ python3 -m venv venv
$ source venv/bin/activate
$ python3 -m pip install --upgrade pip
$ pip install --upgrade pip setuptools wheel
$ pip install twine keyrings.google-artifactregistry-auth
Enter fullscreen mode Exit fullscreen mode

The following is a direnv file.

export GCP_PROJECT="<YOUR-GCP-PROJECT>"

# a credential file path of service account which is granted read/write for Artifact Registry
export GOOGLE_APPLICATION_CREDENTIALS="<YOUR-PATH>/google_cloud_credntials.json"
Enter fullscreen mode Exit fullscreen mode

Create a repository on Artifact Registry

I made a repository artifact-example for Python library at Tokyo region using the below gcloud command. The related official guide is here

$ gcloud artifacts repositories create artifact-example \
    --repository-format=python \
    --location=asia-northeast1 \
    --description="Python private libraries"
Enter fullscreen mode Exit fullscreen mode

Upload some packages to the repository

For testing I prepared very simple python libraries which just return foo or bar string.

A directory structure of these libraries is below.

python_libs
├── artifact-example-foo
│   ├── artifact_example_foo
│   │   └── __init__.py
│   ├── requirements.txt
│   └── setup.py
└── artifact-example-bar
    ├── artifact_example_bar
    │   └── __init__.py
    ├── requirements.txt
    └── setup.py
Enter fullscreen mode Exit fullscreen mode

artifact-example-foo (v1)
init.py

__version__ = '0.0.1'

def foo():
    return 'foo'
Enter fullscreen mode Exit fullscreen mode

artifact-example-bar (v1)
init.py

__version__ = '0.0.1'

def bar():
    return 'bar'
Enter fullscreen mode Exit fullscreen mode

I packaged as wheel format and uploaded it to the repository as the below commands.

$ cd python_libs/artifact-example-foo
$ python setup.py bdist_wheel
$ twine upload --repository-url https://asia-northeast1-python.pkg.dev/${GCP_PROJECT}/artifact-example/ dist/*

$ cd python_libs/artifact-example-bar
$ python setup.py bdist_wheel
$ twine upload --repository-url https://asia-northeast1-python.pkg.dev/${GCP_PROJECT}/artifact-example/ dist/*
Enter fullscreen mode Exit fullscreen mode

I made additional versions each library for testing as v2. On the same way I packaged and uploaded to the repository.

artifact-example-foo (v2)

__version__ = '0.0.2'

def foo():
    return 'foo'

def foo2():
    return 'foo2'
Enter fullscreen mode Exit fullscreen mode

artifact-example-bar (v2)

__version__ = '0.0.2'

def bar():
    return 'bar'

def bar2():
    return 'bar2'
Enter fullscreen mode Exit fullscreen mode

Use these uploaded libraries in client Python apps

I created simple Python apps (Flask) in App Engine, Cloud Functions and Cloud Run in the same Google Cloud project as the repository of Artifact Registry.

Notice: If you create in another Google Cloud project, you might need to grant a service account of Code Build. (The related official guide is here)

App Engine

client_apps/appengine/main.py

from flask import Flask, jsonify
import artifact_example_foo as foo
import artifact_example_bar as bar

app = Flask(__name__)

@app.route('/')
def hello():
    return jsonify(dict(result='ok', foo=foo.foo(), foo_version=foo.__version__, bar=bar.bar2(), bar_version=bar.__version__))

if __name__ == '__main__':
    app.run(host='127.0.0.1', port=8080, debug=True)
Enter fullscreen mode Exit fullscreen mode

client_apps/appengine/requirements.txt

--extra-index-url https://asia-northeast1-python.pkg.dev/<YOUR-GCP-PROJECT>/artifact-example/simple
artifact-example-foo==0.0.1
artifact-example-bar
Flask==2.0.2
Enter fullscreen mode Exit fullscreen mode

In the requirements.txt file, a point is that the Artifact Registry URL is added into the first line and the relevant packages to include private dependencies artifact-example-foo and artifact-example-bar.

Also, the /simple string at the end of the repository URL is needed because of Python Simple Repository API.

In the dependencies body, for testing I specified artifact-example-foo to 0.0.1 (v1) and do not specify any versions in artifact-example-bar

https://cloud.google.com/appengine/docs/standard/python3/specifying-dependencies#private_dependencies_with_artifact_registry

Cloud Functions

client_apps/functions/main.py

from flask import jsonify

import artifact_example_foo as foo
import artifact_example_bar as bar

def main(request):
    return jsonify(dict(result='ok', foo=foo.foo(), foo_version=foo.__version__, bar=bar.bar2(), bar_version=bar.__version__))
Enter fullscreen mode Exit fullscreen mode

client_apps/functions/requirements.txt

--extra-index-url https://asia-northeast1-python.pkg.dev/<YOUR-GCP-PROJECT>/artifact-example/simple
artifact-example-foo==0.0.1
artifact-example-bar
Enter fullscreen mode Exit fullscreen mode

The method of declaring the Artifact Registry repository is the same as the above App Engine one.

https://cloud.google.com/functions/docs/writing/specifying-dependencies-python#private_dependencies_from_artifact_registry

Cloud Run

client_apps/cloudrun/main.py

import os

from flask import Flask, jsonify
import artifact_example_foo as foo
import artifact_example_bar as bar

app = Flask(__name__)

@app.route("/")
def main():
    return jsonify(dict(result='ok', foo=foo.foo(), foo_version=foo.__version__, bar=bar.bar2(), bar_version=bar.__version__))

if __name__ == "__main__":
    app.run(debug=True, host="0.0.0.0", port=int(os.environ.get("PORT", 8080)))
Enter fullscreen mode Exit fullscreen mode

client_apps/cloudrun/requirements.txt

--extra-index-url https://asia-northeast1-python.pkg.dev/<YOUR-GCP-PROJECT>/artifact-example/simple
artifact-example-foo==0.0.1
artifact-example-bar
Flask==2.0.2
gunicorn==20.1.0
Enter fullscreen mode Exit fullscreen mode

The method of declaring the Artifact Registry repository is the same as the above too.

client_apps/cloudrun/Dockerfile

# refs: https://cloud.google.com/run/docs/quickstarts/build-and-deploy/deploy-python-service

FROM python:3.10-slim

ENV PYTHONUNBUFFERED True
ENV GOOGLE_APPLICATION_CREDENTIALS "key.json"

ENV APP_HOME /app
WORKDIR $APP_HOME
COPY . ./

RUN pip install --upgrade pip
RUN pip install keyrings.google-artifactregistry-auth
RUN pip install --no-cache-dir -r requirements.txt

CMD exec gunicorn --bind :$PORT --workers 1 --threads 8 --timeout 0 main:app
Enter fullscreen mode Exit fullscreen mode

I needed to enables to access the Artifact Registry repository while installing dependencies in Docker, so the following command allowed to copy the service account credentials.

ENV GOOGLE_APPLICATION_CREDENTIALS "key.json"
Enter fullscreen mode Exit fullscreen mode

The deploy script is for your reference.

The Response of these client apps

The following JSON is the response of theses client apps.
You can see that the library foo's version is used 0.0.1 (v1) and the library bar's version is used 0.0.2 (v2) as expected.

{
    "bar": "bar2",
    "bar_version": "0.0.2",
    "foo": "foo",
    "foo_version": "0.0.1",
    "result": "ok"
}
Enter fullscreen mode Exit fullscreen mode

Conclusion

By using Artifact Registry I could specify a version each library, so I would improve to manage python packages and to control a system quality finely than current the pip command pip install -t.

Other references

Top comments (0)

Read next

sunilkumrdash profile image

We made an AI SWE that solved 48.60% of issues on the SWE bench, 100% open-source.

Sunil Kumar Dash -

grantdotdev profile image

Advent of Code '24 - Day 13 Claw Contraption

Grant Riordan -

foyzulkarim profile image

A Beginner’s Journey Through the Machine Learning Pipeline (1)

Foyzul Karim -

shadow_b profile image

Automating Data Analysis with Python: A Hands-On Guide to My Project

shadowb -