Lately, I build a system over microservices on Google Cloud. I often use Python with App Engine, Cloud Functions and Cloud Run.
There are some private libraries in common across each microservice.
Referring to here official documents(App Engine, Cloud Functions), we had deployed each microservice using the pip command pip install -t
to copy dependencies into a local folder.
After a couple of week since introducing the above pip command operation when deploying, I realized that it was hard to control a system quality, because the above pip command was difficult to specify a dependency version each library in our team environments. So we could not feel free to update some libraries which had been used by many microservices.
In this article, I introduce how manage private Python packages using Artifact Registry and solve those quality control issues.
All source code which I wrote about in this article has been uploaded below for your reference.
https://github.com/koshilife/artifact-registry-python-repo-example
Set up tools and a virtual environment
The following are versions of gcloud, Python and direnv when I tried.
$ gcloud app update
$ gcloud --version
Google Cloud SDK 374.0.0
alpha 2022.02.22
beta 2022.02.22
core 2022.02.22
gsutil 5.6
kubectl 1.21.9
$ python3 --version
Python 3.9.10
$ direnv --version
2.28.0
I made a directory for an example app and set up a Python virtual environment. You need twine
which is for publishing Python and keyrings.google-artifactregistry-auth
which handles authentication with Artifact Registry repositories. The official guide is here for your reference.
$ mkdir hello-artifact-registry
$ cd hello-artifact-registry
$ python3 -m venv venv
$ source venv/bin/activate
$ python3 -m pip install --upgrade pip
$ pip install --upgrade pip setuptools wheel
$ pip install twine keyrings.google-artifactregistry-auth
The following is a direnv file.
export GCP_PROJECT="<YOUR-GCP-PROJECT>"
# a credential file path of service account which is granted read/write for Artifact Registry
export GOOGLE_APPLICATION_CREDENTIALS="<YOUR-PATH>/google_cloud_credntials.json"
Create a repository on Artifact Registry
I made a repository artifact-example
for Python library at Tokyo region using the below gcloud command. The related official guide is here
$ gcloud artifacts repositories create artifact-example \
--repository-format=python \
--location=asia-northeast1 \
--description="Python private libraries"
Upload some packages to the repository
For testing I prepared very simple python libraries which just return foo
or bar
string.
A directory structure of these libraries is below.
python_libs
├── artifact-example-foo
│ ├── artifact_example_foo
│ │ └── __init__.py
│ ├── requirements.txt
│ └── setup.py
└── artifact-example-bar
├── artifact_example_bar
│ └── __init__.py
├── requirements.txt
└── setup.py
artifact-example-foo (v1)
init.py
__version__ = '0.0.1'
def foo():
return 'foo'
artifact-example-bar (v1)
init.py
__version__ = '0.0.1'
def bar():
return 'bar'
I packaged as wheel format and uploaded it to the repository as the below commands.
$ cd python_libs/artifact-example-foo
$ python setup.py bdist_wheel
$ twine upload --repository-url https://asia-northeast1-python.pkg.dev/${GCP_PROJECT}/artifact-example/ dist/*
$ cd python_libs/artifact-example-bar
$ python setup.py bdist_wheel
$ twine upload --repository-url https://asia-northeast1-python.pkg.dev/${GCP_PROJECT}/artifact-example/ dist/*
I made additional versions each library for testing as v2. On the same way I packaged and uploaded to the repository.
artifact-example-foo (v2)
__version__ = '0.0.2'
def foo():
return 'foo'
def foo2():
return 'foo2'
artifact-example-bar (v2)
__version__ = '0.0.2'
def bar():
return 'bar'
def bar2():
return 'bar2'
Use these uploaded libraries in client Python apps
I created simple Python apps (Flask) in App Engine, Cloud Functions and Cloud Run in the same Google Cloud project as the repository of Artifact Registry.
Notice: If you create in another Google Cloud project, you might need to grant a service account of Code Build. (The related official guide is here)
App Engine
client_apps/appengine/main.py
from flask import Flask, jsonify
import artifact_example_foo as foo
import artifact_example_bar as bar
app = Flask(__name__)
@app.route('/')
def hello():
return jsonify(dict(result='ok', foo=foo.foo(), foo_version=foo.__version__, bar=bar.bar2(), bar_version=bar.__version__))
if __name__ == '__main__':
app.run(host='127.0.0.1', port=8080, debug=True)
client_apps/appengine/requirements.txt
--extra-index-url https://asia-northeast1-python.pkg.dev/<YOUR-GCP-PROJECT>/artifact-example/simple
artifact-example-foo==0.0.1
artifact-example-bar
Flask==2.0.2
In the requirements.txt file, a point is that the Artifact Registry URL is added into the first line and the relevant packages to include private dependencies artifact-example-foo
and artifact-example-bar
.
Also, the /simple
string at the end of the repository URL is needed because of Python Simple Repository API.
In the dependencies body, for testing I specified artifact-example-foo
to 0.0.1 (v1) and do not specify any versions in artifact-example-bar
Cloud Functions
client_apps/functions/main.py
from flask import jsonify
import artifact_example_foo as foo
import artifact_example_bar as bar
def main(request):
return jsonify(dict(result='ok', foo=foo.foo(), foo_version=foo.__version__, bar=bar.bar2(), bar_version=bar.__version__))
client_apps/functions/requirements.txt
--extra-index-url https://asia-northeast1-python.pkg.dev/<YOUR-GCP-PROJECT>/artifact-example/simple
artifact-example-foo==0.0.1
artifact-example-bar
The method of declaring the Artifact Registry repository is the same as the above App Engine one.
Cloud Run
client_apps/cloudrun/main.py
import os
from flask import Flask, jsonify
import artifact_example_foo as foo
import artifact_example_bar as bar
app = Flask(__name__)
@app.route("/")
def main():
return jsonify(dict(result='ok', foo=foo.foo(), foo_version=foo.__version__, bar=bar.bar2(), bar_version=bar.__version__))
if __name__ == "__main__":
app.run(debug=True, host="0.0.0.0", port=int(os.environ.get("PORT", 8080)))
client_apps/cloudrun/requirements.txt
--extra-index-url https://asia-northeast1-python.pkg.dev/<YOUR-GCP-PROJECT>/artifact-example/simple
artifact-example-foo==0.0.1
artifact-example-bar
Flask==2.0.2
gunicorn==20.1.0
The method of declaring the Artifact Registry repository is the same as the above too.
client_apps/cloudrun/Dockerfile
# refs: https://cloud.google.com/run/docs/quickstarts/build-and-deploy/deploy-python-service
FROM python:3.10-slim
ENV PYTHONUNBUFFERED True
ENV GOOGLE_APPLICATION_CREDENTIALS "key.json"
ENV APP_HOME /app
WORKDIR $APP_HOME
COPY . ./
RUN pip install --upgrade pip
RUN pip install keyrings.google-artifactregistry-auth
RUN pip install --no-cache-dir -r requirements.txt
CMD exec gunicorn --bind :$PORT --workers 1 --threads 8 --timeout 0 main:app
I needed to enables to access the Artifact Registry repository while installing dependencies in Docker, so the following command allowed to copy the service account credentials.
ENV GOOGLE_APPLICATION_CREDENTIALS "key.json"
The deploy script is for your reference.
The Response of these client apps
The following JSON is the response of theses client apps.
You can see that the library foo's version is used 0.0.1 (v1) and the library bar's version is used 0.0.2 (v2) as expected.
{
"bar": "bar2",
"bar_version": "0.0.2",
"foo": "foo",
"foo_version": "0.0.1",
"result": "ok"
}
Conclusion
By using Artifact Registry I could specify a version each library, so I would improve to manage python packages and to control a system quality finely than current the pip command pip install -t
.
Top comments (0)