DEV Community

Jayanth MKV
Jayanth MKV

Posted on

GCP publish python package in production

#ai #gcp #python #pythonpackage

GCP: Publish Python Package in Production

This guide explains how to use Google Artifact Registry to manage shared Python code as a package. This approach eliminates code duplication between your Cloud Functions and server.

Step 1: Structure Your Shared Code

Create a new Python package for your shared logic (e.g., common_logic).

common_logic/
├── setup.py
├── common_logic/
│   ├── __init__.py
Enter fullscreen mode Exit fullscreen mode

Step 2: Create setup.py

Define your package configuration in a setup.py file:

from setuptools import setup, find_packages

setup(
    name="common_logic",
    version="0.1.0",
    packages=find_packages(),
    install_requires=[
        "pandas>=1.3.0",
    ],
    author="Your Name",
    author_email="your.email@example.com",
    description="Common logic for app",
)
Enter fullscreen mode Exit fullscreen mode

Step 3: Set Up Google Artifact Registry

  1. Enable the Artifact Registry API: 
   gcloud services enable artifactregistry.googleapis.com
Enter fullscreen mode Exit fullscreen mode
  1. Create a Python repository: 
   gcloud artifacts repositories create python-packages \
       --repository-format=python \
       --location=us-central1 \
       --description="Python packages repository"
Enter fullscreen mode Exit fullscreen mode

Step 4: Configure Authentication

  1. Create a service account: 
   gcloud iam service-accounts create artifact-publisher \
       --description="Service account for publishing to Artifact Registry"
Enter fullscreen mode Exit fullscreen mode
  1. Grant necessary permissions: 
   gcloud artifacts repositories add-iam-policy-binding python-packages \
       --location=us-central1 \
       --member="serviceAccount:artifact-publisher@${PROJECT_ID}.iam.gserviceaccount.com" \
       --role="roles/artifactregistry.writer"
Enter fullscreen mode Exit fullscreen mode
  1. Create and download a key: 
   gcloud iam service-accounts keys create key.json \
       --iam-account=artifact-publisher@${PROJECT_ID}.iam.gserviceaccount.com
Enter fullscreen mode Exit fullscreen mode

Step 5: Build and Upload Package

  1. Install build tools: 
   pip install build twine
Enter fullscreen mode Exit fullscreen mode
  1. Build the package: 
   python -m build
Enter fullscreen mode Exit fullscreen mode
  1. Configure twine for Artifact Registry: 
   cat > ~/.pypirc << EOL
   [distutils]
   index-servers = common-logic-repo
   [common-logic-repo]
   repository: https://us-central1-python.pkg.dev/${PROJECT_ID}/python-packages/
   username: _json_key_base64
   password: $(base64 -w0 key.json)
   EOL
Enter fullscreen mode Exit fullscreen mode
  1. Upload the package: 
   twine upload --repository common-logic-repo dist/*
Enter fullscreen mode Exit fullscreen mode

Step 6: Use the Package

In Cloud Functions

  1. Create a requirements.txt file: 
   --index-url https://pypi.org/simple
   --extra-index-url https://oauth2accesstoken:${ARTIFACT_REGISTRY_TOKEN}@us-central1-python.pkg.dev/${PROJECT_ID}/python-packages/simple/
   common-logic==0.1.0
Enter fullscreen mode Exit fullscreen mode
  1. Use the package in your Cloud Function: 
   from common_logic import ...

   def cloud_function(request):
       # Your cloud function code using the imported functions
       pass
Enter fullscreen mode Exit fullscreen mode

In Server Code

  1. Add to your server's requirements.txt: 
   --index-url https://pypi.org/simple
   --extra-index-url https://oauth2accesstoken:${ARTIFACT_REGISTRY_TOKEN}@us-central1-python.pkg.dev/${PROJECT_ID}/python-packages/simple/
   common-logic==0.1.0
Enter fullscreen mode Exit fullscreen mode
  1. Use it in your server code: 
   from common_logic import ...
   # Your server code using the imported functions
Enter fullscreen mode Exit fullscreen mode

Step 7: CI/CD Integration

  1. Add the service account key as a secret in your GitHub repository.
  2. Update your Cloud Build configuration: 
   steps:
     - name: 'python'
       entrypoint: pip
       args: ['install', 'build', 'twine']

     - name: 'python'
       entrypoint: python
       args: ['-m', 'build']
       dir: 'common_logic'

     - name: 'python'
       entrypoint: twine
       args: ['upload', '--repository', 'common-logic-repo', 'dist/*']
       dir: 'common_logic'
       env:
         - 'TWINE_USERNAME=_json_key_base64'
         - 'TWINE_PASSWORD=${_ARTIFACT_REGISTRY_KEY}'
Enter fullscreen mode Exit fullscreen mode

Step 8: Version Management

  1. Update the version in setup.py.
  2. Build and upload the new version.
  3. Update requirements.txt in both Cloud Functions and server code.
  4. Deploy both components.

Best Practices

  • Use semantic versioning for your package.
  • Pin specific versions in requirements.txt.
  • Test new versions thoroughly before deploying.
  • Keep a changelog of version changes.
  • Use environment variables for PROJECT_ID and LOCATION.
  • Include comprehensive documentation in your package.

Common Issues and Solutions

Authentication Errors

  • Verify service account permissions.
  • Ensure key.json is properly encoded.
  • Check .pypirc configuration.

Package Not Found

  • Verify repository URL format.
  • Check if the package was successfully uploaded.
  • Ensure requirements.txt uses the correct URL format.

Version Conflicts

  • Pin specific versions of dependencies.
  • Use virtual environments for testing.
  • Document dependency requirements clearly.

Top comments (0)

Read next

mikeyoung44 profile image

New Voice Command System Tackles Variable-Length Speech for Improved Live Transcription

Mike Young -

asmyshlyaev177 profile image

First really useful AI tool - AI code review

Alex -

hamzakhan profile image

🛠️ Battle of the Backend: Go vs Node.js vs Python – Which One Reigns Supreme in 2024? 🚀

Hamza Khan -

mikeyoung44 profile image

Enhancing LLM Performance at Scale with CDN-Based Knowledge Injection

Mike Young -