DEV Community: Marjori Pomarole

Enabling OpenCensus Tracing with Django and Google Cloud Run

Marjori Pomarole — Thu, 09 Sep 2021 21:27:20 +0000

Make sure the service worker you have set up on your Cloud Run service has IAM roles cloudtrace.agent. if you are using terraform, it will be set up as such:

resource "google_project_iam_binding" "service_permissions" {
  for_each = toset([
    "logging.logWriter",
    "cloudsql.client",
    "clouddebugger.agent",
    "cloudtrace.agent"
  ])

  role       = "roles/${each.key}"
  ...
}

Install the required pip packages from OpenCensus and google-cloud-trace. Which is the new hotness that merged OpenTracing and OpenMetrics into one beautiful package. I have added these to my requirements.txt file.

...
opencensus-ext-django==0.7.5
opencensus-ext-stackdriver==0.8.0
opencensus==0.7.13
...

Then pip install -r requirements.txt

Now we simply have to configure django settings to include the opencensus middleware and tell the package to export it in StackDriver (now Google Cloud Operations/Monitoring) format.

On my core settings file /core/settings/base.py, we have:

MIDDLEWARE = [
    ...
    'opencensus.ext.django.middleware.OpencensusMiddleware',
]

OPENCENSUS = {
    'TRACE': {
        'EXPORTER': 'opencensus.ext.stackdriver.trace_exporter.StackdriverExporter()',
        'PROPAGATOR': 'opencensus.trace.propagation.google_cloud_format.GoogleCloudFormatPropagator()',
    }
}

And let's build the new image with gcloud builds submit or by using a Cloud Trigger. And we are done! Now you can check the Google Trace console to see your traces in action.

Bootstrapping Label Studio on Google Cloud with Terraform

Marjori Pomarole — Sun, 29 Aug 2021 22:04:43 +0000

Say you need to quickly label tons of data for an ML learning pipeline. You have people on your team or university lab ready to start labeling, but you are struggling to quickly put a tool up and running for them to label concurrently.

Label Studio is a great tool for this. In this post, my goal is to show you how to deploy label-studio as well as show how to create terraform files to quickly bootstrap a Django application running on Google Run connected to a Postgres instance on Google SQL.

The code can be found in this pull request.

First login to your google cloud account.

gcloud auth application-default login

Build the docker image and push it to Google Build with this cloudbuild.yaml script:

steps:
  - id: "build image"
    name: "gcr.io/cloud-builders/docker"
    args:
      [
        "build",
        "-t",
        "gcr.io/${PROJECT_ID}/${_SERVICE_NAME}",
        "${_SERVICE_FOLDER}",
      ]

  - id: "push image"
    name: "gcr.io/cloud-builders/docker"
    args: ["push", "gcr.io/${PROJECT_ID}/${_SERVICE_NAME}"]

substitutions:
  _SERVICE_FOLDER: .
  _SERVICE_NAME: label-studio

images:
  - "gcr.io/${PROJECT_ID}/${_SERVICE_NAME}"

I used direnv to setup the PROJECT_ID env variable to be the Google Project created.

Set up the terraform files to bootstrap the project. Below are the terraform I created to get this all up, but you will be needing to run the following commands to let the resources be created:

terraform init
terraform apply -var="database_password=<DB_PASSWORD>" \
                -var="project=<PROJECT_ID>" \
                -var="database_user=<DB_USER>" \
                -var="domain_name=<DOMAIN_NAME>"

After about 10-15 min, all the resources will be created, and you will have Label Studio up and running to create users and start labeling!

main.tf

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 3.80"
    }
  }
}

provider "google" {
  project = var.project
}

locals {
  service_folder = "service"
  service_name   = "label-studio"

  deployment_name        = "label-studio"
  label_studio_worker_sa = "serviceAccount:${google_service_account.label_studio_worker.email}"
}

project.tf

Creating the service account that will be used to run the application as well as enable the APIs and services that will be used.

# Enable services
resource "google_project_service" "run" {
  service            = "run.googleapis.com"
  disable_on_destroy = false
}

resource "google_project_service" "iam" {
  service            = "iam.googleapis.com"
  disable_on_destroy = false
}

resource "google_project_service" "cloudbuild" {
  service            = "cloudbuild.googleapis.com"
  disable_on_destroy = false
}

# Create a service account
resource "google_service_account" "label_studio_worker" {
  account_id   = "label-studio-worker"
  display_name = "Label Studio SA"
}

# Set permissions
resource "google_project_iam_binding" "service_permissions" {
  for_each = toset([
    "logging.logWriter", "cloudsql.client"
  ])

  role       = "roles/${each.key}"
  members    = [local.label_studio_worker_sa]
  depends_on = [google_service_account.label_studio_worker]
}

database.tf

# The Cloud SQL postgres
resource "google_sql_database_instance" "label-studio-postgres" {
  name             = "label-studio-sql"
  database_version = "POSTGRES_13"
  region           = var.region

  settings {
    tier = var.database_tier
  }
}

resource "google_sql_user" "users" {
  name     = var.database_user
  instance = google_sql_database_instance.label-studio-postgres.name
  password = var.database_password
}

service.tf

Run the service, setup IAM policy and create the domain mapping.

# The Cloud Run service
resource "google_cloud_run_service" "label-studio" {
  name                       = local.service_name
  location                   = var.region
  autogenerate_revision_name = true

  template {
    spec {
      service_account_name = google_service_account.label_studio_worker.email
      containers {
        image = "gcr.io/${var.project}/${local.service_name}"
        env {
          name  = "DEBUG"
          value = "True"
        }
        env {
          name  = "LOG_LEVEL"
          value = "DEBUG"
        }
        env {
          name  = "DJANGO_DB"
          value = "default"
        }
        env {
          name  = "POSTGRE_USER"
          value = "postgres"
        }
        env {
          name  = "POSTGRE_PASSWORD"
          value = google_sql_user.users.password
        }
        env {
          name  = "POSTGRE_NAME"
          value = "postgres"
        }
        env {
          name  = "POSTGRE_HOST"
          value = "/cloudsql/${google_sql_database_instance.label-studio-postgres.connection_name}"
        }
        env {
          name  = "POSTGRE_PORT"
          value = "5432"
        }
        env {
          name  = "GOOGLE_LOGGING_ENABLED"
          value = "True"
        }
      }
    }

    metadata {
      annotations = {
        "autoscaling.knative.dev/maxScale"      = "1000"
        "run.googleapis.com/cloudsql-instances" = google_sql_database_instance.label-studio-postgres.connection_name
        "run.googleapis.com/client-name"        = "terraform"
      }
    }
  }

  traffic {
    percent         = 100
    latest_revision = true
  }

  depends_on = [google_project_service.run, google_sql_database_instance.label-studio-postgres]
}

# Set service public
data "google_iam_policy" "noauth" {
  binding {
    role = "roles/run.invoker"
    members = [
      "allUsers",
    ]
  }
}

resource "google_cloud_run_service_iam_policy" "noauth" {
  location = google_cloud_run_service.label-studio.location
  project  = google_cloud_run_service.label-studio.project
  service  = google_cloud_run_service.label-studio.name

  policy_data = data.google_iam_policy.noauth.policy_data
  depends_on  = [google_cloud_run_service.label-studio]
}

resource "google_cloud_run_domain_mapping" "default" {
  location = var.region
  name     = var.domain_name

  metadata {
    namespace = var.project
  }

  spec {
    route_name = google_cloud_run_service.label-studio.name
  }
}

variables.tf

variable "project" {
  type        = string
  description = "Google Cloud Platform Project ID"
}

variable "region" {
  default = "us-central1"
  type    = string
}

variable "database_user" {
  default     = "postgres"
  type        = string
  description = "PostgresSQL user."
}

variable "database_password" {
  type        = string
  description = "PostgresSQL database user password."
}

variable "database_tier" {
  default     = "db-f1-micro"
  type        = string
  description = "PostgresSQL database tier."
}

variable "domain_name" {
  type        = string
  description = "Domain name where service will be served from."
}

Google Cloud Log severity in Django deployed on Cloud Run

Marjori Pomarole — Sat, 28 Aug 2021 13:50:39 +0000

When deploying a Django/python application to Google Run, logs sent to console will automatically get picked up by Google's Ops agent (based on Fluent Bit).

The only problem is that log severity does not get picked up automatically if you are using python's logging library.

Even when logging like the following:

logging.warn('This should have WARNING level');

Logs on Google's Log Explorer will all appear with level default.

To fix this, you will have to configure Django/python's LOGGING settings to use CloudLoggingHandler:

LOGGING = {
    'handlers':
        'console': {
            'level': get_env('LOG_LEVEL', 'WARNING'),
            'class': 'logging.StreamHandler'
        }
    },
    'root': {
        'handlers': ['console'],
        'level': get_env('LOG_LEVEL', 'WARNING'),
    }
}

// When running in an environemnt without gcloud auth credentials
if get_env('GOOGLE_LOGGING_ENABLED', False):
    try:
        import google.cloud.logging
        from google.auth.exceptions import GoogleAuthError

        client = google.cloud.logging.Client()
        client.setup_logging()

        LOGGING['handlers']['google_cloud_logging'] = {
            'level': get_env('LOG_LEVEL', 'WARNING'),
            'class': 'google.cloud.logging.handlers.CloudLoggingHandler',
            'client': client
        }

        LOGGING['root']['handlers'].append('google_cloud_logging')

    except GoogleAuthError as e:
        logger.exception('Google Cloud Logging handler could not be setup.')

Now we successfully get to parse the severity metadata to create alarms and metrics dashboards.