DEV Community: Alvaro David

Secure Web App access with GCP Identity-Aware Proxy

Alvaro David — Fri, 21 May 2021 22:35:11 +0000

Security is an important part of any architecture, especially for remote workers.

During the pandemic, many challenges have appeared for security teams, I saw a lot... a lot... a lot of cloud admins creating VPNs for remote workers in order to access to internal apps.

Imagine this: workers for financial teams, managements teams, people that never heard about VPNs... and now they have to create them to be able to work! Poor people from DevOps, trying to explain and create a peering for each worker on the company =(

So, what if I tell you that you can secure your apps with no VPN client required? Yep, it's possible with GCP Identity-Aware Proxy

The objective for today is create a web app which only authorized accounts can access and it should also be able to consume internal services.

1.- Only authorized accounts can access the app hosted in App Engine.

2.- Only App Engine can consume an internal service. (It could be a VM or cluster, I just chose Cloud Run to make it simpler).

3.- No one can access directly the internal service.

If we look deeper to this architecture we can found more built-in resources on GCP that can help us achieve our objective.

4.- Service-to-service authentication is the ability for one service, which can be an App Engine service, to invoke a Cloud Run (fully managed) service.

5.- Identity-Aware Proxy (IAP) : Use it when you want to enforce access control policies for applications and resources. We will focus in this part.

The Code

For the internal service let's use the API we made in GCP Cloud Run: containers without Dockerfile.

# Just remember to add `--no-allow-unauthenticated` flag 
# to secure the API. 
# This means only authorized services can request this API, 
# in this case App Engine is using its Service Account.

gcloud run deploy my-go-api-service \ 
  --image gcr.io/$PROJECT_ID/my-go-api:v0.1 \
  --region southamerica-east1 \
  --no-allow-unauthenticated \
  --platform managed

For the service-to-service authentication add the roles/run.invoker permission to the App Engine Service Account.

gcloud run services add-iam-policy-binding my-go-api-service \
  --member='serviceAccount:[Your-app-engine-service-account]' \
  --role='roles/run.invoker'

How service-to-service authentication works is an extensive topic, it could be a new post, but just keep this in mind:

The client service has to get a token to be able to request the protected service.

This token can only be generated inside the GCP internal networking.

This means that only instances running on GCP can get this token, it's not possible to get this token outside GCP (like from you computer for example).

With this in mind, the internal request has to be done from App Engine (inside GCP). If we do the request from the client side (Chrome) we won't be able to get the token because Chrome is running on your computer (outside GCP).

So.. what can we do now?

Server Side Rendered

Server-side rendering (SSR) is the process of rendering web pages on a server and passing them to the browser (client-side), instead of rendering them in the browser.

I decided to use NuxtJS for the SSR because I'm familiar to VueJS and it's so simple to use.

I'm absolutely not a frontend guy, so probably I won't use the best practices for the client side (web app), please take it easy :D.

Use the Get Started from NuxtJS to create a SSR project.

<!-- 
*******************************************
I only made three modifications: 
- Get the "token" and 
- Request the internal service via "axios"
- Show the message from the internal service
*******************************************
-->

<!--index.vue-->
<template>
  <div class="container">
    <div>
      <h1 class="title">Cloud Run says:</h1>
      <h2>{{rsp.message}}</h2>
    </div>
  </div>
</template>

<script>
export default {
  async asyncData({ $axios, $config }) {

    // Getting token
    const token = await $axios.$get(`http://metadata/computeMetadata/v1/instance/service-accounts/default/identity?audience=https://my-go-api-service-[your-hash]-rj.a.run.app`, {headers: { 'Metadata-Flavor': 'Google'}})

    // Request internal service
    const rsp = await $axios.$get(`https://my-go-api-service-[your-hash]-rj.a.run.app`, {headers: { 'Authorization': 'Bearer ' + token}})

    return { rsp }
  }
}
</script>

Add the app.yaml file to deploy on App Engine Standard

runtime: nodejs10

instance_class: F2

handlers:
  - url: /_nuxt
    static_dir: .nuxt/dist/client
    secure: always

  - url: /(.*\.(gif|png|jpg|ico|txt))$
    static_files: static/\1
    upload: static/.*\.(gif|png|jpg|ico|txt)$
    secure: always

  - url: /.*
    script: auto
    secure: always

env_variables:
  HOST: '0.0.0.0'

And deploy to App Engine Standard

# Build our project
yarn build

# Deploy to App Engine Standard 
gcloud app deploy

Great! Our internal service is protected and we can consume it from App Engine, but the Web app is still open to everyone, let's secure it.

Identity-Aware Proxy

Simpler for cloud admins: Secure access to apps in less time than it takes to implement a VPN. Let your developers focus on application logic, while IAP takes care of authentication and authorization.

IAP lets you establish a central authorization layer for applications accessed by HTTPS, so you can use an application-level access control model instead of relying on network-level firewalls. Docs

Sounds great but why to use it instead of Firebase Authentication, for example: Firebase Firestore Rules with Custom Claims - an easy way.

Simple, with Firebase Authentication anyone on the internet can register to your app, if they can access the app content is another story.

With IAP you implement a zero-trust access model, this means accounts that are not listed on you policy won't be able even to see the HTML, they will receive this message:

And if you use Workspaces you can limit the access to only accounts from your organization.

First we have to enable IAP on our project

gcloud services enable iap.googleapis.com

Then configure the OAuth consent screen, basically Google displays a consent screen to the user including a summary of your project and its policies and the requested scopes of access.

Go to the Identity-Aware Proxy page and select the resource you wish to modify by checking the box to its left, in this case App Engine.

Now let's add an account to our IAP-secured Web App Users list

gcloud iap web add-iam-policy-binding  \  
  --member='user:alvardev@example.com' \
  --resource-type='app-engine' \ 
  --role='roles/iap.httpsResourceAccessor'

So when this account enters to the web app the HTML will be displayed

That's it!

Our Web app is secured, no VPN client required and we consume an internal service (message: "Hello world! v0.2").

Thanks @lucasturci for the review!

GCP Cloud Run: containers without Dockerfile

Alvaro David — Sat, 06 Mar 2021 08:49:08 +0000

Containers allow us to run our code wherever we want, Docker is a popular option to build our containers. However, sometimes you love Dockerfile and sometimes (or more) you hate Dockerfile.

"44% of the containers in the wild had know vulnerabilities for which patches existed. This just shows you that writing your own Dockerfile can actually be quite fraught with errors unless you're an expert or maybe you have a platform team that provides Dockerfiles for you." Src: Buildpacks in Google Cloud

Writing a Dockerfile could be complicated, considering image size, security, multi-stage, versions and so on, there are too many ways to do that!

Sure, we can handle all that things to get the perfect image but if you are using containers, sooner or later you will have to deploy your container in a Kubernetes cluster (like Cloud Run or GKE on GCP) and Kubernetes has deprecated Docker.

We are Devs, we should focus on the code...

So... Here comes Buildpacks to save the day!

Our main objective for today is to deploy a simple API on Cloud Run without a Dockerfile using Buildpacks.

The Code

We're going to deploy a simple API using Go

package main

import (
    "encoding/json"
    "log"
    "net/http"
)

// Response definition for response API
type Response struct {
    Message string `json:"message"`
}

func handler(w http.ResponseWriter, r *http.Request) {

    response := Response{}
    response.Message = "Hello world!"
    responseJSON, err := json.Marshal(response)

    if err != nil {
        log.Fatalf("Failed to parse json: %v", err)
    }

    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(http.StatusOK)
    w.Write(responseJSON)
}

func main() {
    log.Print("starting server...")
    http.HandleFunc("/", handler)
    http.ListenAndServe(":8080", nil)
}

Using Dockerfile

Let's begin with the 'popular' way, I mean using a Dockerfile

# Source: https://github.com/GoogleCloudPlatform/golang-samples/blob/master/run/helloworld/Dockerfile

FROM golang:1.16-buster as builder

# Create and change to the app directory.
WORKDIR /app

# Retrieve application dependencies.
# This allows the container build to reuse cached dependencies.
# Expecting to copy go.mod and if present go.sum.
COPY go.* ./
RUN go mod download

# Copy local code to the container image.
COPY . ./

# Build the binary.
RUN go build -mod=readonly -v -o server

# Use the official Debian slim image for a lean production container.
# https://hub.docker.com/_/debian
# https://docs.docker.com/develop/develop-images/multistage-build/#use-multi-stage-builds
FROM debian:buster-slim
RUN set -x && apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
    ca-certificates && \
    rm -rf /var/lib/apt/lists/*

# Copy the binary to the production image from the builder stage.
COPY --from=builder /app/server /app/server

# Run the web service on container startup.
CMD ["/app/server"]

Don't forget the .dockerignore!

# The .dockerignore file excludes files from the container build process.
#
# https://docs.docker.com/engine/reference/builder/#dockerignore-file

# Exclude locally vendored dependencies.
vendor/

# Exclude "build-time" ignore files.
.dockerignore
.gcloudignore

# Exclude git history and configuration.
.gitignore
.git

With no doubt somebody will say:

-"Hey! your Dockerfile is incomplete! with multi-stage your image will be smaller"

And... yes, you're right! with this Dockerfile the image is 82.6 mb, I found somewhere a long time ago a Dockerfile with multi-stage for Go, it's here just for the record, and the image size was reduced to 13.7 mb.

Great right? Not at all, First I don't remember where I found the multi-stage Dockerfile or if the post even exists today, second for a maintainer it could be difficult to understand the process if an update (security patch) is required.

Using Buildpacks

Buildpacks is an open-source technology that makes it fast and easy for you to create secure, production-ready container images from source code and without a Dockerfile, following the best practices. Src: Announcing Google Cloud buildpacks—container images made easy

The first thing you should do for using Buildpacks is to remove the Dockerfile and the .dockerignore.

That's it! you keep what it matters: your code.

To install buildpacks go to: https://buildpacks.io/docs/tools/pack/

To build the image use this:

pack build my-go-api:v0.1 --builder  gcr.io/buildpacks/builder

To test the image use this:

docker run --rm -p 8080:8080 my-go-api:v0.1

# In another terminal
curl --request GET --url http://localhost:8080/

# Response expected
# {"message": "Hello world!"}

Awesome! we have our image running locally, now let's deploy to Cloud Run

# You can push the image that you built
# but I want to show you that buildpacks can be
# used with gcloud too.

export PROJECT_ID=$(gcloud config list --format 'value(core.project)')

gcloud builds submit --pack image=gcr.io/$PROJECT_ID/my-go-api:v0.1

# Deploy to Cloud Run
gcloud run deploy my-go-api-service \
  --image gcr.io/$PROJECT_ID/my-go-api:v0.1 \
  --region southamerica-east1 \
  --platform managed

curl https://my-go-api-service-[your-hash]-rj.a.run.app
# {"message": "Hello world!"}

Here is the repo with all the code in this post, check the branchs.

Hope this post helps you!

GCP Cloud SQL Secure connection

Alvaro David — Wed, 02 Dec 2020 18:32:25 +0000

Update

It's Google I/O 21! and new features are coming :D

One of them is about Cloud Run and Secret Manager, so here comes the update:

Yep, ENV_VARs are a little bit dangerous because in some way they would be in a plain text in some place (like the Cloud Run Console or a file if you use volumes).

But now we can set our Envs directly from Secret Manager, that means we won't be able to see the secrets on the console (or in a file).

# Just add those secrets as envs when you deploy your service.
# For this example I'm using 'latest', it's recommended to use a specific version. 

gcloud beta run deploy pets-api \  
  --image gcr.io/Project-A/pets-api:v0.1 \  
  --update-secrets=db_password=db_password_secret:latest

Another tip I want to show you is from Wietse Venema from his book Building Cloud Applications with Google Cloud Run.

"To add another barrier to prevent direct connections [to our Cloud SQL instance], you can request a SSL/TLS for those [connections]:"

gcloud sql instance patch --require-ssl my-pets-instance

That's it guys!
Remember to follow the Best Practices from documentation.

A secure connection is mostly a networking topic because it takes infrastructure tasks. GCP has resources and tools that we can use to resolve those tasks as Devs.

In this case, we have a API that returns a list of pets from a MySQL Database.

Our objective is to connect our API from Project A to a MySQL Database on Project B using the ready-to-use tools from GCP.

Sure, there are other ways to connect to a DB instance like VPN, VPC for privates IPs (for serverless resources you can use the Serverless VPC Access connector e.g. Cloud Functions), Firewall and etc. However, as developers we can take advantage of GCP resources, not need to worry about infrastructure, just code!

By the way, this connection can be used in only one GCP project too

Architecture

In this architecture we have two GCP Projects (A and B)

As you can see in the image:

Our API is running on Cloud Run in Project A.
Our Cloud SQL instance (MySQL) is running on Project B.

Both are managed by GCP, so we are cool. Here comes security tools:

Cloud SQL Proxy

The Cloud SQL proxy allows you to authorize and secure your connections using Identity and Access Management (IAM) permissions. The proxy validates connections using credentials for a user or service account, and wrapping the connection in a SSL/TLS layer that is authorized for a Cloud SQL instance. Docs

Secret Manager

Secret Manager is a secure and convenient storage system for API keys, passwords, certificates, and other sensitive data. Secret Manager provides a central place and single source of truth to manage, access, and audit secrets across Google Cloud. Docs

Time to code!

Enough theory, let's code a little bit :D

Database (On Project B)

Actually, create a Cloud SQL instance with MySQL on GCP is very simple, just follow this Quickstart

Schema

A simple table for Pets

CREATE DATABASE petsbook;

USE petsbook;
CREATE TABLE pets (
    id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, 
    name VARCHAR(255) NOT NULL
);

INSERT INTO pets (name) values ("Mel");


SELECT id, name from pets;
-- 1, Mel (My pet)

Pets API (On Project A)

main.py

from flask import Flask, jsonify
import sqlalchemy

app = Flask(__name__)


def init_connection_engine():

    db_user = "my-super-user"
    db_pass = "my-super-password"
    db_name = "my-pets-database"
    db_socket_dir = '/cloudsql'

    # <PROJECT-NAME>:<INSTANCE-REGION>:<INSTANCE-NAME>"
    cloud_sql_connection_name = "Project-B:southamerica-east1:my-pets-instance"

    db_config = {
        "pool_size": 5,
        "max_overflow": 2,
        "pool_timeout": 30,  # 30 seconds
        "pool_recycle": 1800,  # 30 minutes
    }

    pool = sqlalchemy.create_engine(
        sqlalchemy.engine.url.URL(
            drivername="mysql+pymysql",
            username=db_user,
            password=db_pass,
            database=db_name,
            query={
                "unix_socket": "{}/{}".format(
                    db_socket_dir,
                    cloud_sql_connection_name)
            }
        ),
        **db_config
    )

    return pool


@app.before_first_request
def create_connection():
    global db
    db = init_connection_engine()


@app.route("/", methods=['GET'])
def get_pets():
    """
    This method returns a list of pets from DB on Cloud SQL
    """
    pets = []

    with db.connect() as conn:
        pets_result = conn.execute("SELECT id, name from pets;").fetchall()
        for row in pets_result:
            pets.append({"id": row[0], "name": row[1]})

    # Response
    return jsonify(pets), 200

Implement the Cloud SQL Proxy

[On Project A] Create a Service Account.

# Create Service Account (on Project A)
gcloud iam service-accounts create pets-api-cred \
  --description "Service Account for Pets API Service" \ 
  --display-name "Pets API Service Account"

[On Project B] Grant Cloud SQL > Client role to the Service account you created.

# Grant Cloud SQL Client role (on Project B)
gcloud projects add-iam-policy-binding Project-B \ 
   --member=serviceAccount:pets-api-cred@Project-A.iam.gserviceaccount.com \ 
   --role="roles/cloudsql.client"

[On Project A] Build and deploy the API on Cloud Run

# Build image
docker build -t gcr.io/Project-A/pets-api:v0.1 . 

# Push to Container Register 
docker push gcr.io/Project-A/pets-api:v0.1 

# Deploy to Cloud Run
gcloud run deploy pets-api \ 
  --image gcr.io/Project-A/pets-api:v0.1 \
  --region southamerica-east1 \
  --platform managed \
  --allow-unauthenticated \
  --add-cloudsql-instances Project-B:southamerica-east1:my-pets-instance \
  --update-env-vars INSTANCE_CONNECTION_NAME="Project-B:southamerica-east1:my-pets-instance" \
  --service-account pets-api-cred@Project-A.iam.gserviceaccount.com

--add-cloudsql-instances and --update-env-vars indicates to Cloud Run where the Cloud SQL Instance is located.
--service-account indicates to Cloud Run to use that Service Account.

This is a powerful advantage! because we don't need to download any key for the Service Account (e.g. a JSON file).

Let's try our Pets API

curl https://pets-api-[your-hash]-rj.a.run.app
# [{"id":1,"name":"Mel"}]

It's working!
But... wait a minute... the user and password are exposed on the code .-.

Yep, we can use environment variables to hide the information but in some way the user and the password would be in plain text in some place.

The idea is that all sensitive information should be hidden for everyone, everywhere, even on the CI/CD.

Implement Secret Manager

All our sensitive informations (e.g. user and password) should be protected, Secret Manager is a great resource for that.

First, create our Secrets

# Enable Secret Manager
gcloud services enable secretmanager.googleapis.com

# Create a secret for each sensitive information
gcloud secrets create db_user_secret --replication-policy="automatic"
gcloud secrets create db_password_secret --replication-policy="automatic"
gcloud secrets create db_name_secret --replication-policy="automatic"
gcloud secrets create cloud_sql_connection_name_secret --replication-policy="automatic"

# Create a version (contains the actual contents of a secret) for each secret
gcloud secrets versions add db_user_secret --data-file="db_user_secret.txt"
gcloud secrets versions add db_password_secret --data-file="db_password_secret.txt"
gcloud secrets versions add db_name_secret --data-file="db_name_secret.txt"
gcloud secrets versions add cloud_sql_connection_name_secret --data-file="cloud_sql_connection_name_secret.txt"

# Add a secret version from the contents of a file on disk.
# You can also add a secret version directly on the command line, 
# but this is discouraged because the plaintext will appear in your shell history.

# Grant the Manager Secret Accessor role to our Service Account
gcloud projects add-iam-policy-binding Project-A \ 
   --member=serviceAccount:pets-api-cred@Project-A.iam.gserviceaccount.com \ 
   --role="roles/secretmanager.secretAccessor"

Then we have to access to the information by code. Docs

...
def init_connection_engine():
    # Create the Secret Manager client.
    client = secretmanager.SecretManagerServiceClient()

    # Build the resource name of the secret version.
    db_user_secret = f"projects/{PROJECT_ID}/secrets/db_user_secret/versions/latest"
    db_password_secret = f"projects/{PROJECT_ID}/secrets/db_password_secret/versions/latest"
    db_name_secret = f"projects/{PROJECT_ID}/secrets/db_name_secret/versions/latest"
    cloud_sql_conn_name_secret = f"projects/{PROJECT_ID}/secrets/cloud_sql_connection_name_secret/versions/latest"

    # Access the secret version.
    db_user_response = client.access_secret_version(request={"name": db_user_secret})
    db_password_response = client.access_secret_version(request={"name": db_password_secret})
    db_name_response = client.access_secret_version(request={"name": db_name_secret})
    cloud_sql_conn_name_response = client.access_secret_version(request={"name": cloud_sql_conn_name_secret})

    db_user = db_user_response.payload.data.decode("UTF-8")
    db_pass = db_password_response.payload.data.decode("UTF-8")
    db_name = db_name_response.payload.data.decode("UTF-8")
    db_socket_dir = '/cloudsql'
    cloud_sql_connection_name = cloud_sql_conn_name_response.payload.data.decode("UTF-8")

    db_config = {
        "pool_size": 5,
        "max_overflow": 2,
        "pool_timeout": 30,  # 30 seconds
        "pool_recycle": 1800,  # 30 minutes
    }

    pool = sqlalchemy.create_engine(
        sqlalchemy.engine.url.URL(
            drivername="mysql+pymysql",
            username=db_user,
            password=db_pass,
            database=db_name,
            query={
                "unix_socket": "{}/{}".format(
                    db_socket_dir,
                    cloud_sql_connection_name)
            }
        ),
        **db_config
    )

    return pool


@app.before_first_request
def create_connection():
    global db
    db = init_connection_engine()

...

Great! Everything is set, our connections is secure and our sensitive information is protected :)

All the code is on GitHub
https://github.com/AlvarDev/Cloud-SQL-Connection

Hope it helps you!

GCP Cloud Logging: The Basis

Alvaro David — Fri, 18 Sep 2020 06:57:46 +0000

Recently the Google Cloud Platform Blog posted an interesting article about Tips and tricks for using new RegEx support in Cloud Logging, this is a powerful feature but in some way many developers are not even familiar with Logging!

Logging and monitoring are an important part of your project, you have to know what is going on inside your application and I'm not just talking about a print() or console.log() or System.out.println() or etc.

Rich log content (obviously anonymized and following the data protection laws) will help us to understand the behavior of our applications and improve them.

So, this post is about how to create and manage our logs on GCP Cloud Logging in a simple way for devs.

Architecture

This is an architecture that shows a basic pipeline for our logs.

Architecture description:

Cloud Run: The example is a Docker and Cloud Run is the easiest way to deploy a container on GCP.
Cloud Logging: Here is where we are going to manage our logs.
Logs export (Sink): One advantage to use Cloud Logging is the ability to export our logs to:
- Pub/Sub: Basically, we can export our logs wherever we want, just publish the log and we are done.
- BigQuery: Logs also can be analyzed to get insights about our applications and create ML models with BigQuery ML
- Cloud Storage: Save our logs in a cheaper way, this could be a good option for systems audit.
Create Metrics: We can create dashboard and alerts based on logs. For example: if we get too many 403 errors we can create an alert to send us an email or notification.

For this post we only will focus on Cloud Logging, for the other resources I will leave links for documentation at the end of the post or create more posts about it if you want :)

GCP Resources and permissions

The Cloud Logging client could be used in any place (on-premise for example) and the best way to represent that is using docker. However, we need to enable the Cloud Logging API and create a Service Account to grant logWriter permission to our application.

# Get project information
export PROJECT_ID=$(gcloud config list \
  --format 'value(core.project)')
export PROJECT_NUMBER=$(gcloud projects list \
  --filter="$PROJECT_ID" \
  --format="value(PROJECT_NUMBER)")

# Enable services on GCP project
gcloud services enable logging.googleapis.com

# Creating a Service Account for our API
gcloud iam service-accounts create logging-basis-cred \
  --description "Service Account for Logging Basis Service" \
  --display-name "Logging Basis Service Account"

gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member=serviceAccount:logging-basis-cred@$PROJECT_ID.iam.gserviceaccount.com \
  --role="roles/logging.logWriter"

gcloud iam service-accounts keys create \
  --iam-account logging-basis-cred@$PROJECT_ID.iam.gserviceaccount.com logging-basis-cred.json

# For CI/CD you can encrypt this service account 
# using KMS (https://cloud.google.com/security-key-management).

# NEVER save a service account on a repository.

The Code

I use python in this example but you can use the client that you prefer, the concepts are the same.

main.py

from flask import Flask
from google.cloud import logging
from google.cloud.logging.resource import Resource
import random
import os

app = Flask(__name__)

# resource_type: "cloud_run_revision" (from Dockerfile),
# GCP Resource that we use.
# If not set this log can be found in
# the Cloud Logging console under 'Custom Logs'.
# or using the less efficient global restriction
resource_type = os.environ['RESOURCE_TYPE']

# service_name: "logging-basis" (from Dockerfile),
# service_name is a Cloud Run property
service_name = os.environ['SERVICE']

# region: "us-east1" (from Dockerfile)
region = os.environ['REGION']

# Log Name
log_name = 'choose_side'

'''Stackdriver Logging config (This could be a Class)'''
logging_client = logging.Client()
logger = logging_client.logger(log_name)  
resource = Resource(
    type=resource_type,
    labels={
        "service_name": service_name,
        "location": region
    })


@app.route("/", methods=['GET'])
def choose_side():
    """
    This method chooses a random side and return a simple message.
    :return: String message to the client
    """

    # Getting the side
    side_random = random.randrange(2)
    side = 'Dark side' if side_random == 0 else 'Light side'
    struct = {
        'sideRandom': side_random,
        'side': side,
    }

    # Sending log to Stackdriver Logging
    logger.log_struct(struct, resource=resource, severity='INFO')

    # Response
    return "You're the {} [{}]".format(side, side_random)

Dockerfile, requirements and other files are on the GitHub repository of this post: https://github.com/AlvarDev/Stackdriver-Loggin-Basis

Build and deploy

This part is not required by the Cloud Logging, I just wanted to show you how easy is to deploy a container on GCP :)

gcloud auth configure-docker

docker build -t gcr.io/$PROJECT_ID/logging-basis:v0.1 .

docker push gcr.io/$PROJECT_ID/logging-basis:v0.1

gcloud run deploy logging-basis \
  --image gcr.io/$PROJECT_ID/logging-basis:v0.1 \
  --region us-east1 \
  --platform managed \
  --allow-unauthenticated

# Expected response:
# Service [logging-basis] revision [logging-basis-00001-tij]
# has been deployed and is serving 100 percent of traffic at
# https://logging-basis-[your-cloud-run-hash]-ue.a.run.app

Test

The code is working

Logging

This is why we made all that work, now we have the ability to query our logs and export them if needed.

On the GCP Console, Logs Viewer section, we can run the following query:

And this is the result...

There are our logs! In this example a log is a jsonPayload but they could be just a string if you prefer.

Now we can query our logs, the post that inspired this tutorial is a good example. As I told you, it's possible to export this logs to different resource, I think monitoring is a mandatory one because setting an alert for undesired events will help us a lot.

Hope it helps.

Some links to complement this post:

Firebase Firestore Rules with Custom Claims - an easy way

Alvaro David — Mon, 24 Aug 2020 04:34:55 +0000

Firebase Firestore rules are a great tool to provide access control and data validation in a simple and expressive format, it has a great documentation and videos.

Custom Claims provides the ability to implement various access control strategies, including role-based access control, in Firebase apps. These custom attributes can give users different levels of access (roles), which are enforced in an application's security rules.

Both together are awesome but I found that the implementation could be a little bit confusing if you are not familiar with Firebase Authentication, so in this post we are going to create a strategy where we can take advantage of both of them.

For this post I will simplify as much as possible the structure of the Collections and Documents.

The Case

We have a Collection of articles with a title and a content.

And we want that:

Editors can update those articles
Viewers can read those articles

So we create a Collection of access (roles).

As you can see we reuse the Firebase Authentication User UID as a document id.

Without Custom Claims

If you are not using Custom Claims, your Rules will look something like this:

There are two functions that read the Collection of access in order to know if the user (UID) can read/update the article.

This works, but there are two reads, one for the request itself and one more to check the access level, so we are duplicating the number of reads and if an API wants to use this logic the code will have to read that collection too.

We can improve that.

With Custom Claims

Ok, first at all: What is a claim?

Basically is the information that Firebase Authentication will return each time you validate a token.

For example:

{
    "aud": "my-project",
    "auth_time": 1234567890,
    "email": "alvardev@example.com",
    "email_verified": true,
    "exp": 1234567980,
    "firebase": {
        "identities": {
            "email": [
                "alvardev@example.com"
            ],
            "google.com": [
                "1234567890123456789012"
            ]
        },
        "sign_in_provider": "google.com"
    },
    "iat": 1234567890,
    "iss": "https://securetoken.google.com/my-project",
    "name": "Alvaro David",
    "picture": "https://urlformypicture.com/image",
    "sub": "123abc123abc123abc123",
    "user_id": "xIR9cPwbHBQb"
}

This information can be read by the Firestore Rules.

The strategy is to add some fields to this information and avoid to do a second read on our collections.

Custom claims are only used to provide access control. They are not designed to store additional data (such as profile and other custom data). Please read the docs.

In this case, we will add the 'level' field:

{"level": "editor"}  // or viewer

So our rules will be like:

Now it looks cleaner.

Sounds good... but how to implement that?

In both cases (with and without claims) the start point is the Collection of access, every time we update a document in this Collection we have to set the Custom Claim.

The perfect resource for this task is Firebase Functions! Every time a document is update we can trigger a functions that can set the custom claim based on the content updated.

index.js

const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp();

exports.updateAccess = functions.firestore
  .document('access/{accessId}')
  .onUpdate((change, context) => {

    const newValue = change.after.data();
    const customClaims = {
      level: newValue.level
    };

    // Set custom user claims on this update.
    return admin.auth().setCustomUserClaims(
               context.params.accessId, customClaims)
    .then(() => {
      console.log("Done!")
    })
    .catch(error => {
      console.log(error);
    });

});

deploy

firebase deploy --only functions

Now our claim looks like:

{
    "aud": "my-project",
    "auth_time": 1234567890,
    "email": "alvardev@example.com",
    "email_verified": true,
    "exp": 1234567980,
    "firebase": {
        "identities": {
            "email": [
                "alvardev@example.com"
            ],
            "google.com": [
                "1234567890123456789012"
            ]
        },
        "sign_in_provider": "google.com"
    },
    "iat": 1234567890,
    "iss": "https://securetoken.google.com/my-project",
    "name": "Alvaro David",
    "picture": "https://urlformypicture.com/image",
    "sub": "123abc123abc123abc123",
    "user_id": "xIR9cPwbHBQb",
    "level": "editor"  // There you are!
}

That's it! with these few steps we implemented a Custom Claim to enforce our Firestore Rules, those claims are easy to update due to the Firebase Functions triggers.

Hope it helps and my the source be with you :)

PoseNet - TensorFlow.js on GCP

Alvaro David — Mon, 29 Jun 2020 06:04:23 +0000

TensorFlow.js is a JavaScript Library for training and deploying machine learning models in the browser and in Node.js.

Fortunately, TensorFlow.js has ready-to-use models that can be used by anyone, especially for developers with no knowledge of ML.

One of them is PoseNet, a standalone model for running real-time pose estimation in the browser using TensorFlow.js.

For browsers PoseNet works perfectly, the challenge comes when we want to us it on the backend, I made a simple API to use the model and I want to share with you a couple of issues that I had to resolve in order to deploy the API on GCP. (complete source at the end of the post)

TensorFlow vs Docker

Docker is a powerful tool. However, use TensorFlow.js on a docker is little bit harder, messages like Your CPU supports instructions that this TensorFlow binary was not compiled to use... are very frequently.

Production considerations

From documentation: The Node.js bindings provide a backend for TensorFlow.js that implements operations synchronously. That means when you call an operation, e.g. tf.matMul(a, b), it will block the main thread until the operation has completed.

For this reason, the bindings currently are well suited for scripts and offline tasks. If you want to use the Node.js bindings in a production application, like a webserver, you should set up a job queue or set up worker threads so your TensorFlow.js code will not block the main thread.

So... How to resolve this?

Both issues have their own solution, build a detailed Dockerfile and use worker threads for example, but there is one resource that will resolve both with one shot:

Cloud Functions

Cloud Functions instances already have TensorFlow compiled. (One example that I found was How to serve deep learning models using TensorFlow 2.0 with Cloud Functions)
Each instance of a Cloud Function handles only one concurrent request at a time. This means that while your code is processing one request, there is no possibility of a second request being routed to the same instance. Thus the original request can use the full amount of resources (CPU and memory) that you requested. (Source: Auto-scaling and Concurrency)

Some consideration to take are:

Set the max capacity of memory available.
Starting a new Cloud Functions instance involves loading the runtime and your code. Requests that include Cloud Functions instance startup (cold starts) can be slower than requests hitting existing instances.

And... yes, the request is going to be slower than running on a Kubernetes for example. But this solution is a Quick win for developers, we are using a trained model in a Serverless environment.
Developers with no knowledge of ML can take advantage for this technology in an easy way.
For future posts I will use the other solutions: detailed Dockerfile and worker threads. Be patient :).

That's it, a start for developers into the ML world.

Code

This API receives a "gs://" path from Google Cloud Storage.
I made it that way because Cloud Functions can be invoked indirectly in response to an event from Cloud Storage or Pub/Sub.

Source: https://github.com/AlvarDev/posenet-nodejs-gcp

index.js

// Express
const bodyParser = require("body-parser");
const express = require('express');
const app = express();

// Utils
const path = require("path");
const os = require('os');
const fs = require('fs');

// GCS 
const { Storage } = require("@google-cloud/storage");
const storage = new Storage();

// Tensorflow
const tf = require('@tensorflow/tfjs-node');
const posenet = require('@tensorflow-models/posenet');
const { createCanvas, Image } = require('canvas');

/**
 * Validates that the request body has the "image" attr.
 *
 * @param {Object} req Cloud Function request context.
 *                     More info: https://expressjs.com/en/api.html#req
 * @param {Object} res Cloud Function response context.
 *                     More info: https://expressjs.com/en/api.html#res
 * @param {Function} next Function to be execute after validateReq
 */
function validateReq(req, res, next) {
  if (!("image" in req.body)) {
    res.status(404).send({ message: "image GCS path not found" });
    return;
  }

  next();
}

/**
 * Estimate the poses from an image.
 * 
 * @param {String} imagePath Image local path
 * 
 * @returns {Object} poses estimations
 */
async function estimatePose(imagePath) {

  const net = await posenet.load({
    architecture: 'MobileNetV1',
    outputStride: 16,
    inputResolution: { width: 640, height: 480 },
    multiplier: 0.75
  });

  const img = new Image();
  img.src = imagePath;

  const canvas = createCanvas(img.width, img.height);
  const ctx = canvas.getContext('2d');
  ctx.drawImage(img, 0, 0);
  const input = tf.browser.fromPixels(canvas);
  return await net.estimateSinglePose(input, { flipHorizontal: false });
} 

/**
 * Request Handler.
 *
 * @param {Object} req Cloud Function request context.
 *                     More info: https://expressjs.com/en/api.html#req
 * @param {Object} res Cloud Function response context.
 *                     More info: https://expressjs.com/en/api.html#res
 */
async function getPose(req, res){

  // Getting Image path, Cloud use a Regex
  const attrs = req.body.image.split('/');
  const bucketName = attrs[2];
  const filename = attrs[attrs.length - 1];
  const imageGCS = req.body.image.replace(`gs://${bucketName}/`, "");
  const imagePath = path.join(os.tmpdir(), filename);

  // Downlaod from GCS
  try {
    await storage
      .bucket(bucketName)
      .file(imageGCS)
      .download({ destination: imagePath });

  } catch (err) {
    console.log(err.message);
    res.status(404).send({ message: "File not found" });
    return;
  }

  const pose = await estimatePose(imagePath);

  // Remove image
  fs.unlinkSync(imagePath);
  res.status(200).send(pose);
}

app.use(bodyParser.urlencoded({extended: true}));
app.use(bodyParser.json());
app.get("/", (_, res) => { res.send("Hello World!"); });
app.post("/get-poses", validateReq, (req, res) => {
  getPose(req, res);
});
app.use((err, req, res, next) => {
  console.error(err.stack)
  res.status(500).send({message: "Something went wrong"})
})

exports.app = app;

// For localhost test
// app.listen(8080, () => {
//   console.log(`App listening on port 8080`);
//   console.log('Press Ctrl+C to quit.');
// });

package.json (dependencies)

"engines": {
    "node": ">=10.0.0"
  },
"dependencies": {
    "@google-cloud/storage": "4.7.0",
    "@tensorflow/tfjs-node": "2.0.1",
    "@tensorflow-models/posenet": "2.2.1",
    "body-parser": "1.19.0",
    "canvas": "2.6.1",
    "express": "4.17.1"
  }

To test locally you can uncomment the app.listen() method

// For localhost test
app.listen(8080, () => {
  console.log(`App listening on port 8080`);
  console.log('Press Ctrl+C to quit.');
});

execute

node index.js

test locally

curl -X POST \
    http://localhost:8080/get-poses \
    -H 'Content-Type: application/json' \
    -d '{"image": "gs://your-bucket/karate.jpg"}'

deploy to Cloud Functions

gcloud functions deploy app --runtime nodejs10 --trigger-http --allow-unauthenticated

GCP Cloud Functions with a Static IP

Alvaro David — Mon, 22 Jun 2020 03:29:01 +0000

I always recommend to go to the documentation at first place. However, sometimes there are some concepts that are not so clear for everyone. In this case, network concepts for developers.

When we hear Serverless we forgot almost everything about DevOps, networking, memory and so on, just worry about the code and that's ok.

But now we have a requirement: the client API only accepts requests from a whitelisted IP.

This is the schema for a 'traditional' architecture:

Cloud Functions will resolve most of the architecture but as you can see, some resources from a 'traditional' architecture are still necessary to achieve our objetive and that's where I want to help.

Create a Simple HTTP Function

main.py

# This function will return the IP address for egress
import requests
import json

def test_ip(request):
    result = requests.get("https://api.ipify.org?format=json")
    return json.dumps(result.json())

deploy

gcloud functions deploy testIP \
    --runtime python37 \
    --entry-point test_ip \
    --trigger-http \
    --allow-unauthenticated

test

curl https://us-central1-your-project.cloudfunctions.net/testIP
# {"ip": "35.203.245.150"} (ephemeral: changes any time)

Networking

So this is the part that some devs going crazy, we're going to use a VPC (Virtual Private Cloud) that provides networking functionalities to out cloud-based services, in this case our Cloud Function.

VPC networks do not have any IP address ranges associated with them. IP ranges are defined for the subnets.

# Create VPC
gcloud services enable compute.googleapis.com

gcloud compute networks create my-vpc \
    --subnet-mode=custom \
    --bgp-routing-mode=regional

Then we have to create a Serverless VPC Access connector that allows Cloud functions (and other Serverless resources) to connect with a VPC.

# Create a Serverless VPC Access connectors 
gcloud services enable vpcaccess.googleapis.com

gcloud compute networks vpc-access connectors create functions-connector \
    --network my-vpc \
    --region us-central1 \
    --range 10.8.0.0/28

Before we can use our functions-connector we have to grant the appropriate permissions to the Cloud Functions service account, so the Cloud Functions will be able to connect to our functions-connector.

# Grant Permissions 
export PROJECT_ID=$(gcloud config list --format 'value(core.project)')
export PROJECT_NUMBER=$(gcloud projects list --filter="$PROJECT_ID" --format="value(PROJECT_NUMBER)")

gcloud projects add-iam-policy-binding $PROJECT_ID \
--member=serviceAccount:service-$PROJECT_NUMBER@gcf-admin-robot.iam.gserviceaccount.com \
--role=roles/viewer

gcloud projects add-iam-policy-binding $PROJECT_ID \
--member=serviceAccount:service-$PROJECT_NUMBER@gcf-admin-robot.iam.gserviceaccount.com \
--role=roles/compute.networkUser

Ok, we have the connector and the permissions, let's configure our Cloud Function to use the connector.

# Configurate the connector
gcloud functions deploy testIP \
    --runtime python37 \
    --entry-point test_ip \
    --trigger-http \
    --allow-unauthenticated \
    --vpc-connector functions-connector \
    --egress-settings all

If you make a request to our Cloud Function you will see this message: "Error: could not handle the request" that's because our VPC doesn't have any exit to the internet.

In order to be accessible to the outside world we have to:

Reserve a static IP.
Configure a Cloud Router to route our network traffic.
Create a Cloud Nat to allow our instances without external IP to send outbound packets to the internet and receive any corresponding established inbound response packets (aka via static IP).

# Reserve static IP
gcloud compute addresses create functions-static-ip \
    --region=us-central1

gcloud compute addresses list
# NAME                 ADDRESS/RANGE  TYPE      PURPOSE  NETWORK  REGION       SUBNET  STATUS
# functions-static-ip  34.72.171.164  EXTERNAL                    us-central1          RESERVED

We have our static IP! 34.72.171.164

# Creating the Cloud Router
gcloud compute routers create my-router \
    --network my-vpc \
    --region us-central1

# Creating Cloud Nat
gcloud compute routers nats create my-cloud-nat-config \
    --router=my-router \
    --nat-external-ip-pool=functions-static-ip \
    --nat-all-subnet-ip-ranges \
    --enable-logging

Awesome! now let's try our Cloud Functions with a new request

curl https://us-central1-your-project.cloudfunctions.net/testIP
# {"ip": "34.72.171.164"} (our static IP!)

Yay! everything is working :) a little recap:

We have deployed a simple Cloud Function (HTTP).
Created a VPC to provide networking functionalities to our Cloud Function.
Created a Serverless VPC Access connector to allow our Cloud Function to use VPC functionalities (like use IPs for example).
Granted permissions to the Cloud Functions Service Account to use network resourcing.
Configured the Cloud Function to use the Serverless VPC Access connector and redirect all the outbound request through the VPC
Reserved a static IP.
Created a Cloud Router to route our network traffic.
An finally create a Cloud Nat to communicate with the outside world.

Hope this post helps you and let me know if you have any questions or recommendations.

Source code: https://github.com/AlvarDev/functions-static-ip