DEV Community: Alex Mammay

Effective Go on GCP: Lean containers with ko on cloud build

Alex Mammay — Thu, 02 Sep 2021 23:56:00 +0000

What is ko?

ko is a technology that allows for building simple go applications into a container.

As long as the application does not need any os level package dependencies, ko is an easy and streamlined way to create
simple go applications as container. To read more check out the github repo for ko.

Using ko on gcp

We will focus on using cloud build, gcr and cloud run for our usage of ko in this example. There is a couple extra steps
involved in being able to use ko on cloud build, but the process should be straight forward.

Using the cloud-builders-community repo, we will grab the ko builder and run it through our projects cloud build
pipeline. I have this step in my makefile

.PHONY: setup-ko-gcr
setup-ko-gcr:
    git clone git@github.com:GoogleCloudPlatform/cloud-builders-community.git --depth=1;
    gcloud builds submit ./cloud-builders-community/ko --config=./cloud-builders-community/ko/cloudbuild.yaml
    rm -rf ./cloud-builders-community

Now when this is done, we will have the ko builder available to us under our private gcr repo
of gcr.io/[project-id]/ko.

Next we will look at our applications cloudbuild.yaml file.

steps:
  #  using our ko builder we will build our application that lives in ./cmd/ko
  - name: gcr.io/$PROJECT_ID/ko
    entrypoint: /bin/sh
    env:
      - 'KO_DOCKER_REPO=gcr.io/$PROJECT_ID'
    # we write the result of ko publish to a txt file so we can persist the variable between steps
    args:
      - -c
      - |
        echo $(/ko publish --preserve-import-paths ./cmd/ko) > ./ko_container.txt || exit 1

  # Deploy container image to Cloud Run
  - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
    entrypoint: /bin/bash
    args:
      - -c
      - |
        gcloud run deploy ko \
        --image=$(cat ./ko_container.txt) \
        --region=us-central1 \
        --platform=managed

From us running gcloud builds submit --config=./cmd/ko/cloudbuild.yaml we will have our image uploaded to gcr and
deployed to cloud run!

You can view the full reference repo here https://github.com/amammay/effectivecloudrun

Effective Go on Cloud Run: Graceful Application Shutdown

Alex Mammay — Tue, 24 Aug 2021 01:45:00 +0000

On gcp cloud run we have the opportunity to gracefully shutdown our application in case gcp decides to scale down our service. With golang we can capture the SIGTERM signal that google will send us to and use that signal to gracefully
shutdown our http server. You can read more about that here

Let us take a look at the go code for handling this.

Source code -> https://github.com/amammay/effectivecloudrun

Signal Handling

First we create a base context, this is the context that we will use for application level dependencies (db connection as an example)

// create our base context to work with
ctx, cancelFunc := context.WithCancel(context.Background())
defer cancelFunc()

Now we setup our http server with a base context extraction function

httpServer := &http.Server{
    Addr:    ":" + port,
    Handler: mux,
    // we register our base context function, this will be the first piece of context added to all incoming calls
    // if this context where to be cancelled, it would cancel all subsequent context driven functions therefore to
    // allow for a clean and mostly graceful disconnect
    BaseContext: func (listener net.Listener) context.Context { return ctx },
}
// upon shutdown cancel our base context
httpServer.RegisterOnShutdown(cancelFunc)

This will go ahead and allow us to capture one of two signals, os.Interrupt (SIGINT) for local development, and SIGTERM for when gcp wants to shut down an instance of our service.

// setup our shutdown signal
shutdown := make(chan os.Signal, 1)

signal.Notify(
    shutdown,
    os.Interrupt,    // Capture ctrl + c events (SIGINT)
    syscall.SIGTERM, // Capture actual sig term event (kill command).
)

The final piece of the puzzle to go ahead and wait for that signal on a separate go routine and attempt a graceful shutdown. In this example we will give it 9 seconds to cancel all ongoing context driven operations, if it happens to clock out after 9 seconds, the httpServer.Shutdown call will be forced to error out.


// setup our errgroup is listen for shutdown signal, from there attempt to shutdown our http server and capture any errors during shutdown
g, ctx := errgroup.WithContext(ctx)
g.Go(func () error {
    // on an seperate go routine we will wait and listen for our shutdown events
    o := <-shutdown
    log.Printf("sig: %s - starting shutting down sequence...", o)
    // we need to use a fresh context.Background() because the parent ctx we have in our current scope will be cancelled during the Shutdown method call
    graceFull, cancel := context.WithTimeout(context.Background(), 9*time.Second)
    defer cancel()
    // Shutdown the server with a timeout
    if err := httpServer.Shutdown(graceFull); err != nil {
        return fmt.Errorf("httpServer.Shutdown(): %w", err)
    }
    log.Printf("server has shutdown gracefully")
    return nil
})

log.Printf("starting server on %q", httpServer.Addr)
if err := httpServer.ListenAndServe(); err != http.ErrServerClosed {
    return fmt.Errorf("httpServer.ListenAndServe(): %v", err)
}
return g.Wait()

Real use cases

With the above code tied together we will showcase how it performs and how it affects the lifecycle of the application
in regards to handling ongoing operations.

What happens if you where not using the request context for an upstream network/context driven call? In this example we
are using a fresh background context. To see what happens we do the following, start server, make curl request to http://localhost:8080/noncancelablerequest and immediately send a ctrl + c event to our application.

mux.HandleFunc("/noncancelablerequest", func (writer http.ResponseWriter, request *http.Request) {

    log.Println("starting work")
    req, err := http.NewRequestWithContext(context.Background(), http.MethodGet, "https://httpbin.org/delay/10", nil)
    if err != nil {
        http.Error(writer, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
        return
    }
    _, err = http.DefaultClient.Do(req)
    if err != nil {
        log.Printf("http.DefaultClient.Do: %v", err)
        http.Error(writer, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
        return
    }
    fmt.Fprintf(writer, "<h1> hello world <h1/>")
})

Produces the following result.

2021/08/23 21:29:45 starting server on ":8080"
2021/08/23 21:29:50 starting work
2021/08/23 21:29:51 sig: interrupt - starting shutting down sequence...
2021/08/23 21:30:00 run(): httpServer.Shutdown(): context deadline exceeded

Let's fix that to work correctly now.

mux := http.NewServeMux()
mux.HandleFunc("/cancelablerequest", func (writer http.ResponseWriter, request *http.Request) {
    log.Println("starting work")

    // very important we are using the request.Context()
    // since we are using the requests' context which inherits from our context generator function
    // if we SIGTERM the server, this upstream api call will get canceled
    req, err := http.NewRequestWithContext(request.Context(), http.MethodGet, "https://httpbin.org/delay/10", nil)
    if err != nil {
        http.Error(writer, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
        return
    }
    _, err = http.DefaultClient.Do(req)
    if err != nil {
        log.Printf("http.DefaultClient.Do: %v", err)
        http.Error(writer, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
        return
    }
    fmt.Fprintf(writer, "<h1> hello world <h1/>")
})

The result of running our http server and then sending a curl request to our endpoint of http://localhost:8080/cancelablerequest and then immediately hitting ctrl + c will go ahead and start the shutdown sequence and finish gracefully.

2021/08/23 21:26:41 starting server on ":8080"
2021/08/23 21:26:47 starting work
2021/08/23 21:26:49 sig: interrupt - starting shutting down sequence...
2021/08/23 21:26:49 http.DefaultClient.Do: Get "https://httpbin.org/delay/10": context canceled
2021/08/23 21:26:49 server has shutdown gracefully

The full reference source code is

package main

import (
    "context"
    "fmt"
    "golang.org/x/sync/errgroup"
    "log"
    "net"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"
)

func main() {
    if err := run(); err != nil {
        log.Fatalf("run(): %v", err)
    }
}

func run() error {

    mux := http.NewServeMux()
    mux.HandleFunc("/cancelablerequest", func(writer http.ResponseWriter, request *http.Request) {

        log.Println("starting work")
        req, err := http.NewRequestWithContext(request.Context(), http.MethodGet, "https://httpbin.org/delay/10", nil)
        if err != nil {
            http.Error(writer, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
            return
        }
        _, err = http.DefaultClient.Do(req)
        if err != nil {
            log.Printf("http.DefaultClient.Do: %v", err)
            http.Error(writer, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
            return
        }
        fmt.Fprintf(writer, "<h1> hello world <h1/>")
    })

    mux.HandleFunc("/noncancelablerequest", func(writer http.ResponseWriter, request *http.Request) {

        log.Println("starting work")
        req, err := http.NewRequestWithContext(context.Background(), http.MethodGet, "https://httpbin.org/delay/10", nil)
        if err != nil {
            http.Error(writer, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
            return
        }
        _, err = http.DefaultClient.Do(req)
        if err != nil {
            log.Printf("http.DefaultClient.Do: %v", err)
            http.Error(writer, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
            return
        }
        fmt.Fprintf(writer, "<h1> hello world <h1/>")
    })


    // create our base context to work with
    ctx, cancelFunc := context.WithCancel(context.Background())
    defer cancelFunc()

    port := os.Getenv("PORT")
    if port == "" {
        port = "8080"
    }
    httpServer := &http.Server{
        Addr:    ":" + port,
        Handler: mux,
        // we register our base context generator, this will be the first piece of context added to all incoming calls
        // if this context where to be cancelled, it would cancel all subsequent context driven functions therefore to
        // allow for a clean and mostly graceful disconnect
        BaseContext: func(listener net.Listener) context.Context { return ctx },
    }

    // upon shutdown cancel our base context
    httpServer.RegisterOnShutdown(cancelFunc)

    // setup our shutdown signal
    shutdown := make(chan os.Signal, 1)
    signal.Notify(
        shutdown,
        os.Interrupt,    // Capture ctrl + c events (SIGINT)
        syscall.SIGTERM, // Capture actual sig term event (kill command).
    )

    // setup our errgroup is listen for shutdown signal, from there attempt to shutdown our http server and capture any errors during shutdown
    g, ctx := errgroup.WithContext(ctx)
    g.Go(func() error {
        // on an seperate go routine we will wait and listen for our shutdown events
        o := <-shutdown
        log.Printf("sig: %s - starting shutting down sequence...", o)
        // we need to use a fresh context.Background() because the parent ctx we have in our current scope will be cancelled during the Shutdown method call
        graceFull, cancel := context.WithTimeout(context.Background(), 9*time.Second)
        defer cancel()
        // Shutdown the server with a timeout
        if err := httpServer.Shutdown(graceFull); err != nil {
            return fmt.Errorf("httpServer.Shutdown(): %w", err)
        }
        log.Printf("server has shutdown gracefully")
        return nil
    })
    log.Printf("starting server on %q", httpServer.Addr)
    if err := httpServer.ListenAndServe(); err != http.ErrServerClosed {
        return fmt.Errorf("httpServer.ListenAndServe(): %v", err)
    }
    return g.Wait()
}

Effective Go on Cloud Run: The Metadata Server

Alex Mammay — Fri, 20 Aug 2021 02:01:23 +0000

Metadata Server

The metadata server is a simple way to retrieve key pieces of information in regards to the context of where your code
is running at.

The most common usages of the metadata server is retrieve some pieces of information around where your code is running
at, and the identity that is running the code. According
to here we have the following available for us

Source code here -> https://github.com/amammay/effectivecloudrun/tree/main/cmd/metadata

Project id

How to get it?

package main

import (
    "cloud.google.com/go/compute/metadata"
    "log"
)

func main() {
    // get our gcp project id
    projectID, err := metadata.ProjectID()
    if err != nil {
        log.Fatalf("metadata.ProjectID(): %v", err)
    }
    log.Printf("our gcp project id is %s", projectID)
}

How can I use it?

package main

import (
    "cloud.google.com/go/compute/metadata"
    "cloud.google.com/go/firestore"
    "context"
    "fmt"
    "log"
)

func main() {
    // get our gcp project id
    projectID, err := metadata.ProjectID()
    if err != nil {
        log.Fatalf("metadata.ProjectID(): %v", err)
    }
    log.Printf("our gcp project id is %s", projectID)
    client, err := useProjectID(context.Background(), projectID)
    if err != nil {
        log.Fatalf("useProjectID(): %v", err)
    }
    // TODO use firestore client
}

func useProjectID(ctx context.Context, projectID string) (*firestore.Client, error) {
    client, err := firestore.NewClient(ctx, projectID)
    if err != nil {
        return nil, fmt.Errorf("firestore.NewClient(): %v", err)
    }
    return client, err
}

Project number

package main

import (
    "cloud.google.com/go/compute/metadata"
    "log"
)

func main() {
    // get our numeric project id (auto generated at project creation from gcp)
    numericProjectID, err := metadata.NumericProjectID()
    if err != nil {
        log.Fatalf("metadata.NumericProjectID(): %v", err)
    }
    log.Printf("our gcp numeric project id %s", numericProjectID)
}

Cloud run geo-graphical region

package main

import (
    "cloud.google.com/go/compute/metadata"
    "log"
)

func main() {
    // get the region our code is running in
    region, err := metadata.Get("instance/region")
    if err != nil {
        log.Fatalf("metadata.Get(instance/region): %v", err)
    }
    log.Printf("our code is running in region %s", region)

}

Cloud run instance id

package main

import (
    "cloud.google.com/go/compute/metadata"
    "log"
)

func main() {
    // unique container instance id
    instanceID, err := metadata.InstanceID()
    if err != nil {
        log.Fatalf("metadata.InstanceID(): %v", err)
    }
    log.Printf("the instance of our cloud run instance is %s", instanceID)

}

Identity tokens (used to call other services that can validate an identity token)

package main

import (
    "cloud.google.com/go/compute/metadata"
    "log"
)

func main() {
    // get OIDC token to call other services that can validate an identity token
    identityToken, err := metadata.Get("instance/service-accounts/default/identity?audience=https://some.cloud.run.url.com")
    if err != nil {
        log.Fatalf("metadata.Get(instance/service-accounts/default/identity): %v", err)
    }
    log.Printf("recieve identity token that is %d bytes", len(identityToken))

}

Access Tokens (used to call GCP Api's)

package main

import (
    "cloud.google.com/go/compute/metadata"
    "log"
)

func main() {
    // get access token to call gcp api's with, can pass scopes as an query param
    accessToken, err := metadata.Get("instance/service-accounts/default/token?scopes=https://www.googleapis.com/auth/drive,https://www.googleapis.com/auth/spreadsheets")
    if err != nil {
        log.Fatalf("metadata.Get(instance/service-accounts/default/token): %v", err)
    }
    log.Printf("recieve access token that is %d bytes", len(accessToken))

}

How to get it?

You can ping the metadata server in two separate ways, using the client library or using a standard http client. For
this example we are going to use the client library for ease of use.

You can install the metadata server client library with below

go get cloud.google.com/go/compute/metadata

or you can just use the http client and add a specific header detailed here

Effective Go on GCP: Structured Logging

Alex Mammay — Thu, 19 Aug 2021 00:21:35 +0000

Structured Logging

Structured logging is a way to level up your insights into your application that is running on cloud run. Within this
example we will go over 3 different styles of logging and the incremental benefits you would get for each one.

Source code is here -> https://github.com/amammay/effectivecloudrun

Standard library logging

Our basic logging approach is as simple as it gets, we will use the golang std library logger under the log package. If
you are new to go, this is most likely what you have been using so far. It is as simple as



package main

import (
    "log"
)

func main() {
    log.Printf("hello %q", "alex")
}

When you use this logger, the output will be directed to STDERR and cloud run will pick up the log statement and add a
tiny bit of information around it, here is an example of what is produced by gcp logging when we would log
this log.Printf("hello %q! im an standard logger from the golang standard library", request.UserAgent())



{
  // text payload is our actual log message
  "textPayload": "2021/08/18 01:30:11 hello \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36\"! im an standard logger from the golang standard library",
  // insert id is automatically calculated
  "insertId": "611c62a30008cc0be6f91983",
  // resource is automatically inferred due to the log entry being written to STDERR
  "resource": {
    "type": "cloud_run_revision",
    "labels": {
      "service_name": "logging",
      "project_id": "mammay-labs",
      "revision_name": "logging-00003-wil",
      "location": "us-central1",
      "configuration_name": "logging"
    }
  },
  // timestamp is auto calculated
  "timestamp": "2021-08-18T01:30:11.576523Z",
  // instance id label is automatically added
  "labels": {
    "instanceId": "00bf4bf02dd076018e4f332daec97d8ebce3bb2a4da1d941bc9f30e3773efd19ff329eb7e3778f1461e05af587de3d79095acfe02468a6d875c5552845"
  },
  // gcp will monitor stdout and stderr
  "logName": "projects/mammay-labs/logs/run.googleapis.com%2Fstderr",
  // another timestamp that is auto calculated
  "receiveTimestamp": "2021-08-18T01:30:11.829264005Z"
}

So what we have shown for the standard library logger is the most primitive way that you log something on gcp, but you
are running this on gcp! You might as well get the most bang for your buck and beef it up with some structured logging
statements.

Basic Structured logging

Our basic structured logging implementation is very flexible with regard to how you would want to implement it. If your
team needs a large amount of flexibility/control over a logger that you would be distributing to other teammates this
would be a building block to achieve such a goal.

The gist of what you want to accomplish is to write your logs as json to STDERR. There is a schema you must follow when
writing the entries, and you can view the requirements
here gcp structured-logging

The most basic way you can implement something that would adhere to the structured logging schema would be like...



package logx

import (
    "fmt"
    "net/http"
)

func info(r *http.Request, message interface{}, projectID string) {
    get := r.Header.Get("X-Cloud-Trace-Context")
    traceID, spanID, traceSampled := deconstructXCloudTraceContext(get)
    traceID = fmt.Sprintf("projects/%s/traces/%s", projectID, traceID)
    entry := logEntry{
        Severity: "INFO",
        Message:  message,
        HttpRequest: &httpRequest{
            RequestMethod: r.Method,
            RequestUrl:    r.URL.String(),
            UserAgent:     r.UserAgent(),
            RemoteIp:      r.RemoteAddr,
            Referer:       r.Referer(),
        },
        Timestamp:    time.Now(),
        Labels:       map[string]string{"labels": "rock"},
        SpanID:       spanID,
        TraceID:      traceID,
        TraceSampled: traceSampled,
    }
    writelog(&entry)
}

func writelog(entry *logEntry) {
    if err := json.NewEncoder(os.Stderr).Encode(entry); err != nil {
        fmt.Printf("failure to write structured log entry: %v", err)
    }
}

// taken from https://github.com/googleapis/google-cloud-go/blob/master/logging/logging.go#L774
var reCloudTraceContext = regexp.MustCompile(
    // Matches on "TRACE_ID"
    `([a-f\d]+)?` +
        // Matches on "/SPAN_ID"
        `(?:/([a-f\d]+))?` +
        // Matches on ";0=TRACE_TRUE"
        `(?:;o=(\d))?`)

func deconstructXCloudTraceContext(s string) (traceID, spanID string, traceSampled bool) {
    // As per the format described at https://cloud.google.com/trace/docs/setup#force-trace
    //    "X-Cloud-Trace-Context: TRACE_ID/SPAN_ID;o=TRACE_TRUE"
    // for example:
    //    "X-Cloud-Trace-Context: 105445aa7843bc8bf206b120001000/1;o=1"
    //
    // We expect:
    //   * traceID (optional):          "105445aa7843bc8bf206b120001000"
    //   * spanID (optional):           "1"
    //   * traceSampled (optional):     true
    matches := reCloudTraceContext.FindStringSubmatch(s)

    traceID, spanID, traceSampled = matches[1], matches[2], matches[3] == "1"

    if spanID == "0" {
        spanID = ""
    }

    return
}

This accomplishes a number of things. First lets look at what the output looks like in gcp logging

Severity mapping, can be easily used for cloud monitoring to send notifications based on reported severity from log statements.
Decorate log message with http request data for advanced logging queries.
Provide structured json as a log message that can be as complex as needed. This also provides opportunity for advanced logging queries.
Trace ID + Span ID support, this will allow us to correlate log entries to cloud trace for better insight to fine-grained application behavior.



{
  // auto generated insert ID
  "insertId": "611c679300037f00b8a3abce",
  // our log message gets wrapped in json payload if you send in a json friendly message
  "jsonPayload": {
    "timestamp": "2021-08-18T01:51:15.228952576Z",
    "message": "information message"
  },
  // fill in the http request to give extra context around what our application is handling
  "httpRequest": {
    "requestMethod": "GET",
    "requestUrl": "/structuredlogger",
    "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36",
    "remoteIp": "------"
  },
  // auto generated due to the nature that we are writing to STDERR
  "resource": {
    "type": "cloud_run_revision",
    "labels": {
      "project_id": "mammay-labs",
      "configuration_name": "logging",
      "revision_name": "logging-00003-wil",
      "location": "us-central1",
      "service_name": "logging"
    }
  },
  "timestamp": "2021-08-18T01:51:15.229120Z",
  // pure info severity message, can be anything from https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#LogSeverity
  "severity": "INFO",
  "labels": {
    // custom label we added
    "labels": "rock",
    "instanceId": "00bf4bf02d728741a6c4e41e7cabe309fe6857d005b0a31df24a74786e7f909f47e95c96545a213ea7789f7016f8e2ff0b47fc2732718fe030372f2792"
  },
  "logName": "projects/mammay-labs/logs/run.googleapis.com%2Fstderr",
  // add cloud trace id 
  "trace": "projects/mammay-labs/traces/c83f888de20ad443abe581370e2a3beb",
  "receiveTimestamp": "2021-08-18T01:51:15.474376268Z",
  // add our span id 
  "spanId": "3688206018754987042",
  // notify gcp that we should sample this trace and aggregate data into cloud trace
  "traceSampled": true
}

Now within gcp here is an example of what log entries with various severities looks like.

As you can see the value-add structured logging has here is pretty awesome. In addition, adding the cloud trace and span
id's into the log statements even powers up more gcp products to work in a more insightful way. We will cover cloud
trace + open telemetry in a latter post to get even more quality of life upgrades.

Third party structured logger (Uber Zap)

Using a third party logger such as uber zap + zapdriver provides us with a really nice developer experience. We get the
best of both worlds around ease of use + structured log output. It will produce the same structured log output like
before, but just provide an easier way to configure its behavior without you having to write a whole bunch of custom
code for it.

To setup uber zap lets take a look at some code




package main

import (
    "fmt"
    "github.com/blendle/zapdriver"
    "go.uber.org/zap"
    "go.uber.org/zap/zapcore"
    "log"
    "net/http"
)

func uberzaplogger(projectID string, onGCE bool) http.HandlerFunc {

    var config zap.Config
    // if on the cloud we will use a production config
    if onGCE {
        // create our uber zap configuration
        config = zapdriver.NewProductionConfig()
        // set the min logging level to debug for this demo
        config.Level = zap.NewAtomicLevelAt(zap.DebugLevel)
    } else {
        // running locally we will use a human-readable output
        config = zapdriver.NewDevelopmentConfig()
        config.Encoding = "console"
        config.EncoderConfig.EncodeLevel = zapcore.CapitalColorLevelEncoder
    }

    // creates our logger instance
    clientLogger, err := config.Build()
    if err != nil {
        log.Fatalf("zap.config.Build(): %v", err)
    }

    wrapTraceContext := func(header string) *zap.SugaredLogger {
        traceID, spanID, sampled := deconstructXCloudTraceContext(header)
        fields := zapdriver.TraceContext(traceID, spanID, sampled, projectID)
        setFields := clientLogger.With(fields...)
        return setFields.Sugar()
    }

    return func(writer http.ResponseWriter, request *http.Request) {
        logger := wrapTraceContext(request.Header.Get("X-Cloud-Trace-Context"))
        logger.Debug("debug message")
        logger.Info("info message")
        logger.Warn("warn message")
        logger.Error("error message")
        // calling any below will cause application to quite from the behavior of uber zap
        // logger.DPanic("critical message")
        // logger.Panic("alert message")
        // logger.Fatal("EMERGENCY message")

        fmt.Fprintf(writer, "<h1> uber zap is saying hello %q", request.UserAgent())

    }
}

Overall at the end of the day my goal is to be the most productive and efficient as possible, I heavily lean to option 3
with using uber zap for all of my logging needs due to ease of configuration and integration into gcp.

Setting up open telemetry for golang on google cloud platform

Alex Mammay — Sun, 13 Jun 2021 23:19:33 +0000

If you are looking to start capturing deeper insights around the work your application is performing, open telemetry is the way to go for capturing data points while using a consistent api surface.

Here is a quick snippet for how you can setup app engine/cloud run/cloud functions/GKE with open telemetry in GCP. This will leverage using the X-Cloud-Trace-Context header that comes in from every request from the GFE.

Source code for the propagationgcp pacakge can be found here

https://github.com/amammay/propagationgcp

package main

import (
    "cloud.google.com/go/compute/metadata"
    "context"
    "fmt"
    texporter "github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/trace"
    "github.com/amammay/propagationgcp" // ablility to parse X-Cloud-Trace-Context
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/stdout"
    "go.opentelemetry.io/otel/propagation"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    "go.opentelemetry.io/otel/trace"

    "log"
    "net/http"
    "os"
    "time"
)

func main() {
    if err := run(); err != nil {
        fmt.Fprintf(os.Stderr, "run(): %v", err)
        os.Exit(1)
    }
}

func run() error {
    projectID := ""
    onGCE := metadata.OnGCE()
    if onGCE {
        // infer our project id from the metadata server
        id, err := metadata.ProjectID()
        if err != nil {
            return fmt.Errorf("metadata.ProjectID(): %v", err)
        }
        projectID = id
    }
    // init open telem, while allowing us to defer the teardown and flushing of our exporter
    tracerShutdown, err := initTracer(context.Background(), projectID, onGCE)
    if err != nil {
        return fmt.Errorf("initTracer(): %v", err)
    }
    defer tracerShutdown()

    mux := http.NewServeMux()
    mux.HandleFunc("/", func(writer http.ResponseWriter, request *http.Request) {
        ctx := request.Context()
        //create our custom span of work
        ctx, span := trace.SpanFromContext(ctx).Tracer().Start(ctx, "hello-world")
        defer span.End()
        writer.Write([]byte("hello world"))
    })

    // add http middleware for tracing
    otelWrapper := otelhttp.NewHandler(mux, "requesthandler")

    port := os.Getenv("PORT")
    if port == "" {
        port = "8080"
    }
    httpServer := http.Server{
        Addr:         ":" + port,
        Handler:      otelWrapper,
        ReadTimeout:  45 * time.Second,
        WriteTimeout: 45 * time.Second,
    }
    log.Printf("starting server on %q", httpServer.Addr)
    return httpServer.ListenAndServe()
}

func initTracer(ctx context.Context, projectID string, onGCE bool) (func(), error) {
    var tpos []sdktrace.TracerProviderOption
    // if our code is running on GCP (cloud run/functions/app engine/gke) then we will export directly to cloud tracing
    if onGCE {
        exporter, err := texporter.NewExporter(texporter.WithProjectID(projectID))
        if err != nil {
            return nil, fmt.Errorf("texporter.NewExporter(): %v", err)
        }
        tpos = append(tpos, sdktrace.WithBatcher(exporter))
    } else {
        exporter, err := stdout.NewExporter(stdout.WithPrettyPrint())
        if err != nil {
            return nil, fmt.Errorf("stdout.NewExporter(): %v", err)
        }
        tpos = append(tpos, sdktrace.WithBatcher(exporter))
    }

    tp := sdktrace.NewTracerProvider(tpos...)

    otel.SetTracerProvider(tp)

    propagator := propagation.NewCompositeTextMapPropagator(
        propagation.TraceContext{},
        propagation.Baggage{},
        propagationgcp.HTTPFormat{}, //adding this propagator allows for extracting X-Cloud-Trace-Context
    )
    otel.SetTextMapPropagator(propagator)

    return func() {
        err := tp.ForceFlush(ctx)
        if err != nil {
            log.Printf("tracerProvider.ForceFlush(): %v", err)
        }
    }, nil
}

How I write and test Go HTTP services after 1 year

Alex Mammay — Wed, 05 May 2021 01:42:07 +0000

Had the inspiration to write this after reading https://pace.dev/blog/2018/05/09/how-I-write-http-services-after-eight-years.html

Figured it would be fun to take what i have learned so far from others, and my own self and write it out as a point of reference. So props to all that i have learned from so far! You guys rock!

Reference source code found here on github

Rest api app structure

Supporting resources

https://pace.dev/blog/2018/05/09/how-I-write-http-services-after-eight-years.html

As a note this is a somewhat opinionated section...

Server struct

App server level dependencies are fields on our server struct (think something like a database, or router).

package main

type server struct {
    router    *mux.Router
    firestore *firestore.Client
}

Main func just calls our run function, so we can handle fatal errors in a nice, easy to consume manner.

package main

func main() {
    err := run()
    if err != nil {
        fmt.Fprintf(os.Stderr, "run(): %w\n", err)
                os.exit(1)
    }
}

// run is a nice function that does all our server setup and starts listening for requests
func run() error {
    port := os.Getenv("PORT")
    if port == "" {
        port = "8080"
    }

    client, err := firestore.NewClient(context.Background(), "TODO-PROJECT")
    if err != nil {
        return fmt.Errorf("firestore.NewClient(): %w", err)
    }
    s := newServer(client)

    server := http.Server{
        Addr:         ":" + port,
        Handler:      s,
        ReadTimeout:  30 * time.Second,
        WriteTimeout: 30 * time.Second,
    }
    return server.ListenAndServe()
}

Our server implements the http.Handler interface that way we could easily swap router implementations.

package main

func (s *server) ServeHTTP(writer http.ResponseWriter, request *http.Request) {
    s.router.ServeHTTP(writer, request)
}

Handlers/Middleware

Our handlers just hang off the server struct as a way to provide a closure for a handler func.

package main

func (s *server) handleFindDog(someService *dogs.DogService) http.HandlerFunc {
    //define our request/response needed structs within the closure 
    type DogTypesResponse struct {
        Dogs []*dogs.Dog `json:"dogs"`
    }

    // you could do some setup processing here as well
    // this would only be executed while the server is booting, and will be guaranteed to be finished before receiving requests.
    serviceConfig := getServiceConfig()

    return func(w http.ResponseWriter, r *http.Request) {
        //...Do request processing, maybe use the DogService we have access to as well :)
        response := &DogTypesResponse{Dogs: byType}
        s.respond(w, response, http.StatusOK)
    }
}

In addition, middleware is just regular old go code, nothing special here. Just do our middleware processing and either
end the request chain or continue to the next handler.

package main

func (s *server) isAllowed(next http.HandlerFunc) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        // if user is not allowed, then just send a 404
        if !userAllowed(r) {
            http.NotFound(w, r)
            return
        }
        // otherwise process the request to the next function
        next(w, r)
    }
}

Applying the middleware would be something as easy as

package main

// apply our middleware
r.HandleFunc("/find", s.isAllowed(s.handleFindDog(dogService))).Methods(http.MethodGet)

Routing

Our requests routing is defined all withing a routes method that hangs off our server struct, that way you can easily
identify how all the routes are structured for our application

package main

// as a fyi we are using github.com/gorilla/mux as our router
func (s *server) routes() {

    dogService := dogs.NewDogService(s.firestore)

    func(r *mux.Router) {
        r.HandleFunc("/find", s.handleFindDog(dogService)).Methods(http.MethodGet)
        r.HandleFunc("/{dogID}", s.handleGetDog(dogService)).Methods(http.MethodGet)
        r.HandleFunc("", s.handleCreateDog(dogService)).Methods(http.MethodPost)
    }(s.router.PathPrefix("/dogs").Subrouter())

}

Logging

Using one of these products should get the job done

Testing

Supporting resources

https://testwithgo.com/

Testing is a very important part to the development of an application. Knowing what kind of testing to do, at what time
can help boost your productivity short term and long term.

The three main types of testing we will focus on are

Unit Testing - Testing the smallest unit of code, mocking all external deps.
Integration Testing - Testing how code integrates with 3rd party services (database as an example)
Functional Testing - Testing how your end user would interact with your application all together.

Now before we dive in to testing tips and common things you can do to help your testing journey, we will just go over
some Opinionated rules that i generally follow.

Naming Conventions

Generally i follow this article for how i name my
files/variables/functions and so on for testing.

The main bullet points that i think are the most useful would be the following

General File/Package naming conventions.

xxx_test.go is the file name for a test
example_xxx_test.go is the file name for examples
Tests within a package that is non-executable should most likely prefer doing black box testing. ( ie. package dog_test)
- The only reason for this is for the sake of testing an api layer that closely resonates to other consumers of that package.
- In addition, you should use your best judgment for the statements above, and feel free to diverge where you seem its necessary or required

Function Naming Conventions

func TestXxx(*testing.T) this is at a mininum enforce by the go tool
A common pattern is func TestStruct_Method_Params(t *testing.T)

Now all together!

Go Code

package dogs

import "fmt"

type Dog struct {
    Name string
    Age  int
}

func NewDog(name string, age int) Dog {
    return Dog{Name: name, Age: age}
}

func (d Dog) GetFormattedDesc() string {
    return fmt.Sprintf("%s is %d years old", d.Name, d.Age)
}

Test Code

package dogs

import (
    "testing"
)

func TestDog_GetFormattedDesc(t *testing.T) {
    type fields struct {
        Name string
        Age  int
    }
    tests := []struct {
        name   string
        fields fields
        want   string
    }{
        //Table driven test
        {name: "Properly formatted name", want: "Oscar is 1 years old", fields: fields{Name: "Oscar", Age: 1}},
        //add as many test cases you want :)
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            d := Dog{
                Name: tt.fields.Name,
                Age:  tt.fields.Age,
            }
            if got := d.GetFormattedDesc(); got != tt.want {
                t.Errorf("GetFormattedDesc() = %v, want %v", got, tt.want)
            }
        })
    }
}

Variable Naming Conventions

standard syntax for failed assertion would be

Regular Failed Assertion
t.Errorf("FunctionCall(%q) = %s; want %s",arg,got,want)

Error Handling
t.Errorf("FunctionCall() = %v; want nil",err)


package dogs

import "testing"

func TestThing(t *testing.T) {

    // any special args that you would to showcase
    arg := "foo"
    // call the function/method and store it as the result
    got := SomeFuncCall(arg)
    // the expected result
    want := "some other thing"

    //assertion
    if got != want {
        t.Errorf("Thing(%q) = %s; want %s", arg, got, want)
    }

}

Integration Testing

Before we dive into integration testing examples some important notes that I think is important to call out.

Peter Bourgon said it
best here

If for any reason the tests need extra OS level setup done prior to them running, skip the tests, and print out some
useful piece of information around why it was skipped. This can help get new developers get up and running with a
project alot easier.

Reasoning is that once a developer clones the code base and runs a go test ./... from the root of the repo, they will
immediately see what else the tests need to fully run, sort of like self documenting the expected dependencies of the
system right of the bat!

For an example of integration testing we will be using test containers to
provide implementations of our dependent services.


package dogs_test

import (
    "context"
    "github.com/amammay/gotoproduction/dogs"
    "github.com/amammay/gotoproduction/internal"
    "github.com/matryer/is"
    "github.com/testcontainers/testcontainers-go"
    "testing"
)

// integration testing a service with a db interaction
func TestDogService(t *testing.T) {

    // skip test if docker is not available, very important that go test always work!
    testcontainers.SkipIfProviderIsNotHealthy(t)
    ctx := context.Background()

    // set up out testing container
    fsContainer, err := internal.CreateFirestoreContainer(ctx)
    if err != nil {
        t.Fatalf("createFirestoreContainer() err = %v; want nil", err)
    }

    // run a clean up method to terminate the container when this test and all sub tests are completed
    t.Cleanup(func() {
        err := fsContainer.Terminate(ctx)
        if err != nil {
            t.Fatalf("fsContainer.Terminate() err = %v; want nil", err)
        }
    })

    // cool way to dynamically allocate an endpoint for us to use, we could have multiple integration test running in parallel that would never have a port collision
    endpoint, err := fsContainer.Endpoint(ctx, "")
    if err != nil {
        t.Errorf("fsContainer.Endpoint() err = %v; want nil", err)
    }

    // home made firestore testing client that has a util method for clearing all data
    fsClient := internal.NewFirestoreTestingClient(ctx, t, endpoint)

    service := dogs.NewDogService(fsClient.Client)
    t.Run("Create", testDogService_CreateDog(service, fsClient))
    t.Run("Find By Type", testDogService_FindDogByType(service, fsClient))
    t.Run("GetDog By ID", testDogService_GetDogByID(service, fsClient))
}

func testDogService_CreateDog(ds *dogs.DogService, fsClient *internal.FsTestingClient) func(t *testing.T) {
    return func(t *testing.T) {
        // test our service
        ...
    }
}

func testDogService_FindDogByType(ds *dogs.DogService, fsClient *internal.FsTestingClient) func(t *testing.T) {
    return func(t *testing.T) {
        // test our service
        ...
    }
}

func testDogService_GetDogByID(ds *dogs.DogService, fsClient *internal.FsTestingClient) func(t *testing.T) {
    return func(t *testing.T) {
        // test our service
        ...
    }
}

A bit more about testing containers

Leverage testing containers for easily spinning up external resources needed for testing

go get -u github.com/testcontainers/testcontainers-go


package internal

import (
    "cloud.google.com/go/firestore"
    "context"
    "fmt"
    "github.com/testcontainers/testcontainers-go"
    "github.com/testcontainers/testcontainers-go/wait"
)

// CreateFirestoreContainer will spin up a new firestore emulator container
func CreateFirestoreContainer(ctx context.Context) (testcontainers.Container, error) {
    req := testcontainers.ContainerRequest{
        Image:        "ridedott/firestore-emulator:latest",
        ExposedPorts: []string{"8080/tcp"},
        // An example for waiting for some endpoint in the container to be healthy before proceeding. 
        WaitingFor: wait.ForHTTP("/").WithPort("8080"),
        // An example for waiting for a certain log to be produced from the container
        WaitingFor: wait.ForLog("Dev App Server is now running."),
        // combining multiple strategies together
        WaitingFor: wait.ForAll(wait.ForHTTP("/").WithPort("8080"), wait.ForLog("Dev App Server is now running.")),
    }
    fsContainer, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
        ContainerRequest: req,
        Started:          true,
    })
    if err != nil {
        return nil, fmt.Errorf("testcontainers.GenericContainer(): %w", err)
    }

    return fsContainer, nil
}

Functional Testing

So from the functional requirements of our system, we can say that we have a web service that consumes and produces json
for some crud'y operations for our dogs db.

The approach for the functional test is very similar to our integration test in terms of test setup.

Our main course of divergence is that instead of creating an instance of our service, we are working against our server
struct.

We are able to test our routing logic since our app server implements the ServeHTTP interface, and we invoke that with
our test recorder and test requests. In addition, we are able to test that our handler is processing the correct
payloads, and returning the correct json responses.

There are a couple different ways to approach the functional tests for http services, but that is for a different
segment.


package main

import (
    "bytes"
    "context"
    "encoding/json"
    "github.com/amammay/gotoproduction/dogs"
    "github.com/amammay/gotoproduction/internal"
    "github.com/matryer/is"
    "github.com/testcontainers/testcontainers-go"
    "io"
    "net/http"
    "net/http/httptest"
    "strings"
    "testing"
)

// more of a functional style of test that tests the rest endpoint + dog api layer
func Test_server_dogs(t *testing.T) {

    // skip test if docker is not available, very important that go test always work!
    testcontainers.SkipIfProviderIsNotHealthy(t)
    ctx := context.Background()

    // set up out testing container
    fsContainer, err := internal.CreateFirestoreContainer(ctx)
    if err != nil {
        t.Fatalf("createFirestoreContainer() err = %v; want nil", err)
    }

    // run a clean up method to terminate the container when this test and all sub tests are completed
    t.Cleanup(func() {
        err := fsContainer.Terminate(ctx)
        if err != nil {
            t.Fatalf("fsContainer.Terminate() err = %v; want nil", err)
        }
    })

    // cool way to dynamically allocate an endpoint for us to use, we could have multiple integration test running in parallel that would never have a port collision
    endpoint, err := fsContainer.Endpoint(ctx, "")
    if err != nil {
        t.Errorf("fsContainer.Endpoint() err = %v; want nil", err)
    }

    // home made firestore testing client that has a util method for clearing all data
    fsClient := internal.NewFirestoreTestingClient(ctx, t, endpoint)
    // construct a instance of our server application
    s := newServer(fsClient.Client)
    t.Run("create dog handler", test_handleCreateDog(s, fsClient))
    t.Run("get dog handler", test_handleGetDog(s, fsClient))
    t.Run("find dog handler", test_handleFindDog(s, fsClient))
    t.Run("find dog handler, no dogs found", test_handleFindDog_nonFound(s, fsClient))
}

func test_handleCreateDog(s *server, fsClient *internal.FsTestingClient) func(t *testing.T) {
    type dogCreateReq struct {
        Name string `json:"name"`
        Type string `json:"type"`
    }
    return func(t *testing.T) {
        fsClient.ClearData(t)
        is := is.New(t)

        dogReq := &dogCreateReq{
            Name: "Oscar",
            Type: "Golden Doodle",
        }
        jsonBytes, err := json.Marshal(dogReq)
        is.NoErr(err) // json.Marshal error
        buffer := bytes.NewBuffer(jsonBytes)

        request := httptest.NewRequest(http.MethodPost, "/dogs", buffer)
        recorder := httptest.NewRecorder()
        s.ServeHTTP(recorder, request)

        result := recorder.Result()
        is.Equal(result.StatusCode, http.StatusOK) // correct status code set
        body, err := io.ReadAll(result.Body)
        is.NoErr(err) // io.ReadAll error

        is.True(strings.Contains(string(body), `"dog_id":`)) // verify we got back an ID

    }
}

func test_handleGetDog(s *server, fsClient *internal.FsTestingClient) func(t *testing.T) {
    return func(t *testing.T) {
        ...
    }

}

func test_handleFindDog(s *server, fsClient *internal.FsTestingClient) func(t *testing.T) {
    return func(t *testing.T) {
        ...
    }

}

func test_handleFindDog_nonFound(s *server, fsClient *internal.FsTestingClient) func(t *testing.T) {
    return func(t *testing.T) {
        ...
    }

}

Serverless microservices on Cloud Run with go and oto

Alex Mammay — Sun, 08 Nov 2020 21:02:13 +0000

source code on github

The problem?

Developing microservices that run on cloud run has been a little bit of a challenge, usually for inner project authenticated service communication you had to worry about a couple things from within the application.

- Service discovery
- Auto Authentication

You usually had to have had some sort of service discovery strategy within your application code. For authentication to leverage gcp IAM to keep your service to service communication secure, you also had to make sure to ping the metadata server to get an identity token with the correct audience set.

Even though those steps where trivial, you still had to write the code or use/create some sort of library to help mitigate the code duplication between your services.

As for some rpc framework to use to auto generate your client/server/message and stubs/definitions, the main choice was gRPC.

While gRPC is quite the feature complete rpc framework that really shines in scaling out complex systems. If you are just working with a handful of services, oto might make a solid alternative for you to get you stack up and running.

How can we improve?

To solve the service discovery and auto service authentication issue we will use runsd

As for more info around oto, well its better to just quote one of the creators of it, Mat Ryer reference blog post

rather than obfuscate the message data (humans struggle reading binary) we wanted a more user friendly JSON API which was more familiar to developers. There are a range of great packages that allow you to expose JSON services alongside the binary gRPC ones, but they work by proxying to the binary port, rather than providing a standalone solution within themselves.
So in our case, developer comfortability and familiarity (ours and our future API consumers) is more important than most of the technical arguments that you might make in favour of gRPC. Oto uses Go interfaces to describe the API, and in all honesty, we generally try to use Go for as much as we can.

runsd setup

So first we will take a look at our dockerfile to square away our runsd implementation.


FROM golang:1.15-alpine as builder

WORKDIR /workspace

# Retrieve application dependencies.
# This allows the container build to reuse cached dependencies.
# Expecting to copy go.mod and if present go.sum.
COPY go.* ./
RUN go mod download

# Copy local code to the container image.
COPY . ./

ARG build_target=client

# Build the binary.
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -v -o /app ./$build_target/


# add runsd to the image
ADD https://github.com/ahmetb/runsd/releases/latest/download/runsd /runsd

# gives runsd the correct permissions
RUN chmod +x /runsd


# bare bones distroless image
FROM gcr.io/distroless/base
COPY --from=builder /app /app

# copy runsd to our final image
COPY --from=builder /runsd /runsd

# the most important part is adding runsd before we start our app
ENTRYPOINT ["/runsd", "--", "/app"]

oto breakdown

First we will look at ./definitions/taco.go that contains our service definitions for our client and server to communicate between each other.


package definitions

// TacoService contains all knowledge around consumption of tacos
type TacoService interface {

    // EatTaco handles keeping track of eating tacos
    EatTaco(request EatTacoRequest) EatTacoResponse
}

// EatTacoRequest is the request for TacoService.EatTaco.
type EatTacoRequest struct {
    // Name is your name
    Name string
    // All of the Tacos you have consumed 🌮
    Tacos []string
}

// EatTacoResponse is the response for TacoService.EatTaco.
type EatTacoResponse struct {
    // Your current taco consumption status
    TacoConsumptionStatus string
}

Now we can run our ./generate.sh script to download our templates and generate our code. The templates get synced down to the ./templates/ directory and the auto generated code goes into ./generated/client and ./generated/server/.

Lets implement the server inside of ./server/main.go


package main

import (
    "context"
    "errors"
    "fmt"
    "github.com/amammay/otorun/generated/server"
    "github.com/pacedotdev/oto/otohttp"
    "log"
    "net/http"
    "os"
    "time"
)

func main() {

    if err := run(); err != nil {
        fmt.Fprintf(os.Stderr, "%v\n", err)
        os.Exit(1)
    }

}

func run() error {

    // creates new oto base http server
    oto := otohttp.NewServer()

    // register our taco service to the oto server
    server.RegisterTacoService(oto, &tacoService{})

    mux := http.NewServeMux()

    // map the path prefix /oto/ to the oto server and our custom middleware
    mux.Handle("/oto/", userAgentMiddleware(oto))

    // cloud run sets a port env variable that we should respect
    port := os.Getenv("PORT")
    if port == "" {
        port = "8080"
    }

    // create our http server and bind our handle to our mux
    httpServer := &http.Server{
        Handler:      mux,
        Addr:         ":" + port,
        ReadTimeout:  30 * time.Second,
        WriteTimeout: 30 * time.Second,
    }
    log.Printf("Starting server on %s", httpServer.Addr)

    return httpServer.ListenAndServe()

}

// userAgentMiddleware logs the user agent middleware for all requests that come into oto
func userAgentMiddleware(h http.Handler) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {

        if agent := r.UserAgent(); agent != "" {
            log.Printf("info: request from %s", agent)
        }

        // send the request into oto
        h.ServeHTTP(w, r)
    }

}

// tacoService implementation
type tacoService struct {
    // TODO add database
}

// EatTaco will tell you how many tacos you have consumed
func (t *tacoService) EatTaco(ctx context.Context, request server.EatTacoRequest) (*server.EatTacoResponse, error) {
    if request.Name == "" {
        return nil, errors.New("EatTacoRequest.Name is required")
    }
    if len(request.Tacos) <= 0 {
        return nil, errors.New("EatTacoRequest.Tacos is required")
    }

    return &server.EatTacoResponse{TacoConsumptionStatus: fmt.Sprintf("%s has consumed %d tacos", request.Name, len(request.Tacos))}, nil
}

and now the client inside of ./client/main.go


package main

import (
    "context"
    "fmt"
    "log"
    "net/http"
    "os"

    "cloud.google.com/go/compute/metadata"
    "github.com/amammay/otorun/generated/client"
)

func main() {

    // if we are running on cloud run we will leverage runsd for service discovery
    tacoHost := "http://tacoserver/oto/"
    if !metadata.OnGCE() {
        tacoHost = "http://localhost:8080/oto/"
    }

    // create a new client from our auto generated client
    c := client.New(tacoHost)
    c.Debug = func(s string) {
        fmt.Println(s)
    }
    // create our taco service
    tacoService := client.NewTacoService(c)

    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {

        // consume some tacos
        tacoResponse, err := tacoService.EatTaco(context.Background(), client.EatTacoRequest{
            Name:  "Sammy Sosa",
            Tacos: []string{"Chicken", "Chorizo"},
        })
        if err != nil {
            http.Error(w, err.Error(), http.StatusInternalServerError)
            return
        }
        fmt.Fprint(w, tacoResponse.TacoConsumptionStatus)
    })

    // cloud run sets a port env variable that we should respect
    port := os.Getenv("PORT")
    if port == "" {
        port = "8081"
    }

    log.Fatal(http.ListenAndServe(":"+port, nil))

}

So now we can travel over to the gcp console and start to get some items deployed.

Let's start with getting our GCP projects cloud run hash. You can do that by doing the following

Cloud run
Create Service
Service name = hello-world -> next
Deploy one revision from an existing container image = gcr.io/cloudrun/hello -> next
Up to you if you want it to be public or private and then hit create.
Grab the hash from the url of the service, it will be in the format of this https://hello-world-HASHVALUE-uc.a.run.app

Now back to our local terminal we can get everything deployed out by running cloudhash=HASHVALUE ./buildanddeploy.sh

The last item we need to do is to grant the service account that is powering the client the Cloud Run Invoker role.

Now lets test the connection


curl -H \
"Authorization: Bearer $(gcloud auth print-identity-token)" \
$(gcloud run services describe tacoclient --format 'value(status.url)')

We should get back that Sammy Sosa has consumed 2 tacos.

If you poke around the logs on the tacoserver you can see our user agent middleware printing out the runsd agent that made the proxying requests.

2020/11/08 20:50:48 info: request from runsd version=0.0.0-rc.9; Go-http-client/1.1

Happy coding!

VS Code setup for golang in 2020

Alex Mammay — Thu, 01 Oct 2020 01:40:59 +0000

My IDE setup usually change quite a bit overtime, so i figured it would be quite nice to document my current setup as a snapshot in time around the current development ecosystem that i am currently in.

My day to day tech stack usually revolves golang, and containers. I guess you could call that stack the 2020 serverless stack at this point really... Between using intellij ultimate and vscode i generally have a pretty consistent setup between either one.

Theme

Color Theme: One Dark Pro
Icon Theme: Material Icon Theme
Font: Jetbrains Mono, Font Ligatures enabled

  "workbench.colorTheme": "One Dark Pro",
  "editor.fontFamily": "'JetBrains Mono', Consolas, 'Courier New', monospace",
  "editor.fontSize": 13,
  "editor.fontLigatures": true,
  "editor.letterSpacing": 0.4,
  "editor.smoothScrolling": true,
  "workbench.iconTheme": "material-icon-theme",

Settings for go

  "go.lintOnSave": "file",
  "[go]": {
    "editor.formatOnSave": true,
    "editor.codeActionsOnSave": {
      "source.organizeImports": true
    }
  },
  "go.docsTool": "gogetdoc",
  "go.formatTool": "goimports",
  "go.useLanguageServer": true,

Extensions

Cloud Code (googlecloudtools.cloudcode) For all your serverless GCP needs (really nice way to run a cloud run like environment locally with it leveraging minikube)

Docker (ms-azuretools.vscode-docker)
GO (golang.go)
gotemplate-syntax (casualjim.gotemplate)
Golang postfix code completion (yokoe.vscode-postfix-go) This might be the most important extension to have installed. I find myself having more productive rhythm when using postfix tools across any language since i worry more about the core pieces of code and any thing else can be an afterthought.

Intellij IDEA key mapping (k--kato.intellij-idea-keybindings)

GCP Api Gateway: Firebase Authentication

Alex Mammay — Wed, 23 Sep 2020 18:51:43 +0000

Google Cloud Platform Api Gateway

github repo --> here

What is API Gateway?

As per the documentation, Api gateway is a fully managed gateway for serverless workloads.

So really at the end of the day, that boils down to a serverless gateway for you serverless api's... Man that's alot of serverless.

What does API Gateway do?

API gateway will act as the middleman between an end user and your services. You describe your services according to the OpenAPI specification,
upload the specification to api gateway center and then finally deploy the spec to a gateway. API Gateway also provides a suite of utilities, such as monitoring, logging and authentication.

When should I use API Gateway?

If you are going to be using the serverless GCP eco-system(Cloud Functions, Cloud Run, App Engine), then API Gateway will be complimentary to those serverless products.

Why should I use API Gateway?

The most important question to answer is "why", why should I use API Gateway?

Security - Your core application can be deployed and protected by GCP IAM, that way the only direct interactions with your services will be done with the api gateway.

Externalized Configuration - You have an external way to manage application authentication, service url mapping, and API documentation that lives outside the context of the application.

Less Code - Your application itself won't have to worry about validating JWT/Api Keys since those will be handled at the gateway layer, and the result will be forwarded to your application. The less code you write, the fewer bugs there will be 😬

Observability - All your performance metrics will roll up to a single, easy to view dashboard with all your KPI's such as request latency, error rates, requests per second and more.

Securing Cloud Run Services with Firebase Authentication.

First we will take a look at our openapi specification file to get an understanding of our api. It is pretty straight forward, a single endpoint named /greet that will echo back the users name but in a "greeted" format.

openapi2-run.yaml

swagger: '2.0'
info:
  # Title of the api gateway
  title: gateway
  description: Sample API on API Gateway with a Cloud Run backend
  version: 1.0.0
schemes:
  - https
produces:
  - application/json
# The cloud run service url, this could also be defined per path as well in case you have multiple cloud run services that
# make up a single gateway
x-google-backend:
  address: "YOUR-CLOUD_RUN-URL"
securityDefinitions:
  firebase:
    authorizationUrl: ""
    flow: "implicit"
    type: "oauth2"
    # Replace YOUR-PROJECT-ID with your project ID
    x-google-issuer: "https://securetoken.google.com/YOUR-PROJECT-ID"
    x-google-jwks_uri: "https://www.googleapis.com/service_accounts/v1/metadata/x509/securetoken@system.gserviceaccount.com"
    x-google-audiences: "YOUR-PROJECT-ID"
paths:
  /greet:
    get:
      summary: Greets the user
      operationId: greet
      # define that our service uses the firebase security definition
      security:
        - firebase: [ ]
      responses:
        '200':
          description: A successful response
          schema:
            type: object
            # our object look like `{name: ""}`
            properties:
              name:
                type: string
                description: The users name

Now we will take a peek at the code that will fulfill the contract of the api.

cmd/routes.go

func (s *server) routes() {
    s.router.HandleFunc("/greet", s.handleAuth(s.handleGreeting()))
}

//handleGreeting will fetch the UserInfo struct that is stored in context from our auth middleware and use that to greet the person that called our api
func (s *server) handleGreeting() http.HandlerFunc {

    type person struct {
        Name string `json:"name"`
    }

    return func(writer http.ResponseWriter, request *http.Request) {

        writer.Header().Set("Content-Type", "application/json")

        // fetch the token user object that is stored in context
        userObj := request.Context().Value(gatewayUserContext).(UserInfo)

        // greet the user 👋
        p := person{Name: fmt.Sprintf("Hello 👋 %s", userObj.Name)}
        decoder := json.NewEncoder(writer)

        if err := decoder.Encode(p); err != nil {
            http.Error(writer, err.Error(), http.StatusInternalServerError)
        }
    }
}

And just for a quick review of the Auth Middleware.

cmd/auth.go

const gatewayUserInfoHeader = "X-Apigateway-Api-Userinfo"
const gatewayUserContext = "GATEWAY_USER"

type UserInfo struct {
    Name          string   `json:"name"`
    Picture       string   `json:"picture"`
    Iss           string   `json:"iss"`
    Aud           string   `json:"aud"`
    AuthTime      int      `json:"auth_time"`
    UserID        string   `json:"user_id"`
    Sub           string   `json:"sub"`
    Iat           int      `json:"iat"`
    Exp           int      `json:"exp"`
    Email         string   `json:"email"`
    EmailVerified bool     `json:"email_verified"`
    Firebase      Firebase `json:"firebase"`
}
type Identities struct {
    GoogleCom []string `json:"google.com"`
    Email     []string `json:"email"`
}
type Firebase struct {
    Identities     Identities `json:"identities"`
    SignInProvider string     `json:"sign_in_provider"`
}

// handleAuth is a piece of middleware that will parse the gatewayUserInfoHeader from the request and add the UserInfo to the request context
func (s *server) handleAuth(h http.HandlerFunc) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {

        encodedUser := r.Header.Get(gatewayUserInfoHeader)
        if encodedUser == "" {
            http.Error(w, "User Not Available", http.StatusForbidden)
            return
        }
        decodedBytes, err := base64.RawURLEncoding.DecodeString(encodedUser)
        if err != nil {
            http.Error(w, "Invalid UserInfo", http.StatusForbidden)
            return
        }
        decoder := json.NewDecoder(bytes.NewReader(decodedBytes))
        var userToken UserInfo
        err = decoder.Decode(&userToken)
        if err != nil {
            http.Error(w, "Invalid UserInfo", http.StatusForbidden)
            return
        }

        ctx := context.WithValue(r.Context(), gatewayUserContext, userToken)
        h.ServeHTTP(w, r.WithContext(ctx))

    }
}

You can see the rest of the code for the application in the github repo.

Enable API's and Create service account

Make sure you enable the following api's on your project

gcloud services enable apigateway.googleapis.com
gcloud services enable servicemanagement.googleapis.com
gcloud services enable servicecontrol.googleapis.com

Lets create our api gateway service account

gcloud iam service-accounts create api-gateway

gcloud projects add-iam-policy-binding YOUR_PROJECT_ID --member "serviceAccount:api-gateway@YOUR_PROJECT_ID.iam.gserviceaccount.com" --role "roles/run.invoker"
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID --member "serviceAccount:api-gateway@YOUR_PROJECT_ID.iam.gserviceaccount.com" --role "roles/iam.serviceAccountUser"

Build and Deployment

Since we are going to be using cloud build to roll everthing out, lets get our container built and deployed with gcloud builds submit

cloudbuild.yaml

steps:

  # Run the docker build
  - name: 'gcr.io/cloud-builders/docker'
    args: [ 'build', '-t', 'gcr.io/$PROJECT_ID/greeter', '.' ]

  # push the docker image to the private GCR registry
  - name: 'gcr.io/cloud-builders/docker'
    args: [ 'push', 'gcr.io/$PROJECT_ID/greeter' ]

  # deploy to cloud run
  - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
    entrypoint: gcloud
    args: [ 'run', 'deploy', 'greeter', '--image', 'gcr.io/$PROJECT_ID/greeter', '--region', 'us-central1', '--platform', 'managed', '--no-allow-unauthenticated' ]

images:
  - 'gcr.io/$PROJECT_ID/greeter'

Now that we have the cloud service deployed, we just need to get our cloud run url gcloud run services describe greeter --format 'value(status.url)'
and to verify its secure we can try to curl the endpoint curl $(gcloud run services describe greeter --format 'value(status.url)') and we should get a 403.

The next step is to take that url from our endpoint and plug it into our api spec as the address for the x-google-backend

# The cloud run service url, this could also be defined per path as well in case you have multiple cloud run services that
# make up a single gateway
x-google-backend:
  address: "YOUR-CLOUD_RUN-URL"

Now we are turning to the home stretch, we just need to deploy our gateway/config


# create api config
gcloud beta api-gateway api-configs create echoconf \
  --api=gateway --openapi-spec=openapi2-run.yaml \
  --backend-auth-service-account=api-gateway@YOUR_PROJECT_ID.iam.gserviceaccount.com


# create gateway with config
gcloud beta api-gateway gateways create gateway \
  --api=gateway --api-config=echoconf \
  --location=us-central1


#get hostname from gateway
gcloud beta api-gateway gateways describe gateway \
  --location=us-central1 --format 'value(defaultHostname)'

Now if we curl the gateway endpoint with

curl "https://$(gcloud beta api-gateway gateways describe gateway --location=us-central1 --format 'value(defaultHostname)')/greet"

we will get a 401 since we didn't attach a firebase identity token to request.

But once you attach the token to the request as either a query param of access_token or as an Authorization header with bearer token

curl "https://$(gcloud beta api-gateway gateways describe gateway --location=us-central1 --format 'value(defaultHostname)')/greet?access_token=ACCESS_TOKEN"

We will get back our greeted message! { "name": "Hello 👋 Alex Mammay" }

Firebase Emulator Suite: Running golang/python/java firestore event triggers locally with the emulator

Alex Mammay — Tue, 08 Sep 2020 20:35:20 +0000

Purpose

The firebase team has done a really awesome job at providing a rich and feature complete environment for running and testing out various products all locally (firestore emulator, real time database emulator, pub/sub emulator).

Most of the support/docs/guides out there mainly cover using the firebase-tools cli and using Node.Js.

This post is mainly to show that it's possible to recreate a nice local development setup with the other languages that GCP Cloud Functions supports, such as Java, Python, and Go.

for the purpose of this write up, we will be using golang

Setting up the emulator

Grab the latest firestore emulator with using firebase-tools cli by running firebase setup:emulators:firestore.

Invoke the help menu of the jar with java -jar ~/.cache/firebase/emulators/cloud-firestore-emulator-v1.11.7.jar --help, as of the time i am writing this the current version is v1.11.7 so thats what this sample will be with.

Run the emulator with java -jar ~/.cache/firebase/emulators/cloud-firestore-emulator-v1.11.7.jar --functions_emulator localhost:6000 this will set the callback url for functions events to localhost:6000 and thats where we will be running our local function at.

Register our function resource with the firestore emulator. We want to watch for all document writes to the root collection of skills/{id} and the name of our Function is called "WriteSkills" .

curl --location --request PUT 'http://localhost:8080/emulator/v1/projects/dummy/triggers/WriteSkills' \
--header 'Content-Type: application/json' \
--data-raw '{
   "eventTrigger": {
       "resource": "projects/dummy/databases/(default)/documents/skills/{id}",
       "eventType": "providers/cloud.firestore/eventTypes/document.write",
       "service": "firestore.googleapis.com"
   }
}'

As a quick schema reference you can use the following.

curl --location --request PUT 'http://[HOST:PORT]/emulator/v1/projects/[PROJECT_ID]/triggers/[FUNCTION_NAME]' \
--header 'Content-Type: application/json' \
--data-raw '{
    "eventTrigger": {
        "resource": "projects/[PROJECT_ID]/databases/(default)/documents/[RESOURCE_PATH]",
        "eventType": [EVENT_TYPE],
        "service": "firestore.googleapis.com"
    }
}'

the event types for firestore can be

"providers/cloud.firestore/eventTypes/document.create"
"providers/cloud.firestore/eventTypes/document.update"
"providers/cloud.firestore/eventTypes/document.delete"
"providers/cloud.firestore/eventTypes/document.write"

Our function code for golang will be


package main

import (
    "cloud.google.com/go/firestore"
    "cloud.google.com/go/functions/metadata"
    "context"
    "fmt"
    "github.com/GoogleCloudPlatform/functions-framework-go/funcframework"
    "log"
    "time"
)

type FirestoreEvent struct {
    OldValue   FirestoreValue `json:"oldValue"`
    Value      FirestoreValue `json:"value"`
    UpdateMask struct {
        FieldPaths []string `json:"fieldPaths"`
    } `json:"updateMask"`
}

type FirestoreValue struct {
    CreateTime time.Time   `json:"createTime"`
    Fields     interface{} `json:"fields"`
    Name       string      `json:"name"`
    UpdateTime time.Time   `json:"updateTime"`
}

func main() {
    // create a firestore write event in a go routine that will be ran by the time local dev server is spun up
    go func() {
        ctx := context.Background()
        firestoreClient, err := firestore.NewClient(ctx, "dummy")
        if err != nil {
            log.Fatalf("firestore.NewClient: %v", err)
        }
        defer firestoreClient.Close()

        if _, err := firestoreClient.Collection("skills").NewDoc().Create(ctx, map[string]interface{}{
            "skill":     "Running",
            "timestamp": firestore.ServerTimestamp,
        }); err != nil {
            log.Fatalf("firestoreClient.Collection.NewDoc().Set: %v", err)
        }
    }()
    funcframework.RegisterEventFunction("/functions/projects/dummy/triggers/WriteSkills", WriteSkills)
    if err := funcframework.Start("6000"); err != nil {
        panic(err)
    }
}

func WriteSkills(ctx context.Context, e FirestoreEvent) error {
    meta, err := metadata.FromContext(ctx)
    if err != nil {
        return fmt.Errorf("metadata.FromContext: %v", err)
    }
    log.Printf("Function triggered by change to: %v", meta.Resource)
    log.Printf("Old value: %+v", e.OldValue)
    log.Printf("New value: %+v", e.Value)
    return nil
}

now we will run it by invoking

export FIRESTORE_EMULATOR_HOST=localhost:8080 && go run .

and we shall see an output of

Serving function...
2020/09/08 16:16:58 Function triggered by change to: &{firestore.googleapis.com projects/dummy/databases/(default)/documents/skills/3wUTxvMAawFlJPQmxwYv  }
2020/09/08 16:16:58 Old value: {CreateTime:0001-01-01 00:00:00 +0000 UTC Fields:<nil> Name: UpdateTime:0001-01-01 00:00:00 +0000 UTC}
2020/09/08 16:16:58 New value: {CreateTime:2020-09-08 20:16:58.461011 +0000 UTC Fields:map[skill:map[stringValue:Running] timestamp:map[timestampValue:2020-09-08T20:16:58.459Z]] Name:projects/dummy/databases/(default)/documents/skills/3wUTxvMAawFlJPQmxwYv UpdateTime:2020-09-08 20:16:58.461011 +0000 UTC}

Util Time!

Finally i wrote a little util https://github.com/amammay/firebase-emu that will register your go function against the firestore emulator automatically and then add a entry into the funcframework for you.

package main

import (
    "cloud.google.com/go/firestore"
    "cloud.google.com/go/functions/metadata"
    "context"
    "fmt"
    "github.com/GoogleCloudPlatform/functions-framework-go/funcframework"
    "github.com/amammay/firebase-emu/fsemu"
    "log"
    "time"
)

func main() {

    go func() {
        ctx := context.Background()

        firestoreClient, err := firestore.NewClient(ctx, "dummy")
        if err != nil {
            log.Fatalf("firestore.NewClient: %v", err)
        }
        defer firestoreClient.Close()

        if _, err := firestoreClient.Collection("skills").NewDoc().Create(ctx, map[string]interface{}{
            "skill":     "Running",
            "timestamp": firestore.ServerTimestamp,
        }); err != nil {
            log.Fatalf("firestoreClient.Collection.NewDoc().Set: %v", err)
        }
    }()
    event := fsemu.EmuResource{ProjectId: "dummy", Address: "http://localhost:8080"}
    emuRegisters := []fsemu.EmuRegister{{
        TriggerFn:    WriteSkills,
        TriggerType:  fsemu.FirestoreOnWrite,
        ResourcePath: "skills/{id}",
    }}

        //register firestore emulator resources
    if err := event.RegisterToEmu(emuRegisters); err != nil {
        panic(err)
    }
    if err := funcframework.Start("6000"); err != nil {
        panic(err)
    }
}

type FirestoreEvent struct {
    OldValue   FirestoreValue `json:"oldValue"`
    Value      FirestoreValue `json:"value"`
    UpdateMask struct {
        FieldPaths []string `json:"fieldPaths"`
    } `json:"updateMask"`
}

type FirestoreValue struct {
    CreateTime time.Time   `json:"createTime"`
    Fields     interface{} `json:"fields"`
    Name       string      `json:"name"`
    UpdateTime time.Time   `json:"updateTime"`
}

func WriteSkills(ctx context.Context, e FirestoreEvent) error {

    meta, err := metadata.FromContext(ctx)
    if err != nil {
        return fmt.Errorf("metadata.FromContext: %v", err)
    }
    log.Printf("Function triggered by change to: %v", meta.Resource)
    log.Printf("Old value: %+v", e.OldValue)
    log.Printf("New value: %+v", e.Value)
    return nil
}

Firebase Emulator Suite: Using Firestore And Cloud Functions offline

Alex Mammay — Sun, 19 Jul 2020 16:24:15 +0000

source code can be found here --> https://github.com/amammay/firebase-emulator-workshop

Firebase Emulator Workshop

Goal of this little project is to show case some examples of how to leverage the
firebase emulator suite in accelerating your local development experience by not using your production database/triggers!

Background Information

First lets get a better understanding of the products we are going to be using and more importantly, how we can use them to easily get our app up and running locally.

Firebase

If we can get our firebase products working independently from the client app that will be a pretty quick win. This will just prove out that our configurations are looking good, and we can run the infrastructure in a silo'ed environment.

Functions

Inside of functions/index.js you will see we have two functions available, one is a firestore event trigger, and one a https.onCall Function

Firestore Event Trigger this watches for new documents created on a collection called buttonPresser, and upon that new document created
it will just add a field to the document {server: 'side'} to show that the cloud function is properly working.

exports.buttonHit = functions.firestore.document('buttonPresser/{docId}').onCreate(async (snapshot, context) => {
  await snapshot.ref.update({server: 'side'})
})

https.onCall this is an invokable function, that as long as the user has been properly authed into the application, allow them to delete all documents in buttonPresser collection.

exports.deleteAll = functions.https.onCall(async (data, context) => {
  if (!context.auth) {
    throw new functions.https.HttpsError('unauthenticated', 'No Auth provided')
  }
  const allDocs = await firestore.collection('buttonPresser').listDocuments();

  const deleter = async (ref) => await ref.delete();
  let promiseArray = [];
  for (const firestoreElement of allDocs) {
    promiseArray.push(deleter(firestoreElement));
  }
  await Promise.all(promiseArray);

})

Firestore

Our firebase database rules are pretty simple, we allow any authed users to read and create new documents into firestore, but we don't allow them to delete directly.

rules_version = '2';
service cloud.firestore {
  match /databases/{database}/documents {
    match /{document=**} {
      allow create, read: if request.auth != null;
    }
  }
}

Emulators

The emulators we are mainly using is the firestore emulator and the functions emulator for nodejs.
From the root of this repo we can run firebase emulators:start to see our emulators come online. To verify everything is talking to each other,
lets attempt to trigger our firestore event trigger in the emulator ui by going to http://localhost:4000/firestore and creating our buttonPresser collection

Upon us hitting save we should see our cloud function add the data into the document we just created

Now at this point we see the emulators are happily talking to each other and now we can shift focus over to the client application.

Client Application

Let us take a look at this regular angular application we have that is pointed to our live production database, and make it so it can be pointed to our firestore emulator suite.

Within src/environments/environment.ts lets add new a field of emulator:true and then register a new firestore configuration provider.

What the code below will do is to check if we want to use the firestore emulator,
if we do then go ahead and provide the host name and ssl config to use for talking to the emulator. The firestore emulator by default will run on port 8080, but you may want to double check your config in your firebase.json file.

src/app/app.module.ts


import {AngularFirestoreModule, SETTINGS} from "@angular/fire/firestore";

@NgModule({
  declarations: [
    ...
  ],
  imports: [
    ...
  ],
  providers: [{
    provide: SETTINGS,
    useValue: environment.emulator ? {
      host: 'localhost:8080',
      ssl: false
    } : undefined
  }],
  bootstrap: [...]
})
export class AppModule {
}

So that will get us to point to the emulator for firestore, but what about the functions emulator?

Well if we take a look at our usage of the firebase functions service we will check our env config again to see if want to use the emulator, if we do, then provide the base url for it use to talk to the emulator.

src/app/app.component.ts

public ngOnInit() {
    if (environment.emulator) {
      this.aff.useFunctionsEmulator('http://localhost:5001').then(() => console.log('Using functions emulator'))
    }
}

NOTE: The overall concept is the same for the other firebase client sdk's but the syntax will change between implementations.
The main goal is to really just tell the firebase sdk's that you want to use a different url for that service to point to.

Now All Together!

Within the root package.json you will see a script start:all you can invoke this by running npm run start:all this will spin up the emulators and run the angular project in watch mode.

Now navigate to localhost:4200 you will need to login with the google auth provider.

From there you should see some seed data already loaded into the web app.

Now if we open our dev tools and click Add Button Press we will see that we are talking to our local firestore database

Finally, lets clear all the documents by hitting Delete All

Firebase Emulator Suite: Using production data inside the emulator

Alex Mammay — Sun, 12 Jul 2020 03:04:27 +0000

So you have a lot of info in your live (prod/test) database that you want to use with the new fancy firebase emulator UI. Here is how i went upon wiring up the data.

Create storage bucket to sync into
- gsutil mb gs://[targetstoragebucket]
Export your prod database into a storage bucket.
- export the database with this command https://cloud.google.com/sdk/gcloud/reference/beta/firestore/export as a sample if you want to sync the whole database you can do gcloud beta firestore export gs://[targetstoragebucket] and you can pick and choose root level collections by doing gcloud beta firestore export gs://[targetstoragebucket] --collection-ids='customers','orders'
Create seed-data folder in your working directory mkdir seed-data
Once the export is done, we can sync it down to our filesystem. gsutil rsync -r gs://[targetstoragebucket] seed-data/
Now here is where we need to do a bit of patching, if we just try to run firebase emulators:start --only firestore --import seed-data it will error on us, so lets do a bit of cleaning up
lets cd seed-data and then mkdir firestore_export and touch firebase-export-metadata.json and then finally move our export into the firestore export folder, in my case it was mv 2020-07-12T02:45:39_47881/ firestore_export.
create content of firebase-export-metadata.json file, so lets open it in a editor and point out some important bits

{
  "version": "8.5.0",
  "firestore": {
    "version": "1.11.4",
    "path": "firestore_export",
      "metadata_file": "firestore_export/2020-07-12T02:45:39_47881/2020-07-12T02:45:39_47881.overall_export_metadata"
  }
}

make sure to note the version field (version of firebase-tools you are using, and the metadata file, this is that folder we used the mv command for.

and just as final screen shot of the important stuff in case you want to compare.

now if we boot up the emulator with firebase emulators:start --only firestore --import seed-data lets do a compare of the live database and our emulator

Woow!