DEV Community: yosan

Test with Firestore Emulator on GitHub Actions

yosan — Wed, 14 Jul 2021 21:17:22 +0000

I'm developing a web site using Next.js hosted by Cloud Functions. It uses Firestore as main database. And some Firestore-Trigger functions exist.

Now, I wanted to test my functions and firestore.rules with Firestore Emulator. Due to manage its codes in GitHub, I wanted to use GitHub Actions for test.

I tried some ways to find more simple way to achive it. As a result of various investigations, I think preparing all-in-one GitHub Actions instance and using firebase emulators:exec command is most simple way.

Here is how I setup auto testing system.

Setup local test

First of all, write tests and setup configurations to be able to run test in my local machine.

Here is what I committed.
https://github.com/yosan/tennico/pull/13/files

I'll explain some points in it.

Use firebase emulators:exec

Add this setting to package.json.

  "scripts": {
    "test": "firebase emulators:exec —only firestore 'jest --silent'"
  },

firebase emulators:exec is useful command. It downloads, launch, and stop Firebase Emulator automatically.

While executing tests, many Cloud Functions logs are shown on our console so that it's hard to read test results. So I use jest --silent.

Devide Cloud Functions' index.ts

Usually we import an index.ts which exports all of Cloud Function's implementations to wrap firebase-functions-test. But if a function which serves Next.js is contained in it, it becomes difficult to import index.ts because Next.js needs many settings to do so. Therefore, I separated Next.js function and other functions. Only import functions except Next.js one.

src/functions
├── index.ts
├── tsconfig.json
├── __tests__
├── firestore <- Not Next.js functions
└── web <- Next.js function

Mock algolia

My functions use algolia to register my Firestore document to index. Mocking alogolia client is little hard for me. This is how I mocked it.

let mockSaveObject: ReturnType<typeof jest.fn>
jest.mock('algoliasearch', () => {
  mockSaveObject = jest.fn().mockReturnValue({})
  return {
    default: () => ({
      initIndex: () => ({
        saveObject: mockSaveObject,
      }),
    }),
  }
})

Setup GitHub Actions

This is the diff of setup.
https://github.com/yosan/tennico/pull/14

Workflow Settings

I made settings referring to Cloud Functions runtime environment.

This was my first time to use GitHub Actions. I felt good because workflow settings was simple and I didn't have to write Dockerfile.

Firebase Emulator needs to project ID selection to launch and some credential is needed for it. I specified it from "GitHub > Settings > Secrets".

name: CI
on: [push]
jobs:
  test:
    runs-on: ubuntu-18.04
    env:
      FIREBASE_TOKEN: ${{ secrets.FIREBASE_TOKEN }}
      GCLOUD_PROJECT: ${{ secrets.GCLOUD_PROJECT }}
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v1
        with:
          node-version: '10.18.1'
      - uses: actions/setup-java@v1
        with:
          java-version: '15.0.2'
          java-package: jdk
          architecture: x64
      - name: Install dependencies
        run: yarn
      - name: Test
        run: yarn test --project ${GCLOUD_PROJECT}

Before I wrote this settings, I found many articles which said using Docker Compose is needed to use Firebase Emulator on GitHub Actions.

Maybe there are some difference we can do between them. But I wanted to do was only to test Cloud Functions and firestore.rules and it seemed that it would become more complex by Docker Compose. So, I did't use it.

functions.config()

If .runtimeconfig is pushed, functions.config() can return value. But we shouldn't do that if we published our code on public repository in GitHub.

I set default value to resolve runtime error. For example, when I want to refer algolia API key in Cloud Functions environment variables, I write code like this.

const client = algoliasearch(
  functions.config().algolia?.app_id ?? '',
  functions.config().algolia?.admin_key ?? ''
)

Ofcource, the client will return error when it calls Algolia API. But we can avoid it by mocking client.

Finally

It works!

Export Firebase Firestore data to BigQuery

yosan — Tue, 21 Jul 2020 12:30:09 +0000

This article is about "how to export Firestore data to BigQuery".

Use Firebase Extension if your service is before published

I haven't used yet but I think you can use this.

Firebase Extensions | Export Collections to BigQuery

This extension can sync Firestore data with BigQuery in realtime.

But if your service is already published, you have to move existing Firestore data to BigQuery by other method.

Use gcloud if your service is already published

You can use "gcloud" command (and "bq" command installed at the same time) to export it.

Create service account

Set roles refering these documents.

Command Example

When this commands are executed, BigQuery table is replaced with latest firestore data.

# Set service account key file
gcloud auth activate-service-account --key-file key.json

# Set project
gcloud config set project my-firebase-project

# Export Firestore data to Storage by collection group
gcloud firestore export gs://firestore-export-for-bq/2020-06-29T10:17:23.011+09:00 --collection-ids=user,shop,product

# Import data to BigQuery by collection group
bq load --source_format=DATASTORE_BACKUP --replace=true --projection_fields=name firestore.user gs://firestore-export-for-bq/2020-06-29T10:17:23.011+09:00/all_namespaces/kind_user/all_namespaces_kind_user.export_metadata
bq load --source_format=DATASTORE_BACKUP --replace=true --projection_fields=name,owners firestore.shop gs://firestore-export-for-bq/2020-06-29T10:17:23.011+09:00/all_namespaces/kind_shop/all_namespaces_kind_shop.export_metadata
bq load --source_format=DATASTORE_BACKUP --replace=true --projection_fields=name,price firestore.product gs://firestore-export-for-bq/2020-06-29T10:17:23.011+09:00/all_namespaces/kind_product/all_namespaces_kind_product.export_metadata

You can filter like privacy data by using projection_fields.

Schedule commands using Google official image in Docker Hub

Google publishes official Docker image to use gcloud commands.

google/cloud-sdk - Docker Hub

I think we can create scheduled BigQuery exporting easily by using this image with Cloud Run.

Export user records in Firebase Auth to Google Spreadsheet

yosan — Thu, 09 Jul 2020 00:09:26 +0000

I published node project to export user records in Firebase Auth to Google Spreadsheet.

https://github.com/yosan/FirebaseAuth2Sheet

You can get user email's without downloading them to local. It is good for privacy when this code is executed in some server.