DEV Community: Zentered

Time Series Data with TDEngine and GraphQL

Patrick — Mon, 27 Nov 2023 21:18:57 +0000

Motivation and Introduction

As part of the software team at Nevados we are building an operations and monitoring platform for the Nevados All Terrain Tracker®. A solar tracker is a device that orients a solar panel toward the sun. Every solar tracker constantly sends status information and readings, such as the current angle, temperature, voltages, etc. to our platform and we need to store this information for analysis and visualization. If the tracker is configured to send data every 5 seconds, we have 17,280 data points per tracker per day, 518,400 data points per tracker per month. That sums up a lot of information. This kind of data is called "time-series data" and as for all complex problems in software, there are several solutions (Time Series Databases) for it. The most famous ones being InfluxDB and TimescaleDB. For our platform, we decided to work with TDEngine, a relatively new product that is optimized for IoT applications and works with the SQL query language.

There were several arguments for this decision: TDEngine

is open source
is optimized for IoT applications
uses SQL, which is a language we are familiar with
is available as a managed service and we can focus on building our application
is easy to run locally via Docker

In this article, we'll go through the setup of a TDEngine database and tables and how to craft a GraphQL schema that allows us to query the data from various clients and applications.

Getting started with TDEngine

The easiest way to get started with TDEngine is to use their cloud service. Go to the TDEngine and create an account. They have a few public databases we can use, which is great to put together a demo or experiment with queries.

If you want to run TDEngine locally, you can use the Docker image and Telegraf to retrieve data from various sources and send them to the database, such as system information, ping statistics etc.

version: '3.9'

services:
  tdengine:
    restart: always
    image: tdengine/tdengine:latest
    hostname: tdengine
    container_name: tdengine

    ports:
      - 6030:6030
      - 6041:6041
      - 6043-6049:6043-6049
      - 6043-6049:6043-6049/udp
    volumes:
      - data:/var/lib/taos

  telegraf:
    image: telegraf:latest
    links:
      - tdengine
    env_file: .env
    volumes:
      - ./telegraf.conf:/etc/telegraf/telegraf.conf

Check out the official documentation for the Telegraf configuration and the TDEngine documentation on Telegraf. In short, this would look something like this to connect to an MQTT topic:

[agent]
  interval = "5s"
  round_interval = true
  omit_hostname = true

[[processors.printer]]

[[outputs.http]]
  url = "http://127.0.0.1:6041/influxdb/v1/write?db=telegraf"
  method = "POST"
  timeout = "5s"
  username = "root"
  password = "taosdata"
  data_format = "influx"

[[inputs.mqtt_consumer]]
  topics = [
    "devices/+/trackers",
  ]

Instead of setting everything up locally and waiting for the database to fill with information, we'll use the public database for this article, which contains ship movements from the 5 major US ports.

Using TDEngine with public ship movement data

By default, the tables in TDEngine have an implicit schema, which means the schema adapts to the data that is written to the database. This is great for bootstrapping, but eventually, we want to switch to an explicit schema to avoid issues with incoming data. One thing that takes a little time to get used to is their concept of Super Tables ("STable" for short). In TDEngine there are tags (keys) and columns (data). For each key combination, a "table" is created. All tables are grouped in the STable.

Looking at the vessel database, they have one STable called ais_data which contains a lot of tables. Usually, we do not want to query on a per-table basis, but always use the STable to get accumulated data from all tables.

TDEngine has a function DESCRIBE which allows us to inspect the schema of a table or STable. The ais_data has the following schema:

The STable has two keys and six data columns. The keys are the mmsi and the name. We can use regular SQL statements to query the data:

SELECT ts, name, latitude, longitude FROM vessel.ais_data LIMIT 100;

ts                        name      latitude   longitude
2023-08-11T22:07:02.419Z  GERONIMO  37.921673 -122.40928
2023-08-11T22:21:48.985Z  GERONIMO  37.921688 -122.40926
2023-08-11T22:25:08.784Z  GERONIMO  37.92169  -122.40926
...

Keep in mind that time-series data is usually very large, so we should always limit the resultset. There are a few time-series specific functions that we can use, like PARTITION BY which groups results by key and is useful to get the latest update individual keys. For example:

SELECT last_row(ts, name, latitude, longitude) FROM vessel.ais_data PARTITION BY name;

ts                        name      latitude   longitude
2023-09-08T13:09:34.951Z  SAN SABA  29.375961 -94.86894
2023-09-07T18:05:01.230Z  SELENA  33.678585 -118.1954
2023-09-01T17:23:24.145Z  SOME TUESDAY  33.676563 -118.230606
...

I recommend reading their SQL Documentation for more examples. Before we move on, head to "Programming", "Node.js" and retrieve your TDENGINE_CLOUD_URL and TDENGINE_CLOUD_TOKEN variables.

GraphQL with Nexus.js, Fastify and Mercurius

GraphQL is pretty well known these days and there are lots of good articles about it. We chose the technology as we collect and process information from different sources and GraphQL allows us to transparently combine them into a single API.

We'll use the amazing Fastify framework (by now the default choice for Node.js applications) and the Mercurius adapter. The teams of Mercurius and Fastify worked together for a seamless experience and it's a great choice GraphQL APIs with a focus on performance. GraphQL Nexus is a tool to build/generate the schema and resolvers, so we do not have to write everything by hand.

There's a bit of setup code etc. to be done, which I'll skip here. You can find a full example on GitHub - tdengine-graphql-example.

I want to elaborate on two things in this article that are rather specific:

the TDEngine Query library
the GraphQL schema with Nexus

TDEngine Query library

TDEngine has a Node.js library that allows us to query the database. This makes it easy to connect and send queries, unfortunately the responses are a little difficult to work with. So we wrote a little wrapper:

'use strict'

import tdengine from '@tdengine/rest'
import { tdEngineToken, tdEngineUrl } from '../config.js'
import parseFields from 'graphql-parse-fields'

const { options: tdOptions, connect: tdConnect } = tdengine
tdOptions.query = { token: tdEngineToken }
tdOptions.url = tdEngineUrl

export default function TdEngine(log) {
  this.log = log
  const conn = tdConnect(tdOptions)
  this.cursor = conn.cursor()
}

TdEngine.prototype.fetchData = async function fetchData(sql) {
  this.log.debug('fetchData()')
  this.log.debug(sql)

  const result = await this.cursor.query(sql)
  const data = result.getData()
  const errorCode = result.getErrCode()
  const columns = result.getMeta()

  if (errorCode !== 0) {
    this.log.error(`fetchData() error: ${result.getErrStr()}`)
    throw new Error(result.getErrStr())
  }

  return data.map((r) => {
    const res = {}
    r.forEach((c, idx) => {
      const columnName = columns[idx].columnName
        .replace(/`/g, '')
        .replace('last_row(', '')
        .replace(')', '')
      if (c !== null) {
        res[columnName] = c
      }
    })
    return res
  })
}

This returns a TDEngine object that can be passed into GraphQL context. We'll primarily be using the fetchData function where we can pass in a SQL query and get the results back as an array of objects. TDEngine returns the metadata (columns), errors and data separately. We'll use the metadata to map the columns into a regular list of objects. A special case here is the last_row function. The columns are returned as last_row(ts), last_row(name) etc. and we want to remove the last_row part so the attribute maps 1:1 to the GraphQL schema. This is done in the columnName.replace part.

GraphQL Schema

Unfortunately there is no schema generator like Postgraphile for TDEngine and we don't want to write and maintain a pure GraphQL schema, so we'll use Nexus.js to help us with that. We'll start with two basic types: VesselMovement and Timestamp (which is a scalar type). Timestamp and TDDate are two different types to display the date as a timestamp or as a date string. This is useful for the client application (and during development), as it can decide which format to use. asNexusMethod allows us to use the type as a function in the VesselMovement schema. We can resolve the TDDate right here in the type definition to use the original ts timestamp value.

import { scalarType, objectType } from 'nexus'

export const Timestamp = scalarType({
  name: 'Timestamp',
  asNexusMethod: 'ts',
  description: 'TDEngine Timestamp',
  serialize(value) {
    return new Date(value).getTime()
  }
})

export const TDDate = scalarType({
  name: 'TDDate',
  asNexusMethod: 'tdDate',
  description: 'TDEngine Timestamp as Date',
  serialize(value) {
    return new Date(value).toJSON()
  }
})

export const VesselMovement = objectType({
  name: 'VesselMovement',
  definition(t) {
    t.ts('ts')
    t.tdDate('date', { resolve: (root) => root.ts })
    t.string('mmsi')
    t.string('name')
    t.float('latitude')
    t.float('longitude')
    t.float('speed')
    t.float('heading')
    t.int('nav_status')
  }
})

For time-series types, we use the Movement or Series suffix for a clear separation of relational and time-series types in the interface.

Now we can define the Query. We'll start with a simple query to get the latest movements from TDEngine:

import { objectType } from 'nexus'

export const GenericQueries = objectType({
  name: 'Query',
  definition(t) {
    t.list.field('latestMovements', {
      type: 'VesselMovement',
      resolve: async (root, args, { tdEngine }, info) => {
        const fields = filterFields(info)
        return tdEngine.fetchData(
          `select last_row(${fields}) from vessel.ais_data partition by mmsi;`
        )
      }
    })
  }
})

GraphiQL is a great tool to test the API and explore the schema, you can enable it by passing graphiql.enabled = true in Mercurius. With the query, we can see the latest movements of vessels grouped by mmsi. Let's go a little further though. One of the biggest advantages of GraphQL is that is is a transparent layer to the client or application. We can fetch data from multiple sources and combine them into the same schema.

Unfortunately, I wasn't able to find an easy/free API with extensive vessel information. There is Sinay, but they only provide the name, mmsi and imo in their Vessel response (which we already have in TDEngine). For the sake of the example, we assume we do not have the name in our database and we need to retrieve it from Sinay. With the imo we could also query CO2 emissions for a vessel or another API could be used to retrieve an image, the flag or other information, all of which can be combined in the Vessel type.

export const Vessel = objectType({
  name: 'Vessel',
  definition(t) {
    t.string('mmsi')
    t.string('name')
    t.nullable.string('imo')
    t.list.field('movements', { type: 'VesselMovement' })
  }
})

As you can see here, we can include a list field movements with the time-series data from TDEngine. We'll add another query to fetch the vessel information and the resolver allows us to combine the data from TDEngine and Sinay:

t.field('vessel', {
  type: 'Vessel',
  args: {
    mmsi: 'String'
  },
  resolve: async (root, args, { tdEngine }, info) => {
    const waiting = [
      getVesselInformation(args.mmsi),
      tdEngine.fetchData(
        `select * from vessel.ais_data where mmsi = '${args.mmsi}' order by ts desc limit 10;`
      )
    ]

    const results = await Promise.all(waiting)
    return {
      ...results[0][0],
      movements: results[1]
    }
  }
})

🎉 and here we have a working GraphQL API returning rows from TDEngine for a specific vessel we requested. getVesselInformation() is a simple wrapper to fetch data from Sinay. We'll add the TDEngine results into the movements attribute and GraphQL will take care of the rest and map everything to the schema.

Note: SQL Injection

As with any SQL database, we need to be careful with user input. In the example above we use the mmsi input directly, which makes this query vulnerable to SQL injections. For the sake of the example, we'll ignore this for now, but in "real world" applications, we should always sanitize user input. There are several small libraries around to sanitize strings, in most cases we only rely on numbers (pagination, limit etc.) and enums (sort order), which GraphQL checks for us.

Thanks to Dmitry Zaets for pointing this out!

Optimizations

There are a few things that go beyond the scope of this article, but I want to mention them briefly:

Pothos as spiritual successor to Nexus.js

When we started the project, Nexus.js was the best choice to generate our GraphQL schema. Although stable and somewhat feature-complete, it lacks maintenance and updates. There is a plugin-based GraphQL schema builder called Pothos which is a bit more modern and actively maintained. If you're starting a new project, I probably recommend using Pothos instead of Nexus.js.

Thanks to Mo Sattler for pointing this out!

Field Resolvers

As you can see in the Vessel resolver above, both data sources are immediately fetched and processed. This means if the query is only for the name, we still fetch the movements for the response. And if the query is for the movements only, we still fetch the name from Sinay and potentially pay for the request.

That's a GraphQL anti-pattern and we can improve the performance by using the field information to only fetch the data that is requested. Resolvers have the field information as the fourth argument, but they're pretty difficult to work with. Instead, we can use graphql-parse-fields to get a simple object of the requested fields and adjust the resolver logic.

SQL Query Optimizations

In our example queries, we use select * to fetch all columns from the database even if they're not needed. This is obviously pretty bad and we can use the same field parser to optimize the sql queries:

export function filterFields(info, context) {
  const invalidFields = ['__typename', 'date']
  const parsedFields = parseFields(info)
  const fields = context ? parsedFields[context] : parsedFields
  const filteredFields = Object.keys(fields).filter(
    (f) => !invalidFields.includes(f)
  )
  return filteredFields.join(',')
}

This function returns a comma-separated list of fields from the GraphQL info.

const fields = filterFields(info)
return tdEngine.fetchData(
  `select last_row(${fields}) from vessel.ais_data partition by mmsi;`
)

If we request ts, latitude and longitude, the query would look like this:

select last_row(ts, latitude, longitude) from vessel.ais_data partition by mmsi;

With only a few columns in this table this might not matter much, but with more tables and complex queries, this can make a huge difference in application performance.

Time Series functions

TDEngine has some time-series specific extensions that should be used to improve performance. For example, to retrieve the latest entry, a traditional SQL query:

SELECT ts, name, latitude, longitude FROM vessel.ais_data order by ts desc limit 1;

Takes 653ms to execute, while the "TDEngine" query takes only 145ms:

SELECT last_row(ts, name, latitude, longitude) FROM vessel.ais_data;

There are configuration options for each table to optimize for last_row/first_row functions and other cache settings. I recommend reading the TDEngine documentation.

Conclusion

The simple version: In this article, we've set up a TDEngine time-series database and defined a GraphQL schema to allow client applications to connect & query data.

There's a lot more to it. We have a boilerplate project to combine complex time-series data with relational data in a transparent interface. At Nevados, we're using PostgreSQL as a primary database and retrieve time-series data the same way as in the movement example above. This is a great way to combine data from multiple sources in a single API. Another benefit is that the data is only fetched when requested, which adds a lot of flexibility to the client application. Last but not least, the GraphQL Schema works as a documentation and contract, so we can easily tick the "API Documentation" box.

If you have any questions or comments, please reach out on BlueSky or join the discussion on GitHub.

GitHub (Preview) Deployments with Google Cloud Platform

Patrick — Tue, 30 May 2023 18:43:17 +0000

Motivation and Introduction

Build previews aren't anything new or innovative. Heroku had them for years. If you're building on Google Cloud Platform however, things are a little more difficult. Google Cloud Build can be used to deploy to Cloud Functions, Cloud Run, Kubernetes, App Engine etc. etc. This is what makes previews slightly complicated, as Cloud Build doesn't know where you're deploying your application or service to. They have a GitHub App to connect the repositories and show the build status in GitHub, but that's it. No previews, comments or any of the "normal" stuff we're used to from products like Heroku, Vercel or Fly.

There's an "easy" way to post a comment on a GitHub Pull Request from Cloud Build with a simple HTTP request. But why easy when you can use the GitHub Deployments API.

With this API, you can create Environments, have different variables per environment and show build & deployment status directly on a Pull Request. And once the deployment is done on your CD platform, you can add the URL to the deployment status. This is what we're going to do in this article.

Again, there's another way of doing this by using the Deployments API directly with HTTP requests. This can be tricky sometimes though and you'll need to define a Personal Access Token to authenticate with GitHub. Instead, we decided to put together a little Node.js service that uses GitHub Apps to generate an app token and interact with the Deployments API. This allows a little finer control over the deployment process.

Disclaimer

This article is exclusively written for Google Cloud Platform and GitHub. Basic understanding of Google Cloud Platform, Google Cloud Build and IAM permissions is assumed. You should have a Google Cloud Platform project set up.

GitHub Deployer (Node.js & Docker)

GitHub Deployer is a tiny Node.js service that authorized with a GitHub App and interacts with the GitHub Deployments API. Commands are predefined and some information needs to be built in to the Docker image, as otherwise the runtime configuration in Cloud Build would get too complex.

The Docker image is not ready built; you'll need to build it yourself and push it to your registry. The best place is Google Artifact Registry where the images for your application are stored as well (Container Registry is deprecated):

If not done already, create an Artifact Registry repository
Create a GitHub App and install it on your organization
- Create a GitHub App
- Required permissions: 'pull_requests', 'deployments', 'metadata'
Copy the App ID (from the App Settings screen), and the App Installation ID (after clicking "Configure" you'll find the installation ID in the URL. No idea where else that can be found)
Create & download a private key and convert it to base64 (base64 -i your.private-key.pem)

Run the Docker image locally

docker run -it -e REPO_NAME=test -e REF=main -e ENVIRONMENT=test us-central1-docker.pkg.dev/[PROJECT]/docker/gh-deployer:latest create

Required environment variables:

REPO_NAME: required the name of the repository (e.g. gh-deployer)
REF: required the branch, tag or SHA to deploy (e.g. main)
ENVIRONMENT: required the environment to deploy to (e.g. preview)
TRANSIENT_ENVIRONMENT: optional/false if set to true, the deployment will be deleted after a certain time
DESCRIPTION: optional a description of the deployment

The following commands are available as first argument:

create - create a new deployment
pending - the build is pending
in_progress - the build is in progress
queued - the build is queued
success - the deployment was successful
error - something went wrong
failure - the deployment failed

With Cloud Build the options are currently a bit limited. There's no pending state as Cloud Build needs to start in order to create the initial deployment. queued doesn't make much sense either, so in our own setup, we're using the following:

gh-deployer/create to create the transient deployment for a commit
pgh-deployer/ending directly after create
Run kaniko build to build the Docker image for deployment
gh-deployer/in_progress after the build is done and before the image is deployed
gh-deployer/success after the image is deployed

Again, with Cloud Build we don't know where the deployment is going to be, so there are two ways to pass the deployment URL to the success step:

In a build step, write the URL into /workspace/deployer_environment_url
Pass a second argument to the Docker image after success

We use Cloud Run for our deployments, so here's the build step to retrieve the deployment URL for a given Pull Request number:

- name: gcr.io/google.com/cloudsdktool/cloud-sdk
    env:
      - PR_NUMBER=$_PR_NUMBER
    script: >-
      gcloud run services list --project [project] --filter preview-$PR_NUMBER
      --format "value(status.address.url)" > /workspace/deployer_environment_url

Cloud Build Configuration / Terraform

cloudbuild.yaml

Here's a complete exmaple using the cloudbuild.yaml configuration file. We use Kaniko to build and Cloud Run as deplyoment target.

If you work with Terraform, there's a Terraform file afailable as well: preview.tf

steps:
  - name: 'us-central1-docker.pkg.dev/[project]/docker/gh-deployer:latest'
    env:
      - REPO_NAME=$REPO_NAME
      - TRANSIENT_ENVIRONMENT=true
      - DESCRIPTION=Deploying to Google Cloud Run
      - ENVIRONMENT=preview
      - REF=$REF_NAME
    args:
      - create
  - name: 'us-central1-docker.pkg.dev/[project]/docker/gh-deployer:latest'
    env:
      - REPO_NAME=$REPO_NAME
      - ENVIRONMENT=preview
      - REF=$REF_NAME
    args:
      - pending
  - name: 'gcr.io/kaniko-project/executor:latest'
    args:
      - '--destination=us-central1-docker.pkg.dev/[project]/docker/[name]:$SHORT_SHA'
      - '--use-new-run'
  - name: 'us-central1-docker.pkg.dev/[project]/docker/gh-deployer:latest'
    env:
      - REPO_NAME=$REPO_NAME
      - ENVIRONMENT=preview
      - REF=$REF_NAME
    args:
      - in_progress
  - name: gcr.io/google.com/cloudsdktool/cloud-sdk
    args:
      - run
      - deploy
      - preview-$_PR_NUMBER
      - '--image=us-central1-docker.pkg.dev/[project]/docker/[name]:$SHORT_SHA'
      - '--region=us-central1'
      - '--memory=512Mi'
      - '--platform=managed'
      - '--allow-unauthenticated'
    entrypoint: gcloud
  - name: gcr.io/google.com/cloudsdktool/cloud-sdk
    env:
      - PR_NUMBER=$_PR_NUMBER
    script: >-
      gcloud run services list --project [project] --filter preview-$PR_NUMBER --format "value(status.address.url)" > /workspace/deployer_environment_url

  - name: 'us-central1-docker.pkg.dev/[project]/docker/gh-deployer:latest'
    env:
      - REPO_NAME=$REPO_NAME
      - ENVIRONMENT=preview
      - REF=$REF_NAME
    args:
      - success

GitHub Actions

By using the Deployments API, you can trigger GitHub Actions on deployment_status. This makes it easy to run quality assurance checks, end-to-end tests etc. on each preview build. Here's an example for running Playwright on each preview using the environment_url that is passed in for the success state:

name: preview-qa

on: deployment_status

jobs:
  playwright-qa:
    if: ${{ github.event.deployment_status.state == 'success' }}
    timeout-minutes: 60
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: 20
          cache: 'npm'
      - run: npm ci
      - name: Install Playwright Browsers
        run: npm exec playwright install --with-deps -y
      - name: Run Playwright tests
        run: pnpm exec playwright test
        env:
          BASE_URL: ${{github.event.deployment_status.environment_url}}
      - uses: actions/upload-artifact@v3
        if: always()
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 30

Optimizations

the basic image gcr.io/google.com/cloudsdktool/cloud-sdk is relatively large. You can build your own small gcloud image with Cloud Run components to speed up the builds.

Conclusion

Comments on GitHub Pull Requests are easy at first, but when you have multiple deployments per Pull Request, failing builds etc. they don't offer a lot of flexibility and are hard to maintain / update. The GitHub Deployments API offers an elegant way to create and manage deployments from any third party CI/CD system. With the GitHub Deployer we tried to streamline the interaction with the deployments API and take care of some side effects or pitfalls that are impossible to solve with just the HTTP API.

Another benefit of using the GitHub Deployments API is that you can use the deployment_status trigger for GitHub Actions and get the environment_url directly with the event payload.

You can find a little more detailled installation / build instructions and all the code on the GitHub repository

If you have any questions or comments, please reach out on BlueSky / Twitter or join the discussion on GitHub.

Fastify DX and SolidJS in the Real World

Patrick — Wed, 20 Jul 2022 08:56:33 +0000

Motivation and Introduction

SolidJS ranked #1 in the 2021 "State of JS Front-End Frameworks", so we wanted to see what the fuzz is about and give it a proper chance with a side project. We started with a simple Single Page Application (SPA) and a few components but wanted to add data with GraphQL. For GraphQL we needed some sort of authentication, to identify users. This quickly turned more complex day by day. The biggest pitfalls and challenges were understanding Reactivity in SolidJS and the use of context providers. Last but not least, we wanted to add some server-side rendering capabilities, which lead to a three week rabbit hole of reading SolidJS code, Fastify Vite and lots of trial and error. Fortunately, Jonas Galvez was already working on Fastify DX a new full stack framework based on Fastify and Vite, where he added support for SolidJS as well. A nice side effect was that we not only have server-side rendering, but also async rendering, streaming web components/html streaming1 available, which is pretty awesome. As most of this is new and a lot of those technologies are barely documented past "Hello World" or "Todo List", we decided to extract the most important/difficult parts of our project into this "Real World Application with SolidJS and Fastify DX".

TLDR; You can skip ahead and browse through the code on GitHub: Fastify DX with SolidJS

Overview of The Real World Application

Foreword: SolidJS looks a bit like React, but it really isn't. Things like .map() shouldn't be done, as it's not "reactive". Make sure you read up on SolidJS Reactivity and the basic concepts, otherwise you'll run into a lot of issues that are simple to avoid.

Let's start with data. We live in amazing times and it's really easy and cheap (or free) to get started with storing and working with data online. Take for example a PlanetScale MySQL-compatible database, Fastify Node.js Server, Prisma database mapper and a GraphQL connector like Mercurius and you have an entire backend stack. For this example we assume you already have a backend or you want to connect to a 3rd party database like the GitHub GraphQL API.

To retrieve data from your own or a 3rd party API, you most likely need access tokens, this is where Auth0 comes in. You can either retrieve JWT tokens for your own back-end and decrypt them on your API, or retrieve access tokens for the social providers such as GitHub, Twitter, etc. to interact with their API. In our example we decided to use the GitHub GraphQL API to simply retrieve a list of popular repositories for the logged in user, and let them browser to the repository detail page.

To sum up the architecture:

Front-end: SolidJS
Server-side rendering & streaming: Fastify DX
Data layer: GraphQL
Backend: GitHub GraphQL API
Authentication: Auth0 with GitHub Social Connection

Let's get started with the authentication setup.

Getting started with Auth0

Hint: There's a env.example file in the repository which you can rename to .env and store the variables from the next few steps.

Auth0 provides the auth0-spa-js package which offers two ways to authenticate users:

Usually the login with redirect is preferable, but there are some issues retaining the state / session when coming back from the login, so we're currently using "login with popup". The function opens "Auth0 Lock", a pre-defined UI that has all the different authentication mechanisms that are enabled for the application, and can be customized in the Auth0 settings. In order to retrieve GitHub access tokens, we need:

Auth0 Single Page Application to handle the login on our web app
GitHub OAuth App to communicate with Auth0 securely and issue GraphQL API tokens
Auth0 Machine to Machine Application to retrieve access tokens for the GitHub GraqhQL API through Auth0

Auth0 Single Page Application

This is the primary Auth0 application we'll use. Create a new Single Page Application in Auth0 and copy the client id and domain into .env, (variables starting with VITE_AUTH0_). Make sure you add http://localhost:8080 to the Allowed Callback URL, Allowed Web Origin and Allowed Origin.

Auth0 application client id and secret

GitHub OAuth App

Go into your Developer Settings and create a new OAuth App. Name, homepage etc. are not important, but the Authorization callback URL needs to point to your Auth0 Tenant. You can get the domain in your Auth0 application settings: https://<auth0-app-DOMAIN>.auth0.com.

where to find the Auth0 domain

GitHub OAuth App settings

Auth0 Social Connections

Go to Authentication -> Social and Create Connection. Select GitHub. Pick a name and copy the Client ID from your GitHub OAuth App. Go back to the GitHub OAuth App settings and click Create a new client secret. Copy that into the Client Secret in the Auth0 connection details. For this app we only need public_repo as a permission. You should enable Sync user profile attribute at each login to retrieve the user details from GitHub.

After saving, go back to your Single Page Application, to the tab Connections and add the GitHub Social Connection. You should disable the Database connection to avoid issues later.

Auth0 Machine to Machine Application

We use a Machine to Machine Application to retrieve the Access Token for the GitHub API through Auth0. This can't be done directly on the client.

Go back to Applications, Create Application and pick Machine to Machine Application. Enter a name, and copy the client id, secret and domain into .env (variables starting with AUTH0_). Go to APIs and Authorize the application to the Auth0 Management API. The only permission needed is read:user_idp_tokens.

Auth Summary

The above steps look complicated, but they're actually just a few clicks and values to copy&paste to have a complete and secure user authentication setup. The UI and UX comes out of the box from Auth0, so there's very little we need to do for a great signup experience.

We created two "Auth0 Applications", the Single Page App to handle all things login/user/etc. and the Machine to Machine App/Client to talk to the Management API for the GitHub Access Tokens. We also created a GitHub OAuth App and connected that with Auth0, so they can talk to each other and retrieve tokens. This will allow us to:

handle user login and signup through "Auth0 Lock"
synchronize user data from GitHub to the Auth0 profile
issue JWT tokens for our own web app and back-end
retrieve GitHub access tokens for the GraphQL API on our web server

Now, to the code!

Fastify DX

Fastify DX is rad. You can use it with Vue, React, Svelte etc. and it runs on our most beloved Node.js framework Fastify. Here's all we need to start a server:

import Fastify from 'fastify'
import FastifyVite from 'fastify-vite'
import FastifyDXSolid from 'fastify-dx-solid'

const port = process.env.PORT ?? 5000
const logger = process.env.LOGGER ?? false

const server = Fastify({
  logger: logger
})

await server.register(FastifyVite, {
  root: import.meta.url,
  renderer: FastifyDXSolid
})

await server.vite.ready()
await server.listen({ host: '0.0.0.0', port: port })

There's not much going on here; we're starting up Fastify and use the FastifyVite plugin to build and render the Vite app on the fly. But since this is a fully functional web server, we can also add our code to retrieve the GitHub Access Tokens for the UI that we'll need later:

import { ManagementClient } from 'auth0'

server.post('/api/token', async (request, reply) => {
  const userId = request.body.userId

  const management = new ManagementClient({
    domain: process.env.AUTH0_DOMAIN,
    clientId: process.env.AUTH0_CLIENT_ID,
    clientSecret: process.env.AUTH0_CLIENT_SECRET,
    scope: 'read:user_idp_tokens'
  })

  const user = await management.getUser({ id: userId })
  if (user && user.identities) {
    return reply.send({ token: user.identities[0].access_token })
  }
  return reply.send({ ok: true })
})

In our vite.config.js we'll add the SolidJS plugins and everything else needed to render and build the SolidJS App. We'll also add the unocss engine. Some modules such as solid-app-router or solid-headless require special handling for SSR, so they have to be added into the noExternal array in the config.

import { defineConfig } from 'vite'
import viteSolid from 'vite-plugin-solid'
import unocss from 'unocss/vite'
import viteSolidFastifyDX from 'fastify-dx-solid/plugin'
import { join, dirname } from 'path'
import { presetAttributify, presetUno, presetTypography } from 'unocss'
import { fileURLToPath } from 'url'

const path = fileURLToPath(import.meta.url)
const root = join(dirname(path), 'src') // 'client', 'src', ...

export default defineConfig({
  root: root,
  plugins: [
    viteSolid({ ssr: true }),
    viteSolidFastifyDX(),
    unocss({
      presets: [presetUno(), presetAttributify(), presetTypography()]
    })
  ],
  ssr: {
    noExternal: ['solid-app-router']
  }
})

To get your app up and running, copy the context.js, index.html, index.js and root.jsx from the src/ folder or directly from the Fastify DX Solid Starter Kit (or the TypeScript version here: Fastify DX Solid-TS).

When running pnpm dev, you should get a basic web server up and running already, so let's move forward to the SolidJS app. Any errors, refer to the example code or the Fastify DX starter kit, there are a few tiny details which are easy to miss (so best just copy & paste).

SolidJS

At the time of putting together this application, this was the documented example for renderToStream:

// node
renderToStream(App).pipe(res)

// web stream
const { readable, writable } = new TransformStream()
renderToStream(App).pipeTo(writable)

That's not much to go with, so this has been quite a challenge, mostly solved by Jonas with Fastify DX. With SSR and streaming, there come a few new issues that require special attention; the fact that some parts are pre-rendered on the server and have a different context and state. Fastify DX provides context.js, a way to share state between client and server, which is super helpful. Using the context providers properly and taking care of the reactivity chain is a must.

To make things slightly easier, we require auth for our entire app, so there are no authenticated/unauthenticated routes to deal with. We also need the data (GraphQL) context pretty much everywhere, so we'll wrap our app in those two contexts:

<Auth>
  <GraphQL>
    <App />
  </GraphQL>
</Auth>

The tricky part is that auth and GraphQL are both async, means we'll have to wait for the auth client to be initialized and authenticated, and then initialize and authenticate the GraphQL client. This was very painful to figure out and lucky for you I've gone through that ordeal so you can just copy&paste the solution:

return (
  <Auth0
    domain={import.meta.env.VITE_AUTH0_DOMAIN}
    clientId={import.meta.env.VITE_AUTH0_CLIENT_ID}
  >
    <SiteRequiresAuth>
      <Router url={props.url}>
        <Routes>
          {
            // eslint-disable-next-line solid/prefer-for
            props.payload.routes.map((route) => (
              <Route
                path={route.path}
                element={
                  <DXRoute
                    state={props.payload.serverRoute.state}
                    path={route.path}
                    payload={props.payload}
                    component={route.component}
                  />
                }
              />
            ))
          }
        </Routes>
      </Router>
    </SiteRequiresAuth>
  </Auth0>
)

Since we're on the front-end, process.env doens't work; we use import.meta.env instead. Vite automatically exposes all variables that are prefixed with VITE_. The Auth0 context provides a client and [reactive Signals](https://www.solidjs.com/guides/reactivity#introducing-primitives], but those is only available within the Auth0 context provider, so we'll need a wrapper function SiteRequiresAuth to get the initialized Auth0 context, fetch the GitHub access token and initialize the GraphQL client. Only then do we show any content of the site.

function SiteRequiresAuth(props) {
  const auth0 = useAuth0()
  const [accessToken] = createResource(() => auth0.userId(), githubAuth)

  return (
    <Show
      when={auth0.isInitialized() && auth0.isAuthenticated()}
      fallback={<Login auth0={auth0} />}
    >
      <Show when={accessToken()}>
        <GraphQLProvider accessToken={accessToken}>
          {props.children}
        </GraphQLProvider>
      </Show>
    </Show>
  )
}

At this point I'd recommend to read up on Basic Reactivity and get familiar with the primitives.

We use the control flow functions <Show>, as they update when the signals (ie. isInitialized()) change or when the asynchronous resource accessToken is ready. As long as the Auth0 client is not initialized or authenticated, we'll show the Login component. Once those are ready, we'll have to wait again for the accessToken and then finally show the rest of the site.

The context providers are fairly useful now, as the variables are available through the entire site. Anywhere, you can just const auth = useAuth0() and run auth.user() to fetch the user details for example.

Fetching Data

Now that we have an authenticated GraphQL client, we're able to run queries, mutations etc. easily. Here's an example to retrieve the 10 "most starred" repos:

import { gql } from '@urql/core'
import { useGraphQL } from '../contexts/GraphQL'

export default async function repos() {
  const gqlClient = useGraphQL()
  return gqlClient()
    ?.query(
      gql`
        query ($number_of_repos: Int!) {
          viewer {
            name
            repositories(
              orderBy: { field: STARGAZERS, direction: DESC }
              first: $number_of_repos
            ) {
              nodes {
                name
                stargazerCount
              }
            }
            login
          }
        }
      `,
      {
        number_of_repos: 10
      }
    )
    .toPromise()
}

We're retrieving the (authenticated) client context and run the query, simple.

import dataGitHubRepos from '../data/github-repos.data'
const [repos] = createResource(dataGitHubRepos)

createResource is a SolidJS reactive component that processes async data and hydrates the components. This can be async/sync/streamed/....

Pages & Routing

Fastify DX follows the same routing principles as Next.js and Remix. The first page is /pages/index.{js|ts} and other pages can be linked to by using solid-app-router. <Link href="/dashboard">Dashboard</Link> would link to /pages/dashboard.{js|ts} and <Link href="/articles/awesome-solidjs">SolidJS Article</Link> would link to /pages/articles/[id].{js|ts}. SSR, Streaming etc. can be fine-tuned by exporting variables in the page. Check out the examples for streaming, SSR, etc. in the fastify dx starter kit

Components

Components work fairly similar to React, you can either import or lazy() load them. Make sure you use reactive primitives such as Signals, Memos, Resources etc. and pass them to the component. SolidJS re-renders the component when the values change. "Debugging" with console.log is fairly useless, as most variables are only updated when the reactive signals kick in. Sometimes just using <div>{val()}</div> helps to see what's going on, but we're definitely looking for better ways to debug code in SolidJS.

Useful Links, Caveats and TODOs

One issue at the moment is the hard refresh. The state is lost, the cookies gone and the application shows our login screen. That's fairly annoying during development.

Another dent in the developer experience are unhandled client-side errors. They break the reload/fresh cycle and the app becomes unresponsive. A full reload is needed, causing the login screen to show again.

Fastify DX also offers data function and a really amazing shared context, which we'd like to add some more examples for.

Questions, discussion and feedback

Everything looks simple after the fact, but this was a lot of work and effort to piece together all the components and figure out how things fit together. We're hoping to resolve the last few issues and have a boilerplate or starting point for your next SolidJS application, that you can use and not go through the same issues again.

Special Thanks to Jonas Galvez for the amazing work at Fastify DX and the long, late night discussions we had.

Thanks to the entire SolidJS team for this new exciting framework. Feels great to work with and we're looking forward to build more apps with SolidJS.

You can find the source code here: Fastify DX and SolidJS Example on GitHub.

If you have any questions or comments, please reach out on Twitter or join the discussion on GitHub.

MDX 2 with Next.js and Embed Components (ESM version)

Patrick — Fri, 01 Apr 2022 00:00:00 +0000

Motivation and Introduction

We've crafted quite a few websites now with Next.js and we're really loving the experience and outcomes. Thanks to Opstrace we were able to Open Source a huge component to render product documentation on your own Next.js page: Next Product Docs.

The entire Markdown ecosystem is evolving though and we're happy to welcome the release of MDX 2 which brings a set of really cool new features, such as:

📝 Improved syntax makes it easier to use markdown in JSX
🧑‍💻 JavaScript expressions turn {2 \* Math.PI} into 6.283185307179586
🔌 New esbuild, Rollup, and Node.js integrations
and much more. You can read the announcement here.

As it is in an ecosystem, some parts are moving faster or slower than others, so moving to MDX 2 isn't without a few hoops to jump.

We're going to integrate unified/remark/rehype for improved Markdown handling and extensibility with useful features such as GitHub Flavored Markdown, external link handling etc. and we're also integrating embeds to show off your work on 3rd party platforms such as Twitter, Instagram etc. rehype is a tool that transforms HTML with plugins, while remark transforms Markdown.

In this article we're using two different kind of embed mechanisms:

remark-embedder is awesome because it converts simple links into embeds based on the oEmbed format. Everything is done during build/render time, so the embed code is rendered directly into the compiled html without any client-side JavaScript. You can find a list of supported oEmbed providers here.

mdx-embed takes a different approach and offers components that can be imported in mdx. They have a different set of components/embeds.

The good news: you can use both approaches in the same file if you wish to. Here's an example rendered with remark-embedder:

https://twitter.com/PatrickHeneise/status/1508503730295037954

Bonus: it works with ESM.

Getting Started

You can either start from scratch or upgrade your existing Next.js/MDX project. As a reference, we created a repository on GitHub with a working example.

Setup

We're listing the required packages, use your preferred package manager to install those.

The basics (MDX loader for Next.js):

    @mdx-js/loader @next/mdx

remark/rehype plugins

    rehype-external-links remark-gfm remark-frontmatter

List of remark plugins: https://github.com/remarkjs/remark/blob/main/doc/plugins.md#list-of-plugins
List of rehype plugins: https://github.com/rehypejs/rehype/blob/main/doc/plugins.md#list-of-plugins

remark-embedder

If you want to include oEmbeds during build time:

    @remark-embedder/core @remark-embedder/transformer-oembed

mdx-embed

And last but not least "mdx-embed" for embedding components:

    mdx-embed

Next.js Config

We're integrating mdx handling via webpack and the @mdx-js/loader. You'll need to add the pageExtensions and webpack config. Feel free to add / remove remark/rehype plugins here.

import remarkFrontmatter from 'remark-frontmatter'
import remarkGfm from 'remark-gfm'
import rehypeExternalLinks from 'rehype-external-links'

import fauxRemarkEmbedder from '@remark-embedder/core'
import fauxOembedTransformer from '@remark-embedder/transformer-oembed'
const remarkEmbedder = fauxRemarkEmbedder.default
const oembedTransformer = fauxOembedTransformer.default

const nextConfig = {
  reactStrictMode: true,
  experimental: { esmExternals: true },
  pageExtensions: ['md', 'mdx', 'jsx', 'js'],
  webpack: function (config) {
    config.module.rules.push({
      test: /\.mdx?$/,
      use: [
        {
          loader: '@mdx-js/loader',
          options: {
            rehypePlugins: [rehypeExternalLinks],
            remarkPlugins: [
              remarkGfm,
              remarkFrontmatter,
              [remarkEmbedder, { transformers: [oembedTransformer] }]
            ]
          }
        }
      ]
    })
    return config
  }
}

export default nextConfig

JSX Page

You can create a new page (ie [post].jsx). We're able to use a dynamic import to load the contents of the MDX file as a regular component. We're also dynamically importing mdx-embed, as the module is not ESM compatible yet.

import dynamic from 'next/dynamic'
import { getSlugs, getMeta } from 'utils/posts.js'
import { MDXProvider } from '@mdx-js/react'

export default function Post({ post, meta }) {
  // import mdx
  const Post = dynamic(import(`content/posts/${post}.mdx`))

  // dynamic import because not ESM compatible
  const embeds = dynamic(() => import('mdx-embed'))
  const { CodePen, Gist } = embeds

  const components = {
    CodePen,
    Gist
  }

  return (
    <div>
      <main>
        <h1>{meta.title}</h1>
        <article className="prose">
          <MDXProvider components={components}>
            <Post />
          </MDXProvider>
        </article>
      </main>
    </div>
  )
}

MDX Content

This is the file that's being rendered. It's a simple MDX file using regular Markdown YML frontmatter, rather than an export. You can change that to your liking. The components need to be imported again, otherwise there's an error, you can find the full list of supported components on the mdx-embed page.

---
title: hello world
date: '2022-04-01'
preview: showing off twitter embeds
---

import { CodePen, Gist } from 'mdx-embed'

This is a demo post.

### Twitter Embed

https://twitter.com/PatrickHeneise/status/1508503730295037954

### The Gist ...

<Gist gistLink="PatrickHeneise/bbca1a8c4816f92aa3796db41a4a6203" />

### And the Pen

<CodePen codePenId="PNaGbb" />

In addition to those embed components, you can also add your own JSX components. As an exmaple, we created a small <Gallery> component taking an array of images and displaying that in a grid:

The Markdown for this is very simple:

### Component

<Gallery images={['cyprus1.jpeg', 'cyprus2.jpeg', 'cyprus3.jpeg']} caption="This
is a caption" />

This opens up a whole new world of possibilities when it comes to Markdown content.

Summary & Feedback

In this article we explored the basics of and upgrade to MDX2. We also learned how to add rehype/remark plugins to add features to our Markdown content during render/build time and how to use the mdx-embeds to create an easy and rich experience with 3rd party platforms. You can find the minimal example blog on GitHub.

We hope you enjoyed this article and it'll help you on your journey with Next.js and MDX2. If you have any feedback, comments, ideas about this article, please share them here or let us know on Twitter. We'd love to hear from you.

Issue Ops Meet Event Management

Patrick — Fri, 04 Mar 2022 00:00:00 +0000

Motivation

Events (meetups, conferences, talks, ...) are a crucial platform for people to share information, collaborate, and grow. They help to make friends, get condensed and focused information on a specific topic and learn new things. I owe my entire career to events and the people I met along the way.

In 2011 I had the opportunity to visit my first developer meetup BerlinJS. I was new in the JavaScript world and really loved the inclusivity and friendliness in that community. Right after going back to Barcelona I founded BarcelonaJS. Since then a lot has happened and I have been involved in many events and meetups, always trying to find new ways to make the community better, more inclusive.

Unfortunately the tools and infrastructure for event management have been very centralized and restrictive. On meetup there is the "organizer" who is paying for the group, and they're the one responsive on what's going on. While traveling, I've seen groups with thousands of members but not a single event (neither in person nor virtual), with some people trying desperately through the comments or discussions to get people together.

We've been looking for ways to make the event management more accessible and decentralize the process. Making it easier for individual organizers by allowing more people to collaborate in a community.

The beginning of GitEvents

GitEvents was born in December 2014 as a joint project between BarcelonaJS, Node.js Barcelona and the London Node User Group (LNUG). As developers, we're collaborating on code every day on GitHub, so we thought it would be the perfect platform to also manage events from there. Looking back, the tooling around it was terrible, but the best at the time: a node.js server were GitHub Webhooks were sent to. A GitHub Issue represented a talk for an event, a "Milestone" represented the upcoming events, and "talks" were added to "events" by assigning the issues to milestones and labeling them. We then created json files for each event with semantic schema.org data, which we then used to generate a website.

GitEvents Resurrections

Fast forward to December 2021, we're back with a new version of GitEvents, as nothing has improved (or got worse) in the event management space. The idea is still the same: Issue-based event management. We've simplified the architecture; no server is needed any more and no complicated webhook configuration. Instead we're using GitHub Actions to process issues and events that happen on the repository. When the GitEvents Action is installed in a repo, it can set up the required labels and issue templates with an easy setup script. GitEvents runs with the permissions from a GitHub App, so no bot account is needed, and it's clearly visible what it does when it labels or comments on issues. One of the main features, inspired by Steffen, is the "auto invite" of everyone who interacts with the organization. And since it's based on Open Source Repositories on GitHub, everyone can add talks, event proposals etc. and a team of organizers can moderate.

We've started using GitEvents on the meetups we currently co-organize, such as CyprusJS and the Cyprus Developer Community. We'd love to hear from you if you have any feedback or ideas for improvements. And we're happy to help you set up GitEvents for your own meetup.

GitEvents Features

Auto-invite everyone who interacts with the GitHub organization, be it a comment, issue or discussion. This helps to create an inclusive space and makes everyone feel welcome
creation of the required labels and templates to run a meetup, inspired by popular meetups like BerlinJS, BarcelonaJS etc.
Code of Conduct template and consent check for events, talks, sponsors, attendees, organizers, etc.
iCalendar export of events as .ics file in the same repository, which users can subscribe to in their calendars

GitEvents Roadmap

More modular/flexible approach for 'auto invite', issue management, ics generation etc.
Add atom/xml/json feeds for events
Discord integration
Snippets to copy&paste the event schedule to other websites
Simplify the event/talk relationship and connect everything properly
Add a parser for GitHub Issue Forms instead of relying markdown frontmatter
Much more.

Conclusion

We're just getting started (in-the-middle-but-somewhat-post-pandemic?), but it really helps getting people together and involve everyone in the process. Events are a great way to get people together either virtual or in real life, and we're excited to see what you can do with it.

http://github.com/gitevents
https://gitevents.org

We'd love to hear your feedback!

Authenticated requests with Vite + React 18 + Hummingbird Swift API

Patrick — Tue, 28 Dec 2021 00:00:00 +0000

Motivation and Introduction

As we're exploring Swift on Server, we wanted to see how that integrates with our authentication solution. Vite has been the hype the past months, so we were eager to try this out as well. We built a small demo project to log in to Auth0, obtain a JSON Web Token (JWT), and use the token to authenticate requests to the Swift API. For the API, we picked Hummingbird as it's a bit lighter than Vapor.

You can check out the code on GitHub

Let's get started

Auth0

As a first step, let's create an Auth0 Application and note down the variables. If you haven't signed up for Auth0 yet, you can do that for free, and then create a "Single Page Application (SPA)" in the Auth0 Dashboard.

To allow requests from localhost, add http://localhost:3000 to the allowed callback URLs, web origins and logout URLs. If you deploy this application to a cloud provider, the URLs need to be added here as well:

For the frontend (React), add these values into the .env file:

VITE_APP_AUTH0_DOMAIN=<app-id>.<region>.auth0.com
VITE_APP_AUTH0_CLIENT_ID=<your-auth0-client-id>

Note: you can find detailled instructions about Auth0 with React in the Quickstart.

For the backend (Hummingbird/Swift), we need the "JSON Web Key Sets" of your application. You can find the endpoint in the Application Settings at the bottom "Advanced Settings" -> "Endpoints". It should look more or less like this:

.env:

JWKS_URL=https://<app-id>.<region>.auth0.com/.well-known/jwks.json

Afterwards, go to "Users" and add a test/dev user.

Vite/React/Windi

Vite is the "Next Generation Frontend Tooling" and we wanted to test first-hand what that means. We used an "Opinionated React Template" created by Omar Elhawary as a base, since it's fairly close to the structure we're used to from Next.js. This includes React 18, Vite and some other tooling. Instead of Tailwind we use WindiCSS which has great support for Vite:

Install the package

npm i -D vite-plugin-windicss windicss

Add the vite.config.js config file:

import WindiCSS from 'vite-plugin-windicss'

export default {
  plugins: [WindiCSS()]
}

Add windi to your main.js/ts

import 'virtual:windi.css'

For the authentication, we'll use the Auth0 React SDK:

npm install @auth0/auth0-react

And add the Auth0Provider to your main.jsx/tsx:

import 'virtual:windi.css'

import { createRoot, hydrateRoot } from 'react-dom'
import { BrowserRouter } from 'react-router-dom'
import { Auth0Provider } from '@auth0/auth0-react'

import { Routes } from '@/config'

function App(): JSX.Element {
  return (
    <BrowserRouter>
      <Auth0Provider
        domain={import.meta.env.VITE_APP_AUTH0_DOMAIN!}
        clientId={import.meta.env.VITE_APP_AUTH0_CLIENT_ID!}
        redirectUri={
          typeof window !== 'undefined' ? window.location.origin! : ''
        }
      >
        <Routes />
      </Auth0Provider>
    </BrowserRouter>
  )
}

const app = document.querySelector('#app') as Element
const root = createRoot(app)

if (app.hasChildNodes()) hydrateRoot(app, <App />)
else root.render(<App />)

The first page that's loaded is index.jsx/tsx, so we'll add the useAuth0 helper to that page and require authentication:

import { useAuth0, withAuthenticationRequired } from '@auth0/auth0-react'

function Home(): JSX.Element {...}

export default withAuthenticationRequired(Home, {
  onRedirecting: () => <div>Redirecting you to the login page...</div>
})

The helper provides several states such as error, isLoading as well as the user data and logout action:

const { isLoading, getAccessTokenSilently, error, user, logout } = useAuth0()

Hello {user.name}!

To make authenticated requests with a JWT, we'll use getAccessTokenSilently() and pass the audience. Prefixed with Bearer, we have a valid authentication token for our API:

const token = await getAccessTokenSilently({
  audience: `https://${import.meta.env.VITE_APP_AUTH0_DOMAIN}/api/v2/`
})
const response = await fetch(url, {
  mode: 'cors',
  method: 'GET',
  headers: {
    Accept: 'application/json',
    'Content-Type': ' application/json',
    Authorization: `Bearer ${token}`
  }
})
const data = await response.json()

Hummingbird (Swift) API with JWT

In this example we don't use any unauthenticated requests; all requests to the API need to have an Authorization header. The easiest way is a simple Middleware to decode the token:

import Foundation
import Hummingbird
import HummingbirdAuth
import JWTKit

struct JWTPayloadData: JWTPayload, Equatable, HBAuthenticatable {
  enum CodingKeys: String, CodingKey {
    case subject = "sub"
    case expiration = "exp"
  }

  var subject: SubjectClaim
  var expiration: ExpirationClaim
  // Define additional JWT Attributes here

  func verify(using signer: JWTSigner) throws {
    try self.expiration.verifyNotExpired()
  }
}

struct JWTAuthenticator: HBAsyncAuthenticator {
  var jwks: JWKS

  init(jwksUrl: String) throws {
    let jwksData = try Data(
      contentsOf: URL(string: jwksUrl)!
    )
    jwks = try JSONDecoder().decode(JWKS.self, from: jwksData)
  }

  func authenticate(request: HBRequest) async throws -> JWTPayloadData? {
    guard let jwtToken = request.authBearer?.token else { throw HBHTTPError(.unauthorized) }

    let signers = JWTSigners()
    do {
      try signers.use(jwks: jwks)
      let payload = try signers.verify(jwtToken, as: JWTPayloadData.self)
      return payload
    } catch {
      print("couldn't verify token")
      throw HBHTTPError(.unauthorized)
    }
  }
}

Since we're using two different ports (3000 for the vite client, 8080 for the hummingbird server), we'll also need to enable Cross-Origin Resource Sharing (CORS). You can add both middlewares to your Application+configuration.swift`:

`swift
self.middleware.add(
HBCORSMiddleware(
allowOrigin: .originBased,
allowHeaders: ["Accept", "Authorization", "Content-Type", "Origin"],
allowMethods: [.GET, .OPTIONS]
))

let jwtAuthenticator: JWTAuthenticator
guard let jwksUrl = env.get("JWKS_URL") else { preconditionFailure("jwks config missing") }
do {
jwtAuthenticator = try JWTAuthenticator(jwksUrl: jwksUrl)
} catch {
print("JWTAuthenticator initialization failed")
throw error
}
self.middleware.add(jwtAuthenticator)
`

You can then use the auth0 user id in requests to request user-specific data etc.:

swift let jwtPayload = request.authGet(JWTPayloadData.self) let userId = jwtPayload?.subject

Hummingbird does not load .env variables out of the box, so we'll use a Makefile to load the environment and build/run the server:

`makefile

!make

MAKEFLAGS += --silent
include .env
export $(shell sed 's/=.*//' .env)

start:
swift run Server

build:
swift build -c release

install:
swift package resolve
`

In the first step, we've created the .env file already for the server, so the JWKS_URL should be available, otherwise make start will throw an error, as the precondition fails.

Putting it all together

Open two terminal windows and run:

npm run dev

to start the vite development server on port 3000 and:

make start

to start the swift API on port 8080. Open your browser on http://localhost:3000 and you should be redirected to an Auth0 login screen:

After logging in, you can make an authenticated API request and get some data back:

Summary

As of today, React 18 is still in beta, the initial setup was a bit tricky, but once we finally got it working, it was really pleasent. Especially the Next.js-like routing and pages/components structure made it very easy to transition. Vite is super fast, it's simply amazing how the Hot Module Replacement (HMR) works. The SPA flow with Auth0 is quick and painless as well, and the Access Tokens are refreshed automatically whenever needed.

Given we're still getting started with Swift on Server, the authentication middleware for Hummingbird was quite the challenge. We made it work with JWT-Kit which offers a lot of the needed functionality (especially fetching JWKS and RS256 encryption).

The final outcome is a fast, reliable and strictly typed API that can be deployed on Google Cloud (CGP) and Amazon Web Services (AWS) and a Single Page Application (SPA) that can be deployed to a simple Storage bucket like S3 or Cloud Storage. The API can be used with Cross-Origin headers, or routed with a load balancer on the same domain (ie. /api). The application starts with a nice, branded login/signup window and easily integrates with social providers by just enabling them in the Auth0 console. Auth0 credentials are stored in a cookie and a JWT access token can be requested on demand when API requests are made.

Special Thanks

the contributors of JWT-Kit - https://github.com/vapor/jwt-kit
Adam Fowler for the Hummingbird Project - https://github.com/hummingbird-project
Omar Elhawary for https://github.com/oedotme/render

You can find the source code here: Swift API Demo on GitHub. If you have any questions or comments, please reach out on Twitter or start a discussion on GitHub.

Deploying a Swift API on AWS Elastic Container Service with CodePipeline

Patrick — Tue, 14 Dec 2021 00:00:00 +0000

Motivation

After getting started with Swift on Server and deploying a Swift on Server API on Google Cloud Run with Google Cloud Build, we wanted to see how to get the Swift API running on Amazon Web Services (AWS). AWS offers a lot more products and flexibility, which comes at the cost of simplicity, so we created a new Terraform project to consistently re-create the environment. Here's an overview of the puzzle pieces roughly in order:

CodeCommit as a mirror of the GitHub repo
CodeBuild to build the Docker image and push it to the registry
CodeDeploy to orchestrate the deployment and traffic allocation to ECS
CodePipeline to glue Commit, Build and Deploy together in a nice deployment pipeline
Elastic Container Registry (ECR) to store the Docker images
Elastic Container Service (ECS) to run Docker containers
Elastic Application Load Balancer (ABN) to manage the traffic between instances and allow "Blue/Green Deployment"
and a few more 3-letter acronyms and other AWS products like KMS, IAM, S3, CloudWatch...

Please mind the /api folder. For prototyping we've decided to work in a single repo, while the server bits are stored in the /api subfolder (which makes things again a little more difficult, as this needs to be specified in various places).

TLDR; Mistakes made, 🤦 and lessons learned

You can find our Terraform project on GitHub

for ECS Blue/Green builds, the "image definition" is not "imagedefinitions.json" but imageDetail.json: - printf '{"ImageURI":"%s"}' ${REPOSITORY_URI}:latest} > imageDetail.json
to deploy from CodePipeline to ECS, the "blue/green" strategy is required
CodePipeline uses S3 to pass build artifacts from one step to the next
a taskdef.json and appspec.yaml are necessary build artifacts from the build to the deploy step
it's appspec.yaml (y*a*ml with the 'a') not appspec.yml
Docker container port (ie 8080) is used everywhere except for the Load Balancer Listener
Fargate requires network mode awsvpc and public IPs, or private IPs and a lot more configuration with firewalls, networks etc.
IAM permissions are a mess
snake_case should be used for all Terraform names

Getting down to the nitty gritty

Setting up this project was all but smooth. There are a lot of little details and a single wrong number can cost you hours of debugging. The AWS error messages are misleading, the user/developer experience a mess, each of the products uses different UIs, all in all very chaotic. At least with Terraform we're able to write "Infrastructure as Code" and provide a working Terraform project that you can use as a starting point. Some product specific information first:

AWS CodeCommit

CodeBuild uses GitHub OAuth, which gives Amazon access to all public and private repos. We're a little paranoid, so that's not something we wanted to do. Instead, we created a CodeCommit repo and push the changes there, until AWS switches to the new GitHub App experience, which lets you choose individual repos.

In order to push to CodeBuild, you need to add your ssh public key to your IAM user and add CodeCommit origin to your git config:

CodeCommit setup: https://docs.aws.amazon.com/codecommit/latest/userguide/setting-up-ssh-unixes.html?icmpid=docs_acc_console_connect_np
IAM console "security credentials": https://console.aws.amazon.com/iam/home#/users/?section=security_credentials

You can probably skip this with the AWS Connector for GitHub where you can connect a GitHub Repo as Source in the Pipeline. We're still working that out with Terraform.

AWS CodeBuild

CodeBuild was one of the easier tools to use. Just add your buildspec.yml. Important are the build artifacts that are exported to S3 and used by the deploy step.

Elastic Container Registry & Elastic Container Service (ECS)

To get a head start for your builds, you can push your latest image to ECR:

aws ecr get-login-password --region <region> | docker login --username AWS --password-stdin <account_id>.dkr.ecr.<region>.amazonaws.com
docker push <account_id>.dkr.ecr.<region>.amazonaws.com/<image>:latest

ECS was the worst part to get up and running, since there are a lot of different components playing together. There are containers, tasks, services, task definitions, image definitions and more. If you encounter any issues, it's worth looking into the service, then tasks, to see if the container boots up correctly.

Let's Deploy

We've published the entire Terraform project here: Terraform ECS Blue/Green Deployments with CodePipeline. You need to download the Terraform CLI. If you're new to AWS and you don't have the AWS CLI installed, you can find instructions here.

terraform init
terraform validate
terraform plan
terraform apply

Terraform outputs the url of the load balancer, you should be able to navigate to that URL and get the 'hello world' response.

Clean up:

terraform destroy

Feel free to fork/clone the repo and adjust it to your needs.

Summary

AWS comes with a lot of benefits and great features, but the UI across the products is inconsistent and there are dozens of configuration options, permissions and other things to take care of and its easy to miss an important detail. Setting up the project in Terraform makes things a little easier, but it still takes time to write down all the configs. There are some pre-built Terraform modules for AWS which we didn't use, as we wanted to learn how things worked under the hood. In comparison, deployment to GCP was definitely a lot easier and faster to set things up. On GCP there are a lot of "beta" products though and things can change, or sometimes don't work as expected.

Special thanks and further reading ...

Special thanks to "vinycoolguy2015", "snow-dev" and "capital one" for their articles and resources on the topic. Here's a list of links to articles and repos that helped put together this project:

Thanks for reading! You can check out all the code on GitHub and read more about our "Swift on Server" series:

If you have any questions or comments, please reach out on Twitter or start a discussion on GitHub.

Deploying a Swift API on Google Cloud Run with Google Cloud Build

Patrick — Fri, 03 Dec 2021 19:59:30 +0000

Motivation

Recently we started a Swift API proof of concept and wanted to see if we can deploy the service with "the tools we know" from Node.js: continuous integration in the cloud and deployment to serverless / managed k8s platforms. TL;DR: It works! And very similar to working with Node.js services.

Dockerfile

Let's start with the Dockerfile, the current standard for k8s/Cloud Native services. If you created your new awesome Swift API with the Vapor CLI/template, there's a Dockerfile already in the folder. We decided to remove the package updates to speed up the builds a bit. We also removed the user/group specifics and rely on system defaults:

FROM swift:5.5-focal as build
WORKDIR /build
COPY ./Package.* ./
RUN swift package resolve
COPY . .
RUN swift build -c release

WORKDIR /staging
RUN cp "$(swift build --package-path /build -c release --show-bin-path)/Run" ./
RUN [ -d /build/Public ] && { mv /build/Public ./Public && chmod -R a-w ./Public; } || true
RUN [ -d /build/Resources ] && { mv /build/Resources ./Resources && chmod -R a-w ./Resources; } || true

FROM swift:5.5-focal-slim

RUN export DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true && \
    apt-get install ca-certificates

WORKDIR /app
COPY --from=build /staging /app

EXPOSE 8080
ENTRYPOINT ["./Run"]
CMD ["serve", "--env", "production", "--hostname", "0.0.0.0", "--port", "8080"]

We still need to make sure the ca-certificates package is installed, as PlanetScale uses those. If you use a different database, you might be able to skip this step too.

Artifact Registry

Google deprecated the Container Registry in favor of the Artifact Registry, which can also handle Node.js, Java and other packages, not just Docker containers. You have to enable the Artifact Registry for your project. There are a few minor changes to the URLs. If you want to connect locally to the registry (push/pull), you need to authenticate for your region. You can read more here: https://cloud.google.com/artifact-registry/docs/transition/changes-docker.

Cloud Build

For continous deployment of our Swift API, we're using Cloud Build pretty much the same way as for a Node.js API:

build the container image
deploy the image to Cloud Run
cache the image in the Artifact Registry

The easiest way to set this up is to go to Cloud Build Triggers and "Create a Trigger". Set a name, "Push to branch", and pick the Source Repository. You may need to authenticate to GitHub/Bitbucket and set permissions for Google Cloud Platform to access your repo. At the end you should set the "Substitution variables", which are environment variables that you can inject in the runtime, such as the DATABASE_HOST or other information. We're going to check out how to use the Secret Manager with Swift in a future post. See the screenshot for the trigger settings:

For the trigger to work, a cloudbuild.yml file is required in the root of the project. We first try to fetch the latest image for caching, then we build the Dockerfile, push it to the registry and deploy the service to Cloud Run. Important here is to set the --port=8080 parameter, as Vapor doesn't use the PORT environment variable at the moment. Here is a list of Cloud Run regions, we recommend us-central1 or europe-west1.

Unfortunately the builds take quite long, be prepared for 20-30 minute build times. To save yourself time and energy, build (docker build . -t swift-server) and test run (docker run -p 8080:8080 swift-server) the image on your local machine first, and make sure the server starts correctly.

cloudbuild.yml, replace <repo-name> and <image-name>:

steps:
  - name: 'gcr.io/cloud-builders/docker'
    entrypoint: 'bash'
    args:
      - '-c'
      - |
        docker pull ${_REGISTRY}/$PROJECT_ID/<repo-name>/<image-name>:latest || exit 0
  - name: 'gcr.io/cloud-builders/docker'
    args:
      - build
      - -t
      - ${_REGISTRY}/$PROJECT_ID/<repo-name>/<image-name>:$SHORT_SHA
      - -t
      - ${_REGISTRY}/$PROJECT_ID/<repo-name>/<image-name>:latest
      - .
      - --cache-from
      - ${_REGISTRY}/$PROJECT_ID/<repo-name>/<image-name>:latest
  - name: 'gcr.io/cloud-builders/docker'
    args:
      ['push', '${_REGISTRY}/$PROJECT_ID/<repo-name>/<image-name>:$SHORT_SHA']
  - name: 'gcr.io/cloud-builders/gcloud'
    args:
      - run
      - deploy
      - <repo-name>
      - --image=${_REGISTRY}/$PROJECT_ID/<repo-name>/<image-name>:$SHORT_SHA
      - --port=8080
      - --region=us-central1
      - --memory=512Mi
      - --platform=managed
      - --allow-unauthenticated
      - --min-instances=0
      - --max-instances=5
      - --set-env-vars
      - ^;^ENV=production; DATABASE_HOST=${_DATABASE_HOST}; DATABASE_PORT=${_DATABASE_PORT}; DATABASE_USERNAME=${_DATABASE_USERNAME}; DATABASE_PASSWORD=${_DATABASE_PASSWORD}; DATABASE_NAME=${_DATABASE_NAME}
images:
  - '${_REGISTRY}/$PROJECT_ID/<repo-name>/<image-name>:$SHORT_SHA'
  - '${_REGISTRY}/$PROJECT_ID/<repo-name>/<image-name>:latest'
timeout: 2000s

Note: See environment variables for more information on ^;^....

Cloud Run

In this article we use Cloud Run, a managed Kubernetes product on Google Cloud Platform. It allows cloud native, serverless deployments of Docker images directly from Cloud Build.

If you have the Dockerfile and cloudbuild.yml files in your repo, and Cloud Build finished the build job with the little green ✅, your Cloud Run service should already be deployed and running:

Click on the service and you should see the service details, including the URL of your app (ie. swift-api-demo-123.run.app.):

Clicking the link should show you a simple but satisfying It works! in the browser. If you add /todos to the URL, you should see the item you added during development, or you can POST to the address to create a new item. In our Cloud Build config we defined --min-instances=0 which means the app will go into standby after ~15 minutes if there are no requests, which is very cost effective. You can increase the number to have "always on" services. --max-instances sets the maximum number of instances, for auto scaling. The region, memory, cpu settings etc. can be adjusted either with substitution variables or hard-coded in the cloudbuild file. Substitutions are great if you want to deploy to different regions, for example.

As this proof of concept is successful, with Cloud Run we can also perform gradual rollouts such as canary deployments or blue/green deployments of your Swift API, run everything on your own domain and create high availability services 🥳.

Summary

This is very exciting. We're essentially using the same tools and workflows we already use for production apps in Node.js, and get the same logging, monitoring and scaling for our apps. To sum up the setup:

Create a Dockerfile and build/run the app locally to verify it works
Add the Dockerfile and cloudbuild.yml to the repo in the project root folder
Go to the Cloud Build history and check if your build is green
Go to the Cloud Run overview to check the status of your service

Afterwards the workflow for developers is simple:

Write code
Commit, push and merge
Wait...
Check out your feature live in production

Thanks for reading! You can check out all the code on GitHub. If you have any questions or comments, please reach out on Twitter or start a discussion on GitHub.

We'll be working on authentication, performance and semantic versioning next, stay tuned!

Getting Started with Swift on Server

Patrick — Fri, 03 Dec 2021 19:47:59 +0000

We've been working with Node.js for over 10 years now and decided to pick up a new challenge. For "strong typing" features, people are exploring Rust and Go as alternatives to Node.js with TypeScript. Swift on Server has been around for a few years now, and Swift is an interesting language to learn for mobile development. It feels quite similar to JavaScript with the new async/await features and is easily readable.

There are some interesting http frameworks (think Express.js, Fastify, etc.) to simplify working with SwiftNIO such as Vapor, Hummingbird and Perfect. There's even a full CMS (Feather).

The Swift Server Work Group names a few benefits on why Swift is great as a server-side language:

Small footprint
Quick startup time
Deterministic performance

Small footprint: One of the main goals of a modern cloud platform is to maximize resource utilization by efficiently packing services into a single machine. Cloud services built with Swift have a small memory footprint (measured in MB)–especially when compared to other popular server languages with automatic memory management. Services built with Swift are also CPU-efficient, given the language’s focus on performance.

Quick startup time: Swift-based applications start quickly since there are almost no warm up operations. This makes Swift a great fit for cloud services, which are often re-scheduled onto new VMs or containers to address platform formation changes. Using Swift also helps streamline continuous delivery pipelines, since you incur less wait time for new versions of the service fleet to come online. Finally, quick boot times make Swift a perfect fit for serveless applications such as Cloud Functions or Lambda with negligible cold start times.

Deterministic performance: Swift’s use of ARC (instead of tracing garbage collection) and its lack of JIT gives it an important edge in the cloud services space. While tracing garbage collection technologies have vastly improved in the last few years, they still compete with the application for resources which triggers non-deterministic performance. The absence of JIT means no runtime optimization or de-optimization. It’s challenging to debug non-deterministic performance, and language-induced non-deterministic performance can both confuse and mask application-level performance issues that could otherwise be addressed.

Those all sound great, so we wanted to see if we can use modern, Cloud Native tools to build and deploy a Swift Server application. But first we needed to build one. We decided to go with Vapor and PlanetScale, to add some complexity to the regular "Hello World" app.

Install

Swift and Vapor

On macOS, if you have the Developer Tools and Xcode installed, you should be good to go already. You can check if Swift is properly installed:

  » swift --version
swift-driver version: 1.26.9 Apple Swift version 5.5.1 (swiftlang-1300.0.31.4 clang-1300.0.29.6)
Target: arm64-apple-macosx12.0

If you don't have Swift installed yet, follow the Installing Swift guide for macOS, Linux or Windows. You'll also need Vapor, which you can easily install with brew:

brew install vapor

PlanetScale

Install the PlanetScale CLI. For macOS, via brew:

brew install planetscale/tap/pscale mysql-client

Full PlanetScale CLI reference

Creating your first Swift on Server / Vapor project

This is as simple as npm init:

vapor new swift-api-demo

To go beyond the hello world steps, let's install Fluent for MySQL and skip Leaf during the project initialization.

cd swift-api-demo
open Package.swift

This will open Xcode and download/install vapor dependencies. You should see the following screen:

Now simply hit the "Run" button (▶︎, or ⌘R). In the Xcode terminal you should see a notice that the server has started ([ NOTICE ] Server starting on http://127.0.0.1:8080). And voilà, you can curl your server instance:

 » curl localhost:8080
It works!%
 » curl localhost:8080/hello
Hello, world!%

VScode

Instead of XCode, you can also write Swift applications in VSCode. There is a vscode extension available for the language support. We love "auto format on save", which is a little tricky, but can be achieved by downloading and compiling swift-format:

VERSION=0.50500.0  # replace this with the version you need
git clone https://github.com/apple/swift-format.git
cd swift-format
git checkout "tags/$VERSION"
swift build -c release
ln -s /usr/local/bin/swift-format .build/release/swift-format

The last line will create a symlink in your binaries folder (macOS specific, you may need to use a different path). Afterwards, format on save should work wth the SwiftFormat extension.

Create the PlanetScale Database

After installing the PlanetScale CLI you need to log in, create a database and a password:

    pscale auth login
    pscale database create swift-server-db
    pscale password create swift-server-db main production-password

The default database has no "production" branch, which means schema changes can be applied directly on main. Once you promote your branch to production, you need to follow a "feature branch flow", just like in Git:

Create a branch, then open a SQL shell to create a new table:

    pscale branch create swift-server-db todos
    pscale shell swift-server-db todos

We're creating the table todos with an autogenerated id and a title:

CREATE TABLE `todos` (
  `id` INT AUTO_INCREMENT PRIMARY KEY,
  `title` varchar(1024) NOT NULL,
  `created_at` datetime(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6),
  `updated_at` datetime(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
exit;

These changes are now in a branch. You can open a Deploy Request and merge it into main:

pscale deploy-request create swift-server-db todos
pscale deploy-request deploy swift-server-db 1

If you want to use DBeaver or any other MySQL UI, you can connect to the branch directly with pscale connect <db-name> <branch-name>.

Connecting the Pieces

Identifiers (IDs)

Vapor/Fluent work with UUIDs as default. MySQL doesn't have native support for those, so it's recommended to switch to Integer IDs:

Sources/App/Models/Todo.swift:

import Fluent
import Vapor

final class Todo: Model, Content {
    static let schema = "todos"

    @ID(custom: "id")
    var id: Int?

    @Field(key: "title")
    var title: String

    init() { }

    init(id: Int? = nil, title: String) {
        self.id = id
        self.title = title
    }
}

Vapor with PlanetScale

After creating a password on PlanetScale, you should be able to Connect securely to your database. You can find the credentials (host, user, password, db name) on your Dashboard:

Create a branch, then open a SQL shell to create a new table:

    pscale branch create swift-server-db todos
    pscale shell swift-server-db todos

We're creating the table todos with an autogenerated id and a title:

CREATE TABLE `todos` (
  `id` INT AUTO_INCREMENT PRIMARY KEY,
  `title` varchar(1024) NOT NULL,
  `created_at` datetime(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6),
  `updated_at` datetime(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
exit;

These changes are now in a branch. You can open a Deploy Request and merge it into main:

pscale deploy-request create swift-server-db todos
pscale deploy-request deploy swift-server-db 1

If you want to use DBeaver or any other MySQL UI, you can connect to the branch directly with pscale connect <db-name> <branch-name>.

Connecting the Pieces

Identifiers (IDs)

Vapor/Fluent work with UUIDs as default. MySQL doesn't have native support for those, so it's recommended to switch to Integer IDs:

Sources/App/Models/Todo.swift:

import Fluent
import Vapor

final class Todo: Model, Content {
    static let schema = "todos"

    @ID(custom: "id")
    var id: Int?

    @Field(key: "title")
    var title: String

    init() { }

    init(id: Int? = nil, title: String) {
        self.id = id
        self.title = title
    }
}

Vapor with PlanetScale

After creating a password on PlanetScale, you should be able to Connect securely to your database. You can find the credentials (host, user, password, db name) on your Dashboard:

Wrapping it up

Overall we're pretty excited how things have developed in the Swift on the Server world and how Vapor works. There are some really cool features coming up in Swift 5.6 and everything is under active development. The Vapor Community on Discord is super friendly and helpful!

As a Proof of Concept, we're able to run a http service and connect to a MySQL database. We can store and retrieve data for a simple todo list. That's all pretty nice, but it's still just "localhost" development. We wanted to know how to get this up into the cloud, so we wrote another article on how to deploy your swift api to the cloud. TL;DR: this is pretty simple and we were able to use Cloud Native tooling such as Cloud Build and Cloud Run to automate the entire deployment pipeline from GitHub to Production.

You can find the source code here: Swift API Demo on GitHub. If you have any questions or comments, please reach out on Twitter or start a discussion on GitHub.

TODO

A few things we need to do some more research on:

Write tests and implement a TDD workflow
How can we run Vapor Migrations on PlanetScale?
How can we use/automate semver tags with XCode?

Serverless v3.0.0

Patrick — Tue, 23 Nov 2021 00:00:00 +0000

Breaking Changes: v2.0.0 to v3.0.0

Some time ago we wrote about Serverless deployments v2.0.0 and a lot has changed and improved since then, so it's time for an update.

One of the biggest improvements to the workflow is GitHub Actions. No need for external CI/CD systems to generate new versions any more. GitHub also improved Releases, and Conventional Commits reached v1.0.0.

Continous Deployment of Serverless Applications

Let's walk through the details of a fully automated continous integration and deployment system with all those fancy new toys. The one thing that hasn't changed since the last article is our Git workflow: we recommend the Git Feature Branch Workflow. The merge strategy doesn't matter much, rebase, squash and merge all work with the following tools and flows.

Use Conventional Commits

We use Semantic Release to calculate version numbers based on commit messages. The 'de facto' standard for these commit messages has been defined in Conventional Commits. In short, your commit messages start with a keyword, a scope and then a short description of the change. Commit messages should be written in present tense ("closes", not "closed"):

fix(accounts): calculation error on totals, closes #123 to create a Patch Release or
feat(users): create and update user details, closes #123 to create a Minor Release

You can find a quick summary here.

Add Semantic Release and GitHub Actions

Actions is a CI/CD product from GitHub. You can define "workflows" as simple yml files to go through certain steps such as linting, testing, building etc. They work for pretty much any project / programming language. Every Action ususally has a trigger ("on") and jobs. You can read more about Events that trigger workflows, Building and testing Node.js Actions or even Custom JavaScript Actions.

We usually use a simple "test, build & release" Action, for example for our integration Actions like Cloudflare Preview URL.

Below you can find a slightly more complex Action that restricts permissions and runs checkout, npm install, linter, codestyle checks, tests, coverage reports and semantic release. See Hardening GitHub Actions for more information about security/permission steps taken in this Action.

./github/workflows/publish.yml:

name: Semantic Release

on:
  push:
    branches:
      - 'main'

permissions:
  actions: none
  checks: none
  contents: write
  deployments: write
  issues: read
  packages: none
  pull-requests: write
  repository-projects: none
  security-events: none
  statuses: none

jobs:
  publish:
    env:
      CI: true
      NODE_ENV: test

    runs-on: ubuntu-latest
    timeout-minutes: 15

    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v2.4.1
        with:
          node-version: '16'
          cache: 'npm'
      - name: Install Dependencies
        run: npm ci --ignore-scripts
      - name: Run Linter
        run: npm run lint
      - name: Validate Codestyle
        run: npm run codestyle
      - name: Test
        run: npm test
        env:
          DISABLE_REQUEST_LOGGING: true
          LOG_LEVEL: error
      - name: report coverage
        uses: codecov/codecov-action@v2.1.0
        with:
          token: ${{ secrets.CODECOV_TOKEN }}
      - name: Semantic Release
        run: npx semantic-release
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

The magic line to convert our Conventional Commit messages into Releases is run: npx semantic-release. This will run semantic-release after the tests, lint etc. are done and generate a release with fancy notes based on the commits that have been merged. For example, if you currently have a v1.2.0 deployed, and merge a fix, semantic-release will publish v1.2.1.

You can see how Semantic Releases look like in our open source repos. The important piece here is, that each release also comes with a tag. This is interesting and useful for a variety of reasons, for example compliance ("what has been deployed when?"), and of course the developer experience (DX). You can see the currently deployed version in each environment, without checking cryptic 6-digit SHAs. The tag is also the ingredient required for the next step:

Automated Deployment

In Google Cloud Build (or AWS CodeBuild or Azure Pipelines) you can select a source repository for deployments and pick "tags" as a build trigger. .v* for example will deploy every new release. If you want to deploy a Major release in a new environment, you can switch the previous trigger settings to .v1.* to only deploy v1 releases, but not v2. This is a nice way to test breaking changes.

Cloud Build allows you to continously deploy your application. On GCP this works with a cloudbuild.yml file:

steps:
  - name: 'gcr.io/cloud-builders/docker'
    entrypoint: 'bash'
    args:
      - '-c'
      - |
        docker pull us-central1-docker.pkg.dev/$PROJECT_ID/api-demo/api:latest || exit 0
  - name: 'gcr.io/cloud-builders/docker'
    args:
      - build
      - -t
      - us-central1-docker.pkg.dev/$PROJECT_ID/api-demo/api:$TAG_NAME
      - -t
      - us-central1-docker.pkg.dev/$PROJECT_ID/api-demo/api:latest
      - .
      - --cache-from
      - us-central1-docker.pkg.dev/$PROJECT_ID/api-demo/api:latest
  - name: 'gcr.io/cloud-builders/docker'
    args:
      ['push', 'us-central1-docker.pkg.dev/$PROJECT_ID/api-demo/api:$TAG_NAME']
  - name: 'gcr.io/cloud-builders/gcloud'
    args:
      - run
      - deploy
      - api-demo
      - --image=us-central1-docker.pkg.dev/$PROJECT_ID/api-demo/api:$TAG_NAME
      - --region=us-central1
      - --memory=256Mi
      - --platform=managed
      - --allow-unauthenticated
      - --min-instances=0
      - --max-instances=5
images:
  - 'us-central1-docker.pkg.dev/$PROJECT_ID/api-demo/api:$TAG_NAME'
  - 'us-central1-docker.pkg.dev/$PROJECT_ID/api-demo/api:latest'
timeout: 1800s

$TAG_NAME is the version number, so all your images and deployments have a semantic version number and you can clearly identify what's running where. There are four steps here:

fetch last image from cache (if there's a cached image)
build the code and Dockerfile
push the image to the registry
deploy the image on Cloud Run

Note: We're using the new Artifact Registry instead of the deprecated Container Registry, so the registry urls are slightly different. You can create your Artifact Repository here and adjust the registry region in the cloudbuild file.

Go live on managed k8s

Google Cloud, AWS and Azure all have managed Kubernetes products which are great to focus on software development and leave the k8s configs, helm charts and autoscale yamls to the DevOps Pros. In the previous section we defined a step to deploy to Cloud Run. For GCP, you may need to enable the API, billing etc. on your project:

enable Billing (requires a credit card)
enable the Cloud Build API
enable the Cloud Run Admin API
enable the Artifact Registry API
create a Repository in the Artifact Registry (Format: Docker, Region: your choice)

If the build is green, you should see a new service in the Cloud Run console. When you click on the service, at the top you see the new public URL (ie. app-name-123.run.app) for your service. Link a domain to that URL and 🎉, your API is now live.

Summary

Let's recap a everyday workflow:

Developer picks an issue and works on this amazing new feature for your product
Developer pushes changes to a feature branch and opens a Draft Pull Request. A Pull Request Preview is automatically deployed via the GitHub/Google Cloud Build integration and changes can be tested
Developer marks the Pull Request as "Ready for Review"
Team reviews the PR, discusses the changes and approves (hopefully ;))
Pull Request is merged into main by the developer or reviewer
GitHub publish.yml Action is triggered ("push to main") and runs tests, lint etc.
If everything goes well, npx semantic-release executes, calculating a new version number, summarizing the release notes and creates a new tag and release
Google Cloud Build trigger is fired on "new tag", builds your app and pushes the Docker image to the Artifact Registry
Cloud Build starts the Cloud Run deployment with the new image (which has the version number)
Cloud Run starts the new service and changes the traffic allocation to the new instance
The change is live, start over with the next feature 😎

Cloud Run enables you to split traffic between multiple revisions, so you can perform gradual rollouts such as canary deployments or blue/green deployments. Combined with automatic rollbacks, you can skip your staging environment alltogether and continously deploy to production. Developers can easily revert changes by merging a commit to main.

Thanks for reading! If you have any questions or comments, please reach out on Twitter or start a discussion on GitHub.

Universal/Isomorphic Web Apps on Google Cloud Run

Patrick — Mon, 28 Sep 2020 11:55:43 +0000

TL;DR

In this article we'll learn how to launch a JavaScript application on Google Cloud Run (fully managed) with:

Server Side Rendering (SSR)
TLS (managed by Google)
Global Content Delivey Network (CDN)
resources and services in the same network (no added network delays)
No cold starts*
No CORS (avoid preflight requests)

We'll also demonstrate how to get close to dev/prod parity with a local development setup.

You can find all relevant configuration files and code example the Universal Apps on Cloud Run GitHub Repo.

The concept of universal/isomorphic apps is that the first page is rendered on the server and delivered to the client in plain HTML and CSS, while additional JavaScript is delivered after, to allow the "application like" usability known from Single Page Applications. By caching the rendered pages on the CDN, we aim for fast initial page load with low First Input Delay/Largest Contentful Paint (Web Vitals). By avoiding CORS preflight requests, we skip the additional OPTIONS request to the API which usually adds additional delay to each ajax request.

This article requires basic knowledge of N*xt.js and Node.js as we'll be building on top of that.

(*) The min-instances setting is currently in Alpha and should be available in Beta soon, which allows to keep a certain number of instances running.

Introduction

Single Page Applications (SPA) are easy for developers and great for many things, but when it comes to web performance and search/SEO scores, Server Side Rendered (SSR) applications still perform much better.

For a recent project, we looked into Cloud Run as an easy-to-use, scalable infrastructure. We chose Nuxt.js and Vue.js for simplicty over Next.js and React.js. Data is delivered by a Node.js API. Next.js with SSR requires a build step and a web server, while the API also requires a separate environment. In this article we call them web and api services.

To achieve a fully automated deployment pipeline we use Google Cloud Build and Semantic Release to version and build Docker images based on our code on GitHub.

Google Cloud Run is an easy and reliable infrastructure for running Docker containers and they recently added Load Balancer and CDN support for the managed service, which means there is zero devops overhead on Kubernetes (k8s); everything is managed by Google. For advanced configurations they still offer Cloud Run for Anthos to screw and tweak with a custom Kubernetes config, but we wanted to focus on product development rather than infrastructure, and Cloud Run (managed) makes that possible.

Universal Web App with N*xt

Nuxt.js

Set up your N*xt App as usual with a generator or boilerplate. For this article we used create-nuxt-app with Universal mode and axios(a Promise based HTTP client) support:

npx create-nuxt-app web
yarn create nuxt-app web

In our example we want to retrieve a travel itinerary from our API by using an axios request like $axios.get('/travels') and add images or other assets with /assets/yourimage.png. We'll add the prefix /api later in our config.

Usually you would need to set up a proxy in axios to rewrite /api/ and /assets to the correct URLs, but this will be handled by Google Cloud's Load Balancer, so there is no need. Instead, we set the environment variables API_URL and API_URL_BROWSER in our local Docker Compose setup to overwrite the axios configuration. These are set in next.config.js to avoid issues with the live version on Google Cloud.

For the local setup to work, BASE_URL and API_URL_BROWSER are set to the nginx proxy, while API_URL is used for the internal SSR requests from Nuxt directly to the API service. On GCP (Google Cloud Platform) these adjustments are not needed.

At this point, the web app isn't working, because the api and assets are not reachable by Next when running npm start, so we'll move on to the other pieces and get back to this later.

Node.js API with Fastify

Fastify is an amazing Node.js framework for API development. It works very similar to Express. It's built for speed and has some great convenience functions built in, such as automatic OpenAPI/Swagger docs generation, input and output schemas and validation and a great plugin system. Here's a basic Node.js server set up with fastify:

const { env } = require('./config')
const Fastify = require('fastify')
const AutoLoad = require('fastify-autoload')
const { join } = require('path')

const app = Fastify({
  disableRequestLogging: true
})

app.register(AutoLoad, {
  dir: join(__dirname, 'routes')
})

if (env === 'development') {
  app.register(require('fastify-http-proxy'), {
    upstream: 'http://localhost:3000',
    prefix: '/api',
    http2: false
  })
}

module.exports = app

In this demo, we are using /routes/travels/list.js to automatically generate an endpoint GET /travels and deliver some travel data. These are locations we'll be travelling to in the coming months of 2020, so if you're nearby, give us a shout.

Two things which are important here:

disableRequestLogging - Google Cloud Run does that already, so there's no need to log requests in Fastify
fastify-http-proxy - this is a little tricky. In our Docker Compose environment, all internal requests (requests posted by next directly to the API for server-side rendering) still have /api/ in their path, so we need to proxy /api/travels to /travels with this little hack. For external requests, our nginx server and Google Load Balancer rewrite the path.

This should be straightforward. Once your API delivers data on localhost:3000/travels, let's move to deployment.

Google Cloud

Before we start with the deployment, you need to set up gcloud and create a project on Google Cloud:

Install gcloud CLI https://cloud.google.com/sdk/gcloud
Create a new project on https://console.cloud.google.com/
Set the default project: gcloud config set project universal-apps-cloud-run-demo

You'll need the following services activated:

Cloud Build API (https://console.cloud.google.com/marketplace/product/google/cloudbuild.googleapis.com)
Cloud Run API (https://console.cloud.google.com/run/create)
Network Services / Load Balancing (https://console.cloud.google.com/net-services/loadbalancing/loadBalancers/list)
Container Registry (https://console.cloud.google.com/gcr/images/)

Deployment Pipeline

We'd recommend to use one repository for the api and one for the web service, but for this demo we put both services in one. Once you have the repos set up and pushed the code, go to the GitHub Marketplace and install the Google Cloud Build App. Add the repositories to the integration and connect the GCP projects. When you separate into two projects, don't forget to change the cloudbuild.yaml in both projects to build from root, instead of a folder (- web/. and - api/. to .)

In this article, we skip the Semantic Release setup, you can read and adjust this from a previous article Serverless 2.0.0.

We create two triggers for deployment: one for web and one for api.

You also need to give permissions to Cloud Build to deploy on Cloud Run:

Web requires a substitition variable _API_URL set to the domain you want to run the service on. We're using https://demo.zentered.io/api here. A substition variable is an environment variable that you set during build time. You can deploy the same code with various API_URLs to test new releases for example.

It's important to note that the build on Cloud Build happens with NODE_ENV=production, this means your devDependencies are not installed. Make sure you have all build-dependencies in the dependencies of your package.json.

Once this is done, you can push to your main branch and watch Cloud Build deploy your services to Cloud Run:

If you work with Semantic Release, a GitHub Action can create new Tags/Releases for you. Modify the trigger to build releases only.

Services on Google Cloud Run

Google Cloud Run (GCR) is a fully managed compute platform for deploying and scaling containerized applications quickly and securely. You can focus on your application, wrap them in a Docker container and let GCR do the rest.

In the service details you'll get a URL for both services which ends with .run.app. You should be able to access both services, but Nuxt will not be able to retrieve data from assets or the API yet.

Assets Bucket

To store images and other assets for your site, head over to Google Cloud Storage, create a public bucket and upload some files. At a later stage you might want to have a build step to copy assets from your web service to Cloud Storage.

Load Balancer with SSL and CDN

Quick recap. We have a deployment pipeline to deploy two services, web and api, automatically to Cloud Run. Pusing to the repo triggers a deployment. Both services are public and have their internal *.run.app URLs. We also have a public bucket for assets etc.

Now we're placing the last piece of the puzzle to make everything work together: the Load Balancer for Serverless Network Endpoint Groups (NEGs).

You will need a domain and an A record to an external IP address from Google Cloud that you can create as follows:

gcloud compute addresses create web \
--ip-version=IPV4 \
--global

Get the IP with:

gcloud compute addresses describe web \
--format="get(address)" \
--global

Feel free to replace europe-west1 with any of the following regions that is closer to you:

asia-east1 (Taiwan)
asia-northeast1 (Tokyo)
asia-northeast2 (Osaka)
europe-north1 (Finland)
europe-west1 (Belgium)
europe-west4 (Netherlands)
us-central1 (Iowa)
us-east1 (South Carolina)
us-east4 (Northern Virginia)
us-west1 (Oregon)

See Cloud Run locations for a full list and pricing.

If you need more information about the individual parts, you can head over to the Google Tutorial. Here's a summary of the commands that need to be executed in order:

Network Endpoint Group (NEG)

gcloud beta compute network-endpoint-groups create web-neg --region=europe-west1 --network-endpoint-type=SERVERLESS --cloud-run-service=web
gcloud beta compute network-endpoint-groups create api-neg --region=europe-west1 --network-endpoint-type=SERVERLESS --cloud-run-service=api

Backend Services

gcloud beta compute backend-buckets create assets --gcs-bucket-name=uwa-demo-bucket --enable-cdn
gcloud beta compute backend-services create web-service --global
gcloud beta compute backend-services add-backend web-service --global --network-endpoint-group=web-neg --network-endpoint-group-region=europe-west1
gcloud beta compute backend-services create api-service --global
gcloud beta compute backend-services add-backend api-service --global --network-endpoint-group=api-neg --network-endpoint-group-region=europe-west1

URL Maps (🧙‍♀)

This is where the magic happens. Create a file url-map.yaml with this content. Replace universal-apps-cloud-run-demo with your project id:

defaultService: https://www.googleapis.com/compute/v1/projects/universal-apps-cloud-run-demo/global/backendServices/web-service
hostRules:
  - hosts:
      - '*'
    pathMatcher: path-matcher-1
kind: compute#urlMap
name: web
pathMatchers:
  - defaultService: https://www.googleapis.com/compute/v1/projects/universal-apps-cloud-run-demo/global/backendServices/web-service
    name: path-matcher-1
    pathRules:
      - paths:
          - /api/
          - /api/*
        routeAction:
          urlRewrite:
            pathPrefixRewrite: /
        service: https://www.googleapis.com/compute/v1/projects/universal-apps-cloud-run-demo/global/backendServices/api-service
      - paths:
          - /assets/*
        routeAction:
          urlRewrite:
            pathPrefixRewrite: /
        service: https://www.googleapis.com/compute/v1/projects/universal-apps-cloud-run-demo/global/backendBuckets/assets
selfLink: https://www.googleapis.com/compute/v1/projects/universal-apps-cloud-run-demo/global/urlMaps/web

Then import the URL map on GCP:

gcloud beta compute url-maps import web --source url-map.yaml --global

We create two url rewrite rules for the load balancer, to route all requests for /api to the api service and all requests to /assets to the storage bucket.

SSL certs

gcloud beta compute ssl-certificates create web-ssl-cert --domains demo.zentered.io

HTTPS Proxy

gcloud beta compute target-https-proxies create web-https-proxy --ssl-certificates=web-ssl-cert --url-map=web

Forwarding Rules

gcloud beta compute forwarding-rules create https-web-rule --address=web --target-https-proxy=web-https-proxy --global --ports=443

Enable CDN

gcloud beta compute backend-services update web-service --enable-cdn --global
gcloud beta compute backend-services update api-service --enable-cdn --global

Tada

You can verify your Load Balancer config here: https://console.cloud.google.com/net-services/loadbalancing/details/http/web. It should show the two backend services, the assets bucket and the IP with SSL on the Frontend. In the Cloud CDN tab, all three backends should be listed.

After a few minutes, your SSL certificate should be ready and your website should show. It might take a few minutes, you can check the status with:

gcloud compute ssl-certificates

GCP Cleanup

To roll back / remove all the resources created before, execute the following steps:

gcloud beta compute forwarding-rules delete https-web-rule --global
gcloud beta compute target-https-proxies delete web-https-proxy
gcloud beta compute url-maps delete web
gcloud beta compute backend-services delete web-service --global
gcloud beta compute network-endpoint-groups delete web-neg --region=europe-west1
gcloud beta compute backend-services delete api-service --global
gcloud beta compute network-endpoint-groups delete api-neg --region=europe-west1
gcloud compute addresses delete web --global

Running locally with Docker Compose

Now as everything is running in Production, we'd like to achieve a similar setup to develop efficiently on our application and API. For SSL and path rewrites (/api and /assets), we use nginx:

Head over to https://github.com/FiloSottile/mkcert and download/install mkcert, this is required to generate SSL certificates for localhost:

cd certs
mkcert localhost 127.0.0.1

This is the entire nginx config with both upstreams (web and api), the path rewrite rules for api and assets, and the SSL configuration:

Last but not least, you'll need Docker Compose to run all services together. Entrypoint for the application is https://localhost:8080 which loads your N*xt app with SSL and redirects all requests to /api/ to the api service.

api service: port 3000
web service: port 5000
nginx ssl proxy: port 8080

You need to explicitly add https:// in the browser, as there's no http-to-https redirect yet.

We use environment variables to overwrite Axios behaviour. Internal requests for server-side rendering are sent to http://api:3000/api, while client-side requests to to https://localhost:8080.

Summary

This article shows how to run universal apps on Google Cloud Run (fully managed) with very little operations overhead. There's no need to SSH into machines or take care about regular system updates, and the system can easily scale by changing the max-instances setting in the configuration or by deploying the same services in new regions.

You can clone/download all parts of this tutorial from our GitHub Repo. Please open an issue if you find anything that is not working in this tutorial or reach out to us on Twitter.

Special Thanks

Steren from the Google Cloud Run team
Roman for helping with the url map
Patrick, James, Yann and Jonas for reviewing.