DEV Community: James Moschou

Adding passkeys to a Supabase app without using third parties

James Moschou — Fri, 30 Aug 2024 04:44:12 +0000

Recently I added passkeys to a minimal Next.js application using Supabase as the
backend.

Supabase Auth makes it incredibly easy to implement user management in your
application. It supports SSO, multi-factor authentication as well as traditional
password authentication out of the box. However, at the time of writing it does
not support signing in with passkeys.

Supabase does make it possible to issue your own JWTs to your users. So
theoretically it is possible to implement a custom authentication method. So I
figured if could implement passkey registration and authentication flows, then I
could simply issue a custom JWT and let the user sign in once their passkey has
been successfully verified.

When I was doing research for this I came across many third-party solutions that
claim to support passkey authentication with Supabase. I was hesitant to use
them, however, as it seems they also want to be the system of record for your
users. I preferred not to go down this route and I was also curious to see how
straightforward it would be to implement passkeys manually. Thankfully there is
an open source library called SimpleWebAuthn that
takes care of most of the logic.

My basic requirements were:

Use Supabase Auth to manage users.

Existing users should still be able to sign in with their password, managed
by Supabase.

Integrate with Supabase Row Level Security.

This means that interacting with Supabase as an authenticated user should use
a user token instead of a service role token.

Enable new and exising users to continue to use passwords or other sign-in methods.

At the moment passkeys are advertised to end users as an optional
convenience. Users should not feel obliged to adopt them. Additionally, it is
important to maintain existing sign-in methods for account recovery flows.

Automatically refresh the user's session.

The custom access token should be automatically refreshed to mirror the
default behaviour of a Supabase-managed session.

Here's how I did it...

Application setup

To create the base application I followed the
official guide
for using Supabase with Next.js. I implemented sign up, sign in, sign out and
reset password flows.

The database schema

We're going to need two new database tables to implement passkeys. One for the
passkey data itself, and another to store the unique challenges.

I created a new schema in the Supabase database called webauthn to act as a
namespace for everything related to the WebAuthn specification.

Note: A quick note on terminology. The word "passkey" is a common noun like
"password". It doesn't refer to any particular specification or data
structure. The specification that defines how user agents (browsers),
authenticators (e.g. Face ID) and relying parties (servers) is called
WebAuthn. The WebAuthn specification describes multiple types of credentials
that can be used to authenticate users. A passkey is a public-key credential
that is discoverable.

First create the schema:

create schema webauthn;

The credentials table will store public key information once they have been
verified. This corresponds to the
Credential Record in the
specification.

create type webauthn.credential_type AS ENUM ('public-key');
create type webauthn.user_verification_status AS ENUM ('unverified', 'verified');
create type webauthn.device_type AS ENUM ('single_device', 'multi_device');
create type webauthn.backup_state AS ENUM ('not_backed_up', 'backed_up');

create table webauthn.credentials (
  id                       uuid not null default gen_random_uuid(),
  user_id                  uuid not null default auth.uid(),
  friendly_name            text,
  credential_type          webauthn.credential_type not null,
  credential_id            varchar not null,
  public_key               bytea not null,
  aaguid                   varchar default '00000000-0000-0000-0000-000000000000'::varchar not null,
  sign_count               integer not null,
  transports               text[] not null,
  user_verification_status webauthn.user_verification_status not null,
  device_type              webauthn.device_type not null,
  backup_state             webauthn.backup_state not null,
  created_at               timestamptz default now() not null,
  updated_at               timestamptz default now() not null,
  last_used_at             timestamptz,
  constraint credentials_pkey primary key (id),
  constraint credentials_credential_id_key unique (credential_id),
  constraint credentials_user_id_fkey foreign key (user_id) references auth.users (id) on delete cascade
);

create unique index credentials_pkey on webauthn.credentials (id uuid_ops);
create unique index credentials_credential_id_key on webauthn.credentials (credential_id text_ops);

The registration and authentication flows both work by issuing a challenge to
the user, which is signed by the authenticator and returned back to the server
to be verified.

The challenges table will store each challenge used to register or
authenticate a passkey.

create table webauthn.challenges (
  id         uuid not null default gen_random_uuid(),
  user_id    uuid null default auth.uid(),
  value      text not null,
  created_at timestamptz not null default now(),
  constraint challenges_pkey primary key (id),
  constraint challenges_value_key unique (value),
  constraint challenges_user_id_fkey foreign key (user_id) references auth.users (id) on delete cascade
);

create unique index challenges_pkey on webauthn.challenges (id uuid_ops);
create unique index challenges_value_key on webauthn.challenges (value text_ops);

The user_id column is nullable, because we will not know who the user is when
they are signing in.

Configuration

The backend will require some configuration variables:

// src/webauthn/config.ts

const relyingPartyID = process.env.WEBAUTHN_RELYING_PARTY_ID
const relyingPartyName = process.env.WEBAUTHN_RELYING_PARTY_NAME
const relyingPartyOrigin = process.env.WEBAUTHN_RELYING_PARTY_ORIGIN

export { relyingPartyID, relyingPartyName, relyingPartyOrigin }

The Relying Party ID should be based on your host's domain name, without the
https:// or port number. For example, example.com or localhost.

Passkey registration overview

As passkeys are an opt-in experience, the first thing to implement is the
ability for an already signed-in user to create a new passkey for themselves.

At a high level, the passkey registration flow looks like this:

The client sends an API request to the server to generate a unique challenge and passkey creation options.
The client creates the passkey on the user's device using the retrieved creation options.
The client sends the authenticator attestation response to the server.
The server verifies the attestation response against the challenge that was generated earlier.
Upon successful verification, the server stores the new credential in the users account.

Start the passkey registration flow

When the user clicks Create Passkey, the application will send a POST request to
the server to obtain the
PublicKeyCredentialCreationOptions
dictionary. This dictionary contains a crypographic challenge, which is
generated on the server.

We need to add a new route handler to the Next.js app. Since this is an
authenticated endpoint, we need to make sure the user is signed in:

// src/app/api/passkeys/challenge/route.ts

import { createClient } from '@/utils/supabase/server'

export async function POST() {
  const supabase = createClient()
  const {
    data: { user }
  } = await supabase.auth.getUser()
  if (!user) {
    return NextResponse.json({ error: 'Not authenticated' }, { status: 401 })
  }

  // ...
}

Now we can create the options dictionary using the
generateRegistrationOptions() function.

import { generateRegistrationOptions } from '@simplewebauthn/server'

const options = await generateRegistrationOptions({
  rpName: relyingPartyName,
  rpID: relyingPartyID,
  userName: user.email,
  userDisplayName: user.user_metadata.display_name,
  attestationType: 'none',
  authenticatorSelection: {
    residentKey: 'preferred',
    userVerification: 'preferred',
    authenticatorAttachment: 'platform'
  }
})

By default, SimpleWebAuthn generates it's own user IDs for privacy. I've opted
to use the existing ID from the auth.users table, as it is already a UUID and
doesn't contain any personally identifying information. You can read more about
how to use custom user IDs
here.

import { isoUint8Array } from '@simplewebauthn/server/helpers'

const options = await generateRegistrationOptions({
  // ...
  userID: isoUint8Array.fromASCIIString(user.id)
})

While the generateRegistrationOptions() takes care of generating the
cryptographic challenge, we still have to save it so we can retrieve it later in
the verify route handler.

import { saveWebAuthnChallenge } from '@/webauthn/store'

const challenge = await saveWebAuthnChallenge({
  user_id: user.id,
  value: options.challenge
})

Finally, we return the options dictionary to the browser:

return NextResponse.json(options, { status: 200 })

We can improve this to prevent the user from re-registering the same credential
twice. All we have to do is pass any existing credentials in the
excludeCredentials property when calling generateRegistrationOptions():

import { listWebAuthnCredentialsForUser } from '@/webauthn/store'

const credentials = await listWebAuthnCredentialsForUser(user.id)

const options = await generateRegistrationOptions({
  // ...
  excludeCredentials: credentials.map((credential) => ({
    id: credential.credential_id,
    type: credential.credential_type,
    transports: credential.transports
  }))
})

Complete the passkey registration flow

Once the user has created a new passkey on their device, the browser will call
our verify endpoint to register the passkey.

The first thing to do is to retrieve the challenge that was created earlier.
Since the user is already authenticated during this flow, we can look the
challenge up by its user_id field:

// src/app/api/passkeys/verify/route.ts

import { getWebAuthnChallengeByUser } from '@/webauthn/store'

export async function POST(request: NextRequest) {
  const supabase = createClient()
  const {
    data: { user }
  } = await supabase.auth.getUser()
  if (!user) {
    return NextResponse.json({ error: 'Not authenticated' }, { status: 401 })
  }

  const challenge = await getWebAuthnChallengeByUser(user.id)

  // ...
}

Challenges should only be valid for one attempt in order to prevent replay
attacks. Once the challenged has been retrieved, immediately delete it from the
database regardless of whether it will be sucessfully verified or not:

import { deleteWebAuthnChallenge } from '@/webauthn/store'

await deleteWebAuthnChallenge(challenge.id)

Now we can call the verifyRegistrationResponse() function with the attestation
response received from the client and the expected challenge:

import { verifyRegistrationResponse } from '@simplewebauthn/server'
import { relyingPartyID, relyingPartyOrigin } from '@/webauthn/config'

const data = await request.json()
const verification = await verifyRegistrationResponse({
  response: data,
  expectedChallenge: challenge.value,
  expectedOrigin: relyingPartyOrigin,
  expectedRPID: relyingPartyID
})

const { verified } = verification
if (!verified) {
  return NextResponse.json(
    { error: 'Could not verify passkey' },
    { status: 401 }
  )
}

If the credential was verified successfully we can store it in our database:

import { saveWebAuthnCredential } from '@/webauthn/store'

const { registrationInfo } = verification

const values = {
  user_id: user.id,
  friendly_name: `Passkey created ${new Date().toLocaleString()}`,

  credential_type: registrationInfo.credentialType,
  credential_id: registrationInfo.credentialID,

  public_key: registrationInfo.credentialPublicKey,
  aaguid: registrationInfo.aaguid,
  sign_count: registrationInfo.counter,

  transports: data.response.transports ?? [],
  user_verification_status: registrationInfo.userVerified
    ? 'verified'
    : 'unverified',
  device_type:
    registrationInfo.credentialDeviceType === 'singleDevice'
      ? 'single_device'
      : 'multi_device',
  backup_state: registrationInfo.credentialBackedUp
    ? 'backed_up'
    : 'not_backed_up'
}

const savedCredential = await saveWebAuthnCredential(values)

Finally, we can return some data to the browser so it can update the user
interface of the Settings page.

const passkeyDisplayData = {
  credential_id: savedCredential.credential_id,
  friendly_name: savedCredential.friendly_name,

  credential_type: savedCredential.credential_type,
  device_type: savedCredential.device_type,
  backup_state: savedCredential.backup_state,

  created_at: savedCredential.created_at,
  updated_at: savedCredential.updated_at,
  last_used_at: savedCredential.last_used_at
}

return NextResponse.json(passkeyDisplayData, {
  status: 201,
  headers: {
    Location: `/api/passkeys/${savedCredential.id}`
  }
})

The registration user interface

The Settings page contains a typical data table to show existing passkeys and a
Create Passkey button.

The page route is a server component that fetches whatever passkeys already
exist for the user and passes them to a client component:

// src/app/dashboard/settings/security/page.tsx

import { createClient } from '@/supabase/server'
import { listWebAuthnCredentialsForUser } from '@/webauthn/store'
import { redirect } from 'next/navigation'
import { PasskeysSection } from './passkeys-section'

export default async function SecuritySettingsPage() {
  const supabase = createClient()
  const {
    data: { user }
  } = await supabase.auth.getUser()
  if (!user) {
    redirect('/signin')
  }

  const credentials = await listWebAuthnCredentialsForUser(user.id)
  const passkeys = credentials.map((credential) => ({
    credential_id: credential.credential_id,
    friendly_name: credential.friendly_name,
    credential_type: credential.credential_type,
    device_type: credential.device_type,
    backup_state: credential.backup_state,
    created_at: credential.created_at,
    updated_at: credential.updated_at,
    last_used_at: credential.last_used_at
  }))

  return <PasskeysSection initialPasskeys={passkeys} />
}

Define a client function to perform the end-to-end registration flow. Calling
this function will either return the new passkey if it's successful or throw an
error:

// app/webauthn/client.ts

import { startRegistration } from '@simplewebauthn/client'
import { sendPOSTRequest } from './helpers'

export async function createPasskey() {
  const options = await sendPOSTRequest('/api/passkeys/challenge')
  const credential = await startRegistration(options)
  const newPasskey = await sendPOSTRequest('/api/passkeys/verify', credential)
  if (!newPasskey) {
    throw new Error('No passkey returned from server')
  }
  return newPasskey
}

This function will be called from the onClick handler of the Create Passkey
button:

// src/app/dashboard/settings/security/passkeys-section.tsx

import { useState } from 'react'
import { toast } from 'sonner'
import { createPasskey } from '@/webauthn/client'

export default function PasskeysSection({
  initialPasskeys
}: {
  initialPasskeys: {
    // ...
  }[]
}) {
  const [passkeys, setPasskeys] = useState(initialPasskeys)
  const [creating, setCreating] = useState(false) // controls loading indicator

  const handleCreatePasskey = async () => {
    try {
      setCreating(true)
      const passkey = await createPasskey()
      setPasskeys((prev) => [...prev, passkey])
    } catch (error) {
      if (error instanceof Error) {
        if (error.name === 'NotAllowedError') {
          // This request has been cancelled by the user.
          return
        }
        toast(error.message)
      }
    } finally {
      setCreating(false)
    }
  }
}

When the button is clicked, the created passkey is optimistically added to the
table's state.

If the user explicitly aborts the flow, a NotAllowedError error will be
thrown. The handler needs to explictly check for this case in order to avoid
showing an error message to the user.

That's it! Now a user can sign in and create their own passkeys.

Passkey authentication overview

The flow for authenticating with a passkey is quite similar to the registration
flow. We'll need two endpoints to start and complete the flow, as well as some
client-side code to perform the flow.

At a high level, the passkey authentication flow looks like this:

The client sends an API request to the server to generate a unique challenge and passkey authentication options.
The client signs the challenge on the user's device using their passkey.
The client sends the authenticator assertion response to the server.
The server verifies the assertion response against the challenge that was generated earlier.
Upon successful verification, the server generates an access token for the user.

The main difference is that the user will not be authenticated in the endpoint
handlers. The verify endpoint will need to determine who the user is by looking
them up from the matched credential.

This presents a major issue: how do you associate the cryptographic challenge
with the user if you don't know who they are yet? Most passkey tutorials that I
could find on the web assume that your web framework provides some sort of
mutable session object, which you can attach the challenge to. There is no
such session object in Next.js app directory route handlers, so I opted to
implement a simple cookie-based session.

Start the passkey authentication flow

I initially allowed users to sign in with a passkey by adding a Sign In with
Passkey button to the sign-in page. When the user clicks this button the
application will send a POST request to the server to obtain the
PublicKeyCredentialRequestOptions
dictionary.

The route handler will be roughly the same as the start registration route
handler. Since the user will not be authenticated, so there is no need to
validate the supabase user:

// src/app/auth/passkey/route.ts

import { generateAuthenticationOptions } from '@simplewebauthn/server'
import { relyingPartyID } from '@webauthn/config'
import { saveWebAuthnChallenge } from '@webauthn/store'

export async function POST() {
  const options = await generateAuthenticationOptions({
    rpID: relyingPartyID
  })

  const challenge = await saveWebAuthnChallenge({
    value: options.challenge
  })

  // Store the challenge ID in the "session"
  cookies().set('webauthn_state', challenge.id, {
    httpOnly: true,
    sameSite: true,
    secure: !process.env.LOCAL
  })

  return NextResponse.json(options, { status: 200 })
}

Complete the passkey authentication flow

The verify router handler is also similar to its registration counterpart.

First retrieve the challenge using its ID stored in the "session".

// src/app/auth/verify/route.ts

import { getWebAuthnChallenge } from '@/webauth/store'

export async function POST(request: NextRequest) {
  const challengeID = cookies().get('webauthn_state')?.value
  const challenge = await getWebAuthnChallenge(challengeID)
}

Again, we should delete the challenge immediately to prevent replay attacks.

import { removeWebAuthnChallenge } from '@/webauth/store'

await removeWebAuthnChallenge(challengeID)

Next, we can retrieve the credential that was alledgedly used. I say alledgedly
because we haven't verified anything yet.

The request body will contain an id field which will correspond to the
credential_id field of the credential. Be careful! Here the credential_id
field is the one set by the authenticator, which is different to our internal
primary key.

import { getWebAuthnCredentialByCredentialID } from '@/webauth/store'

const data = await request.json()
const credential = await getWebAuthnCredentialByCredentialID(data.id)
if (!credential) {
  return NextResponse.json(
    { error: 'Could not sign in with passkey' },
    { status: 401 }
  )
}

With the expected credential in hand, we can verify the authentication response,
and return the result to the browser.

import { verifyAuthenticationResponse } from '@simplewebauthn/server'
import { relyingPartyID, relyingPartyOrigin } from '@/webauthn/config'

const verification = await verifyAuthenticationResponse({
  response: data,
  expectedChallenge: challenge.value,
  expectedOrigin: relyingPartyOrigin,
  expectedRPID: relyingPartyID,
  authenticator: {
    credentialID: credential.credential_id,
    credentialPublicKey: credential.public_key,
    counter: credential.sign_count,
    transports: credential.transports
  }
})

const { verified } = verification

Before we return the result to the browser, there is some housekeeping we need
to do. Namely, we need to update the sign_count and last_used_at fields on
the credential record in the database:

import { sql } from 'drizzle-orm'

if (verified) {
  await updateWebAuthnCredentialByCredentialID(credential.credential_id, {
    sign_count: verification.authenticationInfo.newCounter,
    last_used_at: sql`now()`
  })
}

The authentication user interface

Define a client function to perform the end-to-end authentication flow. Calling
this function will either return a successful result or throw an error:

// app/webauthn/client.ts

import { sendPOSTRequest } from './helpers'
import { startAuthentication } from '@simplewebauthn/client'

export async function signInWithPasskey(
  useBrowserAutofill?: boolean = false
): Promise<void> {
  const options = await sendPOSTRequest('/auth/passkey')
  const authenticationResponse = await startAuthentication(
    options,
    useBrowserAutofill
  )
  const { verified } = await sendPOSTRequest(
    '/auth/verify',
    authenticationResponse
  )
  if (!verified) {
    throw new Error('Could not sign in with passkey')
  }
  return { verified }
}

This function sould be called from the onClick handler of the Sign In with
Passkey button.

Issue an access token to the user

Currently a user can create and use their own passkey. But nothing happens when
they sign in. That's because we still need to create a new user session.

In order for a user to sign in to Supabase and access data using Row Level
Security, we need to issue our own JWT to the user. Fortunately, Supabase
supports this use case and provides the same JWT signing secret that they use
within the dashboard.

In our application we'll define two new environment variables in order to issue
our own JWTs:

SUPABASE_AUTH_JWT_SECRET=secret_copied_from_supabase_dashboard
SUPABASE_AUTH_JWT_ISSUER=https://exampleapp.com/webauthn

Now we can create a function to build a JWT payload and sign it using our
secret:

// src/webauthn/session.ts

import { User } from '@supabase/supabase-js'
import jwt from 'jsonwebtoken'

const jwtSecret = process.env.SUPABASE_AUTH_JWT_SECRET
const jwtIssuer = process.env.SUPABASE_AUTH_JWT_ISSUER

export function createWebAuthnAccessTokenForUser(user: User) {
  const issuedAt = Math.floor(Date.now() / 1000)
  const expirationTime = issuedAt + 3600 // 1 hour expiry
  const payload = {
    iss: jwtIssuer,
    sub: user.id,
    aud: 'authenticated',
    exp: expirationTime,
    iat: issuedAt,

    email: user.email,
    phone: user.phone,
    app_metadata: user.app_metadata,
    user_metadata: user.user_metadata,
    role: 'authenticated',
    is_anonymous: false
  }

  return jwt.sign(payload, jwtSecret, {
    algorithm: 'HS256',
    header: {
      alg: 'HS256',
      typ: 'JWT'
    }
  })
}

We can deliver this access token to the user agent by calling setSession() on
the supabase client. This will take care of storing it in the appropriate cookie
in the user's browser:

// src/webauthn/session.ts

import { createClient } from '@/supabase/server'

export async function createWebAuthnSessionforUser(user: User) {
  const accessToken = createWebAuthnAccessTokenForUser(user)

  const supabase = createClient()
  const { error } = await supabase.auth.setSession({
    access_token: accessToken,
    refresh_token: '' // dummy value
  })
  if (error) {
    throw error
  }
}

The advantage of calling setSession() is that it will store the custom access
token the same was that Supabase-issued tokens are stored. This should mean that
Supabase clients will work transparently.

A major downside of storing the access token this way is that Supabase clients
will not be able to refresh them, as they do not actually correspond to any
session in the auth.sessions table.

A possible enhancement would be to define our own sessions and
refresh_tokens tables within the webauthn schema, mirroring those in the
auth schema. Then when we initialise the Supabase client, we could supply a
custom accessToken() function that will take care of refreshing the session
for us.

Automatically suggest a passkey friendly name

We can make the experience of creating a new passkey a bit nicer by
automatically suggesting a friendly name instead of "Passkey created {date}".

To do this we'll make use of the Authenticator Attestation GUID, which is
contained in the aaguid field of the attestation object. This value describes
the make and model of the authenticator used to create the passkey. For example,
"iCloud Keychain".

The FIDO alliance maintains a list of metadata statements for known
authenticators. We can use the SimpleWebAuthn library to conveniently look up a
metadata statement by AAGUID.

Unfortunately, when I was testing this on my MacBook, the statement
corresponding to iCloud Keychain was not included. For this we have to go to a
community supported database at
https://github.com/passkeydeveloper/passkey-authenticator-aaguids.

Altogether, our function for retrieving a default friendly name given an AAGUID
looks like this:

// src/webauthn/metadata.ts

import { MetadataService } from '@simplewebauthn/server'
import additionalMetadata from './additional-aaguids.json'

MetadataService.initialize({ verificationMode: 'permissive' }).then(() => {
  console.log('MetadataService initialized')
})

export async function authenticatorDescriptionWithAAGUID(aaguid: string) {
  const statement = await MetadataService.getStatement(aaguid)
  if (statement) {
    return statement.description
  }
  return (additionalMetadata as Record<string, { name: string }>)[aaguid].name
}

Now when registering a new passkey, we can provide a better default friendly
name:

// src/api/passkeys/verify/route.ts

const { registrationInfo } = verification

const description = await
authenticatorDescriptionWithAAGUID(registrationInfo.aaguid)

const friendly_name =
  description ?? `Passkey created ${new Date().toLocaleString()}`

Sign in with Passkey via browser autofill

Another user experience enhancement is to implement
Conditional UI.

This lets us remove the Sign in with Passkey button from the sign-in page and
replace it with an autofill prompt that appears when the user focuses the
username input.

Adopting Conditional UI with the SimpleWebAuthn library is a matter of passing
true for the the useBrowserAutofill argument of the startAuthentication()
function.

// Conditional UI
useEffect(() => {
  let cancelled = false
  setTimeout(() => {
    if (cancelled) {
      return
    }
    signInWithPasskey(email, true /* useBrowserAutofill */)
      .then(() => {
        router.push('/dashboard')
      })
      .catch((error) => {
        if (error instanceof Error) {
          if (error.name === 'NotAllowedError') {
            return
          }
          console.error(error)
          toast(error.message)
        }
      })
  }, 0)
  return () => {
    cancelled = true
  }
}, [])

Warning: React Strict Mode is enabled during development, which causes all
effects to be run twice. I found that calling navigator.credentials.get()
twice on Safari caused the macOS Sign In dialog to become undismissable. To
work around this I wrapped the entire effect inside a timeout and only proceed
if the effect was not cleaned up.

Summary

It's entirely possible to implement passkeys in a Supabase app. The
SimpleWebAuthn library makes it incredibly easy to implement the specification.
Most of the implementation is just wiring it up.

I chose to add passkey registration to the Settings page. A better experience
would be to present an interstitial to the user immediately after they sign in
with a password. The user should also be to able to enter their own friendly
name in order to recognise the passkey later.

Because the application session is implemented using a custom access token,
Supabase cannot automatically refresh it. The next step would be to generate a
refresh token along with the access token and implement session autorefresh.
That is outside the scope of this article though!

Making headless components easy to style

James Moschou — Thu, 22 Aug 2024 00:48:07 +0000

Is a headless component simply an unstyled component, or is there more to it?

The web already separates style from content by requiring styles to be defined
in CSS instead of HTML. This architecture allows each web page to adopt a global
design standard without defining any page-specific styles.

As the web evolved into an application platform, developers sought ways to make
their growing codebases more maintainable. Nowadays, the defacto strategy for
organising application code is to define small, lightweight components that can
be composed together. Thus, the component became the unit of composition in
modern web development.

Components often define both their HTML and CSS in the interest of encapsulation.
While this makes them easier to compose, they can be more difficult to
incorporate into an existing design system cohesively. This is especially true
for third-party components that are imported from external vendors.

Headless components solves this challenge by reintroducing a separation between
content and style. However now the separation is along the component boundary as
opposed to between HTML and CSS. They key to creating a great headless component
lies in designing the component's interface such that a developer can
clearly and easily apply their own styles.

Forward relevant props

In the most basic sense, a headless component is simply an unstyled component.
Developers must be able to apply their own CSS to the HTML elements that the
component defines.

For simple components, this may simply be a matter of forwarding the className
prop to the root element so that developers can use class selectors in their
CSS.

If your component has the same semantics as a native HTML element, you can use
the ComponentProps type from React to ensure that all relevant props are
forwardable. Remember to omit any props that you don't want the user of
your component to be able to override.

import { type ComponentProps } from 'react'

function SubmitButton({ ...props }: Omit<ComponentProps<'button'>, 'type'>) {
  return <button type="submit" {...props} />
}

Provide predefined classes

For components that contain one or more child elements, developers will probably
want to style each element individually.

One strategy to support this is to rely on
CSS combinators.
For example, a headless gallery component might be styled like this:

/* Root container */
.gallery {
}

/* Gallery items container */
.gallery > ul {
}

/* Gallery item */
.gallery > ul > li {
}

/* Next and Previous buttons */
.gallery button {
}

But this approach creates a huge problem because now the internal HTML structure of
the component is part of its public API. This prevents you from modifying the
structure later without potentially breaking downstream code.

A better strategy is to predefine classes for each major child element. This way
developers can use class selectors without depending on any particular HTML
structure:

.xyz-gallery {
}

.xyz-gallery-next-button {
}

.xyz-gallery-previous-button {
}

.xyz-gallery-items-container {
}

.xyz-gallery-item {
}

Remember to prefix your classes so that they don't clash with the
developer's own styles.

Support custom layouts

Providing predefined classes is perhaps the quickest way to enable developers to
style your component. However, a disadvantage with this approach is that the
HTML structure cannot be customised.

This may not matter. After all, plain HTML is already pretty flexible in how it
can be rendered. However sometimes developers reach for additional HTML in order
to accomplish certain designs. If you view the source code for almost any
website, you can expect to see a multitude of unsemantic <div/> elements,
whose sole purpose is to define flex or grid layouts, visually group child
elements within a border or create new stacking contexts.

You can support such uses cases by splitting your headless component up into
multiple related components. This way developers are free to add their own
layout elements to the component. For example, a developer could embed the Next and
Previous buttons from the gallery example within a custom flexbox container:

<Gallery>
  <GalleryItems className='gallery-items-container'>
    {data.map((item) => (
      <GalleryItem key={item.id}>{item.content}</GalleryItem>
    ))}
  </GalleryItems>
  <div className='gallery-buttons-container'>
    <GalleryPreviousButton>
    <GalleryNextButton>
  </div>
</Gallery>

.gallery-items-container {
}

.gallery-buttons-container {
  display: flex;
  gap: 0.5rem;
  justify-content: flex-end;
}

These kinds of components are typically implemented using
context to pass
data between themselves. They require more work to design, implement and
document. However, their resulting versatility often means the extra effort is
worth it.

Allow components to be overridden

A small number of use cases require that a headless component manages the layout
of its child components. An example might be a heirarchical tree view that
allows its items to be reordered via drag and drop. Another use case might be to
allow single-page applications to replace the default anchor element with a
custom link component that facilitates client-side routing.

An advanced strategy for allowing developers to define custom layouts is to
allow the actual child component being rendered to be overriden via props:

<TreeView
  nodes={[...]}
  components={{
    CustomRow,
    CustomDragPreview: (props) => <div className="drag-preview" {...props} />
  }}
/>

This grants the developer full control over what is rendered in each child
component, while allowing the headless component to manage its overall
structure.

You can even allow developers to customise the root element of your component
via a prop. For example, this button component allows a developer to render it
as something else:

import { type ElementType } from 'react'

function HeadlessButton({ as, ...props }: { as?: ElementType }) {
  const Component = as ?? 'button'
  return <Component {...props} />
}

For example, in order for assistive technology to treat the button like a link,
the developer can specify that an anchor element should be used to render the
button:

<HeadlessButton as="a">Actually a link</HeadlessButton>

Summary

Headless components are much more than components that don't contain any
styles. Great headless components are fully extensible and allow the developer
to customise the entire internal HTML structure.

REST API date format best practices

James Moschou — Tue, 09 May 2023 03:40:20 +0000

Date and time information is so common in APIs that they can make or break your API's developer experience.

Most people have an intuitive concept of dates and times, based on their culture, educational background and life experience. Unfortunately, this natural intuition can bias people towards designing poor APIs that don't scale across use cases, geographies or end users.

Different use cases require different approaches

There is a lot of overly simplistic advice within engineering practice, such as "always normalize dates to UTC" or "always use "ISO 8601". This is good advice for many use cases, but not all. Product management can help engineering understand the API's specific use case in detail so that the team can make informed decisions on how to design the API together.

Use cases for date and time information fields usually fall under two broad categories: system dates and dates used by humans. The use case that your field falls into informs how you design it.

System dates are generated on the server

System dates mark the precise time certain events took place within the system. System dates are also commonly referred to as "timestamps". Two ubiquitous examples are the times a record was created and modified.

System dates enable use cases such as sorting a list of records chronologically or auditing the system accurately.

When it comes to system dates, millisecond-level precision and unambiguity are must-have requirements. Being able to make before and after comparisons are more important than being able to describe, say, which day of the week the event happened on.

To summarise the characteristics of system dates:

Generated by the system
Millisecond-level precision
Unambiguous interpretation

Dates used by humans are interpreted within a context

Dates used by humans typically represent real-world events and are usually entered into the system by the user. Examples include dates of birth, arrival and departure times for flights, a recurring time to turn on your home's heating, and so on.

Unlike system dates, they will almost never need to be specified to the millisecond. In fact, they may not have a time at all. It all depends on the situation.

It is impossible to separate a date entered by a user from the user's cultural context. This cultural context will include the user's language, geographic location, and community. The user's language informs how to represent date and time information as text. The geographic location informs within which timezone to interpret a point in time. The user's community determines which calendar system is used to track years, months and days.

To summarise the characteristics of dates used by humans:

Specified by the user
Different levels of precision, depending on the use case
Interpreted according to the user's cultural context

Formatting system dates in APIs

JSON does not have a native date type, so APIs typically represent dates as either numbers or strings. This leads to a number of trade-offs. For system dates, the developer experience should be optimized for precision and correctness.

Approach 1: Use a numerical value to represent the elapsed time since an epoch date

System dates imply a scientific, linear view of time. Most programming languages internally represent dates as the number of seconds that have elapsed since that have elapsed since 00:00:00 UTC on 1 January 1970. This is known as Unix time and is a sensible choice for representing a date as a number.

You can see what today's date as a Unix timestamp looks like using unixtimestamp.com.

Timestamps in an API might look like this:

{
  "created_at": 1682648103
}

Advantages of this approach:

The value is opaque to the developer, discouraging them from performing error-prone date arithmetic.
Widespread support across programming languages to parse and serialize values.

A downside to this approach is that it is not readable by humans. Does 1682648103 refer to a date in the future or the past? This will make it hard for developers to debug and troubleshoot requests and responses that use this format.

Approach 2: Use the display string format

If we want to provide the developer using the API with more readability than a Unix timestamp, we can format the date as a string. A very naive approach would be to format dates the same way they would be displayed to the user.

This might look like:

{
  "created_at": "8/4/2023, 11:51:16 am"
}

You could describe this as the "quick and dirty" approach. But it is a bad approach for APIs:

It's ambiguous whether 8/4/2023 refers to the 8th of April or the 4th of August.
Not all users will expect the same date format depending on their locale.
It's unclear within what time zone the time should be interpreted.
A developer using this API would probably find this format unfamiliar.
System libraries probably lack out-of-the-box ways to parse and serialize dates in this format, placing more work on the developer.

This approach is mentioned for demonstration purposes, but do not design your API this way.

Approach 3: Use an internationally recognized standard

Fortunately, there is already an internationally adopted standard for formatting dates and times as strings: ISO 8601.

A timestamp formatted using ISO 8601 looks like this:

{
  "created_at": "2023-04-28T01:52:25Z"
}

Advantages of this format include:

No ambiguity between day-month or month-day ordering.
The timezone offset can be included or the value can be explicitly denoted as UTC.
It's readable by humans, aiding in development and troubleshooting.
Most language standard libraries provide ways to parse and serialize dates in this format.
Can easily be converted to the user's locale on the client.
Readable by developers during development or debugging.

The above example ends with the "Z" character, indicating that the value is in UTC time. Time zones can be hard for developers to reason about, so consider only returning values in UTC time so that different values can be easily compared with each other.

An alternative to the ISO 8601 standards is RFC 3339. The ISO 8601 standard allows for multiple variations in the format, whereas RFC 3339 is sometimes described as a "profile" of ISO 8601 that is more strict. In most scenarios, these standards are interchangeable and the differences are too nuanced to go into here.

Formatting dates used by humans in APIs

Computing has been around for less than a century, yet humans have been describing dates and times for millennia. So it's no surprise that trying to convert human dates to a computer representation is complicated!

When designing APIs to communicate dates, you need to consider how much of the cultural context is known upfront by the server, how much is specific to each client, and to what degree your API need to be able to convert information between different contexts.

Converting between different contexts, or even just supporting multiple contexts, might require you to keep and communicate a normalized representation through your API.

Approach 1: Use reduced precision formats of ISO 8601 where possible

The ISO 8601 standards allows for lower levels precision than what is needed for timestamps.

For example, this is what a date-only field would look like in an API:

{
  "membership_expiry": "2023-09-24"
}

Approach 2: Encode each date component as a separate field

Depending on the use case it may make more sense to design the API to match the user experience.

An example of this would be formatting date of birth fields as objects with separate subfields for day, month and year. This matches common UX patterns:

In an API this might look like:

{
  "date_of_birth": {
    "day": 3,
    "month": 7,
    "year": 1980
  }
}

This API design is assuming that all users have western concepts of birth dates and use the Gregorian calendar.

If this is not a fair assumption for your application, an advantage of the object format is that you can include a discriminator field, to inform how the date should be interpreted.

{
  "date_of_birth": {
    "calendar": "gregorian",
    "day": 3,
    "month": 7,
    "year": 1980
  }
}

As an example of a different cultural context, Thailand uses the Thai solar calendar, in which the current year at 543 years ahead of the Gregorian calendar. Furthermore, many people only have their year of birth listed on official records.

The date_of_birth field may look like this, with month and day becoming optional fields when calendar is equal to "buddhist".

{
  "date_of_birth": {
    "calendar": "buddhist",
    "year": 2523
  }
}

Document how to interpret dates that can be interpreted in multiple ways

Since non-system dates are often less precise than system dates, they can introduce subjectivity into your business logic that should clarify for your API's consumers.

For example, if a user's membership expires on 7 May 2023, does that mean it is still valid throughout the day on 7 May, or did it already expire the night before at 12:00 AM? In which timezone did it expire — the user's timezone or the system's timezone?

Regardless of what decisions your business logic makes, you should include those decisions in your API's documentation.

Should you include the timezone in the API?

Generally, interfaces don't display timezone information as it would be repetitive and cluttered. Nonetheless, the APIs powering such interfaces do need to consider timezone information in order for times to be displayed accurately.

Approach 1: Always normalize dates as UTC

Since end users don't consume APIs directly, it can make sense to transmit dates as UTC and rely on the application to convert them into the appropriate timezone when displaying them.

This is advantageous because the application is often better positioned to determine what the most appropriate time zone is. For example, it could be the user's timezone determined by the operating system, or it could be a user preference owned by the application. Many B2B apps even have a shared preference for the entire account so that dashboards and reports present the same information to all users.

The following API has a purchased_at field containing a UTC date, which the paplication can transform into an appropriate representation for display as needed.

{
  "purchased_at": "2023-03-24T11:00:10Z"
}

Approach 2: Include the timezone as a separate field

Normalizing dates to UTC works well for dates in the past. However, for scheduled events in the future, UTC is not a good fit since timezones and timezone offsets can change in the meantime and the resulting date may not be what the user expected.

Having a separate field for timezone also makes sense where it is important to display to the user. Examples include a calendar event for a virtual meeting where the attendees live in different countries or a flight itinerary where the arrival and departure airports are in different cities.

An API with a separate field for timezone may look like this:

{
  "departure_time": "2023-09-24T11:00:00",
  "departure_timezone": "Australia/Sydney"
}

It's best practice to format timezones as values from the IANA timezone database instead of reinventing the wheel. For example, Australia/Sydney. This ensures widespread compatibility with different systems.

Approach 3: Encode the timezone offset with the date field

If "Australia/Sydney" refers to a timezone, the timezone offset would be Australian Eastern Standard Time (AEST), or UTC+10:00. Alternatively, it may be Australian Daylight Saving Time (AEDT), which is UTC+11:00.

Since the timezone offset can change throughout the year for a given timezone, it is not appropriate to have a timezone offset as a standalone field. However, there are certain situations where it makes sense to encode the offset as part of the date field.

An example might be an album of sunset photos taken around the world. A user would expect each photo to be taken around the same time of day, i.e. dusk. It would be strange if some photos were displayed at 2 am due to timezone conversion. Here it doesn't make sense to normalize all photo timestamps to UTC, or even a single timezone. Yet it is also important to convey the precise time each photo was taken.

{
  "taken_at": "2023-01-020T18:30:01-05:00"
}

This format allows both the local time to be easily obtained and the absolute point in time to be calculated.

Approach 4: Separate fields for local time and UTC time

Often it can be simpler for the developer to have multiple fields available to them, for each specific use case:

{
    "taken_at_utc": "2023-01-020T18:30:01-05:00"
    "taken_at_local": "2023-01-020T23:30:01Z"
}

Naming date fields

In the above examples timestamp fields all end with the suffix _at. This is a good convention to adopt as it provides an additional signifier to the developer that the value of the field should be interpreted as a timestamp.

Other examples that do not follow this naming convention are not timestamps, and so need to be parsed differently.

Summary

Hopefully, this article illustrates that good design is always dependent on the use case.

That being said, there are some general principles that can be followed as a shortcut:

System dates should be formatted using the ISO 8601 standards, be in UTC time, and have the _at suffix.
Other dates should use the less precise formats from ISO 8601 unless the use case would benefit from a more specialized format.
Consider the timezone of the event and whether it needs to be communicated separately through the API.
Avoid assuming the user's cultural context. Instead, design APIs to be as inclusive as possible.

The best-designed APIs will be the result of a collaboration between product and engineering. A tool like Criteria can facilitate deep collaboration amongst a cross-functional team to produce world-class APIs.

How to write JSON Schemas for your API Part 1: validating data

James Moschou — Wed, 29 Mar 2023 00:17:24 +0000

All APIs should treat data coming from the client as untrusted until it can be validated.

By validating incoming data, you can ensure that it is safe, and can be interpreted and processed correctly by your server. Your validation rules, therefore, need to be communicated to the client via documentation so that they know how to format their data properly.

A common challenge many API teams face is keeping their documentation up to date as their API evolves. By using JSON schema as both the validation engine and the communication tool, the two always stay in sync since there is only one source of truth.

This series of articles will explain what each JSON Schema keyword means with realistic examples so that you can communicate as much information to your customers as possible.

Background on JSON

JSON is typically used to describe data as a collection of labelled fields or key-value pairs.

This form

May be represented as an object in JSON like this:

{
  "api_name": "Example API",
  "api_version": "1.0.0",
  "description": "An example API to demonstrate JSON Schema."
}

JSON can also describe a sequence of unlabelled values called an array.

[
  {
    "message": "This is an object value in an arary."
  },
  "This is a plain string value in an array."
]

JSON also refers to the data type of individual fields, that is strings (text), numbers (both whole numbers and fractional), boolean (yes/no) and empty values (null and undefined).

Objects and arrays are the compositional features of JSON. Objects can contain arrays and vice versa. This allows complex data structures to be composed of smaller structures.

General purpose validation keywords

type

JSON Schema answers the question: what type of JSON data do we expect?

We know that JSON data can be an object, array, string, number, boolean or empty value. This brings us to our first and most important validation keyword: type.

In most cases type literally says the type that we expect: "array", "boolean", "integer", "null", "number", "object", "string".

In practice, most JSON schemas that are used to validate requests and responses will specify that the payload must be an object. This is good practice from an API design point of view because it allows for the API to evolve incrementally by adding new fields without breaking existing clients.

This JSON Schema says that we always expect the JSON value to be an object.

{
  "type": "object"
}

If the data is allowed to take more than one form we can also specify more than one type in an array.

A use case for this might be to describe a configuration option. For example, the user could specify the name of each setting individually or specify a preset via its name:

POST /print_jobs

{
  "printer_settings": "black_and_white"
}

POST /print_jobs

{
  "printer_settings": {
    "paper_size": "A4",
    "color": true,
    "double_sided": true
  }
}

The JSON Schema for the value of printer_settings would look like this:

{
  "type": ["string", "object"]
}

enum

The printer example exposes a potential issue in the printer settings schema. What happens if the user specifies a paper size that doesn't exist?

We can describe all the possible values upfront using the enum keyword.

{
  "type": "string",
  "enum": ["A3", "A4", "A5", "US Legal", "US Letter"]
}

Usually enum is used with string values, which translates well to many programming languages. However, JSON Schema is more flexible and the enum keyword can be used with any type of value.

const

Occasionally your API may have fields that, when specified, always have the same value no matter what. This can be described using const.

{
  "type": "string",
  "const": "must be this value"
}

Using const is equivalent to using enum with only one value.

Why would you have this? A field where you already know the value doesn't convey any new salient information.

Constant values can be useful in requests where you want to the user to be explicit about the effect of the API call. They also allow you to expand the use cases of the field into an enum in the future.

For example, a POST /shareable_links API will make a resource available to anyone on the Internet. Calling this API would have drastic consequences if the the user never meant for everyone to be able to see the resource. Making the user explicit about the effect can reduce confusion and misuse of APIs.

POST /shareable_links

{
  "visibility": "public"
}

Later on, a "private" visibility option could be added to the API request in a backwards compatible manner.

Validating textual values

minLength and maxLength

If your API expects textual data, you can specify the minimum and maximum number of characters that are allowed.

Some APIs may restrict text to a maximum length based on what the underlying storage layer has the capacity for.

An example of a minimum length might be a phone number with a full area code where there is always a certain number of digits. If there are not enough digits, you know that the supplied value is not a valid phone number.

pattern

The phone number field example highlights another way we can check the integrity of the data. Phone numbers only contain digits and perhaps spaces or punctuation marks. We can forbid unallowed characters by using pattern, which is a regular expression.

Regular expressions are too big a topic to go into here. The following schema specifies that the string value must be a series of zero or more characters, where each character is a digit, space, parenthesis or dash.

{
  "pattern": "^[\\d ()-]*$"
}

Regular expressions can also go beyond describing what characters are allowed, and also specify what order they must appear in.

This example describes a US phone number formatted as (XXX) XXX-XXXX.

{
  "pattern": "^\\(\\d{3}\\) \\d{3}-\\d{4}$"
}

format

There are common use cases where the text value actually has a specific, well-known meaning such as an email address. We can use the format keyword to describe this.

{
  "type": "string",
  "format": "email"
}

This says that the text value must be an email address, not just any sequence of characters.

Validating numeric values

minimum, maximum, exclusiveMinimum and exclusiveMaximum

If a field can be set to a number, we can define the range that we expect the number to be within.

A very common use case is to disallow negative numbers:

{
  "type": "number",
  "minimum": 0
}

The keywords minimum, maximum, exclusiveMinimum and exclusiveMaximum corresponds to the mathematical inequalities >=, <=, > and < respectively. You can use these keywords in any combination, though typically you would either use the exclusive or non-exclusive one as needed.

multipleOf

You can specify that the number must be a multiple of something using multipleOf.

Say your Calendar API accepts a meeting duration in minutes, but only in 15-minute increments, you can use multipleOf to disallow 20-minute meetings. This means your API can avoid having an awkward number_of_quarter_hours field and instead have an easier-to-use minutes field that still respects the constraints of your application.

{
  "type": "integer",
  "multipleOf": 30
}

Validating arrays

items

When a field value is an array, you can specify the data structure of each item in the array using items.

Say your API allows adding multiple tags to an element, you can validate that each tag is a non-empty string.

{
  "type": "array",
  "items": {
    "type": "string",
    "minLength": 1
  }
}

uniqueItems

Continuing with the tags example, it doesn't make sense for an element to have two tags that are identical. We can use uniqueItems to disallow duplicate items in an array.

{
  "type": "array",
  "items": {
    "type": "string",
    "minLength": 1
  },
  "uniqueItems": true
}

minItems and maxItems

Just as we can limit the length of strings and numbers to a certain range, we can limit the number of items in an array to a certain range too.

Say in a survey builder we want to allow the creation of a new survey of questions, but each survey must have at least one question. We can disallow empty arrays by setting minItems to 1.

{
  "type": "array",
  "minItems": 1
}

Similarly, we can set an upper limit on the number of questions using maxItems.

prefixItems

Say we want to introduce an additional constraint on new surveys, where the first few questions are always the same, but the rest of the questions can vary.

We can use prefixItems to specify a sequence of schemas that the first items in an array must satisfy.

This example requires that all surveys ask for name and address up front.

{
  "type": "array",
  "prefixItems": [
    {
      "type": "object",
      "properties": {
        "question": {
          "type": "string",
          "const": "What is your name?"
        }
      }
    },
    {
      "type": "object",
      "properties": {
        "question": {
          "type": "string",
          "const": "What is your address?"
        }
      }
    }
  ],
  "items": {
    "type": "object",
    "properties": {
      "question": {
        "type": "string"
      }
    }
  }
}

contains

We can use the contains keyword to specify that at least one item in the array must satisfy an additional constraint.

For example, we can specify that a team that consists of team members must contain at least one member who is an admin.

{
  "type": "array",
  "items": {
    "type": "object",
    "properties": {
      "name": {
        "type": "string"
      },
      "isAdmin": {
        "type": "boolean"
      }
    }
  },
  "contains": {
    "type": "object",
    "properties": {
      "isAdmin": {
        "type": "boolean",
        "const": true
      }
    }
  }
}

maxContains and minContains

We can also specify the number of times the contains keyword would match using maxContains and minContains.

For example, if a team must have exactly one admin user, we can specify this.

{
  "type": "array",
  "items": {
    "type": "object",
    "properties": {
      "name": {
        "type": "string"
      },
      "isAdmin": {
        "type": "boolean"
      }
    }
  },
  "contains": {
    "type": "object",
    "properties": {
      "isAdmin": {
        "type": "boolean",
        "const": true
      }
    }
  },
  "minContains": 1,
  "maxContains": 1
}

unevaluatedItems

The unevaluatedItems keyword of JSON Schema is used to define a schema that applies to items in an array that do not match any of the previous schema definitions.

A very simple use-case of this keyword is to disallow any unexpected values.

This schema describes an array with exactly two string values and nothing else.

{
  "type": "array",
  "prefixItems": [{ "type": "string" }, { "type": "string" }],
  "unevaluatedItems": false
}

You typically wouldn't use items and unevaluatedItems together, since items will evaluate every value in the array.

Validating objects

A JSON object is made up of pairs of keys and values. Each pair is called a property, which can be separately validated using nested schemas.

properties

Validate each value in an object using properties.

This example describes a calendar event structure with a name, start time and end time:

{
  "type": "object",
  "properties": {
    "name": {
      "type": "string"
    },
    "start_time": {
      "type": "string",
      "format": "date"
    },
    "end_time": {
      "type": "string",
      "format": "date"
    }
  }
}

required

By default, all properties in an object are treated as optional. To make some properties mandatory, use required.

Let's update the calendar event structure to add an optional location property, while making the other properties mandatory.

{
  "type": "object",
  "required": ["name", "start_time", "end_time"],
  "properties": {
    "name": {
      "type": "string"
    },
    "start_time": {
      "type": "string",
      "format": "date"
    },
    "end_time": {
      "type": "string",
      "format": "date"
    },
    "location": {
      "type": "string"
    }
  }
}

dependentRequired

Whether a property is required or not is not cannot always be decided upfront. The dependentRequired keywords allows you to specify scenarios where a propery becomes mandatory if another optional property is also included.

For example, you may have a Food Delivery API that allows specifying a delivery address when making a food delivery order. If an address is not specified it is a pick-up order. If the delivery address is specified, then an additional property specifying whether the courier has the authority to leave the delivery in a safe place must also be specified.

{
  "type": "object",
  "dependentRequired": {
    "delivery_address": ["authority_to_leave"]
  },
  "properties": {
    "delivery_address": {
      "type": "object"
    },
    "authority_to_leave": {
      "type": "boolean"
    }
  }
}

additionalProperties

Sometimes your data structures require more flexibility than what is normally possible by specifying fixed fields upfront.

The additionalProperties keyword allows you to validate the values properties where you don't know the names upfront. This is useful to allow consumers of your API to define their own properties, such as with a custom metadata field.

This example lets clients include custom fields in a metadata object, but only for simple values like strings, numbers and booleans.

{
  "type": "object",
  "properties": {
    "metadata": {
      "type": "object",
      "additionalProperties": {
        "type": ["string", "number", "boolean"]
      }
    }
  }
}

patternProperties

While you don't always know the property names upfront, as with the custom metadata field, you can still constrain them in some way.

This is especially useful to group custom properties in a certain "namespace" to maintain forwards compatiblity with future versions.

This example allows additional custom properties on a contact record, but requires them to start with the user_defined_ prefix:

{
  "type": "object",
  "properties": {
    "name": {
      "type": "string"
    },
    "email": {
      "type": "string",
      "format": "email"
    }
  },
  "patternProperties": {
    "^user_defined_": {}
  }
}

propertyNames

If you just want to validate the names of properties, but allow any values, you can use propertyNames.

A very common use case is to only allow property names made up of certain characters, excluding spaces, punctuation and other non-alphanumeric characters.

{
  "type": "object",
  "propertyNames": {
    "pattern": "^[A-Za-z_][a-za-z0-9_]*$"
  }
}

minProperties and maxProperties

For flexible data structures where you don't know all the properties up front, you can limit the number of custom properties to a certain range using minProperties and maxProperties.

This example describes a color pallete structure that contains between 3 and 5 color objects, keyed by their hex code.

{
  "type": "object",
  "patternProperties": {
    "#[0-9a-f]{6}": {
      "type": "object",
      "properties": {
        "name": "string"
      }
    }
  },
  "minProperties": 3,
  "maxProperties": 5
}

unevaluatedProperties

A JSON Schema is a system of constraints. Everything is allowed by default, unless you explicitly disallow it.

You can specify a minimum set of constraints that all properties must satisfy, regardless of whether they are known or unknown properties, using unevaluatedProperties.

Of course, if you simply want to disallow unknown properties, set unevaluatedProperties to false.

{
  "type": "object",
  "properties": {
    "name": {
      "type": "string"
    }
  },
  "unevaluatedProperties": false
}

Validating binary data

contentMediaType

Sometimes the data your API works with cannot be expressed by JSON. However, to be consistent with the rest of the API, it may be encoded as a string and wrapped inside an outer JSON structure.

You can use contentMediaType to specify that a string value should be interpreted as different type of data.

For example, you may have a Content Management System that returns a post's content as HTML that can be rendered directly on the client's website.

The JSON schema might look like this:

{
  "type": "object",
  "properties": {
    "post_slug": {
      "type": "string"
    },
    "post_content": {
      "type": "string",
      "contentMediaType": "text/html"
    }
  }
}

contentEncoding

With the embedded HTML example, the HTML text could be represented as a string using the same encoding as the enclosing JSON document, which would typically be UTF-8.

For other mime types, e.g. images, the binary data must be converted to a string representation using a different encoding, which can be specified using contentEncoding.

The following schema indicates that a string contains a PNG image, encoded using Base64:

{
  "type": "string",
  "contentEncoding": "base64",
  "contentMediaType": "image/png"
}

contentSchema

There are some scenarios where a data structure can be expressed as a JSON data structure, but is nonetheless transmitted as an encoded string.

An example is JSON Web Tokens, which are used to store authentication and authorization information about a user. Different JWTs can make different claims about the user, so we can require certain claims to be present using contentSchema.

This example describes a JWT that requires the issuer (iss) and expiration time (exp) fields in its claim set.

{
  "type": "string",
  "contentMediaType": "application/jwt",
  "contentSchema": {
    "type": "array",
    "minItems": 2,
    "prefixItems": [
      {
        "const": {
          "typ": "JWT",
          "alg": "HS256"
        }
      },
      {
        "type": "object",
        "required": ["iss", "exp"],
        "properties": {
          "iss": { "type": "string" },
          "exp": { "type": "integer" }
        }
      }
    ]
  }
}

Conclusion

We've seen how JSON Schema can be used to specify and communicate validation rules for JSON data, including text, numbers, arrays, objects and even binary data.

Future articles in this series will cover how you can use JSON Schema to include human-readable metadata for documentation purposes, specify dynamic data structures that vary based on certain conditions, and organize schemas into a library of reusable elements.

Getting video to autoplay using Next.js and Tailwind

James Moschou — Tue, 12 Oct 2021 07:21:08 +0000

Recently I added a demo video to the homepage of https://criteria.sh.

In this post I'll explain why React's asynchronous layout makes it a bit more difficult to get autoplay working, and how to modify the Dialog component from Headless UI to get autoplay working in React.

The requirements are:

The video should start hidden.
Upon clicking the call-to-action, the video should open in a fullscreen dialog.
When the dialog appears, the video should start playing automatically with sound.

Implementing the dialog

I couldn't find a lightweight lightbox component that I liked and leveraged Tailwind CSS. So I ended up using this modal and modified it to be fullscreen.

The relevant code is below. The <Dialog /> component implements the show/hide functionality and is wrapped inside a <Transition.Root /> component, which provides the animation.

import { Fragment, useState } from 'react'
import { Dialog, Transition } from '@headlessui/react'

export default function VideoPlayerDialog() {
  const [open, setOpen] = useState(true)

  return (
    <Transition.Root show={open} as={Fragment}>
      <Dialog as="div" onClose={setOpen}>
        {/* ... */}
      </Dialog>
    </Transition.Root>
  )
}

Hosting the video

I haven't worked much with video on the web and my first instinct was to commit the video file directly to the Git repository!

I wanted to do better and after some research discovered Mux. I liked their developer-centric model and their pricing plan comes with $20 free credit. Considering I only have one video, it was effectively free for me.

Mux provides a guide for integrating it with a React app here. They recommend not to use the autoplay attribute, and instead call video.play(). This is a side-effect, so naturally I called it from within an effect.

import { useRef, useEffect } from 'react'

export default function VideoPlayer() {
  const videoRef = useRef(null)

  useEffect(() => {
    if (videoRef && videoRef.current) {
      videoRef.current.play()
    }
  }, [videoRef])

  return (
    <video
      controls
      ref={videoRef}
      style={{ width: "100%", maxWidth: "500px" }}
    />
  )
}

When this component is rendered React will execute the effect, which will play the video. Right? Wrong.

Safari video policies

In Safari I got the following error:

NotAllowedError: The request is not allowed by the user agent or the platform in the current context, possibly because the user denied permission.

The reason is that Safari prevent websites from playing videos, especially ones with sound, without the user's consent. The way browers infer this consent is whether the code executed as a result of a user gesture. For example, if the code executed inside a click handler then that would indicate the user probably clicked a Play button.

This policy prevents websites from playing unwanted media. But in this case the user explicitly clicks a call-to-action to play the video.

Understanding asynchronous rendering in React

When you make changes to React's state, using setState() or the setter returned from useState(), React may batches these changes into one update operation in order to optimise performance. This means that effect code that runs after a DOM update may not run in the same context as the code that originally changed the state.

You can see this in action using some logging:

<button
  onClick={() => {
    console.log('Begin click handler')
    setOpen(true)
    console.log('End click handler')
  }}
>

If rendering was synchronous, then you would expect the following code to execute between the begin and end markers.

  useEffect(() => {
    console.log('Playing video from useEffect')
    if (videoRef && videoRef.current) {
      videoRef.current.play()
    }
  })

Instead this is what gets logged:

Begin click handler
End click handler
Playing video from useEffect

This indicated that useEffect is being called asynchronously after the click handler, not within it.

Modifying the Dialog component

After much experimenting I discovered that if I modified how the dialog was shown and hidden I could get it to work. Specifically, I changed the unmount prop to false on the <Transition.Root /> component:

import { Fragment, useState } from 'react'
import { Dialog, Transition } from '@headlessui/react'

export default function VideoPlayerDialog() {
  const [open, setOpen] = useState(true)

  return (
    <Transition.Root show={open} as={Fragment} unmount={false}>
      <Dialog as="div" open={open} onClose={setOpen}>
        {/* ... */}
        <VideoPlayer shouldPlay={open} />
        {/* ... */}
      </Dialog>
    </Transition.Root>
  )
}

This causes the dialog, and hence the video element, to remain in the DOM even when it is not visible.

I also added back the effect to start playing the video:

export default function VideoPlayer ({ shouldPlay }) {
  const videoRef = useRef(null)

  useEffect(() => {
    if (shouldPlay && videoRef.current) {
      videoRef.current.play()
    }
  })

  return (
    <video
      controls
      ref={videoRef}
    />
  )
}

Since the video is no longer removed from the DOM completely, it would sometimes start playing with sound in the background when you navigate away from the homepage then return to it. The shouldPlay prop prevents that from happening.

Conclusion

Video on the web is incredibly finicky. I don't know why this works, but it works.

Naming in Swift Part 3: Boundary Words

James Moschou — Mon, 26 Jul 2021 06:00:56 +0000

With the introduction of async/await to Swift, this article may have already aged poorly.
In fact I wrote the first draft of this for Objective-C, before Swift even existed!

There are some words that appear in pairs through Swift APIs:

start and stop
start and finish
begin and end

I am calling these boundary words, as they delimit the boundarys of certain actions, events or processes.

Although theses words are considered synonyms of each other in plain English,
they take on distinct semantics and should no longer be considered synonyms in Swift.

Furthermore, due to this specialisation, there are certain pairings that make sense and some pairings that you don't see as often.
For example, you typically won't see start() paired with end(), nor begin() with finish() even though these constructs would be perfectly fine in ordinary English.

About start and stop

APIs use start and stop to convey a long-running task that is happening in human-perceivable time.

Here are some examples:

URLProtocol has startLoading() and stopLoading()
AVCaptureSession has startRunning() and stopRunning()
UIImageView has startAnimating() and stopAnimating()

Things that can be stopped can usually be started again. This means the object will have state.
Where you see startPerformingTask and stopPerformingTask you often also see isPerformingTask.

About finish

Sometimes start is paired with finish.

Here are some examples:

AVAssetWriter has startWriting() and finishWriting(completionHandler:)
PHContentEditingController has startContentEditing(…) and finishContentEditing(completionHandler:)

Unlike stop, finish is not simply the opposite of start.
It specifically denotes that there is some sort of final phase to the task.
For example, flushing the contents of a buffer, writing to the disk, or cleaning up resources.

About complete

You see APIs with complete when they have completion handlers.
The concept of completion specifically denotes the boundary of the task between the caller and callee.

Here are some examples with both finish and complete:

AVAssetWriter.finishWriting(completionHandler:)
HKWorkoutBuilder.finishWorkout(completion:)

In these APIs, finish is refering to the final phase of the task that needs to be performed before control is transferred back to the caller in the completion handler.

About begin and end

These pairs normally appear within the same lexical code block, and are used to group operations into batches, transactions or other constructs.

Here are some examples:

UITableView has beginUpdates and endUpdates
MTLCaptureScope has begin and end
OSSignposter has beginInterval and endInterval

These APIs were much more common in Objective-C prior to the introduction of block syntax.
In Swift these would be good candidates to improve using closures.

Counter-examples

When writing this I did find a lot of examples in Apple's APIs that contradicted my definitions above.

For one, in the Swift Standard Library, collections have a start index and end index.

These definitions aren't hard rules, but I believe they are more consistent than not and that makes them worthwhile to know.
Many design patterns that have corresponding vocabulary already defined.
If possible, why not use a common vocabulary — it will only make your APIs feel more at home.

Naming in Swift Part 2: Name, Title, Identifier & ID

James Moschou — Fri, 09 Jul 2021 07:24:19 +0000

Let’s start this series with a meta discussion about how to name properties that themselves are used to name, describe or otherwise identify things.
On Apple platforms, words used for these purposes are name, title, identifier and more recently id.

About name

Lots of things have names, especially things that are displayed in lists and other collections.
It is incredibly common for developers to give such types a string property called name, which gets displayed in their user interface.
This seems sensible at first, but it is actually a wrong usage of the word if you accept that words in APIs take on more precise meanings than they do in ordinary English.

When writing Swift APIs, calling a property name carries the implication that its value will be unique within some namespace.

Here are some examples:

Notification.name
UIFont.fontName
NSManagedObjectModel.entitiesByName

When you see a type that has a name property, you will typically also see a way to look up values of that type via its name.
(Incidentally the convention is to name the argument label [type]Named: rather than [type]WithName:.)

This requirement that properties called name have unique values is what makes them unsuitable for storing user-facing text.
Users can't guarantee uniqueness when entering a value unless the interface also performs validation and rejects non-unique values.
But this would be a poorer experience than if the user could simply enter any value they want.
(File systems come to mind as one of the few scenarios where this does happen.)
To get around this you typically model types as having both a user-facing text property and an internal identifier property.
These properties should be called title and identifier respectively.

About title

Title is used for user-facing text that distinguishes an item from others.

Here are some examples:

UIMenuItem.title
UIViewController.title
NSUserActivity.title

Sometimes title might seem a overly formal or plain silly,
especially when you are inclined to say name in ordinary English.
But for non-unique, user-facing text, title is correct.
Even if the corresponding field has a prompt that says something like "give your new BLANK a name",
title would still be the correct word to use for the property!

When you see a property called title you often also see a property called localizedDescription next to it.
Both are used for describing (as opposed to identifing).
The difference between the two is in their usage — think short headings versus long sentences.

About identifier

Properties that contain unique identifiers are called, well, identifier.

Here are some examples:

CNContact.identifier
EKEvent.eventIdentifier
URLSessionConfiguration.identifier

Both name and identifier imply uniqueness; values can be used as keys for lookup.
The difference is that properties called name are typically human readable (if only for development and debugging purposes),
whereas properties called identifier are typically opaque and generated randomly.

About id

Swift 5.1 introduced the Identifiable protocol to the standard library.
Here you can see how the Swift programming language allows APIs to become more concise, or as some would say, Swifty.

APIs should be designed with the reader in mind. Abbreviations and initialisations should usually be avoided.
ID is a fairly common initialisation and you can argue that shortening identifier to ID is sensible.
Doing so is much less problematic in Swift than in, say, Objective-C because the var id: ID property is statically associated with the Identifiable protocol, which provides additional semantic context.

There are some APIs written in Objective-C that do use id instead of identifier, for example NSManagedObject.objectID
and CKRecord.recordID. Notice that the types of these properties are not NSString but the more semantic NSManagedObjectID and CKRecordID classes respectively.

Oddballs

If you scour the frameworks that belong to the iOS and macOS SDKs you will see that these definitions mostly hold up.
That said, there are always exceptions. Properties called localizedName have semantics closer to title than name. In the new MusicKit framework, the Playlist type has a property called name, but it seems like it should have been called title. But please don't let a few inconsistences discourage you from designing your own APIs rigorously.

Naming in Swift Part 1

James Moschou — Fri, 09 Jul 2021 07:22:49 +0000

While there is lots of writing on programming style, syntax and formatting, I've seen very little centred on vocabulary.
Most writing that I've come across tends to focus on vague outcomes like "make names clear and consise" without actually telling you what words to use.

I originally meant to write this series for Objective-C before Swift came onto the scene.
Nonetheless I think this will still be relevant. After all, Swift was designed to be interoperable with Objective-C and owes it a lot of its design.

Like human languages, programming languages have their own etymology.
Swift's etymological roots can be found in the Cocoa and Cocoa Touch APIs written in Objective-C.
While the vocabulary comes from Foundation and related frameworks, I would argue that the development of this vocabulary was influenced by a feature of the language, specifically message passing.

The influence of message passing

Message passing, being the method of communication between objects in Objective-C, actually comes from Smalltalk.

In languages that use message passing, the message is a top-level concept. In Objective-C, a message is called a selector and can be stored in a variable of type SEL.
Message passing can be contrasted with method calling, where methods are bound to a type and must be called on an instance of that type.

In Objective-C a selector can be sent to any object, regardless of the object's type.
Objects need to be prepared to receive any selector, even if they don't know how to respond to it.
This behaviour is on full display in the application responder chain where objects may be asked to handle the cut:, copy:, paste: and other actions.

Since objects need to be prepared to handle any selector, you could argue that selectors have a greater burden to convey their semantics than methods.
Selectors do not have the benefit of being bound to a type that can provide additional semantic context. The name of a selector must be unambigious and make sense on its own.
I believe these constraints had a large influence on the verbosity that Objective-C became known for.

I like to use an analogy with organic chemistry. Acetic acid and ethanoic acid both refer to the same chemical.
But only "ethanoic acid" unambigously descibes the chemical's molecular structure using the IUPAC systematic naming conventions.
Names in Objectice-C work in a similar way in that their verbosity encodes important semantic information.
In this sense I would describe Objective-C APIs not as verbose but as precise.

Swift vocabulary today

Programming is a technial profession, and like most technical professions it has specialised language and jargon.
While every profession comes with its own jargon, what is unique to programming is that the creation of new language isn't
something that's confined to acedemia and research — it is literally our everyday jobs.

Programming concepts really only exist in our heads, so giving something a name is what makes it real and understandable.
For something so integral to our profession it is a shame that we don't give much guidance beyond an acknowledgment that naming is hard.

Static typing, generics and type inference enable Swift APIs to be much more concise than Objective-C without losing semantics.
Yet the importance of getting the name right isn't diminished and much of the vocabulary from Objective-C remains relevant today.

This series will focus on the precise meanings of words that are treated as synonyms in ordinary English but have distinct meanings when used in Swift APIs.